On Methodical Problem Solving

Seeing the sky through the trees

Co-workers who have solved problems with me over any period of time have heard me talk about “building the matrix”. Sometimes its an actual table on a whiteboard (virtual or on the wall) while other times it’s in my head or in a notebook, but I’m almost always building a matrix when I’m working a problem.

And when I encounter someone working a problem and see them flailing around, I encourage them to slow down and develop a matrix, as a way of moving forward.

What’s in the Matrix?

The matrix can be as simple as a list of steps tried and their results. Or it can be more, tracking fields such as the state of different components during that test, any hypothesis and a place for tracking results.

When the team gets going too fast and not in a forward direction, I like to focus the process by asking a few questions:

  1. What part(s) of the system do these actions test?
  2. Have any variables changed?
  3. What do we expect will happen?
  4. What will the results mean?
  5. What will we do next?

Let’s break this down some:

1. What part(s) of the system do these actions test?
Many of our systems today are large and complex and it’s not certain where the problem lies. Thinking about and answering this question can help develop a good test.

2. Have any variables changed?
It does us little good if the ground is shifting under our feet without us being aware of it. We can’t always be certain of every single variable to track, but with practice, we can be aware of many of the significant ones.
This can be tricky, however, because some unobvious things like the day of the week or the phase of the moon can be important variables to a process.
This is also important for fulfilling your change management obligations.

3. What do we expect will happen?
Stopping to answer this helps insure that we understand the system that we’re testing, but it’s sometimes true that we won’t know the answer.
It’s often only important to stop long enough to think about the question, even if we don’t answer it. It’s my opinion that even when we don’t come to an answer, the subconscious begins working on it and may contribute to an intuitive leap forward (or sideways!) during future steps.

4. What will the results mean?
It’s OK if we don’t know this answer (yet). If we aren’t able to answer this question at all, it may be a symptom that we’re still flailing and not moving forward. Not knowing the answer may also mean that the test is meaningless and should be skipped.
But it’s a good question to ponder after the test is performed. Where do these results point us? What do they mean? What are the implications of a positive result or a negative one?

5. What will we do next?
I don’t always insist on this one—it can be useful in developing the larger test plan, but isn’t absolutely needed.

There’s a sixth question that can be asked at any point along this path: Why? When asked and considered from time to time, this one little word can help promote understanding in ways that many collections of large ones can’t.

Wrapping up

Move forward. I don’t encourage stopping everything until a full and complete matrix is developed. We should know our systems sufficiently that we can start early, go hunting in the right “direction” and get results.

Communicate. One benefit of a shared, virtual matrix is that multiple groups can work some tests without stepping on others’ toes. But communication is key here (where is it not?) in making certain that unexpected variables aren’t being changed.

Allow for intuition. This one topic may require a complete separate post. Allow for intuition. It may be that someone on your team can “see” through opaque parts of the system and has a hunch for what’s wrong. Let that guide you in developing a test. You may end up skipping over several unproductive tests and land right at the important one.
This sort of problem solving can bring difficulties into a group, however. Frequently the one with the intuition can have difficulties expressing their thought process and their actions may seem haphazard. You, as the leader, can help facilitate this if it does.

Use the results. Any event like this can benefit from an after-action-report (whatever your group calls it) and the team should certainly use the steps and results to continuously improve your processes and your product.

Keep the results. Don’t throw away the matrix or the test steps. Incorporate them into your knowledge base and your documentation. You may find that this jump-starts the process the next time something goes “bump” unexpectedly.

Posted in Management, System Administration | Tagged , | Comments Off on On Methodical Problem Solving

Case Study: Security Controls in small, non-traditional Organizations

Several unfortunate events were used together to defraud a 501(c)(3) of nearly all their operating funds during a transition from one managing group to another, highlighting the needs for stronger controls in a wider spectrum of organizations than usually considered.

The Scenario

The attacker gained control of a person’s email account at a large consulting firm, presumably because the company’s high-profile name and contracts made them a more important target for attackers. The employee using the compromised account was using the company account for personal purposes, one of which was as a team member working to transition an unrelated 501(c)(3) from one managing group to another.

The attacker monitored this individual’s email communication and began to piece together the intended transition plan. When the outgoing head of the 501(c)(3) communicated to the employee (and the attacker) that she was going on an extended and remote vacation, the attacker put their plan into motion.

The outgoing head was using a common public email platform and the attacker created a similarly-spelled email address that was then used to spoof the outgoing head.

The spoofed email address was used to create confusion around the timing of the funds transfer and the attacker used this supposed “change in plans” and their control of both the spoofed email address and the team member’s compromised corporate email address to misdirect the funds from the intended account to the attacker’s.

The intended recipients, unaware of the change in plans, didn’t know the funds were arriving early and so didn’t miss them when they didn’t arrive. By the time the senders started asking questions and the outgoing head returned from vacation, the funds had been transferred from the attacker’s account to other accounts and not even the FBI was able to recover the misdirected funds.

The leaders of the 501(c)(3) disclosed the fraud to their supporters and were able to raise almost twice the amount lost, ensuring the continued operation of their charity for another year.

The attackers, meanwhile, left us with several valuable lessons that we can benefit from.

Lessons Learned

Even in a small non-profit organization run by part-time people, a few extra security controls might have led to a completely different outcome.

  1. Require that personal email be used for personal purposes.
    The large consulting firm had a high-profile name and contracts, making them a big target for the attackers. It’s doubtful that their purpose was to defraud the 501(c)(3) when they compromised the corporate email account—but the attackers took advantage of the situation when it presented itself.
  2. Require standard email account security measures.
    Use two-factor authentication and unique passwords passphrases that are hard-to-guess and easy-to-remember.
  3. Verbally confirm both payment amounts and destinations.
    Implement a dual-control policy where a wire-transfer request won’t be initiated without verbal confirmation from another person, while ensuring that the parties are talking to the right persons.

Additionally, employing some best-practice security tips would have helped.

  • Pay attention to email addresses—sometimes the changes can be subtle—and spoofing an address or even a domain is quite easy.
  • Be aware of changes in grammar, spelling, word usage and even formatting from frequent correspondents. When looking back, the spoofed emails had tell-tale signs that they weren’t from the outgoing head.


The funds your organization deals with may not have many digits, but even if you’re not a large target, you can still become collateral damage in someone else’s attack. Without building a full security department or adopting onerous and complicated security policies, your organization’s security posture can benefit from periodic risk analysis and the implementation of the right controls to address the concerns.

Posted in Other Thoughts | Tagged , | Comments Off on Case Study: Security Controls in small, non-traditional Organizations

Off the Shelf: June 2021

Most of June’s reading was for fun, but each stands out for something unique.

I first put I sing the body electric! on my e-reader a few months ago because I knew I’d soon be needing a collection of short stories, things a little more self-contained than a longer novel, and Bradbury delivers.
“Night Call, Collect” may be my favorite of the collection — a man stranded on Mars attempts to keep his future self company through hundreds of electronic recordings that call him and interact with him years later, but ends up driving himself nearly (completely?) mad. A thoughtful story that made me stop and muse about some things for a few days after finishing.
“Tomorrow’s Child” explores the love of a father and the love of a mother for their unique child, born into a different dimension from the parents. After a year of trying to bring the child into the parents’ dimension, the scientists announce they can transport the parents to their child’s dimension, but cannot go the other way. Faced with the options of being united with their child (and isolated from the regular world) or of staying estranged from their child (but living in the regular world), they make a choice.

Continue reading

Posted in Other Thoughts | Tagged | Comments Off on Off the Shelf: June 2021

Off the Shelf: May 2021

Looking back at the month, I finished more books in May than I had thought, including two that I’ve been working on for a long time and two that I rushed since the Library wanted them back sooner rather than later.

I first heard about Paper Trails (Blevins, 2021) from a WSJ book review that caught my eye in March of this year. I’ve long been interested in the visual representation of data (see Tufte’s works) and here was an entire book telling a different perspective on the story of “how the west was won” through the dry data of when and where US Post Offices were opened in the 1800s. My local library didn’t have it yet, but I put a hold on it and it arrived quickly (and they wanted it back rather soon, too). Blevins takes a potentially dry subject and breathes life into it, resulting in a fascinating story.

Continue reading

Posted in Other Thoughts | Tagged | Comments Off on Off the Shelf: May 2021

Broken Everywhere

I catch some good-natured and well-deserved ribbing from my co-workers and colleagues about this precept, but frankly, I take it in stride. It’s true that sometimes the new hires don’t understand it right away, but once explained and demonstrated, I don’t generally get much push-back.

Stated simply, “Broken Everywhere” means:

A single device or installation should never stop working as a result of a change to (or near) a similar device or installation somewhere else. Continue reading

Posted in System Administration | 1 Comment

Ban “Interesting”

Let’s ban the word “interesting” (at least for a while). Think about it for a moment. When was the last time you used the word when it actually[1. Let’s ban “actually” next.] meant something?

“What did you think about the incident management presentation?”
“Oh, it was interesting.”

Does that tell you anything? It’s a cop-out answer, a filler word Continue reading

Posted in Other Thoughts | Tagged , | Comments Off on Ban “Interesting”

Never too early—rarely too often

We’ve got a rather large project going on at the moment and a shrinking window to get it done. The deeper we dig into this mound of work, the more and varied things we pull out. What started as a “you’re almost there already” project is now starting to look rather daunting.

My strategy? Limit the scope and focus solely on that scope. If it’s not in that scope, it gets left out and doesn’t get done. Continue reading

Posted in Leadership | Tagged | Comments Off on Never too early—rarely too often

A Manager’s Smile


We moved offices this past week, relocating the IT department (and one other) into a new space. The new space is light and airy and is laid out on the open office plan.[1. I believe that the only fans of the open-plan offices can be found in the workplace design profession. However, they simply cannot back up their claims of productivity increases. As Tom DeMarco and Tim Lister wrote many years ago, “The only method we have ever seen used to confirm claims that the open plan improves productivity is proof by repeated assertion.” (emphasis in the original)

DeMarco, Tom, and Timothy R. Lister. Peopleware: Productive Projects and Teams. Addison-Wesley, 1987, p 53.] Initially we were skeptical but we’re making the best of it and it appears most everyone is settling down and adjusting well.

We have two new conference rooms, a large dedicated IT workroom, lots of community (but very little individual) storage, an awesome kitchen and a very welcoming lunch space. Continue reading

Posted in Other Thoughts | Tagged , | Comments Off on A Manager’s Smile

5 Reasons Why Not to Use Your Work Computer to Do Your Personal Taxes

  1. You don’t have local admin.
    TurboTax requires[1. See “Administrator Rights In Windows 7 And Vista – Turbotax Support“. Web. 8 Feb. 2017.] it and you probably don’t have it, nor should you.
    Admins take away local admin because users frequently don’t look before clicking and because users install things they shouldn’t, possibly introducing malware or putting their organizations at risk of license violations.
  2. You just saved, on a device you don’t own, your sensitive personal information.
    Enough of your (and your family’s) sensitive personal information[2. See  NIST’s “Guide to Protecting the Confidentiality of Personally Identifiable Information (PII)” (pdf). Web. 8 Feb. 2017.] that even a mildly competent criminal would find it easy to exploit. And you saved it for them all in one spot. With pointers to it.
    There are methods to encrypt this file and make it harder for them to exploit it. But it’s still possible.
    Oh, and that PDF you saved, just in case? It has everything the criminal needs, too. Continue reading
Posted in Other Thoughts, System Administration | Comments Off on 5 Reasons Why Not to Use Your Work Computer to Do Your Personal Taxes

Know when to STOP

After a frustrating day at work dealing with a repeat of a problem from this past spring, I related some of the situation to my crew of hecklers advisers at the dinner table:

Stop Sign in Australia

By Matthew Paul Argall [CC0]

A software package common to all our endpoints has a third-party plugin that only half of our users utilize. The package incorporates some templates that—in our situation—were created in a different version of the package on quite a different platform. The package and plugin relationship is complicated: only a particular version of the plugin will work with any given version of the package.

Of the users that utilize the plugin, only a few are in the same workgroup with another user. And, of course, there’s not a user group or a communication mechanism. The only common point of contact is the workgroup that takes the output from this package and does some massaging, formatting and editing before sending back to the users.

The IT workers on my team started with each user of the software package and plugin—and blaming the templates. Or the packages. Or the plugins. (Or the users!) Very soon we had a mashup of instructions with every known mixture of package/plugin/installation possible. I was hearing things like “UserA says that it’s the templates” and “UserB says when they reboot it works” and “UserC deleted their cache, reinstalled the plugin and now it works”.

Whatever. Sure, keep developing individual solutions for each of your installations. How’s that working for you?

Continue reading

Posted in Leadership, System Administration | Comments Off on Know when to STOP