Troubleshooting 101

Get used to the idea that you won't know what's going on. This is a key theme to IT; not only when you're starting out, but even after you've been around for a few years (or decades). Even those of us who've been doing it for 20+ years still run into quite a few WTF moments on a regular basis as the technology constantly changes and vendor support isn't always reliable (and is occasionally even more clueless than you are, about their own product). The skills we've learned and are constantly refining are those skills that you might already know (or at least can learn very quickly).

## WTF is Happening?

Ask questions, get details, then research (often through Google). Researching will lead to new questions, then new details, new research, etc., and then, hopefully, a solution.

The job of IT is not to know the answer. The job of IT is to know where to find the answer.

LEARN THIS. MASTER THIS.

Here's how.

## Ask these questions:

* What is happening?
* Who is it happening to?
* Where is it happening?
* When did it start happening?

### What is Happening?

There is nothing more useless than hearing, "My computer broke." Okay; what broke? What is the user seeing? Is there power to the computer? Is there an error message on screen? Is the computer or monitor plugged in? Is it possibly a logon issue, or an application issue? Perhaps the user can log on Windows, but launching a software program results in a blue screen or an error message like, "Unknown user name or bad password." Try to gather as much detail as possible about the exact issue. Pay special attention to any error messages you find, and try to understand the context. The "Unknown user name or bad password" message, for example, is a pretty common message used by a lot of different programs, as well as Windows, so it may refer to the user's domain account or to a specific program that they are trying to run.

### Who is it Happening to?

This will help narrow down your search and potentially point to a solution. Suppose Bob in Accounting calls to say he's unable to log on the payroll system. If possible, you should ask if anyone else is having the same issue. If Sally, Mark, and Debbie in Accounting are all able to log on to the payroll system, the problem is very likely with Bob, his user account, his computer, his network connection, or something similar. If everyone in Accounting is unable to log on to the payroll system, the problem is likely to be with the payroll system, the network, the Internet (if the payroll system is cloud-based; more on this later), or something similar.

#### Find the Single Point of Failure.

What is the common element that is shared among all users having a particular issue? This will often point you to the solution. For example:

* If one user is having problems with their email, but no one else is, it's not the email system.
* If the user is having problems with their email when they launch Outlook, but they can access email when they log on to the email system's website (i.e., "OWA," or "Outlook Web Access," for Microsoft Office users), it's not the user's account. At this point, it's probably the user's Outlook profile or their Windows user profile. The Outlook profile is a very quick fix (2-3 minutes), while the Windows user profile is a slightly longer fix, but still fairly quick.

### Where is it Happening?

This will be a bigger question when you work for a more geographically-diverse company, but take note of where instances are occurring. If all of your users in Mexico City are having problems logging on a payroll system, for example, but you know that US-based Accounting staff are able to log into that same payroll software, then you might want to look at other services in Mexico City. Are users able to reach the Internet? If your program resides on the network, can your users reach other assets on the network? A local file or print server, for example? Can you reach any of the servers in that area?

### When Did it Start Happening?

Be careful with this one! Correlation does not equal causation, and just because the network team was working on the router last night, that doesn't necessarily mean that's what caused the outage with the payroll software; however, it's still something you'll want to keep in mind. Find out when the issue started occurring. A lot of users won't necessarily know, and of course, if an issue occurs when no one was working in the software, they won't necessarily know there was anything wrong until they start using it.

With all that in mind, however, always try to ask if anything else happened around that time. When was the last time the computer/hardware/software/whatever was known to be working? When was the first time it was known to not be working? Did anything happen right around that time? When you look at logs to try and narrow down the cause, it will be very helpful to know, "we logged into the payroll system last night, and it was fine up until 5pm, but we couldn't log on this morning at 8am." Also, keep in mind that some problems can crop up that don't immediately cause outages until unless another action occurs; for example, when a user logs out or reboots their computer, when the server restarts, after a screensaver locks, etc. So keep an open mind on this.

## Start Troubleshooting

```
Note that not all of these steps will be necessary for smaller issues, but this makes for a good set of habits to get into

```

### Trust, But Verify

I don't want to say users lie, but... they lie. Or maybe, "don't fully understand the problem" would be a better way of characterizing it. Find out \*exactly\* what is going on, and remember that even the smartest person can make the dumbest mistakes.

### Have You Seen This Before?

This is the easiest step, and one a lot of people skip, but if you've seen this before and you know what fixed it the last time, try that fix. Obviously you'll need to use your best judgment here. Unplugging the cable modem and plugging it back in might make sense if you're in a three-person sales office, no one's been able to connect for four hours, and IT hasn't responded to phone calls, voice mails, or emails. On the other hand, if you're in a larger office with 100 people and Internet connectivity is intermittent, you'll want to be more cautious about taking that kind of approach.

Also (and this is important): if/when something gets fixed by IT/vendor support, *ask what they did and how they fixed it*. This is one of the best ways to learn! If possible, spend a little extra time to go over all the details until you understand how to fix the issue in the future, and/or possibly prevent it from happening again.

### Check the Basics, First

I can't count how many times (even in recent years) that I've worked on an issue for hours or even days, then rebooting the server and finding the issue just "magically goes away." There's a reason level 1 help desk people ask you to do the simple stuff; it's because we all miss it.

* Reboot
* Check the cables/physical stuff
* Make sure you're using the right password/credentials
* Check to ensure the bill has been paid (seriously)
* Are any of the hard drives full (more on this later)?
* Is the licensing still valid?
* Are the services all started (more on this later)?
* And so on...

### Look at Task Manager / Resource Monitor

Task Manager is a bit more basic than Resource Monitor, but should suffice for basic troubleshooting. If Task Manager doesn't provide the information you need, and you're comfortable using it, launch Resource Monitor. Either way, check the following:

* See if any processes are frozen
* Are any resources (CPU, Memory, Disk, or Network) being overloaded?
* How long has it been since the system was rebooted?
* Are there a lot of disconnected user sessions still active?

### Check the Event Viewer

This applies to any kind of logs, really (especially if the application itself provides them, but the Event Viewer is a special built-in Microsoft Windows application that tracks events and errors in real time. This can allow you to find specific errors that can then be researched online.

### When Was it Last Working?

If there's a "known good" configuration, you might be able to reverse whatever changes happened between then and now. For example, if you know that everything was good yesterday and a bunch of Windows Updates were deployed last night, you might want to look at removing those Windows Updates. Or if the system was good before a particular piece of software was installed, you could uninstall the software. Maybe a driver update killed something, or a new piece of hardware.

### Can You Find any Information on Google?

Seriously; there's a LOT of information out there on this, and while some of that information can be pretty hard to decipher when you're first starting out, a surprising amount is pretty easy to understand. Just copy and paste the error message into your favorite search engine, and see what comes up. You'll need to use some judgment (which will come in time), but this will often give you the answer, or at least a way to find it. Also, you will be able to find a lot of information in the Event Viewer (more on this later).

### Communicate

This is one of the best (if not THE best) skills to learn. Let your boss know what's going on, at all times. Let any stakeholders (i.e., interested parties) know what's going on. Before you do something that could potentially impact the users, let their managers know, and if needed, let them know why you're doing whatever it is you're about to do, if at all possible. Something like, "Hey everyone, the file server is currently down and we have over a dozen reports of people who can't access their files. I'm going to reboot the server at 2:45pm (in 15 minutes), so if you still have access and have any open files, please make sure to save them." If possible, let your boss (and possibly any really critical staff members) know verbally.

### Drop Back 10 Yards and Punt (if needed)

If you need to, engage your co-workers, the next higher level of IT, or the vendor (assuming you have vendor support). You should always do a reasonable amount of troubleshooting prior to contacting someone else, but you'll need to judge what that "reasonable" amount is. In some high-stress situations, you might be in "triage mode," where you're struggling to keep your head above water and just get the patient out the door. If possible, try to provide as much information as possible, and learn what the fix was. If you have to implement five separate possible fixes at the same time because you absolutely have to bring that server online ASAP due to pressure from higher ups, then it is what it is and you do what you need to. The key is; whenever possible, learn. Every IT person has failed, repeatedly, but what makes a good IT person successful is that they learn from their failures and they don't make the same mistakes twice.

### Document the Fix

This serves two purposes; first, it's a CYA. It's easy for a Monday morning quarterback to complain about, "Why did you restart the file server? I lost a marketing proposal I worked on for eight hours!" If you have documentation on why you did something, this can help you later on if/when political pressure comes your way. Also, it helps identify what happened, how it happened, and hopefully how to fix it or prevent it from happening in the future.

Also, always be prepared to answer the whole what/who/where/when afterward, as your boss and/or other stakeholders will likely ask, if it was a critical issue.