Hi!

> I doubt I can do such thorough debugging. Are there any useful guides/resources one can use to understand and debug the various hardware architectures?

I'm not aware of any good asm or CPU arch books which cover avx more accessibly than the Intel manuals but I imagine someone will correct me if a good reference is available. I'd never heard of avx until today since those instructions are new since the last time I had to look at x86 SIMD.

I would actually just recommend a general text like "Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems" as a first shot. I can't really answer your question in the way you want :( There is no "magic book" which will explain everything you need to know.

I'm an "intermediate level" debugger. I've spent perhaps a few hours a week doing low-level debugging for the last couple of years on x86/x64 Windows and Linux plus maybe 2 embedded archs. I can give some general advice as to what might be a good use of your time if you want to get good at low-level debugging. Obviously, anyone doing stuff like this as their career will need to go a little bit further.

Mainly I am writing this because I think I can answer your question better than saying "Read several thousand pages here: http://www.intel.com/content/www/us/en/processors/architectu.... It's not really a good use of time for most people. It is not an entirely wasted investment in your early 20's, but 5 years down the track you won't remember much of it unless you live in the debugger or work on a code generator.

The way everyone I know does it is to have a good handle on the general case and research (google :) everything else as needed. You can learn the basics in about 40 hours. I know many people who are talented at reversing and debugging, and I can tell you that they do not have (or need) detailed mental models of the SIMD extensions or (speaking for myself) even the FPU. It's cool to know, but with the exception of domain-specific work (codecs, fast math) it is not necessary to keep that information "swapped in". When you are working low-level you will find there is an enormous amount of detail to the world. That's the whole problem, and I can explain this with an analogy: imagine trying to diagnose a cracked engine block with an electron microscope.

The general idea is to step back, know the basics and be prepared to apply detailed analysis as neccesary.

Priorities:

0. Determination, patience and "can do". Common sense and knowlege of general debugging strategies.
1. How to operate the debugger
2. Top 50 mnemonics and references on hand for the rest
3. Calling conventions and ABI (http://en.wikipedia.org/wiki/X86_calling_conventions)
4. Basic OS internals (location of key data structures, heap layout, syscall conventions)

... (everything else)

0. The number one predictor of success is to have a kind of "I can do this" attitude even though you might not necessarily know what you are doing. The confidence that you can figure it out and the willingness to spend the time to do it. You won't always be right, but you won't get far without it. You also need the general principles of divide and conquer, basic logic and how not to fool yourself.

1. Knowing your way around the debugger really well is more useful than knowing reams about, say, the specifics of CPU architecture. So, how to set up symbols and sources, inspect the state of your process/threads. Most crashes can be resolved with a backtrace and looking at a few locals (assuming you have source).

2. If you need to read asm, you only need to know the top 50 or 100 mnemonics (if that). If you look at instruction frequencies you'll find the top 50 mnemonics make up more than 95% of all code by frequency, so you can work quickly knowing just these and look up the remainder as required. I had a reference for that (which I can't find) but a quick-and-dirty analysis (I did this on x64 Ubuntu 10.04) goes:

$ find /usr/bin -type f | xargs file | grep ELF | cut -d: -f1 | xargs -n1 objdump --no-show-raw-insn -d > /var/tmp/allops.lst # dump all instructions from binaries in /usr/bin/ to /var/tmp
$ egrep -i ' [0-9a-f]+:' /var/tmp/allops.lst | awk '{ print $2 }' | sort | uniq -c | sort -rn > /tmp/opfreq.list # get sorted list of mnemonic frequency (highest at the top)
$ head -100 /tmp/opfreq.list | awk '{sum += $0} END {print sum}' # accumulate frequency of top 100 mnemonics
30229337
$ awk '{sum += $0} END {print sum}' /tmp/opfreq.list # accumulate frequency of all mnemonics
30356097

Top 50 is 97%. Top 100 is 99.6%. If you do a more granular analysis (involving addressing forms etc) you'll find a similar conclusion holds.

3. The ABI comes next; basically because you're not going to be able to make sense of function prolog/epilog or the state of your stack without it.

4. Knowing your memory map and OS specifics really help too (so e.g. on Windows how to read the PEB/TIB, syscall convention for your OS, roughly how the heap is laid out, whether a pointer is pointing towards a local, a heap address or a library function). Again, only to a high level really.

---

The normal way to debug something like this (after googling your error, of course) would be to repro the crash, check the call stack, look at the source code for the library and figure out what path takes you to where you crashed. In this case you would work out reasonably quickly out that the crashing eip is in some AVX-optimised LAPACK code and that LAPACK chooses this code at runtime based on the advertised CPU capabilities. Then you would be confused for a bit. Eventually you would figure out you're faulting because AVX instructions don't work but only reach there because they're advertised. Hence Amazon's bug. The whole process is pretty slow, but it's the standard and obvious way of doing it.

However in this case the problem they had really was that the crash was intermittent.

Based on the narrative given, the picloud guys took a more "cloud-like" approach to diagnosing the issue: they ran the "unreliable" code (plus some environment scraping, I'm guessing) across a whole bunch of instances and worked out by google and eyeball what was different about the crashy instances. This is a practical way of doing it :) It's almost a kind of "statistical debugging", if you want to put things into buckets. Most major software vendors now get minidumps when their apps crash and this (statistical debugging) is actually an interesting field of study in it's own right. It could use some more postgrad. See e.g. https://crash-stats.mozilla.com/topcrasher/byversion/Firefox...

---

I'm going to finish this off by explaining what would be better than reading books and references: finding excuses to do it. It turns out that debugging is mostly thankless and only buys you credit in a very limited social circle. Truthfully it's not a good use of your life unless you're the kind of person who enjoys it. Think of it like chess or go problems. If you want to be good at it, you have to find an excuse. Some motivating activities people find for doing low-level work (in no particular order) are:

1) Cracking commercial software, writing game trainers, hacking online games (Download trials, or your game of choice)
2) Writing exploits (Say, check CVEs, figure out if you can repro, debug until your eyes bleed, write an exploit)
3) Improving open source software (find a bug tracker, repro crashes, isolate the bugs)
4) Doing crackmes (see e.g. http://crackmes.de/)
5) Commercial reasons (work on a toolchain, compiler, embedded system ports, your $software)

---

P.S: your complete problem solving breakfast should include repro first, understanding your target, a bit of reasoning and guessing, dynamic analysis (tracing first: Process Monitor/strace/ltrace, debuggers: gdb/ddd/WinDbg/Immunity/OllyDbg, instrumentation: dynamorio/pin), static analysis (objdump/IDA pro) and copious amounts of whatever will make your life easier.

---

If you are interested I can tell you some war stories about debugging problems in distributed systems but this post is already too long.