I once had to debug a very strange error. It was in the middle of thousands of batch processes one of them would just fail. No error message just stone dead. Most of the time when I reran the process it would then work fine.

Eventually I isolated a very weird string that I wolud see in the stderr some time before the error happened. It was just "Err" and then a few digits of hex. I set about searching for this but couldn't find it anywhere in the source code. I eventually found it by running "strings" on all the libraries through grep - it was in a linear algebra addon library. This allowed me to narrow down to just a few plugins which used that library (most of them used a different one) and from there to finally make a reproducible case that I could run locally. When I ran it locally it worked fine. Hmm.

I went to get a coffee and think and came back to my desk and ran it again. It died. Not only did it die but it crashed the IDE I was running it in in a very interesting way. The TUI[1] was stone dead but the process was still there on my computer. However diagonally down about 5 cm or so from the top left corner was a questionmark, superimposed on everything else. I had to kill -9 the process but I was nearly there. What had changed while I was getting a coffee?

This process was trying to calculate some risk on some basket options[2]. The prices of these depend on the correlation between the underlyers and this correlation is marked and remarked constantly by traders throughout the day based on what they observe in the market. The point is that noone marks the entire matrix, if they're doing a trade they will come in, mark a few correlations relating to that trade and go about their business.

Suddenly I knew why the bug was appearing and disappearing - it depended on the correlations. I put the equivalent of a test for null in the code and immediately I didn't get the crash and the weird question mark I got the actual error:
"Cannot Cholesky decompose a singular matrix". Basically sometimes the traders would mark the correlations in a way that would make the matrix non-invertable. We noticed it the most because we traded the most weird stuff so when you pulled out just a few correlations (say coffee commodity price + SBUX + SPX or whatever) the matrix wouldn't invert and when you tried to do the decomposition it would return essentially a null pointer and an error. There was some code that was just dereferencing the pointer so in the terminal it crashed the process with this weird question mark and in the overnight process it just gave the inscrutable error number but that was the real reason.

The investigation process was really epic but the fix was pretty tame - in fact I had already done it - the null check I had put in while trying to track this down was all that was needed.

[1] Think a bit like Borland Turbo Pascal and you're there. If you know you know.

[2] These are trades where you have an option which can be on one or more things and in the event it gets exercised you get something specific. So for example say you don't know in a given year which market is going to do better (Europe or the US) but you think they will both do well. One thing you could do is buy a "worst of" call which will give you a call option which when exercised will give you the performance of whichever one did worst. Why would you want that? Because it will be much cheaper than buying either of the call options outright. In any case the price depends on the correlation between the things in the basket.