'"even what seems like plain stupidity often stems from engineering trade-offs"'

This has truly become something I try to keep in mind, considering a) I've later, sometimes long after starting on someone else's code base, learned a useful rationale for why they did some of the previously more inscrutable things in their code, and b) ended up writing a few things like that myself.

Documentation is key to understanding these systems, but it isn't sufficient. Often you are presented with a nicely documented mega-function, which while anyone can read through, but is very hard to reuse a portion of when needed. In breaking it apart into smaller chunks, you necessarily scatter some of the reasoning about why a particular approach was taken from where it was originally used, or at least where the weird behavior is required. You can either reproduce large chunks of the documentation at many different points in the code base, and hope it doesn't get out of date as the systems it describes in other files is slowly changed, or keep the documentation as fairly strictly pertaining to the code immediately around it, in which case the knowledge of how the systems interact can get lost.

Whenever you encounter code that seems to make no sense, it's better to assume there's some interesting invisible state that you need to grok, than that the programmer was an imbecile or amateur. The latter may be true, but assuming that from the beginning rarely leads to a better outcome.

Edit:

I'll share my favorite example of this. At a prior job, we had a heavily used internal webapp written in Perl circa 1996. It was heavily modified over the years by multiple people, but by the time I was looking in on it in 2012, it was a horror story we used to scare new devs. The main WTF was that it was implemented as one large CGI which eschewed all use of subroutines for labels and goto statements, of which there were copious amounts. The really confusing part was that they were used exactly as you would expect a sub to be used, just with a setting a few variables and a jump instead, so we always scratched our heads as to the reasoning for this. There was even a comment along the lines of "I hate to use goto statements, but I don't know a better way to do this, so we're stuck with this."

Fast forward a couple years, and I'm migrating the webapp to a newer system and Perl, and I discover the reason for this. At some point it was converted to be a mod_perl application, and the way mod_perl for Apache works is to take your entire CGI and wrap it in a subroutine, persist the Perl instance, and call the subroutine each request. The common problem with this is that because of this any subroutines within your CGI can easily create closures if they use global variables. The goto statements really were intended to be used just like subroutines, because they were likely switched to in an attempt to easily circumvent this problem. Now, there are better methods to combat this, such as sticking your subroutines in a module, and having your CGI (and then mod_perl) just call that module, which is what I ended up converting the code to do, but the real take-away is that the original decision, as impossible to defend as it seemed, was actually based in a real-world trade-off, and at the time it was done may have actually been the correct call.