I think the closest I ever came to this was when I stumbled into a bug in git.

We were asking git to do something utterly trivial, like clone, and it was segfaulting. We installed the debug symbols or something (as I recall you can install these separately in Debian) and started trying to see where the function was crashing. The crash was in a parser, and the code was doing,

char *foo = strstr(input, "something constant");

and segfaulting later (but not chasing null!), during the first use of foo. I figured that input must, therefore, be a bad pointer, since the string literal is by definition fine, and the only way to get a bad pointer out of strstr was to give it bad input in the first place.

So, in gdb, I print the input point. It's valid. "Damn" I think to myself "we've gotten unlucky, and whatever triggers the cascade of UB hasn't happened this run. Freakin' heisenbug." So, I told gdb to just continue running the program: it crashed. Same backtrace: dereferencing foo triggered a segfault, and not with a null pointer.

If the input pointer to strstr is valid, how can the output be a bad pointer? strstr is documented as,

These functions return a pointer to the beginning of the located
substring, or NULL if the substring is not found.

So, what gives?

I attempted to debug strstr … that was almost a mistake. strstr is, unfortunately heavily optimized. It might have been this one: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86...

I'm okay with a debugger, but I'm not going the be able to follow where vectorized code is going wrong. Almost stupidly, I told strstr to (f)inish the function's execution, at which point gdb printed the return value: it was a valid pointer!

"Odd. Is this run going to succeed?" I continue execution: it crashes.

I restart the debugging, and run to the completion of strstr: the pointer is correct, but again, crash. How. The segfaulting address isn't the pointer strstr is returning, either, and nothing modifies foo after the strstr.

I single step out of strstr, and immediately print foo. The resulting pointer is wrong. strstr is working fine … but what, the assignment operator is broken? At this point, I'm sure I'm crazy. I disassemble the source, and lo and behold, the disassembly is trivially wrong. Forgive me as I can't amd64 unless I'm looking at it, but the disassembly is something like,

call strstr
cltq
<store foo>

We look the odd instruction up; it sign extends eax into rax. That's … not a valid operation on a supposed char pointer … there's not even technically a value in eax at this point. strstr returned a pointer in rax. eax is just the lower 32 bits of that pointer, and sign extending that back into rax makes no sense. It's like the compiler thought the return from strstr was a signed int … and that's where it hits me: C is shooting us in the foot again.

> and if no declaration is visible for this identifier, the identifier is implicitly declared exactly as if. in the innermost block containing the function call. the declaration

> extern int identifier();

> appeared.

The default return for an undeclared function, in C89, is int. (I'm actually not sure what C11 says about this. The C89 rule appears to be gone, but there doesn't seem to be anything in its place, which is bizarre. "Undeclared" does not appear, except for an unrelated footnote.)

Slap in the proper include for strstr, recompile. No segfault, disassembly is correct.

Upgrade to latest version, disassemble: disassembly is correct. Someone else fixed the bug, in the meantime. Should have just tried upgrading in the first place…

(After typing this up: had to dig up the debugging session. It was strchr, not strstr. Potato, potato. Someday … this should be a blog post…)