[Hardware Hacking General] js22:
Yeah, it's got to be the fsa9480. That plus (possibly) the volume/power/etc buttons are the only possibilities. The fsa switches which lines from inside the phone are connected to the micro-USB pins 2 & 3 (aka D+/D-). But it also has (at least) 2 digital outputs called JIG and BOOT which feed back to the CPU. BOOT presumably causes a hardware reset, so the JIG line is free to determine the boot mode.

We know the normal boot is from OneNand. Looking at table 6.3 of the data sheet tells us that the only pin that matters is OM5. Pins OM[4:0] determine which of the 4 different OneNAND boot modes is used, and that mode is the same regardless of OM5. So they are almost certainly just fixed. If OM5 is 0, the chip boots normally (directly from OneNAND). If it is 1, the ROM will first try to negotiate a UART connection, then try a USB connection, and only then (if I'm reading it right), fail over to a normal OneNAND boot.

So it's hard to imagine any scenario other than pin OM5 connected to the JIG output of the fsa9480.

From the Samsung source code linux/drivers/usb/gadget/fsa9480_i2c.h, there are a few interesting resistor choices (I'm not sure what the 5 bits represent; maybe they are for setting/reading the switch state over i2c) :

1 0 1 1 0 150K UART Cable
1 1 0 0 0 255K Factory Mode Boot OFF-USB
1 1 0 0 1 301K Factory Mode Boot ON-USB
1 1 1 0 0 523K Factory Mode Boot OFF-UART
1 1 1 0 1 619K Factory Mode Boot ON-UART

The 301K case (Factory Mode Boot ON USB) is the familiar "jig" people on xda use. But USB protocol is fairly complex, and since the whole idea of using the fsa switch is very non-USB compliant (the thing sends analog audio over D+/D- lines !) I don't think we can assume anything about what the ROM code does to "negotiate a connection". In addition, I think that all the people who have already looked into this were specifically trying to get into "download mode" (i.e., Loke protocol to talk with Odin). So who knows what else was going on beforehand.

I'm most curious about the behavior with a 619k resistor (Factory Mode Boot ON UART). The nice thing with UART mode is that we already know from TheBeano's thread that there is output at 115200 baud that appears at some point in the boot process. By putting a scope probe on the Tx line and simultaneously watching the text output in a terminal emulator, it should be very simple to see if any kind of negotiating is going on at an early (ROM) bootloader stage. Maybe it involves different baud rate, banging on a key, or pressing/holding a button. But (for someone with time and the right hardware) this is very do-able. If there is any hint of negotiation going on before the NAND-based bootloaders begin, we know we're onto something promising.

Like Odia pointed out, any stuff that we load during the ROM bootloader stage is not being flashed to the OneNAND; it is simply being loaded into RAM and executed. So even with no clue how to write a bootloader, I can imagine writing a "hello world"-grade program to, say, toggle a GPIO. That would clearly establish that the UART bootload procedure works.

So I think its an exciting prospect. Some of it is way beyond my abilities, but there are some easy steps early on that could really generate some intense developer interest.

I've got a I897 and a scope, but no connectors or cables to sacrifice. I may get a breakout board from Sparkfun to mess around with. In the meantime, I'd love to hear if anybody with a resistor/cable combo can sniff out anything interesting.

Also, I'm glad to see some response. I guess my title was a little cryptic, but at first it was just me and the crickets.