[Epic 4G Android Development] mkasick:
It's the same process until the kernel and initramfs is loaded. Specifically:

SoC ROM code loads and boots the Primitive Bootloader (PBL, boot.bin).

PBL searches the device partitions for the Secondary Bootloader (SBL, Sbl.bin), loads, and boots that.

SBL checks the "reboot mode" in param.blk (on stl6) and on a normal boot, loads the kernel stored in bml7 with init.rc, and on a recovery boot, loads the kernel stored in bml8. It also checks if the vol-down, camera, and power buttons are presed, and if so, sets the reboot mode for a recovery boot and loads the kernel stored in bml8.

The loaded kernel (stored on bml7 or bml8) also contains a initramfs, or root file system image. /init, in the initramfs checks the kernel command-line for a "bootmode" parameter set by SBL, which determines whether init.rc (normal boot) or recovery.rc/fota.rc (recovery boot) should be loaded.

As part of executing init.rc (or recovery.rc/fota.rc), the appropriate storage modules are loaded (fsr.ko, fsr_stl.ko, rfs.ko, rfs_glue.ko, j4fs.ko, and param.ko for a BML kernel), and various file-systems are mounted (stl6 as /mnt/.lfs, stl9 as /system, stl10 as /data, and stl11 as /cache for BML).

An MTD kernel differs by the storage drivers used in Linux. Instead of the Samsung proprietary modules, we use the open-source OneNAND MTD driver. Now, our OneNAND chip is different/newer than the one used in other SGS devices, so we actually had to backport the "samsung_audi" driver from the SGSII kernel source tree. Amusingly, the SGSII doesn't use MTD either, but some SGSII development prototype devices did, so somewhere along the line Samsung added that driver to the kernel tree to support their newer OneNAND chips. This is one of the reasons why it took a while to support MTD on our device as the driver that ships with SGS and Nexus S kernels doesn't work for us.

One of the takeaways from the earlier description of the Samsung boot chain is that both PBL and SBL must support I/O to the OneNAND chip, the OneNAND partition table, and the BML partition encoding. Additionally SBL must support STL for flashing via download mode, and for reading/writing param.blk. So, for MTD support, it's critical that retain compatibility with the partition table format, and use BML-encoded partitions for the kernel. Actually, the fact that we're running MTD is essentially invisible to the bootloader, it does the exact same booting sequence until the point that the kernel is booted and "we" have control.

Where MTD differs is in the format of the /system, /data, and /cache partitions. Instead of using RFS (or ext4) on STL, on BML, we use yaffs2 straight on top of MTD. To flash the kernel, CyanogenMod uses a user-space tool, bml_over_mtd, to BML-encode a kernel and write it to the raw MTD device. That's basically it.

I'm not sure why "reboot to recovery" doesn't work yet. As I understand, executing "reboot recovery" from adb or the command-line does work, although I've not verified that behavior myself. There's some bug that must prevent the command from working properly when executed from the UI, that needs to be investigated yet.

As for the three-finger issue, that's an artifact of a fix that we have to use in order to boot into recovery on MTD kernels. The Fascinate, which has the same problem, uses a very similar fix.

Earlier I mentioned that a normal vs recovey boot is determined by the "reboot mode" setting in param.blk. Specifically, when you do a three-finger boot, SBL sets "reboot mode" to boot recovery before loading bml8. Now, it's the responsibility of the recovery kernel to reset the reboot mode in parma.blk back to normal, otherwise SBL will always load bml8 and boot recovery on subsequent reboots/powerups, whether the vol-down, camera, and power buttons are held or not.

Now, the problem is that param.blk is stored on a partition, stl6, that must remain BML/STL/j4fs encoded for bootloader compatibility, and so MTD kernels lack the driver support to mount that partition. Thus, we can't reset the reboot mode setting.

So, the three-finger fix consists of a kernel with a stub initramfs installed to bml8. It's actually a BML kernel, that is, it supports Samsung's storage modules and not MTD. The sole job of this kernel is to reset the reboot mode in param.blk, poke a magic value into persistent memory, and reboot. On reboot, the MTD kernel in bml7 is loaded, checks for our magic value in persistent memory, and boot into recovery. The caveat is that, if you keep the vol-down, camera, and power buttons held, SBL will continue to boot bml8 repeatedly, which continues to reset the reboot mode and reboot the device.

Hope that helps.