[G Tablet Android Development] rajeevvp:
All right--here's the explanation:

We begin with what happens when a user selects "Reboot recovery" in "Advanced".

- The function show_advanced_menu in extendedcommands.c calls android_reboot(ANDROID_RB_RESTART2, 0, "recovery") (CWM-5.5.0.4).

- That function (implemented in android_system_core/libcutils/android_reboot.c) calls the __reboot(LINUX_REBOOT_MAGIC1, LINUX_REBOOT_MAGIC2, LINUX_REBOOT_CMD_RESTART2, "recovery") function in the Bionic C library. (CWM versions 3.x and 4.x call this __reboot function directly.)

- The Bionic C library __reboot() calls the associated system call in the kernel: SYSCALL_DEFINE4(reboot, int, magic1, int, magic2, unsigned int, cmd, void __user *, arg) (in kernel/sys.c of the kernel source).

Now something interesting happens: In that system call, Tap'NTap, or, Malata, has added a hack to the (nVidia-originated) kernel sources (in both the 1.1 and 1.2 kernel patches) to get their bootloader to boot into recovery mode when the tablet restarts. They check to see if a reboot into recovery is requested, and if it is, writes some strings into the second page of the MSC partition which is the Bootloader Control Block (BCB):

case LINUX_REBOOT_CMD_RESTART2:
if(strcmp(buffer, "recovery") == 0) {
misc_write();
}
kernel_restart(buffer);

- The misc_write() hack (in drivers/mtd/devices/tegra_mtd_nand.c) writes 2 different strings based on the bootloader they assume the kernel to run on. For 1.1 bootloaders they write a "force" string (there is a bug in the source here which will be obvious to C programmers, but, it doesn't affect the final outcome):

/* Write "recovery" in the front of this partition */
memset(&boot, 0, sizeof(boot));
// strlcpy(boot.command, "boot-recovery", sizeof(boot.command));
// strlcpy(boot.recovery, "recovery\n", sizeof(boot.recovery)); //force-recovery
strlcpy(boot.command, "[B]force[/B]", sizeof("boot.command"));

For 1.2 bootloaders, they write 2 different strings:

/* Write "recovery" in the front of this partition */
memset(&boot, 0, sizeof(boot));
strlcpy(boot.command, "[B]boot-recovery[/B]", sizeof(boot.command));
strlcpy(boot.recovery, "[B]recovery\n[/B]", sizeof(boot.recovery)); //force-recovery
// strlcpy(boot.command, "force", sizeof("boot.command"));

This 1.2 method seems to be the standard Android technique to boot into recovery because those strings are what CWM also uses. I should emphasize here that currently only the VS supplied 1.2 kernels (in ROM or recovery) write the "boot-recovery" string to force a boot into recovery mode. All the custom kernels we use at present still write the "force" string because they are all based off of VS's 1.1 kernel patch. This means that the custom kernels, whether in ROM or recovery, will reboot into recovery mode only on the 1.1 bootloader. On a 1.2 bootloader, the custom kernels will write "force"--which will be ignored.

- After writing the string(s) into the BCB on MSC, the kernel restarts the machine (using kernel_restart(), see above). The "recovery" argument is not used further by any of the functions which kernel_restart() calls to actually restart the tablet: machine_restart() (in kernel/sys.c) -> arm_pm_restart() -> arm_machine_restart() -> arch_reset() (all in arch/arm/kernel/process.c), and finally the arch_reset() implementation in arch/arm/mach-tegra/include/mach/system.h which actually reboots the tablet.

- After the reboot, the bootloader will check the state of the volume rocker and then the BCB to determine which kernel to load. The 1.1 bootloader will load the recovery kernel on either command--"force" or "boot-recovery"--but, it will erase the BCB only for the "force" command. If the command is "boot-recovery", then it is CWM's job to erase the BCB. This is why a "reboot recovery loop" cannot happen on a 1.1 bootloader by selecting that menu item. (Other code paths in CWM, however, can still cause the boot loop by writing "boot-recovery".) The 1.2 bootloader only understands "boot-recovery" and it does not erase the BCB.

- Now we come back to CWM--this time after the reboot. Everytime CWM shows the main menu, it will clear the BCB by calling the finish_recovery function. finish_recovery() (in recovery.c) will try to zero out the BCB in MSC by calling set_bootloader_message() (in bootloader.c). This function, however, will fail to clear the BCB on the gTab because there is no entry for the /misc partition in /etc/recovery.fstab (on CWM >= 3.x, and, the SMBA-1002 device tree. On CWM-2.5.1.x, the paths are hard-coded in the source, and the misc partition is, again, not touched): the volume_for_path("/misc") call will return a NULL. The CyanogenMod guys disabled this entry a long time ago (see this commit) because of problems with writing to the MSC partition.

Fixes

For CWM >= 3.x, the obvious fix is to add an entry for /misc into /etc/recovery.fstab:

/misc mtd misc

but, I don't think this should be done: The MSC partition is fragile and futzing around with it unnecessarily can result in "hangs". For example, while testing all this out, sometimes writing to MSC would cause I/O errors, and further reads/writes from/to the partition would fail (I blame the poorly written nNvidia/Malata/TnT MTD kernel code for this). Then when I rebooted the tablet, it would just hang--no ROM, no recovery. I had to put the tablet into APX mode and format the MSC partition using nvflash to get the tablet to boot properly.

I recommend that for the gTab, CWM devs just disable the checks for "/misc" in CWM so that it enables a "Format misc" option under "Mounts and storage". Users can use that to get out of the boot-into-recovery loop. Or, just flash my MSC formatting zip file.