[HD2 Android NAND Development] ph03n!x:
Index

Post 1 - Android Boot Process, Space Restrictions in /data, Usual Workaround scripts that load from init.d

Post 2 - The Permanent Solution for Internal space issues in /data, and how it is executed (Step by Step)

Post 3 - Credits, Benefits, Note on Cache, Note to ROM Makers on ClockWorkMod Updater Script, FAQ

Post 568 - Test ROM for Native Mount DataOnEXT with DalvikOnNAND

Post 622 - Test #2 ROM for Native Mount DataOnEXT with DalvikOnNAND

Thanks to quite a lot of inputs, ideas, testing and feedback, I have been able to give a working solution to reliably mount /data to the EXT partition in our HD2’s SD Card before Android initializes. I am redoing the first three posts of this thread to give a complete info that answers the what, why, how questions.

Before I even start, let me make my objectives clear:

Have a system partition that is as big as I want in NAND, so that I can install any bloated ROM I want without chopping down anything, and yet have a /data partition for all my apps and their bloated data

Avoid using internal (NAND) userdata partition completely, so that if the day comes where my /system is full of bad blocks, I will still be able to create a system partition in the ~900+ MB internal NAND my TMoUS HD2 has

With this method, this is how I am using my phone:

Device - TMoUS HD2

Boot Loader - cLK 1.5.0.9

Partitions (in order) and Sizes - Recovery - 10 MB, Boot - 10 MB, System - 500 MB, Userdata - <I didn't bother :D>, Misc - 1 MB, Cache - 2 MB.

Recovery - CWM Based Touch Recovery B8

SD Card - 16GB Transcend Class 6 with 4 GB EXT4 Partition (mounted as /data, of course!)

My idea for this setup is that a user should not be having to get into the boot loader or changing partition sizes for every ROM. With this setup, I am having a close-to-native-Android device setup and can flash any ROM I want, without worrying if my partitions will accommodate it.

So what exactly does this method do? Before I answer that, a brief on the Android booting process – Note that this is simplified to meet this thread’s requirement, and not necessarily completely accurate. When you power on the phone the following actions happen…

The Android Boot Process

Bootloader – In HD2’s case, Magldr or cLK – loads the kernel based on how you have configured the phone.

Kernel – The kernel (zImage) is loaded into RAM along with an initial ramdisk (initrd.gz), which initializes various devices (IO, memory, GPU, etc.), interrupts, and mounts the root file system (/). After this, the first user-space process called init is started.

Init – this is a binary file that is contained within the initrd.gz. The init binary processes init.rc and init..rc , along with other .rc files that are called by these two .rc files. Some of the key functions (from this thread’s perspective) in the order of their initialization/ execution are:

The init process follows the instructions in the init.rc and init.xyz.rc files and creates empty directories including /data. It then mounts the storage devices (partitions in the internal NAND (MTD)) to these empty directories. The NAND partition for system is mounted to /system, followed by the partitions specified for data, cache, etc. The directories for dalvik-cache (/data/dalvik-cache) are also created by the init process after mounting the specified device to /data.

The init process then starts various services including adb, service manager, Volume Daemon (vold) for media like SD Card (FAT partition). Most importantly, the zygote service which initiates the Dalivk-Cache is loaded in this sequence.

As we all know, Android is based on Linux. The boot sequence described above is common for all Linux machines – until the zygote stage. Core Android file like core, framework, services, IME, policy, etc. are executed from the Dalvik-Cache and hence

Initialization of the Dalvik Cache is pretty much where Android comes into the picture

The sysinit/ run-parts part, which runs scripts from the /system/etc/init.d later the Zygote stage. No matter how this is done,

Android has already started loading by the time the boot process comes to executing scripts in /system/etc/init.d

Now that you know what happens when Android boots, you may also have realized that mounting a device to /data happens early in the boot process.

Space Restriction in HD2, and the twist with bad blocks

The 'normal' way to store all OS data (dalvik-cache, Configurations & Settings, Accounts, etc.) and User data (Apps, app data, mails and messages, call logs, etc.) is to have a partition named userdata in the HD2’s NAND, and have it mounted to /data during the init process. The drawback in using the userdata partition is that the space you can have for userdata is inversely affected by the size of your system partition, and whether your phone is an EU/ International HD2 or a TMoUS HD2.

Best case is, if you have a ~100 MB system partition for a really scaled-down, space-optimized OS, you will get about 800+ MB for userdata partition on TMoUS and ~400 MB on an International/ EU HD2. This space might be good enough for a lot of us who only have light-to-medium use of our phones. But for quite a lot of us, this is a bottleneck. As if that is not enough, we have these villainous bad-blocks that creep up slowly, determined to swallow up the internal NAND completely, though it would be several years down the line before that happens :)

The Workaround

The workaround for this method is to have a script in the /system/etc/init.d that mounts the EXT partition in the SD Card to sd-ext, moves a part of /data from the userdata (NAND) to the EXT partition (sd-ext) and creates symlinks (symbolic link) in /data (userdata) to point to the new location in the EXT partition. In this method you will not see an increase in what the phone shows for internal space. The apps you install and then some more will be installed in the EXT partition, which will be symlinked as /data/app and so on.

The other way is to move the mount point (and the contents) for the /data directory from userdata (NAND) to the EXT partition in the SD Card, mount the userdata (NAND) as sd-ext or some other directory, create dalvik-cache (and maybe /data/data - where all user apps' data are stored) in the userdata and symlink that location back to the EXT partition (which is now /data). In this method, you will get the size of your EXT partition as the internal space. Moving parts of /data to NAND will in theory give you better performance, as the most read/ written part of the phone resides in NAND.

The benefit of these methods is that it is very simple and easy to install on any rooted ROM through the recovery. This method balances performance by moving the static part of /data (like the .apk you install) to the EXT partition, keeping the dynamic part (/data/data – where all app data are stored, or /data/dalvik-cache – from where all apks are executed) in NAND.

The drawbacks with this workaround:

This method is initiated AFTER Android boots

. No matter how it is executed or what it does, anything that loads from /system/etc/init.d is executed only after at least some parts of Android loading process is initialized. This leads to weird outcomes – low sound issues, unpredictable behavior of the OS or some apps (Notification Widget toggles not working, etc.), and then some more. There is no reasonable way to predict what issues you may get. In fact you cannot be sure if the issue is with the ROM or if the issue is occurring because of the move2sd script you are using. If you are facing an issue that none others are facing and if you have a move2sd script, then you MUST undo what the script does before you can start troubleshooting.

In some cases, an in place upgrade (upgrading from one version of the same ROM to a higher/ newer version, or upgrading from one ROM to another ROM of the same Android version/ build) will corrupt the apps' data as well as configuration, etc.

Moreover, one move2sd script that works fine with one ROM will not work exactly the same way with another ROM or even an updated version of the same ROM.

There is always some amount hit or miss involved in this workaround!!!