[Miscellaneous Android Development] forbiddenera:
this guide is under construction still and not quite finished - check back soo

Hi guys, I'd like to share some knowledge and as I'm going to start a blog doing so, I felt this would be as good as anywhere to start.

This technique will allow you to create disk images of your chosen size on as many storage devices as you want and that your device supports. There are a few advantages and reasons someone would do this. Storage flexibility, performance, data redundancy, easy backups. The techniques outlined here will likely be obvious to any linux pros, but perhaps not thought about in a mobile context.

While the current instructions and application of this technique is relatively complex, part of my goal in starting this is to gauge interest on whether it would be worth making a tool to do this. Perhaps there is one already, or another how-to. I didn't find any in my research in setting this up, albeit as a linux guru I didn't have to do much.

I will also include an APPENDIX including instructions on how to automatically compresss a filesystems contents (READ&WRITE) to save space with fusecompress. I may also add one for an encrypted filesystem if someone requests it.

DIFFICULTY: Hard for noobs, Easy for Pros.

REQUIREMENTS;

- Android device (rooted) preferably with multiple storage (sub-)systems (one will do if all you want is imaging or compression)

- Currently LINUX MUST BE INSTALLED ON YOUR DEVICE AND HAVE A BIND MECHANISM TO ACCESS THE MOUNT POINT IN ANDROID. As mentioned above, if the interest is there I will port the utils and make an app to manage it. I recommend Galoula Linux Installer w/ Debian stable. This tutorial is based off that setup but will likely work w/ Ubuntu and other Linux methods with few changes. Once all the tools are ported/compiled for Android, a Linux installation will not be required. You could very well compile it yourself..

- Working apt-get package manager and an internet connection.

Linux packages/tools needed & requirements/dependencies:

- lvm2, fuse, fuse-utils, libfuse, gvfs-fuse, libfuse2, libfuse-devel*, gcc/compiling tools*, fusecompress*+, busybox binaries or equivilent, fdisk, dd, mke2fs (or equivilent for your chosen filesystem), and iozone for benchmarking if you want.

- You will need a kernel & linux that supports fuse/loopback devices (most do!) or you can load the loopdev module. You will need at least 1 or 2 or more free loop devices depending on your configuration. There is a kernel parameter to change the maximum. **TODO: Add instructions for that **

- * and +: You will only need libfuse-devel and gcc for compiling fusecompress ONLY IF YOU NEED/WANT COMPRESSION. (If your chosen distro has a working fusecompress package you can use that, however at the time of writing DEBIAN-STABLE has not included an updated package for it. Squeeze had it, wheezy has some patches and updates for it but is not released as a package nor can I get it to work stabily after I spent hours convincing it to compile. I should note that this package is based off the CPP aka 1.9x branch (v2.6) and isn't completely stable compared to the older (slightly slower) but stable 0.9x branch. The APPENDIX instructions fot fusecompress ARE FOR THE STABLE/0.9 BRANCH)

My setup (as a reference):

Motorola XT875 aka Targa aka Bionic running Android 4.2.2, Avatar ROM (based on Cyanogenmod CM10.1) Latest Weekly Beta

16G internal device/SD storage, 16G patriot class 10 MicroSDHC. Dual-core ARMv7 armhf @ 1.2ghz, using DEBIAN-STABLE (Wheezy) under Galoula Linux Installer installed to it's own separate Ext3 partition on the external SD and Android Bind active. Also using my tablet, an a13 clone, with same Linux configuration.

I have a 2GB disk image on the internal, and a 2GB image on the microsd and they are combined as RAID0 (striping) w/ 4k stripes, ext3 fs w/1kb blocksize, 4kb stripe stride and compressed w/LZMA compression level 3-8 depending on my mood. It is setup like this for storing source code (cm-10.1 source actually, I am going to attempt to compile it entirely on mg phone, or at the very least my own modified portions.

Step 1.

--> If you're a pro, install all the stuff above, skim this section (mainly the part on finding an open loop) and skip to Step 3. Otherwise continue reading.

1a.

Boot up your linux distro on your Android. For me this means opening TerminalIDE, launching a term window, typing 'su' to gain root and then 'linuxchroot.sh' to launch the linux environment.

1b.

Then, ensure you have loop/fuse support.

ls /dev/block/loop*

.. should list all your systems loop devices, you can also type:

losetup /dev/block/loopX

(where X is a device number) to find out if the device is free or used by another app.

A response like:

losetup: /dev/block/loop0: 0

generally means it's available, while

losetup: /dev/block/loop1: 0 /mnt/asec/com.some.app/data.img

means another app is using it and you shouldn't.

Remember this as you will need to find anywhere from 1-4+ unused loop devices depending on your desired configuration.

1c. *

Next, it's best to ensure you're distro is up-to-date, this is optional* but recommended.

apt-get update

..then..

apt-get upgrade

..then, again (for good measure: update before you upgrade, upgrade, then update the upgrades..)

apt-get update

Step 2.

2a.

Once that completes, start installing the required packages:

apt-get install lvm2 fuse fuse-utils libfuse libfuse2 gvfs-fuse

Press Y and agree to any dependencies, wait for it to finisn.

2b.

You will need to choose which filesystem to use. I recommend ext3, but you will have to choose which is best for your needs. FAT is supported almost anywhere out-of-box, but is CRAP. NTFS is a bit better but is likely unsupported by your device, exFAT may be an option even. Whichever you choose you will need the appropriate tools (mkfs) to create/format it.

For both FAT and EXT support (or rather creation ability - mounting & read/write support may be limited by your kernel/modules loaded):

apt-get install dosfstools e2fsprogs e2fslibs

Step 3.

In this step you will create the disk image(s), initialize the partition table(s) and file system(s) and the steps will vary based on your goals, so read carefully. Since we are dealing with images, there shouldn't be much risk for damaging your device/data as long as you're careful. I should note, for the less experienced, that incorrect parameters with some of these commands can wipe your device or SD, if you read the directions carefully and only use the disk image(s) with these tools you will be safe, of course, after you put data within the image, it can also be overwritten with some commands.

3a.

You will now decide which, and how many devices to put images and how large they will be, and most importantly, your logical volume (RAID, non-RAID, span/JBOD, etc.) configuration. I will go into brief detail on the different configurations and use cases.

RAID0 aka STRIPING

Uses/Performance/Pros/Cons

Primarily aimed at performance, NO REDUNDANCY PROVIDED.

RAID0 will write one X byte block per Y devices looping in order. This effectively splits/distributes your data evenly among member devices.

Disk size will be size=devices*device size (or *disk img size)

All images (or devices) need to be the same size, extra bytes will be left unused.

Performance (P) (aka throughput) should be/is about (in theory) S*N=P where S is the average device speed (among all members, assuming similar/identical devices & busses, also note that P will never really exceed the slowest devices avg/max speed*number of drives, so slower drives are a bottleneck, try to use equal or identical devices) and N is the number of devices. On a physical hard disk drive, seek times can be lowest avg seek*members at best sometimes, though seek latency isn't an issue on our flash devices.

My benchmarks hardly show any impressive performance increase however my internal and external sd are obv. not matched, and I haven't done a ton of testing/benching/optimising.

Raid0 examples: 2x4GB disk images, each on a class 10 MicroSDHC card will appear as one 8GB device that can perform as fast as both devices can work together, about twice as fast as one, seek times won't decrease much. 4x200GB SATA drives striped will be one 800GB drive that is almost 4x as fast, however seek times may decrease to 1/4.

RAID1 aka MIRRORING

Uses:

Primarily aimed at data redundancy.

RAID1 will write one X byte block to EVERY device. This effectively MIRRORS your data among member devices.

Disk size will be equivilent to the device/img size regardless of the amount of devices it's mirrored to.

WRITE PERFORMANCE will be the same as the slowest single devicd, READ PERFORMANCE can increase up to x*member devices as it can act 'sort of like' RAID0 for reading, it doesn't need to read EVERY byte from EVERY device, it only needs to read 1/N (n=num of devices) from each device reading all simultaneously.

I have not tested mirroring in this setup yet.[/b]

JBOD aka JussaBunchO'Disks

Uses:

Just basically spans the devices for extra space. No performance techniques, striping or mirroring, rolls off the end of one disk (or img) onto the next. Devices/imgs can be any size and there's no performance hit for using devices/imgs on the same bus or device.

OK -- I started this a while ago, and here's my attempt at finishing it.

Not sure if you need fusefs unless you want to do file system compression.

You do need the lvm2 package, however.

Get down to it:

Now you have to create your image files, loop them, and raid them.

To create the images, use this command (this creates a ~100mb image, adjust the size as necesary):

dd if=/dev/zero of=/mnt/sdcard0/.myRaidImage-0.img bs=1024k count=100k

dd if=/dev/zero of=/mnt/sdcard1/.myRaidImage-1.img bs=1024k count=100k

Then, loop them to two free loop devices (see above to find out if they're free):

losetup /dev/block/loop1 /mnt/sdcard0/.myRaidImage-0.img
losetup /dev/block/loop2 /mnt/sdcard1/.myRaidImage-1.img

Before using any of the lvm2 tools below, you should check that your /etc/lvm/lvm.conf is correct, you may have to change the dirs from /dev to /dev/block for example.

Now, use pvcreate to create lvm physical volumes with those images:

pvcreate /dev/block/loop[1,2]

And vgcreate to create a lvm volume group for your lvm volumes:

vgcreate vg0 /dev/block/loop[1,2]

Now, we can create either a JBOD/spanned/contiguous disk, a mirrored disk, or a striped disk.

First, we will create a mirrored logical volume, with a log (in contrast to lvm2's raid1 mode which has no log):

Note: I had to force lvm2 to manage devices in /dev instead of relying on udev (which my device doesn't have), this is quite possibly the case for you as well, the option is in /etc/lvm/lvm.conf.

lvcreate -m 1 --mirrorlog mirrored -l 40%VG vg0

If that is successful, you now have a usable logical volume. To actually use it however, you will have to activate a filesystem on the device.

mkfs.ext3 /dev/vg0/lvol0

Then, you can create a mount point and use it:

mkdir /mnt/mirrored
mount /dev/vg0/lvol0 /mnt/mirrored

NOW YOU CAN USE IT.

The gentoo wiki has a great page on lvm2, and how to use it: https://wiki.gentoo.org/wiki/LVM

Most of it can be applied here with my instructions.

I will come back later and add specific instructions for striping and other modes, including stream compression and perhaps encryption - however - it make take me a while, which is why I posted the link - you should be able to figure it all out with this & that :)