yashsingh1/xdaforums huggingface dataset:
[Desire Android Development] melethron:
@sibere:

Update: Sorry i'm wrong on the speed of the direct ext4 loop mount. I referred to the benchmarks of ownhere. I'm doing my own sqlite insert benches directly on the phone and ext2 loop on fat is MUCH faster than unjournaled ext4 loop mount. Im still not through with all test but considering sqlite write speed (and thus smoothness) this was fasted by now.

EDIT

: Tested umount. I don't get a clean flag after reboot and had 4 errors after reboot.

/ # e2fsck -f /sd-ext/ext2
e2fsck 1.41.6 (30-May-2009)
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 122788 has zero dtime. Fix<y>? yes

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Inode bitmap differences: -122788
Fix<y>? yes

Free inodes count wrong for group #15 (8024, counted=8025).
Fix<y>? yes

Free inodes count wrong (130028, counted=130029).
Fix<y>? yes

userdata: ***** FILE SYSTEM WAS MODIFIED *****
userdata: 787/130816 files (2.0% non-contiguous), 33818/523005 blocks

What ROM and what kernel do you use?

EDIT2

: Its not the ROM or Kernel. It is the Power menu of snq. If you reboot with the snq menue /data isn't unmounted.

But normal shutdown does. I think you should add this info to the main post since a missing umount

HIGHLY

increases the chance of file corruption (1 reboot 4 errors). Log was clean after normal shutdown btw.

EDIT3

Same on "adb reboot" and "adb reboot recovery" and snq "reboot to bootload". Rofl after 4th of this "unclean" reboots i already had a force close of sense xD. You really should warn people

NOT

to use the "power menu". It's as safe as a battery pull.
[Desire Android Development] ownhere:
about loop-on-device benchmark, You must benchmark it in wrong way. :)

"mount -o loop /dev/block/mmcblk0p2" in phone is invalid.

You must mknod,losetup to setup loop-on-device first, then mount -o loop /dev/loop0.

this is my benchmark:

2000 records insert:

ext4-loop-on-file in fat32 partition without journal, without sync: 7 seconds.

ext4-loop-on-file in fat32 partition without journal, with sync: 16 seconds.

ext4-loop-on-file in fat32 partition with journal, without sync: 125 seconds

ext4-loop-on-device without journal, with sync: 13 seconds.

so, loop-on-device with sync is the best balance.

BTW:noop scheduler and split /data/data all is my idea, right? :)

and in my phone, mount with noauto_da_alloc is slower than without it.
[Desire Android Development] Droidzone:
Been trying hard to get Cyanogen to work with DATA2SD. Nothing seems to work.. Tried from scratch, then after a Nand restore of the ext, then after Restoring a DATA2SD based Leedroid, after deleting all the apps etc. Does it work only on Froyo? Or is there something I'm missing here. The file I deleted from Cyanogen was \system\etc\init.d\05mountsd. Tried with and without it.

Maybe I missed some post here which mentioned some additional steps? I know Sibere's mentioned in OP that he tried it on Sense. Maybe someone could add some info?

Iinga? (you're on Cyanogen, rt?) Dan? (Defrost?)
[Desire Android Development] danarama:
I think both iinga and I are defrost.

Go back to your thread on android forums. The code I posted there is exactly what I did. Nothing more, nothing less. Hope that helps
[Desire Android Development] danarama:
Thanks. Can I do that in a terminal emulator?
[Desire Android Development] op8867555:
I think you should do that in recovery mode.
[Desire Android Development] iinga:
If I was to run the command:

/system/xbin/busybox mount -o loop,sync /sd-ext/ext2(4) /data

and then nand backup, would it back up just the ext partition as data?

Then to restore i assume that command would need to be run again?

@Droidzone, checking CM for you mate
[Desire Android Development] Droidzone:
Thanks, Iinga..

Tried multiple times..All the steps seemed to proceed ok, but it just wouldnt increase the space..

Kind of gave up trying to get CM on..

@Dan..Tried your commands too. they were fewer than Sibere's final version, and there were a couple of syntax errors, but corrected those and tried..But finally gave the error "loop device failed to mount"..

So back on Leedroid + DATA2Sd.. Maybe I'll leave it at that.. I'll wait for Iinga's comments..
[Desire Android Development] melethron:
@ sibere:

First of all. Sorry again for my false statement that loop doesn't increase speed. It does ...

But i'm still right about ext4 without journal directly mounted as loop. The reason it didn't work before is that i did a loop with "mount -o loop" and this doesnt work. I manage now to mount an unjournaled ext4 partition directly through loop and it increased the speed of your script by 10-15%.

I'll test a bit more add it as an optional "life-on-the-edge" patch to my ROM and share it then. But the advantages of ext4 with direct loop are:

- easy setup (now loop file required and only 2 lines in a startup script)

- Speed increase if the ext4 partition is aligned (because your loopfile isn't aligned)

- Higher safety bcs the partition gets unmounted (your fat partition doesn't only your loopfile)

- unjournaled Ext4 is faster then ext2 (because of extends)

- Doesn't use FAT

This way it preserves the advantages of loop while adding some more.

Also im sorry again for not doing proper shutdowns test. All the file corruption i had with your script was because i either used "adb reboot recovery" or the powermenu of snq. If i do a normal shutdown the partition is always reported clean by e2fsck. This is something i'm willing to provide to users because it only adds risk when there is a powerloss. I still will not make this the "default" setup of my rom but provide a zip to activate and one to deactivate the mount through loop so user can decide what they want.
[Desire Android Development] ficti0n851:
How to get it working with CyanogenMod

Hi everyone,

My first post, been following this thread for some time, had this working with CM but by the looks of it scripts in /system/etc/init.d are not run at boot without modifying another file.

Add this to /system/etc/init.local.rc

service 01data /system/etc/init.d/01data

user root

group root

oneshot

Make sure permissions on this file are 755

This may not be the best way to do it but it works.

You may or may not need an extra sleep in /system/etc/init.d/01data

This is a great performance increase, and many thanks to sibere and everyone who has contributed.

Personal experience tells me to leave the sync on this as well, if you do manage to break something, such as a bad kernel build it stands a good chance of data loss without sync.

It also seems that BFS/BFQ does give slower performance than CFS/CFQ with my testing.

fic.
[Desire Android Development] danarama:
Thats no good then, as that would require ADB, ergo a PC so thats not a solution.

I'm happy with Nandroid anyways...

Thanks for the info.
[Desire Android Development] danarama:
Syntax errors where?
[Desire Android Development] op8867555:
How about write a script to backup/restore your /data and make it a flashable zip ?

Then you can do it without a pc.
[Desire Android Development] Droidzone:
I think it was on this line:

/system/xbin/busybox mke2fs -, 0 -F -L userdata /sd-ext/ext2

Just left out an m, I guess (probably while copying onto your post)

Should be:

/system/xbin/busybox mke2fs -m 0 -F -L userdata /sd-ext/ext2

@ficti0n851

Will try the Cyanogen method and revert..

By the way..The file \system\etc\init.d\05mountsd

That has to be removed, right?

A line on that file has a comment:

# mount ext[234] partition from sd card

Since Sibere doesnt use ext, I'm guessing this is the file which is to be removed.
[Desire Android Development] ficti0n851:
Hi, since that script is not in init.local.rc I'm not too sure if it has any effect on this apps2sd method, should be ok to remove 05mountsd.

Try it with or without, I'm a bit over the top with roms and kernels, I strip everything down to bare minimum so I would not have that file.

fic.
[Desire Android Development] danarama:
I'll edit that. Its a typo. , is next to m on a keyboard ;)
[Desire Android Development] danarama:
That is a good idea, but you give me too much credit. I have not yet learnt linux at a command line level :)
[Desire Android Development] sibere:
Not at all. If you backup with tar, you will have to restore with tar in recov mode.

Most of the time there is no point to backup everything.

usually /system doesn't change unless you flash new rom. /cache can be left empty, /data on nand same, so once you've backed up all you system, you actually only need to do a /data (ext) backup. This can easily be done with tar.

in recovery mode, mount /system /sdcard and mount /sd-ext (can be done with clockworkmod menu)

then mkdir /sd

/system/xbin/busybox mount -o loop /sd-ext/ext2 /sd

cd /sd

tar czvf /sdcard/ext2.tgz *

to restore do the same method to mount ext2. Once everything is mounted do;

cd /sd

rm -r * (to clean the file, you can reformat it also if you prefer, see the OP for this)

tar xzvf /sdcard/ext2.tgz

sync

cd /

/system/xbin/busybox umount -d /sd
[Desire Android Development] sibere:
Hum I don't and never compared roms dude.

I said from start (OP) that I developed this on a standard rooted htc sense froyo rom.

If it's behaving differently on other roms, there's nothing I can do about it.
[Desire Android Development] sibere:
Well, once again I've said in OP that I use a rooted htc sense froyo rom.

My kernel is based on RichardTrip V5f kernel (sense kernel too)

If one use a different rom or tweaks his rom with unreliable add-ons, it's their own responsability.

I can't warn everyone with all the bugs and issues brought by all different rom and kernels. This would be and endless job! ;)
[Desire Android Development] Droidzone:
You're awesome! It worked perfectly. The only mod needed was the addition in the file. Cool.
[Desire Android Development] sibere:
No worries.

I've tried ext4 on device without sync and without loop mount but I wasn't happy with the performances I had on my phone. Could be a Filesystem misalignment, a bat format or mount option I don't really know. Did you publish in your thread the way you've formated/aligned/mounted your ext4 partition? I would like to give it another try :)

Unfortuantely ATM I haven't been able to unmount mtdblock5 at boot b/c it's used before I try to unmount it. To sort that out I must tweak the rom to modify the ramdisk used @ boot to mount mtdblock 5 and it's a pain in the a*s.

I haven't been able to use data=ordered too but I think that I can solve this with an update of my kernel (I may have forgotten 1 option when building the kernel)

Regarding the FAT, I really don't think that it's a problem. As long as the ext2 loop file is unmounted correctly, there is no reason to get data corruption. After all, I'm not sure that /sdcard is cleanly unmounted too ;)

I've briefly looked though all the sqlite tests but I'm not sure that injecting thousands of datas reflects everyday's life :) But year, if the system can handle that, there should be no problems in normal situations.
[Desire Android Development] sibere:
All right just read your post.

After a few tap benches, CFQ and noop are at par. Maybe cfq is even better in some tests and noop in others. I now have to "feel" if noop brings smoothness to the usage but this sounds very interesting !
[Desire Android Development] melethron:
@sibere:

Almost forgot... the quick and improved version of your data2sd:

busybox e2fsck -yvf /dev/block/mmcblk0p2
busybox tune2fs -O ^has_journal /dev/block/mmcblk0p2
busybox losetup /dev/block/loop0 /dev/block/mmcblk0p2;
busybox mount -t ext4 -o sync,noatime,nodiratime,nosuid,nodev /dev/block/loop0 /data;

EDIT: short explanition about this. "Mount -o loop" doesnt work if it used on a PROPER device (it works in your script because of the loopfile - but it doesnt work in ownhere's script because he uses it on a REAL partition). To mount a real partition as a loop device you need to asign it a loopdevice and then mount that one.

Feel free to make a script out of this by adding the "cp" and "chmod" stuff.

Apart from that i did more tests with proper unmount. If it is mounted WITHOUT sync the system will also not unmount properly (device or resource busy ;) ) .

So this script is safe if:

- "sync" option is NOT removed

- snq powermenu NOT used

- adb reboot / abd reboot recovery / adb reboot bootloader NOT used

- battery NOT pulled

As long as all this is watched there should be no data loss. Shouldn't be so hard to copy+paste this info to the OP (you dont even need to mention me) and might spare some people for the risk of a data loss.

EDIT: Btw, i still hate FAT :D :D :D
[Desire Android Development] sibere:
got an easier option... link your post on the OP ;)

BTW why do you disable the journal with tune2fs instead of not creating it when you do you mkfs.ext4? mkfs.ext4 ..... -O ^has_journal .... and you shouldn't need the tune2fs line.
[Desire Android Development] melethron:
Thats from my ROM (next version v0.3). This way users don't need to care about this and can do the partition with recovery or whatever. Apart from that i made it possible to patch and unpatch the loop mount option. Flash the "life-on-the-edge" zip then ext4 without journal and loop mount is used. Flash the "life-to-the-edge-UNPATCH" zip and it is mounted non-loop with ext4 with journal.

Will release the ROM to tommorow (more testing first). Also my Benchmark results are finished and i make a big update in the former btrfs thread (may take some time till i want to write a big explanation of the results and need to calculate the mean values).

Best results (considering the sqlite inserts) of safe stuff was ReiserFS .

Best results of the more risky stuff was loop.

(btrfs - even with ssd optimizition - and nilfs was rather disappointing and the increased smoothness was only because of noop scheduler... :( ... i wasted some time on this. But ReiserFS is about 20-30% faster in sqlite than ext4 so that is the best alternative on the "safe" route)
[Desire Android Development] moebius83:
hello guys, really nice work here!

I started trying this mod on my stock rooted htc desire and it works perfectly, I chose the easy way for me by directly partitioning the SD with an ext2 partition and mount it under /data (with sync on), everything worked great but I'm noticing some huge spikes so I started reading the whole thread and I got a little lost :)

I seem to understand that these spikes will disappear if I use a loop file instead of the native ext2 partition (for some reason I thought it was the opposite), this because the loop device will get kernel caching. Is this right?

Also, will I benefit at all by using ext4 without journal instead of ext2?

Isn't having cache due to the use of loop device the same as using "nosync" on the ext2 mount command?

For what I understood, even if I mount the ext2 as a loop file with sync on, the loop device will be cached and not synched so the possible data loss can happen anyway and in the same way as mounting an ext2 partition directly with the "nosync" option.

Correct me if I'm wrong but after reading 98 pages I'm a little confused :)
[Desire Android Development] blowy666:
Hello everybody,

first of all, thanks for this great work!

I´ve tweaked my phone yesterday, and after some 2 hours and some problems, all went fine.

I splitted my memory card into 2 partitions with fat32...the one for data2sd with a size of round about 1,7GB.

After work, I have also "installed" the "big boost" from this site: http://androidforums.com/desire-all...sd-unlimited-internal-memory-your-desire.html

Now, I have a little question.

Everytime I install an app, my phone will lag...during the installation the phone is almost NOT usable, and afetr installation of the app I´ll have to wait a few seconds before I can use my phone in normal way.

Is there a work-around for this?

Or is the problem my memory-card, cause it is only class 2 ?

By the way, it is an 32 GB card.

Would be great if anybody could help me...?

And, last but not least: sorry for my english... ;-)
[Desire Android Development] sibere:
It has been shown that loop cache and file system cache are different. Apparently loop cache gives a bit more performance. If you don't want to change everything, you can at least remove the sync option. The risk of data loss is still marginal unless you remove the battery before the cache is committed to the SD card.

If you want the loop cache, then check the post linked on the OP.

EXT4 with no journal is more or less equal to EXT2 on the performance battlefield :)

I personally have ext2 ATM and I only plan to move to EXT4 to test the performance gain with the stripes and extends offered by ext4.

If I don't have any improvement, I'll keep ext2.

Good luck !
[Desire Android Development] sibere:
Even with class6 you can have a few spike lags so I easily imagine that even with all caches, a class 2 will give bad performance.

A version has been developed to use then NAND and the SD to spread the data access and trying to reduce those issues. look for it (data2ext or something similar)
[Desire Android Development] melethron:
Both will make things faster. Mount without sync and mount through loop with sync.

I tested loop for 2 days on a HD rom (those ROMs have the overall worst performance). While loop helps in some situations (writes to database) it is worse in other situations (a write and read of the data afterwards) so it isn't the "holy grale of smoothness" ;) . But both options are better than a direct mount with sync considered speed.

But it is better to mount through loop with sync: When you mount without sync the device may still be busy if the system is shutting down. Normally unmount forces a sync before the unmount is done but android is a stripped down linux :( . So mounting directly without sync can cause dataloss (and ext2 has no journal so you may get force closes after some reboots).

If you mount through loop with sync on the unmount works fine.

You can mount your setup through loop without the creation of the loop file.

Simply do "busybox losetup /dev/block/loop0 /dev/block/mmcblk0p2"

and then mount "dev/block/loop0" (with sync) instead of the mmcblk0p2.

These lags will not go away completly even on a class 6 sd (which has the same speed as nand) because an sd-card is different from the internal nand (block erase etc). It should be better without the "big boost" (which is only a fake boost and tricks quadrant).

I tested 6 different filesystems in various setups and none will solve the lag issue completly. The reason is also on the software side as explained by a google dev here. Maybe it will be better on gingerbread but there are also lags reported in the 2 threads of the ROMs that are already available.
[Desire Android Development] sibere:
Hum I was just wondering... What is the solution used by apple for their phones? After all, iOs is based on a Unix family OS, isn't it?
[Desire Android Development] melethron:
I tested nand sqlite speed. It has 50-60 sqlite inserts.

In my btrfs thread someone tested journaled ext4 on a class 6 card and the results were completly the same as on my tests with a class 2 card (both around 30). The issue with sqlite inserts isn't the "pure mb/s" of the sdcard but the generall way an sdcard works (LBA provided through an FTL by a controller).

So class doesn't matter at all. The only reason a better card could be better is the controller used for the FTL. But it is close to IMPOSSIBLE to get infos on those. A detailed description of this can be found here .

Btw. I'll think i will use "reiserfs" for data2sd. It has the best sqlite performance. While loop is definitly the fastest solution if you save something only (like clicking on the superuser "accept button") it is slower than reiserFS in situations like scrolling through the "titanium backup list"/"app drawer" etc. So as they are both almost the same considering speed i prefer ReiserFS because it is journaled (but this isn't my final judgement yet).
[Desire Android Development] sibere:
This sounds promising!

The thing is, even though it's a very good FS, ReiserFS is not supported ATM by recovery kernels while ext2/3/4 are.

We have also to keep in mind that users want to do a nandroid backup. if the 2nd partition in formatted to ReiserFS, they will have to do a backup manually with tar.
[Desire Android Development] sibere:
Did you try the other tutorial (around post 3) ?
[Desire Android Development] sibere:
Well ... It'll be show on the backup screen and you will get a huge 2GB file called sd-ext.img in the nandroid backup directory. This why I have limited my storage space to less than 1GB. It's more than enough and the backup won't take too much time to save/restore and too much space on /sdcard :)
[Desire Android Development] mattiadj:
Thanks I'll try and report the situation.

Another question :-) if I upgrade my ROM I must make entire procedure again?

And if I change ROM for example to try miui ROM or other?

sent from the future
[Desire Android Development] sibere:
Not completly. you won't have to format again the SD, but the rest yes. The new ROM installation will wipe everything out. If you just flash a kernel, nothing will be destroyed.
[Desire Android Development] mattiadj:
Yes I imaginated that for kernel radio or mod (everything don't need wipe) all remains intact so when for example new version of leeDroid will be relasaded I'll had to make all again.

sent from the future
[Desire Android Development] wollyka:
if i want to make a ext3 file instead of ext2 i have to type those commands right?

adb push c:\01data /system/etc/init.d

adb shell

chmod 755 /system/etc/init.d/01data

cd /

mount /dev/block/mmcblk0p2 /sd-ext

dd if=/dev/zero of=/sd-ext/ext3 bs=1048576

/system/xbin/busybox mke2fs -F -L userdata /sd-ext/ext3

mkdir /sd

/system/xbin/busybox mount -o loop /sd-ext/ext3 /sd

cd /data

cp -a * /sd

cd /

/system/xbin/busybox umount /sd

exit

adb reboot

and in data01 file i can comment out or remove this line

e2fsck_recvy -p -v -f /mnt/asec/ext/ext2

am i right?

thanks in advance
[Desire Android Development] beanbean50:
looks like a fantastic mod well done...!
[Desire Android Development] danarama:
Not yet.

Turns out it was a school-boy error. I was using an alternative tool to format the partition and it wasn't doing it properly.

Now using Gparted I get a lot further.

Now I get as far as:

mkdir /sd

/system/xbin/busybox mount -o loop /sd-ext/ext2 /sd before No such file or directory.

So close, yet so far... :)
[Desire Android Development] interqd:
hi... anyone tried this in miui rom ?
[Desire Android Development] sibere:
No, because mke2fs will make a ext2 file system that way.

Try with the -j option

/system/xbin/busybox mke2fs -j -F -L userdata /sd-ext/ext3

You should leave the e2fsck line but change it slightly.

I'm gonna update it anyway in the OP for this line:

/system/bin/e2fsck_recvy -y -f /mnt/asec/ext/ext2
[Desire Android Development] _bryan_:
done every thing and failed with miui

then i try to mount in recovery

i get this

/sbin/sh: /system/xbin/mount: Permission denied

but in step 7 and step 9 i can mount

why
[Desire Android Development] sibere:
Please read the tutorial more carefully

the command is /system/xbin/busybox mount
[Desire Android Development] fogbav:
Please let me ask my question again ... there was no answer on my last 3 tries ...

Is there a possibility to switch compression "on" on the EXt2 file System ?

It would nearly double the transfer rate from and to SDCard ... bad thing it would cost some cpu ticks on the SnapDragon - but 1 GHz CPU - no problem ...

Any comments on this ?

I'm not a linux pro ... so could someone point me there how to do that ?

e2compr schould do the magic, or i am woring ? how to implement for testing ?

Thanks for any answer ... 8)

How to get this beast in Leedroid Kernel thing ? 8)