yashsingh1/xdaforums huggingface dataset:
[Desire Android Development] melethron:
Well, ext3 is actually ext2+journal. The main difference considering lag (caused by sqlite inserts) is hardware barriers. Ext3 uses barrier=0 as default and ext4 uses barrier=1 as default. The difference is then this:

Barrier=0 on ext4 makes it

even better

than ext3. Example bench (lower is better):

Ext4 (barrier=1): 1690.87s

Ext3 (barrier=0): 108.77s

Ext4 (barrier=0): 96.48s

But i compared barrier=0 and barrier=1 on my sd-card and it didn't make a difference for me .... But since some people have a real bad performance on ext4 and some have no issue with it i think the difference lies in the controller then.......

If a sdcard controller doesn't support barriers then one wouldn't notice any slowdowns even if barrier=1 is set. So i think that your sd-card supports barriers while mine doesnt....

With that said i make "barrier=0" the default mount option in the next version (v0.5). Its a bit more risky but this shouldn't be a issue on an ordered journal.
[Desire Android Development] woti23:
12500 inserts in about 20 minutes seems not the typically usage of a smartphone?

is every insert synced from db memory to disk after db is written, so that would say something about the time from insert in db to availabillity on disk? otherwise it would say more about db cache/memory performance
[Desire Android Development] hajen78:
I am not sure but dosent this mean that ext3 is just as good as ext4 ?

i just thougt that maybe it would work on my phone with ext3 instead of ext4 wich dont work on my phone ?

Edit* I just treid ext3 and that worked, so i would like to know if that is as good as ext4, because then i will backup now ????
[Desire Android Development] anko184:
Is it possible to add data2whatever in a working Desire rom, as with sebere's script?
[Desire Android Development] hajen78:
Now i got it to work on ext4, but when i run a quadrant test it stands still on I/O 3/4 database writes, and it stands there for about 2-3 minuttes, is that normal ?
[Desire Android Development] moebius83:
you mean a stock rom? it's possible but it's strongly suggested that you use a stock rom with init.d properly enabled
[Desire Android Development] anko184:
No i meant a rom that is already running on desire. I now use Z rom from robocik but it puts files on ext during flash and if i run d2we then the phone boots but sense can't start (not FC).

So i wanted to know if there is a way to use d2we as sibere does with some copying. Check OP on his thread.
[Desire Android Development] sibere:
You know how strange benchmarks can be.

But I just tried to move everything from FAT32+file+loop(sync) to a 1MB aligned ext2+loop(sync).

Although quandrant score remains the same, my TAP benchmark has dropped quite significantly .

But now, even when writing heavily with dd (with a adb shell) to /data, I have no spike lags at all.

I've tried many block size from 4k to 1M and 20 to 30Mb each time. No spike lags. (To test I was swapping the main menu from left to right and right to left).

As a reference, my kernel has cgroups cpu scheduler and atm the I/O sched is CFQ.

Edit: I have managed to freeze the phone but I had to use 1K block for 200000 (200MB) times which should be a really unusual and remote situation. ;)

I'm going to try with deadline too and let you know if it's better or worse.

Last Edit: No difference btwn deadline and CFQ. Probably due to the loop device.
[Desire Android Development] Terepin:
I'm going to ask here: which EXT version is best for my ROM?

Thanks in advance.
[Desire Android Development] melethron:
Yes thats not typical but i felt a strong correlation between sqlite inserts and lags. The higher the insterts the better the smoothness.

If you are on ext4 atm could you try one bench with barrier and one without.

i just compared (remount):

/dev/block/mmcblk0p2 on /data type ext4 (rw,nosuid,nodev,noatime,nodiratime,barrier=1,data=ordered)

and

/dev/block/mmcblk0p2 on /data type ext4 (rw,nosuid,nodev,noatime,nodiratime,barrier=0,data=ordered)

on both i got 36 writes/s in tap benchmark. Since your card was slower on ext4 this could be the difference. This may also explain why i have a higher score on writes with ext4 on a class 2 than others on a class 6 card.

This test takes a bit longer than the others but it shouldnt take 2-3 min. If you have adb can you try (while android is running/system is up - not in recovery and not at boot):

adb shell busybox mount -o remount,rw,nosuid,nodev,noatime,nodiratime,barrier=0 /data

And then run the benchmark again and see if the smoothness changes for you.

Ownhere and i noticed the same. Direct loop mount is better than loopfile. But i had same "tap benchmark" db writes/s if i did:

-ext2 loopfile on fat

-ext4 loopfile on fat

-ext4 loop direct

-ext2 loop direct

-tapbench data dir symlinked to RAM

I didnt compare the other results (only db write).

Btw. Im not sure about scheduler on loop devices. I dont know how loop behaves in conjunction with scheduler.
[Desire Android Development] sibere:
My DB writes ATM is 183(6)Writes/s but again the main thing is more space and no lags, not the higher quadrant or TAP bench score isn't it? ;)

So far in this configuration, I have no lags. Maybe I feel a little slow down of the phone but it's lot lagging. I'm now gonna do the simlinks for the "unsaved settings" bug and I'll leave it this way for a while to test on a longer term basis. If I have no lags even when installing big apps from the market, I'll stick with this configuration. :)
[Desire Android Development] melethron:
Ext 3 is the same as ext 2 but with a journal. Ext4 is a bit different and has also a journal.

EXT4 is the best in general (extends feature for large files).

If there a slowdowns (compared to ext2 or ext3) these are caused by hardware barriers. I think this may be hardware related and if barrier causes the slowdowns that some have from ext4 i will change it in v0.5.

I cant test this myself because i noticed no slowdowns on ext4. But others do ...
[Desire Android Development] Terepin:
I was asking just because I'm using Z 0.6 and I'm having lot of laggs with class 6 card. I tried changing from ext2 to ext4, but I saw only small improvements in Quadrant. Also tried different kernels. coutt's was the worst, LeeDroids and ziggy's the best (~1420). Also, ziggy's BFS kernel feels snappier than others. I guess my Desire is the limit. And those laggs are driving me crazy. Unfortunatelly, I can't use your script because of Z.
[Desire Android Development] woti23:
middle over 5 tries - db writes, 32GB sandisc class 2

TAB bench

on nand i get - 54 writes / s

on ext4 with barrier=1 - 13 / s

on ext4 with barrier=0 - 12 / s

write from ram 52MB

(50 files <10k | 36 files between 10k and 100k | 6 files between 100k and 200k | 6 files > 200k)

to ext 4 with barrier =1 - 2,7MB/sec

to ext 4 with barrier=0 - 3,2MB/sec

dont know what this TAB bench comes from with this low values, but the write speeds with "real live"-files from /data/data seems plausibile
[Desire Android Development] melethron:
Yeah i dont care about numbers. With a 2 line script i can get the highest possible r/w score in a bench (mkdir in /dev, symlink) ^^

The reason i care about tap benchmark write scores that there is a really STRONG relation between the sqlite inserts and the smoothness. And as this google dev tells the same i think im right on this....

Tap benchmark db writes is a good estimate value of smoothness. The only issue is that the values for "loop" cant be counted normally because they skip the fsync() or commit () (flush to ram and not to disk). As i already said this helps in SOME situation in "REAL LIFE" because you dont have to "wait" until the commit() is done (and it will be done when loop cache is flushed to disk). This way the real disk flush will happen in the background and not in the main thread (aka the app that just lags - the delay you WOULD notice is actually just there but happens in background). But then again it causes slowdowns if the written value is recalled. On my card i didnt feel a overall improvement from "loop". If some get better results, well its in the script and can be used with life-on-the-edge....

Atm im really interrested how barrier settings have an effect for others. For me unjournaled ext4 is faster than ext2 and journaled ext4 faster then ext3. So if you have slowdowns on ext3/4 i would be interrested if barrier=0 makes a difference for you.

Also, did you compare "deadline"?

Apart from that i have no lags at all with journaled ext4, barrier=1 and deadline on a non HD rom. I do most script testing with leedroid and i dont get any lags (ill downloaded "speeking santa" for about the 30 times by now :D ) . On DHD roms i do get some lags simply because it uses more ram etc. My interest is not what is the best setup FOR ME (i'll use ReiserFS and swap partition on my DHD rom if im done with the testing). What im interested in now is why some people get horrible lags with data2whatever on leedroid while i and others dont get them ..... THIS is what interest me atm.... and since i cant visit those people and check the smoothness of there phone ^^ .... i'll try to get an estimate through "tap benchmark".....
[Desire Android Development] woti23:
well but then i get no lags (but i use most of the time the phone for ssh thingies ;-)

and have extrem low db writes except on nand.

but then: with sqlite other issues come into the szene:

sqlite has a journal (for rollback to before powerfail eg)

sqlite has also default autocommit (commits for db are auto done when connection is closed) and

sqlite has transactions (break out of autocommit and wait until cache is full)

its said that reiserfs with its variable block size would fit sqlite well. could be the reason that i had the best test result after nand with reiserfs (28db writes/s)

i don't want to doubt the TAP bench devs, but i think this is a really really difficult approach to test
[Desire Android Development] melethron:
Thanks for the test.

for me:

nand : 53,8

ext4 barrier=0/1: 36

reiserfs journaled: 40,2

unjournaled ext4: 57

loop in every setup and ram are all ~200

Hmm so its not barrier that causes the 3 x write difference.... :(

What scheduler did you use ...
[Desire Android Development] sibere:
Yep and I felt no difference. I'm sure that it's because I'm using a loop device.

When I did my stress tests with DD, I did a time busybox dd if=/dev/zero of=/data/testfile bs=4k count=8000 && time sync && time sync to get a synced FS and the full time needed by the dd operation. This is more than 30Mb of data and the system has no choice but to flush it to the SD with the sync command.

During this, I was playing with the phone and even during the "sync" command, which lasted +/- 8 secs, I was feeling no lag at all and this is the main target for me after the extra space.

My card is a 8GB Transcend Class 6
[Desire Android Development] hajen78:
I try asking again when i use data2whatever on ext3 or ext4 it hangs on I/O writes 3-4 minuttes in quadrant, is this normal, i dont think about the score i get, but i cant imagen that this is normal ? Am i doing something wrong here ?
[Desire Android Development] melethron:
The ROM seems also make a huge difference. After i aligned the partitions and use noop or deadline i didnt manage to get lags in leedroid. Every setup worked fine. But thats different from a DHD rom. There i noticed the difference in smoothness. And reiserfs felt best (and got the best score).

But i still wonder why people get lags on leedroid with data2whatever ( ... or those HUGE slowdowns on the quadrant db write test ...).
[Desire Android Development] melethron:
As i already said (you may have missed my answer) this test is the slowest for me but its not THAT slow (~10sec).

Could you download that benchmark:

http://xdaforums.com/showthread.php?t=850454

and post the values of the "DB writes" (its in the "benchmark log" or in "view detailed results"). Make 3-5 runs for this.

And if you have a adb it would be nice if you do the following (no restart or anything needed - just do it after the bench):

adb shell busybox mount -o remount,rw,nosuid,nodev,noatime,nodiratime,barrier=0 /data

And then bench again.

As i said loop helps on pure writes... even if it is flushing to disc there is no commit or fsync you need to wait for in the thread you are in.
[Desire Android Development] moebius83:
since the conversation here moved to performances I decided to give a shot to the TAP benchmark:

EXT2 (sync, but using a loop mounted partition, NOT file): 183 writes/s, 276 read/s

the phone feels smooth, it lags sometime , especially when I open the Settings, but it's acceptable.
[Desire Android Development] melethron:
Loop score is the same as for RAM because the benchmark only syncs to the loop device and not to the real sd.
[Desire Android Development] woti23:
used noop

ram i get about 190
[Desire Android Development] woti23:
when i have low writes, (ext4,btrfs about 10-13) i get 280 dbread (journaled, barrier=1)

at nand with 54 dbwrite i get only 195 dbread

strange isn't it?

maybe i'm going to loop mount /data/data and do an auto rsync-backup (from /data/data to the 3rd partition) at boot after fscheck (takes 2-3 min more at boot, but if fscheck fails i have the /data/data of the last boot before)
[Desire Android Development] melethron:
Read ahead? Controller handling vs yaffs2 ....

About backup: one week scheduled titanium backup is enough for me. For loop just don't drop the phone (in a way the battery jumps out) and stay away from restart tools. Loop is only really risky on powerloss ...
[Desire Android Development] moebius83:
and system crashes...
[Desire Android Development] woti23:
i don't pull out the battery at bootloops. i repair bootlops always in the shell.

the only thing that could happen would be a wrong mounted partition that is too busy to umount. will happen, i'm sure.
[Desire Android Development] melethron:
... and system crashes :)

But this shouldnt be something that happens to often. I only managed to get one one time. And that was when i made a cpu stress test at a too high overclock.
[Desire Android Development] woti23:
readahead NAND 4 (htc default, does not get better for me with higher values)

readahead microsd partitions: all 64 (or 128 get same values) tested is the optimum

backup: your right once a week would do, but i like to have small different backups from different places on the phone to different medias at different times. smaller risk when good backup is overwritten by bad backup (yeah, happens)
[Desire Android Development] melethron:
Same for me. Bootloops are easy to fix in most cases. And if you need to reboot then you can still umount /data before. If you mount with sync the partition should always get unmounted. I didnt manage to get it unclean one single time.
[Desire Android Development] hongkongtom:
confirmed worked on Modaco MCR r9 and r10 (for v0.2e)
[Desire Android Development] hongkongtom:
both v0.3 and v0.4 didn't work for MCR r9 and r10, (might be I did something wrong),

if melethron could make a v0.2 verson with dalvik-cache moved back to internal would be highly appreciated. experiencing some lag with my MCR r10 and 16gb class4 SD.
[Desire Android Development] sibere:
After many tries and app installations I can say that lags are equals to loop+file+FAT32.

But now that I've moved to Loop+EXT2, I'm not going to roll back to the previous state ;)

I'm gonna try dalvik on NAND to see if the spread of load reduces lags during app installs.
[Desire Android Development] Richy Freeway:
Using a stock froyo rooted with init.d, created aligned fat32 and ext4 partitions with gparted. Completely wiped phone, flashed image, then flashed 0.4c. Reboot.

All I get is the HTC logo cycling.

This is the contents of the data2whatever log file.

Any suggestions?

Edit : Just tried it with the 0.2 version and I get exactly the same problem.

I'm using the 2.29.405.5 v3.1 rom from http://xdaforums.com/showthread.php?t=773997
[Desire Android Development] redecs:
Same rom, and same problem here. Using v4.0c.

I've attached the logcat file. There is a problem with mount the /data partition from what i can see.

// edit: logcat was to big, and i misse the error say that. attached smaller one now.
[Desire Android Development] melethron:
Hmm i cant see what changes could cause 0.3 and 0.4 not to work. Ill flash the rom the next min (nandroid before and then i flash it).

The changes to dalvik-cache aren't good as they are because they can cause issues if someone has MANY apps. I'll rework this. Apart from that i wont continue the old version. I may make a version without alignement but this will include the changes i made in 03 and 04.

It should be faster at least in theory because:

flush to loop -> flush to file -> file flush to disk

should take more time then:

flush to loop -> flush to disk

But im not sure about this. Still direct loop mount is easier to setup so ...

Btw if you move dalvik to nand watch that it doesnt get full. I'll only move the system@ ... dex files to internal in the next version. Dex files take about 1/2 the size of the apps so it can cause issues if you have more than 300mb of apps.

This seems like it cant access the ext partition. Can you do a "adb logcat" while it boots.....
[Desire Android Development] melethron:
Logcat would help greatly.... but you forgot the attachment ;)
[Desire Android Development] redecs:
seen that, logcat added.
[Desire Android Development] melethron:
Issue on MCR and the stock ROM is missing ext4 support. I will change the format after repartitioning to ext3. I'll release 0.4d in the next hour.

In the meantime you should format the sd-ext with ext2 (while in

recovery

do):

"adb shell" and then:

mke2fs /dev/block/mmcblk0p2
tune2fs -O has_journal /dev/block/mmcblk0p2

Then you can reflash (after full wipe) the ROM and data2whatever (works with 0.4c then since the partition is aligned and wont be formated in ext4).

If you dont have "adb" or another tool like gparted or something you need to repartition the sd-card with recovery (sorry) and wait for 0.4d.

EDIT: well i would recommend to wait for 0.4d in any case. Will be released the next minutes.
[Desire Android Development] redecs:
I've done a wipe, reflashed the ROM and your script and the executed the two commands.

The phone gets stuck with the "Unlocking SIM card and setting up your phone" message.
[Desire Android Development] melethron:
:/ .... the logcat looks fine now. Partitions get mounted and it should work. Can't see where the issue lies.... Looks like i have to flash another ROM (just flashed MCR to see if it works :/ - it worked by the way) ...

Hmm i wanted to update my ROM first so i dont have time to flash another ROM now .... I'll do this in the next days .....

EDIT: OP updated with 0.4d by the way.
[Desire Android Development] _bryan_:
I have EXT4 aligned if i use 0.4d then the script will format ext partition to ext3?

but ext3 slower than ext4
[Desire Android Development] hongkongtom:
Dear melethron , thx for the great works, 0.4d works on MCR r10 WIP, a question to ask though, will Nandroid backup still work? ( I use clockworkmod 2.5.0.7)
[Desire Android Development] vivmar:
hi melethron

don't know why but after some perfect hours of utilization,of rom miui-au with your script,the problems started.

I've got 10 minutes of crazy reboots one after another without reasons.Then leveled off.

This morning same thing for 15 minutes,disapperared some apps and widgets from home...???

tryed different kernel but no luck..

can u che my log ,maybe u easily find the problem?

thx
[Desire Android Development] melethron:
sorry for late answers. I was working on my ROM.

It wont change ANYTHING if it is already aligned. So dont worry :)

Also note that for some ext4 is slower than ext3 (but i have no idea why :confused: )

Yes. All Clockwork version back up the complete sd-ext partition (mmcblk0p2) so it works fine with data2whatever.

AmonRa also works since 2.0.1.

What filesystem do you use?

Also data2whatever shouldn't cause reboots (but still those reboots caused some issues on the partition). . Did you do something special before it rebooted. Did you flash a new kernel or overclocked or something?
[Desire Android Development] vivmar:
EXT4

I remember that i was playing with setcpu and he started with reboots.

kernel was the stock in miui-au at 1152mhz (my phone work with this mhz)

nothing else...
[Desire Android Development] melethron:
It is possible that your phone is stable with one scheduler on 1152 and not with another. Also the kernel can be different. My cpu is stable at 1113 on a SVS Kernel but not on a HAVS Kernel. Also there are different UV levels that even affect the OC. If the cpu clock jumps up to oc from a low clockspeed with uv then it can get unstable. Also the stresstest of setcpu is heavy (i also thought my cpu is stable on 1152 until i did this stress test ;) ).

So i'm pretty sure the issues lies there.

But still this reboots arent good for your partition. E2fsck fixed all errors but this can still cause some issue. You should revert the changes you did in set cpu.
[Desire Android Development] hajen78:
In both cases it says DB writes 0 (0)
[Desire Android Development] melethron:
:eek::eek::eek:

i HIGHLY recommend that you use ext3 instead....

Wipe everthing and then do "adb shell":

mke2fs -m 0 -L userdata /dev/block/mmcblk0p2
[Desire Android Development] hongkongtom:
wired things happened, felt quite slow with my class 4 16gb toshiba sd and changed it to a newly bought class 10 16gb KingMax, but don't know why, just can't get it to work with either 0.2e or 0.4d. force closes!!!!!

tried copied everything to the new sd and Nandroid restore, started up but force closed at sense.

tried wipe everything just left clean fat32 and ext3, doesn't work either.

have anyone encountered this kinda changing sd card issue?
[Desire Android Development] sibere:
Some issues have been reported with class 10 cards on the desire. Could be more a hardware than a software problem here.
[Desire Android Development] moebius83:
I noticed that since changing the scheduler to deadline (instead of noop) I'm having much less issues, no FC, no sudden reboots.

just my 2 cents
[Desire Android Development] melethron:
I did stress tests with "noop" on my card and it didn't drop a single i/o request. But i thought that this might not be the case with every sd-card. Sadly there are some kernels out there that don't have deadline... hope there are no issues because of this.
[Desire Android Development] melethron:
Well this doesn't have to be related with the class directly. I think its more an issue of the controller (FTL etc)...
[Desire Android Development] hongkongtom:
Oh, what is controller then? Anyway I could solve this?
[Desire Android Development] melethron:
Its something inside the microSD. A microSD is actually a NAND flash. But it doesnt "act" like one. It simulates a LBA (acts like a hard disk). The manufactors of microSD (or ssd devices in general) keep the info how the controller works secret.

For additional info check page 3 and 4 on this:

http://www.usenix.org/event/lsf08/tech/shin_SSD.pdf

and there is also some info in this article:

http://lwn.net/Articles/353411/
[Desire Android Development] hongkongtom:
Wow, really appreciated melethron for the info, but looks like this thing just too technical for me (looks like cryptic to me :S ) so, what can I do? Should i just leave my newly bought class 10 card alone? Or is there other way to fix this?
[Desire Android Development] s-murphy:
melethron,

Good work ;), this is MIUI_Au 0.12.24 R4 with your latest script.