yashsingh1/xdaforums huggingface dataset:
[Desire Android Development] HTF:
Some roms manually put things in the cache too if you have the partition table with a larger cache (oxygen does this for example with dex files).
[Desire Android Development] woti23:
only for people that can customize for their own configuration, please

if you put wildcards and -r in strings in it, you will have no fun ;-)

helps a little to test i/o

#!/system/bin/sh

from='/cache/android.data'; # source of file to copy,
# should be in RAM, should exist
n=5; # how often do yo want to
# repeat the test per filesystem
size='28MB'; # size of copy source in MB
btrfs='/system/sd';
ext4='/data/data';
nand='/data';
fat32='/mnt/sdcard'; # filesystems

echo -e "
copying $n times $size ";

for j in $btrfs $ext4 $nand $fat32; do
{
echo -n "
$j";
cat /proc/mounts |grep \ $j\ ;
rm $j/android.data &>/dev/null;

for (( i=0; i<$n; i++)); do
{
time (
cp -a $from $j &>/dev/null && sync;
rm $j/android.data; )
}
done;

}
done;
[Desire Android Development] melethron:
Additional info about UBIFS. To use it we need a UBI volume on the MTD NAND. That means booting without /system and recreate the mtd volumes as ubi. I will not mess with this .... maybe some day ... but not at my level of expirience atm.
[Desire Android Development] K900:
Just looking at some roms, it seems to me that compatibility is one hell of a problem...
[Desire Android Development] melethron:
I wanted to share some info about the scheduler.

While noop>bfq>cfq considering pure I/O benching. Supercurio had some concerns about this. Here is what he said in IRC

I think he really has a point here. But also there might be some improvements on other schedulers like noop if the queue size would be smaller. But this has to be done in Kernel source ....
[Desire Android Development] ownhere:
Beyond the scope of this title 'data2xxx'

Nevertheless, still thank you for your efforts, I read a lot of information on the FLT.

About data2xxx, I think there are several factors to consider:

1. stability, not based on experimental techniques, and can guarantee that under normal use and the incidents remained stable phone

2. generality, can be handled most of the Recovery or the application

3. performance, can not have too much Lag

4. battery life, avoid excessive power consumption

Based on the above points, I think data2btrfs or data2nilfs2 not good programs, for ordinary users, a2sd+ enough.
[Desire Android Development] ownhere:
additional information:

about sandisk microSDHC wear-leveling, I found a pdf doc:http://www.sandisk.com/Assets/File/OEM/WhitePapersAndBrochures/RS-MMC/WPaperWearLevelv1.0.pdf
[Desire Android Development] woti23:
so chances are, if we leave /data/data /data/app in mtd, the sdcard lasts longer than the mtd?

(or the other way round: mtd gets corrupt before the sdcard levels out?)
[Desire Android Development] woti23:
btw: google ditches yaffs2 for ext4

http://thunk.org/tytso/blog/2010/12/12/android-will-be-using-ext4-starting-with-gingerbread/
[Desire Android Development] melethron:
Because of 3 i do this ....

Because of 4 nilfs is not an option for me any more

and because of 1 and 2 i will never make a zip for this ..... this is meant for expirienced users that are aware of the risk. But what i think about your stability concerns .... well i posted that in your thread.

Thanks. But i did know that already. Thats way there should always be some free space (unpartitioned or unused) that wear leveling can take place. And also because of this its better to have 1 (!) partition so that the wear leveling can spread all across.

mtd is RAW nand wich has no wear leveling at all, so yes. But do you think you still use your desire in 10 years?

The reasons i see is this:

1. yaffs2 is outdated ****.

2. btrfs is still beta.

3. Using internal sd-cards will be cheaper than optimizations for RAW nand (software wear leveling). Thats what Samsung is already using (galaxy/galaxy s) and thats what HTC is using with their HTC 7 phones. The reason is rather simple. The reason it is cheaper is simple: MTD should have non or at least only few bad blocks. While the "bad block" count on production of SD-card doesn't really matter since it has free blocks (~ +7% extra blocks) to replace the bad ones. Some people already have problems with boot image on the desire due to bad blocks. Thats why snq made it smaller some time ago. So the mtd IN the sdcard can produced cheaper. Making sd in general cheaper.

BTW: Ext4 is the best (NON experimental) fs for ssd/sdcards. The question would be better if you ask WHY YAFFS2 in the first place.
[Desire Android Development] sibere:
That is comforting me in the idea of data2SD

I will try to move to ext4 noloop when I've got time and when I'm home :)

I just wonder what format/mounting options they're gonna use ;)
[Desire Android Development] melethron:
blocksize 4096 for format. Mount: noatime,nodiratime. Else is default.

And NO ... this is not your data2SD. No loopdevice and no ****TY fat. Well but its more like yours than ownhere's though...

Apart from that it is slightly different on SGS. /data doesn't get unmounted on their phone. So they make more than 1 partition since it doesnt matter. They mount cache/dalvik seperate WITH data=writeback cause corruption there doesnt matter.
[Desire Android Development] sibere:
you should read more carefully the OP of my thread again when I say that ext2/3/4 straight to the 2nd partition is another option instead of fat32/ext file and loop. (part 7-comments)

Anyway, I've started to play with ext4/nojournal/noloop and most of all, partition and ext4 data alignment to 128Ko blocs.

Results are uneven. ****ty quadrant gives me BAAAD score (more or less 700 I/O), Jbench is more optimistic, and dd is just fantastic with block from 1k to 1M !

I need to find a better benching tool :)
[Desire Android Development] melethron:
Op updated and completly new script.

About the other options ... i saw that ... But you still recommend the unsecure way. And all those that run your 01data are running this:

/system/xbin/busybox mount -o [B][COLOR="Red"]loop,sync[/COLOR][/B],noatime,nodiratime,nosuid,nodev /mnt/asec/ext/ext2 /data

Also try dalvik on internal for speed. That was a really nice idea of ownhere.

Most realistic I/O bench is "time (cp /whatever/* /whatever && sync)".
[Desire Android Development] melethron:
i have btrfs @ /data atm. And market works fine so i thinks its the logical partition. Btrf2 is really smooth so far. I only have lags when i download at full speed with (and my connection over wifi is almost as fast as my crappy class 2 card .... so didn't expect else.)

Btw: thanks for all the testing.

Edit:

This might also intrest you if you are still using btrfs:

Since a sd-card should have a FAR less expensive controller than a SSD harddisk this options should still improve things.
[Desire Android Development] ownhere:
I recommend that all developers do when the benchmark, have joined the sqlite3 insert test.

This is the real performance bottleneck.

Internal memory(NAND) can be achieved even if the average write speed of 5M, they still do, when inserted into sqlite3 surprisingly slow.

In the android system, almost every application have to use sqlite3, sqlite3 data files stored in the partition will continue to bear the torture fsync.

This is why sibera or ownhere's script use loopfile can get more smooth.

And I think this is why Cyanogen origin A2SD script not symlink /data/data/ to SD ext.

BTW:my script V4 use data=Ordered + loopfile-sync every effort to avoid security issues, Do you really read my V4 script? no WriteBack in mmcblk0p3. I deliberately disable journaling on mmcblk0p2, because it rarely changes, and will not cause FC in most cases.
[Desire Android Development] melethron:
And again .... sqlite is exactly were most corruption was found in the stress test the Samsung Galaxy S devs made. They did exactly what you did ... a loop device for sqlite database. That was one of the first speed improvement they made. They changed it because some users had force closes.

If you don't believe me i quote another user from your thread:

(link).

What you do has been done month ago. Supercurio and Unhelpful from Samsung Galaxy S devs have made so many test and my script is based on there result (with a slight adjustment for the desire bcs of /data unmount). I also posted that many times in your thread now and wouldn't have made a own script if you had listened to well known concerns. You do stuff that has been proven as DANGEROUS FOR DATA LOSS. For my own phone i rather use the experimental btrfs filesystem were no major bugs had been found by many linux user that already use it than a script that has been proven dangerous for month now. If you want to use it .... go on. If Rom devs want to use it ... well go on. I WONT.

So one more time: YES this is slower ..... but there will be no users with force closes this way ....... thats why i made this script .... and not for performance ....
[Desire Android Development] ownhere:
Calm down, I know your point of view. I just feel you misunderstood my point of view, so to make a clarification.:)

On the safe umount, symlink / data / data to SD ext is not safe I think. but your script at http://xdaforums.com/showthread.php?t=867049 do. This may be a bit contradictory .

We are very concerned about data security, but I hope the security situation in trying to better the user experience, and nothing more, Your security is far more concerned about more than performance, I understand
[Desire Android Development] melethron:
No offense. Im calm. ;)

But i really think we should relly on findings from other devs here on xda.

I also know that my old script is bad. I left it there for those that still want it but said that there will be no support. This script was before i was in irc of SGS devs. Also robocik asked me if it is safe and he should use it and i said NO. Also about the test of SGS devs i place a quote here.

Here he explains the test they made (snipped quote):

and this is what i just asked some min ago:

As he said above. Corruption was rare on normal user experience. But they wanted stabilty as their main goal. Maybe your script will not corrupt data for 100 users. But i don't wanna be the 101th ;)

I really appreciate all the efforts you make. But think about. Most data that can go lost is not really important. But what if it's a safegame a user needed hours to achieve it .... or maybe a license key that someone has already lost and only maintains the license with titanium backup .....

I know that my script is slow and will be a bit laggy on a slow sd card. Its only meant to be the most secure way to go. I made backups (on my laptop and on dropbox) before i startet all this testing so i feal safe enough to try btrfs. But i would also not recommend to use it for daily use. And if i do this then i maintain several backups.

EDIT: just removed my old script.

Also please go on with your work. You have some nice ideas. :)
[Desire Android Development] ownhere:
Thanks for your UBIFS information.

This is great FS for NAND.

We can start with replace /data(mtdblock5) and /cache(mtdblock4) with UBIFS over UBI to test performance.
[Desire Android Development] HTF:
I am liking the look of btrfs more and more, going to give it a try myself shortly and will report back.

Could you please link to the kernels in the OP as I had to re-read through the whole thread to find them again.
[Desire Android Development] coutts99:
Just been reading up on this before I read this post, was going to have a play with it myself.
[Desire Android Development] sibere:
Yep i still recommend even tho I wrote a BIG disclaimer. ;)

My phone is running on ext4 no journal no loop atm

As per the cp bench I don't really agree. if you do a cp from /sdcard/ to ext /data, the bench will be wrong because the SD will be used R/W. If you do it from nand to sd, the bottleneck can be the NAND read speed.
[Desire Android Development] woti23:
mkdir /data/tmp

mount -o size=50000000 -t tmpfs tmpfs /data/tmp

50MB temporary filesystem in RAM

cp your testfiles to data/tmp and than make your speedtests through copying from there to your to be tested filesystem
[Desire Android Development] melethron:
Completly rewrote main post ... Kernels there and much more info.
[Desire Android Development] yukkio:
I'll probe this when i will be at home. It Sounds great :)

BR.

EDIT: If u update your script I have to do a wipe all or just overwrite the script?
[Desire Android Development] melethron:
@th3: Wow, thanks for sharing all your thoughts.

I didn't check all your links yet but will do.

About the bad performance on SSDs please keep in mind that:

- The controllers of modern SSD are getting better and better in simulating a ROTATING device. This makes all ssd optimization perform even worse.

- We can't even compare this with new usb-sticks. USB bus allows performance of up to 60mb/s while the specification of microSD cant get above 25mb/s. A good USB stick should also have a far better controller then micro SD.

-We can't even compare microSD on its own. There may also some difference between a cheap class 2 and a state-of-the-art class 10 sd considering the controller.

Considering these point all we can do is testing (i didn't find any test with btrfs on microSD's). I also made many assumption about how nice nilfs would be only to find it SUCKS heavy in battery usage (+1%/h even when the phone is idle).

- Exactly the point about small files and sqlite performance is what we need here. Cause data2sd is about moving /data/data to external and this folder contains the sqlite files. This is why data2sd/data2ext is laggy on a poor microSD.

Hope i didn't miss something. I'll read through your post again and check the links. Also i suggest you simply try it and compare it to data2ext4 and share your findings.
[Desire Android Development] melethron:
Dont know yet. For now only "dalvik-cache" is on internal. If there are some changes to this then you need a wipe. If i only make minor changes then there is no need to.
[Desire Android Development] sibere:
Ho year very clever ;)

That's actually the case on my desire for browser cache :D
[Desire Android Development] melethron:
How large did you make it? Do you "feel" an actual improvement

Btw did you try moving the app-cache to mtd?

I want to this later but want to edit a HD rom first that i can use it with btrfs (no install on sd-ext and using the almost empty /data instead -> then copy with init.d to btrfs (btrfs in kernel then))
[Desire Android Development] melethron:
@th3:

I just read through your post again and want to address some more points.

Yep its really hard to find info. But i found this leaked information:

www.dtt8.com/images/micro-sd specification.pdf

According to this the channel count is max 4 for microSDs. Also the min channel count should be 2 as long as either read or write is higher than 6,25 (25/4). For my class 2 this applies. I think we should at least estimate that there are at least 2 channels on all microSD (well might be only one on smaller/older cards).

I also had thoughts about this but at this point i'm completely unsure about this. I really would like to hear what you think about my thoughts since your post looks like you may have some knowledge that might help.

Since we should estimate that we have at least 2 channels on the sd-controller we have some kind of Raid-0 between 2-4 nand banks.

If we consider this the scheduler shouldn't do anything and leave r/w scheduling to the sd controller and this leads us to: noop

But im really unsure about this and supercurio of sgs dev was also sceptical about this. He said that this depends also on the queue size (other stuff slows down bcs I/O has more priority). I also have no expirience with Raid-0 on linux so i have not the slightest idea if im right on this.

HOWEVER i have compare "noop" with "cfq" for now and at least under heavy sd card usage noop performs MUCH better. But i didn't check any other scheduler yet. And also should test if noop is a good way to go if there is NO heavy load on the sd.

I also looked into the queue settings. If someone can help me how those affect scheduling i would be very pleased:

bash-3.2# cat /sys/block/mmcblk0/queue/minimum_io_size
512
bash-3.2# cat /sys/block/mmcblk0/queue/optimal_io_size
0
bash-3.2# cat /sys/block/mmcblk0/queue/physical_block_size
512
bash-3.2# cat /sys/block/mmcblk0/queue/rq_affinity
1
bash-3.2# cat /sys/block/mmcblk0/queue/nomerges
0
bash-3.2# cat /sys/block/mmcblk0/queue/nr_requests
128
bash-3.2# cat /sys/block/mmcblk0/queue/hw_sector_size
512

Btw didn't change any value set (some are hardware at least the last one ;-) )
[Desire Android Development] navalynt:
I compiled the CM6.1 kernel for the CDMA Desire with support for ext2, ext3, ext4, reiserfs, xfs, jfs, btrfs, and nilfs2. I have it attached along with my init.d folder which has my data2sd script I've been slowly optimizing the past couple of weeks.

I've purposely noted the hell out of the script to try and make it easy for others to follow along and adapt to their own purposes.

Can someone zip up the binaries for btrfs and perhaps include their script(s) as well so I have a baseline for at least optimizing the data2sd portion of my own script to incorporate btrfs?
[Desire Android Development] sibere:
Wow yep very good job indeed.
[Desire Android Development] sibere:
I didn't do anything regarding browser cache in a tmpfs. My desire sense rom came with this feature.

Actually /dev, /mnt/asec and /app-cache are a tmpfs
[Desire Android Development] HTF:
There is an error in the OP, you state that the image you post proves it should be faster than EXT4 but that is not true as the smaller bars are better and that is number of seconds to complete the benchmark and not number of operations. This means according to that graph EXT4 is faster for SQLite, the article also talks about this.

I however do not see how they end up with those results, in theory EXT4 should not really be faster and not by that margin.

Edit: also you have only linked the data2ext4 file and not the btrfs
[Desire Android Development] NeoPhyTe.x360:
Me like I/O tweaked like this for the moment, Have noticed little improves, over all in Stability, and maybe 30-40 points in quadrant, but not much noticeable

#!/system/bin/sh
#
# NeoPhyTe.x360 Optimization Script

PART=`mount | cut -d " " -f3`;
BLK0=`ls -d /sys/block/mmcblk0*`;
BLOCK0=`ls -d /sys/block/mtdblock0*`;
BLOCK1=`ls -d /sys/block/mtdblock1*`;
BLOCK2=`ls -d /sys/block/mtdblock2*`;
BLOCK3=`ls -d /sys/block/mtdblock3*`;
BLOCK4=`ls -d /sys/block/mtdblock4*`;
BLOCK5=`ls -d /sys/block/mtdblock5*`;
LOOP0=`ls -d /sys/block/loop0*`;
LOOP1=`ls -d /sys/block/loop1*`;
LOOP2=`ls -d /sys/block/loop2*`;
LOOP3=`ls -d /sys/block/loop3*`;
LOOP4=`ls -d /sys/block/loop4*`;
LOOP5=`ls -d /sys/block/loop5*`;
LOOP6=`ls -d /sys/block/loop6*`;
LOOP7=`ls -d /sys/block/loop7*`;
VIRTUAL=`ls -d /sys/devices/virtual/179:0*`;

echo "TWEAKS: use CFQ as i/o scheduler and decrease latency";

for i in $BLK0 $BLOCK0 $BLOCK1 $BLOCK2 $BLOCK3 $BLOCK4 $BLOCK5 $LOOP0 $LOOP1 $LOOP2 $LOOP3 $LOOP4 $LOOP5 $LOOP6 $LOOP7 $VIRTUAL;
do
echo "cfq" > $i/queue/scheduler;
echo "0" > $i/queue/rotational;
echo "4096" > $i/queue/read_ahead_kb;
echo "1" > $i/queue/iosched/back_seek_penalty;
echo "1000000000" > $i/queue/iosched/back_seek_max;
echo "0" > $i/queue/iosched/slice_idle;
done;

exit;
[Desire Android Development] NeoPhyTe.x360:
I think the article is done without SSD option of btrfs, sure.
[Desire Android Development] sibere:
ext4 when copying all the dalvik-cache from a tmpfs to /data is half speed compared to NAND :(

ext4 even aligned+nojournal is not a solution for me ATM.
[Desire Android Development] woti23:
yes, its setup in the init.rc and its content directory com.android.browser/cache gets the permissions and the ownership of the /data/data/com.android.browser there

if this fails (eg with command chownto missing) then you have the

"cannot create webViewCache" lines in the logfile
[Desire Android Development] HTF:
Yes I think you are correct.

Just out of interest is the kernel you posted for sense or aosp?
[Desire Android Development] woti23:
how did you mount /data as ext4, which options?

did you create the tmpfs new after reboot,, because its gone then?

i have the opposite, tmpfs to nand is about 1,5 times slower than tmpfs to ext4

and i have only a class 2 card in phone
[Desire Android Development] sibere:
ext4 was aligned to a 128Kb stride and a 4096b block size, build without journal.

mount was noatime,nodiratime,nosuid,nodev,data=ordered

But the mount command was reporting data=writeback barrier=1 tho.

Obviously tmpfs was new after boot, with 37Mb of dalvik-cache data.

I did the time cp with a time sync at the end and added the 2 times to get the full write to sd time and compared with the same to /cache (nand)

Even ext2 is slower than nand is this "real life" situation. On the opposite, when writing to sd with dd, I've got a better score with ext2/4 than NAND

I've tried both with cfq and noop as io scheduler but didn't notice any major improvement.

Even though it's not reliable, quadrant gives me about 700 I/O with ext4 and 4000 with ext2.

What options did you use to format the ext4 partition?

I'm gonna try again ext4 without the stride option
[Desire Android Development] woti23:
i don't know what you mean with 128k stride?

i did partition with gparted/parted commandline, which has a align-check

aligned it automatically with gparted to MiB.

mkfs.ext4 without options creates default filesystem with b=4096

and journal

mount -o data=ordered,noatime,nodiratime,barrier=1,nodev,nosuid /dev/... /mountpoint...

mount shows this options
[Desire Android Development] sibere:
dunno why but even with data=ordered, after mounting, I have data=writeback.

Could be a kernel issue in ext4 options that I have to address.

tried again ext4 with the OS running

copied 38mb of dex files from tmpfs to /data = about 17sec (ext)

same to /cache = about 11s (nand)

and this is with a class 6 cards which gives me about 10Mb/s max on the FAT32 partition.

edit: with a 30mb file from tmpfs to ext4 = less than 4 sec and to nand= less than 8sec

but on a bunch of 10mb of 300Kb files, NAND is slightly better than ext4 on SD (by 1 sec or so)

I'm going to try this bench tool designed for SGS lagfix users : http://xdaforums.com/showthread.php?t=850454

It looks quite good because it's doing many different tests like file copy, database stress tests etc.
[Desire Android Development] woti23:
yeah it has detailed tests, but in result it compares different phones and not different filesystems, from desire standard to galaxy s with ext2
[Desire Android Development] sibere:
Yep but the developer is working on a direct I/O version, bypassing the cache. Could be interesting :)
[Desire Android Development] melethron:
Sense ... sry i update op. Thought it was clear what those .....

Stop it ... people might notice ..... i just looked for some nice pics xD

Lol, well i really didn't take a close look at the article. But i really notice some major improvements if data is written to /data/data. One example is if you click on a button that save some settings. I always had lags on this when i used ext4.

Still should remove this .... sorry for accidental fooling people ... i fooled myself with this....

I just searched an explanation why the lags are gone on /data/data writes....and find one by looking at this pic ....

Hey sibere mount doesn't accept this options. You need to set it with tune2fs. Thats at least what ownhere said when i asked him why doesn't simply use the mount option instead of tune2fs ....
[Desire Android Development] woti23:
well, that would be strange, because these options are in the man pages of mount

at mount options for ext4 ...
[Desire Android Development] woti23:
it was the logical partition or android can't count numbers higher than 4 ;-)

market working with btrfs on mmcblk0p3 (3rd partition primary) instead of mmcblk0p6 (2. logical on 4th partition extented)