yashsingh1/xdaforums huggingface dataset:
[Desire Android Development] sibere:
I have used loop so far because I have run some I/O test on my phone (not quadrant) and I always had a better result with loop than with direct format.

ownhere post is great but he's not always 100% right. For instance, the fact that ext2 partition when mounted on /data is not unmounted properly by the system in wrong. You can read my post here http://xdaforums.com/showpost.php?p=9704670&postcount=9 to see the tests I did. I do confirm that the FS is unmounted by the system on shutdown.

About ext4 4K block size, you can do the same with EXT2 (my actual bloc size is 4096).

But I'm reading with great interest the thread about alternatives FS for NAND data storage. The erase block size is different for different devices and manufacturers. But at least if you align blocks to a multiple of 128kb, you will get a noticeable improvement in I/O than no alignment at all.
[Desire Android Development] melethron:
y i have nilfs2-tools. I only meant that -t auto doesn't work. -t nilfs2 works fine.

I also got scared by the warning and googled the warning: http://www.spinics.net/lists/linux-nilfs/msg00595.html

Quoting a dev:

Reading this im willing to use it.
[Desire Android Development] melethron:
ownhere got this (wrong) information from me don't blame him... :o :o :o

You proofed me wrong on this and i admit that i was only guessing ....

After reading cyanogens post i thought that he tried to mount on /data before or tried to create an init.d/rc0.d to make a clean unmount. I didn't dare to think that any good dev would just say "... give a **** about a proper umount and use a journling fs". I mean wtf ....
[Desire Android Development] sibere:
Do you mean that you/he/they stated without trying ? ;)

I never take a guess for granted :)
[Desire Android Development] woti23:
i have a sandisc 32 GB class 2

what if seen there are only two 32 GB micro sdhc available for now (at least in middle europe)

sandisc class 2

kingston class 4

kingston is double the price of sandisc (about 150 : 80 euro)
[Desire Android Development] Terepin:
Are you sure?
[Desire Android Development] woti23:
your correct, must be new, wasn't available in germany 14 days ago, now is
[Desire Android Development] LoQo_:
http://datacomp.sk/a-data-sdhc-class-10-turbo-series-32gb_d67884.html LOL! in spain, a-data class6 16gb 60 € :eek:
[Desire Android Development] woti23:
link goes to a 32 GB class 10 non micro sdhc, well you could half cut it in two micro sdhc to fit ;-)
[Desire Android Development] LoQo_:
jajajajaja xD
[Desire Android Development] Terepin:
I bought this for my Desire: http://datacomp.sk/a-data-micro-sd-class-6-16gb-citacka_d36289.html

16GB Class 6 card + USB Flash Key reader. I took last piece they had. Cool, huh?
[Desire Android Development] melethron:
There are some dd benches in this that measure write and read per socond. But this is not only about pure read/write speed but also sqlite access.

Woti and me both have a smoother expirience when /data/data is on btrfs partition. But we both have only a class 2 sd. Would be nice if someone compares ext4 and btrfs on a class 10 card. Could be the other way around then..... but im not sure ....

Class 2.... thought so .... we need some class 10 tests ...
[Desire Android Development] HTF:
I would do some test on my class 6 card but sadly I use an AOSP rom so there is no kernel for me to test. I will try and compile my own kernel with it in but it will take me ages to get the source etc.
[Desire Android Development] woti23:
http://github.com/richardtrip/cm-kernel

this is from kali- kernel for froyo aosp + cm mod rom

should get you the source with git
[Desire Android Development] HTF:
I am already getting the source for the vork kernel, problem is my connection is super slow and it is taking ages :(
[Desire Android Development] melethron:
Updated OP with the remark that the kernels are for "sense".

And also update the script to create the needed folder (didn't add this in the first place because i included it in my zip with wich one i flash the my script) :

if [ ! -d /system/data_internal ];
then
busybox mount -o remount,rw /system;
busybox sleep 1;
busybox mkdir /system/data_internal;
busybox chown 1000:1000 /system/data_internal;
busybox chmod 771 /system/data_internal;
busybox mount -o remount,ro /system;
fi;

BTW: Im happy if anyone has improvement/bug fix suggestions for the script to make.
[Desire Android Development] K900:
Building VorkKernel for my MIUI...
[Desire Android Development] HTF:
Also building the vorkkernel myself XD
[Desire Android Development] K900:
Guess you'll be faster, my pc is very slow at times :)
[Desire Android Development] sibere:
I have tried to unmount the NAND /data at boot time in a script before mounting the ext one but the mtdblock5 remained mounted. Is the remount working for you? Can you try an busybox umount /data in your script and tell me the results?
[Desire Android Development] HTF:
Mine is taking ages too so I would not bet on that.

Edit scratch that my compile failed :( Could you please upload yours when it is done?
[Desire Android Development] melethron:
Thats strange .... i do umount /data in my script:

snipped quote:

echo " +++ Mounting devices";
busybox umount /data;
busybox sleep 1;
busybox mount -t auto -o noatime,nodiratime /dev/block/mmcblk0p2 /data;
busybox mount -t auto -o noatime,nodiratime /dev/block/mtdblock5 /system/data_internal;

If i do it while system is up and running i get

# umount /data
failed.
# busybox umount /data
umount: can't umount /data: Device or resource busy

but logcat on boot is "clean".

I//system/xbin/busybox( 65): +++ Changing scheduler:
I//system/xbin/busybox( 65): [noop] cfq bfq
I//system/xbin/busybox( 65): +++ Mounting devices
I//system/xbin/busybox( 65): +++ Moving app from internal to SD-Card
[Desire Android Development] sibere:
well yes it's strange. Obviously I was asking at boot time not when running ;)

My rom is HTC sense 2.2 (2.09.405.8) and I really don't understant why a /system/xbin/busybox umount /data isn't working and remains mounted. Could it be because I don't allow the 1 second pause you've added ?
[Desire Android Development] melethron:
Question at everyone that is better in scripting than me. I want to merge my ext4 and btrfs script so that both are compatible (for my ROM):

If i do:

busybox mount -o noatime,nodiratime,[COLOR="Red"]ssd_spread [/COLOR]/dev/block/mmcblk0p2 /data;

....will the mount fail if done with ext partition because of the option ssd_spread? Or will it simply mount without that option?

I also browsed "/proc" and "/sys" to see if i can "cat" the partition type somewhere to make a "if-then-else" mount part depending on the type... but i didn't find anything ....

Other option would be:

mount (auto)

mounts | grep btrfs

umount

then mount depending on type ... but i think mount -> umount -> mount is "bad" coding. Since mount recognizes the partition there must be some way to get the type of it .....:confused::confused:
[Desire Android Development] woti23:
maybe

-s Tolerate sloppy mount options rather than failing. This will ignore mount options not supported by a filesystem type. Not all filesystems support this option.

or --fake
[Desire Android Development] woti23:
or mount readonly first without options, check which filesystem is mounted, and make readwritable with options after
[Desire Android Development] balthor80:
Ok !

I have ReflexTSenseHD + data2btrfs + couttstech kernel

All seems ok

What should I try ? ;)

edit :

Quadrant freeze in "Database writes" in I/O..
[Desire Android Development] AdamG:
I dont suppose anyone has tried formatting /data directly and modifying init to support mounting as a different file type other than yaffs?
[Desire Android Development] woti23:
maybe you could download tap benchmark from here

http://xdaforums.com/showthread.php?t=850454

its "details on performance on /data"

and we could compare with someone with a other data2sd/ext/whatever
[Desire Android Development] mosolicious:
Could you please post the vork kernel when you've built it? I'd like to test it for MIUI and Oxygen
[Desire Android Development] snq-:
i really hope nobody has
[Desire Android Development] woti23:
i have 2050 at first run, rcdhmix, no overclocking, uv, cfq

it takes a while for it to go over database writes
[Desire Android Development] balthor80:
(I have a class 6 sdcard)

First :

Overall Performance: 2531(139)

Database Performance: 307(12)

FileRead/Write Performance: 1391(58)

StressTest Performance: 833(126)

DB writes: 10(2) Writes/s

DB reads: 205(2) Reads/s

DB deletes: 244(18) Deletes/s

File writes 10k: 2079(321) kB/s

File writes 1M: 5226(899) kB/s

File reads 10k: 4471(172) kB/s

File reads 1M: 9496(103) kB/s

File write Overhead: 2.895(0.812) ms

File read overhead: 1.183(0.087) ms

StressWrite 1M: 3933(378) kB/s

StressRead 1M: 5681(102) kB/s

StressWrite 10k: 852(160) kB/s

StressRead 10k: 1518(439) kB/s

Second :

Overall Performance: 2509(111)

Database Performance: 342(12)

FileRead/Write Performance: 1374(29)

StressTest Performance: 793(106)

DB writes: 8(2) Writes/s

DB reads: 234(18) Reads/s

DB deletes: 271(3) Deletes/s

File writes 10k: 2117(241) kB/s

File writes 1M: 5180(102) kB/s

File reads 10k: 4507(80) kB/s

File reads 1M: 9060(264) kB/s

File write Overhead: 2.793(0.538) ms

File read overhead: 1.115(0.051) ms

StressWrite 1M: 4305(640) kB/s

StressRead 1M: 5975(156) kB/s

StressWrite 10k: 930(174) kB/s

StressRead 10k: 1213(317) kB/s

Thrid :

Overall Performance: 2395(105)

Database Performance: 362(3)

FileRead/Write Performance: 1265(22)

StressTest Performance: 768(103)

DB writes: 8(4) Writes/s

DB reads: 265(2) Reads/s

DB deletes: 270(2) Deletes/s

File writes 10k: 1875(70) kB/s

File writes 1M: 4486(346) kB/s

File reads 10k: 4382(84) kB/s

File reads 1M: 8308(197) kB/s

File write Overhead: 3.103(0.263) ms

File read overhead: 1.078(0.052) ms

StressWrite 1M: 4167(575) kB/s

StressRead 1M: 6189(219) kB/s

StressWrite 10k: 861(182) kB/s

StressRead 10k: 1176(300) kB/s

I have seen some galaxy S with 4000+... :(

And quadrant :

no oc : 1927 - 2285 - 2355 - 2255

oc @1190 : 2600 - 3058 - 2535 - 2772
[Desire Android Development] HTF:
Details of Performance on /data

Iterations: 6 - Single Test Duration: 1000

Timestamp: 20101216T191909

Overall Performance: 2528(106)

Database Performance: 320(9)

FileRead/Write Performance: 1201(20)

StressTest Performance: 1007(104)

DB writes: 40(1) Writes/s

DB reads: 207(10) Reads/s

DB deletes: 233(10) Deletes/s

File writes 10k: 892(8) kB/s

File writes 1M: 2310(65) kB/s

File reads 10k: 4829(191) kB/s

File reads 1M: 10282(161) kB/s

File write Overhead: 6.879(0.159) ms

File read overhead: 1.098(0.083) ms

StressWrite 1M: 1007(23) kB/s

StressRead 1M: 9892(97) kB/s

StressWrite 10k: 530(52) kB/s

StressRead 10k: 2409(405) kB/s

This is what I got using oxygen rom using just standard apps2sd+ and no data2sd/ext etc. Also using a class 6 card.

Comparing this with the one above, I would say that this will gain increased write speeds across the board with database performance being very similar to stock.

I will say though that the rom I am using is not sense based so there are less background services which effect this benchmark a lot. This leaves me feeling hopeful, now if only I could get a working AOSP kernel so I can do a proper comparison test.
[Desire Android Development] HTF:
Nice benchmarks, they would suggest that the new data2ext does improve speed etc however I am slightly concerned about the database benchmarks. The database write speed is a lot lower than I was getting even though the database read speed was higher.

A 25% database write speed cut may explain some of the lags people are getting.
[Desire Android Development] sibere:
Well at boot unmount command gives me also the "umount: can't umount /data: Device or resource busy" message :( seems like an app or OS component is having access to the /data NAND partition before the SD mount. That doesn't sound good to me.
[Desire Android Development] bernabap:
I saw that too. But how many times did you run the bench? I am not having any lags at all. My average of DB write speed is 26 (27+22+29+28+22+26+28+29+25+20+28+29 / 12 = 26.083333). Please try run at least 10 times and see your average.
[Desire Android Development] mattiadj:
Can anyone tells how implement this on rcmixhd?are present lags or all is very very smooth?

sent from the future
[Desire Android Development] woti23:
class 2 micro sdhc

a2sd (ext4), /data/data btrfs, dalivik nand, cfq scheduler, no overclocking, undervolt

Overall Performance: 2373(208)

Database Performance 240(18)

FileRead/Write Performance: 1174(47)

StressTest Performance: 959(202)

DB writes: 6(1) Writes/s

DB reads: 160(22) Reads/s

DB deletes: 193(18) Deletes/s

File writes 10k: 2432(116) kB/s

File writes 1M: 2953(492) kB/s

File reads 10k: 3224(261) kB/s

File reads 1M: 9216(578) kB/s

File write overhead: 0.726(0.597) ms

File read overhead: 2.016(0.26) ms

Database overall: 240(18)

File overall: 1174(47)

StressWrite 1M: 4245(1853) kB/s

StressRead 1M: 6731(340) kB/s

StressWrite 10k: 850(185) kB/s

StressRead 10k: 1888(594) kB/s

Stress overall: 959(202)
[Desire Android Development] HTF:
My average DB write speed is 43 writes per second: (40+44+39+46+46+43+46+46+43+45+46+41 / 12 = 43.75)
[Desire Android Development] bernabap:
Thanks HTF. Actually there is a considerable difference. But as I said I have no lags at all. I will try now with my uSD class 10 and I will align the partitions as I didnt on my other uSD. Lets see...
[Desire Android Development] melethron:
@All: It's nice that you do all this benches, but im rather interested if you have a smother expirience (considering lags ...). If the phone has quadrant 9000 score but feals like underclocked to 245 it sucks.

Thanks.

Fake doesn't show up in mounts then so that's no help.

And -s doesn't help either. It's not accepted by busybox mount.

It's this for now :

busybox mount -o noatime,nodiratime /dev/block/mmcblk0p2 /data;
mounttest=`busybox mount | grep btrfs`;
if [ -n "$mounttest" ];
then
echo "btrfs Partition found - remounting with optimizations";
busybox mount -t btrfs -o remount noatime,nodiratime,ssd_spread /dev/block/mmcblk0p2 /data;
else
echo "EXT partition found";
fi;

Im not happy with this.

EDIT: Oh ..... did you mean fake mount with ssd and then real mount with ssd if there is no error and mount without if there is one? That would be an option if fake mount gives an error message on fake mount.

Why not?! Remove the ****ty mtdblock03 device. Create a UBI device and format with UBIFS. xD

...well i myself won't mess with mtd yet. at least not before UBIFS is implemented in a recovery kernel. Having no backup of btrfs partition is enough for me.

(quote snipped)

Galaxy S has like 9mb/s on the internal SD (a dev made a quick dd while i was in irc).
[Desire Android Development] bernabap:
For me it seems really smoth with 40data2ext4. My goal is have a FS that can support a lot of applications without lag on startup or loading icons, low battery consuption and no data corruption. Until now I think I got my goal. My quadrant score average is ~1500 with I/O of ~650 but this does not mean nothing for me! I dont have lags on thing like "super user permissions", "click buttons", playing games or scrolling screen.
[Desire Android Development] HTF:
How is it when installing apps?
[Desire Android Development] bernabap:
Just update GMaps and others apps and I did not realize any difference!
[Desire Android Development] HTF:
That's great as that was the only issue I had with the old script.
[Desire Android Development] woti23:
i moved to compare

/data/data/com.Peaceman.Benchmark

on the fly to nand and made a symlink to the directory from /data/data/ to the directory in nand

so my database writes got from 6 to 52/s

and overal to 2487 from 2373

but the filewrite overhead is now (with in nand not btrfs) 6.66 ms (from 0.72ms), could that be a reason for lags?
[Desire Android Development] HTF:
The lower the overhead the better so from what you said nand has a larger overhead. I wonder why the database performance is degraded so much for writes, that imo must be the reason for any lags it is the only time nand does better.

I wonder what version of sqlite android uses because I remember reading something about efficiency improvements in newer versions. I suppose the issue is as usual cross compilation and android may use a modified version of sqlite.
[Desire Android Development] mattiadj:
Which Rom?which SD?how many apps installed?

sent from the future
[Desire Android Development] bernabap:
As I said on my post a page ago:

- Oxygen 1.0.3

- 40data2ext4 script

- uSD class 6 "Turbo" ( I dont know what that Turbo mean), 1 fat32 and 1 ext4

- 211 apps.
[Desire Android Development] woti23:
yes just compared with unjournaled ext2

46 database writes / s

i think there is also some strange thing goin on with this test/benchmark:

shouldn't i get close to SGS (3600) or even more, when i have the test in RAM,

in RAM i get only about 2900
[Desire Android Development] mattiadj:
Woow!!!211apps are terrific!

Can you explain me how set it up on sense hd based Rom?

sent from the future
[Desire Android Development] HTF:
Interesting but way above my head to figure out the cause for the huge database write speed loss.

Just an fyi to my previous post, android uses sqlite 3.6.22, the current sqlite version is 3.7.4 so android is some way behind. I expect there have been performance improvements since 3.6.22 the problem is how to install a newer version and will it break the android sqlite lib? I personally think it might break things otherwise google would have included it in gingerbread I am sure.
[Desire Android Development] woti23:
i put it in RAM - just to compare, and get only 2900 overall

but 253writes/sec - strange
[Desire Android Development] bernabap:
I didnt tested on any HD sense based ROM but I think should work the same way. All I did was: 1- flashed Oxygen ROM, 2- before reboot (from recovery): 2a- replace my busybox with the new one, 2b- remove A2SD script and replace with 40data2ext4 script, 2c- set permissions, 2d- create the folder /system/data_internal. That`s it! 3- reboot.
[Desire Android Development] mattiadj:
I'm a newbie....can you explain and write what commad have you done?

It's required that Rom is only on nand?

sent from the future
[Desire Android Development] anko184:
In a working data2ext rom if i copy all files from ext partition and then make again the partition formated at btrfs,

add the script with the right permissions and copy back the files to the new btrfs partition will it work?
[Desire Android Development] bernabap:
Most of this was quoted from where everything started: http://xdaforums.com/showthread.php?t=852219

What you need is:

* Rom with init.d script support

* Latest busybox build (http://www.busybox.net/downloads/binaries/)

1.17.2> busybox-arm6l

* Boot your phone in recovery

* Flash your ROM (does not reboot!!!)

* Plug on computer

Connect to your phone with "adb shell"

You should get a ~# prompt now.

#mount /system

#exit

Replace you old busybox exec with the new one. It's located in /system/xbin/

$adb push c:\busybox-armv6l /system/xbin/busybox

Add the script to init.d folder:

$adb push c:\40data2ext4.txt /system/etc/init.d/40data2ext4

back to the shell:

$adb shell

#chmod 755 /system/xbin/busybox

#chgrp shell /system/xbin/busybox

#chown root /system/xbin/busybox

#chmod 755 /system/etc/init.d/40data2ext4

#chgrp shell /system/etc/init.d/40data2ext4

#chown root /system/etc/init.d/40data2ext4

#mkdir /system/data_internal

#exit

$adb reboot

Test your self and see if works! Make BACKUPS of EVERYTHING FIRST!!!

Melethron chaged his script today and you dont need to create the data_internal folder any more, the script will create for you!
[Desire Android Development] melethron:
The overhead would fit to my assumption that the reason the lags are better with btrfs is the filesystem. So its not the ACTUAL write speed to /data/data that matters but it's the difference between ext4 journal and btrfs copy on write.

mattiadj. I said in the thread, i said it in ownheres thread and i said it per PM. Just WAIT a bit for ROMs that have everything ready. Explaining the installation for every user just takes time.

I just fixed the last bug of my ROM and then i upload it...... All you need now is either a second ext4 or btrfs partition. And if you don't have linux you can't make btrfs partition now. So just make an ext partition as for app2sd and wait till i uploaded my ROM.
[Desire Android Development] melethron:
this could mess up the permission of the files on the external but im not sure....
[Desire Android Development] anko184:
If you think that's doable then i'll try it and let you know what happened!

I'll take a nandroid first of course just in case, that's why we have it after all.
[Desire Android Development] melethron:
Give it a try.....
[Desire Android Development] woti23:
this works only in an *nix like enviroment

with cp -ap /part1..../* /part2/

you must also preserve the ownerships of files and directories and keep symlinks

cp -ap ... does
[Desire Android Development] melethron:
@mattiadj

Rom takes more time ....

it somehow wont flash /data/local .... everthing else in /data is flashed but the local folder is missing ...
[Desire Android Development] HTF:
Make sure if there are no files in data/local that you put a placeholder file in there which you can delete afterwards or it will not create the folder.
[Desire Android Development] yukkio:
Hi, I've modified Aurax 8.1.3 to support btrfs with this script but 8.1.3 hasnt a kernel with btrfs support so all things in /data/ arent copied. Is this a recovery problem? Do I need a boot.img with btrfs support to copy all data when I install the rom?

BR.
[Desire Android Development] melethron:
Thanks for that.

I already tried this. It is more and issue that there ARE files in there. To be specific: That there are 2 ZIP files in there. The bootaninmation and the downanimation. Im pretty sure now that they are the reason for the issue, because /system/customize/resource is also not flashed (complete folder with the stuff in there is missing after flash) because there is the shutdown.zip.

I tried to ONLY flash this folder with the zips and it is working then.

So i think it could be something with compression, because the archive doesn't get compressed if there are only zip's in. But i really would wonder about this because the other roms ARE ALSO compressed.

Edit: uncompressed doesn't work also....
[Desire Android Development] woti23:
take a look in the updater-script of the rom?

META-INF / com / google / android / updater-script

so you see what the rom install deletes or install or mounts or does symlinks

maybe it mounts another partition to data

mount("MTD", "userdata", "/data");

package_extract_dir("data", "/data");
[Desire Android Development] melethron:
Bit off topic now, but if someone wonders why i didnt make a ROM yet:

I still didn't figure out why this file wont get written on the mtd but while trying to flash over and over again i managed to get an USB BRICK.....

Wouldnt be a problem but:

-i had no working rom at this time on the phone bcs i only flashed /data

-i couldn't flash leedroid from sd because i had no kernel to flash afterwards and btrfs was not recognized ....

-and i couldnt get access the sd because i had no card reader ....

-and couldn't get a card reader bcs i had other stuff to do and apart from that i have no driver license (atm) bcs they made a nice photo of me earlier this year......XD

Got someone that drove me and bought a card reader and now the brick is fixed .....

... but i still don't have solved the mtd problem.....
[Desire Android Development] yukkio:
Incredible... http://twitpic.com/3gqfnp