yashsingh1/xdaforums huggingface dataset:
[Desire Android Development] melethron:
I check #alphrev to get some help on my flash issue:

I did:

fastboot oem partition_test userdata

to check for bad blocks:

INFOTime cost = 3748405 ms
INFOPASS rate = 1181 / 1181 = 100 %
INFOFAIL rate = 0 / 1181 = 0 %

None bad block and my /system is also clean.

But i found my issue:

/dev/block/mtdblock5 151168 151128 40 100% /data

LOL ....

But at least i have some news about UBIFS or messing with NAND in generall:

So if anyone wants to give it a try do what you like with nand as long as you are S-OFF.

So actually flashing a radio is more risky than formating nand to another fs....

But don't forget that you need recovery support of the filesystem you use.

But who knows. Maybe we some soon:
[Desire Android Development] rafi300:
could some one advise on the fact that this script can be replaced in the init.d folder of ReflextSenseHD v1.4.1 rom with fat32 partition and a 2gb btrfs partition?
[Desire Android Development] rafi300:
I'm doing something wrong that this script is not running correctly in reflextsenseHD v1.4.1. Could someone point out what im doing wrong

Here are my steps:

First parition my sdcard with gparted.

1. Create Primary Partition of 12.9gb Fat32

2. Create 2.00gb btrfs partition.

Do complete wipe.

Flash ReflextHDv1.4.1 A2SD+ version

Flash btrfs kernel of coutts.

Plug phone into usb.

update the busybox with command.

"adb shell"

You should get a ~# prompt now.

#mount /system

#exit

Replaced old busybox exec with the new one.

$adb push c:\busybox-armv6l /system/xbin/busybox

Removed 03stuff2sd

$adb rm /system/etc/init.d/03stuff2sd

Added the script to init.d folder:

$adb push c:\40data2ext4 /system/etc/init.d/40data2ext4

back to the shell:

$adb shell

#chmod 755 /system/xbin/busybox

#chmod 755 /system/etc/init.d/40data2ext4

#reboot

and wait....

Than after 10min i get only lockscreen and no rosie?

Please help.
[Desire Android Development] bernabap:
I dont get it, why this 40data2ext4 script is in this thread?
[Desire Android Development] bernabap:
Melethron that 40data2ext4 script on this OP is right? Should not be 40data2btrfs?
[Desire Android Development] melethron:
Hmm, from the first look at the update-script and the init.d scripts i think it should work. Symlinks should be alright from what i see.

Make sure your sure your 2nd partition is mounted with "df" in "adb shell" and check the logcat (adb logcat) of the first boot. If you are unsure just post the results (for logcat till the the btrfs finish message)

OR if you are not that focused on Neophytes HD Rom you wait a bit. I'm just doin the last test with my HD rom (checking both btrfs and ext4). I used Pays HD Rom for that so if you are fine with that one too just wait some min. I'll upload it in the next couple of minutes.
[Desire Android Development] melethron:
Oops ... sry ....

Well i updated both the same time.... only difference between them is this line

busybox mount -t [COLOR="Red"]btrfs[/COLOR] -o noatime,nodiratime,[COLOR="Red"]ssd_spread[/COLOR] /dev/block/mmcblk0p2 /data;

Well i made 1 script for both for my ROM. But there are ROM specific stuff in. When im finished i add one script for both which detects the partition type and then mounts with specific options....

sry again ....

EDIT: I really wondered why that happened bcs im sure i went to btrfs dir... this wont happen again .....
[Desire Android Development] melethron:
Btrfs ready HD rom can be found here.
[Desire Android Development] melethron:
I tested the I/O speed with mount options "compress". While it increases speed on large files it slows things down on small files. Especially on db writes which leads to lags. So spare yourself the time if you wanted to try that.

Also i compared bfq / cfq / noop. Bench its almost the same. But noop feels more smooth. Didn't try "deadline" (not in coutts kernel) but i think it can't be improve from noop.

About noop note this:
[Desire Android Development] Fightspit:
If I understand, the Noop scheduler can improve the smoothness despite of the time needed on I/O request which is true on HDD because it takes a few time to change from a sector to another one. But it seems it is totally irrelevant on a SSD/Sd-card because the access time is really smaller than the HDD's. So the SSD/Sd-card can handle the noop scheduler without being affected by the time needed on I/O request.

EDIT:

I found an interesting paper (which confirms your statement) about the current schedulers (noop, anticipatory, deadline & cfq) and the new ones on SSD.

Look at the page 3 (figures 1 & 2) and 8-10 for the benchmarks

http://dropzone.tamu.edu/techpubs/2009/TAMU-ECE-2009-02.pdf
[Desire Android Development] HTF:
Here is an interesting blog post from one of the android devs, in it there is an explanation of why YAFFS on the nand is a big area of speed loss. From what he is saying we could get a decent improvement in "stuttering" and "lag" by switching it to either EXT4 like the Nexus S or some other nand filesystem (I would personally go for EXT4 due to its high availability in kernels).

Obviously this does mean data2ext can help as well as it means any writes that would normally go to data on the nand go to the sd card on ext.
[Desire Android Development] melethron:
Yep. On a hdd it sucks:

As you can see a "stress test" increses the time needed by almost factor 20. But on the sd-card there is no difference at all. Also it makes more use of the parallel channels on a ssd. This is like a tiny raid-0 device between some nand banks.

The only reason for scheduling in the first was to reduce the time that a the mechanical parts need to move the head. Thats also the reason you do a defrag on a HDD so that files are at the same place. Defragmentation doesnt matter at all on a ssd.

Hmm i didn't mess with internal memory yet. I first do all the bench marks of sd card and then i make some test by formating the cache partition of the internal memory. As IEF from alpha rev said there is no risk on this if you have fastboot. To revert back to normal you can just do "fastboot erase <partition>" and the partition is clean again.

Also note that internal memory works a bit different than an microSD. Internal is mtd RAW nand chips and sd-card is nand with a controller that emulates an LBA. The difference now between our desire and Nexus S is that Nexus is actually the same as the samsung galaxy S. And this has an internal sd-card. Apart from that an sd-card can also be very different (controller, FTL etc). So there might be differences. But apart from that i agree. Of the "common" filesystems EXT4 is the best choice.
[Desire Android Development] sibere:
Do they say options used to format/mount ext4 on the nexus S?
[Desire Android Development] melethron:
I've read the article and it isnt about ext. Must be the wrong link (this article is footnote 60 in "android" wikipedia article and "ext4" is in footnote 61 ^^).

But it still about lags:

They made new api for programmers so that they can use other stuff instead of commit() etc.

But actually this article describes what i already said. Lags are caused by writes to the sd (especially db writes wich has the lowest performance).

The proper article about ext4 is this.

http://thunk.org/tytso/blog/2010/12/12/android-will-be-using-ext4-starting-with-gingerbread/

And they refer there that the Nexus S will use it. Well and the nexus s is actually a samsung galaxy s with an internal memory card. Well and for samsung galaxy s there is the voodoo lagfix which uses ext4 on the internal memory card. Well and my ext4 script is based on this lagfix. So ..... also nothing new on this .....

Apart from that i tried everthing with mount options on ext4. And i doubt that google will use ext4 unjournaled. The only difference between nexus S and my desire will be that i have a class 2 sd and the one in the nexus should be faster. SGS uses a class 6 card.

The main part they did to reduce the lags was avoiding writes to the memory (as explained in detail in the first article). Lags on HD Roms are on most part "bad coding" of htc and also some "bad coding" of google. Google tried to make things better in gingerbread. We'll see if they done it well when the first gingerbread are considered final.
[Desire Android Development] Benee:
Updated my Kernel. Now with btrfs support. If anyone wants data2btrfs for Aosp or CM roms ;). Please report if it is working
[Desire Android Development] melethron:
Time to make a new kernel with ReiserFS :D

I finally benched everything and i can confirm the results for an sd-card are the same as the results for better ssd as phoronix benched. Other than i expected the controller of a cheap microSD is rather good and better than a 3-4 year old ssd. This makes btrfs with ssd optimizition a rather bad solution (btrfs has rather good mb/s but the sqlite performance of btrfs sucks :( ). Ill update the main thread till the end of the week with "in-depth" analysis of the "data2sd lag" issue and a sqlite benchmark of every filesystem. Im already done with all the benching but ill test ReiserFS more in real-life....
[Desire Android Development] Benee:
Ok tomorrow with ReiserFS :D
[Desire Android Development] sibere:
Wow that's what we call a very good reactivity! :D
[Desire Android Development] navalynt:
This is interesting ... I would love to try and script this. Can anyone post the arm compiled binaries here? Otherwise I'll have to try and cross-compile myself when I get home later (but why do it if someone else already has).
[Desire Android Development] NeoPhyTe.x360:
http://www.phoronix.com/scan.php?page=article&item=ext4_btrfs_nilfs2&num=1

I see nilfs2 is not the best in some test ....

So I think, because I have seen people saying this, better is ReiserFS
[Desire Android Development] NeoPhyTe.x360:
Have built nilfs2 toolchain for arm, but I don't know if I have well done

here is it
[Desire Android Development] melethron:
Quoting the test:

"The disk drive we used for carrying out the file-system benchmarks was a Seagate 250GB SATA 2.0 ST3250310AS (7200RPM; 8MB cache). Originally, we were going to also throw a solid state drive into the mix, but with these tests taking longer than anticipated on the newer file-systems, in this article are just the HDD results."

Quoting myself from the OP

:

"Your Sd-card isn't a HDD (as you may have noticed ^^)"

Linking a test done with an SSD from OP:

http://www.usenix.org/event/lsf08/tech/shin_SSD.pdf

Additional link for the most important physical difference between a SDD and a HDD:

http://en.wikipedia.org/wiki/Flash_memory#Block_erasure

Get my point? ;-)

Thanks for this.
[Desire Android Development] NeoPhyTe.x360:
Maybe my compiled tools don't work:

12-11 02:07:46.275: INFO//system/xbin/busybox(64): mkfs.nilfs2: applet not found
[Desire Android Development] th3:
Is this all based on some theory? :P

NILFS2 had so many performance/write related problems for long. It was killing P/E cycles (Program/Erase). I'd be very careful before eulogising it as a perfect solution.

http://www.spinics.net/lists/linux-nilfs/msg00560.html

http://www.spinics.net/lists/linux-nilfs/msg00042.html

http://www.spinics.net/lists/linux-nilfs/msg00039.html

The general structure of this information is correct:

http://lwn.net/Articles/353411/

NAND flash is dependent on controller algorithms, wear levelling and garbage collection far more than filesystem. Even the wordline caches they currently have started implementing have a much greater effect to improve performance, write amplification and P/E cycles.

------------------------------

- Sent via HTC Desire -
[Desire Android Development] ownhere:
This is why in my ROM, specially all the disk IO scheduling policy is set to noop.
[Desire Android Development] woti23:
thanx for the links

after using ext2/3/4, reiserfs on linux for a long time and phone (+btrfs,-reiserfs) now on phone for 1 day, my (private) asume from a practical (pragmatical) point of view:

ext2: i have ditched on servers years ago, won't use it on phone either, as of missing jorunal. id rather buy a new sdcard than the hassle of trying to repair a corrupted ext2

ext3: don't see an advantage over the robust, now in many distros default ext4

reiserfs: ditched on servers years ago as ext3 became useful.

dont see an advantage over ext3/4, mostly i had no luck with repairing a corrupted reiserfs.

ext4: no issues i can see for now, neither on intel 25 sdd on netbook nor on phone

btrfs: seems robust as ext4 after one day, but with the disadvantage that i don't like to have to compile a kernel module when i'm on the road and the partition needs to be manipulated or data must be read outside the phone. in case of ext4: it can be at least read/write on mac os x with fuse as ext2 mounted, or with virtualbox also in etc in case i need some data to survive.

nilfs2 on ubuntu: well, we had a bad start, made my ubuntu kernel crash twice some time after mounted.

another minus: could not find useful tools as binaries. not even for ubuntu.

advantage: its incredible fast in postmark benchmark, almost on par with the intel ssd (with ext4). about 50x write and 60x read speed against ext4, when writing, reading, deleting 50000 random files between 9k and 103k

but this is bench, don't even know if the data is already writen to disk when bench run is over

only nilfs2: postmark benchmark on usb micro sdhc only here filled the rest of the regular space on harddisk temporaly with 23GB, til it was full. as fast as this went i don think it wrote anything to the ssd, but it was no space left. wtf.

no issues with that on the other filesystem tests.
[Desire Android Development] melethron:
What where your config settings for the "garbage collector" for those tests?

also why don't you test it with "real" data and check if it is there?
[Desire Android Development] NeoPhyTe.x360:
And xfs? It give good benchs in some situations

That we need is that format that have minimal posibilities to corrupt, and in the same time fast.

Another things is an "autobackup" each 10-15 hours of the mount (save in ext), if at boot file is corrupted and can't boot, automaticly restore it (with little partitions), and, put the backup in a place where nandroid save it, so, if you run rom, and there is an existent backup, although don't have the mounted inside, it restore your last backup. It's only a idea but if we do it, we don't need another file system without corruption, only the fastest file system.

Finally, I think data2ext doesn't work well? when you install xxx apps, it don't let it install more although you have free space, so i deduce that if "internal memory" is 0, it doesn't let you to instal more apps?
[Desire Android Development] ownhere:
totally agree with you!

nilfs2 eat disk space, and i don't how to fix it.
[Desire Android Development] woti23:
same as yours

protection_period 3600

min_clean_segments 10%

max_clean_segments 20%

the scurile thing is:

i benchmark a micro sd with nilfs2 - it writes on the micro sd, i can see that while the bench is running, and removes it when bench is over. its a 2GB microsd, no problem here

but it fills up my internal sdd in the laptop with 23GB space occupied so sdd seems full, even if i dont no if its really full or just shows full
[Desire Android Development] woti23:
had to make modules_install for xfs first ;-)

the fastest surely is nilfs2 it seems, but for now its "uncanny" to me (hope this is the right word, as english is not my first language ;-)

xfs - no luck at all to benchmark, its running endless and i must kill the process

will try later all variants with 1024byte block size and only 5000 files instead of 50000 as its endless to test

maybe increase max files in directory possible?

and for tested only read/write speed as reported by postmark

ext4, btrfs on microsd all about 40kB/s read, 1,2Mb/s write

nilfs2 on microsd and ext4 on intel ssd (internal laptop disk values to compare) about 50 write (read 60) times faster as above

(240kB/s read, 58MB/s write)

for all test i had (notice i tried a little with changing block to higher, but not for all, above all compared with 512bytes):

Current configuration is:
The base number of files is 50000
Transactions: 500
Files range between 8.85 kilobytes and 101.34 kilobytes in size
Working directory:
/mnt (weight=1)
Block sizes are: read=512 bytes, write=512 bytes
Biases are: read/append=5, create/delete=5
Using Unix buffered file I/O
Random number generator seed is 42
[Desire Android Development] terryhau:
This sounds awesome.

I used data2sd for a while, but it was way too laggy.

Hope this works out.
[Desire Android Development] melethron:
Thanks for the links. I'll take a look at this.

I ABOSOLUTLY agree with you that it is much more controller dependend than filesystem. But we can't change the controler (except buying another sd) but we still can try to optimze the fs.

That is really nice. I've reading in this paper (page 4) that the ssd controller uses 4 channels. This makes BFS/CFQ completly senseless for the sd/internal.

I just checked this on baadnews 0.4b

cat /sys/block/mmcblk0/queue/scheduler
noop anticipatory deadline [cfq]

Quick test with "time cp -a" some mp3s. (files were present and overwriten before the first run). Results seem to be much more steady.

bash-3.2# echo noop > /sys/block/mmcblk0/queue/scheduler
bash-3.2# cat /sys/block/mmcblk0/queue/scheduler
[noop] anticipatory deadline cfq
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp

real 0m46.094s
user 0m0.050s
sys 0m4.950s
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp

real 0m50.477s
user 0m0.070s
sys 0m4.470s
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp
^[[A^[[B
real 0m48.847s
user 0m0.100s
sys 0m4.680s
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp

real 0m50.791s
user 0m0.070s
sys 0m4.060s
bash-3.2# echo cfq > /sys/block/mmcblk0/queue/scheduler
bash-3.2# cat /sys/block/mmcblk0/queue/scheduler
noop anticipatory deadline [cfq]
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp

real 0m42.937s
user 0m0.050s
sys 0m4.240s
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp

real 0m53.734s
user 0m0.060s
sys 0m4.280s
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp

real 0m54.000s
user 0m0.120s
sys 0m6.000s
< Floyd/The\ Dark\ Side\ of\ the\ Moon/* /sdcard/tmp

real 0m59.362s
user 0m0.010s
sys 0m3.760s
bash-3.2#

You really make many thoughts about optimizations. Thank you.
[Desire Android Development] sibere:
+1

I just changed my IO sched from cfq to noop on my phone running data2sd.

I'll see if I feel any difference :)

thks

Edit: huuuge lag when installing an app, I revert back to cfq.
[Desire Android Development] Benee:
I have read something about it. This is a benchmark thing. In benchmark better but in real use the more complex io scheds are better. Tested many io scheds with different slab allocators/rcu and cpu scheds to get the best performance for vorkKernel.
[Desire Android Development] NeoPhyTe.x360:
Have tested bfq and others, and I have got better results with cfq too
[Desire Android Development] melethron:
I never said i use these and i also don't use. I said that these are the DEFAIULT settings and explained why they WILL MESS THINGS UP for our purpose. (Check the posting you refer to again)

If you Bench with these settings the space used also comes from the large amounts of checkpoints.
[Desire Android Development] woti23:
thanks, will test with new settings
[Desire Android Development] Benee:
For me BFQ with slqb is the best combination. In normal use it is smooth and don't produce lags. Add the classic rcu and BFS and yay :D
[Desire Android Development] twicejr:
Will this end up in the roms, you think?
[Desire Android Development] coutts99:
+1 to this!
[Desire Android Development] NeoPhyTe.x360:
Now I have CFQ with slqb, and CFS with autogroups, will try BFQ
[Desire Android Development] Benee:
autogroups doesn't improve anything on android because of lack of tty (but does nothing bad so yeah just leaving the commit). Think I will stop the CFS version soon.
[Desire Android Development] NeoPhyTe.x360:
Could be, but I don't know why CFS seems more "faster""unlaggy" between menus on touch. And for me, maybe is my opinion BFS gives me a little more battery drain with the same config?
[Desire Android Development] melethron:
It won't be also not of use on normal linux if you don't compile with -j64 ^^

The dev got blamed for slow speed if the system is not under full load and he explained that he made it for special situations only (some linked to the blog of him in some thread ... i think somewhere in the coutts99 Kernel thread)
[Desire Android Development] Benee:
Never tested it on a normal PC. All my machines run a BFS zen Kernel.
[Desire Android Development] HTF:
I think you are 100% correct here in that if it works correctly on a phone and if compile for ARM it should that it should be good and that it is only due to default settings that the previous mentioned "problems" exist.
[Desire Android Development] melethron:
Did you get those to work? The files ownhere's sent me were missing mount binaries.
[Desire Android Development] NeoPhyTe.x360:
format with mkfs.nilfs2 etc etc, and then mount with mount -t nilfs2

if kernel have nilfs2 built in, it should work
[Desire Android Development] melethron:
Nice. But did the cleaner service (nilfs_cleanerd) also work? It would be useless without. Full sd then ^^

Checking the nilfs_cleanerd.sh script it states it needs libaries. And this is confusing me:

relink_command="(cd /homrelink_command="(cd /home/ubuntu/kernel/nilfs2/nilfs2-utils/sbin/cleanerd; { test -z \"\${LIBRARY_PATH+set}\" || unset LIBRARY_PATH || { LIBRARY_PATH=; export LIBRARY_PATH; }; }; { test -z \"\${COMPILER_PATH+set}\" || unset COMPILER_PATH || { COMPILER_PATH=; export COMPILER_PATH; }; }; { test -z \"\${GCC_EXEC_PREFIX+set}\" || unset GCC_EXEC_PREFIX || { GCC_EXEC_PREFIX=; export GCC_EXEC_PREFIX; }; }; { test -z \"\${LD_RUN_PATH+set}\" || unset LD_RUN_PATH || { LD_RUN_PATH=; export LD_RUN_PATH; }; }; { test -z \"\${LD_LIBRARY_PATH+set}\" || unset LD_LIBRARY_PATH || { LD_LIBRARY_PATH=; export LD_LIBRARY_PATH; }; }; PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin; export PATH; gcc -std=gnu99 -Wall -g -O2 -o \$progdir/\$file nilfs_cleanerd-cleanerd.o nilfs_cleanerd-cldconfig.o nilfs_cleanerd-vector.o ../../lib/.libs/libnilfs.so -Wl,-rpath -Wl,/home/ubuntu/kernel/nilfs2/nilfs2-utils/lib/.libs)"e/ubuntu/kernel/nilfs2/nilfs2-utils/sbin/cleanerd; { test -z \"\${LIBRARY_PATH+set}\" || unset LIBRARY_PATH || { LIBRARY_PATH=; export LIBRARY_PATH; }; }; { test -z \"\${COMPILER_PATH+set}\" || unset COMPILER_PATH || { COMPILER_PATH=; export COMPILER_PATH; }; }; { test -z \"\${GCC_EXEC_PREFIX+set}\" || unset GCC_EXEC_PREFIX || { GCC_EXEC_PREFIX=; export GCC_EXEC_PREFIX; }; }; { test -z \"\${LD_RUN_PATH+set}\" || unset LD_RUN_PATH || { LD_RUN_PATH=; export LD_RUN_PATH; }; }; { test -z \"\${LD_LIBRARY_PATH+set}\" || unset LD_LIBRARY_PATH || { LD_LIBRARY_PATH=; export LD_LIBRARY_PATH; }; }; PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin; export PATH; gcc -std=gnu99 -Wall -g -O2 -o \$progdir/\$file nilfs_cleanerd-cleanerd.o nilfs_cleanerd-cldconfig.o nilfs_cleanerd-vector.o ../../lib/.libs/libnilfs.so -Wl,-rpath -Wl,/home/ubuntu/kernel/nilfs2/nilfs2-utils/lib/.libs)"

I dont get what it does but all those "ubuntu" stuff ... and gcc ....