yashsingh1/xdaforums huggingface dataset:
[Desire Android Development] sibere:
OK

I've just turned off my phone. rebooted in recovery, mounted the 2nd FAT32 partition and run a tune2fs -l ext2 (that's the name of my ext2 file mounted via a loop device over /data)

Here is the result:

tune2fs 1.40.8 (13-Mar-2008)
Filesystem volume name: userdata
Last mounted on: /data
Filesystem UUID: 296f98d0-21e8-48b6-82ac-6c94c4edf28d
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: ext_attr dir_index filetype sparse_super
Filesystem flags: unsigned_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 60800
Block count: 243056
Reserved block count: 0
Free blocks: 168874
Free inodes: 57516
First block: 0
Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 7600
Inode blocks per group: 475
Filesystem created: Tue Dec 7 15:00:51 2010
Last mount time: Thu Dec 9 17:40:16 2010
Last write time: Thu Dec 9 21:16:43 2010
Mount count: 1
Maximum mount count: 21
Last checked: Thu Dec 9 17:40:10 2010
Check interval: 15552000 (6 months)
Next check after: Tue Jun 7 17:40:10 2011
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size:	256
Default directory hash: half_md4
Directory Hash Seed: 8ff9238f-aef7-48a7-85e1-7d2d45b9e809

You can see that it's reporting that the system is clean. If the ext2 partition wasn't cleanly unmounted by the system at shutdown, it would never report as being clean.

Obviously, if I check the state now that, after booting to android, the FS is mounted over /data, it is reported as being unclean.

It's reporting 1 as mount count because I'm forcing an fsck on boot anyway.

This doesn't mean that I will never get errors tho. I even have some errors from time to time when I reboot my linux home server :)
[Desire Android Development] sibere:
Remember that there is no writeback cache (AFAIK) with YAFFS2 :)

But I should try with sync off and see if the system is still "clean" after a shutdown. If the systems unmounts ext2 properly, then the cache should be flushed by the system before unmounting the filesystem.
[Desire Android Development] melethron:
You are right about the "clean" flag being set while unmounted. But you forgot one thing. The "clean" flag must be cleared when the partition is mounted. You should check if it as also reported while it IS mounted .... if so mount doesn't write proberly to the superblock. I would do this myself but im on ext4 and afaik it isn't cleared on mount then. For me this far it shows also "clean" when it is mounted.

My guess that mount doesn't write the superblock proberly is also supported by the fact that your and my mount count both shows 1. This could be a coincidince (chance for this is 0,147928994083 %) but could also support my assumption.

So check again while you mounted your system.

EDIT: I completly agree with you that data2ext is about space. Considering speed i also expirienced that reading larger amounts of data from data/data is faster with data2ext. But i also expirienced some slowdown sometimes on ext 4. It was better on ext2 and with that im really fine.

Remount done: Mount count still 1. I bet you will also get clean when you mount your ext.
[Desire Android Development] woti23:
i think that won't work with rc.0 as init.d script execution is initiated in

init.rc only for "before booting", its not as a "normal" linux init system

# Execute files in /etc/init.d before booting

service sysinit /system/bin/logwrapper /system/xbin/busybox run-parts /system/etc/init.d

disabled

oneshot

i wonder if a shutdown.rc could be populated, there is a shutdown.bravo.rc but its empty

for init-handling, in the kernel source its in system/core/init

but im not fit with c/c++ to look into

maybe the init binary would have to be modified by a kernel dev to enable shutdown.rc / also possible that htc has motified it to not to be used.
[Desire Android Development] melethron:
This would have been to easy .... :(

Well maybe some Kernel Dev will look into this.... coutts, sibere, anyone ... can you help :o ?

Btw: i still wonder why /system/bin/shutdown is present in ROMs if it's not used ....
[Desire Android Development] woti23:
this works for the script that should be executed part

/system/bin/logwrapper /system/xbin/busybox run-parts /system/etc/shutdown.d

runs every script which is in /system/etc/shutdown.d

if you create /system/etc/shutdown.d before

and put one or more executable script/s that do something in
[Desire Android Development] sibere:
Sorry mate, but, like I wrote in my previous post, it shows unclean when mounted.
[Desire Android Development] melethron:
ROFL! I should learn to read proberly:

Well thats actually good news thanks for helping me reading. ^^

Well so this isn't a problem. Now its time for ext4 (no loop) without journal mounted to data. About the loopdevice: you might wanna read what ownhere has to say about loop.
[Desire Android Development] coutts99:
http://android.git.kernel.org/?p=pl...2af4c72427ad069245d0aa572;hb=refs/heads/froyo

Some interesting stuff in this code here
[Desire Android Development] woti23:
i'm running this now on btrfs partition

/dev/block/mmcblk0p6** on /system/sd type btrfs (rw,relatime,ssd,noacl)

/dev/block/mmcblk0p6 5.2G 1.3G 3.9G 26% /system/sd

tmpfs 85.8M 24.0M 61.8M 28% /cache

a2sd btrfs *

d2sd btrfs *

dalvik-cache in /data mtd

cache in ram 90M

* without loop mounted img filesystem

** logical partition on extented partition
[Desire Android Development] coutts99:
Running what now?
[Desire Android Development] woti23:
conventional (not froyo app2sd) a2sd + data2sd both on btrfs without loopmounted filesystem files

/data/dalvik-cache left in nand

/cache tmpfs in RAM

btrfs instead of ext4
[Desire Android Development] coutts99:
Any speed increase?

Has btrfs got fsck yet?
[Desire Android Development] woti23:
ubuntu has, and debian, gonna look if i find debian-arm has a usable one

btrfsck

usage: btrfsck dev

Btrfs Btrfs v0.19

speed seems on par with ext4 for me, also quadrant about 2000 without overclocking not much lower. difficult to test somewhat for me.

only i noticed in quandrat running that it hangs when doing i/o database write test, but seems to have no noticable effect in result of quadrant

any idea for a speed test to compare i/o on nand vs ext4 vs btrfs?

would it be useful to use time [command] to copy a bunch of the same files from a tmpfs in RAM to the different medias?
[Desire Android Development] melethron:
Did you notice any difference if you dont use the "ssd" option on mount?

Apart from that you can also increase speed if you align your partition to the the "erase blocks" and "pages" of the nand in the sd. With a 16Mb alignement you'll hit every possible size of the "erase block" of flash memory. To do this your partition should start:

start_sector:=32768*n

with "n" being a "natural number" and (512Byte * 32768 = 16MB).

Chose partition size with

partition_size:= m * 16 mb (m also natural) and thus:

sector_end:= (partitionsize*1024*1024)/512 + start_sector

Infos from here (german) and here.

The results on Nilfs2 were this comes from were ~20% more speed.
[Desire Android Development] sibere:
Have have to say that I do agree with him for the main bit of his post. But I also have to say that I don't with others of his statements , specially that EXT2 crashes on a simple reboot on our phones (quote : "just normal reboot will be crashed it"). I just demonstrated that the unmount is done properly on shutdown thus that he's wrong. And why on earth I would like to remove my battery just to prove that my FS is robust? :D

But it's true that ext2/3/4 are not the best FS for nand storage.
[Desire Android Development] sibere:
Very interesting.

you should also have a look here http://www.patriotmemory.com/forums/showthread.php?t=3696 for data alignment .
[Desire Android Development] woti23:
i will do remove the ssd mount option , for now im reading through all the links you provided - interessting!

one question about alignment:

do i have to repartion the whole sdcard, or can i just remove my 4th partition (extended), let it start at a n*32768 sector and add 3 logical partitions (and format them 1 with btrfs and one with nilfs2, one with ext4)?

would be great to compare, as one could use the a2sd script and only altering the partition numbers for all three partitions to test
[Desire Android Development] woti23:
i now tested again btrfs, have no better choice than quadrant (a few runs)

with a new kernel, undervolted, no oc, no gpu+

[lowest 2050] average 2210 to 2380 [highest 2499]

kernel version: 2.6.32.27-sub501 - SD Sandisc 32GB class2 - option ssd on -

no alignment

--------------------

scaling frequency min/max: 245000 / 998400
kernel reported frequency min/max: 128000 / 1113600
current freq: 998400
current gov: ondemand
current read_ahead_kb SD: 64
current read_ahead_kb NAND: 4
current scheduler SD: noop anticipatory deadline cfq [bfq]
current scheduler NAND: noop anticipatory deadline cfq [bfq]
wifi power: Bit Rate=54 Mb/s Tx-Power:10 dBm
lcd-backlight brightness: 20

A2SD (/data/app on SD btrfs):used 172.2M	/system/sd/app (5.6GB total)
D2SD (/data/data SD btrfs):used 94.5M	/system/sd/btrfsdata (5.6G total)
CACHE2RAM (/cache in RAM):used 24.0M	/cache (90M total)
DALVIK (/data/dalvik-cache in NAND):used 120.5M	/data/dalvik-cache
[Desire Android Development] melethron:
Yeah i took a look at it. Since my information was about large ssd-device storages (not flash media) i think a 512kb align would be enough. Thats also the max value in the samsung presentation i linkend (2-4kb pages grouped to blocks of 64-128 nand pages = 128kb-512kb) . And wiki quotes the same.....

>> n*1024

I think n*1024 would be enough. Although more doesn't hurt.

Afaik all that is need for a proper alignment have the sectors start and end at any value for n.

Also if you use the app2sd script i think it should be rewritten. Mounting sd to data directly and then symlink back to dalvik on internal. So /data/data gets not corrupted. If dalvik gets corrupted > wipe it for now. (if dalvik on internal would even be of any use. An aligned nilfs2 might be faster.

To unmount internal anyone still someone might use "Vold" for this, if its possible.
[Desire Android Development] ownhere:
Thank sibere and melethron.

After reading all the posts, I said my point of view:

Safe umount is useful but not enough, Because the phone is a communication tool rather than entertainment tools, So stability is the most important. Once forced to remove the battery to cause a large number of FC is not acceptable(for example:drop phone from table or pants pocket), This is why the iphone and ipad design is not removable battery(I think, Use of non-replaceable battery design, iOS needn't consider the problem of sudden power loss, so it can use a larger cache and log less), So the stock ROM use yaffs2 for NAND FS and Samsung phone use RFS.

In theory, ext4 without journaling have same performance as ext2, even better because Extends feature from ext4.

I tryed nilfs2 before ext4, it totally crash my system, a lots of FC when first boot, and disk space is strange, 500M datas use 1G spaces.I don't know why but seems that nilfs2 is not stable.

And I considered brtfs, but when I search benchmark about brtfs, I found a report about brtfs have bug when a lots of read/write. In kernel source, brtfs is marked with 'EXPERIMENTAL'.
[Desire Android Development] ownhere:
I had a IO statistics, found in /data/app and /data/dalvik-cache a large number of read requests, almost no written request, in /data/data a large number of read requests is also a large number of write requests, so they give /data/app and /data /data using a different file system may be effective, I consider to /data/app use the XFS file system, because he is said to have read and write large files better performance, /data/data use EXT4 or REISERFS
[Desire Android Development] woti23:
same test as above, but btrfs without option ssd

/dev/block/mmcblk0p6 on /system/sd type btrfs (rw,relatime,nossd)

quadrant average -200 lower
[Desire Android Development] woti23:
well, but data/app has not really large files nor has /data/data or dalvik-cache

root@localhost://data/app# >find -size +5000k -print |wc -l

3

(total 156, 106 < 1MB)

root@localhost://data/data# >find -size +5000k -print |wc -l

3

(total 2185, 2105 < 100k)

root@localhost://data/dalvik-cache# >find -size +5000k -print |wc -l

4

(total 289, 230 < 500k)
[Desire Android Development] melethron:
Also thank you.

1. If you used nilfs2 could you upload the compiled tools pls.

2. About the space and the crashes. Are you sure you implemented it right in the kernel? If your space gets more full this means the "garbage collector" doesn't work as it should.

i make an example how it works. O=data writen X=outdated stuff _= erased space:

You start with an empty device:

_____________________

Then stuff gets writen to it.

OOOOOOO_______________

New Data is writen simply "behind" the old (logbased filesystem) and some data gets outdated:

OXOOXOXOOOOOOO______

This is were the Garbage Collecter now starts its work. If the phone is idle it overwrites it THEN:

O_OO_O_OOOOOOO______

This is the big advantage of it since the erase is the bottle neck of an flash memory.

Considering your issues this might be due to the "Garbage Collector" service failing anyhow.

3. About iPhone and the internal battery. A friend of mine really hates this. He had an issue with a badly coded app. System hang. He couldn't do anything. No shutdown -> ONLY hibernate. And no battery pull -> build in. He was expecting an important call. Since he wasn't at home (were he could have used iTunes to restart the phon) he couldn't do anything.... so THIS should be better for communication?

i give you an advice to prevent powerloss: DONT DROP YOUR PHONE :D:D
[Desire Android Development] NeoPhyTe.x360:
We can mount system partitions to XFS per example formating them from...recovery? or is not an option to make totally of system faster? and then modding it on init.rc
[Desire Android Development] ownhere:
EXT2/3/4 store file base BLOCK(1024,2048,4096), any file large than the block size I think is large. :)

example:

Write a 1024Mbytes file to EXT2 partition with 1024bytes block size,that means the file data be write in 1024000/1024=1000 blocks.Kernel need more time to addressing.This is why larger block size better performance.

And, EXT4 use Extends to reduce large file blocks, so I use it replace EXT2.

this is from EXT4 WiKi:

Extents

Extents replace the traditional block mapping scheme used by ext2/3 filesystems. An extent is a range of contiguous physical blocks, improving large file performance and reducing fragmentation. A single extent in ext4 can map up to 128 MB of contiguous space with a 4 KB block size.[1] There can be 4 extents stored in the inode. When there are more than 4 extents to a file, the rest of the extents are indexed in an Htree.
[Desire Android Development] ownhere:
NILFS2 is a good file system for SSD, so I try it first. I compiled my kernel with NILFS2 support.

But the FC is too annoying. I format my SD with nilfs2 default option, no any dangerous tuning, then it FC, I cannot accpet it.With EXT4 do same thing is very stable.

with NILFS2, I try run nilfs2_cleard for garbage collector, but the disk space not return to normal, still full.

iOS without JB is stable for most people, this is enough, most people only need itunes top 10 apps/games. :)

And, in Android, a bad app be able to crash system, Forcibly stripped off the battery may only be resolved.
[Desire Android Development] melethron:
About your issues with nilfs2. The default settings ("etc/nilfcleanerd.conf") for the "garbage cleaner process" do have these settings (and some more):

# Protection period in second.
protection_period	3600

# Minimum number of clean segments
# 0 = continuous cleaning
# > 0 = pause cleaning until less segments are available
min_clean_segments	10%

# Maximum number of clean segments
max_clean_segments	20%

This means there wont be ANY auto erasing/overwriting done before the card isn't 90% full. On top of that there wont be any state present be erased for 1 hour !!!!!!!! . If nilfs2 isn't used for backup porpuse these setting don't make any sense. 30s should be enough to garantue data intigrity on powerloss. Also you can set the min to 30% or something. I'll play with the settings tommorow and make some speed tests on my usb stick.

nilfs2_cleard also wont do anything to data that is marked as delete within the last hour if "protection_period	3600" is set.

Thanks for the utils. I should really learn how to cross compile. For now i did only compiling on my ubuntu.
[Desire Android Development] woti23:
sorry, than i misunderstood your words here:

"I consider to /data/app use the XFS file system, because he is said to have read and write large files better performance"

i didn't know that xfs sees 100kB files as large.

just out of interesst: whats huge_file_support then? 1MB ;-)
[Desire Android Development] woti23:
same here, compiled kernel support for nlsfs2 in ubuntu.

formated usb micro sdhc reader stick

mounted, tryed to run postmark, bumm - kernel panix

formated again, now postmark runs.

remove usb stick without unmount -> kernel panic
[Desire Android Development] sibere:
EXT4 is not a bad idea if it's without journaling (wear/performance issues) but then the FS is not really more robust than EXT2. For block size, 4kb is 4kb. You can have 4kb block on ext2 too. To me, the extent thing is design for HDD because of seek time. in HDDs, if the head must move to an other part of the disk to read the rest of the data, it's a big loss of performance. Extent is just making sure that the next 4kb block will be attached to the previous one.

quote: "Extents replace the traditional block mapping scheme used by ext2/3 filesystems. An extent is a range of contiguous physical blocks, improving large file performance and reducing fragmentation."

On flash memory, random access time is almost 0ms and there is no reason to write the rest of data to the next block. Fragmentation means nothing for flash memory with 0ms access time :)

For the battery, you must have a problem with you phone ;) I droped my phone a couple of times and the back always strayed well clipped to the phone :)
[Desire Android Development] melethron:
WTF... i just did that while mounted (im on ext4 atm to test my rom)

bash-3.2# tune2fs -l /dev/block/mmcblk0p2
tune2fs 1.40.8 (13-Mar-2008)
Filesystem volume name: sdext
Last mounted on: /data
Filesystem UUID: bb959a67-c788-498d-baab-03874d3279ea
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extents FEATURE_I9 sparse_super large_file huge_file gdt_checksum dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: [COLOR="Red"]clean[/COLOR]
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 131072
Block count: 524032
Reserved block count: 26201
Free blocks: 467215
Free inodes: 130280
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 127
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Filesystem created: Fri Dec 17 18:56:43 2010
Last mount time: Fri Dec 17 22:31:10 2010
Last write time: Fri Dec 17 22:31:10 2010
Mount count: 5
Maximum mount count: 39
Last checked: Fri Dec 17 18:56:43 2010
Check interval: 15552000 (6 months)
Next check after: Wed Jun 15 19:56:43 2011
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: c0bc9d96-b87d-4cbc-8cc1-358039adede1
Journal backup: inode blocks

How did you mount that the unclean flag was set?

Busybox? what version?

Or normal mount?
[Desire Android Development] sibere:
whether it's an ext4 feature to be mounted clean... whether something is wrong ;)

I used the busybox available on my thread's OP with options on my scripts.

BTW still not able to unmount mtdblock5 at boot (reported busy). What rom are you using? S-OFF?
[Desire Android Development] kraml:
deleted becaused replied to the wrong thread....
[Desire Android Development] jogumsim:
Hey guys,

I've tried to unmount the /sd-ext partition successfully before "shutdown" on my Desire. Maybe you can make it happen on other phone models or other different sd-ext usage.

First of all, I split a 2GB partition on my sdcard mounted as ext3 to /sd-ext, and I move only 3 dirs to /sd-ext with symlinks:

/data/app ==> /sd-ext/app

/data/data ==> /sd-ext/data

/data/dalvik-cache ==> /sd-ext/dalvik-cache

The /data is still left at /dev/block/mtdblock5 on my phone.

All I've done to safely unmount:

1. modify the /init.rc

service shutdown [B]/system/etc/shutdown.sh[/B]
user root
group root
disabled
oneshot

2. edit the shutdown.sh lauched by the "shutdown service"

#!/system/bin/sh
# stop whole dalvik things then unmount before shutdown
[B]stop zygote[/B]
sync;sync;sync
[B]sleep 3[/B]
umount /sd-ext
/system/bin/shutdown

To unmount a file system safely and successfully not by force, there should be no files opened from it. Before executing /system/bin/shutdown, there are still many processes alived. In my case, the whole dalvik platform will lock my sd-ext partion. The "sotp zygote" command will stop the dalvik vm and all it's child processes. You can't just 'kill' it since it's a system service started by init.

The "sleep 3" command I used here is ugly and stupid. You should do some other elegant checking to ensure the service has been stopped and clean.

Last but not the least, the /data partition still can't be unmount.

This won't work if you mount the whole /data to sdcard. But I think it could be done by finding out which processes with opened files in /data/*. :-)
[Desire Android Development] ownhere:
great idea!

maybe this script is better:

#!/system/bin/sh

stop zygote
sleep 5
cat /proc/mounts|awk '{if ($1 ~ /(loop|mmcblk)/) {print "/system/xbin/busybox mount -o ro,remount " $1 " " $2 }}'|/system/xbin/busybox sh
/system/bin/shutdown
[Desire Android Development] jogumsim:
IMHO, I would suggest not to use remount. I've been through this and get the file system not cleanly unmounted with something reported by e2fsck and tune2fs.

The most important thing is to make sure the related processes/services are down, then the unmount command would be fine.

Leave the sync command is fine bcos I didn't mount with sync option.:)

By the way, it should work to unmount the /data after "stop keystore", at least it works on my Desire. The keysotre service in /init.rc will related to the /data/misc/keystore.

service keystore /system/bin/keystore /data/misc/keystore
user keystore
group keystore
socket keystore stream 666
[Desire Android Development] ownhere:
great again. ;)

umount or remount for me is same. all get clean flag.

put shutdown.sh to ramdisk maybe better. all things be done by ramdisk self.

service shutdown /shutdown.sh
[Desire Android Development] melethron:
Hey,

Data is unmounted if you shutdown normal. I mount sd-ext on /data and as long i do a "normal" shutdown (no reboot - its unclean then) it is "clean" or for ext2/3 the "need recovery" flag is unset. But this would still help to unmount mtd5 then.
[Desire Android Development] sibere:
I think that the clean unmount of /data is a HTC sense rom thing. I'm not sure that it's the same for other roms like AOSP.

In this particular case, this mod is just fantastic to prevent filesystem corruption.

For sense roms, don't bother about patching init.rc, HTC have done it for us :)
[Desire Android Development] melethron:
I think you are right on this. Supercurio of voodoo lagfix said that galaxy S doesn't unmount /data. He thought it was an issue of samsung. But if this also the case of AOSP then it should be an issue of Android in general.

Well it can still be of use to unmount mtd5 on data2sd on sense roms.
[Desire Android Development] sibere:
Yep.

ATM ext loop file for me with dalvik on mtdblock5. running well so far :)