yashsingh1/xdaforums huggingface dataset:
[Epic 4G General] plapczyn:
Yeah, I should have put it in general. I wasn't trying to flame, but generally speaking, quoting ones self doesn't count as a valid citation.
[Epic 4G General] plapczyn:
EXT4_lifecycle = RFS_flash_lifecycle * (EXT4_write_cycles/RFS_write_cycles)
[Epic 4G General] kenvan19:
Thank you so much for your expertise on this matter plapczyn and for clearing up this dispute! I would add a thanks to you but alas, I have used all of mine today hehe! I suppose I may be one of the few (or many, who knows?) people who really enjoy reading technical responses that are beyond my personal understanding as they generally allow me to glean information that was previously out of my grasp! I really appreciate your detailed response!
[Epic 4G General] msponsler:
Great information plapczyn.

But I think a lot of this stems to how ext4 behaves compared to rfs.

To be frank, I know very little about rfs beyond the fact that it's basically vfat with journaling support. To be honest, that sounds horrible...

ext4 on the other hand, I've got a decent grasp on it:

The real danger of writes with ext4 on nand flash comes from the meta-data blocks. Luckily in ext4 (unlike ext3), the meta-data blocks can be (and are) moved. The 128MB per block (in a 4KB block file system) restriction is removed (each 128MB block required a dedicated meta-data block).

Meaning, mke2fs (part of ext4) can MOVE the meta-data blocks around outside the large virtual block group as the are grown and shrank. Which means that the meta-data blocks aren't constantly written to the same spots, spreading out the meta-data writes across the storage.

The delayed allocation feature of ext4, in addition to the block allocator (mballoc) significantly reduces fragmentation -- in addition to vastly increasing performance. Decreased fragmentation means less move(); rename(); write(); delete(); operations to fit your data in the allocated blocks, thus decreasing wear on the nand (re-writing & updating meta-data) -- atleast in comparison to ext2/ext3. See the part above on how I know very little about rfs, I can't speak on how rfs handles fragmentation and block allocation. But considering how fragmented vfat gets...

But let's put some stuff into perspective:

Does ext4 create more I/O overhead (delete(); / write(); operations specifically) than rfs? Possibly. Some very valid questions were raised -- and questions like this NEED to be raised and debated.

journaling doesn't need to be enabled on your phone. That will alleviate a great deal of the writes if you are worried about it.

Is ext4 a good idea for nand flash on Linux running a database from a reliability stand point? Hell. No.

But a lot of writes in android's /system directory running ext4? Not likely. Sure it would wear out, but probably after a few years. Besides, doesn't samsung have wear-leveling in the controller to the nand? All Android and ext4 sees is the logical level. Which would render this whole argument moot.
[Epic 4G General] kenvan19:
To clarify a single point, and I apologize if this is a stupid question, I have read and heard that disabling journaling does increase the risk of corrupted data but how real is that risk (meaning how much of a danger is it really?) and if data is corrupted would it effect the ability of the system to function or would it merely be 'cosmetic' (for lack of a better word) corruption?
[Epic 4G General] h2g2:
Wear leveling for the OneNAND is implemented in software, not in hardware, in the STL as plapczyn mentioned in his post. From what I can tell, STL is responsible for exposing the OneNAND chip as a block device and from there, you can format the device using RFS, YAFFS, EXT4 or whatever.

Another note here is that the type of wear-leveling that RFS and YAFFS do is called dynamic wear leveling which means wear-leveling is only done on write ops as I understand it. SSDs use static wear leveling that is capable of moving, for example, data from blocks that change rarely over to blocks that change frequently in an effort to give said blocks a chance to "rest". The wear-leveling in RFS and YAFFS doesn't do this.

In addition, SSDs include extra NAND chips and only expose some percentage of the total capacity so that extra blocks are available for wear leveling. From what I understand, the OneNAND chip has no "extra" capacity for this purpose. This means that the more total space you allocate with static data, the quicker you'll run into problems because you'll be reusing a smaller set of blocks over and over for writes. This can be overcome with careful partitioning and, of course, maintaining a reasonable amount of free space.

But, dynamic wear leveling is used in USB memory sticks and most flash memory cards as well (including Micro SDHC cards which is why they can stay on FAT/FAT32 without issues). Lots of folks run, for example, web browsers off USB memory sticks for years - I have an old 1GB drive that's several years old and I keep a copy of PortableChrome on it. All the transient data like the browser cache and history files are kept on the stick and I haven't run into any problems yet. Also note that the MLC NAND chips typically found in these devices are only rated for 10,000 P/E cycles instead of the 100,000 for SLC chips like the OneNAND.

I'm sure that someone can concoct some nightmare-scenario or torture test that will easily result in blowing past the P/E cycle limit on some blocks but realistically speaking, it would need to be a LOT of continuous activity to run into those limits. Overall, even with EXT4, the OneNAND chip is going to be far more durable than your average USB memory stick or memory card for your camera. Granted, the usage patterns aren't exactly the same but then again, OneNAND is good for an order of magnitude more P/E cycles vs. the MLC chips found in these solutions.
[Epic 4G General] msponsler:
Journaling is there in order to rebuild data in the event of a power loss mid write.

A journal will rebuild the last write operation, staving off data corruption.

But let me ask you this...do you routinely start copying files on your phone and pull the battery? Probably not. Which is why journaling isn't very important on phones. You just have to wait the extra 10 seconds for the phone to shut down.
[Epic 4G General] kenvan19:
Wonderful! Thank you for clearing that up!
[Epic 4G General] art3mis-nyc:
So question, since Ext4 is efficient for stuff like usb style flash etc, and "bad"(note the quotes cuz of the claims) for phone style flash, wouldnt it be beneficial if the kernel/initramfs supported it to multi-format? essentially a partial "lagfix" where /cache and /data get partitioned to a mount on the SD and the multi writes/reads and the kernel and /system lives on the onenand, while moving the datadb partition to rfs?

more or less a hybrid where you gain the advantages of each, potentially a performance boost and reduce the wear and tear so to speak?

** just thinking of it all would love an explanation if im wrong in my thinking of behaviors of it.

*** Also because CW 3.0.0.5 supports the ability to partition however you tell it, yopu could multi partition this way, also couldnt we technically mimic google and shift to yaffs pretty easy as well(same way we did with ext4)?
[Epic 4G General] plapczyn:
Exactly. It shouldn't matter if the RO partitions (system, kernel, radio) are formatted as EXT4 because the whole concern regarding program/erase cycles is mute. We should use whatever gives the best performance. Also, if it is truely that big of a deal, we could always go to YAFFS the same way we did with EXT4. Since google announced EXT4 as the default FS for 2.3+, I doubt it really was 'necessary' for YAFFS on the NAND. It is possible...that they chose to use a different low level device driver (faster?) and do the wear leveling in the FS layer.
[Epic 4G General] lviv73:
While still on the subject,how come none of these devs use RaiserFS in their roms?Raiserfs suposed to be real fast with small files.There are few Evo/Nexus roms/kernels with Raiserfs implemented and users report big speed boost over Ext4.
[Epic 4G General] kenvan19:
There must be a reason devs have not ported this. Perhaps there are issues?
[Epic 4G General] aal1:
Look, Im pretty techie myself, been a server administrator/architecht for 25+ years....linux, vmware, windows, etc. So I'm not dumb lol. Just wanted to understand a bit more....please bear with me on this....but can someone line-item out what benefit ext4 is going to give me on my epic running froyo?

maybe something like:

epic with dk28 (froyo) non-ext4 : blah blah

epic with DK28 (froyo) on ext 4 : blah blah PLUS blah BLAH brrrb BLAH and its faster or whatever.
[Epic 4G General] msponsler:
ThoughYou could move to reiserfs. I used it for many yeas, and I greatly enjoyed it.

However...reiserfs is a dead project. Hans reiser is in jail for mudering his wife. Reiserfs isn't without its problems though. It works we'll with files under 4kb. But it still uses the "big kernel lock", which is not the way to go IMHO. And reiserfs does suffer from degredation over time.

As far as the benefits of ext4:

Extents replacing block mapping improves I/O performance. It uses h-trees in the meta-data to drastically improve file location time. Actually mathematical genius...

Mutiple block allocators improve I/O time.

Ext4 is multi-threaded...yaffs2, rfs, ext2/3 are not

Allocate on flush, meaning blocks arent written until the data is ready. You'll sacrifice cpu/memory for I/O through put, but it does improve performance while reducing fragmentation.

15 years as a Linux and Solaris admin and engineer here. Always nice to meet another SA / Engineer!
[Epic 4G General] aal1:
hmmm ext4 is multi-threaded how?

let me throw a bit of what i understand here into the equation and ask differently....vmfs3 (esx filesystem) is comparable to ext3 right ? seems like I can multi-thread like a maniac on that filesystem with umpteen vms right? so basically ext4 is just another filesystem, like fat16 and ntfs, etc , and so we are just seeing the benefits of something written for newer SSD hardware in ext4?

Not trying to be any more newbish lol decided to just research more and found this little article....

http://www.phoronix.com/scan.php?page=article&item=ext4_btrfs_nilfs2&num=7

change the last number in the link to review the whole 7 page article.

It looks to me that ext4 is an upgrade over ext3 and shouldnt be such a big worry - as you said, it seems to have less frag problems, can write faster, writes in a different manner in order to get better throughput.

rhel6 pushes ext4 as a default, with btfrs as an option - which the above link basically proves to me with their testing. In short, ext4 is ready for primetime and works well for linux systems. And the others might not be.

So if you want a little more out of your phone, go ext4. want to be safe, stay where you are.

Since most of us wont have this phone in even 1 more year, more performance is the reason we are here anyways lol...so ext4 here I come I guess )

and a big SUP! to a fellow SA ) thanks for the info, I appreciate it.
[Epic 4G General] leatherneck6017:
Does this mean that I shouldn't be using QuickBoot while running an ext4 file system with journaling disabled?
[Epic 4G General] h2g2:
EXT4 isn't multi-threading but it supports file-level locking. YAFFS only supports partition-level locking. That means that only a single thread can write to a YAFFS partition at any moment in time. So long as that thread has a lock open, all other write operations to that partition are blocked until the lock is released.
[Epic 4G General] h2g2:
VMFS is a clustered file system more analogous to GPFS than EXT4.

EXT4 was also an evolutionary advancement to EXT2 and EXT3. I don't believe it was specifically written with SSDs in mind. BTRFS is the long term replacement for EXT*. Once it's ready for primetime, BTRFS with snapshots, checksumming, etc. disabled is probably the best choice for Android phones.

Another alternative is a hybrid approach like flashcache or bcache - utilizing a smaller OneNAND SLC chip for caching and a larger MLC eMMC for capacity. Probably would be too much overhead on the current crop of single-core smartphones but future dual-core designs might be able to pull it off.
[Epic 4G General] lviv73:
So in your opinion Ext4 is the best filesystem?

What about Xfs?Xfs supposed almost as fast as Raiserfs,and isnt Raiserfs been picked up and just been updated as Raiser4?
[Epic 4G General] ecooce:
Journaling is very important,I ran my Moment with ext2 and it got corrupted atleast 3 times,once I had to pull the battery because it froze.Another time I dropped the phone,battery popped out.The third time I couldnt tell what happened,I did get a freeze that lasted 2 minutes,never pulled the battery,waited about 5 minutes after freeze,rebooted,and it still got corrupted,FC's everywhere.
[Epic 4G General] h2g2:
XFS has been deprecated for some time now. I believe SGI created that one for their IRIX systems.

Reiser4 was in development when Hans Reiser was convicted and jailed. Supposedly it is still being worked on but I don't think it ever made it upstream into the kernel.

The Linux devs have pretty much settled on Ext4 as the interim solution while BTRFS gets ready for prime time.
[Epic 4G General] msponsler:
Maybe running ext3 (basically ext2 plus journalling) would have saved the corruption. Journalling will recover from a write being made during a power loss. If the OS wrote corrupted data on "purpose", it'll journal and rewrite said corruptiom during a recovery. Journalling isn't the end all of data corruption. Much like raid 1 isn't really a true disaster recovery method -- although so many "engineers" and management seem to think so. But I digress.

I don't know, but looking at logs would paint a much clearer picture. Misplaced inodes fixed through an fsck should have fixed it, with "lost" files being in the lost and found directory. Who knows...

Either JFS or XFS....One was developed by IBM for AIX and the other was developed by SGI for IRIX. Both are in the linux kernel. I think JFS was the first jounaled file system. It's been a while... the only thing I remember about them is being slower than reiserfs at the time, and losing files to corruption on both xfs and jfs. But that was something like 7 years ago, so my memory is fuzzy.

Btrfs is striving to be what veritas has in vxfs (my personal fav) and sun has in zfs. Btrfs has a few years to go before its fully cooked.
[Epic 4G General] nubecoder:
I regreat that the only thing I can add to this thread is that, I welcome more threads like this one.

Good info presented, sources cited, links given, and a mature discussion to boot

Thank you.

=]

ps- please don't let my post side-track this discussion. [sorry]
[Epic 4G General] h2g2:
In particular, some journaling file systems like NTFS only implement metadata journaling. Metadata journaling can help quickly recover a partition to a consistent state but it won't actually help you recover data. EXT4 does implement a full journal but due to the usage of delayed allocation, data may not be committed to the journal during a system crash.

JFS was IBM, XFS was SGI. Neither are very crash resilient. Performance is decent although XFS was known for high CPU and memory utilization. Both only implemented metadata journaling as well as delayed allocation - not a good combination for crash resilience.

I have to say ZFS is my personal favorite. Hybrid storage pools are amazing. On my work PC, I use EXT4 + Bcache which is coming along quite impressively for a single-developer kernel extension. Bcache is pretty simple and elegant, I'm hoping that once its stable enough to make it upstream, that we might see it implemented in smartphones.
[Epic 4G General] plapczyn:
Thanks. The other thread just had too much contradicting information in it with no references...although, I believe they have ended with the same conclusion.
[Epic 4G General] huytrang90:
Thanks all for good conversation and contributing to this. It helps noobs, app developer like me to learn more about internal working of system and file I/O.

Great work!! We need more threads like this.
[Epic 4G General] UnicronAlpha99:
Yeah this is good stuff... thanks for the factual data.
[Epic 4G General] plapczyn:
Updated post to add info on bad blocks and more links to references.
[Epic 4G General] plapczyn:
Updated OP with some numbers for the 2% reserve area.
[Epic 4G General] Electrofreak:
I wish every thread on this forum was more like this one. Well written, well argued, well done.
[Epic 4G General] plapczyn:
The trick is to put relevant info back into the OP. Otherwise you end up having to read 100 pages of **** to find out that the original post wasn't correct, or was missing key info. But unfortunately posts stay on the front page based on recency of replies, so a post that doesn't stir up a huge debate based on misinformation, or lack thereof, ends up getting lost in the swamp of garbage threads.

...it also helps if you part of your job entails writing technical documentation for the topic you are posting about. :)
[Epic 4G General] sauron0101:
I've had a day to think about this in more detail. Instinctively, I got the feeling that ext4 was fine when the first post about this issue was made.

I was thinking when I walked into a data centre recently. They have been running a Red Hat Linux server now for several years using mainly SSDs with a controller made by a company named SandForce and the SSDs from a variety of vendors. Anyways, they've been running online for years without a hitch or fear of data loss, although they do have extensive backups (advisable for any setup). They also have little downtime as their data centre had a fairly high level of activity (mainly internal customers from what I understood).

Now there are differences between SSDs for servers and the flash memory on your phone of course, but the fundamental principle is the same - they both use the NAND flash technology. I think that for duration of the phone's lifespan, we'll be fine. If a data centre can run things for years with a fairly high level of activity, a phone can certainly do so without any problems.
[Epic 4G General] boominz28:
man after reading this my brain hurts just trying to understand it. I think I will just keep my phone stock and not worry about it lol.

Great thread btw even if I dont understand it.
[Epic 4G General] asapelectric:
I didn't see anyone answer this directly. Like a yes or no?:rolleyes:
[Epic 4G General] DiGi760:
My gut feeling is that if you disabled journaling, then you should probably avoid quickboot and the three finger reboot. Or manually run an e2fsck.
[Epic 4G General] asapelectric:
Why the death grip,too? You can't exucute that until its had time to shut down, giving it time to finish writing.
[Epic 4G General] DiGi760:
I've always noticed the deathgrip to be immediate. Shutdown takes a good 15 seconds. Am I wrong here?
[Epic 4G General] plapczyn:
Deathgrip from power off is fine.
[Epic 4G General] gTen:
Here is the thing though..when running it on a server you are dealing with 32gb - 1tb space, nto to mention in a RAID..when you have that many blocks you can survive quite some time as they shuffle...

so a 32gb intel SSD for servers is rated for 1 petabyte of data

a 64gb intel SSD for servers is rated for 2 petabytes of data

and so on..

We are only dealing with 1gb of available space hence we are more susceptible then an SSD...
[Epic 4G General] plapczyn:
512 MB actually. You are also running MOSTLY out of RAM and you aren't running a website with thousands of transactions a day. The comparison is not very good. If I were running ebay off of my phone you would have a valid point. I'm not.

BTW, I updated the further reading section to the OP.
[Epic 4G General] DiGi760:
What about when the phone is on?

vol. up + cam + power gives an instant reboot.
[Epic 4G General] gzub:
Are there any counters in the /proc filesystem which can allow us to benchmark EXT4 vs RFS at the sector translation level? It seems that we could then perform identical functions on an EXT4 & RFS formatted phone and determine the difference (if any) in writes.

Geoff
[Epic 4G General] h2g2:
Try:

#cat /proc/diskstats

Some docs:

http://code.google.com/p/m8-android...ocumentation/ABI/testing/procfs-diskstats?r=7

http://code.google.com/p/m8-android-kernel/source/browse/trunk/Documentation/iostats.txt?r=7

I haven't read through the docs enough to know how to munge the data but that seems like what you'd need.
[Epic 4G General] MeetFace:
Plap, when I try to run this command through adb, I get "Permission denied" right after typing the "i" in bmlinfo...what am I doing wrong?

I just updated all my J through SDK Manager, and adb devices shows my device properly.

Tx for your help...and your article/thread is a thing of beauty.
[Epic 4G General] Electrofreak:
did you watch your phone for a permissions message after your su from adb? if you haven't ever used su from adb before, Superuser.apk will ask if you want to grant permissions to "Unknown" and you'll have to click Yes or the su will be denied.
[Epic 4G General] MeetFace:
Sweet, I'll try this when I get home.
[Epic 4G General] plapczyn:
You don't actually need the su command. Just ran this on my coworkers unrooted epic. Btw, he had 5 bad blocks and is running stock di18. Never rooted. My phone has three bad blocks. It's been rooted and flashed many times since I got it and I'm running ext4.
[Epic 4G General] rjr162:
I'm curious how btrfs would run once its further developed.. Or get ZFS rolling and see how it does
[Epic 4G General] Darkshado:
I found the article while trying to understand more about lagfixes and using file systems other than RFS on Samsung phones.

From a longevity standpoint, it seems there are no worries to be had with either RFS or ext4, or most other file systems for that matter.

From an integrity standpoint: journaling is better than no journaling, and there are various settings depending on what file system you're using.

Performance-wise: no journaling is better. Big caveat is that you risk corruption in case of a crash or abnormal shutdown.

My situation: I'm running a Galaxy Gio which uses RFS. I'm wondering if formatting, say, to ext4 instead, would yield performance gains. I do want to keep journaling for data integrity.

How would I measure the difference in an absolute, quantitative manner? (Not just a subjective "my phone runs faster")

Goodbye and thanks in advance,

Darkshado
[Epic 4G General] JohnCorleone:
I believe Quadrant Advanced gives you a numerical breakdown on I/O reads and writes. I know "it feels faster" isn't what you're looking for but when running EXT4 you can really tell the difference. I have been running EXT4 with journaling for months and I find it rough to run RFS..at least on the Epic. I will have to see about my other phone tomorrow. I believe SQ Lite RL benchmark will give you some numbers as well
[Android Software/Hacking General [Developers Only]] shad0wboss:
http://www.youtube.com/watch?v=9VqWtf5X2P0

Just saw this video and wondered if its possible to have ubuntu on android devices with 600+mhz processor...btw i've got motorola defy (similar to droid) so if it's already possible and if someone knows about it, please let me know.
[Android Software/Hacking General [Developers Only]] phoenixhel:
I just installed ubunutu on my HTC incredible and it works pretty well. Using the GUI so I can play around with Firefox and other apps. I went to

h$$p://nexusonehacks.net/nexus-one-hacks/how-to-install-ubuntu-on-your-nexus-oneandroid/

Worked like a charm and only took about 45 minutes(mostly time to download the file).

(replace $$ with t, I haven't posted enough to put links yet)
[Android Software/Hacking General [Developers Only]] AndroidSamurai:
Back when me and Haykuro were slapping together the first rooted/custom roms for the TMO G1 we installed Debian on our phones...

Ran like a charm, but that was when things like busybox and init files were harder to come by.

With enough "tweaking" and patience it is possible to run a smaller distro of linux like Debian on almost any android device. Seeing that you have root anyway...