yashsingh1/xdaforums huggingface dataset:
[Captivate Android Development] AJerman:
I think I've replied to this at one point before.

1. You're comparing internet download speed to system speed. The news feeds and pictures all require connecting to the server, checking for new items, downloading, etc, therefore don't really compare to what increases in system speed would bring.

And I don't use the other 3 apps. How about some stock apps that everyone has? The Market app for example. Always took a few seconds to load, and took quite a while to open the Downloads page of apps I already had installed. Both are very fast now. Also, before downloading an installing apps would nearly lock up the phone for a few seconds while it installed. Probably not assisted by Lookout running antivirus as well. Regardless, the system would just completely stop responding to touch for a few seconds when the apps installed, that's non-existent now.

I'm running the stock kernel mounting an ext3 partition for the data folder on a Lexar class 6 8 GB card that tested at 11 MBps write and 17 MBps read.
[Captivate Android Development] ranova:
No, I guarantee you it is NOT bandwith related. Each was tested on my personal wifi connection. It is definitely system lag, as once I tapped on an email, tweet, etc, I could not do anything else (notification panel, scroll on the current frame, etc). If you read in the past few pages, another person was complaining about Touchdown lagging even with the ext hacks, and had to resort back to the NAND trick (see: http://xdaforums.com/showthread.php?p=7437288#post7437288)

I did see a performance increase in marketplace with the ext hack compared to stock (stock download page would take a LONG time), but it still does not compare with the speed of the NAND trick. Most people that are swearing on the Ext tests, have you tried the NAND hack? I really do not mean to bash it, but after testing it myself, and others confirming that it does not beat the NAND trick, I am very skeptical.

The main apps that everyone has do not lag HORRIBLY with stock, ext hack or NAND hack. All the apps I have listed are free (touchdown has a 30-day trial) and are very common apps.

Nuka confirmed that Dolphin HD is fine for him with the Ext hack, but without actual tests with multiple applications, hypotheses do not help anyone.

I'm tempted to reflash to the ext4 hack (I still have the data on my microsd card) and make a video to show you guys :)

I suggest you try out touchdown (try adding your gmail account, but I was having lag problems using my corporate exchange) and Dolphin HD browser (which is a far superior browser to stock).

First, test the application with the ext hack (opening it, browsing, new tabs, read emails, etc), then apply the NAND hack to just that single application (you can use the NAND hack on top of the EXT hack - make sure you force close the app before applying the NAND hack):

mkdir /dbdata/data
cd /data/data
for i in com.nitrodesk.* ; do echo $i ; busybox mv $i /dbdata/data/ ; busybox ln -s /dbdata/data/$i $i ; done

if you want to revert this:

cd /dbdata/data
for i in com.nitrodesk.* ; do echo $i ; busybox mv $i /data/data/; done

for Dolphin HD, replace com.nitrodesk.* with mobi.mgeek.TunnyBrowser with the above code
[Captivate Android Development] ranova:
also please note your microSD -

Size, class and manufacturer

There was also a theory on why some people's quadrant tests take A LONG time during the I/O tests and still end up with scores over than 1500. It is speculated that those with Kingstons (which is what I have) have very slow I/O speeds - which might cause the lag difference with the Ext speed hack
[Captivate Android Development] kennethpenn:
For this type of fix you should buy a SanDisk microSD. Don't worry about class, size, or what brands claim. SanDisk works the best for these mimocan-style fixes.

Kingston is known to have problems... even Class 6 ones.

I have a 16 GB class 2 SanDisk.
[Captivate Android Development] rob989_69:
I'm running a 8GB Class 6 Transcend. I was definately seeing a lag (no more than normal) during the I/O test and super high 1600 + scores.

Has anyone run any other benchmark tests with this kernal? Such as linpack or benchmark?
[Captivate Android Development] Unhelpful:
Computational and graphics benchmarks shouldn't be affected one way or the other by this fix. DB and general I/O is where you can expect to see things change, for better or worse.

FYI, to anybody else building their own kernels, I have noticed that Odin seems to particularly dislike tarfiles made with 7zip in windows. These almost never work for me, while ones made with GNU tar usually do. I'm trying the hooks and the hooks test stuff again, I screwed up the initrd images I was using with them and I *hope* that that is why even the stock initrd was reboot-looping. :/
[Captivate Android Development] MSigler:
If you haven't already, try downloading samsung kies. I'm not really sure why, I'm guessing it has some kind of additional drivers or something, but I did all kinds of removing cards, removing batteries, holding buttons, releasing buttons, etc, and only got the phone to connect with kies open. Or maybe it was opening kies, then opening odin, then closing kies, then connecting the phone or something equally stupid, I honestly tried so many variations I can't recall.

The phone would show up in ADB, would show up as mass storage, and ODIN would either not detect it, or hang at "analyzing files". After messing with kies it worked.

Also another thing that seemed to work was trying to update in kies, letting it hang, then pulling the battery, getting the phone ! computer icons, then doing the "Hold all buttons, release power on second blink" thing to get it into download.

Basically I'm agreeing as hard as I f'n can. Coming from a Nexus One that just runs vanilla android with no BS, this is a nightmare to deal with, and especially to try to help the less experienced guys with.
[Captivate Android Development] _Opiate_:
Lol yes i can agree as I also own a Nexus One and the difference between working on my Nexus and working on the Captivate is ridiculous.
[Captivate Android Development] shep211:
I talked to Mimocan and asked if he could help out. He said he thought we had a fix already. I linked this thread for him. He said he will work on a kernel for us asap.
[Captivate Android Development] rob989_69:
Awesome. It seems for whatever reason his fix realy sped up the phone, not just benchmark tests.
[Captivate Android Development] alphadog00:
Is any of this lag due to Nand versus SD permanently fixable? Do we think Samsung will address it in an update? It sucks to get a phone with 16GB of memory that is unusable to something io intensive - unless we hack it.
[Captivate Android Development] mykenyc:
Lol i'm mad we been bashing our brains out and he about to fix it in no time good job for notifying him.
[Captivate Android Development] TimF:
This is a question from another thread but it is directly related to this discussion.

Is it possible to re-partition the internal memory?

I am a linux novice but Andy2's and CptTaco's reasoning about the SD card makes sense to me. Internal memory should always be faster than an external SD Card.
[Captivate Android Development] XorZone:
NukaCola, did you tried to compile kernel w/o any modifications and verify turn-on lag/call issues?

It will answer if it is wrong source provided by Samsung/toolchain issue or your modifications cause turn-on lag/call issues.

Thanks.
[Captivate Android Development] NukaCola:
I don't see how the voice quality could be affected by enabling ext4 in .config. The tool chain could be an issue, if anyone can find out what tool chain mimocan uses that would be a big help. I have tried 2 tool chains on 2 different hosts with the same results.

The lag thing is present stock (rfs) as well, it just gets worse if you use ext3 and then worse again with ext4. Perhaps the i9k guys don't notice because their roms are already faster than ours (I'm told, need to test a euro rom next perhaps).
[Captivate Android Development] Unhelpful:
So, the scripts-from-init.rc test is a success, but there are still some problems. I tried to test whether init would block on scripts started in this fashion, and it appears it does not. This is probably why my init hooks test fails - the mount script works, but the init script has carried on without it.

This is probably because the script is started as a service. To fix this problem, I need either another means to launch a shell script, or I will need to split init into stages, with each script started as a service setting a property on completion that will cause the next stage to proceed. If init is calling the system mount binary to perform mounts, I might be able to wrap it to launch scripts when called with appropriate arguments. The cons with the first approach are that I don't know if anything is supposed to happen between the init and boot events - I will need to start the script now run on boot via a property. The cons with the second approach are that it won't work if init performs the mounts itself, and it will make the init script ugly... but if it works it will probably be a better solution than using services. A third solution would be to find some way for init to launch scripts instead of using its own internal handlers.

I am also seeing the wake lag issue on my own kernel with the normal internal data partition, so something other than the external sd or ext4 must be causing that, or at least contributing to it.

Sent from my Samsung Captivate
[Captivate Android Development] NukaCola:
Hmmm not good.
[Captivate Android Development] CaptainTaco:
So just got my hands on a Captivate to poke around the partitioning schemes, mounts, and disks... and can honestly say, I'm rather baffled by the file system structure...

Every single partition in the phone seems to be defined as "RFS"... wtf is RFS?... Are they referring to the ReiserFS? And if so, what would possibly compel them to use a Reiser File System for any part of the phone, never mind just the NAND...

I'm baffled that they would choose an alternative over YAFFS2 for the NAND, period. NAND is pure flash, pure flash requires a file system designed to write on to pure flash, Reiser is based on HD's, and worse incorporates Journaling...
[Captivate Android Development] Unhelpful:
RFS is not ReiserFS, it's a proprietary FAT-based robust filesystem. According to Samsung's FAQ, it integrates with the translation layer of their OneNAND flash and adds copy-on-write and a journal to FAT for integrity on crash/shutdown. It appears to perform better on NAND than on SD - the data partition on internal SD is fairly slow compared to partitions on NAND even the raw device seems to be slightly faster.

Sent from my Samsung Captivate
[Captivate Android Development] mykenyc:
Hm, your name seems to contradict your post because that was very insightful never knew that.
[Captivate Android Development] CaptainTaco:
Well good thing lol, they would have had to been killed if they were using Reiser... Anyway, from your description of RFS it seems to be very similar to YAFFS2, which is essentially just that a NAND structured file system with copy-on-write, less the journaling, far as I know. If that is the case, then I could see why it would perform better on NAND than SDcards as SDcards are not true flash memory, they do benefit more from an ext2 or 3. ext4 is overkill for SDcards IMO, adds overhead and additional writes that aren't really gaining you any benefit.
[Captivate Android Development] Croak:
But...but...it is ONE INTEGER BIGGER than ext3!!111!! Ergo, it is better! :)
[Captivate Android Development] _Opiate_:
LOL I was just thinking about being sarcastic and posting this.

Im curious to see what mimocan comes up with.
[Captivate Android Development] alphadog00:
So do we have a 2GB NAND chip and 14GB internal SD chip? If so, why the 130MB limit for apps in NAND? The OS Is only 300MB, and even allowing for working memory, we should be able to use at least 512MB of NAND for apps.

I only ask because some places the 16GB of ROM is listed as 16GB of NAND. I wonder if the teardown shows a 16GB chip.....
[Captivate Android Development] NukaCola:
Well ext4 is better than ext3 for many commonly used values of better. The question is if it is appropriate in this application.

I think most people in this thread have better 'real life' performance with an ext3 partition than ext4. I cant really notice a difference between the two, except ext4 lags more from sleep.

ext4 spanks ext3 in the benchmark though, and in the original i9k thread on modaco people were going nuts over how fast the ext4 kernel was.

This type of performance is about the hardest to benchmark and deal with because you see a lot of conflicting reports, you throw in a few numbers from somewhat related synthetic benchmarks, couple that with some placebo effect and the fact that most people here test these things for short periods of time and together (or not) with other hacks and 'fixes' and it becomes really hard to figure out what *really* works.

As a sidenote. I've run the stock kernel with sdcard ext3 using the userinit.sh hack and it works fairly well, except I have a ton of issues. The phone seems to reset randomly.. but its not a complete reboot. Screen goes black and unresponsive, for 30 seconds or so (could be more or less) then after i mash buttons for a while it comes back on, looking like it was freshly rebooted. I've also got no gps at all, it doesn't even try to turn on. I restored (TI) backups after a 1click-stock-flash. Guess my backups are bad or something.
[Captivate Android Development] XorZone:
That's the output of my df:

/dev: 166668K total, 0K used, 166668K available (block size 4096)

/mnt/.lfs: Function not implemented

/sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096)

/system: 282432K total, 228420K used, 54012K available (block size 4096)

/data: 1963904K total, 405744K used, 1558160K available (block size 16384)

/dbdata: 130240K total, 10572K used, 119668K available (block size 4096)

/cache: 30784K total, 3952K used, 26832K available (block size 4096)

/efs: 6064K total, 4104K used, 1960K available (block size 1024)

/sdcard: 13660864K total, 4090016K used, 9570848K available (block size 32768)
[Captivate Android Development] XorZone:
true.

Now it's clear that it is not your changes and I see only 2 options: toolchain or source itself.

Looking forward to try mimocan's variant. I hope it is toolchain you using.
[Captivate Android Development] NukaCola:
I hope it is the toolchain too, I'm still not sure what tools he used. I have tried two toolchains though. One that was directly linked in the i9k sources README, and one that was recommended in the same (and in ours).

I don't think my 'changes' changed very much. I very much on purpose kept defconfig, except to enable initramfs (needed) and ext4. Thats all I touched in the kernel so far, I expected that to just 'apply the fix' and let us move into more fun territory with the kernel. However it is possible that it is not the true .config that Samsung used, after all they did turn initramfs off... I haven't even really looked at it.

Also the sources could be off from the stuff used to build their binary by a patch or two. I have been meaning to grep/diff around the captivate and the i9k tree a bit but I just don't have the time to sit down with it properly at the moment.
[Captivate Android Development] Unhelpful:
I looked at it a bit, because I wanted deadline for the default I/O scheduler. Other oddities include enabling AT keyboard support and support for PS/2 ports (but not any devices), and 16 unused 8KiB RAM drives (this last one is done by stock as well).

Sent from my Samsung Captivate
[Captivate Android Development] alphadog00:
df wont show disks. fdisk -l will, but not here. I'm not really sure what embedded commands in linux will. Maybe if reboot and look at dmsg for disk probe and mounts.
[Captivate Android Development] CaptainTaco:
That is basically the point I was attempting to make, ext4 obviously has it's benefits over ext3, however this is for actual machines where data being handled is much greater, dealing with actual hard drives over flash memory, etc. For our purposes, aka partitioning an sdcard for less than 2 gigs intended for running not much more than 200MB worth of data, the benefits of ext4 will likely turn into disadvantages, this I can promise. However, yes it will give you better benchmarks most likely, and this is likely due to the delay write incorporated in ext4, as I stated in a previous post, the essentially built in caching of the ext4 filesystem likely tricks benchmarks into thinking it is much faster than it is. The benchmarks will likely see the write time of the file into RAM where it is then delay written to the SDcard.
[Captivate Android Development] Unhelpful:
This sort of thing is exactly why I was saying that we need a filesystem designed for robustness on self-wear-levelling, self-remapping flash block devices. All of the linux filesystems for flash seem to have their FTL built in, have no way to disable it for use on devices that provide their own FTL-like features, and also work only on mtd devices. And all of the filesystems we have for block devices have design assumptions based on rotating media, and have no safety at all under crash or poweroff or provide it by journalling. I'm leaving BTRFS and its CoW out here because I doubt we really want to use whatever ancient version of it comes with 2.6.19. :/
[Captivate Android Development] NukaCola:
In principle agree, however there are real world differences with this mod as well (lag/stutter does lessen or go away, switching between apps is quicker), very likely (at least partly) due to the caching, but also to other fs features. If the benchmark benefits from the caching I'm sure some apps will as well, at some point. I don't think its all caching either since some apps really seem to launch quicker even from a cold start.

I just installed the euro rom, it behaves better than stock captivate for sure, but I think the ext3 hack on stock captivate is snappier (granted very limited testing so far, no benchmarks) than stock i9k rom on the captivate kernel. And Mimocan and the modaco guys saw real performance improvements after implementing this hack as well.

The /data read/write bottleneck is real, we just need to find a nice clean way to work around it.
[Captivate Android Development] NukaCola:
Yeah, I did think about that but figured back-porting a sane version of btrfs to this kernel would be a huge waste of time and we'd have a froyo kernel by the time that was stable anyway. Not to mention I have a track record of making little oopses in file system/vm interaction that have lost data before ;)
[Captivate Android Development] CaptainTaco:
Will check on the captivate when I have one available again, but on my droid incredible this can be done by running

busybox fdisk -l /dev/block/*
[Captivate Android Development] XorZone:
Aha, ty

~ # busybox fdisk -l /dev/block/stl10

Disk /dev/block/stl10: 134 MB, 134742016 bytes
255 heads, 63 sectors/track, 16 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
~ #
[Captivate Android Development] CaptainTaco:
Yea, I just got a hand of my girlfriends Captivate again... the output is certainly hundreds times more cluttered than my Droid Incredible, samsung seems to have decided to make an individual disk for each section of the OS they wanted to segregate... anyway...

from the output any of the /dev/block/stl## is part of NAND al /dev/block/mmcblk# is SD memory, either external or internal. Not sure what the /dev/block/tfsr# is.

Regardless, you get a good picture of how the phone is being organized.
[Captivate Android Development] Unhelpful:
Batching writes and caching recently read or written data to satisfy future read requests are hardly the exclusive domain of journaling filesystems - in particular I believe linux does the latter at the block-device level and filesystems have very little to do with whether it's happening or not.
[Captivate Android Development] CaptainTaco:
I used that description to define in layman's why it would provide false benchmarks. I can elaborate. It is not simply "caching" but how and when it is caching if you want to get technical.

ext3 and "most" (not necessarily all) other file systems allocate the blocks to be written before the data is ready to be written, they then write the entire file (the blocks on the disk where the file will go) full of 0's to reserve that space, after this the data is prepared to be written (cached), and finally written. In other words, you go through the process of allocated blocks on the disk (SDcard in this case) as well as a write process before the data is actually cached.

Ext4 on the other hand does not "pre-allocate" the disk in the manor above. Ext4 does not write the file with 0's like most other file systems. It also does not allocate the blocks until the data is ready to be written. In other words, ext4 will cache the data

first

(once again tricking your benchmarks into thinking that went waaaay faster, in that it did not go through the block allocation and write process it did with ext3) It will then allocate the blocks to be written, and write the file. In the end you end up with similar results (although the blocks on the disk never need to be written with 0's {I'm not going to explain why here, too long, look it up if you interested}) but in the benchmarks eyes it was written much faster because it considered it done after the caching process.

As for the file system having little to do with how it is written at the block level and caching, no, it is the file system that handles how it is cached, and how it is written and allocated at the block level, that is what the file systems purpose is. Linux is written on top of the file system.
[Captivate Android Development] Unhelpful:
Yes, but as I understand how this works in linux, the filesystem requests reads and writes for blocks - and it does decide which blocks are written, and when - but the actual caching of block contents and the decision as to which blocks with known contents to retain in memory happens in other parts of the kernel, I would think mostly VM is responsible for deciding whether to retain data known to be clean or to evict it and possibly have to re-read it later.

Also as I understand it ext3 doesn't do much in the way of pre- or post-allocation, and pretty much allocates data on disk and writes it as write requests come from userspace. Pre-allocation as done in ext4 can "fool" benchmarks but is also of real benefit in that better decisions can be made about where to write data.
[Captivate Android Development] alphadog00:
Looks like the Galaxy guys were doing the same thing a month ago. I don't think anyone ever did find out if there are different NAND chips or not, but the 130MB limit is due to the size of the partition stuff is moved too.

http://android.modaco.com/content-page/312298/got-the-stalling-problem-rooted-try-this/page/100/

The Korean teardown shows a main chip with (4G NAND + 3G DDR + 1G OneNAND) and another chip called MoviNAND.

http://translate.google.com/transla....com/News_List_View.php?nModeC=4&nSeq=1742568

If we have the same layout and OneNAND is fastest, we should still have more space for apps hiding somewhere.
[Captivate Android Development] CaptainTaco:
Interesting. Yea, unfortunately I am unsure how to get a deeper read on how many chips there actually are, and more importantly which chip each "partition" (which are more like virtual disks it seems) is on. fdisk is one of the most informative tools (when combined with mount and df), I'm unaware of a lower level utility... Also not sure how they split their chips, whether it is part of firmware located on the actual chips (likely) or not, in which case it would be near impossible to get a good read on what partition is located on what chip.
[Captivate Android Development] _Opiate_:
I'm just hoping for a kernel that doesn't disrupt voice calls soon. I'd like to continue testing but I can't deal with the mic problem.