yashsingh1/xdaforums huggingface dataset:
[Asus  Eee Pad Transformer Prime] dagrim1:
Ugh, got it to build for arm-v7a but of course it looks (completely) different from the module with the prime. Structure the same, but compiled different enough. So thought I'd compule the library and install it on the prime but that resulted in the prime not booting...

Hmmm
[Asus  Eee Pad Transformer Prime] xTRICKYxx:
I'd love to help, but I have zero knowledge in anything SQLite...

I wish you good luck!

I think everyone is hoping that the slow random write speeds are purely software related. And it makes sense that they are. The SQLite benchmarking thread is giving a lot of details on what is making our performance so sluggish, but not why.
[Asus  Eee Pad Transformer Prime] dagrim1:
Next step, compile complete CM9 source (hopefully including libsql src :P) so I can use that as a base...

Copying the libsqlite.so and libsqlite-jni.so from an AOKP rom worked fine, so perhaps this is a (not the easiest) way to get things to work.

And ah well, otherwise I learnt how to compile a rom :P
[Asus  Eee Pad Transformer Prime] banderos101:
Same here, thanks for taking time out to tinker with our primes...........hmm, im not intirelly sure if i should refrase that??? :P

GD LUCK
[Asus  Eee Pad Transformer Prime] hairdewx:
Are you actually changing rom code to remove fsync?
[Asus  Eee Pad Transformer Prime] dagrim1:
Ehm, basically... Not rom code itself but one of the libraries in the rom (for which I hope the code is in there as well).

Background info:

http://xdaforums.com/showthread.php?t=1000899

Patch for libsqlite.c:

http://xdaforums.com/showpost.php?p=12695464&postcount=1966

diff -urNp external/sqlite/dist.origin//sqlite3.c external/sqlite/dist//sqlite3.c
--- external/sqlite/dist.origin//sqlite3.c	2011-01-07 15:26:32.000000000 +0800
+++ external/sqlite/dist//sqlite3.c	2011-01-07 15:27:55.000000000 +0800
@@ -34761,7 +34761,7 @@ SQLITE_PRIVATE int sqlite3PagerOpen(
pPager->readOnly = (u8)readOnly;
/* pPager->needSync = 0; */
assert( useJournal || pPager->tempFile );
- pPager->noSync = pPager->tempFile;
+ pPager->noSync = 1;
pPager->fullSync = pPager->noSync ?0:1;
pPager->sync_flags = SQLITE_SYNC_NORMAL;
/* pPager->pFirst = 0; */
@@ -65838,7 +65838,7 @@ static void attachFunc(
sqlite3PagerLockingMode(pPager, db->dfltLockMode);
sqlite3PagerJournalMode(pPager, db->dfltJournalMode);
}
- aNew->safety_level = 3;
+ aNew->safety_level = 1;
aNew->zName = sqlite3DbStrDup(db, zName);
if( rc==SQLITE_OK && aNew->zName==0 ){
rc = SQLITE_NOMEM;
@@ -97124,7 +97124,7 @@ static int openDatabase(
** database it is 'NONE'. This matches the pager layer defaults.
*/
db->aDb[0].zName = "main";
- db->aDb[0].safety_level = 3;
+ db->aDb[0].safety_level = 1;
db->aDb[1].zName = "temp";
db->aDb[1].safety_level = 1;

So the basic idea is to limit IO-interaction by disabling fsync.
[Asus  Eee Pad Transformer Prime] hairdewx:
Interesting, I figured it would have been in one of the kernel modules.
[Asus  Eee Pad Transformer Prime] dagrim1:
Ok, I have been able to compile the CM9 source (for the galaxy nexus, because the tutorial was for that one) including the sqlite source...

First test was now 36,6 secs which is pretty nice, have to check for other issues (and will compile for TF201 if I can soon). The semi patch caused a number of force closes so we'll have to see how this one goes.

But at least happy I was able to compile ;)
[Asus  Eee Pad Transformer Prime] RubenRybnik:
Wow ... thanks for posting this ... I've edited the sqlite source and compiled the binary for AOKP B38 below are my setup details and test results. So far everything is stable, no FCs in the past 45 min and things are much faster. I have no idea whats up with Antutu's SD Card reads, that's outrageously high lol ... Can't say I trust that ;o) . Otherwise great hack. I wouldn't recommend running this if you have a need for solid data stability, I haven't seen anything yet, but ending all those syncs may cause problems down the road, I don't know. Since I flash regularly I don't really mind it, loving the speed! Also found some other sqlite optimization tips( Cache size and some others ) I may try to add in after I run it for a day as is and see how stability is. Again, thanks to the OP, great find!( so far )

Device: Transformer Prime(TF201)
ROM: AOKP B38
Kernel: Motleys running at 1.8 set to performance governer for testing

Tests: ( All tests done Run 1 after fresh boot, Run 2 immediatly after, all set to performance governer )

Couldn't get the damn formating to look right in the post ... check the image for results.
[Asus  Eee Pad Transformer Prime] hairdewx:
What about actual use performance? Is web browsing lag gone? What about switching apps?

I'm a little hesitant about the data stability... with my luck I'll have rock solid stability until the ONE TIME I actually need it and lose important data ;)
[Asus  Eee Pad Transformer Prime] dagrim1:
Updated main post...
[Asus  Eee Pad Transformer Prime] djmcnz:
This should be in dev section?

Looks promising, thanks OP.
[Asus  Eee Pad Transformer Prime] Rytkonen:
Sounds very promising. Rooted people, start testing! :D
[Asus  Eee Pad Transformer Prime] Tempie007:
sweet. going to try it tonight!

---------- Post added at 01:33 PM ---------- Previous post was at 01:13 PM ----------

Regarding your screenshot, the 1k inserts: After the SQL Patch those scores are on par with the other devices where we've seen benchmarks from (less than 1 second, which is literally 100 times better.. )

I don't believe in coincidence ;)

Can it be that on those devices FSync is allready disabled, or bypassed only in some cases?

Anyhow i am realy curious if this makes stuff feel better :)
[Asus  Eee Pad Transformer Prime] RubenRybnik:
Posted this in the TFP's AOKP thread in that 'other' forum:
[Asus  Eee Pad Transformer Prime] dagrim1:
If it is, it is not done in the library itself...

I also tried the libsqlite files from the SGS3 for example and copied them to the prime, but that didn't help much... So if fsync is handled differently on those devices it is done in a different place.

This is just a workaround for (part of) the problem... The real issue is elsewhere. I mean, the CM9/AOKP/AOSP roms use the same source for the differnt devices yet the issues are not the same on all of them.

So somewhere in the kernel code? Or another level, the controller for example. As the HTC One X performs decent it seems that it's not (JUST) the tegra3 but some of the other components.

Ah well, maybe tonight I can try the full wipe and reinstall and see if that helps anything.

Booting time doesn;t improve with this mod, titanium restores improve a bit but nothing extreme (20-25% faster is my guess) so there IS more going on. My guess is the used memory is at least part of the problem.

Some SD benchmarking indicating the memory used isn't the best one :/ (Done with the 'Device Performance' benchmarking app)

Asus TF201:

5x200MB@/mnt/sdcard (=internal)

Max Write:	14,4

Mean Write:	12,1

Min Write:	9,4

Max Read:	28,8

Mean Read:	23,2

Min Read:	14,9

5x200MB@/Removable/MicroSd (=external)

Max Write:	14,7

Mean Write:	11,4

Min Write:	9,9

Max Read:	14,6

Mean Read:	12,7

Min Read:	9,0

SGS3:

5x200MB@/mnt/sdcard (=internal)

Max Write:	26,7

Mean Write:	25,3

Min Write:	23,9

Max Read:	43,8

Mean Read:	42,2

Min Read:	39,4

5x200MB@/mnt/extSdCard (=external)

Max Write:	10,6

Mean Write:	10,0

Min Write:	9,6

Max Read:	13,8

Mean Read:	10,6

Min Read:	6,2
[Asus  Eee Pad Transformer Prime] st33med:
Is it possible to make an intelligent fsync where it basically syncs after some clock time after the last db write or inserts an fsync with a high count of writes?
[Asus  Eee Pad Transformer Prime] jazzy__b:
Your script looks pretty much like what I am doing. I don't recognise logger.ko, should I care about that?

That's exactly it. Kernels I've tried to build:

https://github.com/EnJens/kernel_tf201_stock

https://github.com/AndroidRoot/android_kernel_asus_tf201

https://github.com/motley-git/TF201-Kernel

https://github.com/CyanogenMod/android_kernel_asus_tf201

Toolchains I've tried:

https://github.com/android/platform_prebuilt

http://developer.android.com/tools/sdk/ndk/index.html

http://www.mentor.com/embedded-software/sourcery-tools/sourcery-codebench/overview/

I initially tried building on a Mac but when I ran into trouble I created a clean install Ubuntu 12.04 machine in VirtualBox and continued on there. I was using AOKP on my tablet but yesterday flashed to CM9 to try the kernel source above from the CM9 repo. I still haven't had one successful kernel install yet. My current build command looks like this:

make -j4 ARCH=arm CROSS_COMPILE=~/tf201/android-ndk-r8/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-

and my packaging script looks like this:

#!/bin/bash -x
TOP=~/tf201
ANYKERNEL=$TOP/TFP_AnyKernel
BLD=$TOP/android_kernel_asus_tf201
rm $ANYKERNEL/system/lib/modules/*.ko
cp $BLD/arch/arm/boot/zImage $ANYKERNEL/kernel/
find $BLD -name '*.ko' -exec cp {} $ANYKERNEL/system/lib/modules/ \;
cd $ANYKERNEL
rm $TOP/jazzyb.zip
7z a $TOP/jazzyb.zip
[Asus  Eee Pad Transformer Prime] CekMTL:
No worries about logger, it's android logger and was doing some tests with it.

I'm running ubuntu 10.04 64bit in a vm... now that is weird that everything you tried failed...if you have git or source I can try to compile it for you, but apart from that, right now I don't have a clue what it could be....???

Enviado desde mi Transformer Prime TF201 usando Tapatalk 2
[Asus  Eee Pad Transformer Prime] jazzy__b:
Well I'm compiling clean from those repos. Which toolchain are you using? What location did you get AnyKernel from? I would like to eliminate differences in my environment.

Sent from my Samsung Galaxy S II
[Asus  Eee Pad Transformer Prime] CekMTL:
AnyKernel from the _motley .zip file (any new version of kernel).

Toolchain latest is one that has gcc 4.6.3, don't remember the link right now... it was featured about a week ago in front page of XDA.

EDIT1

It's this one : http://www.xda-developers.com/android/experimental-toolchains-and-guide-for-android-kernel-building/
[Asus  Eee Pad Transformer Prime] jazzy__b:
Well I finally got a kernel to boot but only by switching to the exact toolchain that motley is using (gcc version 4.5.1 (Sourcery G++ Lite 2010.09-51)). Very strange.
[Asus  Eee Pad Transformer Prime] jazzy__b:
I have now begun merging in MMC changes from linux-2.6 source tree. The ones I'm interested in begin from May 23rd 2011, which appears to be similar to the Asus MMC kernel source. The change I'm really interested in is the CMD23 implementation from May 27th but I need to merge in most of the intermediate changes first due to dependencies. I have been noting some rough timings from Androbench here:

https://docs.google.com/spreadsheet/ccc?key=0Aq8XkPiL5fmZdDBDNTR0ZEl2eFlLc3FBM2JyTXMzV3c. These are probably not so accurate as I am running straight after boot-up but I am only interested in the more significant changes so I'm ok with that.

I currently have another 53 patches to work through to get to the end of the CMD23 changes. After that I may look to the nvidia sources for other potential improvements.
[Asus  Eee Pad Transformer Prime] dagrim1:
Awesome Jazzy_b and CekMTL! Your efforts are much appreciated and am still keeping an eye on all this :)
[Asus  Eee Pad Transformer Prime] tshoulihane:
My reading of that is there probably is a small bug in the kernel somewhere, and other tool gains are exposing it. Maybe interesting to work out why now you've got some stability (after the potting). Did you try the latest linaro toolchain?
[Asus  Eee Pad Transformer Prime] Tempie007:
realy nice work jazzy!
[Asus  Eee Pad Transformer Prime] _motley:
Good to hear you finallly got it working! I know you are having fun and I don't want to deter from that, but if you want we can work together on this since I already have all these patches merged and working for the most part (complete backport of Nvidia Android 3.1 for MMC).

Hopefully I can find some time tonight to tidy up and push my changes to github (work has not been cooperating lately). I would really like some other eyes on the code to help investigate my last existing problem (MicroSD doesn't work).

Let me know if you would like to work together. I was already going to ask CekMTL to join in on the fun to help troubleshoot since he is awesome at that kind of stuff. There are some others in this thread that seem knowledgeable about the MMC drivers as well that may be able to help. The more brain power and time we can spend, the better off we will be.
[Asus  Eee Pad Transformer Prime] jazzy__b:
That sounds like a plan.
[Asus  Eee Pad Transformer Prime] CekMTL:
You know you can count on me, so.... LOCK'n'LOAD! :)

Enviado desde mi Transformer Prime TF201 usando Tapatalk 2
[Asus  Eee Pad Transformer Prime] saturn_de:
count me in

Gesendet von meinem Transformer Prime TF201 mit Tapatalk 2
[Asus  Eee Pad Transformer Prime] _motley:
Awesome, looks like we have a quorum:cool: I'll see if I can get everything committed tonight to github into an IO branch and I will send a PM. I am on Eastern time in the US. Most changes I have made are committed to my local git, but I need to sort out my last changes in which I may not commit, or most likely will only partially commit since they seem to have caused wake-locks.

These are cardhu board/platform changes for sdhci. I have a feeling we may need to move forward with these commits at some point, because the custom stuff in the MMC driver source (added by Asus) were likely workarounds for stuff that should have just been defined correctly at the platform/board level in the first place. This and the fact that the tegra3 Nvidia kernel was in its infant state when Asus started working on the prime.
[Asus  Eee Pad Transformer Prime] CekMTL:
Yep... I totally agree on that thought... when I compared both kernels... two things popped in my head...

1) how primitive Asus's kernel is

2) what a mess Asus's kernel is

PS+FYI: Starting from this weekend, I will cut my cable internet to my home... gotta cut on expenses... so will only have low speed inet/quota on my cell phone to browse/reply/anything not downloading.
[Asus  Eee Pad Transformer Prime] jazzy__b:
Which nvidia source did you use? The ones I found didn't seem recent and I don't think covered Linux 3.
[Asus  Eee Pad Transformer Prime] _motley:
Agreed. Nvidia's code was still young as far as devices go, they had to do whatever they could to get the peripherals working on a real device instead of just a test board like Nvidia uses to test with. It gets pretty complicated, but they are supposed to be the professionals. I won't bash them too much since it is tough to get a stable environment while racing it to market. The only thing I will bash them on is their debugging need to be turned off! I have removed a lot of printk spam in the release I will post (cpu, battery messages, touch messages etc.) That stuff should use pr_debug and not printk.

Here is what I used for the MMC donor, i.e. android-tegra-nv-3.1

http://nv-tegra.nvidia.com/gitweb/?...gra-nv-3.1;hb=refs/heads/android-tegra-nv-3.1

It is ahead of the HTC OneX and Asus A700 in which I also referenced.

I also referenced Linaro, L4T from Nvidia's site, and some other repos.

This is the 2.6.39 repo that the prime was based on when Asus got the code. I believe it is a cross between 2.6.36 and 3.x for the most part since it is not a mainline Linux kernel:

http://nv-tegra.nvidia.com/gitweb/?...-2.6.39;hb=refs/heads/android-tegra-nv-2.6.39
[Asus  Eee Pad Transformer Prime] Mistar Muffin:
Hey _motley, just wanted to say that I would gladly run a kernel with significant IO improvements even if it meant sacrificing MicroSD for a while. I use MicroSD so infrequently anyways. It would just be so awesome to use the TFP without it ****ting it's pants anytime something writes to disk!

Don't let the MicroSD issue keep you from releasing an alpha with IO fixes!

Sounds like you've assembled quite the team to do ASUS' job for them. Proud to have you around.

-Muffin
[Asus  Eee Pad Transformer Prime] _motley:
PM's sent - code posted for review and testing. Let me know what you guys think!

Some info that is good to know:

SDCard on Dock = mmc1 = SDHCI controller on sdhci-tegra.2

MicroSD = mmc2 = SDHCI controller on sdhci-tegra.0

eMMC internal memory = mmc0 = SDHCI controller on sdhci-tegra.3

CekMTL - I think you were right, the V constants didn't do anything, I think I read a boolean incorrectly when I thought they were used for mmc0.

I have added some printk debug in some interesting areas of the MMC code.

I would like to pull the remaining Asus logic out where possible, but some is still needed for now, i.e. if(!strcmp(mmc_hostname(host), "mmc0") etc.

Hynix H26M64002BNR e-NAND or Kingston KE4BT5D6A

Hynix

¤ Fully compatible with Multi Media Card (MMC) system spec. version 4.41

¤ Full backward compatibility with legacy Multi Media Card system.

¤ Data bus width: 1bit /4bit /8bit

¤ MMC Interface Clock Frequency : 0 ~52MHz

¤ Power: Interface power : VCCQ (1.70~1.95V or 2.7V ~ 3.6V), VCC (2.7V ~3.6V)

¤ Up to 64GB of formatted data storage

¤ Partition management with enhanced storage.

Kingston

¤ Density 32GB

¤ e•MMC Standard v4.41

¤ Vcc (V) 3.3

¤ Vccq (V) 1.8/3.3

¤ Multi-Block Read Sequential (28 MB/s)

¤ Multi-Block Write Sequential (14 MB/s)

From wikipedia - good info on clock speeds and management etc.

http://en.wikipedia.org/wiki/Secure_Digital
[Asus  Eee Pad Transformer Prime] RubenRybnik:
You're the man Motley! Hell of a job!
[Asus  Eee Pad Transformer Prime] leppie:
Where did you get that info?

Also my 64GB is a Hynix too.
[Asus  Eee Pad Transformer Prime] jazzy__b:
3.0.6 alpha flashed; kernel errors reproduced on microSD insert/removed; unpacking kernel sources for comparison... game on!
[Asus  Eee Pad Transformer Prime] djmcnz:
http://www.hynix.com/mail/newsletter_2011_01/eng/sub02.html
[Asus  Eee Pad Transformer Prime] _motley:
Good news

http://xdaforums.com/showpost.php?p=27987403&postcount=982

Lets move our kernel conversation back to the kernel thread and allow the sqllite convo to continue hear. OP has been very gracious to let us carry on here. Thx!
[Asus  Eee Pad Transformer Prime] ThunderRootedDragon:
Ben running this for a few days now...def noticeable difference and seems very stable...no issues thus far.

Question, how do I remove the patch and return the lib files back to stock? Will flashing the ROM over it and wiping dav and cache do?
[Asus  Eee Pad Transformer Prime] id10terrordfw:
As long as it doesn't include the patch, yes.
[Asus  Eee Pad Transformer Prime] dagrim1:
You can just extract those 2 files from the rom and overwrite them on your file if you want to go back to the original... (I usually rename the originals to .org before overwriting them)
[Asus  Eee Pad Transformer Prime] mox123:
any update on this or has this project been discontinued?? im using a tf300 and not sure what i should do to improve i/o

edit:

just applied the patched lib files and went from 118secs to 30secs total, wow big difference, my antutu score went up to 12708
[Asus  Eee Pad Transformer Prime] Parastie:
But do you notice any actual day-to-day difference?
[Asus  Eee Pad Transformer Prime] Tempie007:
there are some applications where you can actually notice some difference.

however is this worth the risk of screwing up application databases?

it has become clear that the jellybean builds resolve most of the I/O disturbance and that asus is able to improve stuff through its kernel - and they will.

so i would rather wait for that!
[Asus  Eee Pad Transformer Prime] demandarin:
here is the memory used in 8gb version of Nexus 7: Kingston KE44B- 26BN/8GB .source is IfizIt teardown. its a 32nm MLC in the Speed Class of 104MB/s. now Performance wise, this memory Multi-Block Read Sequential is 40MB/s. Its Multi-Block Write Sequential is 20MB/s.

now there hasn't been an official teardown of 16gb version(which I have) but based on Kingston spec sheet and the model 8gb used, the 16gb version of Nexus 7 more than likely is using the Kingston KE4BT4B6A. its specs are exactly the same as 8gb version. same everything other than storage size difference.

P.s. the prime actually uses same brand memory and class as Nexus 7(which has been shown to dramatically have better I/O performance based on mine and other peoples scores in rl sqlite benchmark and other benchmarks). Prime read speeds are really rated at 40MB/s and write speeds at 20MB/s(same as nexus 7). The proof is in the official Kingston spec sheet PDF. Which I posted in this thread http://xdaforums.com/showthread.php?p=29226434#post29226434 its post #147
[Asus  Eee Pad Transformer Prime] whycali:
Not sure how it relates but it means something to me. I have an Lg Thrill 4g. Using the stock rom and running the RL Benchmark my Thrill performs about on par with the stock Prime 137 sec overall. If I install the Promethius rom for Optimus 3d from Samno and install his overclock kernel it will run in 15 seconds total (17 seconds without the overclock).

Neither of these configurations cause the Thrill to lag at app install like the Prime does.
[Asus  Eee Pad Transformer Prime] Tempie007:
they use indeed the same chips only difference is size.

cant wait for jb kernel update!
[Asus  Eee Pad Transformer Prime] Tempie007:
my guesss is that the custom rom has fsync disabled .... that big of a difference
[Asus  Eee Pad Transformer Prime] leppie:
Actually the 64GB version uses Hynix eNAND, some 32GB models also have Hynix. There are no public specs for these chips though.

Also 104 is not MB/s. It is the merely the external interface clock which is in MHz. I believe the Hynix uses that clock speed too. Theoretically, if the MMC device is in 8 bit mode, it could transfer 8 bits per clock, but there are many other overheads and timings one needs to adhere to, so you will never see this in practice.
[Asus  Eee Pad Transformer Prime] mox123:
Been using this for a few days seems stale and chrome loads instantly when tablet is booted up from before it would take 5 secs. I just make sure I shut everything down properly. Also using ezoverclock. I'm keeping fsync off until something better comes along. All my essential data is on Google drive and Gmail. So not that bothered. If I come across any other issue I'll let you know. Everything is so much snapier.

Also rl benchmark completed in 21.545 seconds today just now
[Asus  Eee Pad Transformer Prime] Parastie:
I'm going to try the same on my prime. I've backed up some important files, so we'll see how it goes.
[Asus  Eee Pad Transformer Prime] mox123:
I think I found a bug, it only happens in chrome, I'm trying to download a .rar file and it completes to 100% then says download unsucesful, tried again with opera, firefox, nightly, stock browser and the file is 100% sucesful with no errors, could someone else test this?

this is with fsync disabled of course

edit:

it might be my chrome, cause i downloaded another .zip file and it works!?? weird

also when deleting files in es file explorer, they take about a split second, i deleted 800mb worth of files and bam operation complete in less than a second!! fsync on is slowing down alot of things because before if i deleted anything over 20mb it would take a few seconds to a few minutes!
[Asus  Eee Pad Transformer Prime] dagrim1:
Think I had those kind of corrupted downloads as well at times (with other browsers) on stock rom and non-patched-sqlite roms...

Anyway, now on Jellybean roms and the CM9 version don't work on that one (doesn't boot) so perhaps I have to give this a try again with JB version hehe.
[Asus  Eee Pad Transformer Prime] mox123:
im waiting for full jb, ive never been keen on cm roms,

chrome isn't that great a browser, im using firefox nightly and opera, and sometimes stock browser depend what im doing,