yashsingh1/xdaforums huggingface dataset:
[Asus  Eee Pad Transformer Prime] Parastie:
It's not that the files are deleting so quickly, it's more like the OS is not longer waiting and checking to see if it was done or not. fsync forced everything to stop while it checked to make sure the request was done. Now, the OS is just sending the command and going on.

I'm also running this patch, and so far haven't had any issues. I think I've had it for 3 days now? The only issue I had was copying to a USB drive, it said it was done before it was really done. When I pulled the drive out, the last file was partially corrupted. Now I know to use the safe eject and wait patiently.
[Asus  Eee Pad Transformer Prime] Parastie:
Has anyone tried this on the new JB roms? I think it won't work because it is not compiled against the kernals. Anyone know how yo compile with the newer kernel? I'd like to run this again.
[Asus  Eee Pad Transformer Prime] dagrim1:
I'll give it a shot next weekend... haven't been able to look into it yet. Good reminder though ;)
[Asus  Eee Pad Transformer Prime] nizhnyk:
Just another good reminder just in case)) anyone tried it?
[Asus  Eee Pad Transformer Prime] dagrim1:
lol, damnit... Actually wanted to try it out but my keyboard and mouse didn't work in the unix which made it a bit hard (and couldn't bother to get another one atm). New chances this weekend (or maaaaaaaybe even tonight).
[Asus  Eee Pad Transformer Prime] Parastie:
Don't forget! :)
[Asus  Eee Pad Transformer Prime] Parastie:
Any updates? :)
[Asus  Eee Pad Transformer Prime] whycali:
Not sure what you can be looking for here Parastie. if you are using a Jellybean rom with clemsyns kernel and fsnyc disabled you are seeing this already included.
[Asus  Eee Pad Transformer Prime] Parastie:
Fysnc is already disabled? Was that an option that I forgot to check during the 1.51 Wookie install? :)

EDIT:

I'll be damned! You're right! I just ran the SQLite Benchark and got Fsync disabled results. I wonder what's causing the pausing while copying files to external media then...Maybe that'll clear up with SIO? I have to investigate more when I have more time.
[Asus  Eee Pad Transformer Prime] whycali:
Not sure why you have this. as a test I copied the wookie zip file from my prime to a usb stick in the dock to test it and it froze at 218 mb out of 475. Then I tried it to a usb stick that was not full :o and it worked fine and smooth.
[Asus  Eee Pad Transformer Prime] poofyhairguy:
Just wanted to reply and say that the libsqlite.so and libsqlite_jni.so files in the first post don't work with Jellybean like was assumed, so people like me who just got root will have to find another way.
[Asus  Eee Pad Transformer Prime] poofyhairguy:
I found another way:

http://xdaforums.com/showthread.php?t=2230779
[Asus  Eee Pad Transformer Prime] banderos101:
Dont know if this is relavant, but there was made mention on a review on andetech, where he went out of his way to mention that the tegra 3, "did'nt" have a "dual" channel memory interface. Im gonnna assume that tegra 3 uses a "single" or something

Could that cause any of the issues your noticing?
[Asus  Eee Pad Transformer Prime] Tempie007:
The Tegra3 does indeed have a single channel memory interface, but this is for the RAM, shouldnt interfere much with the internal memory speed (besides; other tegra3 devices dont have these glitches)
[Asus  Eee Pad Transformer Prime] banderos101:
Cheers, makes sense :D
[Asus  Eee Pad Transformer Prime] Moist:
So i did a full wipe and then flashed b38, motley and then installed this patch. Worked great! So i then installed voltage control, and changed the scheduler back to noop, saved to the init.d thing, rebooted, and ran a few benchmarks on antutu. IO speed is exactly the same for me now on noop as it was on cfq. The only thing that hasn't changed at all through all this IO business is the write speed. It's stayed at around about 5-7mb/s before and after cfq/noop changes. To me, this /must/ be quite a big problem. To my logical mind, this is(along with the previously terrible read speed) one of the big factors slowing the Prime down. Titanium backup restores take ages!

Just thought i'd give my findings. Thanks for the good work. This looks promising!
[Asus  Eee Pad Transformer Prime] Tempie007:
just installed , sqllite performance is sick.

combined with wookie 1.1, motley 3.0.0 alpha with zram it seems all good n smooth... now lets hope its stable with less lockups!
[Asus  Eee Pad Transformer Prime] tf201:
Can I run this over stock rooted or should I unlock and have cwm in place?
[Asus  Eee Pad Transformer Prime] Tempie007:
Ok i now actually seen some real world improvement of this patch;

In google reader when marking all messages of a feed as read, it would take like 5-10 seconds to actually mark them... (unlike on my sgs3 and desire hd where its almost instant)

On the prime it is now also instant, very nice!

Overall Google Reader is way faster...
[Asus  Eee Pad Transformer Prime] banderos101:
Thats welcoming news, thanks for for the ongoing updates Tempie, these types of tests is one of the reasons why i like xda :)

---------- Post added at 10:08 PM ---------- Previous post was at 10:04 PM ----------

Currently, you need to unlock your prime (invalidates your warranty) in order to install cwm, which in turn allows you to install your rom of choice, via cwm

Edit : Tempie has actually confirmed that you can tf201, the above info is in regards to installing roms, please ignore
[Asus  Eee Pad Transformer Prime] Tempie007:
Well its possible to install while only rooted..

However i wouldnt do that without a proper backup (CWM required)
[Asus  Eee Pad Transformer Prime] banderos101:
Ahh yes, again to my rescue tempie :D, sorry, it is indeed as tempie has said

My bad, i thought i was in the adrowook thread, assuming you were asking on how to install he's rom, meu colpa
[Asus  Eee Pad Transformer Prime] benefit14snake:
after I replaced the files....awesome work!
[Asus  Eee Pad Transformer Prime] banderos101:
Thanks benefit14snake

Thought i'd post my results for a comparison

WITH Patched libsqlite.so

Virtuous Prime Stock Kernel, WITHOUT Patched libsqlite.so

Very encouraging
[Asus  Eee Pad Transformer Prime] djmcnz:
Yes, anecdotally so far it's also given me a perceived speed boost and fewer apparent lock ups.

Will report any further interesting observations.
[Asus  Eee Pad Transformer Prime] boynamedstacy:
This looks pretty awesome dagrim1! I am thinking about giving it a whirl over AOKP 38 and the Motley kernel. Been hearing great things!

Sent from my Transformer Prime TF201 using Xparent ICS Tapatalk 2
[Asus  Eee Pad Transformer Prime] rosebud:
Well.. seems to work.. got my vote :)
[Asus  Eee Pad Transformer Prime] tf201:
Being bold as dr pepper, I just did this on top of stock .21 rooted no safety net. Everything works well so far. I'll update in the future. I did however delete my originals as I did with my buildprop. Would someone mind posting originals for the "bold" lol.

I'm not sure we are going to get a software fix for Max reads and writes but in my experience the ability to move smaller amounts of data quickly not necessarily fast is more important to android os.
[Asus  Eee Pad Transformer Prime] dagrim1:
Cool to hear that people are trying it and even noticing differences (real life differences) :) Love to hear more experiences/results...

Lol, way to go... Wouldn't dare trying this without a safety net (like CWM).

About the originals, you can simply extract them from any rom in the development thread... But here are the ones from the androwook rom :)
[Asus  Eee Pad Transformer Prime] banderos101:
The cajones on this one:D

Fortune favors the bold and all that....what, what...mmmm, yes, jolly good
[Asus  Eee Pad Transformer Prime] Hobbesian:
Right, time for me to piss on people's cheerios. I'm sorry to do this too, but I'd best say it now before someone commits to this without fully understanding the consequences.

Fsync off means that SQLite is unbound by the need to check that data has been written to disk successfully, it also means the operating system can freely re-order the writes before they are comitted to flash memory, which in the event of a hardlock will very likely result in database corruption, of the unrecoverable sort.

Fsync in itself is often used -deliberately- to incur an I/O buffer, so whilst it might seem like a great idea to turn it off, what you're essentially doing is risking the CPU going faster than the device controller can write to memory. This is bad. I cannot begin to imagine how many klaxons would go off if you're working on something remotely important and use this patch.

The optimal solution as someone sortof stumbled on is to have fsync write every so often, usually checkpointing every 300 sec is a fairly safe idea, that way in the chance of a crash, the worst case is it rolls back to the last checkpoint it created and discards any pending commits. Without any form of checkpointing, you're really flying by the seat of your pants.
[Asus  Eee Pad Transformer Prime] dagrim1:
Thanks, I am myself aware of the risks and am atm willing to take them but have added your remark/explanation to the opening post.

I understand the optimal solution but that's way too advanced for me atm...
[Asus  Eee Pad Transformer Prime] banderos101:
"Right, time for me to piss on people's cheerios. I'm sorry to do this too"

Dont worry about that:), its info that needs sharing, and you seem to know what your talking about, so if its a possibility, the patch would need some work done to it to avoid those possibilities you mentioned, or run the risk in its current version

"The optimal solution as someone sortof stumbled on is to have fsync write every so often, usually checkpointing every 300 sec is a fairly safe idea, that way in the chance of a crash, the worst case is it rolls back to the last checkpoint it created and discards any pending commits. Without any form of checkpointing, you're really flying by the seat of your pants"

Is this even a possiblity?
[Asus  Eee Pad Transformer Prime] Peter Galbavy:
- If you look at the history of sync(2) and fsync(2) in the unix world you will see that fsync itself is an optimisation and was introduced to allow processes to selectively ensure that specific file write were "on disk" before continuing with work. The older sync(2) system call was a general instruction to the OS to flush all write buffers to disk and the traditional (15 - 20 years ago) way to use the command line wrapper was to run it twice as the first sync(2) returned immediately and the second then stopped waiting for the first to finish. Hence when your second command returned you knew your data - the whole system - had been written out.

This changed a bit later on and I have not kept up with Linux (as a unix a like) at all myself. That said, in standards-land if you fail to use fsync(2) then all that happens is that writes ("dirty buffers") will be deferred until some indeterminate time in the future. When you close a file this is a strong hint to the OS to mark dirty buffers as ready to be written. There is also normally a flush daemon that sits in kernel space and round-robins the dirty buffers sync-ing them out.

If you end up with too much stuff to write out - where your disk is much slower than your CPU/processes - all that happens is that a write(2) system call will block until there is more buffer space available. Which ironically is like what we are seeing on stock TF201 even though it is probably the fsync(2) call that is blocking :)

For those with an interest in these matters at an academic level there are a ton of books out there and the historic stuff like Comer and Bach are still relevant. I can dig up links if anyone wants. Or just google for "UNIX virtual memory management" and similar.
[Asus  Eee Pad Transformer Prime] banderos101:
Dont know if you wanna go lone ranger on this one dagrim, if not, might help having a few xda eyeballs keeping an eye out for devs who might want to help.

Assuming you want the help

What kind of dev background do you reckon, would be best suited for this "optimal solution"

Thanks for your work, regardless, for your time and effort, when you didnt have too, heres hoping to many breakthroughs:D

---------- Post added at 01:15 AM ---------- Previous post was at 12:58 AM ----------

That right there, that.... right there, was direct, informational and to its purpose, and i probably only understood about 10% of it:D

Nice to know, our little corner on xda is attracting far greater minds then mine, which im sure you can imagine, is .......quite a feat:D
[Asus  Eee Pad Transformer Prime] Peter Galbavy:
Thanks, and while I used to have experience in this field - UNIX low level stuff - it's been many years. I am happy to answer questions and follow-up on things with people but simply understand that my knowledge was current about 10+ years ago and I have avoided Linux wherever possible as I find it far to untidy. The Cathedral and The Bazaar indeed.

Good jumping deity! I have just looked online for those books I was mentioning. The prices for the latest editions are crazy, but for reference here are some Amazon links (.com even though I am .co.uk):

Andrew Tanenbaum's classic Operating Systems that toured his MINIX sources and spawned Linux when Linus got fed up of Tanenbaum's refusal to free the sources; This is the 2nd edition, I have the student edition of the 1st - http://www.amazon.com/exec/obidos/ASIN/0136386776 - there is also a stupidly priced 3rd edition linked from the page above.

Douglas Comer's Xinu books (nothing to do with cult religions): Again, original here but follow the links for newer more expensive editions: http://www.amazon.com/exec/obidos/ASIN/0136375391 - There is a "part 2" which is the networking part which is also a good read.

And a classic from Bach (not that one!) that looks at AT&T System V UNIX: http://www.amazon.com/exec/obidos/ASIN/0132017997

And for "fun" one last classic from Leffler et al http://www.amazon.com/exec/obidos/ASIN/0201061961 - Note that Sam Leffler can also be seen in the credits of many of the early Pixar movies... way down as "systems administrator". Yeah, right. Like calling the Pope the local priest. :P

PS Noting the publication dates of most of those books will give you an idea of when I was doing this stuff. The world has moved on, thankfully, but the principals are still very much the same.

PPS I haven't even mentioned the master of all UNIX and TCP/IP books, but the sadly lamented W. Richard Steven's wrote some of the most readable bibles there are. Including the "UNIX book" featured in Wayne's World :D
[Asus  Eee Pad Transformer Prime] st33med:
Huh, that is a good explanation. I understood that there was little risk in disusing fsync until the database is closed unless you somehow corrupted a database on crash or sudden poweroff. I would still want some sort of intelligent fsycning like after some amount of clock time of no writes to the DB it should flush it or place an fsync after a large amount of operations ("Large" is subjective, depending on the amount of data written or the amount of writes made). Or does Round Robin handle that?

Speaking of Round Robin, there was a semi fix for the huge lag in AOKP that made the default IO scheduler now CFQ. As I understand it, CFQ is meant for low io and mechanical drives, so using a predicitive scheduler would maintain fair access to all processes... would using CFQ somehow defer the long and labor intensive fsync until later? I'm wondering how it managed to at least make the interface faster.

EDIT: sorry if I'm not making much sense, I'm still starting to understand all this.
[Asus  Eee Pad Transformer Prime] Peter Galbavy:
In terms of "when" data is written without explicit fsync, I found this page a good jumping off point for "grown up" Linux - I am going to assume Android is much the same: http://www.westnet.com/~gsmith/content/linux-pdflush.htm
[Asus  Eee Pad Transformer Prime] Rytkonen:
Yes the fsync every now and then is needed, but the most important thing about this patch is that it proves that there are serious problems with the Primes firmware. Is Asus even aware of this?

And I thought the GPS were the major problem with this device. Sigh...
[Asus  Eee Pad Transformer Prime] Tempie007:
some nice information from Peter Galdavy :o

ok, im aware of the risks and i dont care too much about my data at this point, untill this crusade in search of a speeedy device is finished i will be flashing everything that might help find a sollution.

What about the FSync scheduling, is this possible to achieve?

i am 99.9% sure that other device manufacturers done stuff to their firmware somewhat the same as this sql patch : at least handling fsync differently..

a speed boost of 100x and with that boost getting the same benchmark results as other devices... ?? not a coincidence...
[Asus  Eee Pad Transformer Prime] Tempie007:
this should be in the dev section btw :o

---------- Post added at 10:19 AM ---------- Previous post was at 10:10 AM ----------

Some interesting information under the title 'Tuning Recommendations for write-heavy operations' , especially the first 2 paragraphs...

- The first part 'freezing' is a well known issue on the prime, i cannot say if this is related to this in any way.

- The second part however is exactly what i've seen happening in Google Reader before this SQLLite patch.

Going to try proc/meminfo tonight to see if this is actually true.

''

(1)

The usual issue that people who are writing heavily encouter is that Linux buffers too much information at once, in its attempt to improve efficiency. This is particularly troublesome for operations that require synchronizing the filesystem using system calls like fsync. If there is a lot of data in the buffer cace when this call is made, the system can freeze for quite some time to process the sync.

(2)

Another common issue is that because so much must be written before any phyiscal writes start, the I/O appears more bursty than would seem optimal. You'll have long periods where no physical writes happen at all, as the large page cache is filled, followed by writes at the highest speed the device can achieve once one of the pdflush triggers is tripped.

''

Here some more nice discussion about FSync in Android.

http://lwn.net/Articles/421117/
[Asus  Eee Pad Transformer Prime] Peter Galbavy:
OK, now you got me thinking about "what's different". Well, we have ext4 which is a nice, mature journalling file system that shouldn't care much about final writes to disk as long as the changes are written to the journal for later processing.

What I've done is gone in (not rooted) with the terminal emulator and looked at how the ext4 file systems are mounted, specificall /system, /data and /cache.

(Just run "mount" and find the lines with the ext4 in them - I would use "grep ext4" but that's not available in the bog standard path I see)

One my GNote they all use flags like this:

/dev/... /cache ext4 rw,nosuid,nodev,noatime,barrier=1,data=ordered 0 0

while on the TF201 I see some changes:

/dev/... /cache ext4 rw,nosuid,nodev,noatime,user_attr,acl,barrier=1,journal_async_commit,nodelalloc,data=writeback 0 0

The bit in italics can be ignored IMHO, the bolded bits could mean something. I am going to go off and see. Not sure where mount options are set on Android, so that's also to be researched. This stuff may not mean *anything*.

Edit: Quick bit of google gets us this page: http://git.kernel.org/?p=linux/kern...02ac5fa36d7f4c07856fe9cf89391e08986f7;hb=HEAD

Looks like the last one could be the most, erm, impactful - even ignoring the poor english.

However, in all these cases remember that flash memory has very different performance to HDD, even those with a large RAM cache.
[Asus  Eee Pad Transformer Prime] dagrim1:
Interesting reads indeed, and atm a bit above my head (not that into linux and/or c(++) programming)...

But definately going to dive into it some more if I have some spare time somewhere (crap, it's way too fun to mess around with this).

But it still is weird that the issue is only/mostly visible with the prime and not with other devices. So it's not so much the linux way of working but specifically combined with the prime.

Also, the problem itself is not withing sqlite... The SGS3 does NOT have the issue but using the sqlite.so from the SGS3 on the prime didn't fix anything. So surely possible the actual fix lies elsewhere.

And this indeed does not fix everything, there are still moments when things slow down. But it does seem this patch really does help in at least some parts.

@mounting settings:

For the Galaxy S3 it's:

/dev/... /cache ext4 rw,nosuid,nodev,noatime,barrier=1,journal_async_commit,data=ordered 0 0
[Asus  Eee Pad Transformer Prime] jazzy__b:
I agree totally. It is worth clarifying the risk that people may be taking here. This is not just about data loss. When you are running apps on your prime there is a stream of updates being written to storage. Many of these updates are sqlite updates, as the design of Android is for apps to store much of their data in sqlite databases. If you had a hardlock or reboot that loses you data then you could argue that having, for example, 5 minutes of unwritten updates has not cost you anything in the real world as the hardlock or reboot could easily have happened 5 minutes before and you would be in exactly the same position. What you really don't want is to have a half-written database update as a corrupt sqlite file could potentially make *all* of the data in that sqlite file unreadable.

How this works: the fsyncs in sqlite will be there to make sure that the file spends as much time as possible in a consistent state, i.e. at the end of a completed transaction. I don't know the design of sqlite specifically but there may be other metadata written out which will allow it to detect an incomplete transaction and roll back when the file is reopened after a crash. This would be a performance vs. safety decision that has been made at some point in the past. By removing the fsyncs the sqlite updates will go into an operating system buffer that will eventually get written out to real storage. The problem here is that that buffer has no information about the existence or importance of the transactions in sqlite; the only clue it had previously was the fsync. The buffer will simply be written out when it is full or after some operating system timer kicks in, meaning that you probably have a partial transaction written to storage and the remainder will overflow into the next buffer. This means that without fsyncs you are potentially forcing your tablet to spend most of its time with sqlite databases in an inconsistent state on storage. Good luck with that if you have a hardlock, reboot or perhaps even an app crash.

Ideally the layer below fsync would be taking care of write-behind caching but in a way that will always bring it up to the last fsync. This is what I think is probably missing or not working on our Primes.

Can someone with more sqlite knowledge than me comment on whether there is any transactional metadata written out to allow rollback if there is a partial write to storage? If this is the case we could all be a lot more comfortable with this patch.
[Asus  Eee Pad Transformer Prime] Peter Galbavy:
Anyone with a rooted Prime and no important data fancy changing their mount flags to match what we're seeing in Samsung device - specifically changing data=writeback to data=ordered and removing nodelalloc and see if it makes *any* difference with a stock libsqlite.so ?

Where in the filesystem this is is an exercise left for the reader - since it isn't /etc/?fstab anymore ;)

Found this thread from 2010 with some details on tune2fs: http://xdaforums.com/showthread.php?t=819580

---------- Post added at 10:41 AM ---------- Previous post was at 10:31 AM ----------

Last post for now, as I am going home (working in Manila at the moment, it's 17:40 here):

http://lwn.net/Articles/326342/

"One can completely disable delayed allocation on ext4 (nodelalloc option), which should then avoid the above at considerable performance penalty."

Sound familiar?
[Asus  Eee Pad Transformer Prime] dagrim1:
Cool, gonna give this a try later on!
[Asus  Eee Pad Transformer Prime] jazzy__b:
/etc/vold.fstab ?