[MDA III, XDA III, PDA2k, 9090 Software Upgrading] Tuatara:
Performance ... what matters? The 'NOW', or the 'ALWAYS'?

It's a trade-off really. The total time required for the compaction is still going to be the same. For example, using up blocks of the flash memory is going to 'fragment' the space. When you need that space, the issue is: 'is it ALREADY available'?

By running the thread more often (to clear unused blocks back to being available), your device 'continuous over time' performance increases, but by doing the compaction 'all at once' your device's 'instantaenous performance' is higher.

Rant on.

Benchmarks love the latter ... users (should) love the former.

This is why in the new TuMa v1.4 ROM I'm working on, I've gone AGAINST all of the other 'benchmarks' and dumb short-term tweaks and so-called solutions, and looked at device performance overall, and over a long period of time. Performance is now 'managed' rather than 'peaked', so that things are kept at a good rate of continuous usage, rather than trying to maximize an instantaneous time. Benchmarks really should measure the device in two weeks of use ... not right after it's configured.

What good is a fast device right now, if in one weeks it's slow as mud all the time? Yes, the TuMa v1.3 is slowing down - but the degree of slowdown of that ROM isn't to the point of 'unusable' as is the case with some other WM5 versions (for very well known reasons now). Knowledge is always gained, and improvements are always made with enough time and patience.

This is the other point I raised for people trying out the "new ROMs" ... (which in truth haven't addressed ANY performance concerns when you look at the changes done). These "latest and greatest", "just released", "new and improved", and "better and faster than ever" - ROM release adopter junkies are just experiencing "new device syndrome". Of course it'll be faster - DUH - you've just given it a clean slate to start from! Your multi-week old device has just been cleaned of it's junk, and it WILL be so much faster with the new ROM. REGARDLESS. Shyeah ... think about it.

Sorry ... rant off.

The time it takes is effectively a constant. It must erase the blocks of Flash, and organize the filesystem based on this. Erasing is slow ... which is why when it 'piles up' and there's a lot of work to do, the compaction thread really can't finish and return control to the device, until it's done the job. It's akin to stopping a defragment of your hard drive ... sure, some of it is clean & organized, but the remainder is still a disaster. Update that drive in the meantime, and it needs to start all over again.

A lot of questions ...

1). 20-25 minutes is FAR too slow. There is definately something else taking up 'residence' or CPU time during this process. It sounds like you have an application installed, or a device driver, or you've twiddled priorities improperly, incorrectly using temporary storage, resulting in over-use of your flash. A hard-reset and careful re-installation of your applications MAY be warranted to fix this. *more below*

2). Block size is quite probably 256Kb. I think your device also uses the M-Systems DOC 64MB chips.

The DOC G3 64MB (512Mb) consists of two 32MB (256Mb) flash planes that consist of 1024 blocks each, organized in 64 pages, as follows:

Page (Storage Unit) – Each page contains 512 bytes of user data and a 16-byte extra area that is used to store flash management and EDC/ECC signature data. This is the SIZE of block which changes EACH TIME something gets saved to memory on your device.

Block (Erase Unit) – Each block contains 64 pages (total of 256Kb). A block is the minimal unit that can be erased, and is sometimes referred to as an erase block. This is the SIZE of block which must BE ERASED each time we need some more space on the device.

Note: Since the device works with multiple planes, the operational block size is actually 512Kb. This means each time 64KB of space needs to be erased, when a block can bee freed.

3). And now the hard question ...

The compaction thread is effectively attempting to move the random fragmented blocks of 512 bytes of data in each of the Storage Units around into larger blocks of 32KB (256Kb) Erase Units. Otherwise, you'd have parts of your data erased each time you needed more free space in the Flash to use. 8)

Essentially this comes down to a "defragmentation" algorithm, however the problem is that the device is still actively being used at the same time. To solve this obvious problem - the device NEEDS to be paused, the compaction run to completion, blocks are freed up for usage, and then the job can continue. It's quite obvious when you think about it.

For example ... 64 Erase Units, each has 1 Storage Unit usefully allocated in it. The compaction thread now can in theory take each of those 64 Storage Units and put them into one larger 'Erase Unit' Block space. To do this is needs to read ALL 64 units (32KB) into memory, maintain all the blocks UNFREED (since if you reset your device now you'd lose your data). Erase a new UNUSED Erase Unit, write all 64 Block Units into the new Erase unit. Then it can go through and mark in the FileSystem, where the NEW location of these Storage Units is. Once that's done, it can mark those other 64 Storage Units as being available for Erasure (or for reuse of any remaining Storage Units within it). If it detects that they are full, then those 64 Erase Units can be erased, and it goes ahead and does that. Natually all this 'shuffling' of data, with guarantees that you don't lose any in the process, needs to be done CAREFULLY and PROPERLY.

*RANT* ... For (insert deity of choice here) sake ... NEVER EVER HIT THE RESET PIN WHILE YOUR DEVICE IS BUSY COMPACTING!!! You can (and probably will!) lose your data ... have patience ... just wait ... it really is busy doing something VERY important. But otherwise, go ahead ... RESET away ... you'll probably be cursing that you're continually doing HARD-RESETs all the time, and that it's a sh*tty device you've bought. Well ... after all ... you NOW KNOW ... it really IS your OWN fault. :roll:

The issue comes when there is something which CAN'T be paused (a device driver, OS core, etc.). This will still need flash storage space while the compaction thread runs. This is possibly why your compaction takes 20 minutes or more. Remove the 'extra stuff' which could be doing bad things.

Also, there are those (I shouldn't use the word, but anyways) idiotic people who think that by forcing this to the "now" that they're making their devices better. For example - inapprorpiate use of Priority256 registry keys, in raising the priority of a task (to above real-time even!) in order to solve their immediate issue - they believe is a good thing!!??!!

This results in the compaction thread needing to 'back off', then the OS runs the 'real time task' which corrupts the defragmentation process, so when the compaction thread has an 'idle' chance again, it starts up ... starts cleaning ... tries ... gets interrupted ... (cycle repeats).

I'm trying to 'do away' entirely with this issue in TuMa v1.4 by forcing the compaction thread to run more often, at times when it can do so, just below application level requirements, but above idle. So barring any 'fixes' introduced by l-users ... it should maintain an even performance level over time - barring obvious issues, like when the Flash is too full for compaction to run!

In this way, through usage testing by knowledgable users, I have validated 'theory' in Alpha versions, and will now/shortly be moving into 'practice' in Beta testing with a lot more 'general' users becoming my 'unwitting' performance testers in releasing the TuMa v1.4 ROM. I believe theory will be put into practice, and as experienced by the alpha testers, performance should be continuous and stable. 8).

CacheSize is deprecated, and in TuMa v1.4 ALL keys are explicitly specifed, while in TuMa v1.3 FatCacheSize was not. The reason is two-fold. For the BA ROM (since it's unofficial and a patchwork of other ROMs), these ROMs were still using older drivers (WM2003SE) which did not support the new keys.

The explicit 'new' keys override the deprecated key. However, the old key is still kept in there for legacy reasons, or for applications or users who might 'insist' on twiddling them in the future. I'm now resorting to this, and trying to 'protect' people from their own performance tweaking "foibles".

If people are willing to put up with a slightly slower 'overall' device, but have continuous same-level performance ... then I can give them an answer (hopefully) right now, or soon if I ever get this TuMa BA ROM to a point I'm happy with it (probably AKU2).

If they want to be blind adrenaline junkies and need 'speed at any cost' ... just keep doing what they're doing, tweak caches, put off until later all the housekeeping, and then, when it hits the wall, go ahead and try to do it all in 20 minutes of device lockup when it's become all too much to handle.

If they want 'rocket-ship' speeds at all times, they'd better carry around a non-flash based device or a laptop. But they'll look REALLY dumb holding up one of those things to their ear when making a phone call. 8)

[flame retardant suit is on]

Thanks for such a great set of questions ... let's me get the point out as well.

Now ... I wonder who else will actually read this post? Maybe it'll be Wiki'd? Maybe ... maybe it's better if not.