[Sony Ericsson Xperia Mini, Mini Pro, Xperia Pro, A] tobbbie:
I stumbled over a mismatch of reported free space on the ray's yaffs2 partitions (\cache, \system, \data and \data\idd). The related MTD entries are:

dev: size erasesize name
mtd0: 19000000 00040000 "system"
mtd1: 00600000 00040000 "appslog"
mtd2: 06580000 00040000 "cache"
mtd3: 1a400000 00040000 "userdata"

I noticed that applications are reporting e.g. \data having 107MB space free (see first picture - mind that this one ignores \cache directories - I checked to have ~3MB of cache data), while the partition size is 420 MB and used space by all data is ~269MB. This should deliver a free space of ~150MB, but leaves a dead space of about 47MB.

The tool "Diskusage" reports this "dead space" as "System data" (see second picture).

Another tool, "Storage Analyzer" reports everything not accounted for existing files as "free space" (see 3rd + 4th picture - where 4.th reports in menu pop-up different to main page).

I could also not fill anything on \data beyond the reported 103,9MB - so I speculate about where these 47MB are lost?

The same applies for the other yaffs2 partitions, however in largely different amount. It seems that the number of files/directories correlates with the "lost" space - but this is just a guess and I have not scientifically proven that (yet). On the tiny \data\idd partition of 6MB (mtd1) the System Data is 2MB, so roughly 33% of the partition, on \system it is 20MB, so 5% of 400MB and on (the rarely used) \cache partition it is just 2,3MB of 100MB which is 2,3%.

I do care most about the loss of >10% of the \data partition (47MB of 420MB) and how I could get that lost space back for use?

Some research I did is:

yaffs2 is not a "normal" file-system like FAT or others that have table for block(cluster) allocation and directories but a log-file-system that makes sure that low level flash blocks are evenly used. This has implications:

files are not overwritten by a new version of a file. So anything you ever write to such file-system is only over-written if that flash block is the "next" to write to. This has security related impacts where research is done about - a good place to start is here. An article from the same team going more in details but not linked from there is this.

yaffs2 imaging (what you do with CWM recovery) seems to back up the NAND pages and so it does include the wear-leveling of that time when you took the backup. I was hoping that scratching the partitions in CWM and restoring would cure the loss, but it did not - reason above :(

I am not sure (yet) about the situation after a fresh format of the partitions in CWM and how I could only get the files restored (along with their directory and security metadata like user and attributes) but drop the related NAND wear information. I think of making a "normal" tar-backup of the partition and restore that in CWM, just had no time yet to do that. I remember that nameer in the nandroid backup thread had mentioned that it worked for him. However, since the yaffs2 would allocate the same number of files according to the same rules of wear-leveling, I would not expect a different result here. I am not even sure what the "format" command actually does to the file-system and if it would reset the NAND page usage statistics (maybe not).

I would just like to understand where these 47MB are going and if they can ever be re-claimed with standard means? Some wild speculations (not backed up by any research) are:

Is this just the "log" of blocks used and the related wear information? For my feeling this would be much of overhead, but it could be linked to the large size of erase-blocks (256k) in the MTD device!

In FAT we know "cluster size" as the smallest element of space allocation. If that is linked 1:1 to the erase blocks (which I doubt after reading the papers referenced above) then the space could be the total amount of "slack" space at the end of each file.

Papers above (and related tools derived for secure deletion here and here) explain a mechanism to force the yaffs2 file-system to a mode of early reclaiming these erase-blocks and also force to utilize them in a way to have more than one file share an erase-block.

I am reluctant to believe that free space on a partitions file-system is variable in that regard. So it would be hard for me to accept that available space "grows" under tight free space conditions when files are copied around.

I noticed when filling up \data for testing that copying the last few MB to the \data partition took quite long so possibly there is a mechanism as the papers claim, but it would not free anything of my lost 47MB as I could not eat into that space by copying files to \data.

I tend to believe that the slow write of the last few MB to the file-system was just the reuse of the previous just single-file used erase-blocks and utilize them for also for the new written files - not touching the 47MB of "system data".

Finally I tend to believe that the reported "System data" (or lost space) is attributed to file-slack space and so it is not possible to reclaim that in any way.

Can you share anything with me in that regard? I am interested to answer the following questions:

Do you see similar effects on your file-systems, and to which amount?

Are there any tools (also command-line) telling details about yaffs2 space allocation?

Which is the smallest space that a file eats from the storage space in the yaffs2 file-system and how can I find out?

Does a symlink to sd-ext (I am a heavy user of link2sd) also eat that much of space as a file does or not (I guess the latter) - so does it make sense to also move/link small apps with link2sd?

How can I find out which of the "erase blocks" are actually "bad" - so worn out already?

Is any such bad block marked completely "unusable" or just partly?

Could it be that the "lost space" is attributed to such "bad blocks"?

Thanks for reading so far :)