[Motorola Droid 4] Septfox:
Yea, I figured out some things for you to go "well, duh" at.

I think my problem here is that it's essentially acting as a universal minfree setting when set as high as I'm doing. It allows fast memory allocation because it keeps physical memory open and "ready", absorbing launched apps with ease. Bigger values utilize swap more because it slowly recovers the margin between actual memory use and the kswapd mark in the background, stowing things into swap instead of outright dropping them like the LMK would. As I understand from the documentation this is pretty much exactly what it's supposed to be doing.

This worked really great as long as I was using the phone, but at a guess things like systemui were marked as inactive and swapped out while the phone was sleeping. This because it tried to keep up to the minfree limits of memory free, and in addition the extra_free_bytes setting. In my case, this was around 150-180mb by minfree limits plus 256mb extra_free_bytes. So tried to keep 406mb of memory free when it could by swapping and swapping and swapping in the background indiscriminately until it ended up eating important things - systemUI and Phone for example - causing slow resumes after the phone sat long enough for kswapd to quietly do its thing.

I thought I might get around this by simply dropping minfree limits to minimums - leaving extra_free_bytes to do the work in its place - but the huge glaring flaw in this I inconveniently forgot is that things can't be cleared directly from zram when it inevitably fills up (iirc); they have to be decompressed into real memory first. As soon as swap gets full and main memory is exhausted, the phone deadlocks because...well, swap is full and there's no main memory left to clear it with. One would assume that keeping a little bit of additional memory free for such a situation would be an excellent use for min_free_kbytes, but unfortunately the whole direct-reclaim-freezes-everything "feature" makes it godawful slow in normal use when set to useful amounts.

So in the end, it seems the best for my usage is about 128mb extra_free_bytes, maybe 160mb. Keeps enough free via swapping to load up large apps in a reasonable time and without ANRs from Play Services and most of the time Store, but preserves resume speed by not being so overly aggressive about it.

Whew. I feel dumb for not figuring all that out sooner.

Probably. Though my money would be on "other magic" for the above example, since zram seems to sort itself out pretty quickly if given the CPU time and working space.

Speaking of I/O, it seems that buffered random reads off both internal storage and external SD are pretty fast; to the tune of 110mb/s and 160mb/s with a 1mb buffer. Assuming that Link2SD's extra partition is accessed with buffering - probably not, but lets pretend it is - it's not a bad idea at all to get a decent SD card and symlink everything to a dedicated partition. Of course, that wouldn't help with the system apps, but nobody likes them anyway.

Plopping the "slot" images onto a high-performance SD card sounds like fun. I wonder if there's anything actually stopping us from doing it; messing with the mountpoints should be fairly simple, right?