[OnePlus 3 & 3T Cross Device Development] trachyte:
Ok, one last round of tests. This is probably it for me.

5.0.8 -> 9.0.5 -> 5.0.8 in one TWRP session

Same session

Decryption works

Reboot to recovery

Decryption works

Reboot to system

Decryption works

Flashing 9.0.5 firmware with 5.0.8 keymaster.mbn

Recovery

Decryption does not work, -33 aka INVALID_KEY_BLOB

Flashing 5.0.8 over top

Decryption works

Thoughts

This means that 9.0.5 keymaster.mbn is not required for decryption to break. In this test, of the possible suspects (9.0.5 versions of keymaster.mbn and emmc_appsboot.mbn) only emmc_appsboot.mbn was present. That means emmc_appsboot.mbn alone is enough to break the decryption. Also, we were able to recover our data by flashing 5.0.8 back. This hasn't been possible when both of the suspects were flashed together. So... emmc_appsboot.mbn causes the decryption to break, and keymaster.mbn makes the change permanent?

Flashing 9.0.5 firmware with 5.0.8 emmc_appsboot.mbn

Recovery

Decryption works

Flashing 5.0.8 over top

Decryption works

Thoughts

Further confirmation that emmc_appsboot.mbn is the culprit. In this test we applied 9.0.5's keymaster.mbn and the decryption still worked. keymaster.mbn exonerated?

Flashing 9.0.5 firmware

Recovery

Decryption doesn't work (-33 aka INVALID_KEY_BLOB)

Flashing 5.0.8 firmware over 9.0.5

Decryption works

Thoughts

WTF. We flashed all of 9.0.5, rebooted, verified encryption was broken, downgraded to 5.0.8 and decryption worked again. This is not supposed to be possible.

Flashing just 9.0.5 keymaster.mbn by itself

Recovery

Decryption works

System

Decryption works (first prompt)

OS asking for password after decryption: Wrong Password (returns instantly like it's not really checking)

2nd System Boot

Decryption works (first prompt)

OS asking for password after decryption: Wrong Password (returns instantly like it's not really checking)

2nd Recovery Boot

Decryption works

Thoughts

Ok... keymaster.mbn exonerated again, but Android could do the decryption fine but then not verify the password at login? This is getting weird.

Flash 5.0.8 firmware

Recovery

Decryption works

System

Decryption works (first prompt)

Doesn't ask for second password (as is normal when the decryption password also works as the login password)

Recovery

Decryption works

Thoughts

Another confirmation that keymaster.mbn isn't doing any lasting damage by itself.

Flash just 9.0.5 emmc_appsboot.mbn by itself

Bootloop immediately after bootloader unlocked warning

Can boot to fastboot only

Oddly, fastboot reports the bootloader is fully unlocked, including critical, but flashing anything other than recovery fails with "Partition flashing not allowed"

So, yeah, it appears emmc_appsboot.mbn does the heavy lifting when it comes to breaking the decryption. keymaster.mbn is either entirely innocent or perhaps unwittingly piggybacks on emmc_appsboot.mbn's handywork and ends up causing lasting damage that a downgrade can't undo.

I feel like someone who works on the bootloader and keymaster and has all of the source code would know what the problem is just by reading the test results. But without being in that position there's a lot of reverse engineering left. I still think there's a way to recover the data (from a bit-for-bit copy of userdata) and that it probably just takes a couple lines of code fixing some HW module state or regenerating the signing keys with the right parameters. But figuring out exactly what is needed, that's a long slog and I want to have a usable phone again :D. And I will, as soon as I figure out how to unbrick this one :silly: