[OnePlus 3 & 3T Cross Device Development] trachyte:
"I will have no comment until the investigation is complete."

-- A Very Smart Person​

Previous test:

Upgraded firmware 5.0.8 -> 9.0.5

Rebooted to recovery

Results:

4K of first 10MiB of userdata was modified

I was not expecting userdata to be modified so I wasn't tracking precisely what the conditions were during the flash or checking precisely when the modifications occurred. But the sequence was: flash 9.0.5, immediately reboot to recovery and discover that userdata had been modified.

So, obviously it was the firmware that made the changes.

Caught you, firmware!

Now let's replicate that result and narrow down exactly when the userdata is modified and exactly how it is modified. All of the Upgrade Tests below start with: Wipe, Format, Flash 5.0.8 firmware, reboot to system, do initial setup and enable encryption, reboot to recovery. All tests were done with LOS 16.0-20190501 and TWRP 3.3.0-1. This time I checked the first 500MiB of userdata, and also 500MiB of data starting exactly 20GiB into userdata. I also checked the crypto footer, but there were no changes to it in any of the tests besides setting CRYPTO_DATA_CORRUPT and updating the SHA256 sum.

Upgrade Test 1

Flash 9.0.5 firmware (and do not reboot)

No changes

Let recovery sit for 15 minutes

No changes

Reboot into recovery

No changes

Click around in recovery UI but don't change anything

No changes

Reboot to recovery

No changes

Do literally 55 other steps (reboots to recovery, reboots to system, enter correct/incorrect passwords, hard power off, and more) to give every opportunity for something to modify the userdata

No changes

Thoughts

That's strange, but I must have done something differently in this test. I'm sure another test will be able to replicate the original results.

Upgrade Test 2

Flash 9.0.5 firmware (and do not reboot)

No changes

Reboot into system, don't enter password, reboot to recovery

No changes

4 more various steps

No changes

Thoughts

Ut oh, that's twice in a row. Did I cry wolf about userdata being modified? Hmm, I wonder if I had userdata mounted to /data in recovery during my original test.

Upgrade Test 3

Mount /data in recovery, flash firmware 9.0.5, reboot to system, don't enter password, reboot to recovery

First 500MiB: ~500 bytes changed

Middle 500MiB: ~130 bytes changed

Thoughts

Aha! /data needs to be mounted during the flash in order for userdata to be modified. The firmware probably isn't aware of userdata unless it's mounted during the flash. Doesn't quite make sense but the test data doesn't lie. The modifications are a lot smaller than my original test but both the first 500MiB and middle 500MiB changes take place on well-known boundaries. The first changes to both are exactly 1024 bytes in. Clearly the firmware is doing a quick reformat of some type. Let's do this same test again just to make sure /data being mounted is what makes the difference.

Special Test 3.5

Purpose

This was a quick test while resetting to 5.0.8. I was worried that since /data was mounted during the install, and therefore underwent unmounting during the reboot after flashing, it may have had data written to it as part of the unmounting process (update last time used, etc.).

Mount /data in recovery, flash 5.0.8 firmware

No changes

umount /data

No changes

Reboot to recovery

No changes

Thoughts

This test shows that unmounting /data doesn't cause any modifications to userdata. Good!

Upgrade Test 4

Same as Test 3: Mount /data in recovery, flash firmware 9.0.5, reboot to system, don't enter password, reboot to recovery

First 500MiB: ~670 bytes changed

Middle 500MiB: ~130 bytes changed

Thoughts

Yep, clearly /data being mounted during the flash is what is causing the firmware to modify userdata.

Special Test 5

Purpose

Check if the firmware will modify the userdata on every boot, or only the first boot

Restore first 500MiB and middle 500MiB to their original 5.0.8 states.

Reboot to recovery

No changes

Reboot to system, reboot to recovery

No changes

Enter incorrect password, correct password into recovery

No changes

Reboot to system, enter incorrect password

Get 'Password is correct but data is corrupt' error

Reboot to recovery

No changes

Clear CRYPTO_DATA_CORRUPT flag

Thoughts

The firmware isn't modifying the userdata on every boot. Whatever is occurring is only happening on the first boot.

Special Test 6

Purpose

Let's make absolutely sure unmounting /data doesn't cause any modifications to userdata. I mean, what if there are some pending changes and the kernel doesn't write them until some time after the unmounting? The modifications to userdata do occur at locations like 1024, which is where the ext4 superblock starts. Instead of a quick format these could just be normal bookkeeping writes to the superblock during the unmounting. I better check everything to be super positive that there are no pending changes waiting to be written.

Flash 9.0.5 firmware over itself

umount /data

sync

Confirm that /proc/sys/vm/dirty_bytes = 0

Confirm that column 9 of /sys/block/sda15/stat is 0

Confirm that column 9 of /sys/block/dm-0/stat is 0

Confirm that Dirty and Writeback in /proc/meminfo are 0Kb

Reboot to recovery

No changes

Reboot to system, reboot to recovery

No changes

Thoughts

Great, unmounting /data doesn't cause any writes. Super confirmed. I actually did this test twice to be certain.

Special Test 7

Purpose

Wait, I unmounted /data, but the dm-crypt device (/dev/block/dm-0) still existed. What if removing that caused writes to userdata? Unlikely but let's check anyway.

Cross-compile static dmsetup for ARM (goodbye five hours of my life), push to phone

Reboot to recovery, mount /data

umount /data /sdcard

No changes

sync

Wait a minute

Make sure there are absolutely no pending writes (/proc/meminfo, etc.)

dmsetup ls

userdata (253:0)

dmsetup table

userdata: 0 112477112 req-crypt

dmsetup remove userdata

dmsetup ls

No devices found

Result of removing dm-crypt device

First 500MiB: ~144 bytes changed (exactly the same areas as changes in previous tests)

Thoughts

Crap. It wasn't the firmware doing a quick format, it was just the unmounting of /data and subsequent removal of dm-0 causing the superblock to be written to.

So there you go. After all of this testing I've eliminated a lot of possible causes and established one essential fact:

The crypto footer and userdata are not modified during the upgrade

There were multiple tests in this latest round where there were no changes to the crypto footer or userdata but the decryption failed in the same manner it always has.

This is good for me because it means the bit-for-bit copy of userdata I made after upgrading is actually correct. The correct keymaster_blob, encrypted master key, and the data should all be there. It must be the HW module not signing the IK correctly that is causing the decryption to fail. So something is happening to the HW module or KeyMaster or TrustZone that isn't reversed by simply downgrading the firmware back to 5.0.8.

What's next?

1. Test 5.0.8 -> 9.0.5 -> 5.0.8 in one TWRP session, see if decryption still works in that session, and after rebooting.

2. Install the 9.0.5 firmware, but only flash one image at a time, in order to narrow down which image causes the issue.

3. Once the offending image is identified, reverse engineer it to see what calls it's making to the HW module.

I'm going to do 1. I might do 2 and/or 3, not sure yet.

The cause might have something to do with Version Binding (https://source.android.com/security/keystore/version-binding), although the keymaster_blob should have been updated if the key was updated. And for various other reasons, despite version binding looking like a prime candidate, I'm leaning toward it not being the cause.

Edit: [MENTION=4405529]nvertigo67[/MENTION]'s modified firmware is using 5.0.8 emmc_appsboot.mbn and keymaster.bin, so the 9.0.5 version of one of those is responsible, keymaster.bin being the obvious suspect as [MENTION=4405529]nvertigo67[/MENTION] has previously pointed out in this thread.