yashsingh1/xdaforums huggingface dataset:
[OnePlus 3 & 3T Cross Device Development] trachyte:
Ok, more from my research into the 'Password correct but data is corrupt' error.

1) I verified that CRYPT_DATA_CORRUPT was set in the crypto footer flags. The flags were actually 0x00001008. 0x00000008 is CRYPT_DATA_CORRUPT. 0x00001000 has no meaning here so I don't know why that was set.

2) I cleared CRYPT_DATA_CORRUPT in the flags, rebooted into the system, and as expected Android asked for the password again. It fails with the same error, and sets CRYPT_DATA_CORRUPT in the flags so that subsequent boots don't even ask for the password and go straight to the error message.

3) Clearing the 0x00001000 in the flags made no difference. In fact Android also adds the 0x00001000 back into the flags when it shows the error message. I can't find any place in AOSP that sets this, which is strange, but I don't think it makes a material difference.

The underlying reason the decryption fails appears to be that the keymaster_blob in the crypto footer is incorrect. The evidence for this:

1) All other fields of the crypto footer are the correct size and in the correct format. keymaster_blob, however, is shorter than it should be and is mostly zeros. The zeroed area is where data such as the IV and HMAC should be.

2) When attempting to decrypt in TWRP, the error returned is -33 aka INVALID_KEY_BLOB.

Next steps:

It would be really helpful if I had known-good crypto footers that I could compare to. If you have a working phone with an encrypted userdata partition and would be willing to send me the crypto footer (last 16KiB of userdata partition), here is the command to run in recovery:

adb shell
p=`basename \`readlink /dev/block/by-name/userdata\``; s=`grep "$p\$" /proc/partitions | awk '{ print $3 }'`; o=`expr \( "$s" \* 1024 \) - 16384`
dd if=/dev/block/by-name/userdata of=/tmp/crypto_footer skip="$o" bs=1

adb pull /tmp/crypto_footer crypto_footer

And then upload crypto_footer somewhere I can grab it. You'll know you have the right data when the first bytes are C4 B1 B5 D0 or similar.

Having the crypto footer means I would have a slight head start in cracking your device if I had access to it, but is of no use without the device, so use your own judgement about your security situation before sending it.

Once I'm able to compare my crypto footer to known-good ones, I will be able to see exactly what is wrong with the keymaster blob. And from there I'll be able to see if the affected areas can be recreated or if there's no way to recover it.
[OnePlus 3 & 3T Cross Device Development] :
[MENTION=7065487]trachyte[/MENTION]:

You last posting is one of the most interessting readings for a long time!

I'm stiil using 5.0.8 bootloader and keymaster (for details see https://xdaforums.com/oneplus-3t/how-to/guide-cope-9-0-3-5-0-8-firmware-barrier-t3941164). You still need a clean encryption footer with plain 9.0.x bootloader and keymaster.

That being said: if you can't find a way to migrate a 5.0.x footer to a 9.0.x compatible one, you can reset the footer to it's former 5.0.x state and access your userdata partition with 5.0.x fw or the modded 9.0.x fw from tje thtead mentioned above.

I'm crossing fingers, you'll find a way to transparently migrate a 5.0.x footer to a 9.0.x footer and vice versa.

I'm really impressed by your findings!

[MENTION=9350973]All[/MENTION]: [MENTION=7065487]trachyte[/MENTION] still needs a footer from an encrypted 9.0.x fw!!!!
[OnePlus 3 & 3T Cross Device Development] :
It's not the updater it's twrp. Cutrently no logs are needed. Hopefully the next twrp build will have it fixed. Thanx for your offer.
[OnePlus 3 & 3T Cross Device Development] jakito:
Here's a crypto footer from a working 9.0.x firmware OP3.

Edit: changed the crypto footer name to make it more understandable
[OnePlus 3 & 3T Cross Device Development] trachyte:
This is getting stranger. Your keymaster_blob is exactly like mine. A full qcom keymaster blob is:

struct qcom_km_key_blob {
uint32_t magic_num;
uint32_t version_num;
uint8_t modulus[KM_KEY_SIZE_MAX];
uint32_t modulus_size;
uint8_t public_exponent[KM_KEY_SIZE_MAX];
uint32_t public_exponent_size;
uint8_t iv[KM_IV_LENGTH];
uint8_t encrypted_private_exponent[KM_KEY_SIZE_MAX];
uint32_t encrypted_private_exponent_size;
uint8_t hmac[KM_HMAC_LENGTH];
};

We have these in the correct format:

uint32_t magic_num;
uint32_t version_num;
uint8_t modulus[KM_KEY_SIZE_MAX];
uint32_t modulus_size;

Except the magic_num is supposed to be 0x4B4D4B42 (in Android 8.0 and 9.0). Our magic_num is 0x4B4D4B44. I can find zero references to that magic number anywhere online. The 42 -> 44 change suggests somebody incremented the magic number twice. We're two versions ahead? It also doesn't make sense that they would increment the magic number because it's supposed to be "KMKB" for "Key Master Key Blob" and now it's "KMKD". There's also a perfectly good version_num field they could have incremented instead.

The 512 bytes for public_exponent has 414 bytes of data and the remainder are zeros. public_exponent_size and the rest of the fields are all zeros. Then the keymaster_blob_size follows and is 1187. The keymaster_blob_size is supposed to be 2048, and the beginning of 1187 is exactly 2048 bytes from the magic number. And 1187 doesn't correspond to the end of the data or anything, it just points to a spot amongst the zeros.

I think it's a small miracle your phone still boots :laugh:

I'd really like to see clean 5.0.x and 9.0.x crypto footers. Or any crypto footers from Android 8 or 9, doesn't have to be from OxygenOS firmware (although probably not from Samsung since they do things somewhat differently). This is all leading to me flashing 5.0.x and experimenting with that, but I'd like as much information as I can before I do that.

---------- Post added at 12:16 PM ---------- Previous post was at 12:07 PM ----------

Thanks. Was this phone upgraded from 5.0.x or was it a clean 9.0.x install?

Edit: Hmm, XDA made my reply an edit. That was intended for [MENTION=7652161]jakito[/MENTION]
[OnePlus 3 & 3T Cross Device Development] jakito:
It was a clean install. I upgraded but it messed up the phone so much I just got annoyed and formated everything, changing from ext4 to f2fs and back to ext4... hope this helps you :)

Edit: And didn't restore anything btw. I just installed everything from the PlayStore and the LOS was also a clean install. Sorry for the misunderstanding.
[OnePlus 3 & 3T Cross Device Development] :
Thanx God my device doesn't know that... :p

The footer, I've posted was generated with twrp 3.2.1-0 (old non unified version), fs is f2fs, initial encryption was done with nlos-15.1. I've clean flashed 16.0. I've dirty flashed builds after the native oneplus blobs switch with modded fw.
[OnePlus 3 & 3T Cross Device Development] noisy_lightning:
Hi, it's been a while since I ran LOS on my OP3. The main reason I went back last time was the camera quality vs stock was so much worse, and despite weeks or trying I was unable to find solutions to fixing it.

How do we fare now in comparison to stock on photos, have OP released the source code yet for the camera?
[OnePlus 3 & 3T Cross Device Development] jakito:
I use the gcam mod. Can't really talk about the native camera, but the gcam is superb.

Edit: And I believe is better then the OOS native one.
[OnePlus 3 & 3T Cross Device Development] Notor1ouS-:
I'm on Firmware / Modem 9.0.4.

can i update to 9.0.5 on the latest LineageOS build or will i get problems?
[OnePlus 3 & 3T Cross Device Development] :
No, go ahead. The problem is crossing the 5.0.x-9.0.x (or vice versa) border. There are no 9.0.x-9.0.x+n borders.
[OnePlus 3 & 3T Cross Device Development] mottenkiller:
Here is mine.

My recent phone history:

- Stock 9.0.5 (encrypted)

- format /data + wipe /system

- LOS 16 / encrypted today
[OnePlus 3 & 3T Cross Device Development] S4turno:
Quality on custom ROMs will always be worse than stock, but even stock is crap compared to what you can get with a modded Google Camera app: https://xdaforums.com/oneplus-3/how-to/modded-google-camera-hdr-60fps-video-t3658552

The one from Urnyx05 works out of the box. Only slow motion is broken.
[OnePlus 3 & 3T Cross Device Development] felagund:
Last time I went to OxygenOS it was also due to camera performance and Sultan stopping development and having to go clean flash anyways. Stock camera was faster and better than foss camera. However when I tried the gcam ports it was night and day and actually I was having problems swapping the stock for gcam in all lockscreen+gestures so LineageOS was better for lots of reasons and customizing camera shortcuts instead of forcing the stock cam was a major positive.

So you can give it a go and use a recent good quality gcam port.

You will still have lower speed of starting and using the cam but at least no painting effect like on stock and overall good quality and lots of options.

I recently broke my back camera glass and sensor and can't do comparison photos to show.
[OnePlus 3 & 3T Cross Device Development] :
[MENTION=7065487]trachyte[/MENTION]:

Thank you very much for all the effort you put in testing and documenting!!!

Just some thoughts:

Keep in mind that oneplus stated that only locked bootloaders can be transparently updated. I still think all the update trouble is no bug (though oneplus stated to [MENTION=5091734]dianlujitao[/MENTION] they wil "fix" it), but intended. Modifying data on first boot no matter if the passphrase was given "smells" more like intention...

I'm pretty sure the keymaster partition plays in here, too.
[OnePlus 3 & 3T Cross Device Development] dogmatism:
change the default kernel. I have seemed to overcome this bug, by flashing latest madkernel
[OnePlus 3 & 3T Cross Device Development] rohan164:
this post keeping me away to flash LOS on 3t. is it safe to flash?

[MENTION=7065487]trachyte[/MENTION]
[OnePlus 3 & 3T Cross Device Development] trachyte:
There's no problem flashing LOS itself on the 3T. The conditions that cause data loss are: upgrading 5.0.x -> 9.0.x firmware, with an unlocked bootloader, and encrypted userdata. If you avoid that situation you should be fine (but make a backup of course). If you can't avoid that situation then you can use @nvertigo67's modded firmware.
[OnePlus 3 & 3T Cross Device Development] syntesys:
Have someone flashed 20190826 version and booted? I tried it dirty flashing (9.0.4) and it not starts. Reverting to 20190823 works.
[OnePlus 3 & 3T Cross Device Development] Not-Aus:
26.08. worked for me! (9.0.5)
[OnePlus 3 & 3T Cross Device Development] :
Last week updated every day; no issues here either, all builds booting fine... :good:
[OnePlus 3 & 3T Cross Device Development] An68:
No problem here(9.0.4)
[OnePlus 3 & 3T Cross Device Development] dianlujitao:
apparently they don't wanna keep taking care of an old device, the "fix" they promised is a ghost
[OnePlus 3 & 3T Cross Device Development] chotaire:
Dirty-flashing 20190827 over 20190823 (+Opengapps-Nano +Magisk) would result in endless boot. First time I've been having a showstopper issue in uhm.. years? Dirty-flashed 20190823 again, phone will now boot.
[OnePlus 3 & 3T Cross Device Development] :
Not putting any more effort in an old device after nearly 3 years is ok.

Lying to a customer is a completely other story. That's bad manner! That the customer is my favorite device maintainer doesn't make a difference.

BTW: thanx for all the effort you constantly spend on op3/t for years!

Have you already tried to uninstall all Magisk modules and Magisk, updated the rom, boot once to system before reinstalling Magisk and it's modules?
[OnePlus 3 & 3T Cross Device Development] felagund:
I liked my OP1 despite its flaws.

OP3 was ok but OnePlus kept making design decisions I don't agree with and forcing functionality I'm not happy with. Maintainers are the heros here and they make the device what I expected it to be. I have decided my next phone will not be a OP device even when now recently OP6 was heavily discounted on prime day.

I've seen too many maintainers move to the nexus or pixel lines in my last few decided and I'm very thankful to all those who still care to look after our devices as I am not capable to develop.

So thanks!
[OnePlus 3 & 3T Cross Device Development] :
[MENTION=7065487]trachyte[/MENTION]

That some crazy testing you've done there bro :silly:

If you need any help, feel free to ask for it in case somebody here able to help. We're here as a community, not as a pre-installed app in your phone (if you know what i mean).
[OnePlus 3 & 3T Cross Device Development] syntesys:
It's the same for me, I tried with 20190826 and 20190827 without result. Also tried to clean system partition before reflash everything. No module on magisk is present.
[OnePlus 3 & 3T Cross Device Development] S4turno:
After so many forgotten promises and lies, it doesn't surprise me anymore. This is how OnePlus operates.

We can't do much to stop them, but we can choose different brands next time we buy a new phone.
[OnePlus 3 & 3T Cross Device Development] :
Since the update works for some users and don't work for others, it's quite obvious that the failing update is setup dependent. If uninstalling magisk (btw: I strongly recommend unimstalling magisk ith the flashable uninstaller - reformating system without uninstalling magisk to get rid of magisk may work but also may gibe you trouble, because the traces of magis in userdata can interfere) doesn't work, make a clean flash with only gapps nano or pico WITHOUT restoring anything. If that works restore your apps and their data one by one to see what breaks your setup.
[OnePlus 3 & 3T Cross Device Development] syntesys:
Hi and thank you for your reply, but also with magisk uninstaller the problem persists. I think the problem is not magisk related but ROM release related, as [MENTION=6289001]chotaire[/MENTION] had the same problem coming from the same version I have. Clean flash is too much simple...
[OnePlus 3 & 3T Cross Device Development] :
Had you've done enough testing? The only way to certain it's build related is if you clean flash previously working version and then dirty flash the "problematic" build version as you said.

You should have done 2 different testing on it, first wth only rom flashed, second along with magisk with no module installed.

Those two test with NO PERSONAL SETUP APPLIED. Those two test will determine which is at fault; the rom, magisk, or your personal setup.

If you and your other person can't do that because it's too much simple, let's just ignore your experiences and move on :silly:
[OnePlus 3 & 3T Cross Device Development] crypta:
the 2608 nightly also breaks the build here, bootloop.

and yes, with magisk. the same magisk, which doesn't break updates on my aosp custom room on op6.

so the expectation here is, that an update will be possible. and no, restart from scratch is no viable option, except for diagnosis.
[OnePlus 3 & 3T Cross Device Development] syntesys:
Yes, working in clean flash is too much simple, and I don't have to reformat the phone every week for a new relase of a ROM. The rom worked good from the launch, so I don't expect a problem because, as a programmer, I have to mantain the application working without regressions. :D

Told and forgot that, I cleaned all my phone (data, sdcard and all the other) and done a test; the ROM alone boots, but it does not boot if I apply a modification to system partition: if I flash gapps, it does not boot. The last ROM which boots is 20180823.

EDIT: I've extracted a logcat of the boot with the new ROM with only gapps installed. Here attached the file.

Thank you.
[OnePlus 3 & 3T Cross Device Development] JustPlayingHard:
Hi there!

I've gone through the logcat and the one thing that caught my eye was the following: 08-30 14:58:32.580 9669 9669 E AndroidRuntime: java.lang.RuntimeException: There must be exactly one installer; found []

After that, a bunch of services die; device_identifiers, batterystats, appops, power, recovery and display. Seeing as you're a programmer, I'm sure you know these :D

My only guess is that the GApps package is removing something within system that's causing the services to die out and not to have a installer installed or something. That's my guess as I have no clue exactly so I might be entirely wrong haha

EDIT: Just noticed that it is failing to report an app crash and isn't able to handle the crash, might be another reason for it to not boot up?
[OnePlus 3 & 3T Cross Device Development] syntesys:
Yes, found it, looked on Google and someone said a problem in Android.mk (or something similar).

I tried after that to install only ROM and mindthegapps package and after boot with success. Then, installed Titanium backup as system app (the installer script makes only a copy without remove) and no boot, doing the same problem. Tried clean installation of 20190823 with all these packages and the phone booted with success; then tried 20190824 with the mentioned packages and no boot. So, there is something in 20190824 that creates some sort of error.
[OnePlus 3 & 3T Cross Device Development] JustPlayingHard:
Interresting. So there's something broken in the 20190824 builds causing this. Possibly permissions problems, who knows...
[OnePlus 3 & 3T Cross Device Development] Dirk:
Recent OpenGapps packages were broken and have been pulled from SF. I've seen people reporting problems like some of those posted here in multiple ROM threads, all from people who have just clean installed new ROM builds, using new OpenGapps.

https://xdaforums.com/showpost.php?p=80158522&postcount=6183

Try installing again with working Gapps packages.
[OnePlus 3 & 3T Cross Device Development] sky321:
I‘ve seen similar problems on an other device. Last running version from 23rd and all builds after that date stuck at boot. This problems are related to a change from 24th https://review.lineageos.org/c/LineageOS/android_vendor_lineage/+/251804

Just guessing but probably it‘s related
[OnePlus 3 & 3T Cross Device Development] syntesys:
Looking at this, this could be the problem. OpenGapps installs some APK in priv-app folder, mindthegapps no; Titanium backup zip file installs the app in priv-app folder.

It's surely related to this patch.

Thank you.
[OnePlus 3 & 3T Cross Device Development] syntesys:
It's not gapps package problem, because I used with 20190829 version the package which worked when I installed Los 16 first time: a package from June. Also, I used the new package with 20190823 version and worked.
[OnePlus 3 & 3T Cross Device Development] :
Hmm, I would follow you if updating fails for all users. But as a matter of fact updating works for the vast majority of users with this patch applied and is only failing for very few users...
[OnePlus 3 & 3T Cross Device Development] syntesys:
Maybe because they install the base and don't modify their system partition :p Thank you!
[OnePlus 3 & 3T Cross Device Development] mottenkiller:
I tried today:

LOS 16 (2019-08-21) --> LOS 16 (2019-08-29)

device encrypted, opengapps pico installed, no root, no magisk

Flashed dirty and got a bootloop.
[OnePlus 3 & 3T Cross Device Development] BillGoss:
And I dirty flashed 190829 over 190811 and had no problems.

I'm encrypted, /data is ext4, using open gapps nano from 190225, and Magisk 19.3.

Edit: if I were you I'd be doing a clean install without gapps and no changes to the system. I'm guessing it's one of those two that's causing the problem.
[OnePlus 3 & 3T Cross Device Development] vinoxflame:
dirty flashed from 8/25 to 8/29 without issue also....Im decrypted with ext4 filesystem and rooted with magisk
[OnePlus 3 & 3T Cross Device Development] crypta:
it's not gapps. not using these.
[OnePlus 3 & 3T Cross Device Development] :
Are you assuming that all users with working updates doesn't use gapps? Flashing gapps always modifies system.

I'd suggest opening a proper issue report with all mandatory information on gitlab. Without logs and detailed setup imformation (i.e. versions of all used software) this all is blindly guessing.
[OnePlus 3 & 3T Cross Device Development] syntesys:
Nope, I know that gapps modify system but, looking the patch description, it changes the permissions only on priv-app folder, and not every package writes APKs in that folder.

Yes, I'll insert asap an issue report.

Thank you.
[OnePlus 3 & 3T Cross Device Development] :
Please be more precise: I can't gasp that you use "is modiging system" in the meaning of "is installing packages as previleged packages".

GmsCoreSetupPtebuilts doesn't need to be installed as previleged app on some gapps packages? Can you give me the exact name and version of the gapps package, please. It's completely new to me, that the gms setup app doesn't need previleged permissions to work.
[OnePlus 3 & 3T Cross Device Development] syntesys:
I tried with open_gapps-arm64-9.0-nano-20190830.zip and open_gapps-arm64-9.0-nano-201900615.zip versions, with the same result.
[OnePlus 3 & 3T Cross Device Development] :
My question has been: would you please give me the exact name and version of tje gapps package, which doesn't install GmsCoteSetupPrebuilt.apk to priv-app?
[OnePlus 3 & 3T Cross Device Development] syntesys:
The gapps package that makes the rom boot is MindTheGapps-9.0.0-arm64-20190615_031441.zip.
[OnePlus 3 & 3T Cross Device Development] :
What gapps package did you use - if any at all?

---------- Post added at 10:37 ---------- Previous post was at 10:23 ----------

This very package installs to system/priv-app:

unzip -l /mnt/export/data/Downloads/incoming/Android/MindTheGapps-9.0.0-arm64-20190615_031441.zip system/priv-app/\*
Archive: /mnt/export/data/Downloads/incoming/Android/MindTheGapps-9.0.0-arm64-20190615_031441.zip
signed by SignApk
Length Date Time Name
--------- ---------- ----- ----
7205846 02-29-2008 03:33 system/priv-app/AndroidMigratePrebuilt/AndroidMigratePrebuilt.apk
33182 02-29-2008 03:33 system/priv-app/GoogleBackupTransport/GoogleBackupTransport.apk
784625 02-29-2008 03:33 system/priv-app/GoogleFeedback/GoogleFeedback.apk
2125436 02-29-2008 03:33 system/priv-app/GooglePartnerSetup/GooglePartnerSetup.apk
4422984 02-29-2008 03:33 system/priv-app/GoogleServicesFramework/GoogleServicesFramework.apk
31063119 02-29-2008 03:33 system/priv-app/Phonesky/Phonesky.apk
105423886 02-29-2008 03:33 system/priv-app/PrebuiltGmsCore/PrebuiltGmsCore.apk
10973175 02-29-2008 03:33 system/priv-app/SetupWizard/SetupWizard.apk
80572262 02-29-2008 03:33 system/priv-app/Velvet/Velvet.apk
--------- -------
242604515 9 files
[OnePlus 3 & 3T Cross Device Development] helderfp:
Did you installed via OTA. If so that is the cause.
[OnePlus 3 & 3T Cross Device Development] mottenkiller:
Sorry, forgot to mention:

Flashed via TWRP 3.3.1-0

I will send a logcat later.

Atm I am busy due to a health issue. :(
[OnePlus 3 & 3T Cross Device Development] mottenkiller:
Sorry for false alert.

Flashed again today, working fine here.

Idk, what was going wrong... The most propably explanation: I was distracted and impatient.
[OnePlus 3 & 3T Cross Device Development] h4rl4n:
I guys i just performed a clean install with the latest nightly (full wipe), now the phone starts and after 4-5 minutes on the boot animation it just boots back to recovery. Suggestions?
[OnePlus 3 & 3T Cross Device Development] G-Ray:
Same here, I tried to downgrade to 20190823 without success.
[OnePlus 3 & 3T Cross Device Development] h4rl4n:
I also noticed this. If i wipe dalvik cache and system then it boots into system, flash magisk, back to initial situation: boot animation and then recovery, in loop