yashsingh1/xdaforums huggingface dataset:
[General Questions and Answers] xXx yYy:
Can't mentally comprehend what you are telling me. :confused:

FYI:

The partition /userdata gets detected by Android as /data.
[General Questions and Answers] AOSP100:
It's the tmp/recovery.log in the recovery menu after it gave up and said that the OS is corrupted. I tried running fsck.f2fs, but it gives me something like "invalid f2fs superblock on" or "bad magic number". I loaded TWRP (that unofficial build for Android 12), and ran twrp decrypt, but it stopped with "decrypt process ended with signal 6".
[General Questions and Answers] Renate:
If you've got disk-based encryption it will not show up as a valid filesystem.

Be careful, if you do some things it will wipe the keys for decryption and you'll never be able to decrypt it.

I know that on my main device I can do a full raw backup (32 GB) and restore and everything is as it was.

I've swapped between two different backups of different dates and it works fine.

I don't know if a factory reset would clear the keys in the TZEE (Trust Zone Execution Environment).

I don't know everything, but I haven't shot myself in the foot (yet).
[General Questions and Answers] AOSP100:
It's file-based encryption. See https://github.com/LineageOS/android_device_google_sunfish/blob/lineage-21/fstab.hardware for what I mean.

/dev/block/bootdevice/by-name/userdata /data f2fs noatime,nosuid,nodev,discard,reserve_root=32768,resgid=1065,fsync_mode=nobarrier latemount,wait,check,quota,formattable,fileencryption=ice,reservedsize=128M,sysfs_path=/dev/sys/block/bootdevice,keydirectory=/metadata/vold/metadata_encryption,checkpoint=fs

/dev/block/by-name/metadata /metadata ext4 noatime,nosuid,nodev,discard,data=journal,commit=1 wait,formattable,first_stage_mount,check,metadata_csum

As I said, what's in the `keydirectory` regenerates at boot and is not immutable. The `fileencryption=ice` is explained at https://source.android.com/docs/security/features/encryption/file-based
[General Questions and Answers] Renate:
Ok, but do you think that you can save userdata as raw binary, do a fastboot -wipe, then restore userdata?
[General Questions and Answers] AOSP100:
It's what I did, and it's where the pasted recovery.log came from
[General Questions and Answers] Renate:
Ok, just to be clear what I did. I used EDL (my client) for everything.

C:\>edl /lpoke3.bin /r /s0 /c0 backup1.img
-- six months passed --
C:\>edl /lpoke3.bin /r /s0 /c0 backup2.img
C:\>edl /lpoke3.bin /w /s0 /c0 backup1.img
-- now everything works and it's like six months ago --
C:\>edl /lpoke3.bin /w /s0 /c0 backup2.img
-- now everything works and it's like an hour ago --

Would this have worked if I had fastboot -w ?

I don't know. I didn't want to try.
[General Questions and Answers] CXZa:
@alecxs ?
[General Questions and Answers] AOSP100:
Sunfish might not have an EDL or QDL mode. Either that or your EDL tool doesn't detect it even in rescue mode.
[General Questions and Answers] xXx yYy:
The "userdata" partition, which typically stores user data encrypted by FBE on operational devices, might still be encrypted even in EDL mode.
[General Questions and Answers] AOSP100:
How else do I make it bypass all integrity checks and boot straight in? There's no way of monitoring what it's doing until it boots to recovery.
[General Questions and Answers] xXx yYy:
To clarify things:

1. EDL is a special state on Android devices equipped with Qualcomm processors. The command adb reboot edl is a way to reboot an Android device equipped with a Qualcomm processor into EDL mode.

2. EDL mode itself doesn't operate on commands like ADB or fastboot that you might be familiar with on Android.

3. In EDL mode, the device bypasses the normal boot process and presents a minimal interface for interacting.

4. It's important to note that EDL mode itself doesn't necessarily involve encryption or decryption of user data. While FBE on Android encrypts user files on storage, EDL mode typically deals with flashing firmware which resides in a separate partition and might not be encrypted.

5. Extracting the userdata partition directly while in EDL mode on Android devices is generally not possible.

My recommendation:
[General Questions and Answers] Renate:
Of course it's possible to extract the userdata partition. It's just not possible to decrypt it outside the device.

Let's look at this logically. What is the point of encryption?

It prevents people who have the entire contents of your flash from reading your data.

EDL is low level. It can copy the entire contents of your flash. No problem.

It's about the same as unsoldering your flash chip (eMMC or UFS) and reading everything.

The point is, there is other data, (SoC serial number, NVRAM, who knows) that is necessary to generate the keys to decrypt.

If I buy the same model phone as you have and copy over the entire flash (or even solder in your flash) it will not be able to decrypt it.

To answer the question of the OP. Just write the userdata partition back.

Did it work? Great! Did it fail? Sorry, you must have wiped the keys somehow.
[General Questions and Answers] alecxs:
please show a hexdump of first 1024 bytes
[General Questions and Answers] xXx yYy:
Yes, technically, in EDL mode you might be able to read partitions (images ) / the whole from the eMMC of an Android device.

Even if you manage to read partitions / the whole from the eMMC, the data will be in raw format, which is not human-readable and requires knowledge to interpret. You need a tool that offers loopback mounting, where you can map the raw image file as a virtual disk. This allows accessing the data using file system utilities.

Here's an example of how to use loopback mounting in Linux to access data from a raw image file:

# Replace "userdata.img" with the actual filename of your raw image

RAW_IMAGE_FILE="userdata.img"

# 1. Identify File System Type (assuming ext4)

# This might require installing additional packages depending on your distro

file -s "$RAW_IMAGE_FILE"

# Output Example: userdata.img: Ext4 filesystem data (native)

# 2. Use losetup to map the image to a loop device

# Find a free loop device using losetup -f

sudo losetup -f "$RAW_IMAGE_FILE"

# Output Example: /dev/loop0: [userdata.img]: unknown

# Check mapping with losetup -l

sudo losetup -l

# Output Example: /dev/loop0: [userdata.img]: unknown 0 0

# 3. Mount the loop device to a directory (replace /mnt/userdata with your desired mount point)

sudo mkdir /mnt/userdata

sudo mount /dev/loop0 /mnt/userdata

# 4. Now you can access the data in the raw image using file system utilities within /mnt/userdata

# Example: explore the root directory of the partition (might require root privileges)

sudo ls /mnt/userdata

# 5. Once finished, unmount the loop device and detach the image

sudo umount /mnt/userdata

sudo losetup -d /dev/loop0
[General Questions and Answers] Renate:
Might? Definitely.

I don't know why it sounds like you're disagreeing with me.

EDL does low level reading/writing.

The capabilities are limited only by stupid junk that OEMs modify in the stock Firehose loaders.

Xiaomi wants authorization. Motorola doesn't let you read all areas.

Any device lets you unsolder the flash and read to your heart's content.

(This is not stated as a practical matter, just to make the possibilities clear.)

So, extracting a partition is easy-peasy.

Decrypting said data is intentionally hard/impossible, unless they put in an NSA "back door".

Yes, once you have the raw data, if it is FBE you can loop mount it.

That doesn't get you a lot closer to viewing your baby photos.
[General Questions and Answers] AOSP100:
7C 61 C2 C6 EE AE 8F 78 6C 8E 42 DD 9D 58 57 EA E2 4A 13 21 C3 5C 0E 7D BF 1D C5 10 80 AC 41 28 E6 F5 2F D7 DC F8 67 11 92 84 CB 2D 3D 57 E6 F9 50 5E CC 5B 88 87 36 B7 6E BA 05 FA 5B E0 64 26 66 15 BD 58 5D 0E A8 0B FC B6 EE 92 18 7F 20 78 A1 8B 07 90 79 B1 73 F0 30 4E 9E A4 09 7C 6C 3A 45 F5 15 52 80 31 34 87 18 FD 2A F6 20 88 8E 3A D9 44 6A D3 BD DF 25 ED ED 72 1C B0 16 0C 3F 40 47 A2 81 4B 6E A1 F3 01 2F 64 2A 2D A1 DC 4A 82 53 F9 C2 26 A1 1D 38 83 0A 59 B7 37 F1 48 76 22 65 E9 77 14 00 7F 14 E1 B6 85 D7 52 39 B8 5B 3F 2A EC 03 F6 EF 8A 69 BA FF D0 C1 27 1F 08 78 E6 FB 2C 1E A3 6F F1 4F 6F C8 78 D9 31 21 34 CB 83 BD F6 A3 BB 5B 54 31 81 94 E7 22 0F E1 F4 8C 19 B2 DC 07 40 52 F1 0E 96 96 BB 5E 92 98 22 4B 84 AD 56 FF 95 72 D9 BD 27 3C AF 7A CD C3 EB 8E E2 9A 4E 88 53 5E 5F AB BB F4 4F DC 2C 5B 73 59 74 0D 67 17 B5 6D 65 E7 D5 37 2C FF A5 2E 1F DC EE E8 35 12 AB 6B D9 A6 41 D3 8A 94 9A 5B 59 C4 9B FE 1F A2 B0 00 3E C0 90 FE A7 35 2C C0 10 C6 9D 2D EB 98 06 74 FD BD 82 25 3C 9E F3 77 84 AC 1F 54 3C D3 A3 93 DB A7 A6 21 90 AD A3 7D 9F 3E F2 28 BF 09 81 04 B7 F0 EB 73 13 8C B7 BB 45 9E 5E D9 BC 91 07 7F 78 AA 22 C5 A5 C8 D7 79 84 63 62 25 21 BD E7 B1 A4 F0 8D AC E6 E6 87 6A 9D A5 2E B6 F9 34 58 B6 C3 AB 76 E3 30 FB 2B 62 1C CA F4 DD D2 F0 52 91 A8 BC C3 8A E0 1B 84 55 73 7B 2B CD 3E A9 8F BA 4E 66 4F DC 32 4C 8A E5 B4 EA 0F 6E 13 6B AA 32 E7 C2 CC 68 5C 89 60 E7 97 80 9D 3A 07 A3 BD C5 19 CE A4 DE B2 14 15 95 0E EE C4 04 62 1B B5 2B 72 F2 C9 1E 16 E7 4F 20 31 B5 5B AA 8D 7A 34 A8 D9 38 31 95 94 80 BD 0B 23 88 2D 48 84 22 CE FA 89 11 D4 30 0C 06 EB E5 B2 A8 65 AD BA 23 C1 8A 16 D7 0F A1 AB 6F 10 02 FA 66 E9 05 FB B6 0A 23 64 89 D8 D8 6D 23 60 F2 A0 EA 87 4C C7 E5 20 D9 7E C4 AA 7F 18 A4 F5 7C 73 91 96 F3 62 F3 D0 1E 89 B4 32 36 C7 34 F8 E4 9B D4 B3 16 13 58 66 38 0E C3 94 AD 91 CF 42 42 59 F7 EB 1C E7 C6 5C 8F CF 3F 93 6D 5E 13 9A D2 16 B3 34 A5 E4 B2 44 BB 5A 01 3B D0 B6 DC A4 49 71 55 6C D2 8B 46 A2 9F 6B A9 5E 79 D7 B8 FD AA B8 0F 76 16 DA FA 76 35 0F AC F9 2A 9E 2A 86 60 C1 17 B0 01 D2 A1 55 57 08 9C BB 0B 69 B3 58 59 A6 8D EE 97 16 A6 5B 77 45 87 85 FA AD B1 7B BB D1 AB D4 18 66 AC 36 7B F2 1E C1 7B 1F 14 CC F4 CA 3A 4E 4B 38 B9 21 32 EC 01 27 04 4C 77 F7 AD AD F7 9A E6 9F D4 90 79 94 86 F9 2E B2 E6 53 04 4E AA 35 FD 3A C2 18 A2 4A 01 03 75 73 1F 6A 1B 81 FF 6B 54 5F F9 E2 4A 39 61 6A CE 94 02 65 6D 3C FD C5 48 55 BF AC BC C2 F5 83 9E 7A 86 78 F2 17 52 51 42 7A E3 C0 E9 41 D9 00 04 0F 1D C4 A6 47 37 77 C9 64 F8 70 8F E2 4F 8B FF 74 E5 BA B3 47 C9 47 9C 54 B8 B8 29 68 16 59 10 97 FB 01 74 97 58 CC CD C2 85 94 A1 59 29 8F 96 4C 2B 04 96 6E 6C 5A 65 61 CB 1F 52 63 51 48 DD BE AD CD B4 FC 27 D6 E0 F1 31 4F 32 57 44 8C 62 7F 71 DA C3 B1 EC 68 83 A7 09 09 A4 08 21 7E 24 90 3B FD B6 1B 3B EB 79 84 31 49 4C 57 4A 80 A9 D6 97 50 41 64 3E 31 58 71 5B 4E 72 0B 64 6C 58 5C 9A E6 2A 65 2A 76 04 01 B7 AE 11 5E 54 6C A9 59 99 58 0E CC FA 70 7C 20 D8 79 36 0D 73 13 29 08 64 32 20 4D 21 C9 A7 06 B6 3C 8E 5B 89 5D BB DC 17 90 F1 AD 2E FE 4F 93 D5 0D 17 7D 97 60 8B FE 92 C1 73 DE F6 19 3A 84 15 B3 A7 98 AF FF FE 0B 32 3B C5 64 58 A5
[General Questions and Answers] AOSP100:
It's F2FS, but running `file userdata.img` only returns "data".
[General Questions and Answers] Renate:
Mmm, that looks all random to me.
[General Questions and Answers] alecxs:
your partition image is twice encrypted (FDE + FBE) that's called metadata encryption (as you can see in fstab, keydirectory=/metadata/vold/metadata_encryption is needed for the FDE part)

my hope was you would see all zeros for plain f2fs file system (FBE encrypted partition image) but your hexdump isn't f2fs, it's just garbage without the keys. :(

you don't have a backup of metadata partition and you have factory reset the device, so the following applies here:
[General Questions and Answers] AOSP100:
Don't the keys change upon reboot? Also my output for dmctl table userdata is

adb shell dmctl table userdata

Targets in the device-mapper table for userdata:

0-229834712: default-key, AES-256-XTS - 8:15 0
[General Questions and Answers] Renate:
The keys are regenerated upon sign in.

That means that they take your PIN or pattern or a default salt and use it to generate a key which is used to generate a key which is, ah, you get it.

The exact process is above my pay grade.

The one thing that I do know is if I save and restore flash without wiping, then my data is still accessible.
[General Questions and Answers] AOSP100:
I remember reflashing vbmeta upon coming back from PostmarketOS, and it didn't break anything
[General Questions and Answers] Renate:
vbmeta has to do with integrity of boot images. nothing to do with userdata encryption.
[General Questions and Answers] AOSP100:
And that contradicts what some said about the only access to userdata being stored on vbmeta.
[General Questions and Answers] alecxs:
the crypto-footer for FDE is stored in metadata partition, so this part may survive a factory reset. but that's not the only access, FBE keystore is not in userland therefore no way to backup/restore encryption keys.
[General Questions and Answers] xXx yYy:
On all Samsung Galaxy devices shipping with Android 9.0 or higher and Knox 3.3 or higher, FBE protects files in the user data Flash partition: Each file is independently encrypted using AES-256-XTS, with a unique File Encryption Key that is derived from a Primary Key. In FBE, Primary Keys are randomly generated and protected by the TEE-based Keymaster component.

Most FBE keys reside within directories that are themselves encrypted by another FBE key. This creates a chain effect, making it harder to access any individual key.

More info here:

https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/file_based_encryption_enhancements_final_06.10.2019.pdf
[General Questions and Answers] alecxs:
The emphasis here is on "and Knox 3.3 or higher" - I have seen Samsung Galaxy devices that were shipped with FDE and Android 10. But we aren't talking about Samsung here.

TEE is not in userland. Keys are stored in Replay Protected Memory (RPMB) - no way to backup or restore.
[General Questions and Answers] Renate:
keymaster is the ELF executable that does stuff.

keystore (at least on one device of mine) is all zeroes.

Inside keymaster is the string "km_delete_all_keys_rpmb"

I haven't traced fastboot -w to km_delete_all_keys_rpmb (yet).

The only reference I can find is a 90 page thread on XDA https://xdaforums.com/t/xz1c-xz1-xz...o-backup-drm-keys-implemented.3795510/page-60

Isn't "Userland" that new supermarket chain? :p
[General Questions and Answers] alecxs:
are you referring to the keystore partition in "userland" (non-secure "normal" world)? that's a leftover of android 4 security I guess ;)
[General Questions and Answers] xXx yYy:
I'm aware of this: device this thread refers to is a Pixel 4a. :)
[General Questions and Answers] CXZa:
This thread like many similar before is about the current stupidity of using encryption even when it's unnecessary (like always..) that has spread like a plague.

Why relatively normal people need to encrypt anything really?

Encryption should not be the default...
[General Questions and Answers] xXx yYy:
Android encrypts files and disks for one major reason: security. Android's encryption features play a crucial role in safeguarding your data on the device.
[General Questions and Answers] CXZa:
Mostly more harm than good. Device lost. Who cares about your personal stuff really?

If street gangs want your more important stuff, they'll get it for sure...
[General Questions and Answers] xXx yYy:
AFAIK the layout of the memory on a Mediatek device is described in the Scatter file hence it should be easy to access and extract the desired region of eMMC with EDL tool.

Example of a Scatter file:

#Entry format: index, name, start_addr, size, type, checksum

0 preloader 0x00000000 0x00010000 DW_FLASH

1 dsp_bl 0x00010000 0x00020000 DW_FLASH

2 boot 0x00030000 0x00040000 DW_FLASH

3 recovery 0x00070000 0x00080000 DW_FLASH

4 android 0x000F0000 0x0C000000 DW_FLASH

... (other regions)

JOOC: How can this get accomplished on a QualComm device - the Pixel 4a is powered by the Qualcomm Snapdragon 730G SoC, will say where to derive the start_addr & size of the regions? Running command

cat /proc/mtd typically outputs something like this

dev: size erasesize name

mtd0: 0x00020000 0x00010000 boot

mtd1: 0x000F0000 0x00010000 recovery

mtd2: 0x0C000000 0x00020000 system

mtd3: 0x38000000 0x00020000 cache

mtd4: 0x00010000 0x00010000 misc

mtd5: 0x000F0000 0x00010000 userdata

and mentions an erasesize: Is this erasesize equivalent to size what is expected by EDL tool?
[General Questions and Answers] alecxs:
Maybe, but ChatGPT obviously not ;)

Not at all, no signed firehose programmer was leaked by Google (or Foxconn) yet. But maybe @Renate knows workaround? Google devices should kept bootloader unlocked, so you always can at least use fastboot and have TWRP.

ChatGPT error, must be emmc or ufs

Yes. according to the values, size means address (offset), erasesize means size.

never needed, all EDL tools used so far will read GUID Partition Table automatically
[General Questions and Answers] xXx yYy:
1. What is mentioned ChatGPT?

2. The command cat /proc/emmc likely won't work on a Qualcomm device: The /proc/emmc file - what allows to access information about the eMMC storage - is primarily found on devices using MTK chipsets.

AFAIK there actually isn't a file called /proc/ufs on Android devices, even for those using UFS storage.

3. What FastBoot command to use in order to get start address & size of a partition? Please enlighten me. TIA.

FYI: I don't make use of TWRP. :)
[General Questions and Answers] alecxs:
1. it's fine to use ChatGPT (or similar language model) for english translation (and polite to mention). but do not use it to generate content, this is considered spamming. most of your "AI" generated posts are borderline.

2. mtd is outdated, only ancient android devices used it. it's either emmc or ufs.

3. I don't know of any common cmd, fastboot commands are oem specific. some oems implemented to show partitions. you can calculate the offsets on size based on such output. try fastboot getvar all (afair it works for mediatek)

What I said is, fastboot is the only flash access to unbrick google devices, there is no way to unbrick on locked bootloader (not even google does it on RMA)

I know you never used TWRP or rooted a smartphone. if one have fastboot, one can boot TWRP, and with TWRP (or other custom recovery) one has raw access to all partitions.
[General Questions and Answers] Renate:
No, I haven't seen any Firehose loader for Pixel4. If you are looking for a loader for something, get the PK hash and look it up in this table.

Mmm, unpopular maybe, not often used often on user phones. My Netgear 4G modem uses MTD/UBI/NAND and I program it with Firehose.

Qualcomm has those program0.xml files. They are basically overblown, redundant, verbose XML renderings of a GPT. They (kind of) make sense in a factory settings, but for any other use I find them stupid. Just partition your device and then you don't have to be directly concerned what block vbmeta starts at.

"Scatter files" are the same thing except rendered as a table.
[General Questions and Answers] alecxs:
Android 6? :)
[General Questions and Answers] Renate:
Run an EDL client and get the PK hash.

edl.exe /l

edl.py
[General Questions and Answers] alecxs:
ask @xXx yYy he owns a Google Pixel 6a
[General Questions and Answers] xXx yYy:
My last 2 cents here:

By default the command adb pull /dev/block/by-name/userdata does backup the userdata partition in an encrypted manner. Take note that modern Android devices, including Pixel phones, encrypt the userdata partition by default. This encryption scrambles your data, making it inaccessible without the decryption key. Pulling the entire partition with adb just gives you a copy of the encrypted data.

Decrypting a backup obtained through adb pull /dev/block/by-name/userdata on a Pixel phone (or any modern Android device with default encryption) is highly impossible if the encryption key is absent. This key is what scrambles and unscrambles the data in the userdata partition. Without it, the pulled data remains a scrambled mess.

The encryption keys (CE-keys) used to scramble and unscramble your userdata are stored securely on the device's hardware. The userdata partition on your Android device remains encrypted even when booted into recovery mode. Reflashing the stock ROM / flashing a custom ROM doesn't affect these key unless you do a clean flash what wipes all data, including the encryption information associated with your userdata partition. You'll need to set up encryption again after flashing the ROM.

Because the CE-keys on modern Android devices mostly are stored in a special tamper-resistant hardware called TEE I don't think you can access it with EDL tools as propagated here. I'm pretty sure using EDL tools you can't tamper with the trusted execution environment in order to extract the CE-keys.

Worthwhile to know is the fact that the file headers of encrypted and unencrypted files on Android using FBE typically differ. To enable decryption later, the file header itself gets modified, means the modified header contains information or pointers required to decrypt the file's actual content.

Unencrypted files won't have this modified header. They will have a standard file header that allows the device to recognize the file type and access its content directly.

Here's some general information about what a file header might typically contain:

1. Standard Header Fields (Unencrypted):

Unencrypted files would likely have a standard header containing information like:

File type identifier (e.g., JPG for image, PDF for document)

File size

Access permissions

Creation and modification timestamps

2. Changes for Encryption:

When encrypted using FBE, some bytes in the header are be modified to include:

Encryption algorithm identifier

Initialization Vector (IV) used for encryption

File encryption key reference (likely not the entire key for security reasons)
[General Questions and Answers] alecxs:
of course not, nobody propagated that here.

For google phones, not even low level reading/writing in EDL mode is available, because of missing firehose programmer.
[General Questions and Answers] Renate:
Which is a bit surprising. One would thing some OEM would be truly open with this stuff.
[General Questions and Answers] alecxs:
Most if not all OEM try to restrict EDL access nowadays (as you said) - it's even more surprising that Google support isn't able to utilize EDL mode at all. Maybe that's kinda planned obsolescence?

https://xdaforums.com/t/is-an-unlocked-encrypted-phone-actually-secure.4536223/post-87936179
[General Questions and Answers] AOSP100:
Was there anything that I could've done? It suddenly stopped working when I flashed TWRP (that unofficial build), and TWRP didn't recognize it as a valid data partition.
[General Questions and Answers] Renate:
That's because TWRP can't decrypt it.

Wiping the data was what killed it.
[General Questions and Answers] AOSP100:
I mean before wiping the data
[General Questions and Answers] Renate:
That's because TWRP can't decrypt it. Ever.
[General Questions and Answers] alecxs:
actually, TWRP can decrypt, it just depends on the maintainer has implemented encryption or not. all types of encryption FDE, FBEv1, FBEv2, metadata, island/work profile are supported, except for Samsung.