yashsingh1/xdaforums huggingface dataset:
[Android Software Development] alecxs:
[MENTION=7718083]Spaceminer[/MENTION] i have no idea why you can't write back image to partition (remembers me of selinux but it should be permissive anyway), if you can't access /dev/block/bootdevice paths directly with dd (not the dm-0 path) then it's a good protection somewhere in the kernel, guess you have to disable and recompile it. maybe this link help you to figure it out, i have no clue how dm-verity works

https://gitlab.com/cryptsetup/cryptsetup/-/wikis/DMVerity

regarding modification: emmc image can be loop mounted from linux of course, win files can be extracted with sudo tar. repacking isn't that easy (i wrote a script for), but it's not necessary because if TWRP can write to vendor then why you shouldn't?

method of patching offline: if you collect all partitions into another tar, magisk manager can afaik handle this
[Android Software Development] Zackptg5:
Oh duh, I'll take a look at that. Magiskboot should patch all of that though but since you still can't mount vendor, that's making me think it doesn't

I can't download that rom btw, it keeps timing out and my internet is great at the moment so I'll try again later and see if the server gets its act together

Edit: Found the vendor stuff in dtb - nothing unusual though so magiskboot should've been able to patch it
[MENTION=433422]death_jax[/MENTION] can you flash the zip, then dd the boot img and send it over? (

dd if=/dev/block/by_name/boot_a of=/sdcard/boot.img

or whatever the path is)
[Android Software Development] Spaceminer:
No worries. That happens to me sometimes too. They're probably just doing maintenance or overloaded at the moment. The disabler does mount vendor correctly, even when twrp won't, so I know your scripts are doing their job. The changes do technically happen, they just don't stick. I can flash the disabler, immediately copy the fstab and look at it on a PC and see that the changes get made. But as soon as vendor gets remounted or I reboot, the changes are undone. It's mind blowing.
[Android Software Development] munchy_cool:
the recovery.log attached here

https://xdaforums.com/showpost.php?p=81952359&postcount=874

is one after flashing rom+gapps+dfe but before trying to boot into system.

if you want i can flash the rom+gapps+dfe again , try to boot into system , boot back into recovery and then get a log (but i think that wouldnt get us what we are looking for)

also, i was thinking of another check. once it boots into fastboot, i can try flashing stock boot.img from the rom and see if that fixes the issue. that should tell us, if its vendor modification causing the issue or the ramdisk one, right?
[Android Software Development] Zackptg5:
Log looks good. Ya, if you could grab the patched boot img before you do that and upload it here along with the stock one, that'd be great
[Android Software Development] munchy_cool:
flashing stock boot.img after flashing DFE and booting into fastboot, resulted in successful booting into system.

here is the zip file with both the boot.img, los boot.img is from the zip file, the one after flashing DFE was backed up using dd.

https://mega.nz/#!HQtg2YRQ!ECB3_e3OgK8Tb4TRL35Bp0RVbCb5fWEwyef_0_gfWPk
[Android Software Development] Zackptg5:
Looks like magiskboot messed up ramdisk.cpio somehow, I'll look more into it, thanks!

Edit: nevermind, I needed to run it as sudo on my pc, still looking into it
[Android Software Development] Spaceminer:
[MENTION=7285913]aIecxs[/MENTION]

I think you're on to something. I found these interesting little snippets about it.

"Android 4.4 and higher supports Verified Boot through the optional device-mapper-verity (dm-verity) kernel feature, which provides transparent integrity checking of block devices."

And also this...

"dm-verity protection lives in the kernel. So if rooting software compromises the system before the kernel comes up, it will retain that access."

I think we need a custom kernel, or to patch the current one somehow. Fun stuff... There's a verity_key file that sits in the system ramdisk. I tried removing it and the OS bootloops, so it must be enabled in the kernel. I think this also explains why twrp will bootloop if it's installed with the stock OS but not on a GSI. And it's probably why calling and audio don't work right on them either. It's an untrusted system, so I'm guessing the bootloader locks down partitions. (Vendor libs don't load, etc.) I'm curious if it changes the bootstate on a GSI. getprop says my verified state is Green, but if I look deep inside /sys/devices (I forget where exactly.) it says my state is actually Orange. (Bootloader still says green so maybe there's more than one boot state for different parts of the OS?)

The takeaway I'm getting is that, Motorola did the bare minimum to comply with treble requirements (can boot a GSI for testing & has A/B...), while still trying to make it completely unusable to anyone else but themselves. (Eg. Nothing proprietary works right without verity_keys they made.) I'm sure that we can make our own keys, but I have a hard time believing they'll work. I'm going to try it anyways though.

Edit: It looks like FEC, or "Forward Error Correction" is the culprit. FEC is a feature of dm-verity that allows corrupted blocks to be restored to their original state. So if we make changes to vendor, FEC checks against it, sees an error, then reverses it. It's kinda like ECC RAM but for block devices. Did I buy a Motorola or a Samsung? lmao
[Android Software Development] Spaceminer:
FEC confirmed.

[MENTION=6037748]Zackptg5[/MENTION]

I think we have a winner. FEC stops us from changing the fstab. It's definitely enabled in our kernel. (Verity is too.) Google has almost no documentation on it, as is tradition... I managed to dump this from init. strings -n64 gives us some very readable information.

[COLOR="darkred"]%s %u %s use_fec_from_device %s fec_start %llu fec_blocks %llu fec_roots %u ignore_zero_blocks
%s %u use_fec_from_device %s fec_start %llu fec_blocks %llu fec_roots %u ignore_zero_blocks[/COLOR]
<3>fec: <%d> error: %s: failed to open '%s': (unsupported mode %d)
<3>fec: <%d> error: %s: [%llu, %llu): corrupted block %llu (offset %llu) cannot be recovered
<3>fec: <%d> error: %s: RS block %llu: decoding failed (%d erasures)
[COLOR="darkred"]<3>fec: <%d> error: %s: cannot update verity magic: read-only handle
<3>fec: <%d> error: %s: cannot update verity magic: no metadata found
<3>fec: <%d> error: %s: failed to update verity magic to %08x: %s
<3>fec: <%d> error: %s: invalid verity table size: %u; expected [(10 * 2), ((8 * 4096) - sizeof(verity_header)))
<3>fec: <%d> error: %s: invalid number of verity data blocks: %s
<3>fec: <%d> error: %s: not enough arguments in verity table: %d; expected at least 10
v->hash_data_offset <= UINT64_MAX - (v->hash_data_blocks * FEC_BLOCKSIZE)
v->hash_data_offset + v->hash_data_blocks * FEC_BLOCKSIZE <= f->data_size[/COLOR]
<3>fec: <%d> error: %s: invalid hash tree: hash_offset %llu, data_offset %llu, block %u
<3>fec: <%d> error: %s: failed to read hash tree: offset %llu: %s
<3>fec: <%d> error: %s: failed to read hash tree: data_offset %llu: %s
([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*)@([0-9]+)[.]([0-9]+)::([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*)
@([0-9]+)[.]([0-9]+)::([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*)
([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*)@([0-9]+)[.]([0-9]+)
([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*)@([0-9]+)[.]([0-9]+)::([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*):([a-zA-Z_][a-zA-Z_0-9]*)
@([0-9]+)[.]([0-9]+)::([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*):([a-zA-Z_][a-zA-Z_0-9]*)
([a-zA-Z_][a-zA-Z_0-9]*(?:[.][a-zA-Z_][a-zA-Z_0-9]*)*):([a-zA-Z_][a-zA-Z_0-9]*)
[Android Software Development] Zackptg5:
The thing is, the zip should disable verity on the kernel. If you could send me the boot img to take a look it, that'd be great. Also, which twrp are you on?
[Android Software Development] Spaceminer:
Twrp version is 3.3.1-0

boot.img

magisk_patched.img

After I patch our boot/recovery image, the init inside definitely gets modified by magisk, and I can see the backup folder inside the ramdisk so everything looks good. When I flash it I am rooted, but the init found in the root directory, isn't the one that got patched, it's still the original. I don't know if it's the system or recovery init. But I'm guessing it lives on system only because I can edit the rc files there with a root explorer and it sticks on reboot. The init found in root also appears to be the exact same one we find in the stock boot/recovery. So it's a bit confusing for me.

FEC I think is definitely a problem as far disabling encryption goes. The info regarding it that seems useful is...

"The new error correction feature we developed for dm-verity makes it possible for devices to recover from the loss of up to 16-24 MiB of consecutive blocks anywhere on a typical 2-3 GiB on a system partition"

&

"Of course, if the encoding data itself gets corrupted or we lose more than two of the blocks covered by any single code, we cannot recover anymore."

Source: strictly enforced verified boot with FEC

If I read the math on that page correctly, this is linear and ends up being ~256mb of data loss recovery for a device with 32gb of storage. My entire vendor partition is only 227mb, so it could easily restore the whole damn thing. And oem is a cake walk at only 36mb. The 2nd bit up there might be exploitable. This might also be why some people need to format data 2x after installing twrp. I've also noticed that if I uninstall magisk, half my bloat comes back but not all of it. I think that's related to magisk mirrors and FEC block recovery.

2. FEC is susceptible to symlinks... <--- That I'm sure is very useful. I've done some testing with this already. I also found a spot inside /vendor that we can actually write to permanently. /vendor/firmware_mnt

Doing so however, will trigger verity, which in turn corrupts the data partition and requires a format afterwards. Unfortunately symlinks don't stick here, anything else will though.

I suspect .rc files can be used to symlink to the fstab before the mount_all command happens. This would replace it with our dummy fstab that says encryptable before boot completes. The best method in my opinion is probably patching FEC out of the kernel entirely if it's possible. I'm going to try both.
[Android Software Development] Spaceminer:
[MENTION=6037748]Zackptg5[/MENTION]

I've made some interesting discoveries.

It's actually the magisk mirrors that are causing the issues. Magisk needs a system root method to fix it. Systemless simply will not work on our device be it GSI or stock. It breaks twrp on either or simply doesn't root the device. My modded phh works at the system level and does grant 100% root access to all partitions including vendor. All changes are also permanent. Twrp is also working correctly as long as it's installed to the ramdisk, and not rooted with magisk. I'm currently running LOS 16 GSI with phh root, and your disabler worked without a hitch. Vendor fstab says encryptable=ice, ro.boot.verifiedbootstate=orange, and ro.boot.veritymode= eio so it's toast! I guess I need to port phh root to stock... I appreciate all your help!
[Android Software Development] Zackptg5:
I remember topjohnwu talking about needing a system root method a while ago. It's on his to do list but no telling when it'll be implemented. Thanks for the info!
[Android Software Development] death_jax:
I attached:

dmesg+recovery logs catched after installations of Magisk 20.3 and, at the last, your zip file; but without success of boot into Android 10 :(

Link of my boot stock images of Pie and Q (there are many differences between them, as there are many differences in the vendor partitions)
[Android Software Development] Zackptg5:
So all your vendor partition information appears to be in the dtbs in your boot img. The recovery log also showed that the zip didnt' throw any errors and seemingly did what it was supposed to. You'll have to play around with it and magisk but I didn't see any errors on my end
[Android Software Development] JazzMartian:
[MENTION=6037748]Zackptg5[/MENTION]

How might this affect my device as far as init.d launching boot scripts is concerned, since I flash your universal init.d injector zip (which modifies the boot image) before I flash Universal dm-verity, which also modifies the boot image?

This question is particularly of relevance since I have been having issues with universal init.d injector and it seemingly not running boot scripts. I installed the 00test script to the init.d folder, but never found the test.log file that's supposed to appear in the data folder after boot. I'm wondering if universal dm-verity's boot modification renders universal init.d injector useless?
[Android Software Development] Zackptg5:
It shouldn't have any effect on that. I'm surprised people are still using it honestly, I'm pretty much magisk only at this point although I try to make some stuff rootless compatible
[Android Software Development] JazzMartian:
The device is an Nvidia Shield Portable which doesn't have init.d support built in, and the highest system software that was released was Lollipop. I'm sure it makes more sense why I'd be using considering that. osm0sis' init.d solution returns an error whenever I flash it. I know your solution is deprecated, so I don't want to bother you for help with it. But I can't get things working with your solution, it's there other init.d boot script solutions you'd recommend? I'd rather not use an app for it, and am looking for something that I can use to modify the system software to support init.d boot scripts if possible. Thanks!
[Android Software Development] death_jax:
I noticed that my vendor mount point of Android 10 is placed in ramdisk/etc/recovery.fstab, but not in Android 9 :rolleyes:

I also noticed that in my Nokia 7 Plus, in /vendor/etc/fstab.qcom, there is fileencryption=ice that I changed in encryptable (and not encryptable=ice) but, after flashing your zip, fileencryption=ice wasn't replaced correctly.

PS: in your experience, is possible that some applications can cause bootloop because are made for Android 9 but not for Android 10? And how can I investigate about this?
[Android Software Development] Zackptg5:
I don't know of any newer init.d solutions. Why not just use magisk? It's compatible back to jellybean

the recovery.fstab is used for recovery, not booted systems unless yours is somehow different.

You can't change the filencryption entry to encryptable or anything else because they're completely different methods of encryption so you just have to remove it (fileencryption=ice -> =ice):

fileencryption = FBE - file based encryption

everything else = FDE - full disk encryption

On android 7 - 9, many devices used FDE where you could make the encryption optional with the encryptable flag. This is no longer the case as Q is FBE only - either you got it or you don't

See here for more details: https://source.android.com/security/encryption/
[Android Software Development] JazzMartian:
Well, I've grown used to the way things work on the Shield Portable with SuperSu and like it. I'm probably going to be upgrading my phone though within the next few months and have decided to install Magisk on it once I do. I really like the variety of modules on Magisk and there's a chance once I start getting into that scene that I might also install it on my Shield Portable.

Thanks for the help and for your contributions to the scene! I'm a fan of your work.
[Android Software Development] death_jax:
@Zackptg5

As you said, your zip works well...but, in my case, it was necessary to format data (format, not wipe) (although I was already not encrypted), rebooting to TWRP and then doing all steps described in OP again (for the not encrypted and, in my case, for the A/B devices too), but I didn't wipe internal storage.

But, after reboot from TWRP, the phone rebooted itself again after android one logo but before the Nokia logo...after system completed the boot I understood that in the above double reboots it automatically formatted my internal storage too :( .

But...I'm on Android 10 now, without encryption and dm-verity :)

PS: I hope that it will not format data/internal storage on every update!! :eek:
[Android Software Development] chickendeodexed:
mirrors.lolinet.com/firmware/moto/nora/official/RETIN/XT1944-5_NORA_ROW_RETIN_8.0.0_OPPS27.91-176-11-9_cid50_subsidy-DEFAULT_regulatory-DEFAULT_CFC.xml.zip

No log because unable to save, it says saved but when i navigate to directory it was empty.

Stuck at full black screen "N/A" written in upper-top-left corner. Waited for 1 hour each for following two cases. In case 2 the N/A screen reboots or blinks i think twice before staying there.

Case 1- Flash full firmware. Boot TWRP https://www.androidfilehost.com/?w=files&flid=294390 (NORA https://xdaforums.com/moto-e5/development/recovery-twrp-3-2-3-x-moto-e5-plus-t3912189). Install Disable_Dm-Verity_ForceEncrypt_03.04.2020.zip. Reboot to system from inside TWRP.

Case 2- Flash full firmware. Boot into System. Power off. Boot TWRP https://www.androidfilehost.com/?w=files&flid=294390 (NORA https://xdaforums.com/moto-e5/development/recovery-twrp-3-2-3-x-moto-e5-plus-t3912189). Format Data. Install Disable_Dm-Verity_ForceEncrypt_03.04.2020.zip. Reboot to system from inside TWRP.
[Android Software Development] ocgw8ost9a:
Hi. Sorry for asking a noob question, but I didn't know what to search for and 90 pages is very much to read, even in times of social distancing.

I want to have TWRP and Magisk on my unlocked Moto G6 (keeping stock for now). So I flashbooted into TWRP formatted data and rebooted from TWRP into discovery. The phone then says "no command" (this phrase is guessed, as it is written in German). Nothing more happens for a while, after which it reboots normally, encrypts and I am, where I have started. What do I have to do differently?

# removed question already answered #
[Android Software Development] chooks06:
I apologise for not being able to provide a solution.

But I just want to assure you that your thoughts in last question is correct.

The .zip is to be flashed via custom recovery. Fastboot flashes .img.
[Android Software Development] ocgw8ost9a:
Oh, yes. Now I remember. You do a e.g. "adb push" followed by "install" in TWRP.
[Android Software Development] chooks06:
Cheers!
[Android Software Development] ocgw8ost9a:
I figured out, by holding power and then pressing volume up, I enter Android Recovery (motorola/ali_retail/ali). But I cannot get to TWRP again. So I re-flashed TWRP and was able to follow the instruction in this thread https://xdaforums.com/moto-g6/how-to/guide-faster-updated-guide-to-root-t3914228. But now I am stuck at the bootloader (https://xdaforums.com/showpost.php?p=82141057&postcount=98)
[Android Software Development] Rhino42:
if i flash disable force encrypt, and at some point decide, that i want to encrypt my phone again, will that just work (from the android settings just kick on the encryption process), or do I have to roll back via flashing a enabling force encrypt zip or sth. like this? thx in advance!
[Android Software Development] alecxs:
there is no way back. make backup of boot and vendor before
[Android Software Development] TTTT555:
fstab.mt6785

before:

/dev/block/platform/bootdevice/by-name/userdata /data ext4 noatime,nosuid,nodev,noauto_da_alloc,errors=panic wait,check,formattable,quota,resize,reservedsize=128m,fileencryption=aes-256-xts

after:

/dev/block/platform/bootdevice/by-name/userdata /data ext4 noatime,nosuid,nodev,noauto_da_alloc,errors=panic wait,check,formattable,quota,resize,reservedsize=128m,=aes-256-xts
[Android Software Development] fabrizio b92:
Questions guys:

1)Works on htc u12+ with pie stock, twrp and magisk 20.4?

2)instruction flash:Backup(optional), (custom)rom, magisk and last this script right?

3)after flashing this script it could create problems with NFC payment(Googlepay)?

I use several time my smartphone to pay via contactless and i don't want give up paying in this way

Thanks a lot[
[Android Software Development] icecandle0:
After formatting data and rebooting into recovery, it boots to fastboot and I can't do anything after. I think I am not understanding something important. What should I do?
[Android Software Development] amg314:
Some phones actually respond to

fastboot reboot recovery

If you have a copy of the recovery image on your computer, running command

fastboot boot your_TWRP.img

should do the trick.

https://android.stackexchange.com/questions/201212/how-can-i-reboot-into-recovery-from-fastboot
[Android Software Development] death_jax:
Which phone have you? Do you know if you have boot partition separate from recovery partition?

Inviato dal mio Nokia 7 plus utilizzando Tapatalk
[Android Software Development] icecandle0:
OP6 is the phone I'm using
[Android Software Development] death_jax:
Time ago I just used what is in this guide (but for my phone the fastboot reboot recovery command doesn't work)

Just downloaded the right TWRP for my phone and Android version and booted in it with the:

fastboot boot /path/of/TWRP.img

Then the phone reboot itself into it, because that TWRP must be able to reboot itself into recovery after that command, (fo my phone in fact there is a retention zip in order to install the right TWRP version inside booted TWRP)

Inviato dal mio Nokia 7 plus utilizzando Tapatalk
[Android Software Development] amg314:
You can also run command:

fastboot flash recovery your_twrp_name.img

(your_twrp_name.img and fastboot should be in the same folder)
[Android Software Development] death_jax:
No amg314, for my phone (and similar) isn't so simple...

As indicated here

https://android.stackexchange.com/q...rite-to-recovery-using-fastboot-on-a-b-device, there are two type of devices:

- A/B devices that have both boot and recovery in the same (boot) partition

- A only devices that could have separate boot and recovery partitions

Mine, a Nokia 7 Plus (and similar), is of the first type.

So I cannot always flash directly a boot img, because some boot img can cause bootloop, wifi problem, ecc., nor I cannot directly flash recovery img, because my phone haven't a dedicated partition.

So, after some developments, for my phone (and similar) it's necessary to first fastboot boot into a dedicated type of TWRP img and then, once there, it's possible to install a dedicated TWRP zip that effectively installs the TWRP (injecting it into right position, replacing only recovery part, without alter the boot one)

Seems that the OnePlus 6 is of the first type like mine, as described in the guide here:

https://twrp.me/oneplus/oneplus6.html

Inviato dal mio Nokia 7 plus utilizzando Tapatalk
[Android Software Development] amg314:
Android is getting harder, but not better, and not user friendly.
[Android Software Development] death_jax:
Eeeh, you are so right [emoji22]....

Inviato dal mio Nokia 7 plus utilizzando Tapatalk
[Android Software Development] Zackptg5:
It's more complicated then that lol. Google recommends things but has little power over what oems do. Then they completely reverse changes they make year after year. Fragmentation is going to be the death of android
[Android Software Development] death_jax:
I hate fragmentation [emoji2959]

For you, why OEMs causes what you wrote?

Inviato dal mio Nokia 7 plus utilizzando Tapatalk
[Android Software Development] Henry Markle:
Hi

I switched from Moto g4 plus to Xiaomi redmi note 8, and I'm about to unlock the bootloader to install custom roms, but then I saw this thread and now I'm confused..

It seems that there's a problem with this device, because I never had to worry about things like this with my previous device (athene).. what is dm-verity? Why exactly do I need to flash this zip and when? And do I need this every time I flash a new custom rom, or only when I go from stock??