yashsingh1/xdaforums huggingface dataset:
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
When the ROM is created, all parts are set at the same time. Verity, error correction, signatures, etc.

When you try to change that, something breaks or gets reverted by the error correction.

I think decryption can be an issue if you change verity keys, such as going from official to homebrewed ROMs. The signing keys are different and will change the master key for encryption, which is built from several sources, including secret bits from the trust zone. Note, this master key is different than the pin/pattern which encrypts the master key so that it is only usable to those that know the pin/pattern.

I think this is why if you install official los with stock recovery, set up the phone, then install twrp, decryption will break... maybe. It's been a while since I tested this.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] jesslegentil:
How would you install official LOS using stock recovery?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
I think most people would fastboot boot the twrp installer and then install from there.

I'm my earlier post, I meant install los and keep the recovery that comes with it.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] rczrider:
Can you provide a link to your build? I'm guessing it's in this thread, but I went back through and didn't find it. I probably missed it.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] jhedfors:
See here...
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] rczrider:
Oh, I thought it was made / uploaded by SyberHexen and so didn't think the one provided by d4rk3 was the same thing.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gleenfield97:
I just tried the official 3.3.0-1. Decryption (password) works, but no mtp (no Enable/Disable MTP) and adb shell. adb sideload is there, but I haven't test.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] lucasrt007:
great

thanks, more than one tutorial that says flash recovery :rolleyes:
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] jerryspring:
can this (payton) also be flashed onto a payton_sprout? i have already unlocked the bootloader. thanks
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
Heyyo, I am one of the co-maintainers of TWRP for nash and payton's TWRP trees are very similar, so I thought I'd help out here since it seems payton's TWRP trees are quite behind all the improvements that we have on nash.

Please note! I do NOT own a payton, so I can't test these builds I'm making. If you guys wish to test these? Please only use the following command:

fastboot boot twrp-3.3.1-test1-payton-UNOFFICIAL-20200415.img

https://osdn.net/projects/twrp-for-payton/releases/72764

There's a lot of changes that were made. All the commits since Jul 29, 2019 are the ones that I've cherry-picked and modified from nash to be compatible to payton. :good:

https://github.com/ThEMarD/android_device_motorola_payton-twrp/commits/c657c745bdb7d966c8e823713afc6e6cb667f9ff

Please test EVERYTHING.

Flashing ROM

Flashing GApps

decryption

TWRP Backup

TWRP Restore

adb push or sideload

Any issues? Please copy log and kernel log (Settings > Copy Log > Check "copy kernel log" > swipe > attach the files found in /sdcard/

I might not check this forum section, so if you have a reply for me then please mention or quote me (but only one or the other as both will annoyingly give me double notifications for the same post). Thank you.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] big_edd:
This did not boot to recovery, but hung at the boot logo for a while then booted to system.

Lost root.

Rebooted to installed recovery, but could not mount system.

All reboots from then on were bootloops to boot logo only, regardless of active slot.

Ended up dirty flashing same rom over again, so no big deal in the end.

I'm sure the logs are long gone by now, but I'm willing to help if I can.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
Heyyo [MENTION=2997136]big_edd[/MENTION], thanks for the report.. ok, maybe I went a little too overboard with my changes lol so let's dial things back a little and see what we can cherry-pick to payton from nash. :good:

twrp-3.3.1-test2-payton. Just like before, I only recommend doing fastboot boot twrp-3.3.1-test2-payton.img as I cannot guarantee this will work since I can't test it myself...

https://osdn.net/projects/twrp-for-payton/releases/p16788

Github commit history is up to here

https://github.com/ThEMarD/android_device_motorola_payton-twrp/commits/b6b77fb5e74d7628a2ed36c6acfe8ebb53c2ebaf

For anyone who didn't read my last post I'll copy/paste here too...

Please test EVERYTHING.

Flashing ROM

Flashing GApps

decryption

TWRP Backup

TWRP Restore

adb push or sideload

Any issues? Please copy log and kernel log (Settings > Copy Log > Check "copy kernel log" > swipe > attach the files found in /sdcard/

I might not check this forum section, so if you have a reply for me then please mention or quote me (but only one or the other as both will annoyingly give me double notifications for the same post). Thank you.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] big_edd:
Tested OK so far:

Decrypt data

ADB push/pull

TWRP backup & restore

TWRP backup was to internal storage. test2 failed to mount the microSD card.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
Thanks for testing it out [MENTION=2997136]big_edd[/MENTION]. Can you please also try flashing a ROM? It can even be the same version that you currently have installed is fine. Not sure why USB OTG wouldn't work as I didn't even touch that, or was it always not working?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] big_edd:
I prefer to flash roms from the microSD card.

This time I booted into test2 recovery only to attempt the mount.

I repeated this again with the installed recovery (TWRP-payton-3.3.1-Syber-v2.2_11-02). Logs from both instances attached.

I have done a backup to a USB SSD in the past also.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
Heyyo, thanks for the information [MENTION=2997136]big_edd[/MENTION], it looks like the fstab for the mount points of USB OTG and MicroSD Card are incorrect in the TWRP tree for payton hmm... looking through the LineageOS payton tree and also a dump of 9 PPW29.69-40-4 4ca2a of Payton's stock ROM? We indeed need to fix this.

twrp-3.3.1-test3-payton test build is going up! Hopefully this fixes both USB OTC and MicroSD Card mounting/unmounting.

https://osdn.net/projects/twrp-for-payton/releases/72782

Built up to here:

https://github.com/ThEMarD/android_device_motorola_payton-twrp/tree/8cfd5bca6473347e732a01c1f446fe19b26d28ca

If mounting USB OTG or MicroSD Card do not work? Please reboot to your currently installed ROM and please get me the results of this command...

ls -l /dev/block/platform/soc/c0c4000.sdhci/

For anyone who didn't read my last post I'll copy/paste here too...

Please test EVERYTHING.

Flashing ROM

Flashing GApps

decryption

TWRP Backup

TWRP Restore

adb push or sideload

Any issues? Please copy log and kernel log.

Settings > Copy Log > Check "copy kernel log" > swipe > attach the files found in /sdcard/

I might not check this forum section, so if you have a reply for me then please mention or quote me (but only one or the other as both will annoyingly give me double notifications for the same post). Thank you.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] big_edd:
[MENTION=6235294]ThE_MarD[/MENTION]

I did a clean flash yesterday with no issues.

I grabbed the block layout in case you needed it anyway.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] jhedfors:
Thank you for working on this! I like that we have MTP access and can change between slots A & B!

f I want to flash this TWRP to my device, is there a way to do it without a flashable zip, just using the IMG?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
[MENTION=2997136]big_edd[/MENTION] just to confirm you were able to flash ROMs, GApps, TWRP backup, TWRP restore, adb sideload/push, mount/unmount MicroSD Card and mount/unmount USB OTG?

[MENTION=2091002]jhedfors[/MENTION], please if you could also confirm all the above also works on your Payton.

As for installing it? Once fastboot boot into the image you should be able to go...

Advanced

Install recovery to ramdisk (or something like that)

select the TWRP img
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] big_edd:
All of that + Magisk. Everything worked but the usb otg. It would not mount.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] jhedfors:
I tried these steps, and while I did not recieve any errors, when i rebooted the recovery I still had the previous recovery. I am testing the backup/restore now....

BTW, I am not encrypted.

Edit: No error with backup, but when I attempted to restore, it gave an error 255 when trying to restore persist. I have attached the dmesg.log and recovery.log(zip)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
[MENTION=2997136]big_edd[/MENTION], I'm glad the rest of the TWRP build works good! :)

Well? One other idea I had is maybe the kernel driver for USB doesn't like hyphens in the mounting location's filename? Let's try with an underscore instead.

twrp-3.3.1-test4-payton is going up to try and fix the USB OTG mount/unmount again

https://osdn.net/projects/twrp-for-payton/releases/72791

Built up to this point:

https://github.com/ThEMarD/android_device_motorola_payton-twrp/tree/8cfd5bca6473347e732a01c1f446fe19b26d28ca

[MENTION=2091002]jhedfors[/MENTION], checking out your logs? It does seem like the restore everything was going fine and even the majority of the persist partition was restored except for this one file where it had an error and failed out

tar_extract_file(): failed to extract /persist//cache/recovery/log !!!
I:Unable to extract tar archive '/external_sd/TWRP/BACKUPS/ZY224JWHPL/2020-04-19--15-15-08_QQ2A200405005/persist.ext4.win'

Is there a chance you can copy your backup to your main storage and then try to restore it from there, please? Sometimes TWRP is a bit finicky with MicroSD Cards and USB OTG for Backup and Restore.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] xeropressence:
When making a backup, should we be backing up system, system image, or both? Same for vendor, vendor image?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
[MENTION=9086998]xeropressence[/MENTION] I recommend doing image backups for system and vendor, that way it's a perfect backup of those partitions. Especially handy if you're switching ROMs or upgrading/downgrading ROMs as it will ensure the ROM from the backup is properly restored.

I always do TWRP backups of boot, data, system image and vendor image.

I always keep on my PC a backup of persist and firmware, just in case I need to rescue those two partitions.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] xeropressence:
Tried the test4, was unable to mount my usb-c sdcard reader that works normally on a regular rom.

I've attached the logs
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] big_edd:
[MENTION=6235294]ThE_MarD[/MENTION]

Same report as xeropressence but with a usb ssd.

Here are the mounts from my installed recovery with the ssd mounted:
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
Heyyo [MENTION=2997136]big_edd[/MENTION], great find! yeah TWRP was trying to mount it from sdg and not sda... so I'm guessing we definitely need to correct that.

twrp-3.3.1-test5-nash is going up! Let's try USB OTG on sda1/sda and see what happens. :good:

https://osdn.net/projects/twrp-for-payton/releases/72810

[MENTION=9086998]xeropressence[/MENTION] if you would also please test out the test5 build as well.

It is built to this point

https://github.com/ThEMarD/android_device_motorola_payton-twrp/tree/91456d7de5d5fc86fabb4af3fd6b92351fcd0281
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] xeropressence:
Same thing, doesn't mount.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] xeropressence:
Also internal microsd was working in test4 as well.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] jhedfors:
Using test5, I tested restoring "persist" from internal storage, and got the same error as before.

Also, while I was able to install TWRP to the device, it would not boot, and kept returning to TWRP.

Edit: I am guessing it is an issue with the installer functionality as when installing the previous recovery in the same manner, I also could not boot - had to restore the TWRP backup of the boot.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] SyberHexen:
Thanks for the love on Payton twrp.

Last I worked on twrp on this device was back in October.

Looks like I forgot to push anything relevant to gerrit, oops!

My custom build here doesn't have mount or encryption issues.

Maybe something relevant to your work

https://github.com/SyberHexen/TWRP_android_device_motorola_payton/commits/TWRP-9.0
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
The simple explanation is that we have this (below) Qualcomm decryption scheme in the kernel, and need to find exactly what proprietary files TWRP needs in order to use it.

I have started to bring up the vendor tree, but TWRP is still not decrypting. It may have something to do with TWRP also failing to mount most partitions... or it may be the cause of that. I am starting to believe the latter, as I seem to have a similar tree to other devices I have compared to. So perhaps will focus on these errors today. If I can mount by name, may try mounting by location (a temporary solution as this changes for most partitions with A/B slot). Perhaps whatever fixes mounting will also cause decrypt to work... lot of guessing right now.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Or maybe I am barking up the wrong tree. Making TWRP aware of vold was how I decrypted /system successfully...and logcats show that this is how /data is decrypted on a full boot, too!

Compiling with this decrypts /system:

TW_CRYPTO_USE_SYSTEM_VOLD := true

Is there an equivalent for /data???
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mrkh:
No hands-on experience with this yet, but searching for TW_CRYPTO_USE_SYSTEM_VOLD seems to turn up a slightly different version of that config line:

TW_INCLUDE_CRYPTO := true
TW_CRYPTO_USE_SYSTEM_VOLD := qseecomd

I don't know if you've tried this already? There's some explanation here.

It also mentions a debug config line that may give you some more info:

TW_CRYPTO_SYSTEM_VOLD_DEBUG := true
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
This looks good... will review and compile with it for my next test
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
So, confirmed file-by-file that these commits are in my source. A quick addition of the qseecomd flag to BoardConfig didn't solve /data but have more logs now.

Will need some work on init.c I think, and I need to get other partitions mounted

---

Oh, snap... logs are all empty and say "Error: File not found". I'd say something broke on that build :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
You probably already know this, but did you see this on twrp.me?

"The initial version of the app does not support flashing on the Pixel due to the Pixel's A/B partition layout. In addition, the app may not support finding and flashing images from external storage locations. We are working hard to improve these items and bring you more new features."
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
No, but TWRP seems fine as far as A/B goes... Flashes from sdcard fine, backs up to sdcard fine. Just need to decrypt.

If there is anyone set up to build and help get this tree up, please chime in...my actual time in front of a computer is extremely limited, so this is taking a lot longer than it needs to :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
I'm going through the motions to catch up to where you are. Currently I am trying to get the TWRP to build. Once I get there, I'll be able to help test.

---------- Post added at 10:08 PM ---------- Previous post was at 10:01 PM ----------

I think the A/B partition creates a problem with gapps- I think the A/B partition scheme forces you to distribute gapps with the rom, which is problematic and is likely the issue that is holding up pixel development.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
If you want to fork it, I will try to clean up my device tree... it is kind of a train wreck after testing so many things. :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
I cloned it and I was surprised to see goldfish errors and dependencies. I think goldfish is the emulator, so there are probably changes that you haven't pushed back yet.

I'll fork it tonight, regardless if you get a chance to clean it up. I don't have much experience with bringing up the device tree so it'll be a good learning opportunity.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Yeah, I started out in the emulator and it is still set up that way... I need to remove that, but if you want to test as-is, just clone the goldfish repo to satisfy the dependencies. (actually, I don't think there are any emulator specific things left in this tree... it may be in your local env)

I also set it up to use a prebuilt kernel just to cut down the time to compile test changes... in the long run will need to flip it back to building with kernel source as there may be some kernel changes for the final product.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Thinking about these mounting errors...need to test with "ignoreblkid" flags in fstab.

Some other devices have relied on certain partitions being mounted in order to decrypt... so it is possible that solving this will help solve encryption.

Going to shift to attacking this instead of directly working on encryption.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
FYI [MENTION=2790198]gee one[/MENTION] this ^^ breaks normal recovery/kernel logging as well as logcat. I have not added strace to my build yet anyway, so will return to this after working on partitions.

And, the emulator thing got me searching... and somehow TWRP is being built with ro.build.characteristics=emulator still. No idea where that is coming from, unless I screwed up the kernel and am inheriting it from there.

---

Tried adding default.prop to /recovery/root and setting it to "default" from there, but TWRP logs still show emulator. Strange. Not a big deal, but shows how quickly a build environment can spiral out of control when trying to track down issues :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
Are you using "fastboot boot" to boot into recovery or are you flashing the recovery and using the bootloader to get into recovery?

I can build and use the pre-built kernel, but if I use "fastboot boot" it takes me into the OS, but using the kernel and recovery.img. When I fastboot boot into the recovery.img, I am no longer rooted. If I reboot, I am rooted again because it is booting from the boot_a partition.

The A/B partition scheme is different- see this bit- we might have to cherrypick the commits below:

https://source.android.com/devices/tech/ota/ab_implement#kernel

From what I can tell, the recovery is built into the boot partition and the bootloader skips the initramfs to boot into recovery. See my earlier question about if you are flashing or booting.

https://source.android.com/devices/tech/ota/ab_implement#recovery

I'm not sure how twrp handles the A/B partitions. It has the code to do it, but the output of the mkbootimg command is boot.img. I think "make recoveryimage" is the way to build it.

I forked your repo and I will push back the changes as soon as I remember my github credentials. Some notes:

* payton.mk is not called. omni_payton.mk is being used, so changes should go in there.

* as far as I know, twrp does something funny with the fstab. On boot, it takes etc/twrp.fstab and copies it to recovery.fstab. Maybe the TARGET_RECOVERY_FSTAB flag takes care of that now.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I am fastboot FLASHING a full boot image with TWRP on board. I have been using "make recoveryimage" to generate the recovery ramdisk during testing -- didn't see much point in fixing the mkbootimg process until TWRP was working (and maybe not even then).

So my general process to test a change is:

1. make clean/envsetup/lunch/make recoveryimage

2. take resulting ramdisk in recovery.img and repack with stock split_img.

3. Fastboot flash resulting img to both boot_a and boot_b.

4. Boot TWRP.

5. Optionally, fasboot BOOT the root method (working fine alongside TWRP)

I tested with a separate twrp.fstab but ended up just twrpifying root/recovery.fstab and pointing there...no point in keeping two until one is working :)

I guess payton.mk is just a remnant from early testing...
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
OK, I have a bootable recovery! I guess I'll start testing and try to help get this working.

In theory, you could keep the rooted boot.img in boot_a and flash the recovery..img to boot_b. fastboot has --slot and --set-active flags for slot selection.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
True. Which leads to a question down the road... could we just convert a/b into dual boot :) Let's get TWRP working before answering that one!
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
I pushed my fork here

https://github.com/gee-one/android_device_motorola_payton

Not much works and I didn't do much testing other than booting into recovery.

I'll try to test a few things tonight.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I just pushed a recovery.fstab that uses actual block addresses to mount all the partitions that were failing before. It works, and should get us through testing, but is not A/B compliant. Once decrypt is figured out, we will need to revisit this.

NOTE: These are all slot A mounts
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
"not much works" as in functionality?

Other than USB (obv including abd/mtp), and decrypt everything seems to be working for me. Backup/restore, wipe/repair/format, logs, UI, etc
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
Heyyo, a new test build is going up! twrp-3.3.1-test6-payton is rebased on the tree that [MENTION=7887589]SyberHexen[/MENTION] just shared (which includes my fbe decryption fixes). I also added a commit to fix the source-built bootctrl HAL that we require for the source-built update_engine_sideload. If all functions are good on this TWRP branch? Then we can expand upon it with stuff like that pie kernel I saw in one of his other branches and some other commits I have in my branch. :good:

https://osdn.net/projects/twrp-for-payton/releases/72817

It is built up to this commit

https://github.com/ThEMarD/android_device_motorola_payton-twrp/tree/fbce1d39f6ff576f5d4545a144f2ae23966918b2
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] xeropressence:
USB-OTG doesn't seem to work in SyberHexen's build or test6.

Logs attached.

Do you have telegram? Could be faster to debug this issue.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] big_edd:
[MENTION=6235294]ThE_MarD[/MENTION]

usb-otg has been working for me with Syberhexen's twrp for a long time.

Looks like /dev/block/sda needs to be /dev/block/sda1 ?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ThE_MarD:
[MENTION=9086998]xeropressence[/MENTION] I do have telegram. https://t.me/ThE_MarD

[MENTION=2997136]big_edd[/MENTION] can you please provide a link to that build? I'd like to extract it and see what the fstab files look like since for some reason me directly using syberhexen's branch with the one fix for update_engine_sideload commit is still having USB OTG issues :silly: