yashsingh1/xdaforums huggingface dataset:
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
We have fastboot... but "fastboot format userdata" returns an error about /data being an unsupported (RAW) format.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] kraatus90:
But can't we use fastboot erase userdata ? this command works on motog5 to decrypt it.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Hmmm... not 100% sure I tried that or not! I think I did. I assume the G5 doesn't force back to FBE after formatting?

Process for X4 seems likely to involve formatting, booting direct to recovery, patching boot.img/fstab.qcom... otherwise, on initial boot stock fstab will simply reactivate FBE.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] kraatus90:
G5 has FDE i think.

try this , change fstab in boot.img for disabling fbe, flash that boot.img, and in bootloader, run fastboot erase userdata. it should decrypt it. i have no other ideas lol, only found this for op5t.

maybe we should get some help from TWRP dev ?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
This was my thought from the beginning. But the solution needs to be user-friendly...as in, we need the Format Data button back, and we need a flashable fix for boot.img.

TWRP devs I have reached out to aren't aware of anything that would cause the format data option to disappear. I PMd Dees_Troy, and hoping he can shed some light on it.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
Would it help to work with TWRP 3.2 that has support for v2 fstab and newer xt4 FBE policies? Stab in the dark.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
YES! Been so busy messing around with 3.1.1 that I missed 3.2 being dropped :) Thanks. Will update sources today

EDIT: Hmmm, seems that omni bootable_recovery sources are all still 3.1.1.

EDIT2: Ohhhhh... it is only on the android-8.0 branch. Interesting. Guess I will need to start over AGAIN. :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
So, quick test this morning: unpacked stock boot.img, changed fstab.qcom to add flag enabling encryption selection, repacked boot.img... flashed boot.img and fastboot erase userdata.

Result: Device failed to boot (stuck on a dimmer version of the splash screen). And in fact, flashing any other boot.img failed after this. It is as if booting with this flag had altered the system partition and stopped boot. Thinking I had made an error, tried the whole process again (after restoring nandroid) with the same result. So, changing one flag in fstab that should only allow the OPTION to disable FBE after booting causes boot to fail completely and soft bricks the device. So strange!

Anyhow, I am syncing full sources for LOS 15 and TWRP 3.2.0 (probably take good 6-8 hours)... back at it tomorrow.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
From what I was reading it had legacy abilities.

EDIT: I didn't realize rebuilding TWRP was such a big project, I misunderstood what you meant.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Got decrypt working! Still need to test, but it looks good...

adb/mtp is still down, but at this point it seems like we are very close to a fully working TWRP. :)

Unfortunately, the device will not boot normally with this build, it just loops one and then boots back into TWRP. So, need to either solve that in compile or repack the recovery manually like I was doing previously.

Will update OP soon.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Decrypted /data. Can't tell you how glad I am to be done with that ;)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
OP udpated with beta release.

Any brave tester want to give this a shot and confirm decrypt on another device? Obviously, you should have backups ready and be adept at getting yourself out of trouble.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
OP updated to Beta 2

- fixed "Internal Storage" (/data/media/0)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] kraatus90:
Awesome man. since new 26-7 fastboot images are available now. i'm gonna try it. :D
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
Seems to work, already did a backup, restore, etc. Let me know what else I need to try. Thank you!!!

EDIT: I am willing to test ANYTHING you need help with.

EDIT 2: I did get an error stating updates cannot be installed on this device.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
What kind of error... in TWRP?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] kraatus90:
Decryption not working for me unfortunately. it does decrypts and formats everything. but once you boot to system. its encrypted again.

my active slot is B since i took ota.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Ah... fstab is hard-coded for slot-A in this build. Let me work on bringing it up for A/B and that should solve the problem.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
No, TWRP was working just the MOTO update notification when I rebooted. I actually expected it anyway since rooted/unlocked. Thanks for your hard work.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Yes, that will persist, unfortunately. I blocked it but still get a couple full screen popups each day.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] michael28056:
You can freeze Moto updater in titanium backup to fix that.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Did you flash the TWRP image to Slot A as well?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] kraatus90:
Flashed on both slots, even tried changing slot within TWRP. but no success.

i'm restoring everything now, will try decrypting later since i need a working phone. sorry but can't help atm. :p

also buying a sdcard today to test kernel flashing etc.

thanks for hard work on TWRP btw, looks like not many devs have bought this phone just like motox2. :(
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Is it possible you misinterpreted the changes today?

I am not decrypting the /data partition, rather was able to compile TWRP with crypto support, so we are now working with the as-is ENCRYPTED /data partition.

Since no longer needed, I rolled back the patches to disable encrypt-on-boot and disable verity which is why it force encrypts on boot.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] kraatus90:
Oh, and i thought i can decrypt now. :p lol
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
Which is weird since it's a decent, affordable and unlockable phone.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I suppose I could build that back in since we do have the "Format Data" button back in TWRP now, and that does wipe encryption... Though, with the performance efficiency of FBE, I am not sure what benefit there is to unencrypting.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
On the USB front:

Something broke USB storage in beta2. Not sure whether it was related to updating to 3.2.0 or to some other change, but the LUN file is no longer visible to TWRP in /config/usb_gadget/g1

In fact, /config remains totally empty until full system boot... So something is off in init files somewhere.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Strange... flashed a test build today, and when I went back to beta2, /data is not being decrypted! Have tried wiping and flashing repeatedly, but not working. Some funny stuff going on here :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] bwtmanagement:
Tried flashings with adb fastboot and went normal i the output script, but still stock recovery after...am i missing something
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
Can you post a screenshot or what you typed?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I would say yes, you are missing something. :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Still failing on decrypt in 3.2.1 and when I flash back to beta2. Very interesting and frustrating at the same time :)

I noticed that while TWRP seems to be able to mount root level folders and /system, there are many files missing. Looking into this, but for now be aware that this is still in the experimental phase.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] Zalbu:
Can you not install TWRP if you have unlocked the bootloader and rooted? I'm just getting the "recovery partition not found" if I try to flash through Flashify and it's not working via fastboot either.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I don't recommend installing it other than for testing purposes right now -- especially if you are trying to flash to a recovery partition (which this device does not have.)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] bombaglad:
what is the correct way to flash this thing for the best results? is it the same as we know it from all pre-nougat phones (fastboot flash recovery recovery.img), or do we do it differently with all this partitions-mess?

thx
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Again, there IS no recovery partition. So, the correct way would be

fastboot flash boot_a FILENAME

and probably good practice to flash it to both boot partitions:

fastboot flash boot_b FILENAME

But I also stress again that it does not seem to be working consistently. You may have decrypted /data and suddenly lose it -- so be sure to have backups on the external SD card.

---

On the dev front... it seems like 3.2.0 and 3.2.1 are hard coded to /data/media/0/TWRP/ as the default directory for TWRP regardless of encryption. The logs show that TWRP *thinks* it is working with a decrypted partition, mounts it and tries to call the settings file. Want to find a way to allow mounting of /data/media and defaulting to SD card instead.

Part of me feels this may all be solvable in fstab, so will spend some time on it.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] vivek638:
Any update on twrp.. making it stable seems promising?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
I can confirm that I did flash the TWRP image into recovery and it worked, I don't know how, but it did.

i.e. fastboot flash recovery <img name>
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] michael28056:
First post has the latest updates.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] bombaglad:
wait, so you just flashed the img via fastboot to recovery, and were able to boot it?? which build did you use? how about decryption? how about booting to system and then back again?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Whoa, let's pump the brakes here. This is how things go bad. I assure you there is still not a recovery partition on this device. Maybe some version of fastboot can correct for this error, but TWRP lives in the boot partition.

Whether the command works or not (if fastboot recognizes it), it will not affect functionality.

The eventual progression of this build will likely be such that we will be flashing back to stock boot.img and then fastboot BOOTING a special TWRP image to inject TWRP. But at no point do I believe we will be correct in flashing to a recovery partition.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Can you send me dmesg/recovery logs from TWRP (in "Advanced" menu". Want to see how data is mounting/decrypting on another device
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
For some reason, the logs aren't saving. TWRP says they're in data/media/0, but they're not there.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Can you see the files in /data/media/0 from the TWRP file manager?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
Every file/directory when I use data/media/0 is encyrpted:

example emtpy directory:

FS2DJzKxOVDBPgSDHaSbiB
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
As suspected... decryption is dead again. Not sure why since it obviously was working in the intitial Beta1 build.

Perhaps I need to test starting from stock boot.img, booting a TWRP image directly with fastboot and immediately rooting.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
That makes sense, modifying before any initial boot.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] gee one:
Isn't the decryption key tied to the signature/fingerprint of the boot.img? If you change the boot.img, you are probably decrypting with the wrong key? Maybe sign the new boot.img with the same key?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I have tried signed vs unsigned images.

My assumption is that the boot image is verified via OEM key... but it may be that the actual *changes* to the boot (or perhaps even system) partition cause verity checks to fail and forces the TEE bind the keymaster with a new key that does not match the one originally used to encrypt /data. This could concept is a pretty strong protection from exploits.

I believe it is also the same basic mechanism that breaks OTA after rooting.

This is the reason behind my suggestion...a stock system would pass both fingerprint/OEM key verification, but also verity... and TWRP could simply be booted from fastboot and made permanent with scripts.

Will start testing tomorrow.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
So strange working without a separate recovery image.

I have a full image file that contains the kernel and recovery. It boots fine, directly to the OS and to twrp from abd/fastboot.

Now, theoretically fastboot supports booting separate kernel and ramdisk images (which would allow us to boot recovery without flashing)... But I cannot seem to make it work.

If fastboot can't directly boot the recovery ramdisk, I guess it would require changes to init.

Suggestions on fastboot syntax to do this the easy way are welcome :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
As some of you may noticed, I like to learn by figuring out things on my own. But I ran out of patience and started looking around... haha. Check this out.

https://twrp.me/google/googlepixel2xl.html

Pixel 2/XL TWRP is a BOOTABLE zip (fastboot boot) that temporarily boots TWRP. A flashable ZIP is then ADB pushed to the device and executed to patch TWRP into both A and B boot partitions. The guide also states explicitly that the device needs to be flashed back to factory if the stock boot has been tampered with.

So neat to see all the roadblocks I ran into solved on a different device (by the master Dees Troy himself). I will use his Pixel 2 TWRP and script as a guide, and start building.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
I had read that and figured it was already known. I should have posted way sooner. Sorry about that.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
No worries, I never learned anything by cutting/pasting from other work anyway. :) Obviously, Pixel is a Google device, so the guts of the code were never going to do us any good as far as decrypt (different libs)... it is really just the *method* of installation I was happy to see, as it was my theory that it might be the only way to make it work.

As a side benefit, I will get to test out factory flashing to retail and then returning to Android One/Project Fi :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] rogerjbr@gmail.com:
TWRP 3.2.1-0 Released
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Thanks, been on that source for a week or so
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Fully flashed to the retail image to begin testing... other than the super annoying boot animation and a few apps and features enabled, I see very little different from the Fi version. Project Fi itself is preinstalled and works 100% fine. Other than branding, I am not sure the Fi variant is worth fussing over... I will work from retail moving forward.

I have made a flashable ZIP to inject recovery to the boot.img from a temporarily booted instance of TWRP... ready to test. But I am still struggling to make a recovery image that boots to recovery from fastboot...
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] kraatus90:
There's one huge bug with beta2(didnt try previous versions), if i flash su.zip it works, but when i flash my kernel zip after that, root is gone. also after flashing my kernel zip, and rebooting to twrp from bootloader, it hangs in twrp logo. pretty weird lol. and it doesn't save any logs.

i can post my kernel zip if you want to do some testing, it uses stock modules and wifi isn't broken anymore. still testing Energy model for eas, that's why haven't released it on xda yet. :p

for help, you can look into this . it has same partition scheme as x4, only difference is SOC.

Edit : attached kernel zip. :fingers-crossed:
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I am many changes beyond beta2 (in fact that one was reverted since 3.2.0 made it's boardconfig changes obsolete with auto /data/media mounting.)

But nothing posted as I am fairly certain about needing to start from an intact system to pass crypto... so priority at the moment is booting from fastboot.

---

Surprised that adding "initrd=" to cmdline to force the kernel to boot the recovery ramdisk does not work. Neither does this:

fastboot -c <cmdline arguments> boot <kernel.img> <recovery-ramdisk.img>

I would think that either of these solutions would boot to recovery. I am obviously missing something here...
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Hey, cloned your github on this and built an image with that kernel and twrp on board, just for fun (while I beat my head against a wall trying to boot directly to recovery).

FWIW, it compiles and boots fine. :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Well, I have come full-circle on this... Despite my efforts to support native decrypt, it seems the best option is to format data and patch boot to prevent it on first boot.

I believe it can be done properly, but I don't have any more time to commit.

Have confirmed that "format data" provides unencrypted /data... will finalize a twrp build and flashable zip for disabling force encrypt soon.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] boladaso:
Hi, I would like to know if it's working in my X4 Payton, Brazilian version.

XT1900-6

7.1.1

M660_7020.05.01.52.01R

NPW26.7

NPW26.83-42

Security Patch

November 01 , 2017
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] michael28056:
Read the first post.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Not having much luck preventing forced encryption on first boot. Looking at some other devices that are taking this route, and the same fixes for those do not seem to apply to us.

In other news I did get USB OTG working again in my latest build, and I have been in touch with a couple other people who are interested in both helping get TWRP decrypt fixed and working out the rest of the tree/blobs for building LineageOS.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] ebrandsberg:
I'm hoping that Oreo comes out, so that we can use the device tree and such, so that any efforts that are made on LineageOS can focus on Oreo. Everybody agree that this would be the target?
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Little tricky working across 10 time zones, but very happy to have some willing devs helping out now. The future is looking bright for "payton".

Yes. I have already started on proprietary vendor for LOS 15, and know of at least one other person who has the system image and intends to work on it (and hopefully help me solve either TWRP decryption or disabling encryption on first boot)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
Bought another X4 Android One phone. If I can help in any way let me know.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] vivek638:
Get the link of ota pls.. whenever u get the oreo update. Maybe it can help regular moto x4 owners to get oreo.. if possible
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] johnjingle:
I haven't received the update yet, but I will do a logcat before installing to get the URL. It's too bad no one else has posted it yet, I have checked several other sites.
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] vivek638:
I think moto x4 is not that popular rit now. But i can definitely say that its a best budget phone in sub 20k.. But nowadays only people are getting attention of 18:9 ratio phone. Whether it have something good inside or not.. Atleast in india i am not seeing any phone which comes close to this device with this much specs n ip68...
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
I have two people from India helping me on development, which is great! Once TWRP is fully functional for all variants, we will see increased interest in this device. Went through this with the original Moto G 3-4 years ago... took us forever to get all unlockable variants built. But that old device has TWRP and bleeding edge custom Oreo support now. This device will be a great one to hold onto in the long run.

Still struggling with the final stage of TWRP implementation... as are many other newer A/B devices. I feel like this is the week (though I have felt that way for a month) :)
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] vivek638:
Thats a great news... Hoping to see it soon..
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] rehanraza:
Excellent. Just waiting for proper stock firmware IMG before any root attempt
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Well, all attempts to make this work are still leading back to the exact same issue: /data decryption does not work. It is possible to format userdata and remove encryption, but it returns on the first boot of the system, and attempts to change fstab to prevent this have been unsuccessful. The only other SDM660 device with working TWRP is one that has a separate /recovery partition (lucky!).

I have tried dozens, maybe hundreds of "fixes" with virtually zero change from the first day I booted TWRP on the X4. I am quite certain that the solution requires starting from 100% stock, booting a temporary recovery (fastboot BOOT), and using a ZIP installer to inject the recovery ramdisk. With that method apparently unavailable to us, we are simply stuck.

I will be putting further work on TWRP on hold for now... not because I don't think it can be solved, but because I would rather work on other things that can help us solve this problem indirectly. For example, building out a Lineage based ROM with a custom kernel (maybe even the latter alone) will allow us to avoid the forced-encryption issue and at least have fully working TWRP for custom ROM uses.

I will probably take one last look at what I have, clean up some stuff and push a final beta build. If a temporary-boot solution becomes available, I already have compiled a flashable ZIP file for our device.

Going to shift now to using my limited spare time to build out the proprietary files and vendor tree. This is what we really need in order to jumpstart custom ROM development. I may also make a custom stock-based flashable ZIP (there is nothing stopping us from using TWRP to flash ROMS !)

---
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] vivek638:
Thanks for your work mate...
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] vivek638:
Any more work going on for recovery??
[Moto X4 ROMs, Kernels, Recoveries, & Other Develop] mightysween:
Not until there is a way to compile recovery images that will boot to recovery on a stock device... Seems that function was broken somehow upstream and even compiling with older sources still boots directly to the system.

I even tried compiling several versions of Cyanogen/LOS recovery but it also boots directly to the system... So not a twrp issue, but something in common bootable recovery that has changed