yashsingh1/xdaforums huggingface dataset:
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Just uploaded the kernel hiding bootloader unlock built from 10.62 sony sources repacked for 10.80 fw (as sony still did not release 10.80 kernel source) - available in my experimental folder here:

https://androidfilehost.com/?w=profile&uid=5862345805528050190

boot-*-47.2.A.10.80-hideunlock-kernel-10.62.img

The kernels do not have magisk integrated - just patch it from MagiskManager to get the image rooted for use with fota system update (dm-verity option may be tuned that way).

Or just flash it and your preferred magisk version over it.

You may need to take care about dm-verity according to your needs (leave it on if you plan system fota update or simply using only magisk system less customizations or be sure to disable it in case you make any mods directly in system or vendor).

Finally - none of the kernels have been tested at all. I just repacked them to integrate 10.80 stock ramdisk and uploaded...

But I assume they would work as read somewhere that a kernel built for 10.62 worked with 10.80 too.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] lilbrat:
It is, I apologized as I crossed jumped forums.. but in my defense , it does say XZ1c,XZ1 and XZp..lol plus I have the XZ1..
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] SRV.Vietnamese:
I follow your instruction to step 10, then boot to hiding bootloader unlock and get all DRM function worked, but how I can update to new version? Through newflasher or OTA? And how to maintain drm function after it? I try to update from 47.1.A.8.49 to 47.1.A.16.20 with newflasher (all ta files removed) but when I done my XZ1C lost TA and drm key again although I also do boot 47.1.A.16.20 hideunlock version. Moreover, I can't go to twrp when I boot through fastboot
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Klaus N.:
Once done, flashing another firmware won't erase the TA partition you've restored. Restoring the TA partition is a one time task.

And about newflasher and firmware flashing: Before flashing, you've to delete the persist.sin file. And if you want to keep your data, you have also to delete userdata.sin before. The same applies to TWRP: If you want to keep your TWRP, also delete fotakernel.sin before!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] SRV.Vietnamese:
I do. I detele all the TA file fotakernel.sin and persist.sin, but I don't know why I still lost all TA backed up after flash twrp and flash the 47.1.A.16.20 firmware. So I do it again, go back to 47.1.A.8.49, get temporary root (take me one hour), restore TA and update through OTA. But now I'm stuck with 47.1.A..12.75, because there is no hiding bootloader kernel, cannot root, cannot install twrp :(

UPDATE: May because of I flashed the file extracted from FTF, not the firmware downloaded by Xperifirm. But I don't know where to find 47.1.A.16.20 firmware version for Newflasher, because I only find 47.2.A.10.80 with Xperifirm
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Klaus N.:
[MENTION=8186377]SRV.Vietnamese[/MENTION]

Have you alredy booted up your device into system (with your current firmware)? If not, do that, then enable developer options, and in the developer options enable USB-Debugging. Now shut down your device. Then connect your device in fastboot mode (hold vol+ while connecting) with your pc and flash the appropriate TWRP (hint: 47.1.A.XXXX firmware is Android 8.1; you'll find the download link in the op of this thread - choose the Android 8.1 folder and select twrp-3.2.3-0-lilac-1.img) with the command: fastboot flash recovery twrp-3.2.3-0-lilac-1.img

Now disconnect your device from your pc and enter recovery (device must be switched off, then push both <power> and <vol-> and release <power> after vibrating, but keep <vol-> until entering twrp).
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] SRV.Vietnamese:
I was do all the step you tell above before I flash the 47.1.A.16.20 firmware and failed as I mentioned at previous reply. So I downgrade to stock 47.1.A.8.49 again and update through OTA, and now stuck with 47.1.A.12.75 :(

I think the problem because of I flashed the file extracted from FTF, not the firmware downloaded by Xperifirm :(
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] SRV.Vietnamese:
Got all problem solved!!! Detail follow:

The cause was bootloader version mismatch due to the fact that it wasn't being flashed properly. The file boot_delivery.xml must be present in the ./boot/ directory (relative to newflasher.exe and the root of the unpacked firmware). In the old exploitable version 47.1.8.49 provided by [MENTION=4418980]j4nn[/MENTION], this file is properly located and, as a result, the bootloader flashes correctly. However, with this 47.1.16.20 CE1 firmware at ftf format, when I extract it, the boot_delivery.xml file is in ./, the root directory of the unpacked firmware, and as a result gets ignored by Newflasher (using version 13, which is the latest at the moment). The same with Pie ftf firmware.

It did not show up as an issue going from .8.49 and .12.75 because both share the same bootloader version (1_O_79) so it did not matter if it was being reflashed or not. However, .16.20 comes with a newer version of the bootloader (1_O_82), which is why it's essential that the bootloader is flashed.

Thanks to [MENTION=1987749]Aqq123[/MENTION] at #post952.

So I flashed the 47.1.16.20 firmware, then flash TWRP and install Magisk with semc ignored from the optional step 11 follow to [MENTION=9540710]Unbounded[/MENTION] at #post43. Now it work perfectly at latest Pie, 47.2.10.80, TWRP, Magisk and semc ignored
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] 64801965:
Strange! I backup the Ta.img twice, and the two files are different(using "fc" dos command), and I restored with the second one. And then I checked the security menu, it said: somc fido key not provisioned somc attest key not provisioned!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
It's only strange if you restored your original ta and flashed blu-hide kernel and still get not provisioned errors.

Restoring TA on stock kernel does nothing.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] mirhl:
https://www.reddit.com/r/Android/comments/ao3hng/critical_vulnerability_in_android_allows_remote/

http://blogs.360.cn/post/Binder_Kernel_Vul_EN.html

Anybody looking for a new vuln?

(it also seems like trustzone had some trouble)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] 64801965:
At step 10, fastboot boot twrp.img, it stucked at sony boot logo, couldn't go into twrp.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4900593]mirhl[/MENTION], very interesting, thanks.

Particularly the binder one is very nice - could allow temp root for recent devices / fw versions.

The QSEE is nice too, unfortunately it "only" helps to extract a private key. I wonder if we could extract the private key for Android Attestation with it (not sure if it even uses ECDSA).

I am still hoping for something like the 2015 stuff from Gal Beniamini so we could get truly full control of the phone including TrustZone.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Persori:
First of all: Thanks a lot jann for all your effort. You put a lot of time into this projec.t Nice dude!

To make this work with other phones, would you just need their offsets? I'm particularly talking about Xperia Z.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=5435042]Persori[/MENTION], it is not only about offsets, it is very sensitive to timing. Your phone has different cpu. It would require testing on the phone.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Persori:
[MENTION=4418980]j4nn[/MENTION] I would test it on my phone. Do you have an explanation for the testing steps? They don't have to be detailed.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
If you are referring to TA backup, there are already tools to do that from z to xz series.

If you are referring to a blu-hide kernel, you won't be able to flash it as restoring the TA partition on the older devices physically rel-ocks the bootloader, making it impossible to flash anything other than an untouched stock sony kernel.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] bartys:
Any news about xz2 ta backup?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Persori:
[MENTION=4675598]shoey63[/MENTION]

I know, but these tools require a downgrade, while this exploit here does not.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
Only on X and XZ. Z through to Z5 do not require a downgrade, and you did specifically mention Xperia Z.

Downgrading is not a big issue. Sure, it means wiping data, but so does unlocking the bootloader.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=5435042]Persori[/MENTION], as already answered in your specific thread, there is no point.

And what is your current fw version? Lollipop?

Trying oreo exploit would not work. Even the last stage (that setups root credentials via kernel space read write primitives) would need to be modified due to lot older (major) kernel version, requiring kernel debugging.

Reverse engineering kernel offsets, debugging timing with possible need of already unlocked phone to run a kernel patched for timing measurement, porting the last stage for the old kernel - do not get me wrong, but it is not worth the effort.

I really wonder: why you do not want to use existing tools?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] novebis:
Hello again j4nn, thanks to your work I have dared to make the backup of the TA partition. I have unlocked the terminal with the sony code and restored the TA partition again. I think everything is ok since the camera works, I don't know if I should check anything else.

Now before installing android 9 I have some doubts that I would appreciate if you can clarify me,

- I install android 9 using newflaser as I have done for the degradation in oreo and to be able to perform the backup of the TA patrition?

-I understand that once androi 9 is installed, I must restore the TA partition again, repeating the same steps, is that true?

-Does the kernel in android 9 stock have ethernet support?

- Does your modified kernel have it? Or in your case could you add ethernet to your kernel?

I need ethernet on my xz1 and I have no idea how to get it, but thanks for this thread, I think I'm on the right track!

Thank you!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=800675]novebis[/MENTION], if camera worked on oreo after unlock, your drm keys are restored properly with your TA backup.

You would restore TA just once, no need (nor recommended) to restore it multiple times in a row - it would not help anything.

You can use newflasher for flashing required fw.

I have no idea if stock kernel supported ethernet - I gues you mean some usb-ethernet adapter?

Most likely not. And most likely it is not only about kernel support, but fw would need to support it too.

My kernel is stock identical except bootloader unlock hiding patch.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] novebis:
Exactly, it is for use with ethernet usb adapter as I use in other terminals (xperia z5 compact with android stock 5.1.1 compilation 32.0.A.6.200 ethernet works).

But, could you add ethernet drivers with a kernel modification of xz1? I don't know if it is possible to copy the values of the z5 stock kernel.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=800675]novebis[/MENTION], I could enable ethernet usb drivers in the kernel, but it would no longer be considered nearly stock identical.

Moreover, I do not have the equipment to test it - I do not like to release it blindly.

I guess it would be better if it could be built and released separately as an independent project by someone who has the equipment.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] novebis:
[MENTION=4418980]j4nn[/MENTION] I understand that it would not be almost identical, but if only ethernet is added I think it would be interesting to be able to choose between both options indicating the difference, many people would be delighted to be able to use ethenet.

I would certainly try it since I need ethernet. If you can do it I'd appreciate it, I wouldn't mind making a second donation of course.

I have no idea how to do it unless I try if other roms like lineage or existenz have ethernet support, but I would prefer sony stock.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=800675]novebis[/MENTION], thank you for you donation - not sure if thanked already or missed that somehow.

Could you please explain why you need to connect ethernet to your phone?

Just wondering if other way may serve similarly - like for example connecting the phone via usb to pc as a network or modem device (not tested, but assuming that could also be possible, not requiring special usb cable and an usb-to-ethernet adapter).

You know, there is just single purpose of my kernel, to help re-enabling drm conditioned functionality of sony stock fw after restore of locked TA backup.

It would be nice to add multiple features, like double tap to wake, wireguard, special tuning or your ethernet support - I guess many would appreciate that - but that simply is not the aim of my kernels.

I can try to help you, but I do not like to divert the kernel from stock - it could bring incompatibilities.

Let me know, which usb-to-ethernet adapter are you using, so I could check if a kernel module could be built that would load even if not enabled in stock kernel.

You mention that it worked with z5 pure stock kernel and fw without any mods?

I wonder how network is configured on the phone in that case, together with wifi or cellular data?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] novebis:
Ok [MENTION=4418980]j4nn[/MENTION], it does not help me connecting to the PC since I do not always have it and neither does Wi-Fi, so I need a USB ethernet adapter.

I am mainly using an ugreen usb 2.0 otg adapter (asix88772) but I have some other adapter that also works.

With z5 compact stock without modifying kernel (it is not unlocked) and with rom stock of sony andorid 5.1.1 (build 32.0.A.6.200) adapting vats work correctly. It also works on z3 (D6603) with the same android version 5.1.1 (build 23.4.A.1.264).

To operate in the Z5 I activate the airplane mode and connect the usb-ethernet adapter, then I deploy the notification bar and clicking on the usb icon already detects the adapter and I have an ethernet connection. With the Z3 you simply put it on airplane mucus and connect the adapter and already recognize it without doing anything else.

It would be great a kernel module and be able to keep the stock!

PS .: Sorry if I do not answer these next days, in a few hours I have to go on a trip and I may not be able to connect where I am going, but I will be back next week I hope.

Thank you for your help and what you do for the community!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Just released kernel-*-47.2.A.10.80-hideunlock.zip kernels, available here.

Not tested, just built from sony sources finally released today.

The 47.2.A.10.80 fw version provides 2019-06-01 os patch level.

Please follow howto in this post for usage instructions.

Sources updated on my github here too.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Just released kernel-*-47.2.A.10.107-hideunlock.zip kernels (2019-07-01 patch level), available here.

Not tested, just built from sony sources released on 2019-08-09 - this time sony released sources without releasing the fw yet...

The changes in the kernel are quite limited (Qualcomm MSM V4L2 based video driver), so I guess it would work also with the older fw (10.80).

Please follow howto in this post for usage instructions.

Sources updated on my github here too.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
test modules for usb ethernet network support (xz1)

[MENTION=800675]novebis[/MENTION], I've built you the kernel modules to support usbnet (attached).

They are built in xz1 (poplar) kernel tree corresponding to 47.2.A.10.107 fw version.

Unzip the archive (it's not a twrp flashable zip) and push the modules to the phone via adb:

adb push asix.ko /data/local/tmp
adb push ax88179_178a.ko /data/local/tmp
adb push usbnet.ko /data/local/tmp

Having root on your phone, use 'adb shell' with following commands to load the modules:

su
cd /data/local/tmp
insmod ./usbnet.ko
insmod ./asix.ko
insmod ./ax88179_178a.ko

You can check 'dmesg' output for any logs, particularly when you connect the usb-to-ethernet adapter.

With 'lsmod' you should see the modules loaded (it may be sufficient to load just one of the two modules after usbnet.ko depending on the type of usb ethernet adapter you have).

With 'ifconfig -a' you should hopefully see new network interface present.

I am not sure whether the stock fw would automatically configure the interface when connected, via dhcp or whatever - it may not.

Hopefully it helps you.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] d3dm:
I base my answer on my experience with Samsung devices.

I have yet to see version matching between components and it was mainly relying on it being compatible.

Can you only flash stock firmware or can you flash a custom one as well? (dm-verity)

Have you traced the cameraserver to see exactly where it hangs?

I did not follow the entire thread, and I’m currently with mobile access only so I’m sorry if it was discussed before.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Blu-eyes:
I think that the version matching is NOT done in the bootloader, because when I was with the oreo firmware I have reflashed with newflasher to a new Oreo build (Like 3 month new), without flashing the boot directory and after the boot, all TA/DRM related think doesn't worked, Green pics and Keys not provisioned, after another flash including the boot directory it worked like a charm.

For flashing bootloader with dd, I think that it should work, but for testing that, we must foud a partition which is secure to test with. A wrong/damaged tz binary can cause a hard brick?

I was wondering if we could load/chainload a "modified version" of XBL/other binary(as a debugger), (I got this idea from here https://xdaforums.com/showpost.php?p=59174879&postcount=245)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] novebis:
Thank you! :good: I have oreo installed at this time, I will go to 47.2.A.10.107 srock to install the modules!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] mad0701:
Just FYI

I've received the available update information 4 day's via OTA before.

I use the version Central Europe 1 on my phone.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] LazerL0rd:
We had a similar thing on bullhead, where the imgdata was unlocked for the bootloader. Upon downgrade we could change it up and stuff - but an exploit was hard to find in it (the bootloader would need a PNG/JPG exploit which is hard to find) and so nothing came out of it.

Later, Google locked it up and it wouldn't boot with older imgdata then they locked EDL. I was still able to downgrade even tho some weren't, but I think it'll be difficult to find an exploit. I have a feeling they check all signatures match up. Would be easy for them to implement a quick signature check to make sure all BL components match up and avoid corruption.

---------- Post added at 04:50 PM ---------- Previous post was at 04:48 PM ----------

Interesting, seems good.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] pbarrette:
All of the MBN files are signed.

To prevent rollback, the signature can be changed by incrementing the value in the signing certificate's "SW_ID" OU as described here (Qualcomm changed the links) and exploited previously here.

Updated links:

Original document (v1.0).

New document (v2.0)

Here's the info from "xbl_X_BOOT_MSM8998_LA1_1_O_77.mbn" out of "G8441_Customized US_1310-4371_47.1.A.2.281_R8B" (First ever US FW):

Subject:
CN=Yoshino com-2 Attestation Cert
OU=01 0000000000000000 SW_ID
OU=02 0005E0E100010004 HW_ID
OU=03 0000000000000000 DEBUG
OU=04 0001 OEM_ID
OU=05 00020000 SW_SIZE
OU=06 0004 MODEL_ID
OU=07 0001 SHA256
O=Sony Mobile Communications Inc.
L=Lund
S=N/A
C=SE

Here's the info from "xbl_X_BOOT_MSM8998_LA2_0_P_110.mbn" out of "G8441_Customized US_1310-4371_47.2.A.10.80_R7C" (Most recent US FW):

Subject:
CN=Yoshino com-2 Attestation Cert
OU=01 0000000000000000 SW_ID
OU=02 0005E0E100010004 HW_ID
OU=03 0000000000000000 DEBUG
OU=04 0001 OEM_ID
OU=05 00020000 SW_SIZE
OU=06 0004 MODEL_ID
OU=07 0001 SHA256
O=Sony Mobile Communications Inc.
L=Lund
S=N/A
C=SE

So they don't appear to be implementing rollback protection on the bootloader or trustlets, which means that you should be able to write unmodified mbn files to the partition without issue. Because of the signature, though, you really don't want to modify them.

Now, you may note that there's a "SW_SIZE" attribute, which is supposed to define how much data is actually signed. So you'd think that you should be able to modify code after that signed data size. Well, maybe. But:

- All the signatures I've seen have the same data size listed ("00020000") which is about ~130KB.

- The tzxfl MBN file has that same value, but the entire MBN file is only ~45KB in size.

So I'm not sure that "SW_SIZE" is reliable in this case. I assume that they're just signing the entire file and ignoring the "SW_SIZE" value during validation.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4782500]pbarrette[/MENTION], thank you, that are really interesting finds.

I agree the SW_SIZE looks fishy, but it's kind of redundant field as they need to parse size information from each mbn file (or corresponding partition) to actually load it (if not considering the option to just load the entire partition). So it's probably a dummy not used value as you found.

[MENTION=7543898]Blu-eyes[/MENTION], concerning the partition to start testing downgrades - in my opinion, the best one to start with is the 'xfl' one.

It seems that the 'xfl' implements the 'sony service mode', which is the mode used to flash sony stock fw .sin files (the flash mode used with newflasher, green LED light).

Previously I've assumed (and discussed with [MENTION=4120742]munjeni[/MENTION] in pm) that 'xfl' is not used at all, being just an obsolete implementation of fastboot in linux, replaced by fastboot implemented in 'abl' uefi android boot loader. Now I agree with munjeni that it must be the 'flash mode' implementation.

That would mean 'xfl' is loaded only when entering sony flash mode and therefore may not influence normal boot in case signatures/versions would be checked to match.

On the other hand when entering the flash mode, if version matching was implemented, boot into flash mode would fail. In that case we should hopefully be able to fix that back from normal boot the same way as the downgrade of it with dd.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] 64801965:
Does Flashing a new firmware after restoring TA overwrite TA partition? If is the case, when I flash a new stock firmware, backing up TA partition would fail?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] ecksdeeay:
As long as you don't flash ta files excluding the bootloader ta, it's fine.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=8309153]64801965[/MENTION], flashing stock fw does not remove your restored drm keys from TA, it does not matter which version you flash (downgrade or upgrade).

Just be sure to flash the bootloader too while flashing a fw (the complete boot/ subdirectory needs to be present including .ta there, .xml and .sin files).