yashsingh1/xdaforums huggingface dataset:
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
Sadly no bootable results

Even with the AOSP kernel.

You made black magic.

Oshmoun from the SODP team is interested too.

The only bootable result is your compiled XZ2 kernel.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=6031898]MartinX3[/MENTION], really? That is strange. Even when tried with the compiled tools from me?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
Yes, sadly we hadn't any success. :(

The funny part is that I provide AOSP and OmniROM with working kernels on XDA. :silly:
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=6031898]MartinX3[/MENTION], it might be some verified boot issue?

What I tried to achieve was to boot a custom built kernel from usb while faking still locked state and having pure stock not modified complete fw in the phone including kernel and all other partitions.

It may behave different if something with some changes is flashed (even that verity disable in vbmeta /or how it was called/ could make it fail).
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=6031898]MartinX3[/MENTION], didn't you try to rebuild the kernel from my source on github? Simply try to reproduce what I did? That should work for sure in my opinion - try any of these branches:

my-bluhide/52.0.A.3.163

my-bluhide/52.0.A.3.202
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] niaboc79:
Here certainly

vendor.semc.hardware.secd@1.0-service

Envoyé de mon H8266 en utilisant Tapatalk
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=599722]niaboc79[/MENTION], I assume that is from pie fw? xz1c or other yoshino?

Well, I checked only oreo fw, did not really try with pie when found that lot of libs are compiled statically into binaries, it's more difficult to check.

Care to post a disassembly snippet? Or at least a fw version with the address/offset?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] niaboc79:
Yes it's from pie in vendor.

That file is where all apps who are using security look for to check bootloader status, credentials,...

This is also where the drm fix pre-loaded code.

I could disassemble it with ida pro if you want.

https://drive.google.com/file/d/1xiw7IsyQxgET06HWirUkIGwYK3plphq_/view?usp=drivesdk

Envoyé de mon H8266 en utilisant Tapatalk
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=599722]niaboc79[/MENTION], if it is only your assumption, that the key is used there - then no need to disassemble that for me. I could do it too. That's not the issue.

The thing is that I not only could see it in the (oreo) binary of secd (using idapro too), but I even tried it on runtime - that key is not loaded when you restart secd process on oreo.

I've logged it with intercepted lib call.

Please post proof, not assumptions.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] niaboc79:
Well apps like update center and backup restore are using two framework jar who point to this binary.

I was able to get both app working by patching these apps directly or by patching the framework security jar.

Also if you're editing this binary correctly with an hex editor you are able to achieve what you did with your kernel, to get bootloader locked status, which enable ota updates since it gives correct information to the binary who gives correct information to the jar who gives correct information to updaye center app.

It's only a guess that the magic is inside this binary but for me it seems legit.

Envoyé de mon H8266 en utilisant Tapatalk
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=599722]niaboc79[/MENTION], and it's not only secd that I tried to restart on oreo to see if something would load that TA unit.

I even tried to modify that key (to see if invalid device master key would have any influence in the phone) and restarted like half of the daemons running on oreo (at least all that loaded libmiscta or libta). Nothing. The change in TA was not even detected.

Either I've missed something or it's handled on early boot by bootloader.

But when flashed that TA with one byte of device master key changed and rebooted - that resulted with a hard unrecoverable brick. It is for sure checked (either TA consistency or the key itself) on early boot.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] niaboc79:
Restart secd init has never worked on oreo neither for me, a complete clearing of all datas was needed with the drm fix because the credential.db seemed to be only created during first boot after a fresh installation.

But you're more qualified then me about that subject.

For me the easiest way to do a drm fix on pie would be to removed Pie secd binary and related static files and replace them by Oreo ones. I tried but was blocked by some socket creation in selinux. I'm not a kernel dev neither so I wasn't able to fix that part.

As you said the static libraries are not easy to decode unfortunately. But [MENTION=236992]Storm[/MENTION] was able to understand how it worked on oreo and that's how his drm fix used pre init code to bypass the missing drm and create credentials.db without them.

Unfortunately storm stopped all development due to personal things and until now you're the most skilled guy I know who could find a solution.

With our current devices, loose drm isn't that important but with coming Xperia devices, I think that loose post processing for camera would be a problem since Sony finally managed to build a good camera software with raw support.

I know that you might be bored by my various requests [emoji12]

Envoyé de mon H8266 en utilisant Tapatalk
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=599722]niaboc79[/MENTION], I do not know about that credentials db. But if I remember correctly, restarting secd with patched lib changing bootloader status worked for me - I think I could see that change in security screen.

In my opinion, using oreo binaries on pie will not work.

The reason is the TrustZone applets are not compatible - they are somehow loaded from bootloader already. And if you use oreo bootloader with pie fw (or vice versa), it does not work at all - at least one of those combinations can boot, but in security screen, you just get a single word - "None" - that's it. And when you try to take a picture, it just hangs on touch of the take pic button, then it is needed to kill the camera app to get out of it (using the square button listing running apps).

And do not forget one thing: any drmfix cannot restore _all_ functionality - if there is any other key depending on the device master key, you cannot patch that.

Like for example if widevine key was encrypted and to enable it, the firmware (or TrustZone) would need to use device master key to decrypt it, while it would be device specific (enforced in TrustZone), there would be nothing you could patch to achieve L1 widevine decoding support - if sony implemented it like that.

The FIDO key may be another possibility that could be encrypted with device master key, but I do not know how to even test that.

Which sony device now has good camera with raw support?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Somewhere I've seen a security screen captured from a phone running strom's drmfix:

SOMC FIDO Key: NOT PROVISIONED

SOMC Attest Key: NOT PROVISIONED

while all the rest seem to be ok - but who knows - as that security screen does not really tell if anything truly works as I've verified with the Android Attest Key (you may check the Android Attest Key lost thread)...

For me any drmfix is not a complete solution.

Therefore in my opinion it does not make sense to work on it - it is better to implement a temp root exploit and do a TA backup instead.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
Stand-alone stock kernel works with your tools and commit, however we can't get TWRP to work.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
Hmm

Are we sure that it didn't just switched to the installed kernel?

Ugh

It was too long ago for me
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] niaboc79:
The Xperia 1 [emoji6]

Also if we could temp root on Pie it would be perfect [emoji108] but no exploit [emoji17]

Envoyé de mon H8266 en utilisant Tapatalk
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
That is actually quite easy to check - there is different build date that can be seen with kernel version in the about screen.

And another difference that my kernel would show bootloader unlock (rooting) status as "not done" in the service security screen, while the stock kernel would show "done" on an unlocked phone.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Do you mean you could build my kernel from the sources and it worked, while when you build twrp having the same patch to workaround TAMA 'fastboot boot' problem it does not boot?

I assume that twrp is just a kernel with twrp recovery ramdisk built in.

Should not it be possible just to repack my kernel with a twrp ramdisk?

Maybe the twrp (kernel) actually boots but then hangs due to some problem in twrp init?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
Yes, stock kernel fastboot boots using only the merge dtbo into kernel builtin dtb for tama fastboot boot commit as we have disabled dm-verity in vbmeta. Although useless, it was a starting point to show the commit works.

However, using the same Image.gz-dtb in a TWRP recovery image results in simply coming out of fastboot mode,then booting to system.

Compiling an Image.gz-dtb from your bluhide/52.0.A.3.202 repo with your tools and no changes results in the same behaviour.

HOWEVER, if we use the Image.gz-dtb from the boot-H8216-52.0.A.3.202-hideunlock-rooted-standalone.img that you provided for Serajr to test, we have a working fastboot bootable TWRP.

Are we missing someting here? Should not the Image.gz-dtb be exactly the same as when built from your source?

Whilst happy and grateful that the one you made works, it make it impossible to support fastboot boot TWRP for any other device on the Tama platform, as we have only the one you made for Akari, and can't replicate a successful build.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=6031898]MartinX3[/MENTION], [MENTION=4675598]shoey63[/MENTION], I might have an idea, what that "black magic" might have been;)

If you can compile stock kernel with my patch and it boots via 'fastboot boot' as it looks like from shoey63's post, I guess the only missing thing might be this:

# Force kernel to load rootfs
# skip_initramfs -> want_initramfs
./magiskboot --hexpatch kernel \
736B69705F696E697472616D667300 \
77616E745F696E697472616D667300

As you know, my kernel is pre-rooted with magisk - and this is done by magisk (besides other things) when it patches the boot.img to replace init with his own.

Could you please test that?

I believe this could make your twrp with kernel compiled from sources bootable as standalone kernel.img via 'fastboot boot' for all tama platform phones.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Updated post#2 in xzp/xz1/xz1c variants of "rooted kernel hiding bootloader unlock with working fota" threads: added hideunlock rooted kernels for 47.2.A.10.28 fw.

Sources available here on my github (my-bluhide/47.2.A.10.28 branch).

Pre-released versions did not include few most likely security fixes (as can be seen in this commit with changes against 47.2.A.8.24 version).
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
And Indeed you are right! I let Magisk do it's work and repacked the patched zImage-dtb back into the twrp.img and fastboot boot twrp is working:)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
[MENTION=4418980]j4nn[/MENTION]

thank you very much!

I love you!

Btw. can the "magiskboot hack" be done in the source code of the kernel too, as github commit?

Do you replace this call?

https://github.com/sonyxperiadev/kernel/blob/aosp/LE.UM.2.3.2.r1.4/init/initramfs.c#L619

Or something else?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
Just spoke with the SODP team.

Tested with the AOSP kernel the dtbo kernel hack wasn't needed.

Only the single magiskboot hack, which can be done in the source code too.

AOSP Kernel

diff --git a/init/initramfs.c b/init/initramfs.c
index bf3af10c500a..fe91420c49e3 100644
--- a/init/initramfs.c
+++ b/init/initramfs.c
@@ -616,7 +616,7 @@ static int __init skip_initramfs_param(char *str)
do_skip_initramfs = 1;
return 1;
}
-__setup("skip_initramfs", skip_initramfs_param);
+__setup("want_initramfs", skip_initramfs_param);

static int __init populate_rootfs(void)
{

The

skip_initramfs

in the cmdline is the culprit there and if the line got renamed, it can't find the

skip_initramfs

, because it searches now for

want_initramfs

And maybe this works with the stock kernel too :)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4675598]shoey63[/MENTION], I am glad that it started to work finally.

And thank you for your donation - that was not necessary though - I was happy to help (and to take the challenge;).

I have received another donation today, cannot connect a user here. Thank you too for your support.

[MENTION=6031898]MartinX3[/MENTION], I guess your findings about what the magic command does are most likely right.

Obviously, that is something that has been present in magisk, I just extracted it from it's boot_patch.sh script, based on shoey63's description what worked and wondered what could be missing (and as I studied internals of magisk, I remembered that patch).

However, I still wonder, if my patch to merge DTBO might be useful to make the kernel to be "standalone" bootable via 'fastboot boot'?

Could you please try to erase DTBO partition and boot the kernel patched with my patch (and magisk's "want_initramfs") if it still works or not?

I would guess that twrp with merged dtbo could boot even in case of erased DTBO partition but if used just only the "want_initramfs" patch without dtbo merge, it could need correct DTBO partition?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
No problem. I'm sure there are users that should be donating, but haven't done so [emoji6]

That's right. Kernel in the TWRP.img must match the DTBO otherwise you will get a corrupt firmware message and the phone turns off after 5 seconds.

That's why the merged dtbo is so handy. You can fastboot boot on any firmware version and is absolutely necessary for any hope of official TWRP support.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Do you mean that a kernel with merged dtbo can boot regardless any content of dtbo partition, not only an erased one?

Did you test that already?

I wonder if bootloader could possibly corrupt or in other way change some device tree properties in the kernel to be booted via 'fastboot boot' if the kernel contains full DT, i.e. with merged DTBO.

Or if the kernel with merged dtbo is truly independent/standalone?

That would be nice of course...
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] shoey63:
I tested TWRP (which is a standard boot.img with a TWRP ramdisk) embedded with patched 202 kernel/dtb combined, and it fastboot boots fine on my 8.25 firmware which is 2 security patches ahead so it would seem DTBO partition is ignored using this method. When you reboot to system after, it is as if nothing has happened. It appears to be truly standalone.

I have not tried with a wiped DTBO partition.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MurroFX:
Root on latest Firmware 42.2.A.10.28 ?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Yokurt815:
Succesfull backup of drm keys :)

Thank you very much!

My steps:

* got new phone with 47.1.A.5.51

* upgraded to G8441_47.1.A.8.49_CE1 from OP with newflasher (only deleted persist_X-FLASH-ALL-C93B.sin)

* backup TA according OP

* unlock (renders camera useless)

* restore TA according OP (camera works again)

* fastboot hideunlock-rooted kernel

So far Everything fine! , but after FOTA update hideunlock-rooted kernel is not starting anymore. I'll post more details on [XZ1c] rooted kernel hiding bootloader unlock with working fota
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] SunJu22:
Hello j4nn and thank you very much for the hard work achieved and of course the guide that comes with it (thanks also to munted - 42 page long guide? wow).

I will very soon start the procedure. Do you still need testing feedback from users with locked BL about to go through the steps?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] SGH-i200:
Sure you mean 47.2.A.10.28, but 47.2.A.10.45 has been released already! ;)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
the dtbo partition doesn't change often, it contains hardware specific stuff.

[MENTION=4418980]j4nn[/MENTION] [MENTION=4675598]shoey63[/MENTION]

I compiled the .25 april kernel with the dtbo hack and the drmfix hack and changed the in skip_initramfs hack in the kernel code

https://github.com/MartinX3sAndroidDevelopment/sonyxperiadev-kernel-copyleft/commits/52.0.A.8.25

Sadly it doesn't work.

The blue led disappears at sony logo and I get a bootlooping android boot animation without adb.
[MENTION=4675598]shoey63[/MENTION] your twrp.img works.

what did I different?

Or are the dtc overlay tools different on my computer?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=6031898]MartinX3[/MENTION], you should have my build of the tools - just test it and see if it is due to that or something else?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] ecksdeeay:
Having been occupied with personal commitments I was getting round to doing the UL BL > flash > then hide unlock but have gone through all the updates from Oreo to the NOW current 47.2.A.10.45 (G8142).

Would it be best to downgrade again and flash then hide the *10.28 release and call/wait upon j4nn to his magic with the latest fw?

Went through 18 updates from Oreo (sheeeeeesh!) but at least I have the stock decrypted fw via XCompanion ready for flashing (which I forgot to save with the *10.24)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
please poke me with a stick until it hurts.

One of my stupid moments.

I did lay in my bed at night, and suddenly I became wake as ****, because I remembered that I didn't changed in my build environment the line to use the compiled kernel, instead of the extracted one.

Now, since it works, I need to look which of your commits is needed or if in the end we only need the magisk initframs change. :)

Thank you for your work. :silly::victory:
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
[MENTION=4418980]j4nn[/MENTION]

Ahh

Only this is needed to make the twrp bootable by "fastboot boot twrp.img"

https://github.com/MartinX3sAndroid...mmit/ab70b939ed749efdac60d23a3c24a0d6ea6ad4cc

It should use the dtbo partition then.

Thank you for your help. :)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] khanhesham:
[MENTION=4418980]j4nn[/MENTION] could you please make the drm fixed kernel for xzp g8142 47.2.A.10.45 april 2019 patch
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4800473]ecksdeeay[/MENTION] and [MENTION=6250451]khanhesham[/MENTION], I cannot release a version for 10.45 because sony source code has not been release yet - I need to build the kernel from source. Please see also here for additional info.

[MENTION=4800473]ecksdeeay[/MENTION], I wonder how did you do the incremental 18 steps update in a row from oreo to pie 10.45 having bootloader unlocked?

I assume that has not been done via fota system update, because there are not 18 kernel versions with bootloader unlock hide.

Is it possible to use Xperia Companion to download incremental update and export each step (fw version) as a backup?

[MENTION=6031898]MartinX3[/MENTION], I am glad that it finally works.

Just wondering - if you wanted to support all TAMA phones, do you need a kernel (twrp) build for each variant or is there any possibility for "general" kernel, maybe using device specific DTBO partition for example?

I see that with TAMA the number of kernel config variants increased a lot vs yoshino (I would consider that case as too many there already).
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] MartinX3:
Thanks :)

As far as I know every tama device needs its own kernel.

I am not a low level developer, somewhere should be differences.

But I'm not a good help in such stuff.

I'm from the high level and administrator world of IT :)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] ecksdeeay:
I realise the source will take time to be released and then worked on!!

I have yet to UL BL ( but am just about to do it now as I type this having gotten the CN 10.28 fw from Xperifirm to use the latest hide kernel provided (thanks again)

I was unable to find where the delta updates were being loaded to in Windows and do the work of exporting each fw was beyond me.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4800473]ecksdeeay[/MENTION], I am not sure about CN - is not that with google play services/store access removed?

If I remember correctly someone mentioned that as an advantage allowing to install slimmed version or maybe avoid it altogether in simple way - if that is what you want?

But I may be mistaken...

You may download older xzp fw releases on this site for example.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4800473]ecksdeeay[/MENTION], the "fastboot -i 0x0fce oem unlock 0xYOURCODE" should be very fast.

Mine took less than 400ms.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] ecksdeeay:
yeah I'm getting -i invalid option

I wasn't aware of the OPTIONS from the Official guide...

I'm doing it from the Android/Sdk/platform-tools/ folder

Edit; done UL but had to use -s without it connected and it would wait until I plugged it in, in FB mode as the command was < waiting for 'device ID' >

I unplugged it after nothing showing had done

Rebooted and was greeted with my unlocked warning....Done! [MENTION=4418980]j4nn[/MENTION]:good:

Edit, Edit; The camera was working and all looked the same as if locked...It's late so will post the results tomorrow.

Yes the CN fw is missing Play Store and enhancement not working without your kernel.. [MENTION=4418980]j4nn[/MENTION]

I'll try French 10.24 for the time being as that was what I had planned first of all and just upgrade as when needed or whatever!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] ecksdeeay:
Fastboot is giving me the hanging state as shown when trying to UL BL

Downloading 'boot'.img

with a lot of space to the cursor on the same line and nothing

I unplug it and get

FAILED (Status read failed (No such device))

Edit; same as when done via adb and manually putting into fastboot

Edit,Edit; I got it to download in flashmode

Then tried again in fastboot and it reboots into battery charge
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] OmaeKumiko:
Hello, [MENTION=4418980]j4nn[/MENTION]

I am not e English native speaker, and this thread is too long, so I didn't read all theard except your main post. If there are already someone same with me, I am very sorry.

Xz premium, G8142.

My original FW is 47.2.A.6.30_HK, Pie.

After I flash 47.1.A.3.254_RU, Oreo. ( Already move "persist_X-FLASH-ALL-42E5.sin")

And temp root event count, after several reboots:

After I unlock bootloader:

After I restore TA

I have follow your post, done step 1-9.

Camera doesn't output green photos, though the security result shows error and NOT provisioned.

Some confusion need your help:

1. Pie only show SOMC FIDO Key, Oreo show SOMC Attest Key and SOMC FIDO Key, all don't show Android Attest Key, is that normal on xz premium G8142?

2. Security result still show error and NOT provisioned, is it normal? If it is wrong, what should I do? If it is normal, can I just flash the newest stock FW?

Thanks.::highfive: