yashsingh1/xdaforums huggingface dataset:
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=2735402]SXUsr[/MENTION], you know, my work definitely was not about making a profit.

As mentioned in this post (my answer for a help request to exploit google pixel 2 xl locked down by verizon), just counting the time spent, I could make a better profit taking any low paid job than any bounty promised here.

If I wanted everybody to pay, it would be a heavily protected commercial app with online license providing private keys to obtain key parts to exploit particular target.

As already mentioned, if I did not want to get it done for myself, I would not do it, regardless possible reward.

So thank you to all who donate.

And so you know, there was an exceptionally generous donation recently - a person who donated even 100$ (not making any comment or any link to a nickname here).

Thank you very much. That shows there are few who really understand how it was difficult and what value it brings.

And so you know, your money did not get wasted - I've ordered a brand new XZ1 Compact just for testing and development work.

This will allow me finally to use my phone for daily driver while I can develop or test anything on another one.

I believe this phone is great, possibly one of last with such small/acceptable size, now temp-rootable to get it under full control - not sure this repeats anytime soon.

Working on the exploit and testing downgrades and such, I did not have even a contact list in my phone for something like 9 months!

Think about it, having such phone and instead of using it, working so long on an exploit...

Thanks again to all who appreciate the effort, the value it brings and donate/donated something.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] OzzyFyshmaster:
It surely is a rare breed in today's market. Small, reasonably powerful, unlockable & still has a 3.5mm jack.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] eidglfvsgkj:
could I translate and forward this post to coolapk(another developer forum) ?

I'll Attach a link to this post
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Could you do it here please?

Post a translated post in this thread? And just link to it from external forum?

Please do not mirror the files, let them be downloaded from the first post.

Thanks.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] tramtrist:
No pie ota for me yet....
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] OzzyFyshmaster:
Still trying to temp root...

Since I keep coming back to phone after it's been rebooted & just sitting there, I figured I'd try to wrap the process in a script at least to make it easier to start. I'm working off windows primarily, so this is just a simple batch file:

rr.bat

@echo off
echo __________________________________________
echo %date% %time%
adb shell < commands.txt
echo %date% %time%

commands.txt (combination of OP steps 5 and 6)

cd /data/local/tmp
./renoroot
dd if=/dev/block/bootdevice/by-name/TA of=TA-locked.img
chown shell:shell TA-locked.img
sync
sync

hehe...what I'm pretty sure is gonna happen here is that invoking adb shell through the batch is gonna exit the root shell immediately after creating "TA-locked.img".

The problem being that the last part of OP#6 is

adb pull /data/local/tmp/TA-locked.img

Since this would need be done from separate adb process, not sure how to do that. Could I possible just run another adb process with something as dumb as (pseudocode)

while(1) {
del TA-locked.img
echo Dont stop now
adb pull /data/local/tmp/TA-locked.img
echo kill now
pause 1 minute
}

(not sure if adb would allow pulling a file if dd is still making it.?)

Ultimately, coming back the next day to conveniently find renoroot had successfully rooted it and created the TA partitions and this bit of code above did succeed in pulling it down.

This all depends on looping rr.bat above so that it reconnects & restarts renoroot after renoroot triggers a phone reboot. I've verified that firing rr.bat will successfully start the process without any interaction with the phone.

So yeah, back to question: Is there a way to push the file from the phone shell to the computer rather than pulling from the computer?

I suppose, if the purpose here is to maximize the time spent running the root process, I could have the last command in commands.txt never finish, which would keep the root shell alive for me to return and pull the image. Any thoughts?

Edit:

Further thinking...if the backup image is successfully made, why the need to keep the rooted shell open? Should be able to copy the .img regardless, or is it a permissions issue? perhaps chmod/chown the .img file after the dd or two sync commands? I feel like that's really the only stumbling block to fully automating the temp-root process.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=9546098]OzzyFyshmaster[/MENTION], you may have a look at backup-tools, there I've implemented a way to push files from phone to pc.

But it seems not to be always reliable. For some guys, files got truncated. For others (and me) it worked.

I would not automate the TA backup - I believe transfer of the backup is ok to do by hand.

It could be more useful to automate start of the exploit when the phone reboots...

Particularly the way you have implemented the TA backup - it may be racy. Hopefully it would not happen that the TA image would be pulling right when it is saved via dd, possibly with incorrect or incomplete backup.

Either way, when the shell terminates, but root had been achieved, you may get a root shell again from adb shell just by running

cd /data/local/tmp
./renoshell
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] OzzyFyshmaster:
Ah, I thought about dissecting the backup-send stuff, but hadn't gotten around to it.

Totally agree about automating the backup process being questionably useful...I was most concerned that if I needed to retain root to pull the file, the whole process would have to be done in one shot. ...And that pseudocode loop is most definitely racy (unless dd locks the file from other processes reading it?), but that's what the 'del file' & 'echo kill now' were for...it'd just keep re-pulling the img file, even if it was initially pulled before dd had completed. There is the off chance that I might, through some cosmic conspiracy, catch it in a subsequent iteration where it had gotten root and was repeating the dd process.... I could fix that my making a dynamic image file name... :D

lol...this is all a moot point though, with that ./renoshell command. Yeah, I can just loop renoroot and figure out how to terminate once it detects the presence of the .img file. Then I can come back whenever, "./renoshell", and resume step 6.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] mrsatan:
So if you have already unlocked and lost the keys then this is worthless?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] tonsofquestions:
Don't worry about pulling the locked file off of the phone while still rooted - the chown is so that the adb user can get it - which means that you can do it even after another reboot.

It would also be trivial to just run a single command when you showed up at your computer to find it rooted - because once it's rooted, it'll stay that way until it reboots...

As far as the rooting script goes - in my experiments, I found I couldn't do something like that, since the phone wouldn't launch renotrap (the apk) from a locked state - I had to open up the phone once. But that might be because I had a pin/fingerprint, as I was testing that worked. But if it works for you, that's great!

I suspect your later commands (to make the dd call) won't get sent to the rooted terminal, since it's a subprocess. But again, as long as you end up rooted you can run a couple of commands at the computer when you get back and it's done. I think just wrapping rr.bat in a <run, sleep> loop would be sufficient - because if you sucessfully hit root, it won't terminate, and so stay in the rooted shell.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] OzzyFyshmaster:
[MENTION=1891083]mrsatan[/MENTION]

The processes within this particular thread, I think, would not be of any use if the keys are gone from the target phone. This is definitely to back them up prior to loss.

I may be wrong, but I was under the impression losing the keys was not actually a showstopper...that there were ways around it...I'd made the decision to an xz1c based on that understanding. For my part, it was dumb luck to have found this thread before going through with the unlocking anyway.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] OzzyFyshmaster:
[MENTION=4403504]tonsofquestions[/MENTION]

Oh wow...I hadn't even looked at the renoroot script...so if I understand you correctly...renoshell doesn't terminate on it's own it'll just hang out...and whatever looping I've mechanism I'd created with the renoroot script is on permanent pause until I manually come back and...I dunno...type 'exit' or something? (Just guessing here since all I've seen thus far is the the "please wait...", "Events-Overwrites", <reboot> :D)

I think that probably is the likely culprit. I can start the batch with the phone untouched since (re)boot and after a few minutes swipe the screen (zero security), open up renotrap and see the event count well into the tens of thousands.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] lilbrat:
It's only worthless as you said if you have already unlocked you BL.. to those who haven't, then this is another story..
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] tonsofquestions:
Correct. The thing that renoroot does is give you a root shell ... that's it.

You then have a sandbox to do whatever you want - back stuff up, uninstall system apks, mess up your phone permanently - it's entirely up to you.

I had some long attempts as well, though it's also worth double-checking you're on one of the supported firmwares.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Guys, in my opinion, automating this, is not the best option.

I have the best experience when making sure the phone does not have anything to do - the step 3. prepare your phone - so instead of creating scripts to loop after reboot, it may be easier and faster to wait some minutes after reboot, possibly even check with top about cpu load (not sure if that works nice in windows adb shell). In my opinion it is better to unlock the phone to turn the screen on and make sure it remains always on, clean up all the notification events that show up after reboot, ack adb access as charger...

Mostly I do not need to wait long to get a root shell when doing it manually and just eliminating all possible things that could interfere with the exploit.

Here is few of history of my trials that got green and of which I have taken screenshots:

Events count Overwrites
8336 1
5744 1
10770 1
3273 1
1836 3
1600 1
2849 1
761 1
5199 1
2670 1
6903 1
14402 1
4526 1
5470 1
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] OzzyFyshmaster:
$#@!$#@!...what a waste of time...my build number is 47.1.A.2.3

7

4, not 47.1.A.2.3

2

4 as in the OP:

[IMG="FW screenshot"]https://imgur.com/a/sgjPNSf[/IMG]

imgur.com/a/sgjPNSf​

So yeah, nothing I've done thus far is useful or helpful (other than serving to educate me).
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] mrsatan:
Yeah I had no idea this existed but I have had my phone for quite some time now. Its ok.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] mhaha:
This backup method is exactly what I originally had in mind when I made a pledge that I later withdrew due to the conditions of the bounty that I can't fully agree on.

Although I'm currently happy with stock firmware, I'm even happier that we now have the option to truly unlock our devices without penalizing compromises. Thank you [MENTION=4418980]j4nn[/MENTION] for making it a reality. Donated!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] 4rz0:
Would using safe mode help with acquiring root trough these tools?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] eidglfvsgkj:
does the Video image enhancement work?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=3258505]4rz0[/MENTION], safe mode would not help as it does not allow to start any third party app.

And the exploit needs the renotrap running - it's not there just to show the counters - it's the opposite way, counters are there because the app was needed anyway...

Also normal/ultra stamina mode - they have opposite influence than it would be required - less predictive behaviour due to power saving restrictions.

[MENTION=7995887]eidglfvsgkj[/MENTION], I am not sure how to check that - could you/anybody provide more details?

Updated the first post slightly, the most interesting change may be the added info in the optional secd step:

OTA updates did come, but install to be done on reboot failed - tested by [MENTION=9540710]Unbounded[/MENTION], see post#43 and #44 of the attest key thread please - this may confirm the availability of the SOMC Attest Key which may be the key needed to get sony ota updates (just a guess, not sure what exactly this key is used for).
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Unbounded:
Well, here is another user experienced change in Android pie (SG firmware) in Developer options :

As I already have done TA restore in my XZ1 Compact and as I also earlier that after TA restore we do not lose our Android Assets Keys, Because as far as the TA is restored properly we should always have Android Assets Keys as PROVISIONED while other side because of unlocking the bootloader, SOMC and FIDO keys remains NOT PROVISIONED in pie.

So, here is the change in OEM unlocking status in Developer options for pie, well although Android Assets Keys shows PROVISIONED but OEM unlocking status shows that your bootloader has already unlocked.

No problem, after than I flashed the TWRP followed by flashing of ingnore secd and then rebooted (Those people who are running on pie are not advised to flash TWRP on pie, because [MENTION=4297219]modpunk[/MENTION] is building TWRP for pie and it is WIP, I intentionally flashed it testing purposes)

And so, now the OEM unlocking status changed to "Connect to the internet or contact your carrier".

[MENTION=4418980]j4nn[/MENTION], as you mentioned earlier that sony have changed the bootloader in pie and it is not like the exploitable bootloader that we experiment earlier.

It seems like similar kind of observation is also done by [MENTION=4688377]russel5[/MENTION], as he has mentioned here : https://xdaforums.com/showpost.php?p=78138454&postcount=182

And as far as OTA updates are concerned, I am not worried about them after you clarified the issue of installing the official OTA updates with unlocked bootloared or change in secd because of modified vendor partition.

So, I decided to search about the solution and ultimately came up with "Project Treble" that is based upon "Generic System Image" that separates the Vendor implementation from the OEM's system. (look at the attached image)

As I have mentioned in this thread : https://xdaforums.com/xperia-xz1-compact/help/project-treble-xz1-compact-t3866090

I am not sure if this can really be the effective way to update our devices (especially those who have unlocked their bootloaders ) without messing up with existing OS that is provided by sony.

"Project Treble" can solve our issues because we can get instant update of Security patched and can also keep our devices active for daily day drive ( because OTA from sony will probably mess up the vendor partition and OS in almost all single OTA update, no matter how small the update seems. Because, we, with unlocked bootloader should not rely upon sony anymore)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] ParkourPaul:
I've seen reports that the camera is working on the new Pie firmware for users who have lost their DRM keys. I wonder if that decision by Sony was affected by your work here [MENTION=4418980]j4nn[/MENTION]. Good news for everyone if that's the case. Will still backup in case I ever need it though!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] eidglfvsgkj:
setting - display - video image enhancement - "full screen button"

if it work , the video will appeared in two parts

if it not work , the video will just appeared in one part
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
That is indeed very good news.

But do you know how it is with the other drm features?

For example the above mentioned video image enhancements?

Or the WIDEVINE key (which is useful for netflix for example)? I am afraid at least this will not be provided with pie if already lost due to unlock.

It would also be interesting if someone could verify if the quality of photos/videos on pie is really the same comparing still locked device vs unlocked with lost drm keys.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Unbounded:
Unlocked bootloader, on pie with TA restored.

For display options :

--> video image enhancements = NOT working.

-->Color gamut and contrast :

(1) Professional Mode = Working ,

(2) Standard mode = Working,

(3) Super-vivid mode = Working.

--> Smart backlight control = Working

For sony music options :

--> Audio settings :

DSEE HX = Working

Clear Audio+ = Working

Sound effects = Working

Dynamic normalizer = Working

Accessory Settings = Didn't check this thing, because these are customized settings that will vary by user to user and are dependent upon what kind of external device are you using for your own unique usage purposes. So, please check this thing by own and report back if possible.

[MENTION=7995887]eidglfvsgkj[/MENTION], Thank you very much for figuring the testing method of video image enhancements.

Please forgive me for my earlier observations for video image enhancements which were wrong. That is why I have edited post #283

[MENTION=4418980]j4nn[/MENTION], So , it seems like that, for unlocked bootloader, even though TA is restored properly, the only thing in Pie that DOES NOT seem to work is video image enhancements.

Please, pardon me if I am missing anything else for testing.

And feel free to request anything else for testing.

Cheers
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] russel5:
I dont have TA partition, my bootloader is unlocked long time ago, and i had same [MENTION=9540710]Unbounded[/MENTION]

So your TA keys not working at all, one more you can't use backup tool in settings.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Unbounded:
I backed up my TA by renoroot before unlocking my bootloader, and after that I restored my TA.

So, as result my Android Assets keys is PROVISIONED while FIDO and SOMC are NOT PROVISIONED.

back in 8.0.0 , after successfully restoring my TA, everything seemed to be working like stock, as if the device was never modified.

But, after upgrading to Pie, When I went through primary observations of user experiences, everything seems to work normal.

About backup, for me , cloud backup and manual backup both are working. (Somehow it shows that "No data to backup" for calendar, contacts and email accounts, but excepts that it backups everything. Well I am not bothered about data of contacts, calendar and email accounts because they are well synced in multiple google accounts)

As far as I have noticed, The only thing that is NOT working in Pie is video image enhancement.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] mirhl:
https://play.google.com/store/apps/details?id=com.androidfung.drminfo

Check widevine here.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] eidglfvsgkj:
I'm so sorry , because I still put the translated version in coolapk at there

because in my country not anyone can visit xda easily , in some place just can visit xda by some tool (like v2ray) to pass gfw , but coolapk can visit easily . and my translation is not professional enough to post on xda .

so , that's why I post the translated version to coolapk.com

and I hadn't mirror the files , I put the link to this post and wrote “ must download file from original post , mirror the file is not be allowed "
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] c_tho:
ok.

- downgraded firmware to one of the listed versions

- got temp root

- backed up keys

- unlocked bootloader

- restored keys

- camera works

- flash pie

- everything looks good

buuut cannot seem to flash twrp. flashing via fastboot gives me the following:

Sending 'recovery' (30180 KB) OKAY [0.919s]

Writing 'recovery' OKAY [0.253s]

BUT cannot reboot into recovery, via any method. anyone had this?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] russel5:
Yeah, sure, not working on pie. *facepalm*
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] Unbounded:
Well, it at least wasn't working on my device. (I don't know what the hell has happened to my device!!!!????)

But, it is good to know that it is working for you.

Cheers for your device.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] immelman3:
Hi there, looks like I am a little bit late on this (pie fixed everything with drmkeys?) but do you still need testers for secd? I am on locked bootloader on stock firmware if you need it. [MENTION=4418980]j4nn[/MENTION]
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] strangerq:
Just updated my XZ1C up to pie yesterday, and found this thread today:/

I have locked bootloader, and my keys is safe. Am I be able to downgrade-backup keys-upgrade back and unlock bootloader?

I have wait for this implementation last year, and here it is) Please say that it can become real, [MENTION=4418980]j4nn[/MENTION]

Also ping me if any help needed
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=9560859]strangerq[/MENTION], you can downgrade your locked phone even from the recently released android pie - I just tested (with locked bootloader) to upgrade to pie and then downgrade back to 47.1.A.8.49 with my xz1c.

I was a bit afraid that it might not be possible to downgrade from pie as bootloader was upgraded again, but it is still possible to downgrade (obviously with erase of all data).

Just follow the howto to backup TA, restore it (with the same fw the TA backup was created) and then upgrade.

In my opinion it is still very useful to backup TA - even if camera seems to work the same with pie as never unlocked (it still would be worth to compare quality).

I am afraid the other drm functions (particularly WIDEVINE L1 key) would not work on device with unlocked bootloader where keys had not been backed up.

If not for anything else, backup TA in order to avoid losing an option to use widevine L1 for services like netflix for example.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] russel5:
[MENTION=4418980]j4nn[/MENTION] do you test camera when you downgrade your locked phone?

In my home forum, one man write that he downgrade from pie to oreo on pure stock phone without unlocked and other thing, camera not worked (green Pic) and in service menu he saw that security blobs have unknown status. When he come back to pie all works again as it should.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] c_tho:
on this: i notice that i can't download the netflix app from the play store ("not compatible with your device") but if i obtain and install the apk via other means it works fine? so.... i guess google won't give it to my cos my bootloader's unlocked but the actual app is looking at the widevine key? or something?

edit: this isn't a problem obviously, just pointing it out for interest's sake. :)