yashsingh1/xdaforums huggingface dataset:
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
If you need to extract something e.g. sin or img or elf use this easy double click tool, it will extract things to system folder on your windows machine.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] pbarrette:
[MENTION=4418980]j4nn[/MENTION].

Based on the abl image and other research, I think the QFPROM is mapped to a memory region at 0x780000.

Is that readable? I suspect that some bits will be readable at different ELx levels and that others will need higher ELx levels.

I think it might be worthwhile to try to read and interpret all the QFuse information we can get.

In the ABL, it looks like there's a number of branch instructions that point to a location stored in a register.

My guess is that the register is pointing to an instruction address from a TZ applet loaded from the XBL.

So we're basically seeing external function calls in the ABL that look like internal calls, so it gets rather difficult to follow.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4782500]pbarrette[/MENTION], I have the same impression about those indirect/register calls - most likely they are kind of dynamic linking of ABL to UEFI modules provided by XBL, like usb stack, verified boot, etc. function calls. It would be great to somehow resolve the linking, else we are left only with few hints provided by logs there.

I've patched kernel's /dev/mem driver to remove some checks to have kernel-unrestricted physical memory access, but it's not easy to use, because when reading from wrong address, it crashes anyway.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4120742]munjeni[/MENTION], do you think it could be possible to reconstruct full sony stock fw flash while having the previous version and a fota update package for the next version?

Or that the fota-ua simply updates only what is needed, so there is never created a full .sin for each partition?

I would guess the second case unfortunately, while the first case could be useful to be able to 'download' not anymore available stock firmwares...

It still could be possible to reconstruct full partitions images but without the .sin signatures, so not useful for normal flash mode flashing.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
Search google for "redbend fota solution", fota-ua contain redbend strings and I'm believing sony use red bend software fota solution. Stil not understand which format is on there,

On folowing log https://pastebin.com/DqfuN8Vp at the end of log some lines rb_file_open which I can not tell if those files is patched or those files is extracted, thats unclear, but on this link https://connectedworld.com/wp-conte...per_RedBendSoftware_OTAUpdatingSimplified.pdf (in case red bend software solution) format of the ota package is an "delta" format:

which I can't get what that mean or how it looks, I'm believing ota just doing similar thing to git bin diff?

I'm unable to make fota-ua working on virtual machine, made some partitions like system, xbl, cache and data inside newly created gpt disk, but getting error :

Info [619] Package meta-information OK
Info [132] Updater continues in 3 state
WARN [112] Unknown storage type:N/A
ERROR [526] Cannot open device '' (errno=2)
ERROR [64] Couldn't create a disk info
ERROR [183] Unable to create disks info
ERROR [148] dsi is NULL. Double free?
ERROR [343] Unable to destroy disks info
WARN [112] Unknown storage type:N/A
ERROR [203] LUN Error, failed to open device .
ERROR [128] LBM Error, Open LUN for system failed.
WARN [45] LBM partition collect Error, system
ERROR [661] Error from ua_updater_start

This is due filesystem I have created is not one like fota-ua expect. Need to create new disk with excatly the same layout with 3 LUNS , the same guids ...etc, but can't do that since I don't have that device and I lack that informations to be able to do that. I'm believing it will work in case I create the same disk structure. Anybody help me to make the same disk layout (with just smaller size)?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
I have runing all this on vmplayer which runs ubuntu 16, on that ubuntu virtual machine I'm runung another virtual machine using qemu efi and that seccond virtual machine is ubuntu 16 too but aarch64. Qemu support parameters on virtual disk e.g. logical_block_size and physical_block_size, I'm believing I can set it to 4096 to emulate that 3 virtual disks (3 luns) on qemu to make fota-ua happy . This is aarch64 qemu ubuntu image I'm using right now https://gist.github.com/oznu/ac9efae7c24fd1f37f1d933254587aa4

Is block.map unchanged between reboots?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
It does not change between reboots of thesame fota update trial. Like between the 1st main kernel boot doing the veryfy, it stays the same to next fotakernel boot, ...

But I guess when you redownload and start it again, it is changed.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4120742]munjeni[/MENTION], have a look at twrp sources at

https://github.com/omnirom/android_bootable_recovery

- it seems twrp implements something from the fota update as defined in the files we have captured from cache 'recovery' subdirectory from fota-ua process. For example:

uncrypt/uncrypt.cpp:static const std::string CACHE_BLOCK_MAP = "/cache/recovery/block.map";
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] pbarrette:
[MENTION=4418980]j4nn[/MENTION], [MENTION=7543898]Blu-eyes[/MENTION],

FYI - The 0x9A9 TA unit is identified as "TA_KERNEL_CMD_DEBUG_MASK" in the XBL's SobaDxe.pe EFI module.

It also appears that if it's not set to some calculated value, it disables the "DebugPort".

It appears to disable the debug port by calling another EFI module to change the config on GPIO 4 and 5. I recall seeing somewhere (Sony kernel sources on github?) that GPIO's 4 and 5 are the serial port. So it looks like that's the actual, on-board serial header and likely not switched through to USB.

That said, there's also a function that performs "pm_schg_usb_get_typec_status" and returns "soba device is found" after various other checks.

The "sub_" that checks TA unit 0x9A9 has a disconnected "loc_" function which calls this function.

So there's probably some kind of special Sony debugging functionality exposed via the USB port, but it's obfuscated in some way.

---------- Post added at 10:58 PM ---------- Previous post was at 10:38 PM ----------

[MENTION=4120742]munjeni[/MENTION]

Readlinkat: https://linux.die.net/man/2/readlinkat

"If pathname is relative and dirfd is the special value AT_FDCWD, then pathname is interpreted relative to the current working directory of the calling process."

Looking at your "Unknown storage type:N/A" error.

Were your partitions created by using the partition tables from the LUNx SIN files?

It's possible that the fota-ua binary is using the partition GUIDs to determine the partition types and is complaining that it can't determine if the partition is correct or not.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4782500]pbarrette[/MENTION], concerning ta 0x9a9 unit, please see post#1125 and following post, where it has been tested - it indeed enables serial console in kernel, but most likely not on usb cable.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] pbarrette:
Yes. I've been through the thread.

Though I don't think anyone has yet actually tested it with a cracked-open device and test points located.

I mentioned the above because it looks like it also enables what it calls the "Soba Protocol" and that appears to operate over USB.

I think the same unit is controling the enablement of both functions.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Please note, the TA dump you have from me is from XZ2 phone (TAMA platform), while my recent posts/logs are from testing fota update on XZ1 Compact (YOSHINO platform) - completely different Snapdragon chipset, from fw point of view the big difference is that TAMA is A/B boot device, while yoshino is single slot boot device - big differences in partition layout and boot process. Also tama is Verified Boot 2.0 while yoshino is VB1.x.

So the mentioned TA unit - it may be the same with XZ2 but it may not...

The 0x965 (2405) ta unit had value of 0x00 before clicking on fota install. Now on runtime it still has the 0x00 value.

-- edit --

Concerning partitions - in my opinion, the best case for you would be to use stock fw to replicate the disk.

If you create the same size, dd the extracted partition into it (note please, it is probably only the primary copy of GPT, you may need to manually dd the backup copy to the end of the disk - check wikipedia how gpt is stored on disk).

If you sinunpack all the .sin files into raw images (not android sparse images) you can dd those images into their respective partitions.

That will restore all filesystem metadata, like fs uuid or label.

The part-uids would have already been restored from restoring gpt from image.

Do not make changes in the gpt on disk with any tool to keep the original checksums.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
running LOS16 on XZ1c with still locked bootloader

Let me show you some black magic;)

https://youtu.be/TAhSNjxZfhc

FAQ:

- bootloader is still locked, not ever unlocked (I am not aware of any method to do BL re-lock)

- similarly it is not possible to hide BL unlocked warning screen on boot

Other than that, I would not provide any details yet, but you may freely speculate.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] pbarrette:
[FREE SPECULATION]

1 - There's a script or executable on userdata that is called by the stock ROM. You gained temp root and replaced that script to load your own image.

2 - You've managed to exploit ABL.

3 - The Sony signatures don't sign the entirety of the data, just the program headers, so you've replaced a function beyond the signature coverage in one of the images.

[/FREE SPECULATION]

The DXE files are loads of fun, by the way.

UEFI firmware drivers don't have named entry points. Instead, they register GUIDs with the system at load time.

So to call a function in an EFI module, you have to know the GUID of that function.

Have a look here: https://github.com/Freddrickk/ida-efiutils

This one is interesting too: https://github.com/jethrogb/uefireverse

Unfortunately, although there are some well known GUIDs that show up in the ABL and modules, most of the GUIDs are not identified.

Some are probably Sony unique, but others are probably Qualcomm defined.

Basically anywhere you see a "BLR Xd" instruction, you can bet that it's calling some GUID named function in another EFI module.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4782500]pbarrette[/MENTION], thank you for the 'soba' related usb serial analysis and the tips for resolving uefi dynamic calls. Not that fun I am afraid.

I am still very interested to find some TZ exploit - the attack vector through XFL could be a good candidate.

It would be great if you and [MENTION=7543898]Blu-eyes[/MENTION] (or anybody else willing to) could help me with reverse engineering of that TZ related stuff possibly finding a hole somewhere?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
J4nn, done! I have patched fota_ua and it started working. fw_backup in tmp folder is created but actualy with 0 bytes, its probably because I have used first ota package you posted (update_package_renamed), when I get some more time I will try with seccond one. One of the good things this finaly started with some small progress :) Report log is not created have no idea why but here is strace log https://pastebin.com/vmXdhnMu

My block.map file may be a reason for fw_backup with zero size

/root/update_package_renamed
420897111 4096
1
0 102759

fota_ua read directly from update_package_renamed instead of uncrypted blocks, I think its ok since fota_ua reports that package is ok, didn't created log now but my hope on success with things.

Edit:

Using your seccond ota package (fota-blockfile-trunc) giving this error:

============================== ua_package_context ==============================
p_ctx->package.data: 0xffff9bc99000
p_ctx->package.data_size: 420897111
p_ctx->package.signature: 0x0
p_ctx->package.fota_ua_executable: 0x0
p_ctx->package.fota_ua_executable_size: 0
p_ctx->package.meta_info calculated length: 0
p_ctx->package.meta_info:

[COLOR="Red"]p_ctx->result: 5 (PACKAGE_WRONG_HEADER)[/COLOR]
p_ctx->block_map_file: 1

I'm using ua_status file (03 00 00 00 09 00 00 00) because than my vm not rebooting and package skip verification
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
A donation has been received on 2019-10-24 18:59:13 UTC.

Thank you.

A note referencing the "running LOS16 on XZ1c with still locked bootloader" post has been attached to the payment.

Please do not get excited too much - the exploit is powerful, but also extremely dangerous and difficult to apply - if you considered renoroot exploit too complicated, think again - that has been very simple to apply (if not considering the randomness factor due to exploiting a race condition) and considerably safe (and still there has been one hard brick).
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4120742]munjeni[/MENTION], please find content of /cache/recovery before fota-ua start in fotakernel for the main install phase attached.

I can confirm that the content of block ranges with fota update package is identical to the downloaded blob, if the update/system has been previously successfully verified. If the verification fails, block ranges are invalid (that's why I've seen different content there).

Have sent you also android properties from the fotakernel boot for the main install phase. All this should hopefully define a fine state for the installation start.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
So blob update_package_renamed (first one you posted) is ok? Any other blob not working for me.

ua_status seems when 0300000000000000 than fw_backup getting created, I'm done yesterday but somehow I have meeed my lun0 block device and now getting some errors about crc mismatch and something about main header is not 0x92, investigating. Do you have idea how can I write -33 gpt header? I'm tried with dd but dd can't go on that offset where skip is ((total_clusters - 33) * 4096) and bs is 4096 or I'm doing something wrong with dd. Only gdisk is able to correct seccond gpt header but seems its not working as expected.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
Here is my analyse https://mega.nz/#!cY4zwSZL!vy2ESvHkqQmj9kFabbfE1WnXEHESQeazvb_Lw5vpqmM, my hope somebody find something which I not see and which I unable to understand. My hope somebody explain what it does. In my opinion fota-ua is just for "verify" & "prepare" purpose, the rest is done with recovery, I have spent many days on this and this is time for to take a little break from this, see you later!
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4120742]munjeni[/MENTION], in my opinion, fota-ua handles it all - verification and the actual installation too.

As described in PM, for installation you need what I've posted in the xz1c-fota-before-start-main-install.zip plus possibly the android properties sent by PM, i.e. only the ua_status having '03 00 00 00 00 00 00 00' is relevant here.

I've sent you full strace (660MB compressed to 74MB) of successful installation - maybe you can use that as a reference to see what is missing in your setup?

In any case thank you so far for trying to analyse that - it would be very interesting to find out how fota package is encrypted or even be able to decrypt it and possibly use it to construct new full flash from the previous one.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] pbarrette:
[MENTION=4120742]munjeni[/MENTION],

A quick look at your archive tells me that you've got partition data in the "4__" folder in the "/recovery/fota/tmp/fw_backup" file.

At offset 0x1800 in the file, you've got a gzip header.

Strip it down, and you can extract a ~40MB file from that which look like valid disk blocks.

I'm guessing that the fota package contains:

1 - A list of partitions affected.

2 - A list of blocks in those partitions.

3 - A signature, or set of match data to verify what the original blocks should look like.

4 - New data for those blocks.

So it's possible that you're getting a mismatch now because it worked against 1 partition.

So now that partition doesn't match what it expects for original data.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
Good news, fota-ua sucesfully updated some of my virtual partitions, I'm now at the restart of my analyse, figured out some good things, will post details soon!

Edit:

Part 1:

Some lights on this and some things I have figured out

Package verification:
=======================================================
OTA package is firmware customisation specific, so fota-ua check crc of the e.g. system part
from update_package_renamed (see picture -> https://ibb.co/Kryzjy9)
and crc from system partition (curently I'm not figured out how it is done), and if crc didn't match
you getting RedBend error in report log file inside /cache/recovery/fota folder:

Info [156] RedBend: Source signature 0x22dfc18e and CRC 0x92f0af85 mismatch
ERROR [489] RedBend error: 0x8000013E, code = 0x13E (318)

In short, you can't use OTA update on diferent firmare customisations. Even if you mount system rw (thanks@j4nn)
system partition will get modified and OTA package will not work? For the rest I will make archive with README.txt and will write everything I have found with included strace log and dump of the recoery folder including its logs.

Just for test I have changed crc inside update_package_renamed, this is what I have getting:

ERROR [164] Security signature verification failed (result: 3)
ERROR [178] Security Manager error
ERROR [605] Security Error in package

Thats tells that update_package_renamed can't be modified. It might have two meaning,
fota-ua check signature of the sin header from the fota_recovery_renamed (which can be a good thing for us to figure out how to calculate that)
but also might mean another verification mechanism e.g. another crc of the whole package.

I have sucesfully updated FOTAkernel partition with fota-ua but system partition is not updated because RedBend giving crc error since my system partition not match update package. I must use the same system.sin package as j4nn to be able to pass RedBend crc verification. [MENTION=4418980]j4nn[/MENTION] can you please upload your system.img ?
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4120742]munjeni[/MENTION], please try to use sinunpack tool - it's based on your newflasher merged with android sparse image conversion tools project.

With that you should get exact same .img files from .sin files as I do.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
But its the same https://github.com/munjeni/anyxperia_dumper , is not? Except I have removed libsparse and added my own implementation. When I extract sin 7z reporting valid ext4 image, so I think extracted ext4 is valid, is not? Ok I will try.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
Unfortunately no tool is perfect - your anyxperia_dumper unpacks nicely even partition .sin images and works great for any kernel images.

But I did not know about it that time, having unsin.exe for win only has not been acceptable for me in linux.

In any case I've upgraded libsparse, that was needed for some .sin files (possibly tama, not sure anymore) and it works for big .sin sparse images correctly - and due to libsparse you get linux sparse files on filesystems that support that (i.e. 5GB big file may not need to occupy 5GB of disk space if it contains like 2.5GB of real data for example, otherwise you see no difference).

-- edit --

Cleaned up branches of sinunpack on github.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
I'm compile tool right now, going to do one more atempt on system partition

Edit:

Its confusing, I must check this

With sinunpack tool: b5a38c3490f214a9ef52dcb819ba3b91 /data/unpacked/system.img

With anyxperiadumper: c0424cc822ec0d05657e6ae7b421829b /data/system_X-FLASH-ALL-C93B.ext4

From your pm: b5a38c3490f214a9ef52dcb819ba3b91 *system.img

Edit:

you are right, something is changed is sparse image format, I found it Oct 15, 2019 when system.sin from G8441_Customized CE1_1310-7123_47.2.A.10.107_R1C had some special chunks, I thinked sony done something tricky on that and fixed it but sims sparse implementation which I done in tool need to be realy updated :( Strange thing tool produce now valid ext4 image. I will need to compare now ext4 from anyxperiadumper and sinunpack tool, on first look its evidently with two diferent md5 sums, something is missing :)
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
[MENTION=4120742]munjeni[/MENTION], I know, as said, no tool is perfect, but I believe sinunpack does it right. Try it now with your fota, maybe it gets further.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] munjeni:
System pass verification! Now I'm unsure if my sin tool failed or redbend errors is due system partition was in read write mode, I will test that next day. But one thing for sure is if sytem,oem,vendor partitions was in rw mode redbend will report error and I'm believing thats a thing how sony doing check rooted device. When partitions is modified fota no more working thats for sure.

Edit:

All passed verification in 06 00 00 00 09 00 00 00 mode! Time for final step, step 3! I will update our post https://xdaforums.com/showpost.php?p=80690663&postcount=1366 with what I have found new.
[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] j4nn:
^ both. Many times mentioned to users of my BL unlock hiding kernel with fota update support (if booted via 'fastboot boot') that it needs unmodified stock fw and even a single mount in rw mode without any actual change in files makes changes in the partition.

Unfortunately many zip files or even twrp may mount a partition in rw without any reason.

In case of incorrect .sin unpack -> using "corrupted" .img with fota in qemu would fail too obviously.