yashsingh1/xdaforums huggingface dataset:
[Sony Cross-Device General] osm0sis:
[MENTION=4120742]munjeni[/MENTION] I found a couple of sin files that don't seem to be handled by your tool correctly, stating only "is usuported (sic) format!"

Attached so hopefully you can figure it out. :)

[ Attachment removed ]
[Sony Cross-Device General] munjeni:
Hi, only sin version 3 which started after z1 line! Older version like xperia x10 use and etc I not going to support. Version 3 ADDR variant (z1 - z3) and version 3 hibrid (which use mix of ADDR and LZ4 is fully supported (z5-etc)), including all other version 3 variants.

This is example header of the sin version 3 (first byte tells about sin version)

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000 [COLOR="Red"]03[/COLOR] 53 49 4E 00 00 04 00 00 00 00 20 00 00 00 02 .SIN....... ....
00000010 00 00 00 00 00 00 00 24 00 02 A7 CC 90 07 A6 01 .......$..§Ì..¦.
[Sony Cross-Device General] munjeni:
I will add support for version 1 & 2!
[Sony Cross-Device General] munjeni:
New version is out, - 01.May.2017 -> support for sin version 1 & 2, some improvements for ver 3
[Sony Cross-Device General] munjeni:
It would be great if somebody comfirm this https://xdaforums.com/crossdevice-dev/sony/dev-xflasher-command-line-flasher-t2986634 working on newer devices! Its fully working on old models like z1 and some guy tested on xzs (only cache) and it worked, it will work the same like z1 I am pretty sure, but need confirmation :)
[Sony Cross-Device General] osm0sis:
Fantastic!! Great job! Works perfectly. :D

Would it be possible to get Android compiles for x86, mips, arm64, x86_64 and mips64, as well as a Linux x86_64 compile? I'd love to include your excellent tool in the next version of AIK and then AK2, but I'd need those native builds. :fingers-crossed::)
[Sony Cross-Device General] munjeni:
Allready have arm32, linux32 and windows32 version, and it working on 64bit too! I told you that android toolchain which you uses have messed that USE_FILE_OFFSET64 and LARGE_FILESOURCE64 , I simply can't get it working under 32bit matchine since it didn't add support for LONG LONG, fopen64, fseek64 and etc, it simple can't compile that binary. Did you tried arm and linux version? I have tried all 3 and it working on 32bit and also on 64bit machines without any problem (yes I know, it runing in compatibility layer, but who cares, the most important thing is it is working, right?). I can't publish source code because of many reasons which happened to me on this forum, simply promised to myself that I will no more publish any source code. Compiler which I found working for arm is gcc-linaro-5.3.1-2016.05_arm-linux-gnueabi but I tried one from google which didn't worked
[Sony Cross-Device General] osm0sis:
Ah right, I recall this was due to the LFS needed to handle the system.img sin files, right?

Would it be possible for you to make a fork that is only for boot/recovery images, so LFS wouldn't be required, and building with the Android NDK cross-compilers would be easy? Unfortunately I can't depend on compatibility layers.
[Sony Cross-Device General] munjeni:
That would be possible but I no want to redesign whole program which uses long longs, thats lot of work sorry
[Sony Cross-Device General] osm0sis:
Hmm that's too bad, but I understand time constraints. If you ever do find the time, I hope you'll reconsider.

Would a mips build be possible at least, to represent the 3 broad Android architectures?
[Sony Cross-Device General] munjeni:
Can you give me link to the toolchains which you use?
[Sony Cross-Device General] osm0sis:
The exact ones I use won't be any use to you since they're Cygwin-to-Android cross-compilers. That said, they're just the basic NDK stand-alone toolchains, so I can walk you through how to easily build your own for your (Linux, probably?) setup.

Grab the latest NDK for your system from here: https://developer.android.com/ndk/downloads/index.html (currently r14b)

Unpack it, then to make a stand-alone cross-compiler for any Android supported architecture (arm, arm64, x86, x86_64, mips, mips64) you just use the command structure here: https://developer.android.com/ndk/guides/standalone_toolchain.html#creating_the_toolchain

Looks like the main tool is Python-based now (and much simpler than it used to be!), with the shell script as a wrapper, so I'll cut out the middle man; to (re)generate tarballs of the ones I currently use, I would do the following:

python make_standalone_toolchain.py --arch arm --api 9 -v --package-dir ~/;
python make_standalone_toolchain.py --arch x86 --api 9 -v --package-dir ~/;
python make_standalone_toolchain.py --arch mips --api 9 -v --package-dir ~/;
python make_standalone_toolchain.py --arch arm64 --api 21 -v --package-dir ~/;
python make_standalone_toolchain.py --arch x86_64 --api 21 -v --package-dir ~/;
python make_standalone_toolchain.py --arch mips64 --api 21 -v --package-dir ~/;

I always shot low with target API for broad device support, but header defs and library support in the NDK has generally gotten better with each API revision so where you were having issues, you should probably shoot higher - maybe 21 across the board or even newer. You might want to experiment with the different STLs if C++ (gnustl, libc++, stlport) or the new unified headers as well.
[Sony Cross-Device General] munjeni:
Thats waste of time with that toolchain since a lot of diferent api levels + almost imposible getting static binary, I will try to build http://github.com/buildroot it have a lot of toolchains which can be built by building buildroot :) If I get it success I will try to build tool...
[Sony Cross-Device General] osm0sis:
API levels are easy. Like I suggested, you can just go for 21 (or higher) since it's more complete. -static -s works fine for me as well, but yes, whatever you prefer. Thanks for looking into it further! :)
[Sony Cross-Device General] munjeni:
You need mips litle endian, mips64 litle endian?

Edit:

compiled booth variants, hope you have now all need. LFS is working on arm32 and .exe, on all other .64bit probably working, but on all other other I don't know since didn't tested! If you need PIE just simp[le open binary with hex editor and change 17th byte (0x2) to 0x3, solution is hackish but who cares. Download link in first post
[Sony Cross-Device General] osm0sis:
Amazing! Thank you very much for putting in the extra time for this! :D

Only one architecture missing.. can you please also do Android x86 and Android x86_64? :fingers-crossed:

Edit: mipsel and mipsel64 are what Android supports in the NDK. :good:

The mips compiles seem a little small though, usually they're the largest of the 3 broader architectures.. how'd you manage that? Are they dynamic somehow instead of static even though readelf doesn't list any dependencies? :confused:
[Sony Cross-Device General] munjeni:
android x86 and androidx86_64 just use linux32 and linux64 for it! Mips compiled staticaly, to get small size I used strip with command strip and it gone almost half of the real size. All files from archive is staticaly linked, no dependencies! :) Only dynamicaly compiled is .exe but it will work on any woindows after xp including xp, didn;t tried on earlier versions
[Sony Cross-Device General] munjeni:
Tested mipsel 32 bit for LFS, its fully working! So all other will work too! Time for moving attachment to the first post :)
[Sony Cross-Device General] osm0sis:
Oh my god. You're right! How have I wasted so much time on x86 and x86_64 cross-compilers?! :silly::laugh:

Edit: Haha okay, looks like sometime it's not 1:1, some of my Android x86 compiles give "Illegal instruction" errors when I try to run them on Linux i686. ;)

Hopefully all is good with kernel_dump on Android. :fingers-crossed:

Yes! It all looks great! Excellent work sir!! I'll have your genius SIN support added to AIK v2.8 with proper credit for you in my OP, commits, and authors.txt of course. :highfive::victory:
[Sony Cross-Device General] munjeni:
Use better and more generic toolchains like Codesourcery since it is designed to support more embeding linux machines, don't waste time with bugy toolchains which even don't have lfs :)
[Sony Cross-Device General] osm0sis:
[MENTION=4120742]munjeni[/MENTION] I noticed for the sin v2 "kernel-xp.elf" I supplied you kernel_dump seems to add the SIN header back to the output file as a "certificate" (according to kernel_dump) which according to readelf seems to be corrupt. Is this intentional?

ELF Header:
Magic: 7f 45 4c 46 01 01 01 61 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: ARM
ABI Version: 0
Type: EXEC (Executable file)
Machine: ARM
Version: 0x1
Entry point address: 0x8000
Start of program headers: 52 (bytes into file)
Start of section headers: 0 (bytes into file)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 4
Size of section headers: 0 (bytes)
Number of section headers: 0
Section header string table index: 0

There are no sections in this file.

There are no sections to group in this file.

Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x001000 0x00008000 0x00008000 0x4ba5e0 0x4ba5e0 0
LOAD 0x4bb5e0 0x01000000 0x01000000 0x124f98 0x124f98 0
NOTE 0x5e0578 0x00000000 0x00000000 0x0015a 0x0015a 0
<unknown>: 214 0x0000b4 0x00000000 0x00000000 0x00430 0x00430 R 0x4

There is no dynamic section in this file.

There are no relocations in this file.

Dynamic symbol information is not available for displaying symbols.

No version information found in this file.

Displaying notes found at file offset 0x005e0578 with length 0x0000015a:
Owner Data size Description
readelf: Warning: Corrupt note: name size is too big: 68636163
readelf: Warning: note with invalid namesz and/or descsz found at offset 0x0
readelf: Warning: type: 0x3d796369, namesize: 0x00000000, descsize: 0x6c6f7065

Edit: From what I can see with other files it looks like the cert is usually appended in the cmdline (NOTE) section, and doesn't have its own program header?
[Sony Cross-Device General] munjeni:
Thats just cosmetic, the most important thing it extracts all parts of the elf, processing some elfs gives wrong name, but who care, important thing is it working for what we needs - for extracting kernels. Can you post that kernel elf?
[Sony Cross-Device General] osm0sis:
Here you go sir! Thank you for your continued fantastic work! :)

This is the only file I've seen where the certificate somehow gets added as a malformed program header; in my testing with others it's usually attached to the NOTE section where cmdline resides. Not sure if that's on your end or Sony's, but notable nonetheless!

[ Attachment removed ]
[Sony Cross-Device General] munjeni:
Yes thats true! I have harcoded seek to the command line offset since I found that cmdline do not have PH!

Program Header count : 4
PH[0], type=1, offset=0x00001000, virtual=0x00008000, phy=0x00008000, size=0x004BA5E0
PH[1], type=1, offset=0x004BB5E0, virtual=0x01000000, phy=0x01000000, size=0x00124F98
PH[2], type=4, offset=0x005E0578, virtual=0x00000000, phy=0x00000000, size=0x0000015A
PH[3], type=558778707, offset=0x000000B4, virtual=0x00000000, phy=0x00000000, size=0x00000430
0. Dumping outk/boot.img-zImage
1. Dumping outk/boot.img-ramdisk.gz
2. Dumping outk/boot.img-dt
3. Dumping outk/certificate
Seeking to cmdline address = 0x000004EC
4. Dumping cmdline to outk/boot.img-cmdline
Done.

Thats definitelly malformed elf, cmdline somehow gets dumped into boot.img-dt and also seek to commandline gives null byte for command line :) I think somebody who made that .elf acidentally changed order during mkelf.py e.g. instead of mkelf.py zimage ramdisk dt cmdline he done it e.g. mkelf.py zimage ramdisk cmdline dt , or probably it gets corupt during signing, have no other idea but definitelly its strange
[Sony Cross-Device General] munjeni:
Seems xperia xz premium have new sin format and tool is not abble to unpack it, anybody confirm??
[Sony Cross-Device General] loukky:
unsupported this sin files that download from sony xz premium
[Sony Cross-Device General] munjeni:
Thanks, I see, thats sin v4 which is curently unsupported. I will add support soon!
[Sony Cross-Device General] osm0sis:
Can one of you post the kernel.sin so I can add it to my collection and take a look as well? :)
[Sony Cross-Device General] munjeni:
Here you go -> http://munjeni.myiphost.com/kernel_X-FLASH-ALL-42E5.sin

New kernel_dump tool is out! Sin version 4 is supported right now. Also done some cleanups for 64bit things, hope all is ok now. Enjoy!
[Sony Cross-Device General] osm0sis:
Hmm is it just gzipped?

Edit: Interestingly enough the final output is a standard boot.img signed with AVB.
[Sony Cross-Device General] munjeni:
Yes, gzip ustar archive. Sony made new flashing format and also totaly new flashing protocol

Sorry, uploaded again new version of the kernel_dump, forgot to change one function for win version, its ok now
[Sony Cross-Device General] munjeni:
I am preparing to port new sony flashing protocol and found that cms is cert file which is uploaded first, than bin is uploaded... more info you can see in my xflasher thread! Waiting one usb sniff log to start working on it :) Protocol is very simple in comparation to old s1 protocol, it will be easy porting
[Sony Cross-Device General] osm0sis:
Ah, .tar.gz! So it's basically like Samsung's format for Odin then, with a cert instead of md5.

Well that's easy enough, people can handle .tar.gz on their own once the word gets out that's what the new format is actually. :) :good:
[Sony Cross-Device General] munjeni:
ustar-format tar archive and gziped, ustar source you can see here https://www.cs.usfca.edu/~benson/cs326/pintos/pintos/src/lib/ustar.c

Anyway 7z is enought to extract sin v4 :) Since I not depend on 7z I have included sin v4 support on kernel_dump using ustar
[Sony Cross-Device General] osm0sis:
Exactly. tar -xzvf also works just fine as well. :)
[Sony Cross-Device General] loukky:
i think some files are Lost when i unzip v4 sin files
[Sony Cross-Device General] osm0sis:
Nope it's just boot.000 and boot.cms; boot.000 is a standard AOSP boot.img with an AVB signature. I just added AVB support to my Android Image Kitchen a couple days back, so you can check that out if you want to unpack the boot.img and ramdisk. :good:
[Sony Cross-Device General] loukky:
i upload the v4 sin file and v3 sin file,plz test.

https://drive.google.com/drive/folders/0B1ZdurJvmZAaNXdrcmJoemVSbDQ?usp=sharing
[Sony Cross-Device General] osm0sis:
I don't know what you want me to do with those. Rename the sin files to .tgz and open them with 7zip, or even WinRAR works. That's all there is to it.
[Sony Cross-Device General] munjeni:
None of that is working! There is sparse file format like this one https://xdaforums.com/showpost.php?p=72591107&postcount=81 (oem-cn(v4).sin) , it contain 000, 001, 0002 and it is sparse image... see magic byte 0xed26ff3a at start of first file :) Seems I am not done with tool :)
[Sony Cross-Device General] munjeni:
Thanks! I will add support!
[Sony Cross-Device General] munjeni:
New version is out! Full support for sin v4, enjoy!
[Sony Cross-Device General] munjeni:
Made mips 64, mips 32, arm 64, arm 32 , linux 64, linux 32, win version, enjoy!
[Sony Cross-Device General] Surge1223:
that's a horrible method. It won't use ndk bionic headers, nor will it use c++ headers from ndk repo. It's a waste of time not to use bionic and create a toolchain. When you compile aosp it builds most of the static libs for you that you can use to create a bootstrap gcc for native arm and arm64. You only really need lc, lgcc, lm, ld, lz, and lc++.

The effort is defs worth it, plus glibc won't be mixed in at all. Have faith

don't use buildroot, just configure binutils first and compile it for the ld needed later then use binutils libiberty instead of gcc's for the stage 1 bootstrap (which is static)
[Sony Cross-Device General] munjeni:
I no want to waste my time with bionic since I can't get LFS while using 32bit aosp toolchains, it always give me size of the long long like I am using long! Rather than wasting time with it I'm using toolchain which is working (uclibc-crosstools from buildroot). I realy don't care, the most important thing for me is to make my aplication working for needs, and it works!
[Sony Cross-Device General] osm0sis:
[MENTION=4120742]munjeni[/MENTION] would you be able to make Darwin (macOS) builds for i686 and x86_64?
[Sony Cross-Device General] munjeni:
Hi, I never used darwin and macos and realy have no idea about its reguirements, even I have no idea what is diferencie between linux and darwin or etc, realy. I can't make something for something which I never had on my hand. Did you tried maybe its allready working on that os?
[Sony Cross-Device General] osm0sis:
Nope, different binary format: Mach-O instead of ELF. I'm working on a Windows -> Darwin cross-compiler to extend AIK-Linux to work on macOS. You could probably do the same from Linux? Would be nice to be able to support all platforms for all the binaries. :fingers-crossed:

I've been working away at using these:

https://github.com/diorcety/crosstool-ng/tree/darwin-target

https://github.com/mingwandroid/ctng-firefox-builds

https://github.com/AaronWallace/MacOSX_SDKs

This also looks promising: https://github.com/tpoechtrager/osxcross
[Sony Cross-Device General] munjeni:
Thats litle endian right?
[Sony Cross-Device General] osm0sis:
Yeah, Mac's are all Intel now, so they should be LE. The old PowerPC was BE but there's no need to support that, Apple hasn't since 2009. :good:

Sources:

https://developer.apple.com/library...ptual/CFMemoryMgmt/Concepts/ByteOrdering.html

https://en.wikipedia.org/wiki/Darwin_(operating_system)#Release_history
[Sony Cross-Device General] osm0sis:
Yuss, [MENTION=4120742]munjeni[/MENTION], I succeeded in making a cross-compiling toolchain with the osxcross repo! :D

All I had to do was add the required MacOSX10.7.sdk.tar.xz to tarballs/ by creating it from the AaronWallace repo (`tar -cJf ~/MacOSX10.7.sdk.tar.xz MacOSX10.7.sdk` within the MacOSX_SDKs directory), downgrade to LLVM 3.9.1 in Cygwin and follow the build/use instructions listed.

Only build issues I had to work around was to remove the "#ifdef __CYGWIN__" section from cctools' include/foreign/dlfcn.h once it's unpacked, and there was something about an undefined reference to an lto.h entry when it went to link ld, so I ran it as `DISABLE_LTO_SUPPORT=1 UNATTENDED=1 ./build.sh`.

I imagine you'd have even smoother sailing using osxcross on Linux. :good::highfive:

Edit: Updated instructions - I switched from MacOSX10.6.sdk to MacOSX10.7.sdk since 10.6 doesn't have getline. :eek:
[Sony Cross-Device General] munjeni:
I will do it using buildroot toolchain, but when I get free time, now working on few things in the same time, LOS15, LEDE, buildroot under z1c ...etc sorry
[Sony Cross-Device General] osm0sis:
Thank you sir! Whenever you get the chance. :)

Proper LEDE on a phone?! Cool stuff! :D
[Sony Cross-Device General] munjeni:
I'm tried yesterday build toolchain without luck, didn't folow your link, used something diferent. Can you please point me to the right direction about how to create toolchain?
[Sony Cross-Device General] osm0sis:
I did it on Cygwin but it is written for Linux primarily and should work for you.

cd ~;
git clone https://github.com/tpoechtrager/osxcross;
git clone https://github.com/AaronWallace/MacOSX_SDKs;
cd MacOSX_SDKs;
tar -cJf ../osxcross/tarballs/MacOSX10.7.sdk.tar.xz MacOSX10.7.sdk;
cd ../osxcross;
sudo tools/get_dependencies.sh;

This should make sure you've got the needed Linux packages. Note that I had to downgrade to LLVM 3.9.1 for it to work on Cygwin, but that might not be necessary on Linux.

Then just build it.

./build.sh;
#if this gets some LTO issues at link-time you can try with `DISABLE_LTO_SUPPORT=1 ./build.sh;`

./build_gcc.sh;
#this is optional since you'll already have clang

Full documentation on the osxcross repo here:

https://github.com/tpoechtrager/osxcross/blob/master/README.md

If you're able to get that working in the next couple of days I'll release AIK v3.1 rapid fire with it included before I leave on holiday. :) :highfive:
[Sony Cross-Device General] munjeni:
Ok thanksm will check that!
[Sony Cross-Device General] osm0sis:
Surprising that macOS as a desktop OS wouldn't have that.. I asked for you in case there's an easy fix: https://github.com/tpoechtrager/osxcross/issues/146
[Sony Cross-Device General] osm0sis:
[MENTION=4120742]munjeni[/MENTION], Stack Overflow to the rescue, per usual!:

https://stackoverflow.com/questions/4003479/how-to-enable-large-file-support-under-darwin

:)
[Sony Cross-Device General] munjeni:
Ok, so no need fopen64 and fseeeko64 its by default configured for LFS, right? Just need to add flags D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE , right?

But what about FILE *blabla? Seems osx don't have FILE type! This is warning I'm getting:

warning: incompatible integer to pointer conversion initializing 'FILE *' (aka 'struct __sFILE *') with an
expression of type 'int' [-Wint-conversion]
FILE *fp = fopen64(filename, "rb");

Edit:

Ok, probably because fopen64 is used... will try now!
[Sony Cross-Device General] munjeni:
Had hard time building for apple since can't make any static lib with its crosstool, so missed libz and libsparse, but good thing done it this way:

o64-clang -D_FILE_OFFSET_BITS=64 -Wall -O3 -I./include -Iandroid-simg2img-master/include lz4.c unpackbootimg.c zlib-1.2.9/adler32.c zlib-1.2.9/crc32.c zlib-1.2.9/deflate.c zlib-1.2.9/infback.c zlib-1.2.9/inffast.c zlib-1.2.9/inflate.c zlib-1.2.9/inftrees.c zlib-1.2.9/trees.c zlib-1.2.9/zutil.c zlib-1.2.9/compress.c zlib-1.2.9/uncompr.c zlib-1.2.9/gzclose.c zlib-1.2.9/gzlib.c zlib-1.2.9/gzread.c zlib-1.2.9/gzwrite.c untar.c android-simg2img-master/backed_block.c android-simg2img-master/output_file.c android-simg2img-master/sparse.c android-simg2img-master/sparse_crc32.c android-simg2img-master/sparse_err.c android-simg2img-master/sparse_read.c kernel_dump.c -o kernel_dump.x86-apple-darwin11

Done! Hope its ok and working. Attachment on first post! :)
[Sony Cross-Device General] shoey63:
Well I'm sure [MENTION=4544860]osm0sis[/MENTION] will love it. Just want to give a big thanks to you bro. I no longer use flashtool at all because of your tools. That includes my new Xperia X premium.

Pure genius [emoji4]
[Sony Cross-Device General] munjeni:
Bro many thanks to you too, got mail right now and I see a donation, many thanks!
[Sony Cross-Device General] deboopi2:
extracting kernel.elf

Hi I need kernel.elf.

Do I use your tool to extract it from my cell phone

or

I have the rom in a FTF file, when opening with zip I can see the kernel.sin file, is thee a better way to extract kernel.elf from kernel.sin.

or

can I use it to extract kernel.elf from ftf file.

How doI use it, I cannot see where the source is given, is this because you always need to run it from a command bix on the phone, or how should I use ir?

regards
[Sony Cross-Device General] shoey63:
Windows user? Just drag and drop kernel.sin onto unpackimg and let the kitchen do its work. Then double click repackimg you will get an image-new.img which you can use rootkernel tool on.
[Sony Cross-Device General] deboopi2:
Yes, windoze.

I will try it this way.

Thanks
[Sony Cross-Device General] osm0sis:
Wrong thread I think, [MENTION=4675598]shoey63[/MENTION], but yes you can use AIK (which includes kernel_dump) to automate that. :)
[Sony Cross-Device General] deboopi2:
Oh,

I'm getting confused now, what is AIK and is what do I need for getting kernel.elf with

AIK

and with

kernel_dump.

The questions I asked before where concerning kernel_dump.

regards
[Sony Cross-Device General] osm0sis:
The kernel_dump commands are in the OP under How To.
[Sony Cross-Device General] deboopi2:
Ok but the OP gives the commands like this

kernel_dump DESTINATIONFOLDER PATHTOTHEFILE

kernel_dump out kernel.elf

kernel_dump out boot.img

I donn't know where the source is, only the destinationfolder and the pathtofile.

Do I run it against the ftf file or what ?