yashsingh1/xdaforums huggingface dataset:
[Android Software/Hacking General [Developers Only]] androcraze:
EDIT: Wrong post.

---------- Post added at 20:33 ---------- Previous post was at 20:28 ----------

OK. I think I have this....

What you downloaded IS NOT the python script. You downloaded an HTML link, or saved it as one.

Go to this page:

https://github.com/xpirt/img2sdat/blob/master/img2sdat.py

Press the "Raw" button, and then "Save Page As..." and select "Text Files" as the save format.

You can also right-click on the Raw button and "Save Link As..." and it should do the same thing.

EDIT: Use this link:

https://github.com/xpirt/sdat2img/blob/master/sdat2img.py

Sorry.
[Android Software/Hacking General [Developers Only]] PaulPizz:
Was there ever a solution for this?
[Android Software/Hacking General [Developers Only]] mrbox23:
So I'm running the commands described under the Ubuntu 64 section but on armhf architecture(terminal emulator on my phone in a debian chroot). Everything works fine up to rimg2sdat. I have python installed and I just copied make_ext4fs directly from the host ROM(cm12.1). I'm using qemu to run rimg2sdat, it executes, but my final system.new.dat is exactly 314KB. I'm running the Ubuntu 64 commands to the T. Is there source code for rimg2sdat so I can compile it for armhf?
[Android Software/Hacking General [Developers Only]] :
Not that I am aware of, but please share if you come across it. I suspect it is similar to sdat2img but by design just reverts what it previously does.
[Android Software/Hacking General [Developers Only]] blacknikname:
hello people ) need some advice, how to pack dumped dd system image from samsung to dat format
[Android Software/Hacking General [Developers Only]] jenkins-84:
Hey got a issue with Android M after flashing system.new.dat it doesn't seem to read my SD card... Was fine on LP though. Anyone any ideas?? Cheers

edit fixed issues cheers
[Android Software/Hacking General [Developers Only]] Juandr0ide:
Format it on your PC
[Android Software/Hacking General [Developers Only]] jenkins-84:
Was nothing to do with that, it was device issues after flashing when detecting sd but its fixed now
[Android Software/Hacking General [Developers Only]] ehsan_zamani_68:
hi

And let's make a movie of the whole process?

Some did not understand the process

For example

How to achieve " system.transfer.list"

plz help me
[Android Software/Hacking General [Developers Only]] Juandr0ide:
Is it possible to directly convert a system.img backup made with adb? Where can I get the file_contexts on stock ROM? There is one inside the boot.img. Thanks in advance :)

---------- Post added at 10:29 PM ---------- Previous post was at 10:25 PM ----------

system.transfer.list should generate automatically when converting system.img to system.new.dat
[Android Software/Hacking General [Developers Only]] Robin94:
To everyone getting "command not found" and "permission denied", use:

phyton3 ./sdat2img.py system.transfer.list system.new.dat system.img
[Android Software/Hacking General [Developers Only]] zaphodbeeb:
Trying to mount the system.img produced by step 1 I use

sudo mount -t ext4 -o loop system.img output/ as per the OP

However this gives me

mount: wrong fs type, bad option, bad superblock on /dev/loop0,

missing codepage or helper program, or other error

In some cases useful info is found in syslog - try

dmesg | tail or so.

Appreciate any help. Its a marshmallow cm13.0 rom I compiled. Im working on a Ubuntu 15.04 machine.

Dont recall having this problem on any of the lollipop roms I worked on.:(
[Android Software/Hacking General [Developers Only]] Q9Nap:
Try using the simg2img binary to unsparse the system image before mounting :)
[Android Software/Hacking General [Developers Only]] zaphodbeeb:
I tried

./simg2img system.img output.img

but got this error

Invalid sparse file format at header magi

Failed to read sparse file

I think the error may be with step 1 producing a corrupt system.img, hence the error. I tried with sevral CM13 marshmallow roms and get the same error.
[Android Software/Hacking General [Developers Only]] androcraze:
did you try "-t auto" ? Dumb question, but are you sure it's ext4?
[Android Software/Hacking General [Developers Only]] zaphodbeeb:
Thanks but this doesnt help either. It seems to be an outstanding issue with marshmallow roms.

https://github.com/xpirt/sdat2img/issues/6:crying:
[Android Software/Hacking General [Developers Only]] Q9Nap:
Yeah that means it's not an ext4 image then. I've never had success with "-t auto".

It could be ext3, yaffs, or even some proprietary format...
[Android Software/Hacking General [Developers Only]] androcraze:
Which ROM?
[Android Software/Hacking General [Developers Only]] zaphodbeeb:
This one

http://xdaforums.com/oneplus-one/development/alpha-zaphods-cyanogenmod-13-01-11-15-t3238703

And also an uber version of the rom 13th Nov built).
[Android Software/Hacking General [Developers Only]] :
Same with my CM13 builds. Tell the XDA member making these CM13 builds to use the flag, BLOCK_BASED_OTA=false, in the boardconfig.mk file within the device tree he/she is using. It will build the ROM and create the final zip without the block system image. Thus displaying the system directory and all its subdirectories that come with it. Making it much easier to access until a workaround is found here. If he doesn't want to add it to the boardconfig.mk file then he can call it in the terminal prior to building with, export BLOCK_BASED_OTA=false. See this merged commit.
[Android Software/Hacking General [Developers Only]] androcraze:
The ROM I use, UntouchedAOSP for the N5 (6.0.0_r2), converted and mounted fine: http://xdaforums.com/google-nexus-5/development/rom-untouchedaosp-aug-17-2015-t3182377

Your ROM does no such thing for me. sdat2img.py gives no errors, but I'm unable to mount. simg2img says "Bad magic".
[Android Software/Hacking General [Developers Only]] zaphodbeeb:
I guess it could be a bacon, or a CM13.0 only thing in that case. I,ll try another AOSP bacon rom tomorrow.
[Android Software/Hacking General [Developers Only]] kenzukie:
I got error command zero .:confused:
[Android Software/Hacking General [Developers Only]] zaphodbeeb:
I'll try it for OPO cm13 and report back. I tried to edit the boardconfig.mk file previously but it didnt want to play. However it wasnt exactly as you have suggested above so Ill give it a shot.
[Android Software/Hacking General [Developers Only]] :
It will work but that assumes if you know how to build Roms from open source such as Cyanogenmod for example. Have to have a successful build or else it's futile. This isn't something everyone will be able to do but those reading that can will find it useful.
[Android Software/Hacking General [Developers Only]] zaphodbeeb:
:rolleyes:

Working fine for my CM13 build. Thanks so much. :good:
[Android Software/Hacking General [Developers Only]] ehsan_zamani_68:
[IMG="http://uupload.ir/files/mn4g_screenshot_fro.png"]http://uupload.ir/files/mn4g_screenshot_fro.png[/IMG]

Where is my problem?
[Android Software/Hacking General [Developers Only]] androcraze:
You didn't download the .py script, you downloaded an html page instead.

Go to this page:

https://github.com/xpirt/sdat2img/blob/master/sdat2img.py

Press the "Raw" button, and then "Save Page As..." and select "Text Files" as the save format.

You can also right-click on the Raw button and "Save Link As..." and it should do the same thing.
[Android Software/Hacking General [Developers Only]] ehsan_zamani_68:
Thanks, dear

The error first was resolved

new error
[Android Software/Hacking General [Developers Only]] androcraze:
Let's see an "ls" command (a lower-case "L", not a capital "i")... is there a file named "system.transfer.list" in the same folder that you're current in? You have to have all files that you're invoking on the command line, save the output.
[Android Software/Hacking General [Developers Only]] ehsan_zamani_68:
what "system.transfer.list"?

is

# system.transfer.list:

# - version 1: android-5.0.0_r1

# - version 2: android-5.1.0_r1

# - version 3: android-6.0.0_r1

?

What commands should be used here for unpack?

how i should use code?

./sdat2img.py system.transfer.list system.new.dat system.img (ture??)
[Android Software/Hacking General [Developers Only]] androcraze:
sdat2img.py to convert and extract the .img file, img2sdat.py to revert it back.

what do you mean by "repack"?

img2sdat.py:

https://github.com/xpirt/img2sdat

usage:

img2sdat [system.img] [outdir]

img2sdat.py is used when you want to convert everything back. "outdir" is a temp folder where the new system.transfer.list and system.new.dat files will get dumped to.
[Android Software/Hacking General [Developers Only]] ehsan_zamani_68:
sorry

What commands should be used here for unpack system.img 5.x.x?
[Android Software/Hacking General [Developers Only]] androcraze:
You have it right.

Unpack the original ROM zip archive. in the extracted files, you should see: system.transfer.list and system.new.dat (or hopefully something very close to that).

Copy those files and the sdat2img.py script to a folder.

cd to that folder and execute:

./sdat2img.py system.transfer.list system.new.dat raw.img

That should be it to extract.
[Android Software/Hacking General [Developers Only]] ehsan_zamani_68:
i am sorry

I am a novice

new error in extract

[IMG="http://uupload.ir/files/4oky_screenshot_from_3.png"]http://uupload.ir/files/4oky_screenshot_from_3.png[/IMG]
[Android Software/Hacking General [Developers Only]] androcraze:
It will be like before:

./img2sday.py raw.img temp/

or something like that

you need to pull the img2sdat.py from:

https://github.com/xpirt/img2sdat
[Android Software/Hacking General [Developers Only]] ehsan_zamani_68:
What should I do steps?

step 1 create folder outdir

Step 2:

unpack system.img >>>> code: img2sdat [system.img] [outdir]

step 3

copy "sdat2img.py" in [outdir]

convert dat >>>code : ./sdat2img.py system.transfer.list system.new.dat system.img

steps is true??
[Android Software/Hacking General [Developers Only]] androcraze:
1. Extract original ROM zip file.

2. ./sdat2img.py system.transfer.list system.new.dat system.img

3. sudo mount -t auto -o loop system.img /mnt

Then you can view the files in system.img

If you've modified system.img and want to convert it back to a ROM

1. mkdir /temp

2. ./img2sdat.py system.img temp/
[Android Software/Hacking General [Developers Only]] dimusa:
I tried to do that but I had error with the tipe of fs. Why it happens? What I can do to fix it ?
[Android Software/Hacking General [Developers Only]] androcraze:
What ROM? Give me a link.
[Android Software/Hacking General [Developers Only]] dimusa:
I tried to do with this rom: LINK
[Android Software/Hacking General [Developers Only]] androcraze:
http://xdaforums.com/showpost.php?p=63814632&postcount=467

It's a CM13 issue. My results are the same as yours. No errors converting, it just won't mount.
[Android Software/Hacking General [Developers Only]] hedwig34:
Here is a work around to mount a cm13 system image.

First use sdat2img as normal

Check the BoardConfigCommon.mk for the system partition size. I'm trying to mount for the d2vzw. So in my d2-common device tree I look for it and find:

BOARD_SYSTEMIMAGE_PARTITION_SIZE := 1572864000

Take the partition size and divide by the block size which is 512

1572864000 / 512 = 3072000

3072000 is the number of sectors we want to resize the system.img.ext4 to.

you can check the partition information using fdisk -l system.img.ext4. Mine looks like this:

vzw $ fdisk -l system.img.ext4

Disk system.img.ext4: 1572 MB, 1572859904 bytes
255 heads, 63 sectors/track, 191 cylinders, total 3071992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk system.img.ext4 doesn't contain a valid partition table

The use the resize2fs command to change the sectors from 3071992 to the value calculated 3072000.

vzw $ resize2fs system.img.ext4 3072000s
resize2fs 1.42.9 (4-Feb-2014)
The filesystem is already 384000 blocks long. Nothing to do!

It says 'Nothing to do!' however check it again with fdisk and see that it has changed.

vzw $ fdisk -l system.img.ext4

Disk system.img.ext4: 1572 MB, 1572864000 bytes
255 heads, 63 sectors/track, 191 cylinders, total 3072000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk system.img.ext4 doesn't contain a valid partition table

Now you can mount the image normally like mount -t ext4 -o loop system.img.ext4 mount/

This worked for me. Hope it helps.
[Android Software/Hacking General [Developers Only]] Setting.Out:
I have read through this thread, granted I'm a bit sleepy (12:30am here), but I can't seem to quite wrap my head nor my windows machine around this one. I keep getting syntax errors, I'm sure it's my fault, but still I am missing a step somewhere.. I really miss the old way files were compiled and zipped into roms...

Please consider a video or more pictures.. :)
[Android Software/Hacking General [Developers Only]] androcraze:
I think you're at a bit of a disadvantage.

Most folks replying in this thread (including myself) are running some variety of Linux. You can get away with using Windows, but you have to understand the under-pinnings of Linux (and by extension, Android) to appreciate all of the steps that are being posted.

I'm not saying you can't use Windows; it's just that Linux makes things much easier.
[Android Software/Hacking General [Developers Only]] v7:
I'm getting unexpected transfer list version [1] while flashing on OnePlus 2.Is the bug fixed by [MENTION=753984]xpirit[/MENTION]?
[Android Software/Hacking General [Developers Only]] tirta.agung:
Hi there xpirit , I have tried using your guide with success in unpacking YUREKA CM roms, however, starting from last week nightlies, your guide just does not work. It convert the dat to ext4 image, however all files, such as the build.prop or any other files that usually can be edited by a simple text editor, ie gedit, appear broken.

Here is the link to the rom: https://download.cyanogenmod.org/get/jenkins/135330/cm-12.1-20151119-SNAPSHOT-YOG7DAO1M9-tomato.zip

Does anybody have any idea on what have happened?
[Android Software/Hacking General [Developers Only]] tirta.agung:
Hahahahaha, just ignore the above post. I managed to work it out using the old method.
[Android Software/Hacking General [Developers Only]] bigrammy:
Hi bro,

Care to share the fix as I know quite a few are having problems. :confused:
[Android Software/Hacking General [Developers Only]] tirta.agung:
The culprit is the new android encryption method, so even the dat file is unpacked correctly into the raw image, all the file within it is still encrypted. So to overcome this situation, you need first to flash the rom using twrp (note you have to use twrp v. 2.8.7.0), boot your android up to use the new rom, then go back to twrp, and make a backup of your system partition using twrp. For more info, just head to this thread and thanks adomol for it.

So what I'm saying is that you can mix and match adomol and xpirt guide altogether.
[Android Software/Hacking General [Developers Only]] bigrammy:
Many thanks bro you explained it perfectly :good:
[Android Software/Hacking General [Developers Only]] orimai:
how is one supposed to use this python thing?

All i'm getting is traceback errors, even changing directories(os.chdir) is too hard. yes i imported os (i dont get why i even have to do that)

I just used the python installer an ran it

any help is appreciated.

(i'm using windows 7 and python 3.5)
[Android Software/Hacking General [Developers Only]] schoolsux:
What if the rom.zip doesn't contain the "file_contexts"? :S
[Android Software/Hacking General [Developers Only]] tirta.agung:
Unpack your boot.img, search the guide here in xda. You'll find it in ramdisk of your boot.img.
[Android Software/Hacking General [Developers Only]] pawitp:
Sent a pull request with a fix for extraction of Marshmallow images

https://github.com/xpirt/sdat2img/pull/9
[Android Software/Hacking General [Developers Only]] tirta.agung:
It is still not working for me on all android 5.1.x r29 roms, such as this rom here. Furthermore, the first line in "system.transfer.list" is not the android version, coz in this rom, which is a lollipop 5.1.1 rom, it is written as "3" not "2", however, if you open any android 5.1.x r29 or below, it will be written as "2" or "1".

I also did some experiment using the newest arm64 CM 12.1 Nightlies Rom based on android 5.1.x r29, on my YU Yureka device, and here are the steps that I took:

Clean flashing the rom with TWRP 2.8.7.0 (without flashing any other addons, such as gapps). Here clean means by wiping data, system, and boot partition before flashing the rom.

Booted into android.

While android was running, I made a raw image dump of the system partition with the dd command via adb shell.

Then I mounted the above raw image on ubuntu using below command:

sudo mount -t ext4 -o loop system.img output/

I got a garbled system folder. Garbled here means that I can't even access all of the files that usually can be open using gedit, such as the build.prop and those xml files in the /system/etc folder.

Then I rebooted my device to TWRP, and made two other raw image dumps of the system partition with the dd command via TWRP's adb shell, one with system partition mounted and the other with system partition unmounted. And after mounting those raw images on ubuntu, the results were similar with the raw image I made before, I got a garbled system folders.

Still in TWRP and after I mounted the system partition, I went to TWRP file manager, then to system folder, copying the build.prop of the rom to the sdcard, and then copying it to the PC.

And to my surprise, I could open that build.prop with gedit perfectly.

I then made a backup of the system partition using TWRP, and after copying the "system.ext4.win", which I got from TWRP, to ubuntu, I changed the filename "system.ext4.win" to "system.tar" and extracted it.

And upon reviewing the extracted files, I was getting more puzzled, all the files were extracted perfectly, and I can access all files that usually can be open using gedit or edited the apk files.

So, this makes me wonder, I guess the culprit of our problems here is not in sdat2img.py, but on how we mount the raw image after we converted the dat file with sdat2img.py? Cos, in my opinion, a raw system partition dump made by the dd command is just like making a clone of that partition sector per sector or byte per byte, and it fails me when I tried to mount all of those raw images I made above using ubuntu. I wonder, how does the new arm64 TWRP or android mount the arm64 system partition now?

Perhaps somebody in this thread know the solutions?
[Android Software/Hacking General [Developers Only]] pawitp:
You're right that the first line isn't the Android version. It's the version of BBOTA (block-based OTA). I guess the Android version was given just as a rough reference as to what version uses what.

But that has nothing to do with your problem. In fact, the problem you're having has nothing to do with sdat2img at all. Tomato's ROM is built with CyanogenMod's ext4 file compression addon (TARGET_TRANSPARENT_COMPRESSION_METHOD), so normal Linux kernels cannot read it. You'd need to rebuild Ubuntu's kernel with this patch if you want to read the file.
[Android Software/Hacking General [Developers Only]] tirta.agung:
Is this the same as btrfs which also includes lz4 as one of its compression method? So, what if all devs (ie. they who work on any CM's derivatives) enable the transparent compression by default in their rom's kernel (as it is getting more stable after v2), then I guess there is no way for any of us to mount those ext4 raw images created by sdat2img using ordinary Linux distros?

I don't have the bandwidth to build linux kernel and apply the patch myself:(.
[Android Software/Hacking General [Developers Only]] pawitp:
If you want to mount a file system requiring custom kernel support, then the easiest way would be to run a kernel with that custom code. The kernel source code is only ~70MB, less than the size of a ROM package, so the bandwidth shouldn't be an issue.

If you don't want to replace the kernel you're running, you can even try to compile it as user mode linux (UML), though that may be more complicated.

And since I'm feeling nice today, I've uploaded the kernel compiled as UML with transparent compression support and uploaded it to GitHub: https://github.com/pawitp/uml-xcomp.
[Android Software/Hacking General [Developers Only]] tirta.agung:
I will try this tonight, you are really my life saver :good:.
[Android Software/Hacking General [Developers Only]] tirta.agung:
How to patch Ubuntu's kernel with the new CM's XCOMP patches?

Hi there pawitp, its me and sorry to bother you again. I've tried your UML running on ubuntu 14.04, and its run quite good. However, due to lack of UI support, I mean every single edit must be done through UML's command line, I have decided to patch my ubuntu kernel. So here are the steps I have taken:

I have downloaded the kernel source that I'm using with this command:

apt-get source linux-image-$(uname -r)

I just can't afford the bandwith if I have to clone Trusty Tahr's git.

Afterward I tried to patch the kernel source using below command:

patch -fl -i ../PatchDirectory/The.patch

However upon applying some of the patch, I got some errors saying something about hunk failed or that I have to use "--strip". There are six patches that were successfully patched to the kernel:

f73d668898e5996ea7a985b4661e8648716e1fe8.patch

3b432932e3e1ffad2980ee53fe286cbf887fa81f.patch

afb22f6262df1a39c2ef52646839502d61220ffb.patch

ad1310c997bda926a66afc2787227a95cd8909f3.patch

70d3b57af683ce51642123509d9d0a9c7459ee03.patch

48886e1e90dbe348c45989d5eead9f7b6c740f57.patch

And there are five patches that were failing:

d54ef65eda0923f7567ee1effde28bc24e38e501.patch

f86d62e96d862a0999016e1213a7f4f3a611e9c9.patch

559d826ab6703c48708b5146d45fcc9c666e04fd.patch

321fc8332dccc80c9ecd01a0fb68c483c2dbac80.patch

3ec68c7e54e1b45d57c5a4b5fd02e4e25380e398.patch

Furthermore, I guess these are the only patches that are only needed for Trusty Tahr (please correct me if I'm wrong):

https://github.com/CyanogenMod/andr...59d826ab6703c48708b5146d45fcc9c666e04fd.patch

https://github.com/CyanogenMod/andr...21fc8332dccc80c9ecd01a0fb68c483c2dbac80.patch

https://github.com/CyanogenMod/andr...ec68c7e54e1b45d57c5a4b5fd02e4e25380e398.patch

So could you please give me some hint on how to patch all of the 11 patches to ubuntu's kernel source? I guess it will really help newbies like me here in XDA alot. You can even open a new thread just to enlighten this issue.
[Android Software/Hacking General [Developers Only]] pawitp:
- You should use 3.10.x kernel, as that's what the patches are created for. If you use another version, they may not apply cleanly and you will need to manually resolve conflict, which requires knowledge of C and the Linux kernel, not really something teachable in a forum post.

- All the patches should be applied in the order given

- AFAIK, you will need to apply all patches since LZ4 is not included in the mainline kernel, at least in 3.10.