yashsingh1/xdaforums huggingface dataset:
[Miscellaneous Android Development] munjeni:
Yes it can be done for example editing ramdisk and doing some bindmounts, or using overlayfs and many other ways. But we don't stick to the topic, neither you nor me. Topic is super image tools! What need to be done is recalculation about how many free space is remain in an super partition, that way maybe we can make some more work to resize internal super partitions to the minimal size so that partition which is editing have enought free space to unshare blocks, the thing can be probably done the same way as lptools doing that on the fly https://raw.githubusercontent.com/phhusson/device_phh_treble/android-11.0/cmds/lptools.cc . As you can see lptools doing that natively! The whole idea is to make fast tool for super partition edits on the fly, wihout ramdisk modifications, hackings...etc, so natively inside android. Getting an fast tool I see is not possible because of lack of free space for unshare blocks inside super partition. And I will probably get out of this topic and start new topic and leave our tool only for those who need to extract partitions from super image, at least I see its suitable only for that and nothing more.
[Miscellaneous Android Development] Pervokur:
then compress something and add decompression and mounting to init.rc, it's pretty easy to do

Otherwise I don't understand how do you want "decompress"(unshare blocks)system.img without increasing size.

I will write and test script today, it's sounds pretty easy
[Miscellaneous Android Development] munjeni:
Some super partitions have reserved space on every each partition but also some do not have that reserved space, I have implemented resizing in superrepack but it not work on every device, only on those devices which have some reserved free space on each partition.
[Miscellaneous Android Development] Pervokur:
also will not work if OTA firmware have something like this

# Remove all existing dynamic partitions and groups before applying full OTA

remove vendor

remove system

# Add partition system to group qti_dynamic_partitions

add system qti_dynamic_partitions

# Add partition vendor to group qti_dynamic_partitions

add vendor qti_dynamic_partitions

# Grow partition system from 0 to 5761216512

resize system 5761216512

# Grow partition vendor from 0 to 1097568256

resize vendor 1097568256

Not the right order. Because of this there will be 2 segments of each partition .
[Miscellaneous Android Development] munjeni:
Yes! Even overlayfs will not work in case OTA update boot partition
[Miscellaneous Android Development] samrattt:
Is this possible to reparatition f2fs partition??

When I use superrepack, it says Unknown skipping.Any suggestions?
[Miscellaneous Android Development] Mitezuss:
is it working on any stock firmware ? (like motorola ?) with Android 12 ?

And, this patch is permanent ? Or if i re-flash the original firmware the partition will be again RO ?

is it secure ? (if fail can be re-flash and will work all ?) the partitions can be delete or some like that (problems)?

Thanks
[Miscellaneous Android Development] munjeni:
1. I don't know, probably yes

2. Its not permanent

3. if you reflash original fw it will be stock again and RO

4. I don't know if it is secure.

5. If you reflash stock rom it will be fully stock rom

6. You can delete but not using our tool, for deletion you need another tool and don't ask which one, I don't know. I'm not android active for almost 5 years, even I have no idea whats new in android world
[Miscellaneous Android Development] Talon:
Hi

Tried on realme NEO2 A12

- it seems that the process is going "normally" ... there are no errors ... - but in the end there is also no RW ... /

- ...after reboot - all modules and danylist in magisk were turned off - everything else was unchanged....

hope the attached log will tell something useful with help

(this is the log of the first run .... - I ran it 4-5 more times - but nothing interesting happened ... and in subsequent logs there may be 5-7 uninteresting lines ... - definitely not errors !!))
[Miscellaneous Android Development] Mirwan12:
Does this work for LG V60?
[Miscellaneous Android Development] midozalouk:
Does it support EROFS partitions?
[Miscellaneous Android Development] adwerts:
Has anyone been able to make f2fs unpack and rebuild with what tools can I change this
[Miscellaneous Android Development] Skorpion96:
i'm trying to extract super partition and extraction interrupts

superunpack.exe super.ext4.img

---------------------------------------------------------

Super image unpacker v_15 by munjeni @ xda 2020)

---------------------------------------------------------

LpMetadataGeometry magic = 0x616c4467

LpMetadataGeometry struct size = 0x34

LpMetadataGeometry sha256 = 4E31CF64275442F40E25C772A18D1FCCD8B1291232E593F65E522FC7AC07DF03

LpMetadataGeometry metadata_max_size = 0x10000

LpMetadataGeometry metadata_slot_count = 0x2

LpMetadataGeometry logical_block_size = 0x1000

LpMetadataHeader magic = 0x414c5030

LpMetadataHeader major_version = 10

LpMetadataHeader minor_version = 0

LpMetadataHeader header_size = 0x80

LpMetadataHeader header sha256 = BB6DF87B25D60396FD565AA6730D6E5642B21B01CE604A09F6E35FD5634FC8CF

LpMetadataHeader tables_size = 0x1d0

LpMetadataHeader tables sha256 = ECF1D6F88DAFECC3F9805CF439A38A94FA6B14DD07F4B8974CA95E6FFBD209E8

LpMetadataHeader partitions offset = 0x0

LpMetadataHeader partitions num_entries = 0x4

LpMetadataHeader partitions entry_size = 0x34

LpMetadataHeader extents offset = 0xd0

LpMetadataHeader extents num_entries = 0x4

LpMetadataHeader extents entry_size = 0x18

LpMetadataHeader groups offset = 0x130

LpMetadataHeader groups num_entries = 0x2

LpMetadataHeader groups entry_size = 0x30

LpMetadataHeader block_devices offset = 0x190

LpMetadataHeader block_devices num_entries = 0x1

LpMetadataHeader block_devices entry_size = 0x40

Partitions = 4 used, 0 not used, total 4

partition_1_name = system

attributes = 0x1

first_extent_index = 0x0

num_extents = 0x1

group_index = 0x1

partition_group = main

extent num_sectors = 0x756930 (0xead26000 bytes total)

extent target_type = 0x0

extent target_data = 0x800 (dumping offset = 0x100000)

extent target_source = 0x0

Filetype EXT4. EXT4 size = 0xe71c9000

Dumping system.ext4 ...

....................................................

....................................................

....................................................

.................................................Error reading 512 bytes!

Edit: doing the same on Linux fixed, now my issue is repacking,super, I'm sure that all images used to repack are sparse (good then infact I can open them with p7zip fine) but super creation fails, the command work but at a certain point it says

Invalid sparse file format at header magic

Invalid sparse file format at header magic

Invalid sparse file format at header magic

Invalid sparse file format at header magic

But as I explained images are fine (I tried to convert them with simg2img and it tells me that images are not good for him) so I don't understand what is going on
[Miscellaneous Android Development] travelbug:
@Skorpion96

From my point of view and understanding "Invalid sparse file format at header magic" is a typical feedback from command 'lpmake' - I also get that in Linux when I repack the Super-image manually. I also tried getting that fixed or finding a workaround, but nothing to worry I would say as everything works afterwards (y)
[Miscellaneous Android Development] sasohan9:
I am noob 😒 Please help me... 😢

How to unpack super image with a number, like, super.img.0 / super.img.1 .... etc on Win-10...?

When I drag & drop these types of file (super.img.0) on superunpack.exe, it shows: This is not super image!

I also renamed super.img.0 to super.img and tried, but the result was same.
[Miscellaneous Android Development] alecxs:
simg2img super.img.0 super.img.1 super.img
[Miscellaneous Android Development] lupin 3rd:
Hi @munjeni , I have a galaxy s9 rooted with android10. System partition is read only and both your tool that lebigmac tool doen't work (your tool result is "this is not super image" while lebigmac tool result is "Fatal error! No super partition detected". How can set system partition r-w?

I have also error message about dm-verity and dm-verification: "ERROR: Data from 'vbmeta' does not look like a vbmeta header" ... how can I fix? Here there are steps that I have followed https://xdaforums.com/t/official-twrp-sm-g9600-starqlte-chn.3803297/post-88556861
[Miscellaneous Android Development] munjeni:
Things is drasticaly changed in last 3 years and I don't know, and I am not active on android for about 3 years, just search yourself for information I can't help
[Miscellaneous Android Development] amirzarbakhsh:
I want to run it on Android Studio Virtual Device (AVD).

But it doesn't work when to the line:

./superrepack /dev/block/bootdevice/by-name/super system_a

I get the following error:

127|generic_x86_arm:/dev/block/by-name # ./superrepack /dev/block/by-name/super system_a

/system/bin/sh: ./superrepack: inaccessible or not found

Or if I run it in the /data/local/tmp path, it still gives another error:

generic_x86_arm:/data/local/tmp # ./superrepack /dev/block/by-name/super system_a

/system/bin/sh: ./superrepack: No such file or directory

In my android system this path exists (/dev/block/by-name/super) but this path (/dev/block/bootdevice/by-name/super) does not exist.

I've been searching day and night for more than a month for a solution to RW the root folder in Android Studio (AVD), don't you have a solution for it!?
[Miscellaneous Android Development] Renate:
A bit of necro-posting here, but this seems like a good place to engage...

I was looking into all this super stuff and I was modifying directly stuff inside it.

To figure out what was going on I started writing my own tools.

Sure, there are the AOSP OTA tools with a zillion dependencies.

There are Python tools that can't even deal with multiple slots.

So here is a dump of super off one device.

P=Partition, E=Extent, G=Group, D=(block) Device

C:\>lputil super.img
====== Slot 0 ======
P0 system_a (readonly) E0 G1
P1 system_ext_a (readonly) E1 G1
P2 product_a (readonly) E2 G1
P3 vendor_a (readonly) E3 G1
E0 2048 - 4392712 D0
E1 4392712 - 4978120 D0
E2 4978120 - 5775576 D0
E3 5775576 - 6772456 D0
G0 default ()
G1 qti_dynamic_partitions_a ()
D0 super

====== Slot 1 ======
P0 system_b (readonly) E0 G1
P1 system_ext_b (readonly) E1 G1
P2 product_b (readonly) E2 G1
P3 vendor_b (readonly) E3 G1
E0 2048 - 4392712 D0
E1 4392712 - 4978120 D0
E2 4978120 - 5775576 D0
E3 5775576 - 6772456 D0
G0 default ()
G1 qti_dynamic_partitions_b ()
D0 super

====== Slot 2 ======
P0 system_a (readonly) E0 G1
P1 system_b (readonly) G2
P2 system_ext_a (readonly) E1 G1
P3 system_ext_b (readonly) G2
P4 product_a (readonly) E2 G1
P5 product_b (readonly) G2
P6 vendor_a (readonly) E3 G1
P7 vendor_b (readonly) G2
E0 2048 - 4639672 D0
E1 4640768 - 5222440 D0
E2 5224448 - 6021888 D0
E3 6023168 - 7019872 D0
G0 default ()
G1 qti_dynamic_partitions_a ()
G2 qti_dynamic_partitions_b ()
D0 super

====== Usage ======
2048 - 7019872
83.7% full

As you can see, there are actually three slots, the third being some kind of "everything".

Also, the two slots (as far as super is concerned) have the same content.

And all that stuff is using 83.7% of the entire super partition.

It's not easy to see how there is room for two disimilar slots.

The entire super is 4 GB.

I haven't written the extraction bit yet, but I can plug those sector counts into dd and get a valid FS out of super.

I wrote the extraction/replacement bits.

C:\>lputil /x /s0 /pvendor_a super.img v.img
Wrote v.img

C:\>lputil /r /s0 /pvendor_a super.img v.img
Replaced vendor_a

Any thoughts?
[Miscellaneous Android Development] Renate:
So I got my device running rooted without Magisk.

Don't get me wrong, Magisk is a great and clever program.

It's just that if you add a single file to a directory Magisk will have to have a separate mount for each of the old files.

So something as innocuous as adding a keylayout will give you 30 new mounts!

Here is an example (but not instructions) of how I just rooted my device without Magisk

C:\>adb reboot edl
C:\>edl /lpoke5.bin
C:\>edl /r /psuper super.img
C:\>lputil /x super.img /s0 /psystem_a s.img
====== Slot 0 ======
03000 Header v10.2
03100 P0 system_a (readonly) E0 G1
03134 P1 system_ext_a (readonly) E1 G1
03168 P2 product_a (readonly) E2 G1
0319c P3 vendor_a (readonly) E3 G1
031d0 E0 2048 - 4392712 D0
...

First, we do some math: (4392712 - 2048) / 8 = 548833

That's because when we're dealing with ext4 we're talking about 4096 byte blocks vs. 512 byte sectors.

Now we put s.img on a Linux or Android box using ADB or sftp.

$ resize2fs s.img 3G
$ e2fsck -E unshare_blocks s.img
$ mkdir s
$ sudo mount -t ext4 -o loop s.img s
$ cd s
$ sudo rm something_big_and_useless
$ sudo cp something_useful .
$ cd ..
$ sudo umount s
$ resize2fs s.img 548833

Then we get s.img off the Linux or Android box.

C:\>lputil /r super.img /s0 /psystem_a.img s.img
C:\>edl /w /psuper super.img /z

Note that I had to modify the boot image too. But that's a different story.

Edit: lputil.exe is now available (in my sig).

Edit^2: you can also resize2fs -M system.img and it will minimize it without an explicit size.
[Miscellaneous Android Development] munjeni:
I will try to improve superrepack e.g. to add some command line arguments so everybody will be abble to chose which partitions to convert to rw instead of currently converting all to rw, also I will try add in reverse order to make it revertable to ro. Until that enjoy my tool!
[Miscellaneous Android Development] harpreet.s:
This works for oneplus nord , Android 11 also , thanks.
[Miscellaneous Android Development] lebigmac:
I can confirm that new version 2 of superunpack works perfectly over here both with and without custom argument.

Great work @munjeni as expected. Thank you for making these amazing tools available to us.
[Miscellaneous Android Development] lebigmac:
I believe what's missing is the ability to increase embedded partition size on the fly by an arbitrary amount of megabytes just like the size=100 parameter in my script.

It would be great if @munjeni could add this feature to his new tool as well to make it even better. Thanks.
[Miscellaneous Android Development] HemanthJabalpuri:
@munjeni

I have a Realme C12 running stock Android 10 with ReadOnly Android 11 GSI(LeOS in my case).

I ran your script in it and it completed in just 1sec! Is it default behaviour? Amazing!

But, I don't know why I can't able to mount /vendor as rw.

As you can see in above picture, I can able to mount / as rw as expected. And I didn't get no space left error when adding content to files.

Any ideas?

Also this thread needs to be moved to Development and Hacking instead of sony specific forum.

Thanks!
[Miscellaneous Android Development] munjeni:
Main purpose is to convert ro to rw with a minimal and simple work, but deailing with partitions geometry will involve a lot of work, external libraries like sha256, recalculations...etc which I realy have no plan to implement. The simplest and eficient and fast way as in current implementation is for only purpose to convert ro to rw and if aditional space is reguired simple delete bloatwares from your partitions to get some free space for your files, I think its enought.
[Miscellaneous Android Development] munjeni:
Yes its so fast and eficient, in less than seccond conversion is done. Can you show output of the command "mount" ? ANd also dmesg after you trying to remount vendor
[Miscellaneous Android Development] HemanthJabalpuri:
Here you go. But I am back to stock without Magisk and rw. Below I attached mount from RO and without root state. I will root and try again your tool after sometime. But how to grab dmesg when mounting?

In future versions, below options will be helpful

- Flashable zip(as Magisk flashable too)

- Dump status of read only or read write from partitions

With this, quit if same operation is doing for multiple times

- Option to make change back to before i.e., back to read only

- Option to make rw only mentioned partitions

Thanks
[Miscellaneous Android Development] munjeni:
Yes sure, plan is to implement excatly what you have wrote.

Related to your vendor partition your command mount shows that your vendor partition is on dm-28 block device! It might be your fstab have bug so instead of dm-28 its dm-15 so you getting error operation no permited while you trying to remount it rw. I not going to waste my time with unrelated things other than main purpose of this tools (extraction and ro_to_rw conversion) so please further questions and bugs please ask in your device subforum. I'm prety sure that conversion from ro to rw is perfectly done right so the only thing which you need to figure out is how to remount your vendor to rw.
[Miscellaneous Android Development] HemanthJabalpuri:
That dm-X difference is because of I have flashed stock rom again and updated it and got the mount now. Before one is from when I am on GSI with different version of stock rom.

Below picture is when I used lebigmac's tool

Thanks for the great tool. I will no more ask about this in this thread.

Thanks again.
[Miscellaneous Android Development] munjeni:
New version is out. Whats now:

- added arguments. Now you can do things on single partition, change ro to rw, change rw to ro, on all partitions or on singlepartition. Conversion will be not done on partition if partition allready have that what you want e.g. you want it ro and it is allready ro conversion will be skipt.

USAGE: (With no arguments all partitions going to be RW and shared_blocks removed!)

superrepack [image or block device] [search by string] [rw or ro]

superrepack super.img

superrepack /dev/block/by-name/super

superrepack super.img system_a rw

superrepack /dev/block/by-name/super system_a rw

superrepack super.img system_a ro

superrepack /dev/block/by-name/super system_a ro
[Miscellaneous Android Development] HemanthJabalpuri:
@munjeni

It seems like we have to give partition names one by one explicitly to make all partitions RO! Isn't it?

superrepack [img/block] [ro/rw] may be a good option for it.

Making RO all partitions may doesn't make sense. So it is not that much needed right?

Thanks

Edit:-

We can make all RO with below

for i in /dev/block/mapper/*;do
superrepack /dev/block/by-name/super ${i##*/} ro
done
[Miscellaneous Android Development] munjeni:
if I start fulfilling wishes there will be no end. So not going to add any more combinations since current combinations is good enought, for example tool can be put into recovery installer zip file and used with for example with updater-script with very good possibility for extending for any needs. Enought?

For example it can be used in scripting to retrieve partition names, current flags, etc, for example if somebody call it like this:

superrepack /dev/block/by-name/super blabla blalba

it will not make any changes to the partition since partition blabla and flag blabla doesn't exist so in that way you can trought scripting retrieve what you need without changing anything inside partition, for example to retrieve partion names, flags, features, partiton size...etc, just an example! It can be used a lot diferent way. Its fast enought to run it 10 times in just one seccond. Good enought?
[Miscellaneous Android Development] munjeni:
Version 3 is out! Whats done? Implemented return codes so you can know error return code while running superrepack trought script. Also I have implemented more user friendly log which you will be abble to easily handle while executing superrepack by scripts. This is what is done and how it looks with some examples:

Runing superrepack from script:

Return codes:

RETURN_OK = 0

NO_ARGUMENTS = 1

UNABLE_TO_OPEN = 2

NOT_A_SUPER_IMAGE = 3

WRONG_LP_METADATA_HEADER_MAGIC = 4

UNSUPPORTED_METADATA_VERSION = 5

easy to handle retrieved strings trought scripts, some examples:

Partition: partition name | type | partition size in bytes | uuid | s_feature_ro_compat | current modification status

Partition: system_a | EXT4 | 0x76dc4000 | F3E365B4-F1D6-54B8-AB22-40687FCE6F50 | 0x0000407B | allready RO and with shared_blocks

Partition: system_a | EXT4 | 0x76dc4000 | F3E365B4-F1D6-54B8-AB22-40687FCE6F50 | 0x0000407B | unsharing blocks and making RW

Partition: system_a | EXT4 | 0x76dc4000 | F3E365B4-F1D6-54B8-AB22-40687FCE6F50 | 0x0000007B | allready RW and without shared_blocks

Partition: system_a | EXT4 | 0x76dc4000 | F3E365B4-F1D6-54B8-AB22-40687FCE6F50 | 0x0000007B | adding feature shared_blocks and making RO

Partition: system_a | EXT4 | 0x76dc4000 | F3E365B4-F1D6-54B8-AB22-40687FCE6F50 | 0x0000007B | skipping

Partition: odm_b | UNKNOWN | U | U | U | skipping

Whith this you can do good result while using sed and grep for example.

Enjoy!
[Miscellaneous Android Development] lebigmac:
@munjeni thanks for the update.

If only some genius could compile lptools by @phhusson for aarch64

Then we could do as mentioned by @Lossyx in this comment

to give extra free space to each partition

Even my primitive script can do this (if one can get it to work that is :D)

./superrepack /dev/block/sda15
./lptools resize system $((3*1024*1024*1024))
./lptools resize product $((3*1024*1024*1024))
./lptools resize vendor $((3*1024*1024*1024))

Some users such as @uvzen require 1 GB extra space so they will have to delete a ton of bloatware to free up that much space :D
[Miscellaneous Android Development] munjeni:
I have bad news for you! This tool is dangerous! Do not use please! What I have found now while checked modified partition with e2fsck, found a ton of "Multiply-claimed blocks already reassigned or cloned.", when I do e2fsck repair it got repaired but bad news is some files is deleted from partition! Found that by making diferencie trought git tool! So in general tool superrepack is not a ready!!! Also tool superunpack is not a good while extracting partition with parameter 1 (to rw mode) so do not use parameter 1 !!! I will try to investigate further when I get some free time!
[Miscellaneous Android Development] lebigmac:
Maybe the problem is the shared_blocks reduplication process? You are completely skipping it by only changing 1 or 2 bytes directly on the fly in internal memory.

My script change more than 200 MB per partition file!
[Miscellaneous Android Development] munjeni:
I don't know right now. What I think is whole aproach to have rw on dynamic partition not going to work as we think. Instead we probably must use overlayfs, more info -> https://source.android.com/devices/tech/ota/dynamic_partitions/implement
[Miscellaneous Android Development] munjeni:
Practicaly our super partition contain logical ro partitions (deduplicated ext4 partitions), so to have partitions rw partitions must be increased because reduplication will need some free space to duplicate those deduplicated blocks. But how can we know how many free space we have in super partition for it? E.g. whats happen when we increase all partitons does that mean super partition need to be increased too? Where is than free space inside phone for new increased super partition? Thats how I understand things. Maybe I'm wrong? I'm not worked in android for almost 3 years so all this is new for me.
[Miscellaneous Android Development] lebigmac:
Take a look at this random stock super lpdump output:

Total super size: 4294967296 bytes or 4GB

product is 1.38GB

system is 0.95GB

vendor is 0.3GB

total is 2.63GB which means still 1.37GB of free space left inside super!
[Miscellaneous Android Development] munjeni:
You have PM. It will confirm something about our rw aproach if there is errors while e2fsck it, see pm. I'm realy believing we can't have rw partitions inside super partition. I think we need overlayfs for things. And also I think only a way to debloat super partition is to make new one but stay RO.
[Miscellaneous Android Development] munjeni:
Do not use, all partitions contain errors even without modifying anything! It will give you perfomance isues, rom isues, errors, lost files, corrupted files... etc. I know you have it working but believe me its not a good!