yashsingh1/xdaforums huggingface dataset:
[HTC Desire 526] BigCountry907:
The MODEM Partitions

Things start to get interesting here.

There are several partitions related to the modem:angel:

Keep this in mind when picking files.

All of the modem files have to be for the same modem version.

Don't mix and match versions.

Make sure all of the related partitions come from the same build.

The best option would be to flash the RUU first.

Pull a backup off of your device of each partition.

Use the supplied files in the RUU and for the files that aren't in the RUU use YOUR Backup.

THE BACKUP MUST BE FROM YOUR DEVICE----IT CONTAINS DEVICE SPECIFIC INFO

After and only after we have a way to flash unsigned files we might build an engineering version of the modem.

See here.

https://translate.google.com/transl...reen1900/article/details/46442909&prev=search

Just remember this.

if you mess up the modem your device is toast. :(

The SAFE APPROACH

Step #1 Flash the RUU to your device

Step #2 use the dd command to backup fsg & modem_st1 & modem_st2

Step #3 use the Radio image from the RUU to flash the radio

/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/radio.img

We set this before and called it NON_HLOS.bin ( we might change that name )

Step #4 use the fsg.dd backup from YOUR DEVICE

Step #5 use the modem_st1.dd backup from YOUR DEVICE [/SIZE]

Step #6 use the modem_st2.dd backup from YOUR DEVICE [/SIZE]

This is the best information I can provide at this time in regards to these partitions.

I can tell you that they are related to NV Items that aren't a part of the EMMC.

So you can't change them unless you change the NV or you might have a conflict. :confused:

Lets set our generator.

fsg

modem_st1

modem_st2

Set boot flag and read only flag to false for each one.

The filename will be none.

We need to remember to flash these partitions with the backups before you boot the device into system.

Here is what your generator should look like.

:highfive: whew........this one was making me nervous. :cowboy:
[HTC Desire 526] BigCountry907:
Keepin on Truckin On

Looking at the generator We have made much progress now. :silly:

I'm just going to pause for a second and think about something.

When we put this all together we have to consider a couple of things.

Scenario 1

We have the option to flash using the emergency download tools. (Like if you were in a Brick Situation) :eek:

The EDL tools are only made to flash specific Partitions with specific Filenames.

It is in this situation that we might have to fully erase the emmc before we could flash it.

If we do that then this is where our dd backups will come into play.

We have to get the backup data back on the new partitions.

Scenario 2

We have options for flashing a whole 1 file image.

In a case where the device is still operational and we are not using EDL mode.

Or in a case where EDL mode ( Erasing the entire EMMC ) Puts us into a EDL mode where the EMMC shows up as a disk in windows or linux.

We could be able to even dd copy the new built Image onto the device with adb. ( Provided Bootloader is Unlocked. )

To build a partition.xml to generate a complete image every partition that is not just zeros will have a filename and use the backup images.

To build a partition.xml to generate a EDL mode image we will only be using some of the files. Enough to get the device into fastboot.

Then the rest of the partitions are flashed.

So keep in mind what type of image you are trying to make. One of the following 2

EDL

Complete Emmc

Tailor the Generator Filenames to match the kind of image you need to program. :cool:
[HTC Desire 526] BigCountry907:
Next in the Lineup

The Persist Partition

We have this image in the RUU and in the Backup.

We should take a look and compare the two.

Since we know my device is unlocked. :cowboy:

It is very quick to dump the files and compare them.

hexdump -C ~/Desktop/526-Completed/A13-Backup/pull/persist.img > ~/Desktop/526-Completed/A13-Backup/pull/persist.txt
hexdump -C ~/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/persist.img > ~/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/persist.txt
diff ~/Desktop/526-Completed/A13-Backup/pull/persist.txt ~/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/persist.txt

:cowboy:

They are different. Now we need to open them up and take a look.

The persist image from my device has many selinux settings.

I do not see any of this in the RUU Persist file.

Since my device is unlocked I am going to use the persist.img from it.

Persist partition settings

Read only flag = False

Boot Flag = False

Filename = persist.img

And your generator should look like this.

:cool::cowboy:
[HTC Desire 526] BigCountry907:
One more real fast

Lets look at the sec partition.

This is for security.

There is no image for this in the RUU. We do have our backup file.

Interesting enough there is a sec.dat file located in the Template Rom.

Lets take a look at them.

They are different.

I'm not sure which one is better. Have to test them.

Take a look.

Since I know my device is unlocked. And the file off of my device has more data in it.

I'm going to use the backup from my device.

Now go set the generator.

Read only flag = False

Boot Flag = False

Filename = sec.dat

Your generator should look like this.
[HTC Desire 526] BigCountry907:
Cruzing Along

Ok so we have only a few white un reviewed entrys left.

Lets look at the cdma_record

This is a carrier partition.

There is not much data in my backup file so i will use the backup file to restore the partition.

You will not be able to flash this partition in EDL mode.

In other words.

FILENAME = None

Read only Flag = false

Boot Flag = False

Set the generator and highlight it yellow.

I hope you see how we have color coded our partitions

Things are looking good. :good::good:

Our whole spreadsheet is nearly filled in.
[HTC Desire 526] BigCountry907:
VZW-Quality

Another carrier partition.

This one has some selinux entries in it so we might need it.

We will use the file from my backup since we know my device is unlocked.

Read only flag = false

Boot Flag = false

Filename none

color yellow

:cowboy::cowboy::cowboy::cowboy::cowboy::cowboy::cowboy:
[HTC Desire 526] BigCountry907:
Fast 4

Lets take a quick look at our logs.

These partitions are all different logs.

A good place to debug a problem.

Most of the information in these files is old dump data we don't need.

But they all have some info in the header related to the dump.

We may be able to zero these out.

I will have to try on my running device first.

For now we will plan on using the backups if we have to.

Generator info

All Readonly Flags = Fals

All Boot Flags = False

All File names = none

color = light purple ( Since we want to test )

Fresh data could be useful.

Yeeeee Hawwww :cowboy:

Look at that were down to the last four.
[HTC Desire 526] BigCountry907:
The COTA partition

We know this partition is for updates.

It's important.

You should know the drill by now.

Lets take a look.

Wow tricky.

Since I know my device is unlocked and that being unlocked allows me to flash unsigned updates and ruu files.

Best to use the cota.img from my backup.

Those few bytes must be the switch for the signature check. :laugh::silly:

Generator Settings

Read Only = False

Boot Flag = False

Filename= none

color = yellow ( must flash back later )
[HTC Desire 526] BigCountry907:
Appreload --- This is easy

Appreload is the all the apps that come stock.

You may be able to remove some of them.

Were not gonna get into that yet.

The RUU has a appreload.img

We will use the image from the RUU.

Heres what the partition data looks like.

Therefore our settings are

Read only Flag = False

Boot Flage = False

Filename = none
[HTC Desire 526] BigCountry907:
Finishing Up the Partitions

Normally I wouldn't wok on this so late but were so close.

It's going to be a long night for me.

I've been anticipating getting to this point.

I have been through this process fairly quickly myself.

But to lay it all out for someone else to understand turned out to be quite the task.

This should be a big step in development of the msm8909 chipset.

We used the desire 526 as an example but the same applies directly to the msm8909.

However even if you have a different processor, the same method should work well.

Lets do this. This part is interesting.:cowboy:
[HTC Desire 526] BigCountry907:
Sparse Partitions DATA / SYSTEM

So far we haven't had to deal with any sparse images.

Any large partitions like system / data / cache and possibly others can use sparse images.

Basically the system.img we pulled in our backup is a RAW System.img.

The same for Data.

These are the files I'm talking about.

~/Desktop/526-Completed/A13-Backup/pull/system_raw.img
~/Desktop/526-Completed/A13-Backup/pull/data_raw.img

If you did this dd backup you can mount the backup.img as a disk in linux.

You can modify things as you wish wile it is mounted.

To mount them in ubuntu all i have to do is right click on it and select disk image mounter.

After we have mounted the images we will convert them to sparse.

The sparse image is still one image file. But it gets stripped of its zeros and becomes much smaller.

Then when we run ptool.py against our partition.xml file and checksparse.py it will look for this flag.

Yes if your wondering if the partition.xml output for all the completed partitions is correct????:confused::confused::confused:

The answer is yes. Each line updates as you modify the generator.

HERE YOU CAN SEE THE SPARSE FLAG.

if the sparse= true flag is set the image will be automatically split up when we build the flash

You must manually add this to the sparse files you wish to use.

The generator will not output the flag with exception to data and system.

Data and System will always be sparse.

In the next post we will go over creating the sparse images. :victory:
[HTC Desire 526] BigCountry907:
So Were down to the last Two

Alright lets get this going.

Here are the partitions.

We will sue the userdata.img and system.img form the RUU.

They are here.

~/Desktop/526-Completed/A13-Backup/RUU/ROM/system/system.img
~/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/userdata.img

Converting The Images

Hopefully you have kept the same directory structure as i have used throughout this experiment.

If you have just follow the commands below and lets do this.

Step 1 Set Up Directory & Get Tools

Lets make a new directory for our conversion tasks.

mkdir ~/Desktop/526-Completed/Convert-Sparse

You need a couple of binaries to do the conversion.

They are attached to this post.

Download and put them in our new folder.

They should be here.

~/Desktop/526-Completed/Convert-Sparse/make_ext4fs
~/Desktop/526-Completed/Convert-Sparse/simg2img

With these 2 tools we will do our work.

Step 2 Copy our images to the conversion folder.

cp -rf ~/Desktop/526-Completed/A13-Backup/RUU/ROM/system/system.img ~/Desktop/526-Completed/Convert-Sparse/system_raw.img
cp -rf ~/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/userdata.img ~/Desktop/526-Completed/Convert-Sparse/userdata_sparse.img

cd ~/Desktop/526-Completed/Convert-Sparse

ll

Step 3 Mount the system_raw.img Which is currently in RAW format.

cd ~/Desktop/526-Completed/Convert-Sparse
ll

Now with the size of system_raw.img 2684354560 we can run the following command.

cd ~/Desktop/526-Completed/Convert-Sparse
mkdir /sys
ll
sudo mount -t ext4 -o loop system_raw.img sys/
df

Now you can edit away at the build.prop the system apps whatever else you want to modify in system.

Then we can make the sparse image.

Use this command.

sudo make_ext4fs -s -l 2684354560 -a system system_sparse.img sys/

THE SPARSE IMAGE THAT WILL BE CONVERTED TO CHUNKS WAS NOW MADE

HERE IS THE SPARSE SYSTEM IMAGE TO USE IN THE BUILD.

~/Desktop/526-Completed/Convert-Sparse/system_sparse.img

We can do the same with the userdata.img

But the userdata.img in the RUU is already sparse.

So first we have to convert it back to raw.

Use this command.

simg2img userdata_.img userdata_raw.img

To mount the data img.

mkdir data
ll
sudo mount -t ext4 -o loop userdata_raw.img data/
df

To generate the new sparse image.

sudo make_ext4fs -s -l 4294967296 -a userdata userdata_sparse_new.img data/

Go set the data in the generator.

Filename = system.img

readonly = false

boot = false

sparse flag = true

color green

Filename = userdata.img

readonly = false

boot = false

sparse flag = true

color green

Tomorrow we are ready to put it all together.

We are done with the partitions part of this.

Yaaaaaaaaaaayyyyyyyyyyy :cowboy::cowboy::cowboy:
[HTC Desire 526] BigCountry907:
Download links & references

ALL DOWNLOAD LINKS AND REFERENCES HAVE BEEN ADDED TO THE FIRST POST

:cowboy:
[HTC Desire 526] BigCountry907:
The Files from Our Backup

We know everything highlighted in YELLOW is a file coming from a Device Backup.

Some from the S-Off Backup.

Some from your own Backup.

Guess we have to go back and read through the partition Explanations to figure out which is which.

Lets do that. :cowboy:

Board_Info is now marked in red.

Any partitions marked in Red will need to be backups from your device.

You can modify the file. As shown above. But use the backup from your device

Partitions highlighted in blue will be zero.

We do not need a file for these partitions.

Everything marked in light grey will be from the RUU.

The system and data from the RUU as well.

But you have to convert them to sparse as i showed you before.

Ill attach the new colored spreadsheet for your convience.
[HTC Desire 526] loonix:
Amazing work, putting on a clinic here for everyone.

.
[HTC Desire 526] BigCountry907:
You want to build the image ????

We need to add the GPT Table in to our target build.

We can get that from the RUU.

cp -rf ~/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/gpt_main_8g.img ~/Desktop/526-Completed/Build-Target/gpt_main_8g.img

We still need a few more tools.

I have copied them from those publicly available sources I told you about.

They are in a package with some sample xml files.

Download the package and put all the py files in our Build_Target.

Also copy the Factory_partition.xml file to the Build_Target

Your build Directory should look like this.
[HTC Desire 526] BigCountry907:
Build Complete EMMC Image.

A good way to test and see if what we have created is correct is to build the Entire EMMC Image.

Then we can compare the New Emmc Image with the Complete EMMC Image backup we pulled from the device.

We can even flash it to a SD Card or USB Stick and analyze it.

The SD Card or USB Stick will be an identical clone of what it will be on the phone.

Getting Started

Copy the Factory_partition.xml file and make one called partition.xml

Open the file partition.xml in a text editor.

Delete the lines that start with. <partition label=

It should look like this.

Now go in the generator and give every partition that is not zero.

Zero partitions are highlighted blue.

The filename of the file we have in our Build_Target folder.

The partition xml output in our generator gives us this.

Copy and paste that to the partition.xml file.

Your partition.xml should look like this now.

And there you have it. THE IMPOSSIBLE PARTITION.XML file made easy.

Pretty darn sweet right. :good::cowboy::cowboy::cowboy::cowboy:

Now were ready to make the image.
[HTC Desire 526] BigCountry907:
After 15 or 20 minuets take the sd card our usb stick out.

Then put it back in.

If you are in linux you will notice you get spammed with disks opening.

Thats how you know you did it right.

You can even look at the mounted radios ect...ect...ect....

Sweeeeeeettttttttt.
[HTC Desire 526] BigCountry907:
Just don't get to excited yet.

We really need to check things out good before we even think about flashing the phone.

Let me get some coffee and we will do this.

BRB
[HTC Desire 526] BigCountry907:
Just don't get to excited yet.

We really need to check things out good before we even think about flashing the phone.

So a simple command in linux will show us all the partitions and we can check the sizes against our original image.

cat /proc/partitions

8 32 7782400 sdc
8 33 16 sdc1
8 34 4096 sdc2
8 35 1024 sdc3
8 36 27615 sdc4
8 37 256 sdc5
8 38 24255 sdc6
8 39 256 sdc7
8 40 2048 sdc8
8 41 4096 sdc9
8 42 10240 sdc10
8 43 32 sdc11
8 44 1024 sdc12
8 45 1024 sdc13
8 46 1024 sdc14
8 47 1024 sdc15
259 0 1024 sdc16
259 1 1024 sdc17
259 2 1024 sdc18
259 3 1024 sdc19
259 4 1536 sdc20
259 5 60417 sdc21
259 6 10240 sdc22
259 7 5120 sdc23
259 8 1 sdc24
259 9 4383 sdc25
259 10 1024 sdc26
259 11 4 sdc27
259 12 1536 sdc28
259 13 1536 sdc29
259 14 256 sdc30
259 15 22528 sdc31
259 16 16 sdc32
259 17 1024 sdc33
259 18 1 sdc34
259 19 8 sdc35
259 20 128 sdc36
259 21 128 sdc37
259 22 128 sdc38
259 23 128 sdc39
259 24 128 sdc40
259 25 128 sdc41
259 26 128 sdc42
259 27 128 sdc43
259 28 16 sdc44
259 29 64 sdc45
259 30 1281 sdc46
259 31 256 sdc47
259 32 40960 sdc48
259 33 512 sdc49
259 34 16384 sdc50
259 35 16384 sdc51
259 36 20480 sdc52
259 37 30720 sdc53
259 38 10240 sdc54
259 39 10240 sdc55
259 40 16384 sdc56
259 41 3730 sdc57
259 42 49152 sdc58
259 43 32768 sdc59
259 44 32768 sdc60
259 45 262144 sdc61
259 46 2621440 sdc62
259 47 5120 sdc63
259 48 5120 sdc64
259 49 4265967 sdc65
bigcountry907@bigcountry907-G551JW:~$

Since we have set GROW_LAST_PARTITION_TO_FILL_DISK: true

GROW_LAST_PARTITION_TO_FILL_DISK: In partition.xml the size of each partition

is specified. If this field is TRUE, then the last partition size is ignored

and set to 0. Then during patching this size is updated such that the last

partition extends to use all remaining space.

Our data partition is a little bit larger.

There is a bit more space on the sd card.

However everything else looks the same.

Now we want to dump the GPT partition Table off the SD Card and check it against our original GPT table.

In the generator spreadsheet there is a tab called GPT-Decoder_Compare-Sheet

That is a blank copy of the original generator sheet.

We need to populate it with the data from the new GPT.

Read back to the beginning of the thread if you forgot how we did this.

Dump the GPT table from the SD Card and add it to the GPT-Decoder_Compare-Sheet
[HTC Desire 526] BigCountry907:
GPT table for the new image

When we used the python tools the following GPT table files were generated.

~/Desktop/526-Completed/Build-Target/gpt_backup0.bin
~/Desktop/526-Completed/Build-Target/gpt_both0.bin
~/Desktop/526-Completed/Build-Target/gpt_main0.bin

We will want to take a look at the gpt_main0.bin

Lets make a new folder in our project.

Check_GPT_info

mkdir ~/Desktop/526-Completed/Check_GPT_info

Lets copy our gpt_main0.bin to that location

cp -rf ~/Desktop/526-Completed/Build-Target/gpt_main0.bin ~/Desktop/526-Completed/Check_GPT_info/gpt_main0.bin

DETAILED INSTRUCTIONS FOR THE NEXT STEPS START ON PAGE 3 OF THIS THREAD.

REVIEW THE INSTRUCTIONS TO POPULATE THE GENERATOR CHECK SHEET WITH THE GPT TABLE .

Hexdump the new partition table to a text file.

hexdump -C -v ~/Desktop/526-Completed/Check_GPT_info/gpt_main0.bin > ~/Desktop/526-Completed/Check_GPT_info/gpt_main0.txt

Open the gpt_main0.txt in a text editor.

Then copy and paste the partition table into a blank spreadsheet.

So you can set the format correct.

Then copy that formatted data to the generator.

Once you have the spreadsheet setup you can toggle between the original and the new GPT Table and look for differences.

If you line up the rows and switch back and fourth you will see the differences.

Here is a side by side.

Lets look at what is different.

Backup GPT Header Address

ffffe80000000000

DIFFERENT BECAUSE THE DATA PARTITION IS LARGER

The backup header is at the end of the disk.

Header CRC 32

ff10f08b

iF ANYTHING IS DIFFERENT THIS WILL CHANGE

We hope thats why.

Last Usable LBA

deffe80000000000

MAKES SENSE BECAUSE THE DATA PARTITION IS LARGER

Partition Array CRC 32

9ff47d28

iF ANYTHING IS DIFFERENT THIS WILL CHANGE

The UNIQUE PARTITION GUID values are different too.

We want an Identical Partition table in case the security is checking that crc 32 calculation for the partition array.

If the crc is different the rom might not load.

Lets go see what we can d to fix this.

:crying::confused::confused:
[HTC Desire 526] BigCountry907:
Alright I think I got it.

Now HTC must have edited the GPT table to assign the Unique GUID and have the exact structure there looking for.

So basically this is the breakdown.

First run ptool.py and generate the rawprogram0.xml

Second run checksparse.py to generate the sparse images and update the rawpartition.xml

Third delete the ~/Desktop/526-Completed/Build-Target/gpt_main0.bin

Fourth copy /home/bigcountry907/Desktop/526-Completed/A13-Backup/RUU/ROM/firmware/gpt_main_8g.img (Or the GPT Table from your Device)

Rename gpt_main_8g.img to gpt_main0.bin in the Build_Target folder.

now run the msp.py and burn the image to the sd card.

Check things out.

It's looking damm good.

The partition structure is damm good.

I need to run a diff on each patition.

Pull each partition off the sd card.

sdc1

sdc2

sdc3

ect.

ect.

Run diff against original pull from phone and the pull from the sd card.

I will be back.......

Gimmmeeee a bit.:cowboy::fingers-crossed::cowboy::cowboy::cowboy::cowboy::cowboy::cowboy::cowboy::laugh::laugh::silly::silly:
[HTC Desire 526] BigCountry907:
I have checked the images. They all match.

This means the sd card is an exact clone.

I've even heard that if the device is bricked it will boot off a sd card like this.

I'll get more detailed in these last few post later on.

For now the rom we created is ready to go.

Now we just need a way to flash it.
[HTC Desire 526] BigCountry907:
Recover Brick ????

Well I have done it now.

I went and bricked my desire 526.............:laugh:

Glad its on $34.

No really how else can I get to the bottom of this flashing business.

We have a good image and a good partition.xml but the rest I need to research more.

Now don't worry you have to do something pretty stupid to brick the thing.

Believe me .it's harder to brick than you think.

Anyway I made a cool discovery that if you take the emmc_appsboot.mbn that comes in the RUU and rename it to aboot_signed.img then re zip the ruu.

You can flash the RUU and it WILL flash the bootloader. The OEM TEST KEY signed Bootloader. ( YOU HAVE TO BE S-OFF ).

To boot........:silly: it WILL BOOT. Gives a quick warining that LK is unverified but it boots it all the way.

Thats a good sign. Means IF YOU ARE S-OFF you can run UNSIGNED ABOOT.

After discovering this I got curious. I took the emmc_appsboot.mbn from the template rom and flashed it.

Well that bricked me. ( Probably the wrong Device Tree DUUUUUHHHHH :cowboy:)

Interesting enough the initial mode if plugged in usb is qdl diagnostic 900e mode.

Easily get to qdl husb 9008 mode with DEEP FLASH CABLE.

Or make your own cable.

Right now I have NPRG8909.mbn and ENPRG8909.mbn.

Was MPRG8xxx.hex previously for older msm chips.

So we have the programmers....I'm pretty sure.

And we do have all the firehose both signed unsigned and validated.

We are missing the singleimage.

Have to make it. Guess we will see what happens.

:eek: