yashsingh1/xdaforums huggingface dataset:
[Magisk] pndwal:
Do we have vbmeta in boot image where separate vbmeta partition(s) also exist?... You may be right, but I'd assumed it was one or the other...

Yup, in most cases anyway... PW
[Magisk] zgfg:
Only if vbmeta is integrated to the boot.img - because Magisk patches the boot IMG

If vbmeta partitions (two partitions on A/B devices) are separate, Magisk will not patched them

Read the official Magisk Installation instructions from GitHub:

Installation

The Magic Mask for Android

topjohnwu.github.io

See the screenshots

But frankly, my Xiaomi has separate vbmeta partitions (hence not integrated to the Boot), but I never bothered to run fastboot flash vbmeta
[Magisk] pndwal:
Please give details of the particular app and device setup...

Nb. SafetyNet API is now deprecated; are you passing new Play Integrity API deviceIntegeity verdict? PW
[Magisk] pndwal:
Yup, for legacy Ramdisk and legacy SAR boot-type devices this was never needed (for most stock ROMs), but Magisk did automatically patch vbmeta for 2SI devices (l_Ramdisk types running A10+ and all launched with A10+) up to 24.3 as this was needed to patch fstab for magiskinit injection...

Since 25.0, vbmeta patching is no longer done for 2SI devices either as original init is now patched, meaning AVB flags are generally untouched by current Magisk...

There may be cases where Magisk still patches vbmeta partitions (where they exist) for specific devices, but I'm not aware of / doubt this... PW
[Magisk] zgfg:
Xiaomi 11 Lite 5G NE, launched with Android 11, A/B device and Recovery in Boot Ramdisk - I think it's not legacy device (only if by Magisk terminology)
[Magisk] shoey63:
The installation instructions say vbmeta patching is optional, and since

This pull request was merged

I believe not required at all for devices with seperate vbmeta partitions (as far as I am aware), due to vbmeta_header.flags being set to zero in the boot image.
[Magisk] badabing2003:
I've been at this since my post, doing all sorts of tests and adding multiple validations.

SHA1 hashes are confirmed, I used multiple tools just to be certain.

Although highly unlikely, I'll never discount it, anything could go wrong.

PixelFlasher does not allow you to select a patched boot.img for patching.

When you select a boot.img file for patching:

if present existing boot.img file on the phone (download folder) is first deleted

then it is checked to make sure the delete actually succeeded

the same process is done for magisk_patched.img

then boot.img is transfered

then the patching happens

lastly check if the magisk_patched got created

pull the file back.

I will next analyze manually patching a patched file and see if it would exhibit similar results.

Thanks for the feedback.
[Magisk] Sir Backe:
Hello

I'm in big trouble. I installed the latest magisk version which was suggested in the magisk app on my realme X2 pro. Since then, the phone does not boot anymore. It stucks when the yellow realme logo appears. If I wait long enough (half an hour to two hours), it reboots into TWRP. My phone is rooted.

I tried to reinstall TWRP, I tried the install a TWRP backup (without data, since I want to protect the newer data on the phone). I tried to reinstall the latest ozip file, followed by vbmeta and older magisk. Nothing helps. Always same. Stucks on yellow logo for a long time, then it boots into TWRP.

Can you please help me with an idea, what to do to at least get the phone to boot into OS?

Thank you very much!
[Magisk] badabing2003:
@J.Michael

More test results raise more questions.

I did the following tests (from Magisk Manager UI to eliminate boot_patch.sh as potential culprit)

Factory image Aug: oriole-tp1a.220624.021
name the file: boot_aug_9c43d3fa.img
MD5: ccce82c5cb937aa4881cb410e50826b4
SHA1: 9c43d3fa9822dfd1f2eb7cc2768bbfa3c6c0bddf
patched file: magisk_patched_aug-25203_EM84o.img
MD5: 7fcc9e963ec3b2ec92d9d1257a1492e1
SHA1: b376607f5c57f147a6bd8174c81fcae24372f344
SIG: 9c43d3fa9822dfd1f2eb7cc2768bbfa3c6c0bddf
repatch the patched file: magisk_patched2_aug-25203_GWbxf.img
MD5: 7fcc9e963ec3b2ec92d9d1257a1492e1
SHA1: b376607f5c57f147a6bd8174c81fcae24372f344
SIG: 9c43d3fa9822dfd1f2eb7cc2768bbfa3c6c0bddf

Factory image Sep: oriole-tp1a.220905.004
name the file: boot_sep_7f041236.img
MD5: 175dd0866bbe9907e2c994660cfb04ba
SHA1: 7f04123661284ff8d554f01060b00eb6f188a07b
patched file: magisk_patched_sep-25203_Qeg7V.img
MD5: 49812686f9c69443ac6919472936ac51
SHA1: e5c78036b804fb1397d95cc1b105ac7c949837a4
SIG: 7f04123661284ff8d554f01060b00eb6f188a07b
repatch the patched file: magisk_patched2_sep-25203_CH33B.img
MD5: 49812686f9c69443ac6919472936ac51
SHA1: e5c78036b804fb1397d95cc1b105ac7c949837a4
SIG: 7f04123661284ff8d554f01060b00eb6f188a07b

Note: SIG indicates the SHA1 extracted from within the patched image.

As you can see from the results, re-patching a patched file does not change a thing on the file, it remains the same.

Furthermore I realized that there is boot.fingerprint with the firmware version value in the file.

Comparing the two files: magisk_patched_aug-25203_EM84o.img and magisk_patched_aug-25203_EM84o.img

just in case they're binary identical with the exception of the SHA1 part.

They are not.

However if I compare the two files from the previous post (image included again for reference)

Which were binary identical with the exception of SHA1 part

And then finding the fingerprint section you'd see that they both show the patching happened on September boot.img file

To me this scenario is impossible unless Magisk have inserted the wrong SHA1 in the patched image.

9c43d3fa9822dfd1f2eb7cc2768bbfa3c6c0bddf is the SHA1 of August boot.img, how could it be inserted in a patched boot.img file from September?
[Magisk] pndwal:
Not legacy device by any terminology... It's modern 2SI, as is everything released with Android 10+... PW
[Magisk] pndwal:
Those changes fixed A12 compatibility etc, but Magisk still stripped avb_key and avb flags for 2SI devices (ie. running A10+ unless launched with A9) till 24.3 in order to patch fstab... My understanding is that since A9/AVB 2.0, these have been in vbmeta partition(s)...

Since recent magiskinit injection refactoring in 25.0+, AVB flags and keys / vbmeta have been left intact by default... PW
[Magisk] pndwal:
So seems you upgraded magisk?

With twrp you can delete everything in /data/adb to remove a corrupted magisk configuration; it with rebuild itself if system can boot...

If it still fails, restoring stock boot image should fix booting... Or just dirty flash ROM...

Next you can try a clean (initial) magisk installation (use App to patch image, fastboot to (or twrp) to flash patched image)...

Please say if your device uses magisk in boot.img or recovery.img; it will likely be in recovery (recovery mode, ramdisk=no) if an A only device... PW
[Magisk] Sir Backe:
Thank you so much! I will try everything and respond to your question. Due to I'm not so experienced it will take a little time. But I will refer to all next week.
[Magisk] J.Michael:
@badabing2003 I assumed Magisk manager app used boot_patch.sh when patching. Maybe you can put some debug print in to confirm.

How does PixelFlash recognize that it is being asked to patch an already patched image?

I believe it is not a problem to patch an already patched image.
[Magisk] badabing2003:
I wouldn't know but I suppose it does.

Because it is UI driven.

1- you select the firmware

2- you process the firmware, which extracts the boot.img file, gets that hash and records it in db and shows in ui to select it for patching.

3- you select to patch it (which works because in DB it is marked as stock boot.img)

4- When you patch a file it adds it to the db (and of course UI) and marks it as a patched file.

5- Anytime you select a patched file, the UI disables the patch button.

First row is the stock boot.img, Patch button is enabled

Second and third rows are patched boot files, the patch icon as an indicator (package id referencing the source boot) and the patch button is disabled.

I agree, as seen from the tests, Magisk detects this and does not alter the file in any way.
[Magisk] zgfg:
There is no problem to patch an already patched boot.img - screenshot shows how it goes, and the log is attached

If it was previously patched by the same Magisk version, then the repatched img will be binary identical to the one you were patching - they will have the same checksums

But if boot.img was previously patched by another version of Magisk, then the repatched boot.img will be different, ie it will correspond now to this new version of Magisk

Anyway, it's not true that Magisk detects that img was already patched and that it does not do anything then

Magisk always unpacks the boot img, then it patches the ramdisk and finally repacks the boot.img.

However, if previously and now was the same Magisk version, it results with the same patched boot img

You can read all steps in the attached log or better main/short steps on the screenshot - it corresponds exactly to that scenario: repatching boot img that was already patched by the same Magisk version, particularly by the latest Canary
[Magisk] badabing2003:
Thanks,

I was actually looking at that, it proves that the UI is using boot_patch.sh file which is outputting the message Magisk patched boot image detected

I'm surprised that it still unpacks and repacks the file.

What would be the value of doing so? perhaps if newer version of Magisk is running it, it could add features or fix bugs in the patched file?
[Magisk] zgfg:
It has to, why not. It assures the smooth update

Eg, suppose you take Direct update. In that case you don't select the boot.img file - Magisk uses root and it dumps the actual Boot partition.

It unpacks, patches Ramdisk with this new Magisk version, repacks and dumps back to the Boot partition

And it's not only about patching. It parses the unpacked boot img and that way it checks everything, etc

Btw, the same goes if you update Magisk by flashing through TWRP - always the same script/mechanism is used
[Magisk] J.Michael:
How does PixelFlasher invoke boot_patch.sh?

Does it invoke it as an "executable", creating a new process? Or does it "source" it, so changes to the environment will survive?

Does PixelFlasher create an environment variable named "SHA1"?

My copy of boot_patch.sh has a compound statement that tests whether SHA1 has a non-zero length value before setting it to the sha1 of the boot image. If the environment variable is already set, it will not be overwritten. I don't know why.
[Magisk] 73sydney:
this thread is starting to become more a PixelFlasher debug/app development thread than a Magisk general support thread, just saying....
[Magisk] zgfg:
I wouldn't source. The script is supposed to be called as an executable

Eg, my following code is rather primitive but it dumps the active Boot partition (it supports A/B and non-A/B devices), then it patches by invoking the boot_patch.sh script, finally it dumps back to the Boot partition and automatically reboots

Ie, it updates me the Magisk and it also leaves me the patched-boot img in the Download folder

#!/system/bin/sh

# Boot partition
AB=$(getprop ro.boot.slot_suffix)
BOOT_PART=/dev/block/bootdevice/by-name/boot"$AB"

# Boot and patched boot images
DOWNLOAD=/sdcard/Download/
BOOT_IMG="$DOWNLOAD"boot.img
PATCHED_BOOT_IMG="$DOWNLOAD"patched-boot.img

# Magisk path
MAGISK_PATH=/data/adb/magisk/

# Disk-dump boot image to Download folder
dd if="$BOOT_PART" of="$BOOT_IMG"

# Patch the boot image and move to Download folder
"$MAGISK_PATH"boot_patch.sh "$BOOT_IMG"
mv "$MAGISK_PATH"new-boot.img "$PATCHED_BOOT_IMG"

# Flash the patched boot image
dd if="$PATCHED_BOOT_IMG" of="$BOOT_PART"

# Reboot
reboot
[Magisk] badabing2003:
adb shell "su -c 'export KEEPVERITY=true; export KEEPFORCEENCRYPT=true; ./data/adb/magisk/boot_patch.sh /sdcard/Download/{boot_img}; mv ./data/adb/magisk/new-boot.img /sdcard/Download/magisk_patched.img'"

Where {boot_img} is the transferred boot.img file

no sourcing, does it through adb shell which is invoking the boot_patch.sh

nope, PixelFlasher uses MD5 and does not set an environment variable for it, it sets the environment variables for the verity stuff

I'm going to switch to using SHA1 to be better aligned with Magisk, doesn't change it thing other than makes things easier to troubleshoot.

You're referring to this?

I could be wrong, but I think this is done so that if you attempt to patch an already patched file, it does not pick up the sha1 of the patched file and instead keeps to stock (original) boot.img's sha1.
[Magisk] badabing2003:
I see that you're not setting the verity flags,

When I first wrote PixelFlasher, calling patch_boot.sh was not setting the flags unless I set them myself, as opposed to patching it from the UI which was setting the flags properly.

So I included that it the patching call.

One of two things is a possibility.

1- Either the Magisk Manager UI detects and sets the flags when calling the patch_boot.sh

2- Or at the time I first implemented it, it behaved that way.

I should test it again and see if I still need set the flags when calling patch_boot.sh

My assumption would be yes.

But then how is yours working since you're not setting them?
[Magisk] zgfg:
I don't bother with Verity. It's my code for my phones (non-A/B and another A/B) that both have separate vbmeta partitions, hence I fully ignore verity flags, not my business
[Magisk] martyfender:
I happen to use direct install to update magisk and I believe it always patches the existing patched file. I have done this for months. I can't remember the last time I actually patched a fresh boot.img. However, I could be wrong, it could be patching the original backup boot.img. Please clarify if I am wrong.
[Magisk] zgfg:
Of course, it does as stated - it repatches the previously patched one.

Just carefully read in the Magisk app what it prints above when patching or direct updating. Or even save the log and read.

See another my post (later than the one you quoted) about updating Magisk by direct installing or flashing through TWRP - in both cases the current Boot (patched by the previous version of Magisk) is being repatched
[Magisk] pndwal:
You are correct; doesn't use backup...

There is normally no issue, but when major Magisk refactoring occurs there can be incompatibilities and Direct install to update can fail.

Eg. This occurs with many 2SI boot-type devices when updating pre-25.0 Magisk (OnePlus etc) due to new magiskinit method in 25.0+... Users should see messages

"! Boot image $dt was patched by old (unsupported) Magisk"

and

"! Please try again with *unpatched* boot image"...

since More friendly and clear error message #6030 was merged.

In such cases, patching original unpatched image works fine. PW
[Magisk] Sir Backe:
@pndwal

I started working on your tips, but stuck already at the first point. I do not find an /adb folder within /Data. I have an /adb folder within /sdcard, but this one is empty.

Within /data I have:

/lost+found

/media

/misc

/system

/system_de

/unencrypted

layout_version

Could the folder be named different?
[Magisk] badabing2003:
You have to su, just making sure you did.
[Magisk] pndwal:
I don't use TWRP now; perhaps it's not showing all folders? Decrypt issue?

Was this an attempt to update Magisk as I assumed?... Previous Magisk worked properly?... Then you should have/data/adb... I'm not aware of Magisk config being installed elsewhere on any device...

What do you mean by 'without data'?... Is that your issue??

What's that?

Why are you doing this? Custom ROM that requires it? Are you flashing vbmeta with switch options as prescribed in official Magisk Installation Instruction doc? Did you try without?

Do you mean as a zip from recovery here?

As I said, restoring stock boot image should fix booting... Or just dirty flash ROM...

At this point, a full dirty flash may be best since you've messed with vbmeta (and other partitions)? ... I expect this will fix booting, but it shouldn't make things worse in any case...

also,

... Especially, did you have Ramdisk=no on Magisk App home screen?... PW
[Magisk] badabing2003:
I have a Magisk question, as discussed previously, I've been looking into SHA1 inserted in patched images, which is source boot's SHA1 and observed that I don't find this in every patched image.

Could this be device specific?

On Oriole (Pixel 6) I do have it, whereas on Mariln (Pixel XL) and Sunfish (Pixel 4a) I can't find it in the patched image, regardless of how I patch (ui or through patch_boot.sh).

Anybody knows anything about this?

If it is device specific, is it possible to know in advance which models would have this and which wouldn't so that I don't bother checking and validating for models that are not expected to have it.

Incidentally all the patched images are with the latest Canary (25203)
[Magisk] Sir Backe:
Sorry, don't understand what you mean. What means "You have to su...."?
[Magisk] Sir Backe:
Of course I'm not 100% shure if there is a decrypt issue or it doesn't show all folders, but I most likely think it shows everything.

Yes exactly. The app installed latest magisk version successfully (at least it said so), but then the device didn't boot anymore, but stuck when yellow logo appears, and may be after half an hour or an hour it boots into twrp without doing anything from my side.

Yes, previous magisk worked properly.

This was an attempt to go back to a previous twrp backup. TWRP backs up Bootpartition, Data and System-Image.

Since the backup is about 2 months old I tried to reinstall only Bootpartition and System-Image, since I wanted to protect the data on the phone from latest backup to now. So this didn't work. It still stops bootin when the yellow logo appears.

OZip-file is the latest (official) system update (no custom rom) file which you can download from realme website. It contains all files need for a system update.

I installed vbmeta everytime after installing a system update because it is necessary. I'm using the file from topjohnwu. He suggests to install System update, then vbmeta, then magisk. This so far worked properly. But now I can do it and the phone still does not boot again. I install vbmeta with following command:

"fastboot --disable-verity --disable-verification flash vbmeta vbmeta.img"

This worked so far properly.

No, I did not try without vbmeta since I guess this phone needs this.

Yes.

Restoring only stock boot image or also following images?:

fastboot flash boot boot.img

fastboot flash dtbo dtbo.img

fastboot flash system system.img

fastboot flash vbmeta vbmeta.img

fastboot flash vendor vendor.img

Dirty flash rom means just to flash latest ozip file used for system upgrades?

I don't know this. How can I find out? I can only say that I use magisk app with standard settings.
[Magisk] J.Michael:
I'm not clear on how your ROM is delivered, or how it is updated.

In general, if you have any ROM installed, and you have installed Magisk, and you want to update the ROM, the safest procedure is

unhide Magisk app

delete /data/adb/magisk

uninstall Magisk (real Magisk, not app)

uninstall Magisk manager app

update ROM

repeat process for initial Magisk installation

This isn't all necessary all the time on all devices. Lots of people skip some of these steps; almost lots of people don't have problems. But at some point after having a problem, it is easier to start from scratch than to keep trying to find the one flaw in what you already tried.

If you can't boot, you should replace your ROM. The safest choice is the last ROM that worked for you. Just re-install the ROM, and see if you can boot.

If your ROM update is available as something that can be installed on a device that is not already running an older version of the ROM, try installing that. If the update is only available to update an existing installation of the older version, then try that. But see if that boots before trying to install Magisk.

When you have a running system with a Magisk manager app, but without Magisk itself, you can run the app. The first page has details like the "Ramdisk = Yes/No" about which @pndwal inquired.

At this point you can try to install Magisk. Whatever procedure worked for you with the old version of the ROM is a good first choice.

If you used the Magisk manager app to patch a file, make sure you are now using the corresponding file from the new, updated, ROM.

If you were able to extract a file from the original ROM distribution, but you cannot find that file in the update distribution, say something. That's a whole 'nother wall of text.
[Magisk] Chris4711:
Doesn't uninstalling Magisk put you at risk to reveal your data that was hidden so far by Magisk is read by some apps?
[Magisk] zgfg:
Which data, what kind of data, data where are you talking about?!

Magisk modules (if properly installed and configured) help you to hide that Bootloader is unlocked - you may have problems hiding that.

As well as if you are hiding that you have some custom ROM

Modules should be systemless - you remove Magisk, all the changes made by modules are gone

Without Magisk/root, su does no more work and eg you cannot access /data by some root explorer or adb

Internal memory (your photos, etc) is always accessible from Android, with or without root

Hence again, what (kind of) 'data' and where from are you talking about.?!

Be very precise
[Magisk] Sir Backe:
I know the standard procedure for updating a realme X2 pro. It always worked. This time I didn't do anything else than updating magisk through the magisk app. The ROM is supplied OTA for unrooted devices, and as download version on the website for rooted devices with unlocked bootloader.

I now have installed a nandroid backup with /data, /system and /boot.img. Still the same problem. And this normally also worked with any kinds of problems. Just go back to previous installation. This time it doesn't work. It still stops at yellow logo when booting. After about 20 mind it boots again, and after again about 20 mins it boots into TWRP. 🤔

Surprisingly the date and time of the handy are wrong and can't be set correct in TWRP. 🤔

Also surprisingly: since data is backed up to previous version 3 months ago, there is now an adb folder in data, containing magisk... But deleting this I guess will not help, because this is from the previous installation which was working. There must be somewhere else a problem. May be in the communication module?

I'm really lost...
[Magisk] J.Michael:
handy?

I think you should stop trying to fix it with just one more flash. It doesn't matter how many times you've done this -- this time it didn't work.

The first time you installed the ROM, did it wipe out your data? Try installing the ROM again. Not by restoring a backup, as if for the first time. Report on that.