yashsingh1/xdaforums huggingface dataset:
[Android Software Development] Zackptg5:
Oh lol, missed that. The .magisk file is what magisk uses when you're flashing for example, a magisk update. It'll override whatever magisk detects. So it really shouldn't be necessary but more of a just in case thing. I'll have that fixed whenever I release the next stable version. Thanks!
[Android Software Development] nostradomis:
To clarify what I meant, before you even started, was your internal internal storage encrypted or not encrypted? if not, you need to flash an operating system and no-Verity to stop force-encrypt on first boot. after first boot, you have an un-encrypted storage.

formatting does not cause encryption or not ecryption of internal storage instead it does 'mkfs' (like ext4, etc.). the first boot reads the fstab and builds the internal structure accordingly.
[Android Software Development] Arcaea:
Well, before i started doing those thing, twrp couldn't mount data partition, it seems encrypted.
[Android Software Development] nostradomis:
That's fine. I was just trying to understand started. Start over:

install OOS,

install TWRP and DM Verity,

reboot recovery,

format again,

reboot system-- allow first boot to create internal storage and go through initial setup you will now be decrypted.

use MTP to copy files to internal storage

reboot recovery

if you want root install magisk, custom kernel if you want, and DM Verity again,

reboot system and go through setup of root apps, magisk manager and whatever is needed.

reboot recovery

backup- data, boot,system image, vendor image giving it a descriptive name

factor reset

install latest Havoc

install TWRP

install DM Verity

reboot recovery

your internal files are still there because you are decrypted when you created internal storage before

install what you need gapps, magisk, etc.

reboot system and go through initial setup and whatever is needed.

once you are happy with the way they have a good running system, reboot recovery

backup- data, boot,system image, vendor image giving it a descriptive name.

You now have two systems on your internal storage decrypted and you can switch back and forth. To switch, just boot recovery, factory reset, and restore one you want. Create new backups as you make changes. Remember to flash DM Verity after you flash anything that touches boot. Some users flash DM Verity every time they Flash in TWRP. This is good as flashing DM Verity when you don't need it is not detrimental. incremental backups of your internal storage it's also very essential.
[Android Software Development] sga999:
Thanks for the future fix. But I thought I'd ask something else which is really more pertinent to Magisk than your disablers.

Let's assume that I've already used your zip to disable encryption and dm-verity. Then I update to another release of that rom, so after that, I flash your zip again. When the .magisk file was being deleted accidentally (due to the rm command), Magisk did not see the .magisk file, so it showed both checkmarks in the boxes. I think you're saying that Magisk is using the defaults that it sees, since it does not see a file. But I am already decrypted and have previously turned off dm-verity. So do you know why Magisk would be checking both boxes? Again, this Is a Magisk question, but I thought I'd ask you. And I can also try to read the Magisk zip scripts and see what they do.

Which leads me to another question. I see [ -d /cache ] at the beginning of one of your script lines. I don't know much about the script language, but I am able to get an idea of what's going on anyway. I did search for some resource to help me figure it all out, but I couldn't find something that would tell me about that -d.(or -z in some other lines). Can you point me to a good place to find this info?

Finally, without the .magisk file, when Magisk seems to be seeing defaults that I don't see are accurate, I'm wondering why it doesn't cause encryption to occur or dm-verity to be enabled. (I don't want either of those to occur)!. I read in a thread for another device that if dm-verity is enabled and /system is mounted r/w, bricks can occur. Not sure if they mean hard or soft. I'd rather not have either, but the former is the most worrisome. Is that really a possibility? Maybe it depends on the device/rom, so you will be unable to answer that. As far as encryption, obviously, I'm trying to avoid that from happening. If I always flash your zip after flashing a new rom, will there every be any chance of Magisk causing encryption or dm-verity to be in effect upon the next boot?
[Android Software Development] Zackptg5:
The bracket stuff is shorthand for the test command: https://linux.die.net/man/1/test

-d is for 'directory' :)

I'm glad you mentioned that. It looks like that topjohnwu changed the defaults at some point for magisk so they're now false so that may explain the bahavior. I also see that I need to update the .magisk stuff :)

Note that stuff starting at line 58: https://github.com/topjohnwu/Magisk/blob/master/scripts/boot_patch.sh
[Android Software Development] sga999:
Thanks for the info for the test command.

If defaults for Magisk are now false, wouldn't that mean the checkmarks would not be there? Yet they were both there without the .magisk file. Maybe the words enable/disable, preserve/keep confuse me, sort of like double negatives. Magisk wording is 'preserve force encryption' or 'preserve AVB 2.0/dm-verity'. If defaults are false, wouldn't those checkmarks be off?

As for my last paragraph in the prior post, should I be worried about Magisk undoing either my decryption or my disabling of dm-verity? I'm mostly worried about that possibility of brick for mounting /system r/w. Again, that may be specific for the other device I was reading about. But I'd really like to know what the risks are of dm-verity not being disabled. Thanks again.
[Android Software Development] Zackptg5:
That's a good question that I'm not sure the answer to :)

You don't need to worry about it adding back encryption or verity. It either disables it (the KEEP options are set to false), or preserves it (KEEP options are set to true). So if they're already set to false (such as after flashing this zip) then the KEEP options in magisk install don't matter since they'll either be rerunning part of what this zip did or do nothing at all - if those KEEP vars are true, nothing happens.

The .magisk file stuff isn't necessary but I added it anyways just in case of some weird usage or dirty flashing a rom - which I covered in the OP but recommending that this zip is reflashed after anything that would modify the boot img such as a dirty flash (which would completely replace it)
[Android Software Development] sga999:
[MENTION=6037748]Zackptg5[/MENTION], this is FYI. I'm sure you have looked at this code in Magisk, util_functions.sh. I'll show it here and make some comments.

My device is "system-as-root", and after flashing your zip, it is DEcrypted and dm-verity is disabled. Without the .magisk file to cause any overrides, my Magisk zip install output in TWRP says:

- Device is system-as-root

- System-as-root, keep dm/avb-verity

- Encrypted data, keep forceencrypt

Those decisions seem invalid. It looks like the code below decides to set KEEPVERITY=true for a system-as-root device without an override in .magisk file. And it is also setting FORCEENCRYPT=true without an override. But there are several other tests that comprise that decision which are too complex for me to figure out right now.

So this explains why Magisk has both checkmarks checked when running Magisk Manager with no .magisk file. But since you have explained that there are no bad repercussions from these incorrect settings, I won't worry about it.

get_flags() {

# override variables

getvar KEEPVERITY

getvar KEEPFORCEENCRYPT

getvar RECOVERYMODE

if [ -z $KEEPVERITY ]; then

if $SYSTEM_ROOT; then

KEEPVERITY=true

ui_print "- System-as-root, keep dm/avb-verity"

else

KEEPVERITY=false

fi

fi

if [ -z $KEEPFORCEENCRYPT ]; then

grep ' /data ' /proc/mounts | grep -q 'dm-' && FDE=true || FDE=false

[ -d /data/unencrypted ] && FBE=true || FBE=false

# No data access means unable to decrypt in recovery

if $FDE || $FBE || ! $DATA; then

KEEPFORCEENCRYPT=true

ui_print "- Encrypted data, keep forceencrypt"

else

KEEPFORCEENCRYPT=false

fi

fi

[ -z $RECOVERYMODE ] && RECOVERYMODE=false

}
[Android Software Development] Zackptg5:
Oh duh, I forgot about that part - that explains it. I'll drop another test build when it's ready but ya, it doesn't actually enable verity or fec, just maintains present state.

You can see how magiskboot binary does it here: https://github.com/topjohnwu/Magisk/blob/master/native/jni/magiskboot/ramdisk.cpp

Check out the function at line 36 - stuff only happens if the KEEP vars are set to false
[Android Software Development] sga999:
Sorry, I don't understand. I was showing code from Magisk there. I was not saying that was in your code though it may be in both. I was merely trying to explain why Magisk has those two checkmarks on when I don't think they should be. I'll look at your reference, but again, I wasn't saying you should be doing anything.
[Android Software Development] JazzMartian:
Thanks [MENTION=6037748]Zackptg5[/MENTION] for this and all you contribute to the scene.

Does anybody know of a good app that displays the current status of dm-verity, forced encryption, and disk quota?
[Android Software Development] Zackptg5:
Okay everyone, ended up needing to change quite a bit since the last test build - huge shoutout to [MENTION=6376476]fawazahmed0[/MENTION] for the help - you can all follow the process here if you're interested: https://github.com/Zackptg5/Disable_Dm-Verity_ForceEncrypt/issues/8

So here's another test build - I consider this more or a release candidate than anything else so let me know if there's any issues. Think I got everything ironed out now including issues with nvdata fstabs

Edit: Realized that .magisk/.supersu files won't be created if they're not already detected in last test build - fixed and reuploaded
[Android Software Development] radz_:
What's the proper way to update this zip?

I have 5t, oos, custom kernel + magisk
[Android Software Development] sga999:
I haven't tested your newest test zip yet, but I did look at anykernel.sh. I had thought the 'rm' command was in the wrong place (see my post #511), and you seemed to agree. But it is still like that in the new version, I believe.
[Android Software Development] Zackptg5:
You looking at the one included in the zip? It's completely different from the last test release - I fixed that and reworked it completely
[Android Software Development] sga999:
As I said, I haven't tested your new zip, but I definitely did look at anykernel.sh in it. It still has the 'rm' command in the wrong place. So I think you're saying that part of the code isn't being used at all with your rework.
[Android Software Development] Zackptg5:
Ok, its in the right place (line 72). Existing one is copied to installer to be modified before the original is removed
[Android Software Development] sga999:
I apologize. I should have tested first. (I saw the line with 'rm' down lower in the script,which had failed in your recent July 17 version. But it had been near the top in your working one from May with boot_patch.sh. That's what made me think the new onet was incorrect). I will test your new one soon. Thanks.
[Android Software Development] Zackptg5:
Reuploaded attachment in last test build post, see edit message for why: https://xdaforums.com/showpost.php?p=79965399&postcount=527
[Android Software Development] sga999:
I just tested your earlier one from today, and Magisk Manager was still showing the 2 checkmarks. But I was going to look at the scripts and see what was going wrong before posting to you about it. Glad you found it. I will test the latest now.

---------- Post added at 11:47 AM ---------- Previous post was at 11:38 AM ----------

I tried the newest, and it's still not working for me. Your zip says this when flashing in TWRP....

Keep ForceEncrypt: false

Keep Dm-Verify:false

...Yet Magisk_v19.3-zip install says this:

- Device is system-as-root

- System-as-root, keep dm/avb-verity

- Encrypted data, keep forceencrypt

This results in the 2 checkmarks being checked.
[Android Software Development] Zackptg5:
Can you send me the log after flashing the test zip?
[Android Software Development] sga999:
Attached is the recovery.log after flashing your zip.
[Android Software Development] Zackptg5:
Ah, I see the problem. I assumed that .backup/.magisk would be present in all magisk patched ramdisks but it isn't in yours. I'll push another test build out in a few with a fix
[Android Software Development] Zackptg5:
[MENTION=4541967]sga999[/MENTION] New test build up! Fixed .magisk file creation
[Android Software Development] sga999:
Still no luck. I'm attaching recoveryd.log, which is after flashing your disable...zip. Also, I'm attaching recoverym.log after flashing Magisk zip.
[Android Software Development] Zackptg5:
Looks like I'll need to read the magisk docs more on .magisk files. Zip worked fine but magisk then removed the .magisk file from ramdisk and made a new one :/

Magisk also segfaulted - don't know what's up there
[Android Software Development] sga999:
I've been looking at the scripts for Magisk. It looks like get_flags invokes getvar for variables FORCEENCRYPT and FORCEVERITY (and more). gevar contains this line: VALUE=`grep_prop $VARNAME /sbin/.magisk/config /data/.magisk /cache/.magisk`

But only /data/.magisk seems "accessible" when installing Magisk during TWRP, so it seems like your zip file MUST place the .magisk file in /data in order for the subsequent Magisk install to see the overrides.

I haven't looked closely enough yet at your anykernel.sh. But there are some copies at line 167 that should copy from config to /data/.magisk, etc. But for some reason, those must not be happening for my device/rom. I'll keep learning more, but I thought this might give you a hint about what's going on.

EDIT: Just for a test, I added a line to copy the .magisk file at about line 165.

Added this: cp -f $home/config /data/.magisk

Before this elif [ "$ROOT" != "SuperSU" ]; then

It appears to work, but I haven't tested it much. You will probably want to add more lines, making it more like lines 167-172.

Things I don't know:

1. What is $home? Is it /sbin/.magisk?

2. It seemed like I couldn't access /sbin/.magisk/config and /cache/.magisk in TWRP, only /data/.magisk. See 2a. below. However, I know a config file was written there, so it must be accessible. I'm confused!.

3. In TWRP file manager or terminal, I can't see files/folders starting with a dot. Why is that?

2a. When I say files are not accessible, here's why I say that. While in TWRP, in adb shell, I issue:

sed -n /KEEPVERITY=/p /sbin/.magisk/config /data/.magisk /cache/.magisk

Here is the output from that. Only the middle one, /data/.magisk works.

sed: /sbin/.magisk/config: No such file or directory

KEEPVERITY=false

sed: /cache/.magisk: No such file or directory
[Android Software Development] Zackptg5:
Ya, I get where I messed up. So:

the .backup/.magisk file is mounted to /sbin/.magisk/config during boot so that's only accessible during bootmode (this is what I overlooked)

So I'll place it in data/.magisk but I'll still attempt to extract it from .backup since that's the one that magisk itself uses and has some extra vars (SHA1 and RECOVERYMODE)

$home is set at the top of update-binary of installer, it's just equal to the working directory (/tmp/anykernel)

any file or folder with a '.' in front of it is hidden

not all devices have a /cache partition but in that case, /cache should be a symlink to /data/cache
[Android Software Development] Zackptg5:
[MENTION=4541967]sga999[/MENTION] Got a new test build here that'll always create the data or cache/.magisk file in addition to ramdisk one.

I'll need to look into supersu but at least with magisk, order of flashing no longer matters - you can flash magisk before or after this zip. It's actually preferred to flash magisk before this zip now
[Android Software Development] sga999:
Thanks, with minimal testing so far, this looks good.

About accessing files while in TWRP, thanks for explaining why I could not see /sbin/.magisk (or config). In my system, /cache is a symlink to /data/cache. But after flashing your zip, there is no .magisk file there. I am able to copy a .magisk file there myself, and the 'sed' command will work with /data/cache/.magisk, but not without /data in front. So it seems like Magisk could change its getvar() list of files to accommodate this, but that's another story! (Since /data/.magisk does work in the 'sed', that's good enough). Anyway, I thought I'd let you know that your zip does not seem to be putting .magisk in cache. Maybe because /data/cache is required for my system?

As you mentioned, I do see SHA1 in the config or .magisk file if your zip is flashed last (I do not see RECOVERYMODE). SHA1 does not appear when yours is flashed first. My preference is to have yours first because when Magisk is first, it shows that it is keeping dm-verity and forceencrypt. (I do know that that is not what will actually occur, but I'd rather not see that). Also, I had liked the idea that your zip was passing the .magisk file on to Magisk. So other than SHA1 not being there, is there any other reason that I should not flash your zip first?
[Android Software Development] Zackptg5:
So the zip will only create .magisk in cache if data isn't writable for some reason so in your case, it should be /data/.magisk (and in most other cases as well) - cache is more of a last resort kind of thing

The SHA1 is created by magisk itself and magisk only places that into the ramdisk copy (sbin) so everything you're seeing is expected. Seems like I finally got it right this time lol
[Android Software Development] sga999:
Since you said it was preferred to flash Magisk first, I was just wondering what the reasons are. Was it only a concern only for supersu?
[Android Software Development] Zackptg5:
This is a new thing for magisk only. So magisk will create the .backup/.magisk file in ramdisk replacing what's already there. Then flash this zip and it'll modify it.

I was concerned originally that magisk did some extra stuff to remove that this zip didn't but I got that all figured out now
[Android Software Development] ne0t:
Hi, does the newest trial version works with encrypted Official MIUI ROM?

I tried the zip in the OP on my Redmi Note 7 Pro and when the phone rebooted, there was an error msg saying 'the system has been destroyed', I had to dirty flash the ROM to fixed it.

My RN7 pro came encrypted by default and the TWRP can decrypt and mount the data partition just fine.

I'm trying to use your zip to prevent MIUI ROM replacing TWRP with stock recovery.

Thx.
[Android Software Development] Zackptg5:
The new stable is finally here! It's the same as the last test build I posted. I updated the flashing instructions in the OP so be sure to check it out. Only change is:

Root should be flashed before this zip now due to how .magisk/.supersu files work. Flashing root after instead will still work fine too but .magisk file in ramdisk that magisk creates may not be correct
[Android Software Development] Zackptg5:
I have no idea, never messed with miui. Is your bootloader unlocked? Did you format data as described in the OP before flashing anything?

If all you want to is to prevent twrp from being overwritten, you can just delete /system/bin/install-recovery.sh

The only thing this zip could potentially change in system would be any fstabs in vendor/etc if the device doesn't have a separate vendor partition
[Android Software Development] radz_:
Flashing steps is a bit confusing now, most of the kernel devs says - flash the kernel last. OP says - flash this zip last. Enlighten us please.

Can we flash, rom, magisk, decrypt, kernel?
[Android Software Development] Zackptg5:
It depends on the kernel but you're probably fine flashing kernel last. However, I always flash kernel, then magisk, then this zip because magisk backs up boot img before changing it and I'd rather have it back up boot img with custom kernel in it
[Android Software Development] ne0t:
Yes, bootloader is unlocked, TWRP installed and can decrypt the encrypted partition.

I didn't format Data partition because I want to keep the default encryption from official MIUI ROM.

There is no /system/bin/install-recovery.sh here, I've tried search for it.

My problem is all the changes I made (like replacing hosts file) inside TWRP is gone with every reboot.

edit : found 'install-recovery.sh' in /system_root/system/bin, renamed it to install-recovery.sh.bak, reboot to system, reboot to recovery, TWRP still gone.
[Android Software Development] amg314:
You do not need use TWRP to edit system files.

Do it in any root file manager (like Total Commander).
[Android Software Development] ne0t:
I didn't edit it manually, just flash some zip files.

Actually I don't want to root my phone, but because I was stucked, I tried flashing Magisk

19.3 (latest) and my phone keep rebooting to TWRP, flashing Magisk uninstaller fixed that, so I can't root my phone while TWRP is working perfectly fine, except that it get replaced with stock recovery everytime it reboot to system, and somehow this made all the changes made in TWRP idiscarded.

Same problem happened once with my mi 8 lite (also encrypted on official MIUI ROM), but there is an option to disable dm-verity on the TWRP and it worked, since the TWRP I'm using on RN7 Pro doesn't have that option, I tried this zip file, and so far flashing this zip destroy the system.
[Android Software Development] amg314:
Try flash TWRP then reboot to TWRP (without rebooting to system). Atfer that flash Magisk.zip and reboot to system.

I hope, it may help to prevent TWRP bootloop.

But, you will be rooted((
[Android Software Development] ne0t:
already tried that..

I tried these :

- flash twrp, boot to twrp, reboot to twrp just to make sure, reboot to system, reboot to recovery, TWRP was gone.

- flash twrp, boot to twrp, reboot to twrp just to make sure, flash Magisk, reboot to system, failed, it keep rebooting to TWRP (need to flash Magisk Uninstaller to fixed that.
[Android Software Development] amg314:
Try use "one session" TWRP recovery:

In adb run:

adb devices

adb reboot bootloader (switch to fastboot mode)

fastboot devices

fastboot boot recovery [your_TWRP_name].img

This TWRP can be used until next reboot. Atfer reboot you will return to stock recovery.
[Android Software Development] ne0t:
Yup, you're right, until a better solution come, this is exactly the way I do whenever I need to flash something in TWRP.
[Android Software Development] zzzhead:
Something strange is going on with op6t after using this. After setting a pin or pattern, the system always detects pin or password as incorrect.

Tried clean flash on afew roms eg havoc, resurrection remix, same behavior.

If device is encrypted, there's no such issues

Could anyone shed any light on this?
[Android Software Development] ZeroKool76:
You can't decrypt the 6T, end of story. Sorry. #thankop You can decrypt the 6 and 7 fine, guess they decided to take a dump on us. But that should've been obvious when they released the McLaren edition. Nice way to thank those that bought the device when it was released, by releasing an upgraded version a month later.
[Android Software Development] HaRdC0r3:
Is there a way to flash this without root? I'd prefer to have a stock experience.
[Android Software Development] thoiloidainhan:
You just need a working custom recovery like twrp to flash it. At least in my experience
[Android Software Development] vrda08:
Does this works with xiaomi devices?! Is anyone using it? I just order k20pro and I am use to have decrypted device for easy switching between roms..
[Android Software Development] robygt:
I'm using 5/21 build on Mi9 and always worked, I've just seen a newer version (8/2) and will use that from now on.