yashsingh1/xdaforums huggingface dataset:
[Magisk] VR25:
Thanks for all the feedback!

I was going to release the next huge update a few days ago. Unfortunately, it didn't pass all my tests. I'm currently fine turning features and fixing "smart bugs" (those are some little s#$%&*! BTW).

Was tempted to post more info about new features, but you guys can wait a little more... right?

[MENTION=9231878]Terrigible[/MENTION], the upcoming update will include that feature. Thanks.

[MENTION=303421]Yom[/MENTION], greatly appreciated...

Restores will hopefully be faster (and automatic). As soon as you go past the initial Android setup screen, your apps will start popping on the launcher's app drawer.

Already spoke too much... I'll be back soon.

---

EDIT: Module Updated

**2018.9.13 (201809130)**

- Ability to change where to restore backups from.

- Backups of uninstalled apps can be removed selectively.

- Blacklisted Magisk versions 17.0 and 17.1 in addition to 16.7 (adk will refuse to install; run touch /data/.adk to override).

- Block direct downgrade and direct legacy upgrade.

- Cleaner design

- Config keywords 'inc' and 'exc' work with globbing/regex patterns.

- Improved backup/restore algorithms.

- Migrate apps+data before factory reset & auto-restore everything shortly after boot (unless 'noauto' config keyword is set).

- Misc fixes & optimizations

- No more on-boot data moving nor bind-mounting.

- Option to run all scheduled backups on demand (adk wizard, 4).

- Removed standalone post-fs-data.d/adk.sh in favor of \$modPath/service.sh.

- Save a copy of the default config.txt to \$MODPATH & automatically restore it whenever $Config is missing.

- Save temp files to a non-persistent dir (/dev/adk_tmp).

- SELinux mode is no longer altered.

- Updated documentation & debugging constructs/tools (new log formats, better error reporting & handling, and more).

- When the 'wipe' feature is enabled, all Magisk modules, plus adb keys, ssh config & keys and a bunch of other system data (i.e., accounts, contacts, SMSs/MMSs, call logs, saved Wi-Fi networks, settings, etc.) also survive factory resets.

*Release notes*: $Config will be overwritten (new info). Uninstall AND reboot into system before upgrading (installation will fail otherwise, as apps data must be migrated first).
[Magisk] @krishne35:
Magisk 17.1 support any time soon

Coz on pie we can't use 16.0 so
[Magisk] VR25:
Run touch /data/.adk as root before flashing to force install, then test whether it works.

I'm not officially supporting it since the issues don't seem to be on adk's side, but Magisk's itself.

P.S., can't you use 16.6 either? It seems to run pretty stable, as that's been my daily driver for quite a while.
[Magisk] totalnoob34:
I'm trying to install latest version, Magisk 16.6.

I've uninstalled the previous version, and rebooted several times, but the Magisk installation fails. Should I install the module via TWRP?

EDIT: Just tried TWRP, installation still fails. I'll try downloading the .zip again and report back.

EDIT2: No luck. Both TWRP and Magisk installs fail, at

- Mount /data/adb/magisk_merge.img to /dev/tmp/magisk_tmp
! Installation failed
[Magisk] VR25:
Are you getting any descriptive failure messages, or just "installation failed"? Does that happen before "mounting /system /vendor" even shows up? What about install logs... can you grab any? Thanks.

EDIT: logs are really needed here! :D
[Magisk] totalnoob34:
The log is attached. Thanks a lot for the support, I'm really looking forward to using this module!
[Magisk] VR25:
You're welcome. The log had exactly what I needed. Thanks!

Build 2018.9.13-1 is up.
[Magisk] totalnoob34:
I just installed the module in Magisk, and everything went smoothly. I'll report back on bugs and the new features.

Thanks again.

[MENTION=5228676]VR25[/MENTION], I'd like to test the 'wipe' featureon my non encrypted device. If I understood correctly, it additionally backs up the Magisk modules and other stuff. In the Github readme, I see that with this option enabled, one shouldn't perform a factory reset before flashing a new ROM. Does adk still perform a wipe or I should wipe system, cache partitions myself?

EDIT: nvm, just figured it out it wipes most of /system on its own. I should read more carefully.
[Magisk] VR25:
It doesn't wipe /system, only /data. That's done while preserving most of the useful data you would want to keep (i.e., contacts, WI-FI networks, call log, Magisk modules, adoptable storage keys, and much more.).

There's one thing I need to know before enabling the 'wipe' feature by default: "what's left in /data after a TWRP factory reset (when data is encrypted)".

Someone who uses encryption would have to make a nandroid backup, save it somewhere else (OUT of /data), then wipe (not format) /data (with TWRP) and copy everything left (except /data/media) to /data/media/somefolder, so that we can check whether any encryption key is there. If there's one, you can already imagine what would happen if it's not preserved... everything in /data (inc. /data/media) would be lost!
[Magisk] totalnoob34:
I understand.

I just tried doing a ROM flash using adk with wipe enabled. The order of flashing was:

adk (latest, I see it outputs what it wipes)

ROM (which happens to include Magisk 16.6, is that a problem?)

Gapps

I then wiped /cache and dalvik, and rebooted. First off, I didn't go through the usual setup wizard, is that normal? My settings were preserved (it's RR, so all its tweaks, background, etc), as well as Magisk modules.

Being noauto disabled, I expected applications to be automatically restored, but nothing was happening. So after a couple minutes I opened adk from the shell, and tried restoring packages that way. $e/adk/backups had not been renamed to backups.old, if that is relevant. To avoid accidental overwrites, I renamed it myself.

Anyway, trying to restore app/data (option 3) resulted in the following error:

(i) Input pattern(s) to match (i.e., sp.*fy or duk|faceb|whats. A dot matches all)... wiki

Restoring apps+data...
- org.wikipediaError: Can't open non-file: /mnt/media_rw/8580-1BEA/adk/backups.old/apk/org.wikipedia.apk
Error: java.lang.IllegalArgumentException: Unknown package: org.wikipedia

So I thought, fine, I enabled wipe and something must've survived that it shouldn't have. I then proceeded to perform a TWRP factory reset, ran through the setup wizard, installed adk via Magisk Manager, and tried restoring apps this way. No luck, still.

I'm not sure what I did wrong, I assume something went wrong with the restore process. Fear not, I pulled main.log from /data/media/adk/logs, it's attached to this post.

EDIT: I'm available to perform additional tests, feel free to PM me.
[Magisk] VR25:
[MENTION=4628223]totalnoob34[/MENTION], the built-in Magisk is fine.

Your settings, inc. setup wizard were preserved. That's good.

Now, looking at the log, the first thing which stands out is, your folders in /data/app are suffixed by -2OsxMlVF1Ckjp370OlVsjg==, so that the standard "/data/app/com.google.android.gms-NUMBER" is instead "/data/app/com.google.android.gms-2OsxMlVF1Ckjp370OlVsjg==". That's a major issue!

Is this a Resurrection Remix thing?

Also, did you check whether your apps+data were actually backed up to <external storages>/adk/backups?

BTW, I appreciate your willingness to perform further testing. Thanks.
[Magisk] goldencore:
I had an old install of adk which wasn't working for me, so I uninstalled it. There is still a folder /media/adk with some files in it.

I can't delete these files even using a root explorer or the terminal with su. Anyone knows what's causing this?
[Magisk] totalnoob34:
That's good to know.

I honestly have no idea. I can type stuff in the terminal, but I don't dig deeper than that. I suspect it's more an issue with my install, but being my G5 very poorly mantained it could be the ROM's fault as well. I just reinstalled the same ROM, and sure enough the folders in /data/app are all padded with the same -*== string. I'm moving to LOS, I'll see if the situation changes.

What you're saying is that it's installed ROM's fault, right?

Yes, all the apks + appdata were backed up in $e/adk/backups. Do you happen to know why adk didn't rename the folder to backup.old on its own? When is it supposed to do that again?

That's no problem. It's also in my interest to have this module as stable as possible. The concept is an ablsolute game changer.
[Magisk] VR25:
Is the path really "/media/adk" instead of "/data/media/adk"? If so, that's crazy! I wonder how that folder would get there...

Anyway, if that's the case, run su -c mount -o remount,rw / && rm -rf /media/adk && mount -o remount,ro /.

Yes, and that string is the issue.

The renaming from "backups" to "backups.old" happens right after apps+data migration and before "wipe" kicks in. Why that failed is a mad question mark. The apps+data migration subshells likely crashed due to that odd suffix. However that shouldn't have interfered with "backups" migration at all. A TWRP log would reveal the missing pieces of this little puzzle.

That's good to know.

P.S., I suppose your APKs were backed up with that strange suffix. This could be the reason why the manual restore failed as well.
[Magisk] totalnoob34:
I just booted a freshly installed LOS, and a ls in /data/app yields

com.topjohnwu.magisk-DnuoiSwDBMDIeIAmI2gAiw==

I'm not sure what to make of all of this, but I'm starting to wonder if all the builds for my device (LG G5, variant H850) have that suffix. I suppose you could account for this variation, it's entirely possible some other device has that funky business going on.

Can I still pull the TWRP log? I've done other wipes in the meanwhile. Where would I find it?
[Magisk] pedritopl:
It's the same for me on OnePlus 3T with official Lineage 15.1.

My guess is that the suffixes might be related to the data encryption.
[Magisk] goldencore:
Sorry, I am stupid! Was trying to delete in some /emulated folder.
[Magisk] VR25:
No worries!

The encryption, of course... that suffix could be a side effect of it. ?

Currently working on that...

You would have to go through the whole process over again...

Install an app or two, then perform the adk migration/wipe.

Before rebooting into system, grab /cache/recovery/*log* and/or /tmp/recovery.log or just go to TWRP "advanced" section > "copy logs to sdcard". See of you can also get a fresh main.log for comparison.

EDIT: [MENTION=4628223]totalnoob34[/MENTION], install the attached zip from TWRP and use it instead of the latest on the repo. Run touch /data/.adk before the first flashing (forced upgrade). I changed some stuff to handle that odd suffix and stop right away, with a more descriptive error message if apps+data migration fails.
[Magisk] totalnoob34:
Will do. I'll do a standard TWRP factory reset. In the meanwhile, I also posted in my ROM's thread to see what the developer says.

EDIT: [MENTION=5228676]VR25[/MENTION], the recovery.log file is attached. I'm reinstalling adk right now, main.log will follow shortly.

EDIT2: main.log is attached as well.

EDIT3: [MENTION=5228676]VR25[/MENTION], please make a post, this editing thing is complicated)

I used the DEBUG build you posted, this is what I did:

Installed adk DEBUG (I'll refer to it as adk from now on) in magisk, after forcing the update.

Rebooted to TWRP, flashed adk again, no error prompts. I see that apks in $e/adk/backups/apk no longer have the infamous trailing string.

Standard factory reset.

Reboot to system, installed adk, and tried restoring the apps I had.

Now apps restore fine, and no additional prompts are printed. However, appdata doesn't seem to be restored. I did some additional testing with the ROM booted, and adk will restore an app fine if its data is wiped first, but it ignores (Solid Explorer) or corrupts (Titanium Backup, crashing) its data.

Definitely a step forward, I pulled main.log and attached it as mainNew.log.
[Magisk] VR25:
Indeed...

Nice! We're getting there. Now I need to take a look at your /data/system/packages.xml and packages.list files, respectively. Could you upload these? Thanks.

BTW, you're still using encryption, right?

In order to speed up the debugging process and fully support encrypted data as soon as possible, I may have to encrypt my own data. Unfortunately this takes a while and I only have a single device in hand. Will see once it's "safe" to disconnect from the word for a few hours...
[Magisk] totalnoob34:
They're attached. I renamed *.list to *.list.zip so that xda would upload it.

No, I never was.

I get that, I'm doing this troubleshooting on my daily driver /: I don't unfortunately have any other device at hand, otherwise I'd be delighted to do that on it.
[Magisk] VR25:
pkgName[-suffix]

[MENTION=4628223]totalnoob34[/MENTION], could you confirm there are no such trailing strings anywhere in /data/data?

Also, where's the wizard.log file? :p
[Magisk] VR25:
Yeah, got no choice but to dive deeper

Need supplies to survive down there and you got plenty

Gotta gimme some

Don't be evil ?
[Magisk] totalnoob34:
I won't, for this one time :highfive:
[Magisk] VR25:
Found out all I needed to know about the sneaky trailing strings, and exactly how to handle these.

I can make the necessary changes to the manual restore part fast.

The automatic mechanism, on the other hand, will take some time.

Within the next 24 hours or so, a new version should be up.
[Magisk] totalnoob34:
That's very impressive. I'll check out the updated version when it hits the repo.

Thanks a lot for your dedication, this community needs people like you. Kudos, man.

EDIT: If you have the manual-only version on hand, please post it. I've set up a script and renamed the apks as their data folders, and I'm curious to see if the apps finally restore correctly before restoring the old TBK way.
[Magisk] Sander9847:
Wow this seems like a great project!!

While reading the least pages found out I have the same problem; the weird strings on RR even though my rom was never encrypted.

Really looking forward to using/testing this once new build is out ?
[Magisk] totalnoob34:
That's a relief, I was worrying I was the only one. Which device/ROM are you on?
[Magisk] JVDH:
Just wanted to add that this is also the case on my phone. I have a Oneplus6, OxygenOS v5.1.6 (Android v8.1.0), Magisk v16.0 (downgraded from v17.1 due to SafetyNet failure happening randomly after it initially passed) and adk v2018.9.3. Encryption is active.

Guess that makes three of us :)

I too would love to see this work, and I'm very impressed by the work and dedication that already went into it. Respect!

(Sorry for not being able to post logs. Since I'm a real total noob, XDA doesn't let me yet...)
[Magisk] goldencore:
I am on Xiaomi Note 4, using LiquidRemix Rom (AOSP based, 8.1.0) and I also have these strings in the app names. No encryption
[Magisk] totalnoob34:
Heh, fear not. You already know much more than I knew when I picked this username.

Consider updating to Magisk 16.6, which is the latest stable version before v17.
[Magisk] Fif_:
I've done these many times on many devices:

A TWRP data wipe will delete everything but /data/media from /data. The encryption status is left as it is.
[Magisk] VR25:
**2018.9.14 (201809140)**

- Added support for odd package suffixes (/data/app/pkgName<suffix>) causing apps+data backup/restore to fail.
- Fixed 'noauto config keyword not being properly recognized'.
- On nonzero exit code, revert changes to the recovery environment and don't leave magisk image mounted.
- Save \$Pkg symlinks to \$appDataBkps/\$Pkg.lns (formerly \$appDataBkps/\$Pkg/symlinks.list) for faster rsync update checks (faster incremental apps data backups).
- The dedicated factory reset mechanism wipes /cache as well, exc. magisk.*img (for compat. with 'f2fsfix' module) and magisk_mount/.

This can now handle all sorts of crazy suffixes and similar stuff.

These trailing strings are random. Whenever an app is updated/reinstalled/downgraded, its suffix (/data/app/pkgName<SUFFIX>) changes. So, I had to burn my brain a bit more to go over that.

Thanks for the awesome support and enthusiasm!

P.S., if you're going to rely on the "wipe" feature for future factory resets, you don't need to manually wipe/format anything else. That's all handled automatically to simplify your life and save time. Flash adk then your ROM. All Magisk modules and most system settings, contacts, call logs, saved Wi-Fi, etc., are preserved.
[Magisk] Sander9847:
Thanks for the update!

Has it been tested to work normally on trebble/gsi rom switching?
[Magisk] VR25:
You're welcome.

No one has shared a word on that yet. But it should work. Testing is up to those who own these devices. I can't wait to put my hands on one!

You could test adk wipe, then. Still, I recommend taking a FULL /data backup and saving it on a different storage device, just in case.
[Magisk] totalnoob34:
I just did an adk wipe on my installed ROM (LOS 14.1). Everything worked flawlessly, watching apps pop up in the drawer is mesmerizing.

I'd like to make a further suggestion, though: in the adk main menu, option (3) led me to believe it listed all the applications backed up, prompting to select which ones to restore, and instead it lists the applications backed up but not currently installed. I think this behaviour is confusing.

I suggest option (3) to give the user the option to restore backed up app+data that are alreadty installed, or list all the detected backups, or giving a clearer label to that option.

This module is freaking awesome. Thanks a lot for your hard work.
[Magisk] Terrigible:
Man I wish this was available on Magisk 17.1
[Magisk] VR25:
Thank you, [MENTION=4628223]totalnoob34[/MENTION]. I'll make your suggestion a thing. BTW, isn't that username of yours misleading? :rolleyes:

The "not supported" status is entirely based on user reports on earlier adk versions. I'll be testing Magisk 17.1 very soon.

You could force install and test yourself by running touch /data/.adk before flashing.
[Magisk] totalnoob34:
Great! As for the username, it goes back a long way to the good old days of single core CPUs, and less than a gig of RAM. Joining XDA and not knowing what an APK was, I thought I had to be straightforward about it, so here's the total noob story.

I went a long way, but I still have a lot to learn. I guess you never truly stop being a total noob at something ;)
[Magisk] VR25:
That's a very nice little story right there! And you're right. We all have a "total noob inside", whereas PCs usually have an Intel CPU or more.
[Magisk] totalnoob34:
That's right, Mr. BJT ;)
[Magisk] goldencore:
Current version won't install under Magisk 17.1 for me. Not even in recovery. I did use the touch command. The installation aborts with an error.
[Magisk] dingsaller:
same here

(OP3/latest LOS15.1/Magisk 17.1/latest Caesium kernel):confused:
[Magisk] VR25:
[MENTION=2433195]goldencore[/MENTION], [MENTION=8027614]dingsaller[/MENTION], which kind of error? Any descriptive message? You didn't run "touch /data/adk" instead of "touch /data/.adk", did you?

Please, be as specific as possible and provide install logs whenever you can. Thanks.
[Magisk] dingsaller:
[MENTION=5228676]VR25[/MENTION]

first "touch /data/.adk", then tried to flash your module, see attached log

Tried in magisk manager and TWRP
[Magisk] goldencore:
I created the hidden .adk with the touch command.

My log shows the same error as in the previous post. (Uploading somehow fails in the xda app)
[Magisk] caobe:
That issue happens to me too in both TWRP and in Magisk app
[Magisk] VR25:
[MENTION=5794050]caobe[/MENTION], [MENTION=8027614]dingsaller[/MENTION], [MENTION=2433195]goldencore[/MENTION], my bad... the log basically says I didn't instruct the installer to automatically create the path /data/media/adk. Do that manually and adk should install afterwards. Will fix that. Thanks.

EDIT: that's actually not the case. For some reason, the path is not being found/created. Please, check whether it exists and report back.
[Magisk] dingsaller:
[MENTION=5228676]VR25[/MENTION]

:good: added path /data/media/adk manually

and flashed your module successfully!

Thx for your support
[Magisk] goldencore:
I created the folder manually and the installation worked. The folder was not there after the failed installation attempt.

(I set permissions to 777,if that matters).

I started a backup through the wizard and it seems to work. Apps and data were backed up to external storage.

Could you clarify what the wipe option does? Does it wipe anything immediately or is that something that only happens when you do a factory reset? Somehow the Readme leaves me clueless.

Same with noauto. My understanding is that apps won't get restored automatically when a new rom is flashed, but you have to use the wizard. Correct?

Are backups still executed?
[Magisk] totalnoob34:
It left me a bit puzzled too. Following the readme, you first reboot to TWRP, then flash the currently installed version of adk. From my understanding, adk then performs a full config.txt run, and consequently performs 'wipe' if set to do so. That does all the wiping you need (most of /data and /cache) for ROM flashing, so you don't need to wipe anything yourself. I believe wipe is intended as a factory reset alternative, with the advantages that most configurations (refer to the readme for a full list) are preserved.

As for noauto, I never used it, so I don't know how it behaves as far as new backups are concerned.

Just my two cents: I think 'wipe' is not a very effective name for the feature, maybe 'wipeonflash' or something self-explainatory would be more accessible?
[Magisk] VR25:
"wipe" is a factory reset performed by adk when you flash the same version again from TWRP. Flashing from Magisk Manager sets the module to be uninstalled on the next boot and it doesn't trigger a factory reset.

Unlike TWRP and similar "wipe" methods, adk's preserves all Magisk modules, plus most system settings.

During an adk factory reset, if "noauto" is enabled, apps, plus respective data aren't migrated. Consequently, no auto-restore takes place. You have to use the wizard.

All Magisk modules and most system settings are preserved, regardless of "noauto" status.

This config keyword is currently intended for debugging purposes only.

Keep the nice questions and suggestions coming!

Indeed wipeonflash looks much better! How about wipeOnReflash?

P.S., a new version is almost ready. I implemented all suggested changes/features up to this point. Finally had a chance to test Magisk 17.1. It may soon be fully supported. So far, the only thing not working is adk's auto-start mechanism.

EDIT: yes, "noauto" doesn't block automated backups.
[Magisk] goldencore:
Thanks for clarifying these points.

What happens when you flash adk again within TWRP without wipe in config.txt? Will it get uninstalled? I am just curious.

So, as long as you only use Magisk, wipe can be safely put in the config without accidentally wiping your phone? You could even update to a newer version via TWRP, because a wipe is only performed if you flash the same version again.

Is that it?
[Magisk] Sander9847:
I downgraded to magisk 16.6 to try it out and keep getting installation failed error in Magisk and TWRP.
[Magisk] totalnoob34:
Both look good to me, maybe 're-' could be interpreted as having to flash the module twice, but I may be overthinking it.
[Magisk] Tischbein:
[MENTION=6550197]Sander9847[/MENTION] try to create the directory '/data/media/adk/' manually before installing the module.

cp: can't create '/data/media/adk/config.txt': Path does not exist
[Magisk] VR25:
No, it won't. Reflashing from TWRP never triggers the uninstaller.

That's right.

[MENTION=6550197]Sander9847[/MENTION], refer to the previous post.
[Magisk] goldencore:
My observation is that under Magisk 17.1 the backup is not automatically created after the time given by bkpFreq.

Is that what you mean?