yashsingh1/xdaforums huggingface dataset:
[Magisk] VR25:
Even better, copy YouTube data folder to a temporary location (/data/media/) as the first step.

rm -rf /data/media/.*youtube (only if folder exists)

cp -al /data/data/*.youtube /data/media/

To restore, run "M" or "migrator" as root, choose option 2, write yout, press <enter>, and you're done.

You won't have to do the copying part in the next version. There will be a timeout after boot for refreshing snapshots. The default will be two hours or so. That's plenty of time for data restore before backups get overwritten. If you forget to restore within that timeframe, you'll still have /data/media/migrator/backups.old, from which you can restore YouTube's data using the wizard. Two hours after the next boot or after *bkpFreq mark is hit, that folder will also be refreshed.

* Default: 8 hours after the first backup
[Magisk] iroshan:
Are the apps updatable via play store? Meaning does play store links survive?

I just used this and apps recovered well. Thanks.
[Magisk] chickentuna:
Yes. Auto update is fine
[Magisk] C.Hatfield:
VR25 or anyone who can help ....

I rebooted in to recovery (B partition), ran migrator twice, went to boot into A partition of my Pixel XL and everytthing is encrpyted .... I remember seeing something like this when I first got the device but cant recall how to deal with it
[Magisk] MarcoCount:
Thanks for this awesome tool!

Small question: is there any way to use this when having to change data filesystem?

Thanks a lot in advance.

I recall having the same issue once.
[Magisk] VR25:
So, you weren't using encryption before that, right?

Could you describe the exact migration steps you followed?
[Magisk] VR25:
You're welcome.

If you have external storage,

1) Perform a /data nandroid after migration.

2) Save that [TWRP] backup, /data/media/migrator/migrated_data and any other important data to the external storage.

3) Format data.

4) Restore data from the external storage.
[Magisk] Estebanium:
I guess that I completely misunderstood the description of using Migrator. English is not my first language and the description is a bit nerdy so I hope someone can help me find my fault.

I use Migrator for a while now that I installed as a Magisk module. Today I wanted to update my OS so I booted to TWRP recovery:

0. I booted to TWRP and entered my PIN, because my internal storage was encrypted

1. I flashed the same Migrator version and it performed a backup of all my apps.

2. The log showed that Migrator wiped /data at the end of this process.

3. I did not reboot and swithed to the file manager of TWRP to check if /data was wiped. I didn't notice any difference to before so

4. I headed to Advanced Wipe and selected everything except /data, SD and OTG. then

5. I added my ROM (LOS 16), OpenGApps and Magisk to flash queue and flashed all three zips.

6. I rebooted afterwards and I was asked to enter my PIN for encryption. I entered it and the system booted up. There was no initial setup, it was just there. Unfortunately I noticed that no Apps popped up so I rebooted and when I got some errors that prevented me from logging into the OS, I reinstalled everything with a full wipe.

Luckily I used Swift Backup before so I recovered all my apps, but I would like to know what I did wrong.

I am looking forward to your feedback, because I like the concept of Migrator and I would like to use the advantages of it.

Greetings..
[Magisk] C.Hatfield:
Sorry, I was in a rough place & I am still in the SAME position as I stated above. I stopped checking XDA bc I wasn't getting any feedback after several days - I have tried fishing through the directory in TWRP File Exp. [mnt/ system + data] --- I have found the apps backed up by Migrator, etc but decided not to f@ck w/ them just yet.

I finally created a thread in one of the PIXEL XL sections asking for help .... I will link it here & it will give a LOT of info .... I have ALMOST flashed device clean just to get on with things buuut I would really like to hold onto some of that 110GB of data ..... thread link >>>

https://xdaforums.com/pixel-xl/help/little-help-please-t3913425

I have been using Android since it came out & I have NEVER seen much less heard/read about anything like this happening before - I have pictures I took with another device bc it wont save or D/L any-thing bc it thinks theres no directory there to do so !! The data & system partitions are NOT encrypted, just the emulated/0 aka Internal Storage .... everything is a garbeled alphanumeric MESS.

BTW, NO External .... I would've definitely tried that trick! [Pixel XL 128GB]

**To answer your question .... No encryption before > this encryption has nothing to do with the encryption that TWRP hits you with when you have a PIN/Pattern, etc lock screen set up [at least, it seems unrelated to me] ---- As for steps, very simple > I booted into TWRP (latest) ran script twice, first time to back up, 2nd time to FDR (just as it says in directions in Magisk Module scetion in Magisk Manager) ..... wanted to try it out as I was about to move to a different custom ROM
[Magisk] VR25:
[MENTION=6867525]Estebanium[/MENTION], you followed all the steps, except the last: removing incompatible data.

I've been busy lately with other stuff, but I've also been working on a major update that will bring new features and friendlier/simpler usage/controls to the table.

[MENTION=3042765]C.Hatfield[/MENTION] again, this is very strange if your data wasn't even encrypted to begin with.

Did you forget to install Magisk before rebooting to system?

Recall that modern devices come with forced encryption enabled by default.

Unless you patch the boot image to disable that (Magisk does it), data is automatically encrypted.

Migrator doesn't encrypt data, touch /data/media/0 (/storage/emulated/0), neither does it mess with encryption keys (/data/misc/vold/).

I finished reading the other thread - let's continue the discussion there.
[Magisk] lmgc1996:
Oh man! You're the best. Thank you.
[Magisk] Estebanium:
Could you go a little further with your statement. I haven't quite understood it yet.
[Magisk] VR25:
Due to heavy Android ROM fragmentation, some system data from a particular ROM is incompatible with other/different ROMs. You can try running /sh /data/M on recovery terminal to see if it helps. Its purpose is removing the offending data for you. In the next version, this will be a wizard that gives you more troubleshooting options.

If the current implementation doesn't solve the issue, you have to factory reset. You lose system data, but not migrated APKs and respective data that weren't restored.

If you find this unfriendly or overwhelming, wait for the next release. I've been busy lately, but I'm almost done working on it.
[Magisk] Estebanium:
Thank you for the reply, I guess that I didn't understand the folder structure while in TWRP's File Manager. Am I right that if Migrator writes, that it wiped /data that the path in recovery to /data is /data/data?

Yes finally I flashed Migrator, factory reseted my phone, added ROM/Magisk to flash queue and got my new system in place. After installing the Migrator module in Magisk via Magisk Manager, my Apps started popping up again.

I have a question to the up-popping: I noticed that some/all apps lost their data, so the apps came up by theirself but I lost the data in the apps (e. g. I wasn't logged in in Reddit etc.). Was this because I factory resetted my phone?

I am not sure if this applied to all apps, because I used a backup software like Titanium to restore the data of those apps afterwards.

One last question: If I want to save the data of Migrator, because I want to perform a full wipe, where every data except on my external sd card will be lost, what folders/files do I need to backup of Migrator in order to perform a recovery after installing my ROM/Magisk/Migrator?
[Magisk] hyborian:
Great, will wait for your next release with anticipation
[Magisk] C.Hatfield:
[MENTION=3042765]C.Hatfield[/MENTION] again, this is very strange if your data wasn't even encrypted to begin with.

Did you forget to install Magisk before rebooting to system?

Recall that modern devices come with forced encryption enabled by default.

Unless you patch the boot image to disable that (Magisk does it), data is automatically encrypted.

Migrator doesn't encrypt data, touch /data/media/0 (/storage/emulated/0), neither does it mess with encryption keys (/data/misc/vold/).

I finished reading the other thread - let's continue the discussion there.

***I finally got up the nerve to read your post which I got notice of a mere 2 hours AFTER I finally gave up & did full FDR .... Yeah, it sucked but whatever .... I've had to do it once or twice before over the years. I responded further over in the thread I created.
[Magisk] Mickey03:
Can I backup all the installed Magisk modules using your module? If it is possible please help me!
[Magisk] fxsheep:
[MENTION=5228676]VR25[/MENTION] Please omit /data/unencrypted when performing a factory reset (wipe data automatically). On FBE enabled devices it causes problems.My whole data destroyed because of it.
[Magisk] Darkaeluz:
I just realized, this module has been deleted from the magisk repo
[Magisk] Estebanium:
Hey there any news on the update? I recently used the App Migrator and was really disappointed. No App was restored, because it was not possible to restore the data. A very weak result. Would it be possible to share a more detailed description to handle Migrator Magisk? What to do before wiping everything in order to flash a new rom. What to do in TWRP and what not. How to handle the /data folder etc. The description at the first page is very confusing imo..
[Magisk] BigBrosMo:
[MENTION=5228676]VR25[/MENTION] is this project dead or...?
[Magisk] goldencore:
I don't think so, he just has a lot of projects to maintain and maybe even a private life.

He seems a really active developer to me who answers questions very thoroughly on Telegram. But it's true that he has been quiet for a while now. We'll see...

He answered to questions on the acc Telegram channel today, so he hasn't vanished.
[Magisk] sproez:
Problem with uninstall

Hi! i love the app but dont need it anymore cos I'm changing phone and using the old one for something different.

BUT! i disinstalled migrator with magisk and rebooted multiple times, but still keeps working and I have to continuously delete the migrator folder!

Can you help me please? thank you!
[Magisk] handers1234:
[MENTION=5228676]VR25[/MENTION] Sir do you still developing this module? Or you have completely drop it?
[Magisk] VR25:
https://xdaforums.com/showpost.php?p=81561739&postcount=3304
[Magisk] ErbDroid:
Hi [MENTION=5228676]VR25[/MENTION] thank you for this! Repo link in first post is broken BTW :D
[Magisk] VR25:
The latest acc has the stable tag.

This means the new framework is ready and it's time to update the other projects.

I've been working on the new Migrator for a while.

It's no longer just another Magisk module.

Now it also runs without Magisk.

For the sake of speed, compatibility, flexibility and safety, I'm rewriting most of it.

Features such as runtime permissions backup and restore, split APKs support and much more, are already being implemented.

I'll be sharing development updates more frequently (and pre-release builds) on Telegram: https://t.me/migrator_group .
[Magisk] 73sydney:
Please, none of that you can only get x on Telegram stuff thanks mate

Some of us have been through that with other threads and apps on here, and the admins too....

Gentle reminder you cannot direct people to Telegram (or any other site or medium) for apps etc, you must post releases here, its the XDA rules
[Magisk] Estebanium:
Man I hate Telegram and the policy that some devs post only there. What is that Telegram thing? Just no.
[Magisk] knpk13:
What's wrong with telegram? Devs usually tend to post testing builds so they can communicate easily with testers as it's more of an instant messaging app than XDA which tends to be reserved for stable releases.

Not true. BetterBatteryStats is also distributed through a different platform now, if you want an example of a popular app. OmniROM has pretty much exclusively switched to Telegram and their website which is kind of a bummer, I prefer XDA for stable releases and telegram ONLY for testers who want to help more.

https://xdaforums.com/oneplus-5t/how-to/telegram-chat-channels-forward-t3765018
[Magisk] Estebanium:
Then it seems to me that the test versions from most devs have never reached a level that would have been worth a release on xda.
[Magisk] hyborian:
That's oversimplifying it lol. For one, some devs just like the interaction with very active members I guess, they tend to spend much more time over there than here. It shows why social media is huge, people love it more. Not for me, I like the slow interaction here at XDA, no information overload.

For my ROM, TWRP, kernels ... Telegram is where I see quick updates, and no info here at all for some (or the forum has been abandoned). Also discovered some good stuff here on XDA through recommendations in Telegram. So all good, am fine with it either way.
[Magisk] VR25:
Right now, it can backup/restore user apps (including split APKs) with respective data and runtime permissions.

It can also export/import backups to/from external storage.

Backups are exported to and read from a compressed archive (.tar.lzo).

Local backups take virtually no space (hard links).

System* data backup/restore will be added later.

I'll soon make the new migrator available at https://github.com/vr-25/migrator .

* Settings, WiFi access points, Bluetooth pairings, ADB keys, Magisk modules, etc..
[Magisk] 73sydney:
The wholes devs use "Telegram for testing and its easier to communicate" is a weak as piss argument

XDA's entire reason for being is to fiddle with stuff on phones, so if yuore moving that form XDA to anothe rplatform youre actually decreasing the number of people who can/will test and provide feedback

Im getting tired of reporting threads that do this

If i was an admin, id be closing threads at the first mention of Telegram, but thats me, its highly disrepectful to XDA, for giving a thread space, traffic and exposure, to then move it off to another site or medium
[Magisk] VR25:
Migrator - Android Backup Solution and ROM Migration Utility

It can backup/restore apps (including split APKs) with respective data and runtime permissions, system and Magisk data (including settings).

ZERO warranties, use at your own risk!

Install as a regular Magisk module (no reboot needed afterwards, though).

Alternatively, you can extract `system/bin/migrator` from the zip and run it as `sh filepath` (yes, no Magisk installation required).

The documentation is bundled.

Refer to `migrator --help` or `M -h`.

It can be exported to a file: `M -h > file`

If these executables aren't readily available (before a reboot), use `/dev/migrator` or `/dev/M`.

v2020.6.21-beta (202006210)

Rewritten from scratch.

https://github.com/VR-25/migrator/releases/
[Magisk] VR25:
v2020.6.22-beta (202006220)

Exclude code_cache/ from app data backups.

Fixed app data lib symlink issue.

General optimizations

Refresh APK snapshots after successful restore.

Updated help text.

https://github.com/VR-25/migrator/releases/tag/v2020.6.22-beta

For Clarification

This is a Magisk module*.

Unlike the very old versions, it does nothing on its own.

You have to specifically tell it what to do.

Refer to migrator --help or M -h.

There's no configuration or config files of any kind.

It does not use recovery (e.g., TWRP).

Everything is done live.

Rebooting after installation is unnecessary if you're ok with running /dev/migrator or /dev/M instead of their prefix-free counterparts.

* system/bin/migrator can be extracted from the zip and used as is (without Magisk) - as long as the system is rooted.

$ su

# sh filepath
[Magisk] VR25:
Regarding the Bundled NetHunter Terminal

migrator --help says...

The Magisk module variant installs NetHunter Terminal.

Highly recommended, it's always excluded from backups.

If you use another terminal, it must be excluded manually (e.g., "migrator --backup -v termux").

This is because apps being backed up are temporarily suspended.

Before restore, these are terminated.

Thus, not excluding the terminal that runs migrator will lead to incomplete backup/restore.

Why NetHunter Terminal?

It's free and open source, light, powerful, backed by Offensive Security (company) and regularly updated.

The APK is downloaded from https://store.nethunter.com/repo .

You can always compare the package signatures and/or checksums.

The homepage is https://store.nethunter.com/en/packages/com.offsec.nhterm .

P.S., having a bundled terminal also means users have everything ready right away.

EDIT

The latest version won't backup anything if /data/media/migrator/ is missing.

Create that directory manually: "su -c mkdir /data/media/migrator/".
[Magisk] VR25:
v2020.6.24-beta (202006240)

Backup/restore accounts, call logs, contacts and SMS/MMS as well.

Backup runtime permissions directly from the dedicated XML file.

Fixed mkdir issue.

Backup app specific Android IDs. Restore is WIP.

More modular system data backup method

Use `more` instead of `less` (buggy scrolling) for displaying the help text.
[Magisk] VR25:
v2020.6.25-beta (202006250)

Restore app specific Android IDs.

Fixed accounts backup not working.

Optimized parsing of options to avoid conflicts with certain package names.
[Magisk] VR25:
v2020.6.26-beta (202006260)

Backup app data modes (ditch hard-coded values).

Backup battery optimization whitelist (deviceidle.xml) as part of system data.

Backup Magisk data modes and ownership.

Fixed wifi hotspot settings backup not working.

General optimizations

Perform stricter safety checks during SSAIDs restore.

WARNING

This version will fail to restore data backed up with previous versions.

Use the backup creator (backed itself up too): `/data/media/migrator/bkp*/migrator.sh`.
[Magisk] VR25:
v2020.6.29-beta (202006290)

`--export` and `--import` are more advanced and flexible. Refer to the help text for details.

`--export` creates individual archives for each backup.

Automatic `M --ssaid` runs syncronously.

Enhanced SSAID restore.

Major fixes & optimizations

Updated help text.

EDIT

v2020.6.29-beta.1 (202006291)

Fixed "lzop, parameter not set" and SSAID related issues.

WARNING

This version will fail to restore data backed up with previous versions.

Use the backup creator (backed itself up too): `/data/media/migrator/bkp*/migrator.sh`.
[Magisk] VR25:
v2020.7.1-beta.3 (202007013)

Backups are no longer moved to bkp.old/.

Major fixes & optimizations

More flexible backup/restore options (e.g., app only, data only)

New backups replace old ones.

Updated help text.

WARNING

This version will fail to restore data backed up with previous versions.

Use the backup creator (backed itself up too): `/data/media/migrator/bkp*/migrator.sh`.
[Magisk] pedritopl:
Hi!

Running the first step of ROM migration guide from the help description does not work as expected with this version:

# ./migrator -bAmsS + com.android.vending EXTRAS

grep: bad regex '+com.android.vendingEXTRAS': repetition-operator operand invalid

[...]
[Magisk] VR25:
https://github.com/VR-25/migrator/releases/tag/v2020.7.1-beta.4
[Magisk] pedritopl:
Unfortunately I'm still getting the exact same problem with beta.4.
[Magisk] VR25:
Ok, I was able to reproduce the issue.

The next version will bring a fix.

Meanwhile, try M -bAmsS . + com.android.vending.

Notice the dot before the plus sign.
[Magisk] pedritopl:
That was it. Thanks :)
[Magisk] cruncher1234:
I tried the google version on my Redmi Note 5 and migrate did not restore anything.

It creates a .zip file which i then try to install with twrp / orange fox but after restart to system there is no helper and no message and of course the app that i used for testing is not restored.

Any clues ?
[Magisk] KaMyKaSii:
There is no connection between the "Migrate" app and Migrator
[Magisk] cruncher1234:
ok, than it was a misunderstanding. Sorry.
[Magisk] VR25:
v2020.7.8-beta (202007080)

Automatic backups (Magisk, init.d and Tasker flavors)

Major fixes & optimizations

More comprehensive `--list`

System data backup/restore sub-option is now D (formerly S).

README.md has a copy of the help text.

Updated documentation (ROM migration instructions, automatic backup configuration and more).

Verbose is redirected to /dev/migrator.log, which can be exported to /sdcard/migrator.log.bz2 with `M -L`.

Works in recovery environments (particularly useful for emergency backups).

https://github.com/VR-25/migrator/releases/tag/v2020.7.8-beta
[Magisk] VR25:
v2020.7.11-beta (202007110)

Auto backup config suports a new parameter: cmd="post bkp cmd".

Backup/restore apps' device-encrypted data as well (/data/user_de/0/<pkg>).

General fixes & optimizations

Replaced all instances of ${0##*/} with M in README.md.

The sysdata list includes more Bluetooth and accounts configuration files.

https://github.com/VR-25/migrator/releases/tag/v2020.7.11-beta
[Magisk] VR25:
v2020.7.18-beta (202007180)
​

General fixes & optimizations

The default compression method is <none> (faster, output: .tar file). Method here refers to "<program> <options>" (e.g., "zstd -1v").

The decompression/extraction method to use is automatically determined based on file extension (supported extensions: tar, tar.bz*, tar.gz* and tar.lzo*). The user can supply an alternate method for decompressing other archive types.

--list shows install status as well.

Option to restore only apps that are not installed: "-rn".

Updated data migration tutorial and Tasker setup instructions - export/import to/from external storage included.

Cleaner export/import interface.

Export/import uncompressed archives (-c -).

Fixed settings restore.

Using /data/migrator/ as the new data directory. Android 10 complains when it's placed anywhere in /data/media/.

$version is included in verbose.

NOTE

To restore backups from the old location, use the version that made such backups (/data/media/migrator/local/migrator.sh).

Alternatively, you can move migrator's data folder to the new location (mv /data/media/migrator /data/).

WARNING

The ROM migration process has two additional steps to account for the new backup location.

Failing to follow the instructions carefully will lead to data loss!

https://github.com/VR-25/migrator/releases/tag/v2020.7.18-beta
[Magisk] pedritopl:
Hi,

Small bug there. If I download just the binary and I name the binary anything different than migrate.sh then the script is not able to copy itself to the backup export directory, because it looks for the wrong filename. In my case the binary was /cache/migrator-v2020.7.18 and I got this error when exporting the backup:

cp: can't stat '/data/migrator/local/migrator-v2020.7.18': No such file or directory

It should look for migrate.sh in /data/migrator/local/ because that's the name which the copy there has by default.
[Magisk] VR25:
This will be fixed in the next version.

Thanks!

Due to a time machine malfunction, today's build comes from tomorrow.

https://github.com/VR-25/migrator/releases/tag/v2020.7.20-beta
[Magisk] VR25:
https://github.com/VR-25/migrator/releases/tag/v2020.7.20-beta.1
[Magisk] VR25:
https://github.com/VR-25/migrator/releases/tag/v2020.8.17-beta

This is close to a release candidate.

I've ditched TitaniumBackup and been relying on Migrator for a while.

The relatively low number of testers is a bummer, but understandable.

Gotta thank the brave souls who've been testing this thing.
[Magisk] wilsonhlacerda:
Think of somehow publish it on F-Droid. Once you do that I'm sure lot of people will turn their eyes to it and start using it.

F-Droid has users worldwide, a much wider user base than XDA members that read the Magisk section here. And lots of them are privacy conscious, looking for FOSS alternatives for good closed source apps (like Titanium). Migrator is even not listed on official Magisk repo currently (do you plan to submit it again?).

Concerning apps backup tool your Migrator module and OandbackupX (that is also a new fork) are almost the only open source alternatives out there.

Thanks for all your work on such helpful modules like Fbind, Migrator and ACC!