yashsingh1/xdaforums huggingface dataset:
[Galaxy S II Original Android Development] the.gangster:
Hm, I guess we can't make the emulated storage be selectable as a backup'able partition, can we?

(And probably also not for a restore target)

At least that is my impression after reading this (coming from here).

As I knew about the automatic handling of TWRP not to delete the media folders upon factory reset I would have also expected the backup of /data to include all media/# folders, but reading above opened my eyes. After comparing size values shown in recovery for /data with the folder sizes shown in Total Commander it is in deed equal to size of /data minus size of /data/media.

I thought I would have read about possible symlink-entries in twrp.fstab back a long time ago, but don't know if they can make the emulated storage a backup source/restore target.

And a w a r n i n g I just want to make aware of for those who read here:

As the current EMU-builds of TWRP were build on 7.1.1 I am very sure, that the brick-issues (incompatibilities due to selinux stuff) are back with it if combined with CM11 or CM12.1 kernels. So don't flash them unless you want to test emu-builds and remove them, in case you go back to CM12/11 ROMs. Even CM13 will most likely be affected!

(this report from [MENTION=6606927]xNombre[/MENTION] machtes the old issues of the first TWRP-3.0.0-0 build)
[Galaxy S II Original Android Development] the.gangster:
Hopefully [MENTION=414800]arnab[/MENTION] (that's why i quoted you here for) already got the attention of this thread and our thoughts, as he as he posted today in his official twrp thread to also start working on an emulated storage build soon and people are posting everywhere but here to express their expectations.
[Galaxy S II Original Android Development] OllieW:
My results of my first tests with emulated SD card:

- Shrinked and wiped internal sdcard with Lanchon's REPIT (lanchon-repit-20170115-system=1G-data=max-sdcard=min+wipe-preload=min+wipe-i9100.zip)

- Installed "twrp-3.0.3_EMU-0-i9100-TEST-20170128.zip" and "lineage-14.1-20170126-UNOFFICIAL-i9100_emu.zip" (I had to wipe "data" partition to install LineageOS otherwise installation was aborted). Installed latest pico OpenGApps (20170129).

- First boot up without any problems. Settings/Storage shows 14.68 GB usable internal shared storage.

- File manager can't access /data and /sdcard in safe mode. Direct ink to "/data/media" is accessible and content of emulated storage in "0" subfolder (/data/media/0) is shown.

- Stock LineageOS camera app is showing "Insert an SD card before using the camera." and I can't shoot any photos or record any videos to internal SD. There is no option to change location when no external SD card is inserted.

- Stock LineageOS browser app is showing "No SD card - An SD card is required to download 'filename'. - OK" when trying to download a file.

- Screenshots with longer pressed VolDown+Power keys will show error "Couldn't capture screenshot. - Can't save screenshot due to limited storage space."

UPDATE: Everything works perfectly now - the /data partition has to be formatted! Only wiping /data partition will cause the problems described above! The "Format /data" can also be found in "Wipe" menu of TWRP3.0.3-0_emu and has to be confirmed by entering "yes".

All tests are made without an additional external microSD card.
[Galaxy S II Original Android Development] Lanchon:
thanks for your detailed report!

[MENTION=2222292]rINanDO[/MENTION], there seems to be issues with your build :( please take note
[Galaxy S II Original Android Development] AchiNo:
Weird, I have the same build as he does and had none of his issues. Just tested again and still no issues. Maybe not wiped properly?
[Galaxy S II Original Android Development] OllieW:
Normally REPIT wiped all partitions (except /system and /data) completely - I wiped /system and /data manually before installing the i9100_emu build. But I will try again and report! I think that mounting emulated SD with FUSE was not configured correctly after first boot up...
[Galaxy S II Original Android Development] Lanchon:
remember that to really wipe the complete /data partition on the new TWRP (including emu storage) you need to choose FORMAT, not wipe!
[Galaxy S II Original Android Development] max6698:
I've also got emulated storage to work. It was a nightmare for me because I was already using encryption.

None of the standard encryption removal techniques worked for me. If I created an unencrypted /data using ADB and TWRP, the encryption would come back after I booted. I figured the phone knew about the previous encryption even though I wiped system, data and cache. In case it was storing a flag in partition 1 (efs) or partition 4 (param), I restored those partitions from a pre-encryption backup. That worked.

I was finally able to repit with lanchon's tool. Once I had emulated storage working, I restored /data from a TWRP nandroid backup I had taken just before I started this process. When I booted, encryption came back. Does anyone know why it would do that? I don't want to get rid of it, but I'd like to understand what's going on.

I also overwrote the internal SD card partition with zeros using the dd command. As mentioned in an earlier post in this thread, that card is now reported as "corrupted" (see attached screenshot). No big deal, but it would be nice if it didn't appear on the "Storage settings" screen at all.

I used TWRP 3.0.2-1, not the new version in this thread, with lineage-14.1-20170202-UNOFFICIAL-i9100.zip which I built myself using the latest sources.
[Galaxy S II Original Android Development] OllieW:
Thanks Lanchon - that was the trick! After wiping every partition (except external SD holding the new firmware and GApps) and formatting /data partition (not wiping - "Format /data" can be selected in "Wipe", but has to be confirmed by entering "yes"), I can use the emulated internal SD card (mounted as /sdcard) and everything works perfectly. I will correct my previous thread!
[Galaxy S II Original Android Development] max6698:
Hi rlNanDO,

I have a couple of observations about twrp-3.0.3_EMU-0-i9100.zip:

(1) The Format Data option under wipe is back. Hooray! If I had that earlier I probably wouldn't have had massive amounts of grief with my encrypted /data because that seems to be a great way to remove encryption.

(2) TWRP no longer asks for a password to unencrypt /data which means it can't mount an encrypted /data.

(1) helps with encryption, (2) doesn't.
[Galaxy S II Original Android Development] Lanchon:
if you wanted to wipe data, you should have told REPIT about it! instead of making it work overtime to keep your data in /data (and causing more wear on the eMMC storage chip).

just add "+wipe" to the data definition:

lanchon-repit-XXXXXXXX-system=same-data=max+wipe-sdcard=min+wipe-preload=min+wipe-i9100.zip
​
[Galaxy S II Original Android Development] Lanchon:
-using non-emu TWRP with emu storage is DANGEROUS. read thread to find out why.

-i'm pretty sure nothing was stored on efs or param, and restoring them was ill-adviced.

-formatting /data should be enough to clear encryption. in non-emu TWRP formatting /data is achieved by "WIPE data" on the TWRP UI. (in emu TWRP you need to invoke the "FORMAT" option and type "yes".)

-the phone was probably non-encrypted each time you wiped data.

-your build of android was probably auto-encrypting without asking on every boot:

https://www.extremetech.com/mobile/...isk-encryption-mandatory-for-most-new-devices

http://www.androguider.com/2016/10/forcing-encryption-on-android-should-it.html
[Galaxy S II Original Android Development] max6698:
Hi Lanchon,

I know the phone was encrypted because of this df command output:

i9100:/ $ df

Filesystem 1K-blocks Used Available Use% Mounted on

tmpfs 401316 500 400816 1% /dev

tmpfs 401316 0 401316 0% /mnt

/dev/block/mmcblk0p9 1378168 822692 555476 60% /system

/dev/block/mmcblk0p7 100784 6144 94640 7% /cache

/dev/block/dm-0 13578568 1857636 11720932 14% /data

/dev/fuse 13578568 1857636 11720932 14% /storage/emulated

/dev/fuse 15541920 857248 14684672 6% /storage/6776-628

My build of android has never autoencrypted. I have just got to the point where I can build android myself from repositories that I haven't messed with. I wouldn't have a clue how to make an auto encrypting build. Encryption arrived on my phone because I intentionally put it there.

After restoring efs and param, I went through several cycles of booting in to TWRP and system with no encryption. I know that because df showed /data was /dev/block/mmcblk0p10 and not /dev/block/dm-0. When I restored /data from a TWRP backup and rebooted, the phone told me it was encrypting again. I'd really like to know why it did that.
[Galaxy S II Original Android Development] Lanchon:
i cannot speak rationally of your unknown build. but i can say it is standard behavior for android to auto encrypt (unless disabled; and you haven't).

your builds might have never auto-enc while they were non-emu, but might have switched behavior afterwards. non-emu is so old nobody at google cares what those builds do.
[Galaxy S II Original Android Development] OllieW:
One question after switching to emulated internal SD with unofficial i9100_emu build: Are there any newer i9100_emu builds or can the (un)official i9100 build be used with emulated internal SD card in future?
[Galaxy S II Original Android Development] the.gangster:
As you can read in post #5, I switched to emulated storage manually. Meanwhile I flashed several newer unofficial builds already. Non of them were emu-variants. Nevertheless emu- storage was preserved without any issues.
[Galaxy S II Original Android Development] Lanchon:
the point of the OP is that you don't need an emu build.

but some things might not work well:

-first boot after wiping data

-multi-user support

-sdcard showing in the storage and format UIs

the important point is that you keep you emu build of TWRP.

on the other hand, if device maintainers want to make two official builds (i9100, i9100_emu) to account for and fix these issues, then so much the better. but these are small details really (except the wipe thing).
[Galaxy S II Original Android Development] rINanDO:
New TWRP-recovery EMU build is up!

The good news is that we now can backup data and media (optional) and even migrate content from non-emulated storage to emulated storage (data/media/0).

Please note this is still WIP so please TEST this at your own risk.

These are the changes:

* Added MIGRATION in the versioninfo

* Added support for backup and restore including /data/media (See Settings->General and scroll/swipe down, Backup -> Options)

* Added support for migrating non-emulated sdcard to /data/media/0 (See Restore -> Migrate non-emulated storage)

Please note that /data/system/storage.xml is removed after restore/migration, so you need to install an lineage-14.1-emu-build afterwards for now.

* Renamed Internal Storage to Emulated storage

REPIT is unfortunately not working with this build (and earlier builds). I still have no clue why it fails.

How to migrate:

* i9100 with non-emulated storage

* flash TWRP-EMU-MIGRATION

* Backup data and non-emulated storage partitions to Micro SD-card

* flash TWRP 3.2.0

* REPIT with data=max+wipe-sdcard=min+wipe

* flash TWRP-EMU-MIGRATION again

* Restore your last backup and enable "Migrate non-emulated storage"

* flash lineage-14.1-emu-unofficial + GAPPS

* i9100 with emulated storage and multi-user support

Downloads

TWRP-3.0.3-EMU-MIGRATION-20170207.zip

lineage-14.1-20170126-UNOFFICIAL-i9100_emu.zip
[Galaxy S II Original Android Development] laptopu:
[MENTION=2222292]rINanDO[/MENTION] that's great thanks and guide seems easy to follow. What are risks of going from your emulated build to a nightly? And any ideas if nightly could become emulated by default?
[Galaxy S II Original Android Development] the.gangster:
Hi rINanDO,

reading your how-to makes me wonder about the one point that I marked:

if run with the non-emu TWRP the repit with the given syntax will maximize the internalSD where I would have expected you would like to have data=max(imized) and sdcard=minimized.

Or did I miss something?
[Galaxy S II Original Android Development] rINanDO:
Ow my bad.... You are right. I will update my post. Thanks!
[Galaxy S II Original Android Development] PhoenixNZ:
Followed your guide with data = Max + wipe SD card minimum and working beautifully as emulated storage.

Thank you.
[Galaxy S II Original Android Development] Lanchon:
amazing!!! :)

why the extra 2 TWRP flashes?

does REPIT fail under TWRP-EMU-MIGRATION?

does it fail with a '${@: 2}: bad substitution' error?

that might be happening because you could be building TWRP with the broken and unfinished toybox (proprietizable license) instead of the mature busybox (GPL license). toybox exists because google doesn't want really free software (like GPL software) in AOSP. it might be better than nothing, yes, but it's not a valid replacement for busybox. when cyanogenmod became cyanogen Inc, they wanted an OS that they could sell, so they got rid of all GPL software (busybox included), even if they had no proper replacements, at the loss of users. (google: cyanogenmod fiasco.)

then everyone stopped using stupid broken CM recovery and switched to TWRP.

but opengapps wanted to keep supporting CM recovery, so they ended up bundling busybox with gapps:

http://opengapps.org/blog/post/2016/01/11/no-fun-to-play-with-toybox/

i could do the same with REPIT, bundle busybox, but i wont do it. i'm not going to work to hide the problems of other tools: recoveries should have a properly working userland, instead of the mockery of one. IMHO anyway.

maybe you can consider building your TWRP with busybox? then please see TWRP's TW_USE_TOOLBOX flag.

EDIT:

i completely missed this comment...

> REPIT is unfortunately not working with this build (and earlier builds). I still have no clue why it fails.

so yes, that's why the extra 2 flashes.

and yes, my suspicion is toybox.
[Galaxy S II Original Android Development] Lanchon:
couldn't the backing up of data be avoided? why does it need to be backed up and then restored? why doesn't a simple data=max suffice?
[Galaxy S II Original Android Development] Lanchon:
please note that TWRP is interested in adding functionality to backup the sdcard (emulated or not). maybe you can push upstream...

"You may want to backup the internal storage as well but this is not possible in the TWRP GUI. You have to do it manually (e.g. with "tar" when your internal storage is mounted in TWRP).

TWRP is open source software. So you're free to add a backup option for the internal storage and push your request to our gerrit for reviewing."

https://twrp.me/faq/backupexclusions.html
[Galaxy S II Original Android Development] rINanDO:
Is probably enough, but didn't test that.

---------- Post added at 12:43 PM ---------- Previous post was at 12:01 PM ----------

Yes, Thanks! Which repository for busybox/toybox do I need? Just the ones on omnirom?
[Galaxy S II Original Android Development] Lanchon:
i never actually built TWRP, but i'd try lineage's:

https://github.com/LineageOS/android_external_busybox

otherwise, maybe you could just steal a binary?

busybox.net has binaries:

https://busybox.net/downloads/binaries/1.26.2-defconfig-multiarch/

but opengapps uses a patched busybox instead. the patch could be important, IDK. i'd grab a patched one if i didn't have more info:

https://github.com/opengapps/openga...bad49761d7807/scripts/busybox-resources/HOWTO

i know some busybox builds fail to extract zips that where whole-zip-signed, the kind of signature used to sign OTAs and recovery packages. this might be related, or not.

anyway. trouble with opengapps' busybox is that they have old binaries (1.25.0), so better go to the source:

https://github.com/yashdsaraf/busybox

with binaries here:

https://www.androidfilehost.com/?w=files&flid=43278
[Galaxy S II Original Android Development] rINanDO:
Thanks for the info! tried the TW_USE_TOOLBOX -switch but when I set this TRUE, REPIT doesn't parse the filename properly.

Without TW_USE_TOOLBOX-switch (using busybox) it is missing getprop and setprop which I fixed here: https://github.com/rINanDO/android_bootable_recovery/commit/abbdf35a5da99648441941ad7f9ef368e35c99c9

REPIT runs, parses succesfully and copy itself to /tmp. When I run it from that location I get the "bad substitution"-error.

I tried several busybox (omnirom/lineageos/yashdsaraf) repositories (including older branches within each) but they all result to the same error. So I looked were the "bad substitution"-error is generated and I found it here: https://github.com/LineageOS/android_external_mksh/blob/cm-14.1/src/eval.c#L385. I tried older branches down to cm-11.0 but still "bad substitution"..... Maybe the syntax ${@:2} is depricated or I need a specific patch to get this working. What does ${@:2} mean?

processParList() {

local first=$1

local rest=${@:2}

A minor thing: I also get alot of "WARNING: linker: Warning: unabel to normalize """-warnings. My guess this is because of empty lines sh is executing. (the warning comes from bionic/linker/linker.cpp)
[Galaxy S II Original Android Development] Lanchon:
if you build TWRP 3.0.2 with official device tree it should work.

> I tried several busybox (omnirom/lineageos/yashdsaraf) repositories (including older branches within each) but they all result to the same error.

i think you are using toybox "sh", which sucks. adb shell to your recovery and investigate. start with "ls -l $(which sh)" and go from there. you could be shipping several boxes (busy, tool, toy) and who knows what gets linked.

$@ is similar but not equal to $*. in this case, i'm not even quoting $@ because i know all params are safe to not quote. ${var_name:2} is an array subset operator: give me what's in var_name would (var_name is an array) except for the first element.

this particular thing would be trivial to fix, but there would be other issues for sure. toybox sh sucks, and it can't handle complex scripts (such as REPIT which is sort of an implementation of GParted in script).

if you were running busybox sh, then that shouldn't happen. (or maybe busybox has compile switches to exclude stuff like arrays, but i've never seen them.)

btw, we KNOW that the busybox binary shipped with official TWRP 3.0.2.x works fine :)

but beware: i think that binary is dynamically linked.

> A minor thing: I also get alot of "WARNING: linker: Warning: unabel to normalize """-warnings. My guess this is because of empty lines sh is executing. (the warning comes from bionic/linker/linker.cpp)

that has nothing to do with shell scripts. it's the linker haveing trouble. your build setup has issues of some sort.

TEST:

if you get statically linked busyboxes (like the binaries i pointed you to) you can just adb shell and replace them on the running twrp instance (its a ramdrive). beware of the proper execute permissions! or you lock yourself out of the system an you need to reboot TWRP.

so you can just replace busybox binary and flash REPIT and see if that is enough
[Galaxy S II Original Android Development] rINanDO:
New unofficial TWRP-(migration) build is up!

How to migrate:

* i9100 with non-emulated storage

* flash TWRP-EMU-MIGRATION

* Backup non-emulated storage partition to Micro SD-card

* REPIT with data=max-sdcard=min+wipe

* Restore your last backup (with non-emulated Storage) and enable "Migrate non-emulated storage"

* i9100 with emulated storage

Changes:

* REPIT finally works!

Thanks to [MENTION=4141005]Lanchon[/MENTION] for his useful info! Toolbox's-sh was used indeed.

And the linker-warning is new in Android N in the android_bionic-repository. (I suppressed the useless warning)

* Updating /data/system/storage.xml instead of deleting it. YOU CAN MIGRATE FROM LOS-OFFICIAL ROM TOO!

* When migrating it removes the depricated /data/media/0/.android_secure folder, so the crappy CMFilemanager can access it too!

Note: It seems when you backup /data -partition and restore it after formatting that TWRP doesn't backup /data/system/storage.xml. It will be recreated by Android at boot.

Download here:

TWRP-3.0.3-EMU-MIGRATION-20170213.zip
[Galaxy S II Original Android Development] laptopu:
That's amazing thanks! I'm about to try this but got a question, what's best way of me moving to official lineage with emu storage? Should I do your guide then wipe data system cache to do clean install? Or do clean install then move to emu storage?

Also is the data wipe issue addressed so emu storage can be maintained after a data wipe? Is it possible to wipe data but leave emu storage intact?

Lastly what's the possibility of this twrp going official? It's amazing work!
[Galaxy S II Original Android Development] Pier_89:
Hi [MENTION=2222292]rINanDO[/MENTION], could we have a short video showing all steps?

So we won't getting wrong something.

Regards.
[Galaxy S II Original Android Development] shkedov:
Hey [MENTION=4141005]Lanchon[/MENTION] could you please update the breaking news of the TWRP-3.0.3-EMU-MIGRATION-20170213 in the OP? Especially when now it supports the official LOS builds, with no need to use the special builds.

Thank you, and thank you so much [MENTION=4141005]Lanchon[/MENTION], [MENTION=2222292]rINanDO[/MENTION] and [MENTION=6560258]the.gangster[/MENTION]! You guys are an amazing trio!
[Galaxy S II Original Android Development] Shambles22:
Hello, with this method there are improvements?
[Galaxy S II Original Android Development] shkedov:
Quoting from OP dude:

"Advantages of emulated storage

Emulated storage has many advantages over non-emulated storage:

- EXT4 is a modern journaling file system that is much safer than FAT.

- Due to increased safety, mount-time full file system check is not necessary in EXT4; so it mounts much faster than FAT.

- FAT is limited to files that are less than 4GB in size.

- Emulated storage can be encrypted.

- Only devices with emulated storage can support multiple users.

"
[Galaxy S II Original Android Development] Shambles22:
Sorry I don't saw it.. Thanks
[Galaxy S II Original Android Development] GrimKriegor:
Gentlemen, good evening and thank you for yet another great tool.

I've tried finding a source for this fork of TWRP, in order to examine and try to understand the changes made to the system in order to get emulated storage working.

Am I missing an obvious link? Is there a way to study them?

Your time is greatly appreciated.
[Galaxy S II Original Android Development] Nobby1960:
My first experience with an emulated storage build. I flashed the special TWRP, but the repit fails, I had many linking errors. Maybe the repit file was not correct. Where can I download the right file? My partition space before was 1GB/6GB. I attach a screenshot from my space after flash. But the ROM works perfectly, I type this posting in it ;)
[Galaxy S II Original Android Development] rINanDO:
The sources are here:

https://github.com/rINanDO/android_bootable_recovery/commits/android-7.1

https://github.com/rINanDO/android_device_samsung_i9100_emu_twrp/commits/cm-14.1

---------- Post added at 08:04 AM ---------- Previous post was at 07:56 AM ----------

You probably used the old link and you don't need the emulated storage build anymore.

Use: TWRP-3.0.3-EMU-MIGRATION-20170213.zip https://www.androidfilehost.com/?fid=745425885120707778
[Galaxy S II Original Android Development] max6698:
Hi [MENTION=2222292]rINanDO[/MENTION],

thanks for the updated TWRP. This one prompts for a password to decrypt /data, hooray. :)

There's a minor bit of weirdness when displaying filesystems from the command line. Filesystems "/dev/block/dm-0" are encrypted. I now have 3 sdcards lol

~ # df
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 401316 160 401156 0% /dev
tmpfs 401316 20 401296 0% /tmp
/dev/block/mmcblk0p7 100784 6528 94256 6% /cache
/dev/block/dm-0 13578568 2178880 11399688 16% /data
/dev/block/dm-0 13578568 2178880 11399688 16% /sdcard
/dev/block/mmcblk0p11 8048 4148 3900 52% /sdcard0
/dev/block/mmcblk1p1 15312852 857392 14455460 6% /sdcard1
[Galaxy S II Original Android Development] todonormal:
If use semiadoptable storage, 4 sdcards?
[Galaxy S II Original Android Development] laptopu:
How do you do a clean ROM install in emulated storage?