yashsingh1/xdaforums huggingface dataset:
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] nkk71:
Very insightful, and very recursive :eek:

Yes, /data/media/[user number]/ is part of your userdata partition, but it doesn't mean it's part of your /data partition, at least from a certain perspective.

I also would like the possibility a full backup too, instead of having to run *weird* tar backups, and what not, but if you're going to end up backing up /data/media/[user]/ to virtual sdcard, not sure how that's going to work, but yeah, why not, it's going to be a serious mess if not handled properly, and by that I'm saying *removing* the /data/media exclusion is not going to work.

As for the restore, I'm not even going to start on that.

The OP has mentioned you can submit a patchset to gerrit, I would most certainly welcome it. I honestly dislike /data/media/0/0/0 restores that keep happening to me :o
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sorg:
I didn't touch TWRP sources since 2.x version. Last time it required android/cyanogen source pull in order to compile. Since i'm no longer mess with android compilation long time i don't have working environment to compile and test patches. As i've told above, it's pretty easy to add by original developer when you have all environment set and working. Also i never used gerrit. I use github. I will try to find the sources of TWRP 2.x with this option i made if you want.

All modern devices (at least high-end like HTC10) use unified storage where internal sdcard is virtual and part of /data partition. Actually, in recovery there is no special mounts inside /data. So, if /data/media is a part of /data then it will be backed up together with /data if no special check made. If not, then not. So, i don't see a problem here.

These /data/media/0/0/0 happens when TWRP tries to be "smart" while restoring backup. If restore /data as a raw archive then this problem won't happen.

I did backup and restore a whole /data (with /data/media) on SGS3, SGS4, Zenfone 2 and HTC10 and had no problems.

I think it's silly to have backup system but unable to backup user data. And i don't see any sign of change already long time.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] nkk71:
You'll need to submit code changes to gerrit if you ever wanted them merged into official TWRP.

https://twrp.me/faq/datamedia.html

That has nothing to do with TWRP being "smart", that's an Android *issue* to support multiuser environment. Unless I'm mistaken it's about the layout_version file, if not found or old version, then data will be migrated during boot.

If you want to backup a "Data_Image" (ie the entire raw partition), then you can easily unpack the recovery.img, change the fstab, repack it. The backup will be pretty big though.

It's a long time discussion, whether on xda, github, elsewhere, the devs have explained their views. I also don't see any open patchsets regarding the subject, otherwise I would have tested it, because like you, I also like the idea, but here are a few things to consider:

Before restoring a backup, TWRP wipes the information first, then performs the restore.

Now let's take Internal Storage, which is mostly used for pictures, music, downloads, etc.

Should TWRP go ahead and wipe that too before restoring? To quote Dees_Troy:

"... the implications of restoring a backup of media and how restoring that backup may or may not wipe out their most recent precious pictures of their newborn child."

That's a very valid point, don't you think.

Whereas /data (excluding /data/media) contains your settings and downloaded apps+settings, so restoring them to a previous state won't cause serious data loss, consider the much more dynamic nature of your Internal Storage, where you can have quite a few pictures (as an example), taken within a short period of time.

So consider the average user: Goes to recovery, restores an older copy of his Internal Storage... :eek: Where did my new pictures go :confused::eek:

Therefore it was a design decision not to include Internal Storage in the backup.

Don't get me wrong, I love the idea, but the proper implementation of backing up Internal Storage may be a bit more complex than one may think. I may even take a look when I have some time, but off the top of my head:

Don't use compression: pics, music are already compressed, so trying to compress them again is a waste of resources.

Don't wipe existing stuff, instead merge them, but

avoid duplicates

but how, if the file was only moved :eek:

avoid overwriting existing files

unless the existing file is not the same as the one in the backup

in which case rename...uhm, the one from the backup or the one already in Internal Storage :eek:

Backup cache files or not

Do we really want to backup all the files? For example, personally I don't care about my music files since I can sync them again, that would save me quite a few GB from the backup

etc.

Maybe not as easy as I thought :cyclops:

And thanks to MTP, maybe I'll just use a sync manager to keep my Internal Storage synced with my PC, which would probably also be a lot faster than backing up my entire 11GB /data/media folder because I took 3 new pictures ;)
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Zoandroid:
Or.....allow all devices to FULLY access and utilize their External SD Cards, so that the camera and EVERY app can put its files THERE, and then we don't have to worry about what to back up, lost data during an OS re-flash/upgrade, etc. As you may guess, it rankles my curd that device mfgrs moved away from that ability. In so doing, their "control-mongering" creates all these convoluted necessities and complex backup decisions and operations, simply to work AROUND their stupid idea that everything and its brother MUST be internally stored and/or "cloud" synced ( I won't even get started on the whole "cloud" mess ).
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
But this isn't a TWRP/recovery issue...
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sorg:
@nkk71

Making full backup is not a frequent operation, and of course very time consuming.

But time after time, i would like to make a full backup. Also, recently i had to send my HTC10 for repair and for several reasons i decided to wipe it fully before sending. So, in this case full backup/restore would preserve everything like no repair was happen. I don't see much confusion in restore process. If it's full backup then it eventually should remove everything instead of merging. You don't merge system or data partition when restore, right?

For content: i use phones with removable real SD cards (like HTC10), so movies, pictures and music are going to removable SD card. Internal storage is used mostly for games/apps caches/data. Also many apps save settings to this storage. So, your assumption that internal sdcard has only pictures is wrong. At least in my case. And when you backup /data without /data/media you will get a lot of non-working games/apps if because of some problem /data/media was lost.

Well, i see my idea isn't popular, so it's not going to be implemented by original author due to he thinks /data/media is only for pictures and music. Probably he doesn't play much games and don't use some offline GPS/translators and other apps with large data.

May be i will find some time to implement it on 3.x version, although my interests are shifted away from android tinkering. I just want to have TWRP as a Swiss Knife for backup/restore without using ugly ways for additional backups.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Zoandroid:
Well, it applies to deciding what to back up with TWRP. :) Granted, it is just a wish on my part, so we will still have to do things as being discussed above. (Unless of course TWRP devs create some magic workaround for whatever is preventing apps from storing their data on the REAL SD card. :) Like maybe have TWRP automatically build symbolic links to the internal storage data being described above, and put them on the REAL SD card whenever it does a restoration one's backup, so that apps will actually be writing their files to the REAL SD card. ) Never stop dreaming! :)
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
What I did was partially adopt my SD Card, so that I have a 32GB ext4 partition that can be used for app storage, and the rest is set up as portable. On a fresh flash, apps that can store data to SD Card put their data there automatically. For me that's the best of both worlds, and I don't have to worry about backing up that app data since my card doesn't get wiped.

P.S. That ext4 partition should also be usable for MultiROM, once that's updated ;) (right, [MENTION=2605798]nkk71[/MENTION]?)
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Zoandroid:
I hope you do back up the SD cards. I've seen many suddenly fail. :)
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
When mine fail, they just get stuck as read-only - so I still don't lose any data. I just can't add anything new ;). I don't have anything so important on my SD card that I'm worried about losing it anyway :).
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Zoandroid:
That's good. But I've had them simply be no longer recognizable, even from my PC. I got tired of losing important data from them, so I back mine up to my PC every week. Overkill, but I never lose more than a week's data now. :)
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] cajunflavoredbob:
This thread is always good for a chuckle after work.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] ali nz:
How did you do that man?
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
I used Google ;).

EDIT: Search Google for "partition adoptable storage android" and the first 4 links should give you everything you need to repartition your card :).
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] shields360:
Does flashing an RUU send everything back to stock including recovery ?
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] mrlakadaddy:
Yes.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] shields360:
Thanks ! Assumed so but wanted to double check
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] jsaxon2:
I tried to use adoptable storage and TWRP 3.1.0-1 doesn't see any files on internal storage. I am on US Unlocked 2.41.617.4 firmware running RessurectionRemix 3/22 build. Is it just me or has anyone else had this problem?

Attached is recovery log.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
If you're using the "mixed" adopted storage setup, you are correct - TWRP currently doesn't properly recognize it. It'll only see the portable storage partition and the "old" internal storage. That's one of the reasons I didn't migrate all of my data to that partition (plus I prefer having the option to use either). I've alerted Dees_Troy to the issue, but I should probably create an issue on Github for it.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
With [MENTION=2605798]nkk71[/MENTION]'s help, I was able to make a build with working mixed adoptable storage. It's still a bit of a hacky workaround right now, though. I don't want to post any "unofficial" version just for this so you'll have to wait until we can get this fixed correctly/officially ;).
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
I'm trying to build twrp for pmec2 but in twrp recovery touch panel doesn't work, is there any solution to fix it?

Here is the device tree

https://github.com/sabpprook/android_device_htc_pmec2
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
You'll need to build the kernel from source and patch it to fix touch. See here: https://github.com/CaptainThrowback...mmit/3e364f07a7e939f0ad35e04b6d05f6419aaf5f64
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
Thank you, but HTC seems not release pmec2's kernel, i cloud not find it :(
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
It's under HTC 10. Just look for the downloads with "c2" in the kernel name.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Bn Waheed:
When i do backup, the result's size is so big. more than 10gb. any idea why?
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
I get it, but cloud i build android 6.0 kernel on omni 7.0? And will it works on android 7.0?

I knew pme's old TWRP not work on android 7.0.

And is there any short guide for building kernel? This is my first try it :)
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Your device is running Nougat? If so, you'll need to patch the kernel binary directly. If your device is running Marshmallow, you can build in the Omni 6.0 tree and build the Marshmallow kernel source and it should work fine. I'm sure if you search around XDA you'll find a guide. There's also a readme file in the source with some instructions. Since you already have Android source downloaded, you should have everything you need.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
After 4 hours researching still not make the working patched kernel, could you please make one for me to test it?

I have learned how to build kernel but there is no nougat kernel source for pmec2.

Actually, I have try to flash twrp into boot partition, the touch works and success to gain root, but this way is not good.

Anyway, thanks for your help!!
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Are you asking me to do something here, or just commenting? If your device is running Nougat, you need to patch the stock kernel. I directly linked instructions for that, so I'm not sure what kind of issue you're having. If your device is on M, then whatever kernel you built with that source should work fine. I'm not sure what it is you tried and you didn't answer that question in the post you quoted.

EDIT: Actually those instructions I linked aren't what you need, since your kernel is probably just gzipped. That means it's even easier to extract - just use gzip.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] roirraW "edor" ehT:
You're backing up the system twice, once as an image.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
Yes, my device is running on Nougat.

I've read some guides about kernel compile and zImage generate, the stock kernel is just gzipped and some stuff combined with, and

i follow the guide to patch "recovery\x00" to "rec0very\x00" , now i got the patched & uncompressed image/binary.

At this step is fine to me but i have a problem after that, there is no similar kernel sources to rebuild and i have no idea how to pack uncompressed image/binary into TWRP image.

PS. I've tried to pack it as zImage but it not work (boot to recovery mode will immediately back to bootloader)

stock kernel (nougat): https://github.com/sabpprook/android_device_htc_pmec2/blob/android-7.1/kernel

Maybe I should wait HTC release nougat's kernel source for pmec2 :eek:
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Because you also need to re-append the dtb to the kernel. Here are my full instructions for patching a gzipped kernel+dtb:

# Commands for patching stock HTC zImage for working touch in TWRP

# Extract dt.img from combined binary
# For kernels with appended dtb, find the first instance of the dtb header in hex editor (D0 0D FE ED)
dd if=<name-of-kernel+dtb> ibs=1 skip=<offset-of-dtb-header> of=dt.img

# Decompress gzip stock Image file (after renaming to Image.gz)
# NOTE: Ignore "trailing garbage ignored" error as that's the dtb ;)
gzip -d Image.gz

# Replace string 'recovery' with 'rec0very' so htc_get_bootmode() doesn't know we're in recovery
# NOTE: This also breaks "Reboot Recovery" option in TWRP
sed -i -E -e 's/recovery\x00/rec0very\x00/g' Image

# Compress uncompressed Image using best gzip compression
gzip -9 < Image > kernel

# Append dtb back to end of newly patched, compressed Image
cat dt.img >> kernel

EDIT: Made a few edits so the instructions are a bit clearer.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] mcwups1:
When I go to TWRP, running Resurrection Remix, it asks me for a pin, but if I reboot, it doesn't, and there's no security set.

Is this normal?

I've already formatted since RUU, three flashes ago, but I sometimes have to reboot recovery from recovery to access options needed.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Doesn't sound normal. Where's the recovery log?
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
After several tries I success to make a working patched kernel on himauhl !! I will try on pmec2

There is only one thing i'm confuse about, the Image.gz packed under ubuntu won't get the same file as stock kernel (just decompress and compress it)

but under windows (GnuWin32), it will generate indentical Image.gz.

Overall, I could not do it without your help, I really appreciate it.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
One more question, is marshmallow's kernel able to do patch like that?

I've tried Desire 10 lifestyle (a56dj) but when it boot to recovery will reboot immediately.

stock kernel: <length:0x3E37 unknown> + <zImage.gz> + <length:0x38 unknown> + <dt image>
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Android version doesn't matter. And I use Ubuntu for all of this work - I never use Windows for anything Android related, as Android is Linux-based and you're just asking for issues if you use Windows. Also, I patched the Desire 10 Lifestyle kernel already in another thread. So you should be able to just search for that (or find it on my Github).

EDIT: Not all kernels come with combined dtb, so when you extract the image, you should look for a separate one, and if not present, look for the dtb header in the kernel binary. On older kernels gzip compression wasn't used - so you have to use the method I initially linked you to with extract-vmlinux.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] sabpprook:
Yes, I have cloned desire 10 lifestyle device tree from your github and made TWRP image, but touch not work, then I checked the kernel that did not patched, I can't find any pattern "rec0very" in the Image.

And a56dj's boot image is combined with dtb, and zImage also combined with dtb but the dtb are not the same, so I'm confused how should I do it.

I do the patch like this but it doesn't work.

1. <length:0x3E37 unknown> + <patched image gzipped> + <length:0x38 unknown> + <dtb extracted from kernel> -> zImage

2. zImage repack with dtb which extracted from boot image and ramdisk to TWRP image
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
You're right; that kernel isn't patched. The kernel isn't combined, as you can see a separate dt.img in my tree (unless you have a different kernel than what I used). But the issue on that device is that I have no way to compress the kernel after decompressing it and patching it. And the kernel is too large to fit in the tiny 16MB recovery partition on that device while uncompressed. So the only option is to fastboot boot the image, which should be a perfectly fine workaround. That isn't a gzipped kernel, so you can't just use gzip on it. It's packed with vmlinux. Not sure why/how the kernel you're using is different than the one I have in my device tree.

And I don't understand how you're saying you're patching it. I don't know what that paste means. Please explain it differently.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] mcwups1:
At what point would I need to grab a log? When I first boot into TWRP asking for a pin, or after a reboot of TWRP when it actually mounts and I can access it normally?
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
When it's asking for the PIN.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] gsw5700:
On latest version of TWRP.

Perfomed a full nandroid - OK. Rebooted.

Hard reset. Full restore - getting the error message "extracttarFork() process ended with ERROR: 255"

Tried flashing 3..0.3 - rebooted to recovery

Tried restoring again, failed with same error

Any ideas..?
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] 2WildFirE:
Pls give the devs in here a log

Gesendet von meinem HTC 10 mit Tapatalk
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Jeddix:
Boot your restored system once before restoring the data partition.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] gsw5700:
So, restore system partition, reboot into recovery, then restore data...?

Thanks
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] eTaurus:
I think he meant reboot to system aka the ROM.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Jeddix:
This. Restore system and boot partition, then fully boot your rom once. After that reboot into recovery and restore the data partition.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] gsw5700:
Thanks, I'll try this next time
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] firedown_3:
help

Hi all

I see a report when backup rom viper v5.8 on twrp 3.1.0.1 :

" backups of data do not include any files in internal storage such as picture or downloads"

this is what ? please help me .

Thanks
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Not sure what you're asking for. The message is self-explanatory.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] GROOVYJOSHCLARK:
It means that Nandroids are not backing up the internal SD (emulated storage - internal SD) so make sure you copy this to external SD. My normal process is to copy EVERYTHING from the internal SD (for things that my apps are not automatically configured to save to the External SD) to the external SD before wiping.

sent from my HTC 10 running AOSP
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] cajunflavoredbob:
Seriously, I freaking love this thread.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] nuts77:
Hi everyone,

I have spent one hour searching for my issue and haven't found a clear solution so I post the question here. Sorry if it has been answered before...

I have successfully flashed a stock 7.0 / 2.41.401.4 RUU, and flashed TWRP.

Now I am trying to flash the google dialer & pixel launcher zips (so they are installed as system apps), but I want to keep the ROM unrooted.

I have formatted data in TWRP, mounted system, flashed the zip files, everything seems to go fine.

Now when my phone boots, stock ROM boots without any change, as if I never flashed the files... Although the flashed apps do appear in the system/app and priv-app folders in TWRP file manager.

Can anyone tell me what I'm missing ?

Thank you guys for your help...
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Recovery log from flash?
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] nuts77:
Wooops of course, sorry.

Here it is
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Do you have links to the zips? Is your device S-OFF or S-ON? I'm not sure the first zip is flashing properly, but you said when you mount system after the flash and navigate to the folders, you see the new APKs there? Or are they there if you don't mount System in the mount menu? After you flash, can you use the TWRP File Manager to navigate to the root of /system and take a screenshot (Power + Volume Down)? I'm curious whether /system is getting mounted properly.
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] nuts77:
Sure thing. I'll post the screenshot in a few minutes (editing this post).

I found the apks in the system folder after I mount system & flashed the zips.

I'm S-Off, locked (I assume S-Off on the HTC 10 acts as unlocked unless I'm mistaken... Read that in a post by Jcase).

The zips are the ones I found in these posts :

https://xdaforums.com/google-nexus-5/themes-apps/pixel-launcher-7-1-3231428-flashable-zip-t3477798

https://xdaforums.com/v20/themes/root-nougat-google-dialer-app-v20-t3526735

I guess both should work fine since they just push the apks to the system partition.

Thanks for your support :)
[HTC 10 ROMs, Kernels, Recoveries, & Other Developm] Captain_Throwback:
Those zips won't work. See post #3.