yashsingh1/xdaforums huggingface dataset:
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
I much appreciate all the work you put into the new build but, it's taking about 2min longer then previous to do a backup of approx the same size. I can give logs if needed

---------- Post added at 02:30 PM ---------- Previous post was at 02:26 PM ----------

I would say it's not needed. Had this on previous phone. Just try one backup without it and see if it restores then we'll know. I can try later today. Let us know if you try it and the results

PS, the modem is a security measures sometimes you can lose your emi etc and have a really hard time getting it all rite again. Although you can take a separate backup of it just in case for the size of it I just leave it in. But the system image is big and if we don't need it I'd say why back it up. Just taking up unnecessary space

---------- Post added at 02:55 PM ---------- Previous post was at 02:30 PM ----------

Ok I said screw it and tried a backup and restore without system image. And........ It worked.. yeah. So that saves a lot of space. Left the modem in as I mentioned earlier. Did it on K2 to internal.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
It will take longer with comression enabled but you save a lot of space, also I bet system image does not compress well if at all. I save about 2GB on my backup's that included System image.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
Had compession enabled on for all backups and k3 takes quite a bit longer to internal anyways. Did successfully backup and restore without system image on K2 as well to internal
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
Yes System image was not even an option on any of my previous Samsung phones but I'm new to oneplus5. It seems like backing up System image when you have backed up system is like using a belt and suspenders. I guess Oneplus just creates a system image because they can:D

Found this explanation In Nexus thread:

If you're talking about the backup options in TWRP, it's nothing to do with M.. TWRP added the option to make a dump of the stock rom into a system image instead of a back up image (for fastboot flashing) The option is there regardless of what rom you are on.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
Ok I was going to test a backup without system image but now my backup fails??? REBOOTING FROM TWRP BACK TO SYSTEM NO PROBLEM

Recovery Log in 2017-08-18-21-26-08(thats the only thing in that backup file}

Starting TWRP 3.1.1-k3-c0c5c3a1-dirty on Mon Feb 23 02:51:33 1970

(pid 561)

BOARD_HAS_NO_REAL_SDCARD := true

RECOVERY_SDCARD_ON_DATA := true

TW_NO_USB_STORAGE := true

TW_INCLUDE_CRYPTO := true

I:TW_BRIGHTNESS_PATH := /sys/class/leds/lcd-backlight/brightness

I:Found brightness file at '/sys/class/leds/lcd-backlight/brightness'

I:TWFunc::Set_Brightness: Setting brightness control to 255

I:LANG: en

Starting the UI...

setting DRM_FORMAT_RGBX8888 and GGL_PIXEL_FORMAT_RGBX_8888

cannot find/open a drm device: No such file or directory

fb0 reports (possibly inaccurate):

vi.bits_per_pixel = 32

vi.red.offset = 0 .length = 8

vi.green.offset = 8 .length = 8

vi.blue.offset = 16 .length = 8

setting GGL_PIXEL_FORMAT_RGBA_8888

double buffered

framebuffer: 0 (1080 x 1920)

Using fbdev graphics.

TW_SCREEN_BLANK_ON_BOOT := true

I:TWFunc::Set_Brightness: Setting brightness control to 255

I:Loading package: splash (/twres/splash.xml)

I:Load XML directly

I:PageManager::LoadFileToBuffer loading filename: '/twres/splash.xml' directly

I:Checking resolution...

I:Loading resources...

I:Loading variables...

I:Loading mouse cursor...

I:Loading pages...

I:Loading page splash

I:Switching packages (splash)

I:Blacklisting input device: hbtp_vm

=> Linking mtab

=> Processing recovery.fstab

I:Processing '/boot'

I:Processing '/cache'

I:Processing '/recovery'

I:Processing '/system'

I:Processing '/system_image'

I:Processing '/data'

I:Processing '/firmware'

I:Created '/firmware' folder.

I:Processing '/misc'

I:Processing '/modem'

I:Processing '/bluetooth'

I:Processing '/dsp'

I:Processing '/efs1'

I:Processing '/efs2'

I:Processing '/efsc'

I:Processing '/efsg'
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
Restoring an existing backup(Just Restored 8.2 GB of xXx) still works fine, no technical enough to figure out the back up error.

Fixed the backup issue by mounting system, however ran a backup and tried to restore it but it said system read only cant restore, then I hit reboot system and it came up with setup from scratch, managed to reboot to recovery(TWRP) and successfully restored xXx again. Not sure whats going on:D

Reformatted system EXT4 and restored system only now back in full business again:laugh:
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
OK Final Update Sorry for clogging up the thread. (My weird problems went away after reformatting system EXT4 and restoring system from an earlier backup)

Here is what I found.

You can back up and restore different ROM's over each other or over themselves. (I did a Factory wipe between Full restores)

You can restore a partition of your choise(I formatted system and then restored only system from backup rebooted and all was fine)

You don't need to backup System image(saves 3+GB) I backed up boot/system/data/modem - Factory wipe - Restore - Booted System - All OK:cool::cool:

Great stuff everything works as TWRP should.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] sjamie:
Personally, I appreciate your questions and experiences. I find them helpful.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] hachamacha:
I just flashed k3 and gave it a quick try (since I've not got time for more at the moment):

- There's no such thing as a fair time comparison unless you're using a known test dataset so some of the comparisons are not useful. All parameters must be alike as well.

- I only tested against my last backup which was a week old, and it took about 1-2 minutes longer, but for (a number of) reasons , the latest backup was 2 GB larger. Both were compressed.

- I'll look into any reason I can find why my latest backup was so much larger, but over time, even things like camera, mp3's and Titanium Backup will create a lot of new non-compressible data that could account for it so I don't expect to see much.

EDIT: Compared log files from backup from k2 to backup made via k3:

k2: Averaged 25MB/s and entire thing took about 400s.

k3: Averaged 11MB/s and entire thing took about 950s.

That's a pretty huge difference even trying to take into account the things I mentioned ^^ there, so I guess I'm with the others that think that something in regular backup->SD was 'dinged' in this version. I actually didn't mind the time to backup to OTG in k2 since for me, it's rare I do that. [MENTION=3814731]katinatez[/MENTION], feel free to let me know if you want my log files. I've searched them but not found anything very illuminating yet.

I'll test recovery tonight.

Anyway: Thanks much to [MENTION=3814731]katinatez[/MENTION]!
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
Ok so here's the issue I ran into. As I stated you can backup and restore without system image but I couldn't restore backup's I made from earlier with system image just froze at the one plus screen. So I had luckily made a backup of just the system image as well and to get running again I restored the backup without system image then restored the system image backup on top then everything was beuno. Also when running my restored backup without system image I noticed lots of lag. I also couldn't root without it. I uninstalled magisk to root with super su and everything booted fine but super su kept saying no root and I tried many versions. So I think it's needed as it's original intent is to be able to return to stock non rooted system image. A good example would be to take a backup of a freshly installed ROM just system image then if you install mods, etc and something goes eray you can just restore the backup of system image only just wiping caches to start fresh again without losing all your data like you would starting from scratch. I hope that makes sense tried my best.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] tinuz97:
Weird, I get much faster speeds (also K3 /internal)
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] katinatez:
Go wipe/advance wipe/select data/ cache & f2fs, this is just to format back to f2fs. BUt be warned this f2fs needs to be in rom from devs but they are all removing it now. example of fstab for f2fs

https://github.com/MoKee/android_de...mmit/e9809ba2a8987abfe134a4ded6d4aeefdfe2b132

All those that say k2 build is faster, need to uncheck all backup settings, reboot recovery & test speeds again

If your flashing k3 for the first time uncheck your backup settings then flash k3

That's what I'm getting on internal

Edit: Sorry late reply, been driving all day. Just got home.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
I understood that you wanted Enable compression ticked? Is that not the case?
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] katinatez:
You have to un check all settings when jumping to different twrp. Only settings I checked is compression and ignore digest after I flashed k3
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
I cleared by backup settings reflashed K3 image set options like you suggested. I using xXx 4.9 Rom and I did two backups one with system Image and one without.

With System image 8282MB Backed UP on Disk 5991 MB

Without System image 5210MB Backed UP on Disk 2919 MB

So Compression is fine but data rates are confusing and it definitely takes a longer time than without compression enable, as it should.

So for saving storage use compression for speed turn it off.

Please see attached screen shots.

Thank you for a working TWRP:good:

OK so I did a test without Compression

Without system image Compression enabled Backup time 352 seconds

Without system image Compression disabled Backup time 85 seconds

You only get negative numbers when you backup system image, I'm not sure why.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
So end result I will turn off comression waste a bit of storage and gain a lot of speed:D:cool:

Im just running backups with digest enabled, no compression
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] katinatez:
So your opinion on backup speed has increased significantly over previous k2 ,correct for you?
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
I never had k2 working because of my fork data error but with compression disabled this thing is humming away look at my screenshot above no compression no system image.

file System 64 MB/Sec

Average rate 63 MB/Sec

I also did the same with Digest enabled and no compression and I got

file System 51 MB/Sec

Average rate 63 MB/Sec

---------- Post added at 09:31 AM ---------- Previous post was at 09:16 AM ----------

What is digest, I assume it's some sort of file check so I leave it on....
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] kimyeungg:
I just tested both k2&k3's backup in internal storage.

k2 version

No compression: file systems: 73 MB/sec & imaged drives: 65 MB/sec ----- 13544 MB ----- 188 seconds

With compression: file systems: 26 MB/sec & imaged drives: 98 MB/sec ----- 8363 MB ----- 515 seconds

k3 version

No compression: file systems: 43 MB/sec & imaged drives: 65 MB/sec ----- 13544 MB ----- 312 seconds

With compression: file systems: 14 MB/sec & imaged drives: 49 MB/sec ----- 8363 MB ----- 944 seconds

I do restore default setting in TWRP and un check all setting in "backup" after flashing both k2 & k3.

k3 actually takes me longer time to backup in both "compression enabled & without compression enabled" with same system/data/boot files (i didnt backup system image)

btw, thanks for your work :good: maybe i will just stick with k2 version in the moment
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] katinatez:
Uncheck settings in backup section of TWRP before flashing k3 NOT after

Or uncheck settings in k3 & reboot back into recovery, your settings in k2 are sticking.

My point is there is no commits to enhance backup speeds in k2 recovery.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] kimyeungg:
i actually did

here is my step: flash k2.img on k2 > restore default + un check > reboot to recovery to let settings apply > do backup > flash k3.img on k2 > restore default + un check > reboot to recovery to let settings apply > do backup...

:rolleyes:

---------- Post added at 12:33 ---------- Previous post was at 12:28 ----------

you can compare my speed with #1125

mine with k2 (No compression + digest enabled): file systems: 73 MB/sec & imaged drives: 65 MB/sec ----- 13544 MB ----- 188 seconds

#1125 from [MENTION=3297960]thaidroid[/MENTION]

same setting with k3 - No compression + digest enabled

file System 51 MB/Sec

Average rate 63 MB/Sec

storage chipset related? i believe my partition size is bigger than him cuz i got lots of apps installed but still have faster speed during backup
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
Your speed is faster because your Data partition is a lot larger then mine, System is slower to backup so the larger the data is the better the numbers get.

Also if you run the same job twice right after each other you will get different numbers it's not set in stone.

But the big difference is DATA size
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] hachamacha:
Some early morning thoughts about my dawn retry of k3:

I got a little time to play with k3 again about an hour ago. This is what it took for me to get k3 to even begin to do a backup after clearing all the backup tick marks: Note: I'm completely aware that this could be on me, not k3, since I seem to be bucking the trend on k3 here.

Trials:

thoughts:

Anyway: This is a more interesting issue to me than TWRP right now and it appears that gzip uses very little buffers and force flushes the output buffers. Just messing with different ways to run it at the moment. Also comparing to bzip2 and xz and zip but finding gzip (on linux) is quicker.

Seems as if in the era of SD 835, this should be able to be handled so am looking more at the TWRP source and will try these tests (raw compression) on the 835 as well. I recall some of this from having worked on Philz a lot and it was quite a lot quicker after that section was re-written. If I can ever find my source for the compression part, might be interesting to see what was changed.

Bottom line for me & k3 is that k3 is 1/2 the speed, regardless of setup and ticks of k2 to sdcard. It is interesting to me that ticks matter. It implies that either a) offsets to tick #defines have changed (possible) so they're getting wrong values or b) values themselves have been redefined or c) that mount spots have been redefined between k2 and k3. Wondering what is important about unticking boxes between twrp's, and why it doesn't help in my case anyway.

Cheers. (oh yeah: I do have screenies for these but I need to get going for a bit now. Anyway, it's as I said above)
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
That's weird I've never heard of having to uncheck settings before updating TWRP versions. Not saying it's wrong just never heard of it
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] thaidroid:
I hate to be a spoil sport, I tried TWRP 3.1.1-1 Cheesburger no compression

With system image and Digest enabled it took 110 seconds File system ave 82MB/sec, Imaged drives -21MB/sec(wonder why minus probably because an image file included)

Without system image and Digest enabled it took 78 seconds File system ave 77MB/sec, Imaged drives 66MB/sec

Did not test Restore have to go to bed:cool::cool:
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] katinatez:
Now try it over usb-otg
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] hachamacha:
All of this "real life" stuff is really eating into my flashing time ;)..

Anyway: I just got home again, tried just deleting TWRP folder entirely (thinking "could it somehow be those damned ticks?") and re flashed k3. I set up compression and it was still 1/2 the speed of k2.

At this point, I realize I'm happy with k2 and will use it for now. I've tried all the others and k2 (and k3 if it works well for you) are best and most bug-free. Without changing a thing, a tick, etc, I just re-flashed k2 and tested it and it is twice the speed (with compression, to sdcard) of k3. Maybe the k3 speed thing is just a 'me' thing but all this constant flashing is taking more time than I have, so I'm good with k2.

On another note: I tried many compression settings comparing a straight tarball to a gzip'd/bip2'd/xz'd/zipped tarball and I was wrong. It is a heavy load even for my real linux box to do the quickest (gzip) compression. It is about 4 times as long to create a 1 GB random batch of files tarball zipped. I can lesson it by skimping on the compression (tar cvf | gzip -1 {fn}), but amazingly it doesn't help much. I guess it is the tiny buffers that are used regardless of memory available. (or something else).

Thanks.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] katinatez:
When I first set out to make recovery faster, I did most my testing on an exfat USB. I also was getting k2 results being faster, and I thought maybe getting different results since I have been consistently flashing my recovery partition, I think I'm going to wear out that parttition if I keep this up.

I over came these k2 results by unchecking settings while in k3 and reboot back into recovery. Then I started to get the results I was expecting.

As far as commits, I never changed the Omni recovery repo, no default settings we're changed. The commits are exfat in kernel gave me a jump in speed, and switching i/o scheduler and governor, and quadruple the read ahead. Everything is on my GitHub nothing is hidden.

If you enjoy k2, stick with it.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
Can we discuss the system image needed or not needed for a sec. I posted my records earlier and same holds true. Now that I have done a backup without it I can't restore any backups that I have with or without the system image included. I have to flash the backup I made of just the system image on top/ after restoring a backup whether the backup has it included or not. It's like it's not getting restored . And it just started happening after I backed up without it then restored the backup, which worked but was laggy. So I tried restoring another backup I had of many that had the image included but none would boot unless I flashed the backup of just system image after restoring a backup. Can anyone confirm this. Do a backup without if you haven't already then try to restore it. Then try and restore any other backup that you have from before that has everything included in the backup ie, system image. Take logs of your backups and restores. Mine got wiped
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] adrman:
Not sure if it's of any help, but if I remember correctly, when "system image" became an option in TWRP, it was meant to be used to take a backup of an unmodified system prior to making any modifications. At least when I had my N6P, the advice was not to backup the system image, as it was redundant and served no real purpose. I'm not sure if any of this has changed with the 1+5, but thought I'd throw it out there. I'm currently running sans root, so can't test.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
Yes was same on my n6
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] katinatez:
I will try to give you a detailed answer, but first is my question. Is your system image backup done from stock oos? That could be your problem let me explain.

Stock oos has data on f2fs confirmed by [MENTION=6842057]The Flash[/MENTION] here. Also explains why all kernel devs have included all f2fs commits in kernel. I thought it was /data & /cache

https://xdaforums.com/showpost.php?p=73440615&postcount=267

Also official twrp 3.1.1.0 had f2fs on data then they changed to ext4 on official twrp 3.1.1.1

https://github.com/TeamWin/android_...mmit/55828ce985605398f3efb0fcd0f1366d579d8def

Easier to installed this app here to see what file system you have.

https://play.google.com/store/apps/details?id=me.kuder.diskinfo&hl=en

In my case I just restored backup from internal with only boot, system, data and restored just fine. No system image was in backup.

But I also formatted data to ext4 back when official twrp 3.1.1.1 was released. So in my case all my backups which include all partitions including system image restore just fine or backups with only boot,system,data restore fine also.

That's why I added f2fs support to go back to stock oos 4.5.8, and the correct steps are

Go to Wipe -> Advanced Wipe -> Select only Data -> Repair/Format File system -> Change partition type to f2fs

Basically you have to stick to one file system first, then do all your backups, and all your restores will be fine

Or if you have a stock system image of un_modded oos 4.5.8

You would use these steps to go back

Backup /data to PC

Go to Wipe -> Advanced Wipe -> Select only Data -> Repair/Format File system -> Change partition type to f2fs

Restore system image of stock un modded oos 4.5.8

So the answer is no system_image is not needed, just boot,system,data

hope this helped
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] adrman:
I couldn't remember if system image had been added that far back or not. My memory is fading with old age. :)
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
Thanks, see thing is that since I've done a backup and restore without system image I can only restore backups that don't include system image. If I do want to restore a backup that has the image included ie an earlier backup I can I just have to restore my backup I have of system image only as well after I restore the the backup that has it included. That's why I don't understand cause the backup had it included yet I have to flash it on top. I'm running RR and all ext4. So it's like once you are running a backup without it you just don't back it up anymore and you should be fine for future backups
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
All ext4
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] str8str:
Another note i just formatted data to decrypt back when I first got the phone. I'm decrypted.
[OnePlus 5 ROMs, Kernels, Recoveries, & Other Devel] laupuy:
Hi,

are you sure?

I think f2fs was the stock /data for OP3(T) but OnePlus return to ext4 FBE with OP5.

Take a look: https://xdaforums.com/showpost.php?p=73304982&postcount=69 or https://xdaforums.com/showpost.php?p=73441158&postcount=270

and here: https://xdaforums.com/showpost.php?p=73310060&postcount=105 last: https://xdaforums.com/showpost.php?p=73370423&postcount=218

:)