yashsingh1/xdaforums huggingface dataset:
[OnePlus 3 & 3T Cross Device Development] Dirk:
Terminology is getting confused here.

So, Wipe menu, 'Format Data' Button on right, (above Swipe-to-Wipe), removes Encryption.

Wipe menu->Advanced->Data->Format, does not.

Intentional or not it's good to have one method that removes Encryption and another that maintains it across Formatting operations.

This finding explains my own experience yesterday. I used 3.2.3-0 to Format f2fs to ext4 in Wipe->Advanced->Data, without losing Encryption. I wonder how many versions back this facility existed? Did 3.2.1.0 behave this way or is it more recent than that?

[MENTION=4405529]nvertigo67[/MENTION] This explains everything, right?

Using the Advanced Menu retains Encryption after Formatting operations. All of this also explains, as you stated before, why some people are having trouble removing Encryption in some cases. Some people are formatting using the button and others using the Advanced menu.
[OnePlus 3 & 3T Cross Device Development] BillGoss:
I think you are right. I was coming at this from the perspective of the issue with formatting f2fs partitions. So I was using the Advanced Wipe option, not the Format Data button.

So, now I'm off too try the Format Data button.
[OnePlus 3 & 3T Cross Device Development] Sam Nakamura:
Just a thought to avoid confusion, shouldn't the reworked TWRP be named somewhere like 3.2.1-1, if naming remains same as the previous release from 9th of December 2017 there's no way (exept the file size) to distinguish between them, inexperienced users will be confused and they're probably going to report bugs from old release in the new thread...
[OnePlus 3 & 3T Cross Device Development] arpit_nnd:
I am pretty sure this is more related to encryption than F2FS. You are most likely encrypted with passcode requirement for device start disabled. Disable encryption and TWRP will load almost instantly.

---------- Post added at 04:09 PM ---------- Previous post was at 03:47 PM ----------

[MENTION=5091734]dianlujitao[/MENTION] I faced a different issue with encryption on my 3T with Oreo(NOS). I was on EXT4 with encryption & device start passcode requirement enabled.

After updating to your recovery, when I clean flashed and tried to boot a ROM(used OB22 firmware), I got a 'Encryption Unsuccessful' screen. In description it said that although the passcode was correct, data couldn't be decrypted and is corrupt. There was a button to reset device at the bottom. The same screen showed up on every ROM & each time I tried. All this time TWRP was decrypting successfully.

Thankfully, the data on internal storage was still intact and I could back up.

Then, I formatted /data to clear encryption, flashed ROM, set pattern and encrypted again. Later, I clean flashed again to check if issue re-appears and all went normal.

tl;dr

Decryption was unsuccessful on 3T Oreo ROM after clean flashing it with new TWRP, had to clear encryption to boot. All was well after encrypting again.

Unsure if it was caused by ROM or OB22 firmware.
[OnePlus 3 & 3T Cross Device Development] counterfeit187:
Is there a even difference, or is the description just updated?
[OnePlus 3 & 3T Cross Device Development] Sam Nakamura:
The recently released TWRP version is updated according to the OP's description in post #1, so there is a difference to the TWRP 3.2.1 version which was available before today, potential problem I see is that the old build with compatability issues and the new Oreo build are equally named, actually with TWRP3 they introduced the last number after the version number to indicate device specific changes, like it just was the case here with our device...

----------------

Official TWRP version for all official supported devices is currently 3.2.1, the "-1" at the end is the device specific build of this TWRP version, this helps to avoid confusion and points explicitly to relevant changes made compared to version 3.2.1 "-0")
[OnePlus 3 & 3T Cross Device Development] dianlujitao:
As I said in https://xdaforums.com/oneplus-3/one...ecovery-official-twrp-touch-recovery-t3742894, there's no guarantee it works with any pre-Oreo encrypted /data. Use an old TWRP instead.
[OnePlus 3 & 3T Cross Device Development] arpit_nnd:
Naah, it wasn't pre-Oreo encrypted. I had encrypted it from Nitrogen OS(8.1) with OB21 firmware.
[OnePlus 3 & 3T Cross Device Development] dianlujitao:
No idea. I just tested OxygenOS and LineageOS with latest firmware
[OnePlus 3 & 3T Cross Device Development] VincentJoshuaET:
The latest build is fully working with latest OmniROM. Though, I had encrypted using Nitrogen OS before I flashed/switched to OmniROM.
[OnePlus 3 & 3T Cross Device Development] laupuy:
Maybe you need to flash Stock Oreo OOS (or Open Beta) full zip to encrypt correctly your phone on Oreo.

Your last Stock rom was nougat right?
[OnePlus 3 & 3T Cross Device Development] pendragon666:
So, is there gonna be a fix for encryption issues or should i try an older twrp?
[OnePlus 3 & 3T Cross Device Development] Sam Nakamura:
If you going to flash android 8 this TWRP is what you want to use... For nougat use the previously released 3.2.1
[OnePlus 3 & 3T Cross Device Development] pendragon666:
Oh, i didnt realize that this was for oreo. Before 3t it just used to be one twrp for everything :(

Edit: but i am on 3.2.1-0. Twrp page says that's the latest. Which should i flash for nougat sultan los 14.1
[OnePlus 3 & 3T Cross Device Development] helderfp:
you can use the lattest to flash oreo or nougat
[OnePlus 3 & 3T Cross Device Development] Sam Nakamura:
Just a few posts back the OP said this...

https://xdaforums.com/oneplus-3/one...ial-twrp-touch-recovery-t3742894/post75428958

---------- Post added at 01:00 PM ---------- Previous post was at 12:55 PM ----------

Difficult question, the new build of TWRP 3.2.1 is based on latest Oreo sources (read OP), latest official release based on Nougat was the TWRP 3.2.1 build of 9th December '17 which isn't available anymore, so 3.1.1. - 2 remains as "true nougat" build, this however doesn't mean that latest TWRP won't work, you can try but a backup of your internal storage is recommended in case you need to format data when decryption fails...
[OnePlus 3 & 3T Cross Device Development] pendragon666:
Yeap, went back to that version, works like a charm again. Thanks for all the help guys.
[OnePlus 3 & 3T Cross Device Development] :
For some reason I can't flash twrp. fastboot says that it is finished without throwing any errors, however, after rebooting I am still being shown the stock OnePlus recovery. Any ideas? I have tried 3.2.1, 3.1.1-1 and 3.1.1-0..

As far as I can remember, I didn't have this issue before, but I could be wrong. I just updated to the latest OOS beta 31.
[OnePlus 3 & 3T Cross Device Development] BillGoss:
I assume you mean you flashed TWRP in fastboot, not ADB.

Did you boot straight into recovery after flashing or did you reboot the system first?

If you are running stock OB without root (Magisk) or a custom/patched kernel, OOS will restore the stock recovery on system reboot.
[OnePlus 3 & 3T Cross Device Development] :
Fastboot indeed, sorry, fixed.

I booted the system first as that's what fastboot reboot does. So if I understand you correctly, shutting down the phone after flashing TWRP, booting into recovery and flashing Magisk first would prevent this issue?
[OnePlus 3 & 3T Cross Device Development] Sam Nakamura:
If you see the "unlocked bootloader message" after reboot hit the volume rockers, choose recovery - simple & effective...

I someday started to appreciate the "unlocked bootloader warning" just for this convenience :)
[OnePlus 3 & 3T Cross Device Development] BillGoss:
You can also use the volume buttons in fastboot to select recovery instead of startup.
[OnePlus 3 & 3T Cross Device Development] Dirk:
Can't wait to hear your results! :good:
[OnePlus 3 & 3T Cross Device Development] dianlujitao:
true, you've warned "Format Data will ... removes encryption ..." on that page, so it removes encryption. There's no such warning on advanced wipe page, so it maintains encryption.

Note that Format Data always convert fs to ext4, so to remove encryption on f2fs, format data first and convert fs later
[OnePlus 3 & 3T Cross Device Development] :
Old behavior: formating by format data button does not remove encryption, formating by change fs removes the encryption.

Frankly, I don't care which way removes encryption, as long as it's documented.

On the other hand: users are confused/not knowing that there are two different way of formatting, it might be even more confusing, if it will be different from device to device: i.e. on my chagallwifi the format button doesn't clear encryption and the change fs formating is needed to remove encryption and it's vice versa on op3/t twrp...

BTW: thanx to [MENTION=5091734]dianlujitao[/MENTION] for dealing with the broken f2fs formating at all!!! Thanx to [MENTION=2911853]BillGoss[/MENTION] and [MENTION=1763460]Dirk[/MENTION] for testing!!!
[OnePlus 3 & 3T Cross Device Development] Dirk:
I like the way it is now. As long as we know which is which we can disseminate that info as needed.

For people who are currently needing to remove encryption to flash Pie ROMs, and need to encrypt again later, use the Format 'Button' and 3.2.3-0 for now.

(Expecting BG to confirm from his own testing)

As OP says, if you're warned that the operation will remove encryption, you're on the right path! :)

Also mine and nvertigo recommendation...just use ext4!
[OnePlus 3 & 3T Cross Device Development] :
Absolutly! Always wondered...

Just checked on chagallwifi again (twrp-3.2.1-0): format data leaves encryption untouched, changing fs removes the encryption.

The longer I think of this, the more I agree with you: even if quite some devices leave encryption untouched with format data, since the warning states encryption is removed, it should be removed - no matter what we are used to from older builds.
[OnePlus 3 & 3T Cross Device Development] Doc Sniper:
Just for comparison or maybe to adjust the behavior of TWRP.

What exactly happens while using: fastboot format userdata ?

Does this command always format to ext4 or f2fs ?

Or does it use the current format of the flash?

Does it remove the encryption or not?
[OnePlus 3 & 3T Cross Device Development] :
fastboot does what you are telling fastboot to do: https://xdaforums.com/oneplus-3/one...1-20180223-t3754855/post78327860#post78327860

It should clear encryption: https://forums.oneplus.com/threads/how-to-disable-encryption-via-fastboot.460366/
[OnePlus 3 & 3T Cross Device Development] BillGoss:
Well, I finally got to test the Format Data button and, yes, it removes encryption and formats the partition as ext4.

But it took me a while to get there because something very weird happened: when I rebooted back recovery in order to try the Format Data option I couldn't decrypt the data partition. I made sure that I had typed the password correctly, but TWRP kept saying that the password was incorrect.

So I rebooted the system, which prompted me for my password and that worked!

It was only TWRP 3.2.3-1 and the special format fix version that gave me the error. TWRP 3.2.3-0 worked fine.

I eventually fixed things up by formatting /data using the OOS stock recovery and rebooting on stock OOS.

I've attached the TWRP log in case it's of any help.

So, my take away from all of this is:

- 3.2.3-1 will correctly format /data as f2fs (yay!)

- Advanced Wipe will reformat /data to the desired format without losing encryption

- Format Data will reformat /data to ext4 and removes encryption

It's late in the day for me so I won't be testing what a fastboot format userdata does.
[OnePlus 3 & 3T Cross Device Development] Doc Sniper:
Thanks for the information, on the man pages there was no indication that you can specify the FS type.

I think we should either have a switch for the two possible file systems in the format menu or two format options for the two possible FS types.

The encryption (header) should always be removed, which may be a good idea as the encryption or the on-disk-format of the encryption or the footer changes in a newer version of Android.

Therefore, the latest version is used when re-encrypting the partition.
[OnePlus 3 & 3T Cross Device Development] :
I don't even have a man page... ...Our continuing mission: To seek out knowledge of C, to explore strange UNIX commands, and to boldly code where no one has man page 4. [ldp] *lol*

For reference: https://xdaforums.com/showpost.php?p=78334968&postcount=1373
[OnePlus 3 & 3T Cross Device Development] Doc Sniper:
You're right, "fastboot -h" helps.

But my Linux machine was at home and me outside.

So I could only search the web and the page I found was probably outdated.

As you already wrote, my laptop has the same answer as you:
[OnePlus 3 & 3T Cross Device Development] dianlujitao:
You may wanna try the "TWRP_3.2.1-format-fix-3_oneplus3.img" uploaded just now, check the previous post for link

Everything is working as expected from my point of view
[OnePlus 3 & 3T Cross Device Development] Dirk:
Thanks Bill. Stellar work! :good:

Bill is probably catching flies right now, dreaming of Format operations and calling out in his sleep..."what effing password grumblegrumblegrumble.."

:)

It's Linux Jim but not as we know it...
[OnePlus 3 & 3T Cross Device Development] BillGoss:
I was using that version when I got the failed decryption problem (that's the version I meant when I said "the special format fix version"). So, both TWRP_3.2.1-format-fix-1_oneplus3.img and TWRP 3.2.3-1 would not accept my password.

Sadly, I have no idea what caused the issue in the first place.

----

Edit: Oops!

I just realised you had a new fix file *-3. I had used *-1.

I'll try *-3 and report back.
[OnePlus 3 & 3T Cross Device Development] :
Bad news. Thanx for your efforts and your time spended.

Just an idea:

Since most devs strongly recommend ext4, why not revert the encryption footer removal commit and remove the option f2fs from data's fsmchanging menu. For those insisting (for what ever reason) on using f2fs we can provide a script, whch takes the size (just 64 and 128) as argument and creates a f2fs on data with 16384 bytes (or is it blocks?) reserved for the encyption footer.

Or we even drop the script, and tell op3/t users to use fastboot for creating a f2fs on data.

I havn't seen the latest changes from [MENTION=5091734]dianlujitao[/MENTION], but it seems to me, that removing the encryption footer from fstab is causing much more harm then benefit: if I remember correctly we have - at least - also an issue with nandroids (https://xdaforums.com/showpost.php?p=78329371&postcount=1365 not sure if this also relates to the fstab change, but seems probable).
[OnePlus 3 & 3T Cross Device Development] Dirk:
Makes sense to me. Format Data button defaults to ext4 already so f2fs could be removed as an option in Advanced menu. Could save a lot of headaches. :good:
[OnePlus 3 & 3T Cross Device Development] Doc Sniper:
Think of this before you demonize f2fs:
[OnePlus 3 & 3T Cross Device Development] Dirk:
At this point if Google endorses something, that tells me that it's best avoided.

Google - "f2fs, because we got paid to say so because we say so".

That Android Police article is completely bogus. The benchmarks are a comparison of two different phones, rather than the same phone with different file systems. Apples to Oranges.

Here's a better comparison:

https://xdaforums.com/showthread.php?t=2697069

The takeaway from this is that f2fs has better performance on paper, but synthetic benchmarks are not a good indicator of real world performance. If there is any difference real world it's probably negligible to the point of unnoticeable. I think stability and reliability is the priority.
[OnePlus 3 & 3T Cross Device Development] BillGoss:
OK, so I tested with fix-3.

I started with a clean pure stock install of OOS 5.0.7 which gives me an encrypted f2fs data partition with an on-boot password.

Then I installed fix-3 and:

- Did an advanced wipe of /data to ext4 and rebooted. Everything was fine, including the encryption password.

- reflashed fix-3 (a clean install of OOS replaces the recovery)

- Rebooted to recovery and did an advanced wipe of /data to f2fs and rebooted. Everything was fine

- reflashed fix-3

- Restored los 16.0 backup

- Did an advanced wipe of /data to ext4 and rebooted. Everything was fine. Set a new password and required it on boot.

- Restored all my data

- Rebooted to recovery and restored oos 5.0.7 backup, removed lockscreen password, and rebooted. Everything was fine.

So, using fix-3 everything looks good (formatting f2fs, decrypting).

Thanks!

And I'm now done with this. My phone is back to normal, (most of) my data is restored, and I'm on OOS on an encrypted ext4 (changed from original f2fs) data partition.
[OnePlus 3 & 3T Cross Device Development] Dirk:
Well done Bill, nicely done. Looks like we have a winner! Once you've completed setting up you'll be wanting to take a Nandroid at that point? Could you report back on how that goes? (There were issues with a Nandroid Backup in one case with 3.2.3-1 throwing up errors).

Thanks. :good:
[OnePlus 3 & 3T Cross Device Development] Doc Sniper:
This is a completely outdated benchmark from 2014, it's almost 2019 and many parts of the f2fs code have changed in the meanwhile.

Some people like me want to use f2fs because it's optimized for flash memory.

Ext4 is an archaic file system that was built because ext3 was no longer enough, which in turn came from ext2 because this was no longer enough. It may be old and mature, but also outdated for flash memory, because it was designed for spinning disks.

But everyone should have the file system that he deserves ;)

And ext4 has serious bugs too, see here: https://lkml.org/lkml/2018/11/28/152

Also on my computers, Linux machines, I strictly refuse the use of ext4, instead I use BTRFS.

In that case, I would be affected by the said ext4 kernel error ...
[OnePlus 3 & 3T Cross Device Development] :
No one is demonizing f2fs - in deed, I've supplied a workaround for 3.2.1-0 to 3.2.3-0 to put f2fs to data. Which would work perfectly as well if the button in change fs is removed, since the workaround doesn't even use the twrp gui.

I use f2fs, not because it's supperior by any means, but to catch issues first, before delivering a build to users.

I'm testing f2fs since about 3 years on chagallwifi, oneplus one, op3t and on my build host's src and out partition. I found no real life benefit on the mobile devices - benchmarks is an other point: f2fs is faster here (no miracle if you think of samsungs approach to benchmarks). If you compare on this very device (op3t in my case) with a properly trimmed ext4 to f2fs each filled over 75%, ext4 is slightly faster in reading as well as in writing performance. I've seen no difference at all on bat usage.

The picture for the build host is very clear: building los 15.1 with ccache enabled takes about 70 min on ext4, and about 95 minutes with f2fs.

Some companies - like oneplus - have stepped back in their more current devices.

Again: no demonizing - just expirience. If I would have expirienced any real life benefit, I would recommend f2fs on our device.

If the f2fs button in twrp is removed, we have the exact same functionality as we have now; now the button is there but not working...

There are two points, why quite some people think f2fs is faster on our device: if you chang from a 80% used ext4 to a 10% used f2fs it is faster. The other point is some religious placebo; someting lile "f it's developed for emmc devices, it has to be better." If these two points match...

One of the best, non-technical, religious free article on filesystems for mobile devices I've read the last years: https://www.xda-developers.com/f2fs-why-file-systems-matter-interview-stan-dmitriev-tuxera/

My suggestion is not to stop people from using f2fs, in fact I'll help people using it, I'm just saying: better a missing f2fs button then broken backup/restore.
[OnePlus 3 & 3T Cross Device Development] Dirk:
All good points.

Latest test build seems to be on track for having ALL functionality working so everyone can get what they want. Nandroid Backup needs testing but otherwise, great job OP! I assume if it stands up to all the tests it'll be re-released as 3.2.3-1 again?

I'm sure that's true and agree that random number generators synthetic benchmarks would reflect that. In the real world i used f2fs for two years on this phone and have noticed no difference since switching to ext4, (with the exception that i can now load TWRP in half the time).
[OnePlus 3 & 3T Cross Device Development] :
Who is demonizing a fs by bad?

Just claiming it's bad because it's outdated (btw: not true it's activly developed and has many additions for emmc) and only fpr spinning discs, it just a belive. I'm always willing to discuss the technical side of fs implementations, but my time is too valuable to my to deal with bad polemics.

Religious belives are behind all debating.

---------- Post added at 22:43 ---------- Previous post was at 22:40 ----------

Yeah, I was typing when [MENTION=2911853]BillGoss[/MENTION] (thank you, Dude!) has published his findings.

I'm really happy, we doesn't need to come back to my suggestion.
[OnePlus 3 & 3T Cross Device Development] Doc Sniper:
If we assume that f2fs is mature now and has the advantages mentioned by Google and no more relevant errors, why shouldn't we use it as the preferred file system for all phones?

I understand that the (ROM/TWRP) developers will need some more time, but I'm sure f2fs will be the future file system on flash memory.

Just my 2 cents...
[OnePlus 3 & 3T Cross Device Development] :
No question: on the long run we will see better fses (no matter on which hardware basis).

Regarding maturity: what you call outdated and archaic, I would call well tested and proofed to be reliable in millions of mission critical installations. f2fs has some points, but maturity is surely a point for ext4 when comparing the two of them.

Use f2fs, it's a solid fs - but as long as real life expirience doesn't show it's superior, I'm not calling it superior.

In the end I'm really happy about [MENTION=2911853]BillGoss[/MENTION] report, because it makws it easier to switch the fs, and therefor the freedom of choice is much more comfortable for all users. So everybody can test and make an educated decision behind any promisses and believes
[OnePlus 3 & 3T Cross Device Development] BillGoss:
Backup in TWRP worked without a hitch. All good!

Fix-3 is a keeper in my books.
[OnePlus 3 & 3T Cross Device Development] Doc Sniper:
I'm not saying that f2fs is totally superior, I'm just saying that it's more modern in terms of flash memory.

And yes, I'm sure you will not see a big difference in everyday use.

But we buy new TVs, cars, etc , although the old would be enough, but the file system remain the same?

I'm really unsure about the maturity of ext4 after reading about the recent bugs I linked before and so i think like this:

Ext4 is a modern release, but it is an update of decades-old technology. ;);)
[OnePlus 3 & 3T Cross Device Development] Dirk:
It sure looks that way! Thanks again BG. :good:

[MENTION=5091734]dianlujitao[/MENTION]

---------- Post added at 22:43 ---------- Previous post was at 22:33 ----------

Do you want to try the test build? Particularly the Backup as you had errors with that before? I think you'll find it all working with this:

https://androidfilehost.com/?fid=11410963190603866112
[OnePlus 3 & 3T Cross Device Development] dianlujitao:
Thanks a lot again for your feedback, I do understand it's really painful to backup/restore again and again.

The rest is, clean up and wait for TWRP guys to merge my patch, god knows whether they'll gaf