yashsingh1/xdaforums huggingface dataset:
[Xperia Z1 Compact Original Android Development] derf elot:
Hi there,

that's great. To be honest, I've never really though about backing up my sdcard, mainly because I am using it for unimportant stuff anyway (e.g. my music library, which I have backed up elsewhere). I have also never used any link2sd apps myself, as I never even get close to filling up my internal space.

As for this becoming a general TWRP feature: this is really up to the individual device maintainers. The code that needs changing to get this to work is not in TWRP itself (well ultimately it is, but it's the same as for any other data partition), but the in the device-specific stuff.

To make this a feature in my builds I will need some information though. Is the sdext partition always the second one on the SD? Is it always ext4 or could it be some other FS? How about when using other programs/scripts that do the same thing? Do they all behave the same? My main concern here is to make sure that including this stuff doesn't break anything for those not using these scripts, and that what I modified for you can also be generalized for other programs that do the same as link2sd.

Cheers
[Xperia Z1 Compact Original Android Development] CyanoFriend:
Too hard to answer surely correct for my pretty little knowledge,

but i give it a try:

always second one - afaik yes

ext4 or other - depends on android version and version of os, but needs linux-fs in aware of special rights

same behavior - afai(think) yes, same idea (best known -of very few products- link2sd and app2sd)

PS: If it makes a difference, the possibility of backing up the first partition of sdcard with TWRP generally would be a great fortune anyway.
[Xperia Z1 Compact Original Android Development] MSe1969:
Hi derf elot,

1st of all, many thanks for the great work - glad that I could bring now the most recent fixes to my device :)

I've started building CM12.1 for this device myself end of last year, so I could at least, by using the CM gits, get a working ROM with November '16 patch level, after CM stopped their amami builds in October (same story btw when doing repo init for the Lineage git and cm-12.1 branch)

Hope you don't mind answering some questions I have around this:

I came across your git end of January and decided to give it a try by replacing the respective Lineage projects with your repos. The result was booting fine and working, but unstable: had many fc's, so I flashed back my old ROM (see above). I am wondering now, whether I may have taken 'unfinished work in progress', since e.g. the kernel repo was only updated today and you've built it only this weekend or whether my manifest should have replaced further projects (e.g. to point to sonyxperiadev). Now my questions:

- Could you please share your build manifest?

- In case you plan to continue applying AOSP fixes also in future: Would it be possible to put a comment into the readme of one repo (e.g. android_build) to indicate 'work in progress - don't use right now', when you currently work on commits?

Many thanks in advance

M
[Xperia Z1 Compact Original Android Development] MSe1969:
Thanks a lot for the fast answer!

This is really helpful - I need to look deeper into the manifest, I think I may have missed something already in the manifest. And yes, I have built my bad version on January 27th, so that perfectly explains. And yes, the OS first seems to behave normally, but after using a while, it starts to behave weird and fc's happen...

Nevertheless, from now on, I am glad that I will be able to build CM12.1 amami ROMs on current patch level.

Thanks again.
[Xperia Z1 Compact Original Android Development] Micha_Btz:
thanks, works great.
[Xperia Z1 Compact Original Android Development] colarus:
Hello [MENTION=4063475]derf elot[/MENTION],

thank you very much for your Great Rom. I hope you keep up your work for the Z1 Compact.

Had 2 month Jaguar installed, but may Apps I use won't work with it.

Now I very happy with this fully working Rom with the latest security patch level. :good:

Greetings, colarus

PS: Where is your donation Button? ;)
[Xperia Z1 Compact Original Android Development] levone1:
Modified framework jar from 2/4 build to work with Sony camera port, and with signature spoofing for fake gapps, (micro G), if anyone wants...
[Xperia Z1 Compact Original Android Development] levone1:
I also modded services jar for superuser mod, to work with stryder007's Z5 systemui port, and modded my systemui with swapped navbar buttons and a few notification tweaks, (CM navbar/status bar settings won't work anymore if you flash Xperia systemui)...
[Xperia Z1 Compact Original Android Development] derf elot:
I will work on incorporating the February patches this weekend. Unless there is some annoyance like last time, expect a new build on Sunday.

Btw, I already updated the f2fs driver in the kernel to the latest one in the meantime. The driver we had in there was super outdated. And with my latest TWRP build, there seems to be full f2fs functionality (tested on data and cache, will test encrypted data soon).

Cheers
[Xperia Z1 Compact Original Android Development] CyanoFriend:
Great!

And as you are working on f2fs, do you see a chance for ntfs in the future (some TWRP builders seem to did it) ?

And yes, i know i can be annoying :) always looking for the "Eierlegende Wollmilchsau"

https://en.wiktionary.org/wiki/eierlegende_Wollmilchsau
[Xperia Z1 Compact Original Android Development] derf elot:
It can probably be done. But I guess it would only be useful for the external SD? can you point me to a build that has ntfs support?

No worries. Also, I'm aware of that concept (I am German myself ;) ).

BTW, the new twrp build will support the SDcard backup and SDext support.

---------- Post added 10th February 2017 at 12:10 AM ---------- Previous post was 9th February 2017 at 11:59 PM ----------

Hi there. Thank you for the offer, but like I have said before in a different thread (can't seem to find that post myself anymore ;) ) I don't feel comfortable accepting donations. On the one hand, I do this as a bit of fun and also for my own profit (having a new version to run on my phone), on the other hand I would probably feel obliged to work on it even though my own time might not allow it.

But like I also said before, if you really feel like donating something, feel free to give a bit to a good cause/charity on my behalf :)
[Xperia Z1 Compact Original Android Development] CyanoFriend:
Not only for the Sdcard. It would also be useful and much cheaper to backup/restore the sdcard to an external usb-storage.

For example i use an 3TB drive together with the z1c to access large files and it would be very convenient (and cheap) to store the backups there.

Here is a link:

https://twrp.me/devices/oneplusthree.html

Thanks a lot - derf a lot ;)
[Xperia Z1 Compact Original Android Development] levone1:
Scrolling navbar with power, directionals, and launch camera; plus center clock...
[Xperia Z1 Compact Original Android Development] derf elot:
Right, for otg it should be fairly easy to implement. As long as you only use it as backup target and don't plan to format it. :) I'll put that in and send you a version to test.
[Xperia Z1 Compact Original Android Development] derf elot:
Hi there,

just finished making a new build. I have not been able yet to include a fix for one CVE (CVE-2017-0421), because backporting it is quite a pain. At least it's not a critical issue and everything else is fixed. I will work on it again when I have the time :) I also updated the F2FS drivers. You can download it here:

https://www.androidfilehost.com/?fid=457095661767136923

I also made a new TWRP from these sources. Included now is

- ability to backup external SDcard

- ability to mount and backup potential SDext partition (2nd partition as ext4 on external SDcard)

- ability to backup to NTFS or exFAT drives (e.g. via OTG)

- F2FS support

Download: https://www.androidfilehost.com/?fid=529152257862702670

If there's any problems with either of them, please let me know :) I am now using F2FS on cache and data and everything works fine, including encryption and mounting it in TWRP as well.

Cheers
[Xperia Z1 Compact Original Android Development] CyanoFriend:
Installed and tested a little bit. Worked. Thanks!

Installed and tested a little bit. Worked. MANY thanks!

Some remarks:

- Made a backup of the whole system (z1c and sdcard with 2 Partitions) to USB-Disc with ntfs: 110GB in 160min (nearly half of time for creating md5-checksums.

- For backup/restore of large amount of data its strongly adviced to have a fully charged battery/or even better the device attached to inductive charging.

-Tested backing up System and Data only (5.4GB) on a) SDcard (vfat): 799sec

b) USB-Stick (fat32): 1052sec

c) USB-Disc (ntfs): 447sec

- Tested Access to ntfs with TWRP-internal filemanager

- I like the (new) remark during backup "Backups of Daten do not include any files in internal storage such as pictures and downloads"

and last but not least one (even little) point of suggestion:

- in your test-version data-partition was shown as data, first partition of sdcard was shown as sdcard. In your new version these two were both shown as "Data (excl. storage)".

I would suggest (e.g): "data" and "sdcard (first partition)".

Once again, very good job!

Cheers - and despite your work - dont forget your personal environment: tomorrow is valentinse day.

PS:

So (with your big improvements) there seem to be only only two things on my wishlist for TWRP:

- Backups of files in internal storage ("Backups of Daten do not include any files in internal storage such as pictures and downloads")

- Incremental backups (especially for the sdcard)
[Xperia Z1 Compact Original Android Development] derf elot:
Thank you for testing it. Glad it works as it should. I didn't actually test the NTFS stuff, just saw that the exFAT and ntfs-3g fuse driver were now included in the recovery after building finished ;)

Yes, I just saw that myself. There was a commit a few days ago that is resposnsible for both these changes. (https://github.com/omnirom/android_bootable_recovery/commit/01f4e035aa0833252b99dcbda0e08a327ba8bf3d). It should be pretty easy to fix the naming again (and remove the warning message from the SDcard too, it shouldn't be necessary there).

Right, backup of /data/media (aka "internal storage"). There has been a lengthy discussion on the TWRP git about including this feature. (https://github.com/TeamWin/Team-Win-Recovery-Project/issues/276). Basically, not being able to backup internal storage is by design. So instead, there is now a warning message reminding users that the internal stoarge is not part of the backup. Changing it might not be so easy, but I could have a look eventually. As for incremental backups, I think that would require some heavy changes and using something like rsync (or equivalent). Not saying that it's impossible just probably not doable anytime soon.

It's on the TODO list ;). Thanks for the reminder.

Cheers
[Xperia Z1 Compact Original Android Development] MSe1969:
Hi derf elot,

first of all, thanks for your efforts - tried to build including TWRP by adding/replacing all your repos and using branch "cm-12.0-twrp" instead of "cm-12.0" for the device_sony_msm8974-common, device_sony_amami, device_sony_rhine-common repos and replaced android_bootable_recovery with your repo.

All went through creating system.img, userdata.img, boot.img and recovery.img, but failed at packaging the ZIP file:

Package target files: /mnt/storage/Android/lin-12.1/out/target/product/amami/obj/PACKAGING/target_files_intermediates/cm_amami-target_files-aa385a7435.zip
Traceback (most recent call last):
File "./build/tools/releasetools/make_recovery_patch", line 53, in <module>
main(sys.argv[1:])
File "./build/tools/releasetools/make_recovery_patch", line 35, in main
OPTIONS.info_dict = common.LoadInfoDict(input_dir)
File "/mnt/storage/Android/lin-12.1/build/tools/releasetools/common.py", line 184, in LoadInfoDict
d["fstab"] = LoadRecoveryFSTab(read_helper, d["fstab_version"], d["device_type"])
File "/mnt/storage/Android/lin-12.1/build/tools/releasetools/common.py", line 259, in LoadRecoveryFSTab
raise ValueError("malformed recovery.fstab line: \"%s\"" % (line,))
ValueError: malformed recovery.fstab line: "/boot	emmc	/dev/block/platform/msm_sdcc.1/by-name/boot"
build/core/Makefile:1498: recipe for target '/mnt/storage/Android/lin-12.1/out/target/product/amami/obj/PACKAGING/target_files_intermediates/cm_amami-target_files-aa385a7435.zip' failed

Could you please share again your build manifests? Can TWRP and the ROM be built in one run (that was my assumption)? Or do I need to build them separately by having a separate tree for TWRP?

Thanks again, M
[Xperia Z1 Compact Original Android Development] MSe1969:
Hi,

thanks for sharing the manifests. I had a look and the only difference was, that I originally used the following for the recovery portion:

<remove-project name="LineageOS/android_bootable_recovery" />
<project name="derfelot/android_bootable_recovery" path="bootable/recovery" remote="github" revision="android-5.1"/>

Besides using your repo, I also used the 5.1 branch (and not 7.1 as shown by you).

The situation is now as follows:

When I build w/o the twrp stuff and branches, all works fine (it did already before, only the packaging failed, but I could already flash system.img and boot.img via fastboot and all is fine, but the build now also works).

However, when I now replace the recovery repo with the omnirom and 7.1 branch (as shown in your manifest) and do a repo sync --force-sync bootable/recovery and afterwards an ordinary repo sync, I receive the following error, when attempting to build:

make: *** No rule to make target '/mnt/storage/Android/lin-12.1/out/target/product/amami/obj/STATIC_LIBRARIES/libbase_intermediates/export_includes', needed by '/mnt/storage/Android/lin-12.1/out/target/product/amami/obj/EXECUTABLES/applypatch_intermediates/import_includes'. Schluss.
make: *** Warte auf noch nicht beendete Prozesse...

Whether I try 'lunch cm_amami-eng && make -j4' or 'breakfast amami' followed by 'brunch amami' does not make any difference.

Did you maybe change anything in addition locally, when you did the build?

Thanks again and best regards, M
[Xperia Z1 Compact Original Android Development] derf elot:
Mh, I think I'm getting a bit confused about what you are saying. The manifests I posted work fine for either building CM (just repo sync, source build/envsetup.sh and then brunch amami, no breakfast) for the first set, or building only recovery for the second set. I do not modify anything locally.

The TWRP branches are indeed a bit confusing. They don't really mean that you have to use android-5.1 when building in lollipop. The android-5.1 was TWRP 2.8 (if I remember correctly), which was made to work with lollipop. To get the latest version, you need to use the latest branch.

When you say w/o the twrp stuff, does this mean also using the CM recovery? That's very strange, because it should build and pack fine without any changes.

Are you using my cm12.1-twrp branches here or not? They are needed of course to build with TWRP as recovery.

Cheers
[Xperia Z1 Compact Original Android Development] MSe1969:
Hi,

1. Build with your "regular" 'cm-12.1' branches and 'LineageOS/android_bootable_recovery' (i.e. CM recovery) works perfect to build including packaging. That's fine.

2. Using the below listed three repos with branch 'cm-12.1-twrp' (instead of branch 'cm-12.1'):

derfelot/android_device_sony_msm8974-common

derfelot/android_device_sony_amami

derfelot/android_device_sony_rhine-common

AND

replacing the repo LineageOS/android_bootable_recovery with the omnirom one f. twrp by having the below entries in the manifest:

<remove-project name="LineageOS/android_bootable_recovery" />

<project name="omnirom/android_bootable_recovery" path="bootable/recovery" remote="github" revision="android-7.1"/>

leads to the error, which I have posted this morning. Of course, I did 'repo sync' and for the omnirom repo even a --force-sync before attempting another build. I have however not created an entirely new build tree.

What I did, has happened in the following order:

a. See above (twrp branches and replaced CM recovery by Omnirom recovery, but used wrongly the 5.1 branch)

=> This lead to the 1st error with the fstab in my 1st post yesterday around that topic

b. Reverted this (used 'cm-12.1' branches again and removed the omnirom repo and used again the CM recovery) and did, after repo sync and --force-sync, a proper build, which went perfectly well.

c. Tried again to build recovery by doing the steps above ('cm-12.1-twrp' branch and omnirom recovery, as described by you), and got the posted error.

Regards, M
[Xperia Z1 Compact Original Android Development] derf elot:
Do you make clean in between?
[Xperia Z1 Compact Original Android Development] MSe1969:
First, I did not - now I did. Exactly same story, no difference.

Regards, M
[Xperia Z1 Compact Original Android Development] derf elot:
Is this

what you enter to build the recovery with the twrp branches?
[Xperia Z1 Compact Original Android Development] MSe1969:
I have tried 'brunch amami' (expecting it would build the cm.zip as well as the twrp.img), as well as 'lunch cm_amami-eng && make -j4' both times same result. Today, after make clean, I've only tried 'brunch amami' yet.

Regards, M
[Xperia Z1 Compact Original Android Development] derf elot:
Oh right, OK. I thought when you said you tried to "build recovery", you only built recovery. Like I said, doing a full CM build won't work right now using the cm-12.1-twrp branches and TWRP as bootable_recovery. Instead, only build recovery (and what is needed for it, like kernel for example) by doing

source build/envsetup.sh
lunch cm_amami-eng
make -j5 recoveryimage

So build the ROM using the normal branches, then switch to the twrp branches using bootable_recover from omni and only build the recovery.

I will see if I can fix it to do both in one go.
[Xperia Z1 Compact Original Android Development] MSe1969:
Thanks a lot, that makes sense now - and has done the trick!

Build ran successfully and recovery is now flashed.

Regards - M
[Xperia Z1 Compact Original Android Development] MSe1969:
Hi there,

just wanted to let you know, that this version of TWRP shows some "funny" behavior, so I decided to flash again my old TWRP 3.0.0 version, which so far was always working fine.

The issues, which I had observed, included the following:

- Not possible to save default settings (e.g. language, time zone), after next boot into recovery, again default settings

- Sometimes, when entering recovery by pushing volume up button during start up (when LED is purple), screen remains dark (although adb connection via USB cable possible and thus reboot)

- Once, when attempting to do a backup, showed progress was "hanging" before the backup even started (screen however did react, but was not possible to leave or abort the action at all - again, "adb reboot" from PC connected through USB at least brought my device back to work); in that situation, language was set to German, doing backup again with default language setting EN worked ok.

Regards, M
[Xperia Z1 Compact Original Android Development] derf elot:
That's odd. Have you tried my build by any chance? maybe something in yours is flakey. Sounds like TWRP can't access its settings file, normally located in TWRP/.twrps on the internal storage. Got a TWRP log file by any chance?

Just tested mine and it works fine, including changing language, rebooting, making backups etc.
[Xperia Z1 Compact Original Android Development] MSe1969:
Hi,

I have compiled again and flashed again and now all works fine. I think I have simply wrongly flashed the recovery.img:

Instead of "fastboot flash FOTAKernel recovery.img" (which I did now), I have done "fastboot flash recovery recovery.img" before and that was my mistake, so all is fine.

Thanks, M
[Xperia Z1 Compact Original Android Development] frief:
Sorry for the dumb question: On boot the phone (12.1-20170212-UNOFFICIAL-amami) reminds me about "Install Superuser The su binary is out of date".

I'm somewhat lost as to how to update this. Tried two variants of Superuser from f-droid, searched for something like addonsu-arm-signed.zip (download.lineageos.org/extras) for 12.1 hoping to be able to install a trusted zip with TWRP, then tried some root methods of flashtool which didn't work for me (probably a good sign:), shortly looked at SuperUser and phh's SuperUser on Google play with the latter saying "It won't install other needed parts of SU for you" without giving a pointer and the former (ClockworkMod) having been last updated 3 years ago.

Is there a recommended place to look at?
[Xperia Z1 Compact Original Android Development] blue-spiderfire:
Its recommend to use the TWRP recovery, a quite up-to-date one can you find here You can flash this recovery via PC with the ADB command fastboot flash recovery recovery.img. Read on that, if you didn't used the fastboot method before.

Normaly the built in SuperUser is enough but I don't know how to update this. I'm using SuperSu. Flash the zip of latest stable version via recovery and install the app after boot. SuperSu
[Xperia Z1 Compact Original Android Development] levone1:
OP of this thread has built a newer twrp...

https://xdaforums.com/showpost.php?p=71002054&postcount=1626
[Xperia Z1 Compact Original Android Development] hoorah:
I'm using "12.1-20160914" by [MENTION=4063475]derf elot[/MENTION], I bought the phone in October second hand and unlocked the boot-loader and flashed the mod, i'm using the Cyanogen bootloader (CWM?)

I have had nothing but issues, videocall on 'Line' application kills the bat and phone overheats like a hot potato but works fine on Skype .. I've had several battery issues with the phone just dropping 1% a second and overheating till I factory reset it then its like new again. Now the battery drops to 80% and dies, basically get an hours use of it, I wanna revert back to stock rom but keep everything without wiping Internal Storage, how is the best way to do this? And yea I need root.
[Xperia Z1 Compact Original Android Development] levone1:
If you're using CM, you have root. Check Dev options in settings for 'root access' menu. Easiest way to revert to stock is flash prf in recovery. There's a few for download in 'general' forum. However, it probably won't fix battery issue, which is likely due to an old battery. It's well- established that the sign of a battery on it's way out is sudden battery drop. Stock ROM is better on battery in general, but CM ROM does not cause that behavior.
[Xperia Z1 Compact Original Android Development] hoorah:
Just put the Pprf on a sd card and install it via CM Recovery? Which options do I check to clear everything but my internal storage?
[Xperia Z1 Compact Original Android Development] levone1:
CM recovery might not cut it, but you can flash TWRP easily with Rashr app. Download one of the TWRPs here - https://www.androidfilehost.com/?w=files&flid=41716 - and use 'recovery from storage' option in Rashr.

When wiping, choose 'advanced wipe', then check boxes for system, cache, dalvik, and data. Won't touch your internal storage.
[Xperia Z1 Compact Original Android Development] hoorah:
https://xdaforums.com/sony-xperia-z1-compact/general/stock-14-6-1-236-customized-lollipop-5-t3264557

This one is complete stock right?
[Xperia Z1 Compact Original Android Development] hoorah:
I ****ed up the TWRP flash and had to use navifirm/flashtool to fix it....

Whats the difference with the dstriker rom? Still stock or?
[Xperia Z1 Compact Original Android Development] levone1:
Dstriker is stock with mods and improvements. Very good. Totally different than CM.
[Xperia Z1 Compact Original Android Development] derf elot:
Hi,

made a new build for CM12.1 and TWRP.

CM12.1 contains the March security updates.

Download: https://www.androidfilehost.com/?fid=529152257862711712

TWRP is now version 3.1.0. I also removed the ability to backup SDext and SDcard partitions for now as there was some reported problems.

Download: https://www.androidfilehost.com/?fid=529152257862711713

Cheers