yashsingh1/xdaforums huggingface dataset:
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
Hi

I made a backup of my phone using adb, but I didn't know that saving the /sdcard directory would cause corruption issues .. Now I have a 22Gb .ab file and I'd like to extract the /sdcard part of it to restore it. I managed to convert it to a .tar file with abe.jar, but now when I use tar-binary-splitter, there's no /sdcard file, only app data files .. Here's the error I get

lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % java -jar tar-bin-split.jar -split-android backup.tar
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:61)
Caused by: java.io.IOException: Error detected parsing the header
at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry_2_of_2(TarArchiveInputStream.java:285)
at dragomerlin.TarSplitMain.split_android(TarSplitMain.java:234)
at dragomerlin.TarSplitMain.main(TarSplitMain.java:102)
... 5 more
=f' len=8: java.lang.IllegalArgumentException: Invalid byte 73 at offset 0 in 'I=tI
at org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:137)
at org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:168)
at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1007)
at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:996)
at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:326)
at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry_2_of_2(TarArchiveInputStream.java:283)
... 7 more

Is there anything else I can try to get my /sdcard data back ?
[Android Software/Hacking General [Developers Only]] scandiun:
You don't need any additional software to view or extract any tar file, just regular linux commands will show you the contents:

tar -tf backup.tar

The sdcards will be displayed as "shared/0", "shared/1" and so on. What if there is any error just after that command:

echo $?

Which must be 0 if everything is ok, otherwise another number.
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
Here's what it shows when I run "tar -tf backup.tar" and "echo $?"

lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % tar -tf backup.tar
apps/android.overlay.common/_manifest
apps/android.overlay.target/_manifest
apps/android.qvaoverlay.common/_manifest
apps/be.ac.ulb.monulb/_manifest
apps/be.ac.ulb.monulb/a/base.apk
(app data stuff)
tar: Damaged tar archive
tar: Retrying...
tar: Damaged tar archive
tar: Retrying...
tar: Damaged tar archive
tar: Retrying...
tar: Damaged tar archive
tar: Retrying...
(many times)
lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % echo $?
0
[Android Software/Hacking General [Developers Only]] scandiun:
Looks like your backup.tar is incomplete, since doesn't appear to be any shared folder inside.

Did you try android-backup-processor to extract the backup.ab to backup.tar?

Also, it's weird that your "tar" program throws exit code 0 upon error, probably is some older implementation?

Check your version:

% tar --version
tar (GNU tar) 1.34
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
[Android Software/Hacking General [Developers Only]] scandiun:
You can filter also only by shared:

tar -tf archive.tar | grep '^shared/*'
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
Yes I did use android-backup-processor to convert the .ab file to .tar

Here's my tar version

lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % tar --version
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6

backup.ab is a 22.35Gb file, and backup.tar is a 23.29Gb file. When I extract backup.tar, the output is the /app directory only, and it's a 1.4Gb folder, nothing else ..

When I try with the filter :

lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % tar -tf backup.tar | grep '^shared/*'
tar: Damaged tar archive
tar: Retrying...
tar: Damaged tar archive
tar: Retrying...
tar: Damaged tar archive
tar: Retrying...
(a lot more tries)

No output ..
[Android Software/Hacking General [Developers Only]] scandiun:
I see, Apple uses bsdtar instead of gnu tar because of licensing, but you can install it from homebrew: https://formulae.brew.sh/formula/gnu-tar

Then check again:

gtar -tf archive.tar
echo $?

It should display error code different from 0.

It's possible that you can use download gtar without installing it, but no idea.

Also even if the file is a lot bigger than the app directory, either the rest is filled with 00 values, or has damaged entries, or they are in a format not recognized. You would have to open archive.tar with a hex editor with support for very large files, and look at the end of last app file then search the next entry if there's something valid.

For example where it says "(app data stuff)" which app is the latest detected as invalid? So we can start searching for offset.
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
delete this reply
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
Indeed, gnu tar gives an error 2 :

lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % gtar -tf backup.tar
apps/android.overlay.common/_manifest
apps/android.overlay.target/_manifest
apps/android.qvaoverlay.common/_manifest
(...)
apps/com.remarkable.mobile/f/ee98185b-b7a4-4f3d-8159-e51105b7fbda/1957a2ce-2455-4cb3-8bdf-4cfb00277c53.rm
gtar: Skipping to next header
gtar: Exiting with failure status due to previous errors
lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % echo $?
2

com.remarkable.mobile is the latest detected app, but I don't know if it's valid or not ?

Is there an hex editor that you recommend so I can open the .tar file ?
[Android Software/Hacking General [Developers Only]] scandiun:
A-You can check how many files and folders are there for such app:

gtar -tvf archive.tar apps/com.remarkable.mobile

or

gtar -tvf archive.tar apps/com.remarkable.mobile/

And check error with $? after any command

It can be that all the files that are for such app are fine but may be more missing or not. If you want to test, you can covert the app to ab file.

You can create a tar with only that app:

java -jar tar-bin-split.jar -split-android-shared archive.tar

Then, even if has error, check if tar for com.remarkable.mobile exits. Maybe "split-android/com.remarkable.mobile.tar"

Then you can convert to ab file and test:

java -jar abp.jar pack-kk split-android/com.remarkable.mobile.tar com.remarkable.mobile.ab

Then restore it to some android device or Android Studio virtual device or virtual machine:

adb restore com.remarkable.mobile.ab

Check that the app is as when you backed it up. If the app has the same notes and data it means that the app is complete inside the original archive.tar. Otherwise maybe that the app does not support adb backup 100% or that some of its files where in the damaged part. You may need to install the app first if the apk is not included which can be seen in the first command A.
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
But I only care about the /sdcard data, I don't need the app data. I don't understand why I'd have to test the com.remarkable.mobile backup
[Android Software/Hacking General [Developers Only]] scandiun:
It would be to check if the corruption started just after the last app or in the middle of the last one. But anyway if you don't want to test it, I would you recommend using bless hex editor:

bless archive.tar

Then search for apps/com.remarkable.mobile as Text, and the next entry if valid should be shared/ insted of apps/
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
I think com.remarkable.mobile is not the last app of the backup. I'll try your method to see

I get an error when I try to use bless :

lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % bless --folder /Volumes/Lucas/backup\ 07\:22 --file backup.tar
Authorization required

The backup isn't encrypted so I don't see what authorization they're talking about

I found another Hex editor called Hex Fiend, but there are a lot of apps/com.remarkable.mobile text entries, I'm trying to find the last one
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
I found the last apps/com.remarkable.mobile entry. Here's what it looks like after it
[Android Software/Hacking General [Developers Only]] scandiun:
Yes that looks like the last entry, unless the things below are data which I'm not sure.

Search for shared/0 after that entry, or for ustar and see if there's any which I doubt. You can just move the slider down or use keys page up/page down and see if there's any recognizable text. If there's none, it's likely that the data is garbage and is irrecoverable.
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
There is ! And it's way after the last com.remarkable.mobile entry
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
I can also see some shared/0/DCIM entries (which is the data I'm mostly looking for)
[Android Software/Hacking General [Developers Only]] scandiun:
I see, so is possible that the data is recoverable, however there must something wrong between the last apps/ entry and the first shared/0 one, because tar fails in the middle and does not allow to continue. Could be possible to see screenshot of such data? You can send me private message if you think there is something not allowed to be viewed publicly but is to know why tar fails.
[Android Software/Hacking General [Developers Only]] LucasDupuis32:
I sent you a PM
[Android Software/Hacking General [Developers Only]] scandiun:
@LucasDupuis32 glad you could extract all your files with the "--ignore-zeros" option (-i) and everything seems to have been recovered. Did you notice anything missing, any picture or video?
[Android Software/Hacking General [Developers Only]] AAhrens:
I thought I did, as the tar command works fine. I think cygwin installed them all, but I will check.
[Android Software/Hacking General [Developers Only]] ttv:
Well,

I have made progress up to step 5 trying to transfer Angry Birds scores to my Jelly Bean phone. Following the tutorial I decided to use the pax option in Cygwin with this command:

cat ab.list | pax -wd > ab.tar

having created the .tar archive and .list file before as described in the tutorial.

But when I use the command pax gives me the following error on every file on the list:

pax: ustar header field is too small for apps/com.rovio.angrybirds/f/settings.lua

(this is just one instance of the error; the archive consists of many more files which all create the same error)

I have Googled the error but can't seem to find the solution.

Any idea?

Maybe I could use the star option, but did not find a version of star that works on my Mint installation yet.

Thanks
[Android Software/Hacking General [Developers Only]] ttv:
Ok, so I managed to solve this one myself. The problem was with the pax I installed in Cygwin. It gave me the error. I used the Mint VM for the file system that keeps the permissions in tact, but I couldn't find a star-package for it. But then I thought: "Could there be a pax-package for Mint?". Surprise: there is! I followed the guide to the end and guess what: it works! My beloved Angry Birds and Angry Birds Seasons scores have been restored.

Thanks for the help everyone.

Now Rovio: solve this please; I guess there are a lot of sad players with new phones out there.
[Android Software/Hacking General [Developers Only]] scandiun:
The problem is that the GID / UID is just too big to be compatibly recorded, so pax blows chunks. Example of error:

https://trac.macports.org/ticket/28965#comment:7

I'm glad that you finally made it work. star can be compiled under cygwin but takes a lot of time, getting the schily bundle.
[Android Software/Hacking General [Developers Only]] ttv:
I see. But the version of pax that I installed with Mint 13 works ok. So it seems to be related to the version that gets installed with Cygwin. Is the link you are referring to about the Cygwin version? Not sure what the subject of Macports-website is. Still a noob in Linux-land, although I've learned a lot by following this guide :)
[Android Software/Hacking General [Developers Only]] scandiun:
I'm not sure but is possible that owner is coded different when you extract the tar files under linux or under windows. You can see all the information with this

ls -l apps/com.rovio.angrybirds/f/settings.lua
[Android Software/Hacking General [Developers Only]] escribblings:
Well firstly, one massive thumbs up, I now can migrate my Angry Birds scores :D

I do however have 2 questions

Is PAX installed as a default option in Cygwin? - only I am writing a tutorial for another site and I want to keep the instructions as simple as possible - and I cannot remember if I left it as "default" or "installed" when I searched for it on the Cygwin Installer.

Is it possible to run all the command from 1 terminal? - again this is for simplicites sake, at the moment undr windows I am running the Java command under CMD and everything else under Cygwin.

Thanks again.
[Android Software/Hacking General [Developers Only]] scandiun:
Pax is not included in Cygwin base install. You can run all commands from Mintty in a single terminal.
[Android Software/Hacking General [Developers Only]] escribblings:
Thank you - so if I am telling people about Cygwin, I need to tell them to search PAX and select "installed". OK.

RE: Mintty - I did a search on it and this page says that

Does this mean that when I double click the Cygwin shortcut, Mintty is the terminal window that opens?

I ask this because when I navigate to the folder where the adb.exe is located, and run the adb backup command I get the response

-bash: adb: command not found
[Android Software/Hacking General [Developers Only]] scandiun:
Yes, you have to intentionally install pax from setup.exe and Mintty in the default shortcut to "Cygwin".
[Android Software/Hacking General [Developers Only]] escribblings:
OK - so how do I get Mintty to run adb?
[Android Software/Hacking General [Developers Only]] scandiun:
Yes you can. adb must be in the windows' path to run it directly or either provide the full path to it.
[Android Software/Hacking General [Developers Only]] escribblings:
Not at home right now so cannot test immediately - do I need to specify the path if I am trying to run it from its own directory?

Because that is what I was trying? Admittedly I tried adb and not ./adb - would that make a difference.
[Android Software/Hacking General [Developers Only]] scandiun:
Yes you need to specify full path in unix format if adb is not in windows PATH. You can convert from windows path with cygpath:

$ cygpath.exe 'C:\adt-bundle-windows-x86-20130219\sdk\platform-tools'
/cygdrive/c/adt-bundle-windows-x86-20130219/sdk/platform-tools

Or you can add the windows path to Windows PATH and restart cygwin. You can run also ./adb if you first navigate to the corresponding directory.

How to Add a Folder to Path in Windows XP
[Android Software/Hacking General [Developers Only]] escribblings:
Thanks. That should sort that. I'll try it when I get home.
[Android Software/Hacking General [Developers Only]] escribblings:
OK.

./adb works.

I haven't tried adding to the Windows path yet, as I really want to minimise the number of different setup steps that are needed when coming to write my tutorial but have decided I need to put a path in there for ease of use after setup.

I tried $ cygpath 'C:\ADT\sdk\platform-tools' with and without a trailing slash, and putting adb.exe on the end too.

Which gives the response /cygdrive/c/ADT/platform-tools - again, with or without the trailing slash and adb.exe

But none of these, with or without restarting Cygwin allows the adb command to work.

Only ./adb in the platform-tools directory works.

I will try adding it to the Windows path, but would really like to get Cygwin to sort it out.

BTW, I am running 32-bit Vista on this machine.

[edit] Just to confirm, the Windows Path method works. But i would still like to get the Cygpath method working.
[Android Software/Hacking General [Developers Only]] scandiun:
Then run

/cygdrive/c/ADT/sdk/platform-tools/adb
[Android Software/Hacking General [Developers Only]] codelover:
You can add the adb program path to Cygwin on every start-up -or- automatically via .bash_profile:

export PATH="$PATH:C:\ADT\sdk\platform-tools"

Alternatively, you can just drop three files (adb.exe, AdbWinApi.dll, AdbWinUsbApi.dll) to Cygwin /bin/ directory.
[Android Software/Hacking General [Developers Only]] scandiun:
That won't work because PATH is generated each time you start cygwin so will overwrite it. You may better add the line to ~/.bashrc

echo 'PATH=`echo $PATH`:/cygdrive/c/ADT/sdk/platform-tools/' >> $HOME/.bash_profile
[Android Software/Hacking General [Developers Only]] escribblings:
Ok - the first bit works until Cygwin is restarted - but it is a start. Thank you.

I wish to avoid moving the files, in case the end user wants to delve deeper into the ADT, I also want to avoid copying the files due to issues with duplication and possible version mismatch in the future.

Did that and got a load of messages that I previously did not get when Cygwin started.

bash: export: 'Files/Common' : not a valid identifier
bash: export: 'Files/Roxio' : not a valid identifier
bash: export: 'Shared/DLLShared:/cygdrive/c/Program' : not a valid identifier
bash: export: 'Files/Common' : not a valid identifier
bash: export: 'Files/Roxio' : not a valid identifier
bash: export: 'Shared/DLLShared:/cygdrive/c/Program' : not a valid identifier
bash: export: 'Files/Common' : not a valid identifier
bash: export: 'Files/Roxio' : not a valid identifier
bash: export: 'Shared/9.0/DLLShared:/cygdrive/c/Windows/System32/WindowsPowerShell/v1.0:cygdrive/c/Program' : not a valid identifier
bash: export: 'Files/QuickTime/QTSystem:/cygdrive/c/Program' : not a valid identifier
bash: export: 'Files/Nmap:cygdrive/c/ADT/sdk/platform-tools' : not a valid identifier

And adb still not working from any directory

I opened .bashrc (using the command notepad .bashrc as I had not installed any editors like nano into Cygwin (now fixed)) and noticed this at the bottom.

export PATH=/usr/local/bin:/usr/bin:/cygdrive/c/Windows/system32:/cygdrive/c/Windows:/cygdrive/c/Windows/System32/Wbem:/cygdrive/c/Program Files/Common Files/Roxio Shared/DLLShared:/cygdrive/c/Program Files/Common Files/Roxio Shared/DLLShared:/cygdrive/c/Program Files/Common Files/Roxio Shared/9.0/DLLShared:/cygdrive/c/Windows/System32/WindowsPowerShell/v1.0:/cygdrive/c/Program Files/QuickTime/QTSystem:/cygdrive/c/Program Files/Nmap:/cygdrive/c/ADT/sdk/platform-tools:/cygdrive/c/ADT/sdk/platform-tools

I deleted the above, and then just added export PATH="$PATH:C:\ADT\sdk\platform-tools" to the end of the .bashrc and saved it.

That now works, and works on every restart.

but how can I use echo to copy that to .bashrc without adding all the extra Windows paths and screwing it up?
[Android Software/Hacking General [Developers Only]] scandiun:
That is because cygwin imports windows paths which usually contain spaces, causing the issue. You can easily workaround it with this (undo last changes to bashrc):

echo 'PATH=`echo $PATH`:/cygdrive/c/ADT/sdk/platform-tools/' >> $HOME/.bash_profile
[Android Software/Hacking General [Developers Only]] escribblings:
Excellent - 1 last question on this part of the topic.

I note the use of different single quotes - I'm sure they all have specific names - can I confirm that there is a significant difference between them, and where so I find the backward ones that surround echo$PATH on my keyboard.

A different question - this time about Java.

For the Abe.jar, do I need to install the full SDK, or is the JRE sufficient?

Hopefully that covers everything.

Thanks again for your help.
[Android Software/Hacking General [Developers Only]] scandiun:
JRE is enough to run the application. JDK contains also the compiler, libraries and documentation, so it's only necessary if you want to modify abe.
[Android Software/Hacking General [Developers Only]] escribblings:
Excellent, so that just leaves my question about the single quotes:
[Android Software/Hacking General [Developers Only]] scandiun:
Each type of quotation has its own purpose. The >> appends at the end of the file.

V=[COLOR="DarkRed"]'some substrings grouped together'[/COLOR] # assignment
X=[COLOR="DarkRed"]"Put $V to make a longer string"[/COLOR] # substitution and then assignment
Y=[COLOR="DarkRed"]`date`[/COLOR] # run command, assign its output
Z=[COLOR="DarkRed"]'Put $V to make a longer string'[/COLOR] # no substition, simple assignment

For more information: UNIX SHELL Quote Tutorial
[Android Software/Hacking General [Developers Only]] escribblings:
OK. So ' and ` are significantly different, so where do I find ` on my keyboard.

I must be going blind.
[Android Software/Hacking General [Developers Only]] scandiun:
It's in the english keyboards, between TAB and Escape.
[Android Software/Hacking General [Developers Only]] escribblings:
Found it - thank you.

Hopefully that is the end of my inane questions.
[Android Software/Hacking General [Developers Only]] escribblings:
after re-reading the above, and having played around with it, I got your first option (which you seem to have since edited) to work.

instead of

echo "export PATH=$PATH:/cygdrive/c/ADT/sdk/platform-tools" >> ~/.bashrc​

I entered

echo 'export PATH=$PATH:/cygdrive/c/ADT/sdk/platform-tools' >> ~/.bashrc​

And everything seems to be working perfectly.

Thanks once again for all your help.
[Android Software/Hacking General [Developers Only]] scandiun:
Updated with ADB Backup Splitter
[Android Software/Hacking General [Developers Only]] escribblings:
As some of you will know from reading above, I added a path line to my Cygwin's ~/.bashrc so that I could run the adb command from any directory without having to recite it's explicit path every time. That code was

echo 'export PATH=$PATH:/cygdrive/c/adt/sdk/platform-tools' >> ~/.bashrc

But then it all fell apart again when I tried to do the same to invoke the abe.jar. After realising I couldn't path to a .jar script like I could an executable.

After a lot of issues and help from my bro-in-law I managed to cobble together a wrapper for it, and dropped it into the above path (although I presume it would work if dropped in /use/local/bin too.

The biggest headache was solved when I found out that for Java to be able to read the path to the .jar, it had to be in Windows format.

So here is the contents of my abe.sh script

#!/bin/bash
java -jar "C:\adt\abe\abe.jar" "$@"

Which enables me to call

abe.sh [action] [file.1] [file.2]

Instead of

java -jar C:\adt\abe\abe.jar [action] [file.1] [file.2]

From any directory.

Hope this is helpful.
[Android Software/Hacking General [Developers Only]] scandiun:
You can also put abe-run.sh in C:\Cygwin\bin

#!/bin/bash
java -jar "C:\adt\abe\abe.jar" $1 $2 $3

And run directly

abe-run.sh unpack file.ab file.tar <password>

Remove .txt from the end.
[Android Software/Hacking General [Developers Only]] escribblings:
Just in case anyone stumbles across this and isn't sure, that script linked above will only work if you have used the same paths that I have - these are not the default installation directories.

as a side note, I decided to use "$@" as opposed to $1 $2 $3 as I am asking my readers to enter this script themselves, and it just saves a little bit of typing. It has the same effect, and in fact if required could extend the arguments to $4 $5 $6 and beyond.