[Nexus 5 General] danarama:
Contents:

Background of Android storage (Pre-Nexus 5)

How the Nexus 5 is different

You lost me at file system

Did I REALLY need to know ALL that?

A bit more info for the Nexus 5

Deleting data

Wow, all this time I've been stressing, is it really that simple?

Phone won't boot, can I get my data back from internal memory?

Display and Digitiser won't work, can I get my data back from internal memory?

Data corruption

Wiped /cache and now data is lost

Preventing data loss

Usable space

Where are CWM backups / Storage menu doesn't show all my data (Available space is less than adding up used space / What is "system data"?)

USB OTG storage

I have duplicate sdcard data

Other info

Nexus 5 Data (loss / recovery / prevention)

This thread is intended to give you an overview of some of the Storage of the Nexus 5 from a Data Loss and recovery perspective. It is not intended to cover USB sticks or mods for other storage. It is solely to cover day-to-day data concerns and give a background to how these things work

Before we get started...

Here are a couple of threads you should get familiar with before posting on XDA.

A message from the moderators - All users please read

QUESTIONS DO NOT GO IN GENERAL - There is a Q&A Forum - Please Use It

Backround of Android storage (Pre-Nexus 5)

Firstly, I think we need to understand how Android worked historically as this will help us to understand how the Nexus 5 works now.

A legacy android device, lets pretend the Nexus 1 on Android 2.2 as this was a standard configuration at the time. It had 2 major partitions (and several minor ones which are irrelevant to this topic). It has the /system partition and the /data partition. These were partitions of the internal NAND flash memory.

/system is where the Android operating system lives. The user could not delete or change anything in here (unless you were rooted). All the software that came with the phone was installed in the /system partition

/data is where all the userdata goes. Whenever you installed an app from the market, it installed to /data/app and it put all it's important data into /data/data. Also any system settings you changed (Wallpaper, ringtones etc) all were stored in /data/data. When you did a factory reset, it wiped /data and everything in it.

Of course, having these 2 partitions was not enough for everyday use. There was no where to store your music, photos, documents etc. /data is just for app data and settings. So this is where /sdcard comes in

/sdcard is the Android mount point for the External SD card in this legacy android device. This means that when you inserted a Micro SD card, Android used /sdcard as it's internal reference to where the card's storage is. The /sdcard was a necessity before you could take photos. Over time, bigger apps started to put other data here. For example, a GPS / Map application would store its apk (application package) in /data/app and store your personal configuration settings in /data/data but it may download and store offline maps somewhere on the /sdcard. In older devices, the internal Memory (NAND Flash) was usually too small to allow much data on it. Many users would have to root to get more storage space or keep uninstalling apps to keep the "low on space" warnings at bay

How the Nexus 5 is different

Well, the Nexus 5 is substantially different. There are of course SOME similarities. For example, the Nexus 5 still has internal NAND Flash Memory. This is often referred to as the eMMC (Embedded MultiMedia Card) - which still contains the /system and /data partitions, used in exactly the same way.

The main difference is /sdcard. The Nexus 5 is designed specifically to not have an SD card. It has a larger NAND Flash internal memory (eMMC) so it can also have an "internal SD card". This is where people start to get confused. The entire internal memory is an eMMC which is essentially an internal SD card, however a partition of that internal memory is /sdcard.

OK, I know - let me explain. /sdcard is a mount point that Android uses to know where to store /sdcard data. But on the Nexus 5, instead of storing it on a required external sd card, it points to an internal memory partition. Now here is the clever bit. The /sdcard actually points to /data/media/0. So you continue to have your /data partition, but within /data you have:

/data/app

/data/data

/data/media/0

The clever thing is that the file system that android uses for /system and /data is a Linux file system called Extended FS. In our case, we use the Ext 4 file system. This is important to understand because these file systems do not work with Windows so an external SD card would usually be Fat 32 file system, or exFAT so we could plug it into our windows computers and read the contents. What Google have had to do is use the FUSE file system to allow /sdcard (or /data/media/0) to be accessed from windows. Clever stuff. But it has it's pro's and cons...

Ext 4 is seen as a safer file system to FAT, or any of the Windows friendly ones. This is because it has something called journalling which keeps data safer from corruption and errors. Ext 4 also preserves permissions on files, so the data is safer from anything that may be malicious on your phone. Using FUSE with MTP and PTP means we have maximum compatibility across computer OS platforms.

You lost me at file system

All electronic systems that have an operating system and store data use a file system. Think of it in it's simplest form. Imagine a school text book. It has lots of chapters about different things. It has a "Table of Contents" in the first few pages, telling you where each chapter in the book is so if you want to know what page chapter 13 is on, you look in the contents and find the page and go straight there - The alternative is looking through each page individually to find the chapter. Not a quick process.

Well data storage works the same. When you put a file on a hard drive, sdcard, USB stick (or whatever) it is written to a specific location. When it is written to this location, the location is added to the file system. So when you put word.doc onto the drive, The file system is informed of the (very complicated) location of the file. When you tell Windows, Android (or whatever) that you want to open word.doc, the operating system consults the File system and goes to retrieve the data from its true and real (yet very complicated) location on the drive.

Did I REALLY need to know ALL that?

You know what? Probably not but it may go some way to help understand the limitations we will cover later on.

A bit more info for the Nexus 5

One thing many of you have probably noticed is that with the Nexus 5, there is no option to mount the /sdcard as USB Mass storage on your computer. You must use MTP or PTP.

PTP - Photo Transfer Protocol. When you connect your Nexus 5 to your computer using PTP, Your computer sees it as a camera. It will show photos on your "camera" and will set about implementing the default camera options (such as suggesting you import your photos) etc. It won't show documents or other media necessarily.

MTP - Media Transfer Protocol. When you connect your Nexus 5 to your computer, it will be seen as a media player. This should allow you full access to all the files on there, including word documents and the such like.

One of the reasons for this is that because /sdcard points to something using the FUSE file system and is not a true partition, it would be difficult to allow it to be used as USB Mass Storage. It may or may not be possible but the biggest advantage of using MTP / PTP is that the computer and the Nexus 5 can both access the internal memory at the same time. With USB Mass Storage (UMS) this is quite awkward and can result in errors.

The draw back of these protocols is that you cannot give your device a Windows "Drive letter"

Deleting data

This is partially why we needed to understand a little about file systems. So I could explain to you how data is handled when it is deleted.

As I explained earlier, when you write a file to memory, a corresponding entry is written to the file system to advise the OS where the data is. Sure, you may think you are writing the file to /sdcard/documents/work directory on the internal memory, but in reality these directories or folders do not actually exist at a memory hardware level. The data is written to a block and the file system is informed where that block is, how big the file is, what directory it should appear in to the OS etc. When a file is written the memory, the OS see's the available space go down and the used space go up. All this information comes from the file system.

When you delete the file, the actual data is NOT deleted. It remains where it is on the memory. The block is not overwritten. When the OS is told by the user to delete the file, the File system entry is deleted. This changes the free/used space as the file system is no longer accounting for the data, however the truth is the data still exists. When the next request to write a file to the memory comes from the OS, the file system will think the block where the old data was is empty and will overwrite it.

It is this difference between the file system and reality that allows data to be recovered by external software. if you do not write any data to the memory, external software can scan the memory for data whilst bypassing the file system all together. Ff course the window is small. You only have a very limited time to recover data before the file system allows the data to be overwritten with a new entry.

This is not just true of a deleted file. Even formatting the memory (which is actually just re-creating a new, blank file system) leaves all the data in tact behind-the-scenes and can all still be recovered until you start writing data to the memory. Cool huh?

Wow, all this time I've been stressing, is it really that simple?

Awwww snap! You got me. No it is not that simple. All this PC software, example: Piriform recuva only works on a computer drive. In windows, imagine this is anything with a Drive Letter. C: drive etc. The only way to get a drive letter on your /sdcard is to use USB Mass Storage mode, which as previously discussed - is not possible on your Nexus 5.

It is not realistically possible to get your data back once deleted from internal memory. Once gone, it's gone forever*

* Please see This post as pointed out by [MENTION=1566190]efrant[/MENTION] as a possible method which was touted as "not for the faint of heart"

* Some people have reported success with DiskDigger

Phone won't boot, can I get my data back from internal memory?

Let's start by saying, it depends why your phone won't boot. If your internal eMMC (NAND FLash memory) has failed, then probably not. However, if you believe this is not the case then you can get your /sdcard data using adb BUT you need a custom recovery to be flashed, which also requires your bootloader to be unlocked - Read THIS

However, if out of curiosity - you do still want to get your data off, using adb , read below:

Pre requisite is having adb "installed" on your windows PC. Download THIS file and follow the instructions in the readme.

1) Boot into recovery, connect usb and go to "mounts and storage". Toggle the "mount data" options to mount these partitions. Tip, when mounted, the option then becomes "unmount data"

2) Open "cmd" in Windows and type the below code, which will copy all your data to a folder called sdcard on your windows desktop

adb pull /data/media/0/ c:\users\rootsu\desktop\sdcard

Also note, this assumes you have windows vista or newer. Also, it assumes your windows username is rootsu.

That's it, simple.

Display and Digitiser won't work, can I get my data back from internal memory?

You can use adb and a custom recovery to pull data from your /sdcard or even app data from /data/data

Pre requisite is having adb "installed" on your windows PC. Download THIS file and follow the instructions in the readme.

1) Boot into recovery, connect usb and go to "mounts and storage". Toggle the "mount data" options to mount these partitions. Tip, when mounted, the option then becomes "unmount data"

2) Open "cmd" in Windows and type the below code, which will copy all your data to a folder called sdcard on your windows desktop

adb pull /data/media/0/ c:\users\rootsu\desktop\sdcard

Other things you may want to pull.....

adb pull /data/data/com.android.providers.telephony/databases/mmssms.db c:\users\rootsu\desktop\sdcard

adb pull /data/data/com.android.providers.contacts/databases/contacts2.db c:\users\rootsu\desktop\sdcard

Also note, this assumes you have windows vista or newer. Also, it assumes your windows username is rootsu.

That's it, simple.

FYI, if you want to use adb pull in Android (Kitkat+) it's different. Instead of /data/media/0, use /sdcard (yes you could use /sdcard in recovery too, but not all recoveries mount everything correctly to /sdcard).

Otherwise, in Android you'd need to mount /data rw....

adb root
restarting adbd as root
adb shell
mount -o rw,remount /data
exit
adb pull <whatever>

Data corruption

When data becomes corrupt, there's really not a lot you can do. The file system knows where the data is already. If it's corrupt, you're stuck. Most common causes of corruption is dirty unmount (phone shuts off when writing) or dodgy flash / wipe. If your data is corrupt, there really isn't much you can do to restore it. This is why backups are important.

Wiped /cache and now data is lost

It's quite common during a bad /cache wipe that data appears to be lost or even cannot connect to the mobile network. The data is still there however, you just can't see it.

If your bootloader is unlocked, you can usually fix this by using fastboot to flash the stock cache image

fastboot flash cache c:\path_to\cache.img

Of course unlocking the bootloader wipes data so if you haven't, you won't want to. All you can do is try to reformat /cache via stock recovery a few times.

Factory images (to get cache.img) can be found here: https://developers.google.com/android/nexus/images

Preventing data loss

Time to wise up guys. It is possible to recover data off your removable media, but internal memory - very unlikely. No apps on your PC or Android will help with deleted data. So you need to backup.

Dropbox - Use dropbox to automatically upload your photos to online storage.

Foldersync - Use FolderSync to upload important sdcard files to your dropbox account, or better yet - got a computer thats always on at home? Set foldersync to schedule a sync over wifi whilst you're asleep.

I personally use Tasker with Foldersync. I have tasks that upload new photos over sftp to my NAS, no matter where I am and tasks that backup /sdcard overnight to my NAS whilst I sleep.

Useable space

32 GB Nexus isn't 32 GB. 16 GB Nexus isn't 16 GB

Here is how it works.

A true GB is 1024 MB, which is 1024 KB, which is 1024 B. This is how Software (including operating systems) report a true GB (Now referred to as GiB).

Hardware manufacturers however, refer to a GB as 1000 MB, which is 1000 KB, which is 1000 B. So there is a disparity here that gets bigger, the higher the number.

So you need to employ mathematics to work out the TRUE capacity of the Nexus 5.

Take your 32 GB reported hardware, and multiply it by 1000 a total of 3 times. Then, take the result and divide it by 1024 a total of 3 times. This is the total true capacity

32 x 1000 MB x 1000 KB x 1000 B = 32,000,000,000 B

32,000,000,000 B / 1024 B / 1024 MKB / 1024 MB = 29.8 GB

So in summary, any SD card, SSD, Device, HDD that reports to have "32 GB" capacity, it is only ever 29.8 GB (or 29.8 GiB) capacity.

So now it starts to add up a bit more. Then obviously there is a recovery partition, cache partition, System partition and more..... which is excluded from the total "storage"

Here are the main partitions (excluding /data which was 12.6GB) for the N5 (taken from a 16GB) - use "df" from terminal

/system 1009.3M

/cache 689.8M

/persist 15.8M

/firmware 64.0M

However, there are technically 29 partitions, they obviously don't add up to much between the others - use "ls -lR /dev" from terminal

lrwxrwxrwx root root 1970-01-15 02:15 DDR -> /dev/block/mmcblk0p24
lrwxrwxrwx root root 1970-01-15 02:15 aboot -> /dev/block/mmcblk0p6
lrwxrwxrwx root root 1970-01-15 02:15 abootb -> /dev/block/mmcblk0p11
lrwxrwxrwx root root 1970-01-15 02:15 boot -> /dev/block/mmcblk0p19
lrwxrwxrwx root root 1970-01-15 02:15 cache -> /dev/block/mmcblk0p27
lrwxrwxrwx root root 1970-01-15 02:15 crypto -> /dev/block/mmcblk0p26
lrwxrwxrwx root root 1970-01-15 02:15 fsc -> /dev/block/mmcblk0p22
lrwxrwxrwx root root 1970-01-15 02:15 fsg -> /dev/block/mmcblk0p21
lrwxrwxrwx root root 1970-01-15 02:15 grow -> /dev/block/mmcblk0p29
lrwxrwxrwx root root 1970-01-15 02:15 imgdata -> /dev/block/mmcblk0p17
lrwxrwxrwx root root 1970-01-15 02:15 laf -> /dev/block/mmcblk0p18
lrwxrwxrwx root root 1970-01-15 02:15 metadata -> /dev/block/mmcblk0p14
lrwxrwxrwx root root 1970-01-15 02:15 misc -> /dev/block/mmcblk0p15
lrwxrwxrwx root root 1970-01-15 02:15 modem -> /dev/block/mmcblk0p1
lrwxrwxrwx root root 1970-01-15 02:15 modemst1 -> /dev/block/mmcblk0p12
lrwxrwxrwx root root 1970-01-15 02:15 modemst2 -> /dev/block/mmcblk0p13
lrwxrwxrwx root root 1970-01-15 02:15 pad -> /dev/block/mmcblk0p7
lrwxrwxrwx root root 1970-01-15 02:15 persist -> /dev/block/mmcblk0p16
lrwxrwxrwx root root 1970-01-15 02:15 recovery -> /dev/block/mmcblk0p20
lrwxrwxrwx root root 1970-01-15 02:15 rpm -> /dev/block/mmcblk0p3
lrwxrwxrwx root root 1970-01-15 02:15 rpmb -> /dev/block/mmcblk0p10
lrwxrwxrwx root root 1970-01-15 02:15 sbl1 -> /dev/block/mmcblk0p2
lrwxrwxrwx root root 1970-01-15 02:15 sbl1b -> /dev/block/mmcblk0p8
lrwxrwxrwx root root 1970-01-15 02:15 sdi -> /dev/block/mmcblk0p5
lrwxrwxrwx root root 1970-01-15 02:15 ssd -> /dev/block/mmcblk0p23
lrwxrwxrwx root root 1970-01-15 02:15 system -> /dev/block/mmcblk0p25
lrwxrwxrwx root root 1970-01-15 02:15 tz -> /dev/block/mmcblk0p4
lrwxrwxrwx root root 1970-01-15 02:15 tzb -> /dev/block/mmcblk0p9
lrwxrwxrwx root root 1970-01-15 02:15 userdata -> /dev/block/mmcblk0p28

USABLE space for the user is:

26.7 GB for the 32 GB

12.55 GB for the 16 GB

Where are CWM backups / Storage menu doesn't show where all my data is

Please note, rooted users with clockworkmod recovery will find that their backups to not go to /data/media/0/clockworkmod as it is supposed to. The reason for this is CWM has not been updated to this location, instead it uses the legacy location of /data/media/clockworkmod. The Storage menu can see that the space is used but it cannot include it in it's breakdown. Please check this location. You'll need to move it to 0 if you want to access it on a computer

/data/media is the OLD location of /sdcard. All recoveries should use /data/media/0 as /sdcard but CWM itself is antiquated. Really you should upgrade to Philz or TWRP recovery.

A handy app is "Disk Usage" on Play. It can help identify exactly where this data is....

USB OTG storage

The Nexus 5 by default does not automatically mount USB Storage. There are a few options for you however.

1) Buy Nexus Media Importer - No-root required (Read description before purchase)

2) Use OTG USB Helper or Stickmount which are both free but require root

3) Use a custom kernel such as Chaos Kernel with automount of USB OTG

I have duplicate sdcard data

I bet you haven't. All these locations all point to the same place. If you're looking in any of these, you're looking at your sdcard data so don't delete it.

/sdcard

/data/media/0

/data/media/legacy

/storage/sdcard0

/storage/emulated/0

/storage/emulated/legacy

Other info

This is a work in progress. Please let me know if you think I need to reword something or add something.