[Desire Android Development] Serrano:
Issue: Apps disappear after reboot when moved to SD

More information. When trying to install an app from Play Store (com.kreatorz.englishidioms.free) which I guess is configured to be installed to SD by default, it fails with the Play Store message dialog "Couldn't install on USB Storage or SD card". The significant part of the log cat is:

12-11 22:41:53.597: E/Vold(120): Error creating imagefile (Read-only file system)
12-11 22:41:53.597: E/Vold(120): ASEC image file creation failed (Read-only file system)
12-11 22:41:53.597: W/Vold(120): Returning OperationFailed - no handler for errno 30
12-11 22:41:53.607: E/PackageHelper(12760): Failed to create secure container smdl2tmp1
12-11 22:41:53.607: E/DefContainer(12760): Failed to create container smdl2tmp1

Which looks like an error when trying to create the loop image file (Loop::createImageFile).

Mounted filesystems (using a2sd in xdata mode):

rootfs / rootfs ro,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
none /acct cgroup rw,relatime,cpuacct 0 0
none /sys/fs/cgroup tmpfs rw,relatime,mode=750,gid=1000 0 0
none /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
tmpfs /mnt/asec tmpfs rw,relatime,mode=755,gid=1000 0 0
tmpfs /mnt/obb tmpfs rw,relatime,mode=755,gid=1000 0 0
none /dev/cpuctl cgroup rw,relatime,cpu 0 0
/dev/block/mtdblock3 /system yaffs2 ro,relatime 0 0
/dev/block/mtdblock5 /data yaffs2 rw,nosuid,nodev,relatime 0 0
/dev/block/mtdblock4 /cache yaffs2 rw,nosuid,nodev,relatime 0 0
/dev/block/mmcblk0p2 /sd-ext ext4 rw,noatime,nodiratime,barrier=1,data=writeback,noauto_da_alloc 0 0
/dev/block/mmcblk0p2 /data ext4 rw,noatime,nodiratime,barrier=1,data=writeback,noauto_da_alloc 0 0
/sys/kernel/debug /sys/kernel/debug debugfs rw,relatime 0 0
/dev/block/dm-0 /mnt/asec/com.quoord.tapatalkHD-1 ext4 ro,dirsync,nosuid,nodev,noatime,barrier=1,data=writeback 0 0
/dev/block/dm-1 /mnt/asec/com.keramidas.TitaniumBackupPro-1 ext4 ro,dirsync,nosuid,nodev,noatime,barrier=1,data=writeback 0 0
/dev/block/vold/179:1 /mnt/media_rw/sdcard0 vfat rw,dirsync,nosuid,nodev,noexec,relatime,uid=1023,gid=1023,fmask=0007,dmask=0007,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0
/dev/fuse /storage/sdcard0 fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0

No partition mounting errors in dmesg, just:

[ 19.241333] EXT4-fs (mmcblk0p2): Ignoring deprecated nobh option
[ 19.256774] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: data=writeback,nobh,noauto_da_alloc
[ 55.866271] EXT4-fs (dm-0): mounted filesystem without journal. Opts: (null)
[ 56.834594] EXT4-fs (dm-0): re-mounted. Opts: (null)
[ 56.851287] EXT4-fs (dm-0): re-mounted. Opts: (null)
[ 56.968261] EXT4-fs (dm-1): mounted filesystem without journal. Opts: (null)
[ 56.985961] EXT4-fs (dm-1): re-mounted. Opts: (null)
[ 57.024932] EXT4-fs (dm-1): re-mounted. Opts: (null)

As I haven't tried to move an existing app to SD, there is no tmpfs mounted on /mnt/secure (as when I reported the apps disappearing, see quoted text above), so /mnt/secure is part of the rootfs, mounted read-only, and Vold fails when trying to create the image file on the root file system.

Curiously, if I try to explicitly bind mount /sdcard/.android_secure:

root@bravo:/ # mount -o bind /sdcard/.android_secure /mnt/secure/asec
mount -o bind /sdcard/.android_secure /mnt/secure/asec
mount: Permission denied

If I try to create the legacy android_secure to mount it... :

root@bravo:/ # mkdir /sdcard/android_secure
mkdir /sdcard/android_secure
mkdir failed for /sdcard/android_secure, Permission denied

However, using an alternate folder name works:

255|root@bravo:/ # mkdir /sdcard/.secure
mkdir /sdcard/.secure
root@bravo:/ # mount -o bind /sdcard/.secure /mnt/secure/asec
mount -o bind /sdcard/.secure /mnt/secure/asec

And after that, the application which couldn't be installed before, is properly installed.

After installing com.kreatorz.englishidioms.free, mount shows a new entry (fuse device) for the bind mount, and another new entry for the application loop image (dm-2 device) :

root@bravo:/ # mount
mount
rootfs / rootfs ro,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
none /acct cgroup rw,relatime,cpuacct 0 0
none /sys/fs/cgroup tmpfs rw,relatime,mode=750,gid=1000 0 0
none /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
tmpfs /mnt/asec tmpfs rw,relatime,mode=755,gid=1000 0 0
tmpfs /mnt/obb tmpfs rw,relatime,mode=755,gid=1000 0 0
none /dev/cpuctl cgroup rw,relatime,cpu 0 0
/dev/block/mtdblock3 /system yaffs2 ro,relatime 0 0
/dev/block/mtdblock5 /data yaffs2 rw,nosuid,nodev,relatime 0 0
/dev/block/mtdblock4 /cache yaffs2 rw,nosuid,nodev,relatime 0 0
/dev/block/mmcblk0p2 /sd-ext ext4 rw,noatime,nodiratime,barrier=1,data=writeback,noauto_da_alloc 0 0
/dev/block/mmcblk0p2 /data ext4 rw,noatime,nodiratime,barrier=1,data=writeback,noauto_da_alloc 0 0
/sys/kernel/debug /sys/kernel/debug debugfs rw,relatime 0 0
/dev/block/dm-0 /mnt/asec/com.quoord.tapatalkHD-1 ext4 ro,dirsync,nosuid,nodev,noatime,barrier=1,data=writeback 0 0
/dev/block/dm-1 /mnt/asec/com.keramidas.TitaniumBackupPro-1 ext4 ro,dirsync,nosuid,nodev,noatime,barrier=1,data=writeback 0 0
/dev/block/vold/179:1 /mnt/media_rw/sdcard0 vfat rw,dirsync,nosuid,nodev,noexec,relatime,uid=1023,gid=1023,fmask=0007,dmask
=0007,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0
/dev/fuse /storage/sdcard0 fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
/dev/fuse /mnt/secure/asec fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
/dev/block/dm-2 /mnt/asec/com.kreatorz.englishidioms.free-1 ext4 ro,dirsync,nosuid,nodev,noatime,barrier=1,data=writeback 0 0

Secure container public mount points contents (worked properly before mounting /sdcard/.secure):

root@bravo:/ # ls -l /mnt/asec
ls -l /mnt/asec
drwxr-xr-x system system 2013-12-02 03:01 com.keramidas.TitaniumBackupPro-1
drwxr-xr-x system system 2013-12-11 23:28 com.kreatorz.englishidioms.free-1
drwxr-xr-x system system 2013-12-09 10:56 com.quoord.tapatalkHD-1

Secure container imagefile directory contents (only available after mounting /sdcard/.secure):

root@bravo:/ # ls -l /mnt/secure/asec
ls -l /mnt/secure/asec
-rwxrwx--- root sdcard_r 3226112 2013-12-11 23:28 com.kreatorz.englishidioms.free-1.asec

So... I have no clue. Any reason for these permission denied errors when trying to access /sdcard/.android_secure in KitKat? I think that in older Android versions, a tmpfs was mounted on /sdcard/.android_secure for preventing apps to directly access the real folder, but this is not the case. Could this be related to changes in Vold, using fuse to manage external storage mounts, SELinux conf, spore alien invasion?

PS: SELinux is disabled... getenforce shows Disabled...

PS2: Sorry. Just seen that access to .android_secure by root is explicitly denied in check_caller_access_to_name (system/core/sdcard/sdcard.c). Thought it might have been preventing /sdcard/.android_secure from being mounted in /mnt/secure. I was probably wrong.