[Nook Color Android Development] HamNCheese:
After almost a week of beating my nook into submission, I think it's time to get a public thread started to solve the Market and Gapps issues that are preventing us from having a "full tablet experience" on the Nook Color.

My hope is that some folks from other projects (Nookie, Nooter, Honeycomb, etc) may be able to contribute to this solution, given their experiences in getting these apps up and running in those environments.

I have chosen XDA as the platform for this discussion for several reasons. First of all, I'm not big on all the IRC drama I own a company that deserves my attention more than participating in all of that bullsh*t. Second, I feel that having this discussion in the public forum (where google and other sites may link to this content) is the "right thing." And finally, I don't see any reason to keep this discussion private. I see no reason not to invite the whole XDA community to participate. I have only held this information back thus far because I kept thinking that the ultimate solution was only a couple of hours away. I have since realized that this is a sort of catch-22, and someone else may have already solved this on another device.

NOTE

- If any of my information is correct, please feel free to post a reply and I will amend this post accordingly.

The Problem...

Gapps is closed source. While we can observe quite a bit from the "outside" point of view, there are still some things we can't directly examine (ie. we lack the source code.)

My understanding (after days of looking at this) is somewhere along the path to Market 2.x, Google has changed the authentication token / validation mechanism / certificates by which Google identifies the account(s) used to interact with the market. In 1.x, logging into other Gapps (Gmail, YouTube, etc) is enough to initialize this mechanism in such a way that Market "just works." There are still some issues to overcome here (see below), but 1.x works now (for the most part.)

2.x relies on SetupWizard.apk to run and grab authentication tokens as part of the "initial setup" process on the device. On first boot, SetupWizard.apk is set as your "home" app, and given a higher priority than the default launcher. It locks out certain actions (home button) so that the user must complete the process in order to satisfy SetupWizard and make the device usable. On the Nook Color, SetupWizard glitches out and cannot be completed.

Keep in mind the fact that everything prior to Honeycomb was designed to run on phones. Android handsets are not properly provisioned until the Google account and phone are bound together. If they allowed the user to easily bypass SetupWizard, the phone would be stuck in a partially-working state. This makes for very unhappy customers - no Gmail, no Market, no YouTube, you get the idea...

Gapps architecture and functional requirements

The entire Gapps suite exploits the UID and Permissions mechanisms to allow inter-application access to authentication tokens and account-related data. When Android scans apps in /system/app, it examines the AndroidManifest.xml file inside of each APK, and sets entries in /data/system/packages.xml to establish the UID and permissions required by each application. This is a standard practice in Android security - as it is the only way for an application to notify the Android system of its requirements.

A typical packages.xml entry for Vending.apk looks something like this:

<updated-package name="com.android.vending" codePath="/system/app/Vending.apk" ts="1217592000000" version="1820" userId="10031">
<perms>
<item name="com.google.android.googleapps.permission.GOOGLE_AUTH.sierraqa" />
<item name="com.google.android.googleapps.permission.GOOGLE_AUTH.android" />
<item name="android.permission.DELETE_PACKAGES" />
<item name="android.permission.WRITE_EXTERNAL_STORAGE" />
<item name="com.google.android.googleapps.permission.GOOGLE_AUTH.androidsecure" />
<item name="com.google.android.gtalkservice.permission.SEND_HEARTBEAT" />
<item name="android.permission.ACCESS_DOWNLOAD_MANAGER_ADVANCED" />
<item name="android.permission.GET_ACCOUNTS" />
<item name="android.permission.CLEAR_APP_CACHE" />
<item name="com.google.android.googleapps.permission.GOOGLE_AUTH" />
<item name="android.permission.READ_PHONE_STATE" />
<item name="com.google.android.googleapps.permission.GOOGLE_AUTH.sierrasandbox" />
<item name="android.permission.MANAGE_ACCOUNTS" />
<item name="android.permission.ACCESS_DOWNLOAD_MANAGER" />
<item name="android.permission.GET_PACKAGE_SIZE" />
<item name="android.permission.WAKE_LOCK" />
<item name="android.permission.ACCESS_NETWORK_STATE" />
<item name="com.google.android.gtalkservice.permission.GTALK_SERVICE" />
<item name="android.permission.INSTALL_PACKAGES" />
<item name="android.permission.USE_CREDENTIALS" />
<item name="android.permission.ACCESS_CHECKIN_PROPERTIES" />
<item name="com.google.android.c2dm.permission.RECEIVE" />
<item name="com.google.android.providers.gsf.permission.READ_GSERVICES" />
<item name="android.permission.INTERNET" />
<item name="android.permission.READ_SYNC_SETTINGS" />
<item name="com.google.android.googleapps.permission.GOOGLE_AUTH.sierra" />
<item name="android.permission.CHANGE_NETWORK_STATE" />
</perms>
</updated-package>

NOTE - the userId and permissions must match the other Gapps or Vending.apk fails to work and/or crashes regularly

Getting 1.x to work

There are three major problem with Market 1.x on the Nook Color:

1) Since SetupWizard is never run, the provisioning / authentication mechanisms are never initialized in the normal way. Workarounds exist to initialize account settings through other apps (YouTube, Gmail, etc) Once this is done, Market has enough information about your accounts to establish your identity and allow your to download apps.

2) Force closes in Market 1.x are due to incorrect permissions and/or UID settings in packages.xml. In these cases, logcat reveals that Vending.apk has insufficient permissions to write to common Gapps settings:

E/AndroidRuntime( 1800): java.lang.SecurityException: Permission Denial: writing com.google.android.gsf.settings.GoogleSettingsProv ider uri content://com.google.settings/partner from pid=1800, uid=10027 requires com.google.android.providers.settings.permission.WRITE_GSETTINGS

In this case, either UID 10027 does not match the rest of the Gapps, or the packages.xml entries related to com.google.vending do not contain the usual set of Gapps permissions. fix_permissions.sh scripts (and other things that reinstall apks via "pm") can fix this, but this is a hack at best.

3) The "missing apps" problem stems from two causes: a) The device fingerprint and/or screen resolution defined in Vending.apk/AndroidManafest.xml do not match the device (or a significant number of apps.) b) (

SPECULATION

) Certain bits in com.google.android.gsf.settings are created by logging into other Gapps, but this does not totally satisfy Vending.apk calls to obtain authentication tokens and/or certificates. Wiping Google Services Framework seems to clear this enough so that Vending.apk regenerates whatever it needs to proceed with full market access.

Problems with 2.x

2.x presents an even bigger challenge. When Google changed the APKs around, Vending 2.x became more reliant on SetupWizard. While existing token work-arounds seem to enable rudimentary access to the market, SetupWizard seems to be the only way to populate the required certificates that Vending.apk requires to validate the signature on APKs downloaded from the market.

This is not a systemic issue with Android (pm and adb installs work fine - this eliminates PakageManager and anything B&N may have monkeyed with), nor is it something that B&N has altered to block market installs outside of Vending.apk. In fact, they were quite generous to leave pm in place. :)

This Vending.apk signature validation failure is visislbe in logcat, and very clearly shows what is (not) happening.

I/vending ( 1087): [30] DownloadManagerBroadcastReceiver.installFromUri(): Could not verify APK signature.

D/vending ( 1087): [30] DownloadManagerBroadcastReceiver.installFromUri(): dFCIpo0aSlPNdcGmfdWMKBZ4E8g#1456950

At first I had a really hard time finding any information on this error. Google shows post after post about the Nook Color, and little else. At first, I thought B&N was intercepting intents somehow and enforcing their own signature check. It turns out that they have nothing to do with it, and Vending.apk is to blame.

Decoded 2.x Vending.apk reveals the problem:

smali/com/android/vending/DownloadManagerBroadcastReceiver.smali:

.method protected installFromUri(Landroid/content/Context;Landroid/net/Uri;Landroid/net/Uri;Lcom/android/vending/model/LocalAsset;Ljava/lang/String;ZLjava/lang/String;)V

...

const-string v5, "Calling install uri=%s src=%s asset=%s name=%s last=%B"

...

const-string v5, "Could not verify APK signature."

So in other words.... Vending.apk is unable to validate the signature. Why? Because it has no certificates. You can log in and see apps, but you can't install them via the Market. They download, but the call to installFromUri fails.

So... We have a few options from here....

1) Hack Vending.apk to disable signature checks

2) Hack (or somehow enable) SetupWizard.apk to complete the process of setting up Google Accounts

3) Complete SetupWizard.apk outside of the Nook and install a "template" setup that will enable Vending.apk to function normally.

4) Work around this with some hack (ie. a cron or daemon that watches /cache/download(null) and installs packages via pm)

I welcome any suggestions or additional information on how to get this working. Perhaps someone who has worked on a related Nook Color rom could shed some light on how we can get SetupWizard to do its job. Normally, the whole thing is done on first-boot, and never needs to be revisited. If there's some simple work-around (ie. copy certs to somewhere in /data/data) then by all means share it!

EDIT

Just to avoid frustration and duplicate (failed) attempts to push newer Vending.apk or Gapps packages over what is already there...

Here are some things that DON'T work:

* Installing old versions of Gapps (with working SetupWizard.apk), then re-writing newer versions. Same error with Vending, other apps like Gmail work perfectly - Vending works, but APKs won't pass the signature check.

* Installing Gapps packages from other ROMs (Nookie, Archos, etc) - same problem with signatures

* Installing hacks and Gapps signed with test-keys

* Attempting to defeat signature checks via setprop (setprop vending.disable_ssl_cert_check TRUE) - this appears to be deprecated.

* Decoding and re-packaging Vending.apk to bypass (or unconditionally allow) failed signatures (this may still be possible, as there are many approaches to doing this.)

* All of the above while wiping /data/dalvik-cache, /data/data/com.android.vending, /data/data/com.android.packageinstaller, /data/data/com.android.vending.updater, /data/data/com.android.*, /data/data/com.google.*

* All of the above while wiping cache and app data via NookColorTools

Things I have not tried - if you are technically adept enough to try these, feel free - all are viable options:

* Copying SetupWizard-generated stuff in /data/data from similar Froyo rom (CM7, Nookie, etc)

* Dual Boot (SD Card, whatever) install that runs SetupWizard.apk results to /data/data/

* Intercepting the intent to install the downloaded apk and bypass the normal procedure in Vending.apk

* Further research into how Vending.apk may be altered to skip the signature check

* Getting SetupWizard.apk to behave (and complete the setup process the right way.)

Last but not least....

Perhaps we have this whole thing backwards. It may actually be easier to invert this solution, and move B&N bits into a "normal" FroYo environment. As far as identifying the required parts, there are only a handful of non-Linux bits that make the Nook Color what it is. B&N seems to have added few new APKs, a certificate, and some UI resources, rather than stepping all over stock android ones. The fact that one can run Gapps at all is testament to this. Is anyone looking into this? It may actually be the best of both worlds.

Now I think I'm going to go on an Red Bull and Vodka-fueled bender - thanks so much to all who have donated to *that* cause.

:)