[Eee Pad Transformer General] sbiriguda:
Running the Android Compatibility Test Suite on TF101​

Background: what CTS is

The Android Compatibility Test Suite is an automated test harness which tests one or more devices to ensure that they offer a fully-compliant running environment for 3rd-party applications.

Passing these tests is required to use the Android brand name for retail devices and it's also the basic prerequisite for acquiring a Google Play license for OEMs.

Background/2: what CTS isn't

As of the current CDD (Compatibility Definition Document) the CTS still doesn't enforce stability or performance requirements for devices, even though some of its tests are taxing enough for the device to expose subtle problems. Some performance tests are nevertheless included, but they're not run by the main CTS plan, nor are they part of the current CDD.

Background/3: why it's important for TF101 users

Given the way the CTS stresses the system, it is useful as a standardised testing procedure for some of the problems some users are currently experiencing with ICS on TF101. It also can help ensuring that the device works as intended with (most) 3rd-party applications straight out-of-the-box.

Requirements: host

The host should be a Linux-based workstation with the Java 6 JDK installed (either Oracle's or OpenJDK - IBM JDK or GNU gcj are not supported, while Oracle/OpenJDK JDK 7 are untested), a fully working installation of the Android SDK and about 3 or 4 GB of disk space available for downloaded media files.

Test results are best viewed with a recent version of Firefox, otherwise you can post-process them into an HTML file using an XSLT 2.0 engine (tested with Saxon).

Requirements: target device

The target device should be a completely stock and unrooted TF101 running 9.2.1.24. It is possible to run the CTS on custom roms, but the tests assume that the device is running a user build (not eng, not even userdebug) in secure mode. YMMV, expect unpredicted failures.

Required files

You're going to need the following files from the CTS downloads page:

Android 4.0.3 R2 Compatibility Test Suite (CTS)

CTS Media 1.0

The following files are optional:

Compatibility Test Suite (CTS) User Manual (if you want more information about the test harness)

Android 4.0.3 R1 CTS Verifier (used to perform user-assisted tests - I didn't run this yet, if you feel brave go ahead and post the procedure you used)

You may also need a log collecting application, even though the CTS should be capable of collecting device logs on its own (it didn't work for me). LogDog (if you have it) is strongly recommended.

Host-side setup

Download the required files and unzip them into a folder of your choice, then change the first line of media/copy_media.sh from:

#!/bin/sh

to:

#!/bin/bash

so that the script runs successfully on Debian-based systems (the script is full of bashisms which make dash complain and exit with multiple errors).

Ensure the user you'll be running CTS under has the proper privileges to run adb.

Device setup

First and foremost: perform a full Factory Data Reset (either through the Settings menu or using the VolDown+Power chord at boot time and selecting "Wipe Data").

As soon as the system boots after the wipe, perform the first run wizard selecting English (US) as the system language, connect the device to a wireless network, do not add any Google account (yet), and then reboot the device again to regain access to the system bar widgets (known 9.2.1.24 bug).

Once the system is back up, open Settings and perform the following actions:

disable Quick Settings (ASUS customized setting → ensure Use ASUS Quick Setting is unchecked)

disable System bar lock (ASUS customized setting → System bar lock → ensure Enable system bar lock is unchecked)

disable ASUS Sync (Accounts & sync → ensure Launch ASUS Sync is unchecked)

disable the internal GPS to save battery (Location services → ensure GPS satellites is unchecked)

deactivate all screen locks (Security → Screen lock → None)

temporarily enable Unknown sources (Security → ensure Unknown sources is checked)

under Language & input, ensure that the default input method is Android Keyboard

enable USB debugging (Developer options (dismiss the nag dialog) → ensure USB debugging is checked)

enable Allow mock locations (Developer options (dismiss the nag dialog) → ensure Allow mock locations is checked)

Now go back to Accounts & sync, turn off synchronization and add the Google Account you normally use for Google Play, but don't restore anything, nor accept to synchronize anything - you're going to need this just to install WakeLock.

Close the settings, open Google Play Store, go to its settings and deactivate automatic updates and the infamous auto-add for widgets, then install WakeLock.

Now go back again to Settings → Accounts & sync and nuke the account you just set up.

At this point, you're ready to upload CTS test files, so connect the device via USB to your host, disconnect any other Android device (emulators included!), ensure adb is on your PATH and then run the following commands from the directory where you unpacked the CTS archives:

cd media
./copy_media.sh all

This will upload the CTS media files (Big Buck Bunny) on the device, it will take a bit of time.

When it's done, you have to install the instrumentation apks:

cd ..
adb install android-cts/repository/testcases/CtsDelegatingAccessibilityService.apk
adb install android-cts/repository/testcases/CtsDeviceAdmin.apk

Now back on the device, open Settings again and perform the following actions:

enable Accessibility → Delegating Accessibility Service

go to Security → Device administrators → check all android.deviceadmin.cts.* administrators

This is a good time to install LogDog and reboot the system if you wish.

After that, go back to Security and ensure that Unknown sources is unchecked.

Unplug the device from USB and charge it fully.

After the device is charged, plug it back to the host, go to Settings → Developer options (again, dismiss the nag dialog) → ensure Stay awake is checked even though it won't work.

Launch WakeLock, select FULL_WAKE_LOCK as lock type, deselect "Show notification?", select "Start on boot?" and then "Acquire lock".

>>> Press the Home button to go back to the home screen.

Congratulations, you're ready to start the test harness.

Running CTS

On the host, run android-cts/tools/cts-tradefed. It will detect the device and print its android device id, then wait for a command.

To start the full CTS suite, issue run cts --plan CTS. It will take several hours and the battery will

barely

last for the entire duration. DO NOT INTERACT WITH THE DEVICE IN ANY WAY WHILE THE TESTS ARE RUNNING!.

After the test is finished, just type exit to terminate the test harness utility. Test results will be in android-cts/repository/results/<start time>/testResult.xml, open it with Firefox to see them or process them with saxon-xslt to produce an html file.

If you were running LogDog during the tests, archive the logs now and transfer the log archive to the host.

Post-run caveats

Wipe the device to ensure that any leftover is cleared. If you wish to run the tests again, restart from the "Device setup" step.