[Android Software Development] fattire:
DERP (Device Environment Replacement Program)

initial pre-alpha version 0.001

(Aug 3, 2013)

by fattire (twitter: @fat__tire)

tldr?​

Derp is a general-purpose, platform-independent installer, written in python with wxpython, that executes .derp XML-based scripts to walk the user step-by-step through a ROM installation (or do whatever you want.) The idea is to replace text-based walkthroughs, howtos, and installation instructions by requiring a user to do almost nothing but run a .derp script and sit back. Derp walks through a series of scripted steps (as in, say, a ROM installation) and automatically does all the file downloading/adb/fastboot stuff while the user waits and maybe reads what's happening (at the script author's discretion). Derp also pre-installs and keeps Google's Android SDK tools up to date and even gives adb & fastboot a simple UI. .derp scripts are human-readable XML. Embedding bash and python is also supported in Derp, as is restricting scripts or even script parts to certain platforms. Derp runs as root on the local computer (it's an installer after all), and is open source/GPLv3 licensed. It also comes with sample scripts and a built-in tutorial for creating your own.

Still tldr? It's a script-runner thing!

-----

LONGER DISCUSSION...​

WARNING: RIGHT NOW, DEVELOPER TYPES ONLY! This is not for end users...yet. Hopefully people will find bugs and help fix them before an end user uses this on a “live” computer with an actual device. Again, because this is a software installer, DERP AND ITS SCRIPTS RUNS AS ROOT. Never run random .derp or .xml scripts you find on the Internet. This could screw up your device AND your computer, so... treat it just like any other script you’d (not) run as root. Also, the discussion in this forum is how Derp is supposed to work, but of course, there may be (probably are?) bugs.

THE "PROBLEM" AS I SEE IT

Working on the CM wiki, I've grown to appreciate how varied firmware install methods can be. Some devices need rooting. Some need firmware downgrades. Sometimes you can use fastboot. Other times you can’t. Some systems need to unlock the bootloader, etc. etc.

Installing this stuff can be hard. Okay, maybe not for you, but how about your mom or dad? Could your grandparents buy a device today and put CM on it themselves? There's been some chatter on the interwebs about how to make rooting and replacing firmwares easier... some kind of graphical installer seems to be the answer. But there are a million devices out there...

So people have been using text-based HOWTOs, walkthroughs, step-by-step instructions, and/or shell scripts and batch-type files to do this. I thought maybe a generic, unified scripting method might work better that gives the users readable instructions but optionally automatically does technical steps for them.

Hoping to avoid creating yet-another-standard-way-to-do-something, back in February, I searched online for generic installation solutions. But they all seemed to be platform-dependent, or weren’t licensed for general use, or looked really ridiculously complicated....

So (and big caveat here-- I'm not a programmer!) I whipped up a

proof-of-concept

for developers to play with to start thinking about how to address the issue. It took me a few weeks to get going, but ended up sitting collecting dust for months as I worked on other things and occasionally bothered friends to test the latest version.

Derp is not necessarily intended as any kind of final solution per se-- it’s just for further discussion/testing. A totally legit question to be answered: is this in any way even a good idea?

Let's find out.

SO WHAT IS DERP?

It's a general-purpose installer, written in python with wxpython, that executes .derp XML-based scripts to walk the user step-by-step through a ROM installation, optionally doing all the "technical stuff" like downloading files and running commands in the background. Ideally, you wouldn't have to write a long tutorial for every platform on how to do stuff-- a .derp script could BE the walkthrough.

The first thing Derp does is install the latest adb and fastboot from Google. That looks like this:

(Mac version)

Next, when you run a .derp script, it can automatically download and verify ROM files like CyanogenMod or tools or whatever from the Internet, and then install them.

(Linux version)

(Mac version)

As it does all this, the script can provides information and/or feedback to the user via a UI that hopefully looks like a normal installer. What the user sees is written by the script author in simple HTML. As the .derp script runs, the user simply ftaps "Continue" to proceed through the scripted steps.

As mentioned, .derp scripts are written in XML, which is platform-generic and easy for a human to read. The .derp script syntax, explained below, is also very simple. The script author is also free to embed bash shell scripts or python (or both) if advanced stuff is needed.

Worth mentioning too-- sections, steps, text, actions, or entire scripts can be restricted by the type of computer its running on (ie, don't run certain python commands on Mac, but do run them in Linux, or whatever).

And finally-- while my initial thought was to use this for installations of ROMS like CyanogenMod to a device, I'd think Derp can be used for many kind of installations or scripted operations-- even to wrap a UI around a bash or python script to make it easier for users to run without having to open a Terminal and start typing. Derp scripts don't even have to have to be used for anything to do with mobile devices, though it does pre-set up the Google SDK tools for that purpose.

In fact, a Derp script can do NO actions-- simply serve as a click-through set of HTML-based instructional steps for a user to follow by hand. Conversely, it can say nothing to the user but "Stand by, doing everything." and that's it.

FEATURES:

Easy to install. Debian-based Linux just uses sudo dpkg -i derp_0.001-1_all.deb and it's ready to go. Mac users: it’s Derp.app. Done and done.

Derp is GPLv3-licensed and source code is available now. Read license for terms, conditions, and more disclaimers.

Automatically downloads/updates all SDK tools (primarily adb and fastboot) directly from Google at every launch. So the user is always up to date. (also requires users to agree to Google's T&C...)

Uses an XML-based, OS-neutral installation script format that is easy to write and understand. Just about anything you want the user to do-- restart in bootloader mode, unlock the device, etc-- the derp script should be able to do. Even run bash or python scripts from within the script.

XML Tags:

<derp> - the main tag for a derp script.

<section> - a major category for individual steps.

<step> - Put as many of these in a section as you want.

<info> - The stuff the user sees as the script runs. You can add HTML tags to make it look good.

<file> - tell derp a file’s URL, MD5/SHA hash, and local filename. Derp will grab it and verify it for you automatically. These files can be roms, scripts, recovery images, etc. whatever your script needs to do its job.

<action> - valid “action” types include “adb”, “fastboot”, “python”, and “bash”. Future versions of Derp can add more. <action> allows your script to do stuff. Never worry about whether the user installed and set up the latest versions of adb or fastboot properly. They should "just work".

Using the above tags, you can not only have your scripts automatically do full installations, rooting, bootloader unlocks, etc, but simultaneously tell the user what’s happening behind the scenes if you choose. The user feedback is written in standard HTML-formatted text. The user just hits “Continue” whenever you want to move from one step to the next.

Included are example scripts to install CyanogenMod 10.1.2 on stock Nexus 7, Nexus 4, Galaxy Nexus, and HTC One. The latter script, written by Cowmix, demonstrates how to embed python to interact with the user, and they all include bootloader unlocking.

The only things I can think of that can’t be done automatically are steps that requires hands-on (ie, holding down buttons during power-on) or where, say, debugging mode needs to be manually turned on, or the slider needs to be physically unlocked. In the few cases where user involvement can’t be avoided, the <info> tag can be used to walk them in “real time” through that step.

A built-in tutorial on how to write your own .derp scripts explain how the tags work. (The tutorial itself is a .derp script.)

A console window helps you see what derp is doing in real-time...

Also included: a quick-access adb and fastboot text-entry in the console. This lets you start up Derp and type quick adb or fastboot commands without needing a terminal (or to deal with PATH issues)

“Debug Mode” lets you go through the script without invoking the <action> tags. Makes writing scripts easy.

Derp should automatically detect when a device is connected via adb or fastboot and let you know.

You may filter any Derp tag (including <action> tags) by operating system. This means that using a single script, the user can see different text or the script may behave differently depending on the platform. In fact, you can restrict the entire script to a particular operating system(s).

The script doesn’t actually have to “do” anything. It can be used simply to create walkthroughs or tutorials in a much nicer format than a step-by-step text file. Just link to a .derp file and let it walk the user through whatever. Easy to convert a text walkthrough to an interactive click-through just by adding <section> and <step> tags.

WHY MUST DERP RUN AS ROOT?

Remember, Derp is an installer. It needs to do important stuff, and as such it runs as root. I had considered trying to sandbox the parts that "needed" root and only enable it there and ask for permission for a single operation via an "enter your password" type of dialog box. But because the .derp format is so flexible, there were a million potential places where a script author could do varying kinds of trickery-- by breaking out of Derp to execute python code, spoofing directory paths, abusing the embeddable bash scripts, etc. It just didn't seem to make sense to try to anticipate and counteract all that. Playing cat-and-mouse endlessly is pointless.

Again, Derp is an installer

.

Installers get administration permissions

. Just like any installation script you'd run with "sudo" would get. Just like the package installer on OS X. Also, it is much easier to run adb and fastboot with root permission-- you can easily kill all running versions of adb for example, and fastboot seems to prefer it. Plus, it avoids the need for playing with udev configuration stuff in Linux.

This means that, like pretty much every other type of installer,

.derp scripts will have full access not only to your mobile device, but to the computer Derp is running on

. This seems to make the most sense to me, but I invite others to chime in on improving the design if you disagree.

All caveats and considerations apply. Do not run untrusted scripts, and do not run Derp on a "sensitive" computer (however you wish to define that).

WHAT IS MEANT BY "CROSS-PLATFORM"? (IN OTHER WORDS, WHERE'S THE WINDOWS VERSION?!)

I don't have/use Windows. Right now,

Mac & Linux builds are currently available

. Derp still needs to be ported to Windows, but since it’s wxpython, and I tried to make as little dependent on the underlying operating system, 95% of the work is hopefully done. Anyone with Windows who’d like to help, let me know.

I think it should be some minor changes to the setup.py file and a few definitions. Also, not sure if Windows supports the “bash” shell...

ANYTHING ELSE?

Ummm... That’s it. Remember, this is a work in progress and a proof of concept... Again, I dunno if anyone will see the value here, and maybe it will need a complete rethink. There are likely to be bugs, maybe even really bad ones. But after a few months of playing around, I kinda feel it’s ready for other developers to at least see and even try in a secure environment (as suggested, maybe a VM or something).

SHOW ME THE CODE!

The code is on github-- please submit commits-- fixes, new features, whatever-- as well as bug reports there. And again, figure there will be tons of bugs to be squashed.

Enjoy.

fattire (@fat__tire)

THANKS TO...

Big thanks for helping me test w/different devices: cowmix, hashcode, kornyone, ciwrl, utkanos, verygreen, and jeagoss

DOWNLOADS:

Debian-based Linux (Debian, Ubuntu, Mint)

derp_0.001-1_all.deb

MD5: 6e8eabe94cdfdba649ea41198211bb64

SHA512: 307aed0ad79de17793bb445d2b588388bf66b42716de36a055227f555bfc12ab3e61d5f0e3de804eb4c0c560f140a6318ea6dd1608cc78ee84b50336895cdfc2

Mac OS X (Tested on: Snow Leopard, Lion, Mountain Lion)

Derp-v0.001-mac.zip

MD5: b738e0a270f53d274baec0ce121577fb

SHA512: 3cf7d438c4dfd0c5c5d7c2f29fe19a76dcbb728acfe73a24e28cdb3f21624510c94f1c4224ad31118851f17205e4d7152619c15281c98189cb33ccac82c1505a

Source code on GITHUB is here.

SAMPLE DERP SCRIPTS

Nexus 7 stock to CM 10.1.2 installer - included (written by me)

Nexus 4 stock to CM 10.1.2 installer - included (written by me)

HTC One stock to CM 10.1 nightly installer - included (written by cowmix)

Galaxy Nexus stock to CM 10.1.2 installer - included (written by cowmix)

EXTERNAL SCRIPTS BY OTHERS

None yet...?

DONATIONS?

Not to me, please. If you feel the need to give someone money, consider donating to the EFF or the Software Freedom Law Center. It's really a donation to your digital rights. (I'm not affiliated with them except as a huge fan and occasional donor.)

REMEMBER, DERP IS EXPERIMENTAL AND YOU RUN IT AT THE RISK OF YOUR COMPUTER, YOUR DEVICE, AND YOUR VERY EXISTENCE AS A HUMAN BEING. I TAKE NO RESPONSIBILITY FOR WHAT DOES OR DOESN'T HAPPEN. DON'T RUN DERP SCRIPTS YOU DON'T TRUST COMPLETELY. YOU ARE ADVISED, JUST IN CASE, TO ONLY RUN SCRIPTS IN A SANDBOXED VIRTUAL COMPUTER. And let me know what y'all think.