[Frameworks] saurik:
Comments from Developer

So, yeah: I'm the developer of Cydia Substrate, the framework everyone uses on iOS to do runtime code modification. Back in 2011, I gave a talk at Android Open along with a demo of Substrate running on Android 3.0. However, after some in-depth discussions with people there who were interested, I realized that what I had at the time "wasn't sufficient": it was just the core of an implementation, not an end-to-end offering. By the time it had everything I felt it needed to launch--including a comprehensive website filled with documentation, a configuration application to install with it, fully tested support for both ARM and x86, a forward-compatible pure Java API vetted by a bunch of the top people in the iOS modding community (as I feel like breaking APIs after launch is one of the more evil things a framework can do), and an extension that would make sense to end users that they could try (so that trade press wouldn't be horribly confused, as I knew they would report on the release)--it was already 2013.

http://www.youtube.com/watch?v=tA9cnemnQ0A <- Android Open 2011 keynote teaser

As many people then know, I released it in June. A lot of people have tried it (165k installs just from Play, and another 20k downloads of the APK off the site), and many of those people even like it enough to keep it installed and leave positive reviews, despite there being almost nothing available to use with it except WinterBoard (which I really only did as a demonstration). However, I also get comments from people who seem to believe I'm some kind of "interloper" in the world of Android. Additionally, there are the people who leave reviews saying stuff like "this is stupid, we already have Xposed" (sometimes then explicitly adding in the "go home to iOS" kind of spiel). The #1 complaint, however, is "nothing I can do with it", because developers never seem to talk about it or use it much, and the people installing it are all end users. Clearly this isn't the kind of reaction that I thought would happen, especially after having discussed Substrate at length with pulser_g2 before launch (who said that the community here tends to be very good about judging things on their usefulness and technical merits as opposed to having emotional attachments).

http://www.cydiasubstrate.com/ <- Cydia Substrate

Given this, and after an encouraging back/forth I had with some people on reddit's Android subreddit a few days ago in some threads about the analysis I did of that recent Android iMessage client (people who didn't know much about the ways in which Substrate is very different than Xposed in capability and focus), I figured I'd finally make a post on XDA. I kind of had been waiting to make this post as well, honestly (as again: I like things to be more perfect before I release them than maybe people are used to around here ;P), but it seems like I'm now waiting for something that is itself causing the delay (I had really expected to do this in July, before the whole thing got more actively depressing). This is clearly that post ;P. I've responded to a bunch of other threads here talking about Substrate (and the many other Android projects I've released) in the past, but this is the first time I've actually started a thread.

(In specific, Substrate currently doesn't support some Samsung devices due to a change they make to the linker paths, and I wanted to have 100% device coverage before making the inaugural XDA post. However, I'm finding it very demotivating to spend the time to think through all the options I've been considering for workarounds given the overall lackluster reaction to my work, so I'm not even making fast progress anymore: I tend to work on the things that people react positively to, and while I got a lot of positive reactions on the balance from users, I got much less than I expected from developers given how many people use Substrate on iOS and how powerful the framework is. I think, from some conversations I've had, this is largely due to confusion over how Substrate on Android relates to Xposed, which many people seem to think of as the "home-town competitor" "that does the same thing". I thereby figure that I may as well attempt to directly address that core motivation problem, to see if I should even bother continuing spending time helping out in this community, hence this ludicrously long and highly personal post about what is essentially a technical framework ;P.)

[Readers who find the next section boring should skip below to "=== Substrate ===".]

I imagine I (sadly) thereby need to start by defending my history in the Android community, as many people seem to not be aware of much of it; it actually goes back very far, as I had promised the overall mobile community that if Android were ever rooted, I'd immediately start looking at it in earnest (before there was a device, I had already been messing around with the emulator, but the device concepts Google had at the time were more like slightly souped-up feature phones, not competitors to something like an iPhone). So, in 2008, when that first "root console attached to keyboard" mistake was found on the G1 that let you get a root telnetd running by just typing it into any search field, I dropped everything and drove two hours to Los Angeles to pick up a G1 (they were not selling them in Santa Barbara yet, due to T-Mobile not really having a presence here at the time). As promised, I immediately set to work attempting to help out.

As I ran a number of mailing lists already for iOS, I set one up for Android called g1-hackers, which attracted a good number of people, and even a few Google employees who worked on bionic and the kernel. On this list is where the G1's bootloader was first dumped: if you've ever heard the stories about Eddie Dost figuring out how to do it, this is that. In fact, it was from my G1, with a kernel I compiled (following Eddie's direction: I did not know much about flash drivers), that that first Android NAND was obtained (as Eddie had already updated his device and thereby didn't actually have root). Here is a link to the mailing list thread, directly to the post where we finally succeeded and I provided the kernel image I used so that others could perform the same dump on their own devices.

http://www.telesphoreo.org/pipermail/g1-hackers/2008-December/000096.html <- [g1-hackers] G1 boot code

Around that same time, I was also contributing to AOSP, providing a bunch of patches to things like mount and init, as I wanted to be able to get Android devices to a state where they could run something much closer to Debian than Android (I had my eyes set on kind of a hybrid). In the process of doing this, I wrote a guide that for a couple years subsequent were the canonical instructions for getting a bootstrapped build of Debian installed as a chroot under Android. At the time the patch turnaround on AOSP was sometimes over half a year (and almost never shorter than a couple months), which made contributing to the project sufficiently painful that I eventually stopped. If you search through Android's codebase, though, you still find some of my work.

http://www.saurik.com/id/10 <- Debian & Android Together on G1

At the time, I honestly do not remember XDA having yet become "the place" where people spent much time talking about Android: instead, a lot of conversation happened on IRC (which is where the iOS community had already been, and where it remains). There was a channel that I was a part of which included a bunch of people whose names would hopefully be familiar to people around here, including JesusFreke (and, much later, Cyanogen). I got to see the birth of a lot of great websites and tools (such as JesusFreke's smali/baksmali) while participating on that channel. Apparently, I was talking about "Substrate for Dalvik" on that channel in November of 2008 (which is also when I first joined XDA): that's how long I've been staring at this ;P.

During the next couple years, I ended up developing and maintaining a website called Cyrket, which had the mission to allow developers and users to search the contents of the Android Market using their desktop web browser. It also solved a few key problems that developers had with comments, in that you could only see comments for apps your device had access to that were then written in your language. Developers without devices, or with devices that could not see their product (which often included those that paid extra for the ADP1, which could not see copy-protected apps) could not see comments at all. Cyrket presented all of the comments for your application in all regions in all languages (and even used Google Translate to translate them all into your own language).

The way Cyrket had worked is that I scraped the contents of the Market using the same protocol Google's client used, indexed it (supporting find-as-you-type search), and exported it all to the site (well, originally, it was actually just a live client, but then it got really popular ;P). It got me into some mild trouble occasionally with the Android Market team, but overall no one seemed to mind it that much. Cyrket was actually the primary site people used for this purpose for a long time, and I even got the impression that people at Google were begrudgingly using it as it was more convenient than the alternatives. There were a few times where it had to be taken offline (due to changes and rate limits from Google), one time for months, but I'd usually figure out some new way to get it running. Honestly, though: I was really glad when Google finally launched a website for the Market and I was able to stop working on Cyrket ;P (and also glad that Google added most of Cyrket's features for developers to their publishing console, features that Apple actually still doesn't have AFAIK).

http://www.androidtapp.com/cyrket-android-market-browser-back-from-awol/ <- Cyrket Android Market Browser Back from AWOL!

Since those times, I mostly felt the need to get Substrate "awesome" (which started to really come together during 2011, after Cyrket was no longer needed), and so didn't do many larger projects on Android until recently. That said, I have been involved in things related to exploits and security. One of the higher impact things that I did was to release mempodroid, an implementation of the mempodipper exploit described by Jason A. Donenfeld for Linux 2.6.39+, which became the primary method to root devices running Android versions 4.0.0 through 4.0.2. Much more recently, users have been using Impactor, my implementation of the various "Master Key" exploits (based both on bugs described by Jeff Forristal as well as techniques I pioneered against a random AOSP bug).

https://github.com/saurik/mempodroid <- mempodroid README

http://www.saurik.com/id/17 <- Exploit (& Fix) Android "Master Key"

Given all of this, I hope people can get a feeling for just how strange and depressing it feels to me when people seem to suddenly believe I'm some kind of foreign invader :(. (FWIW, I also feel rather awkward having to describe all of this in this fashion, but frankly I'm at a point where I'm realizing that if I don't explain it in this much detail myself, no one else will. While I'm certain I'll get some people responding really negatively with comments like "he's such a blowhard, going on and on about silly little things he did", so far when I've given similar spiels to people in person at conferences, they often go "oh wow, I remember that tool/happening, but didn't remember that that was also you", and so figure that this might go a long way to fixing this weird problem: I'm not just "that iOS jailbreak guy".)

=== Substrate ===

Alright, now with that aside: in time for Google I/O (which was arguably bad timing, as I was then immediately unavailable for days ;P), I finally released Substrate. Substrate (in my clearly biased opinion ;P) is actually really cool: as far as I know, it is currently the only tool available for Android that allows developers to easily modify native code without patching/replacing. I know, for example, that people often ask how to modify features like the holo themes that are implemented in C, and the answer is Substrate: if you can find the code (which is often exposed via a symbol as there are tons of C++ symbols available on most Android builds) you can use Substrate to hook it at runtime in a way that avoids having to patch the files on disk, allows developers to deploy their changes across multiple ROMs, and supports the idea that users should be in charge of the specific features that they have on their devices (as opposed to ROM distributions).

As another concrete example that maybe makes this more obvious: sometimes you download a program from the Play Store (which, incidentally, I have a very hard time not constantly still calling the Android Market ;P) that is pretty much just a massive JNI binary--maybe an OpenGL game or a media player of some sort--that refuses to run on a device that has been rooted. A really common way that developers implement such checks is to do things like verify the existence of files on disk. The simple/common checks are very easy to detect and defeat using Substrate as you can hook the native "open" call from the C standard library, check if the filename is something like /system/xbin/su, and return "nope, not there".

http://www.cydiasubstrate.com/api/c/MSHookFunction/ <- MSHookFunction()

Substrate lets you do this kind of hooking in any system daemon (not just those spawed via app_process). Yes: if there's a program running in the background of your phone, some native service written by the OEM that manufactured the device, you can use Substrate to modify it. A lot of very interesting extensions on iOS involve these kinds of hack; for an extreme example, the software unlocks that we used to have for earlier iPhones involved modifying CommCenter, a native program that initializes the radio hardware: by hooking some of the code in that daemon, it was possible to, at just the right moment, inject a different command sequence over the serial connection to the baseband, exploiting it for the unlock.

http://www.cydiasubstrate.com/inject/android/ <- Android Native Injection

Of course, Substrate also supports hooking Java code (yes, a little like Xposed, which at some level uses the same underlying trick I walked people through in my talk at Android Open 2011). Somehow, though, a lot of developers don't seem to catch all that other stuff that Substrate lets you do, and get hung up on this one part that Xposed also manages, leading to all those aforementioned irritating comments about how "there's no point to Substrate because we already have Xposed": Xposed can't do most of the things Substrate can do (and the developer has even told me that he actively tries to avoid Substrate-like techniques as they are "pretty complicated", so it isn't even moving in that direction). FWIW, on iOS it took a lot of time for Substrate to get these features (it did not have them in 2008 when I first released it): they aren't trivial ;P.

http://www.cydiasubstrate.com/api/java/MS.hookMethod/ <- MS.hookMethod()

Even within the restricted context of modifications to Java, however, I think Substrate has a lot to offer. Again: I actively refused to release Substrate until I felt I had truly nailed a few things, including in particular the Java API (at Android Open 2011, I only supported JNI, which developers there told me would not lead to traction). I was a major proponent of aspect-oriented programming when I was younger, I got into byte-code engineering in college, and I co-published a paper on a Java code modification framework called jMonitor in 2004: this is something I've been thinking about for a long time, and I think the approach I take has some merit in and of itself. I know a lot more can be done (I feel it would be really interesting to have AspectJ-style pointcuts, for example, or the kind of bytecode-level instruction matching that I implemented as part of jMonitor <- features not described in the paper, I think ;P), but I felt a good first step was be to directly leverage the iOS community's six years of experience.

http://www.cydiasubstrate.com/id/6dfa187d-6e04-4f97-b63a-ae75b5338e01/ <- jMonitor [RV '04]

To this end, Substrate provides an API for Java that is very analogous to the API that it provides for modifying C/C++ and Objective-C. The focus is on "I know about some code and I want to modify it", allowing you to not have to think much about the timing or execution details of the program that may be loading that code (so you never have to think about "packages" or "processes" or "applications": you just concentrate on "classes", and thereby don't need a million "helper APIs" to handle each narrow timing case). To enable this, I use the aforementioned ability of Substrate to modify native code to hack features into the VM itself, giving me the ability to instrument events like "a class has been loaded". If you want to hook a method of a class from Apache Commons, and you want to hook that class no matter whether it was loaded as part of an application or dynamically as part of a classloader for a plugin downloaded by an application, this is trivial to express with Substrate. AFAIK, that use case isn't even describable using Xposed.

http://www.cydiasubstrate.com/api/java/MS.hookClassLoad/ <- MS.hookClassLoad()

This kind of VM-level modification and runtime code generation support (that is heavily flexed on iOS Substrate, and thereby has had years of in-the-field testing; so far Android has exposed just one bug in its ARM reassembler after release, and that was only in the qemu emulator for some reason) also means that Substrate's implementation of hooks is highly efficient: to compare again to Xposed, every time a method that has been hooked is called via Xposed, there is a linear-time search through a linked list doing a rather heavyweight comparison to determine which method it was after the fact; with Substrate, every call is direct, there are no lookups, and there are no comparisons, so you can hook an arbitrary number of methods with no slow down, so even very small methods that are called very often can be hooked without issue.

Additionally, with Substrate I wanted to address a specific pain point that many people would bring up when I'd give talks: "how is this secure, and how do I control what apps can use these features". This became even more important, as I wanted Substrate extensions on Android to be easily deployable via conventional means, such as the Play Store (yes: Cydia Substrate itself is in the Play Store, as I believe it is important for these kinds of features to not just be in the hands of developers on forums, but to be used by end users everywhere). To this end, I integrate into the Android security model, providing a special permission that applications must have to install a Substrate extension. This helps enable the idea that Substrate mostly "gets out of the way", becoming more of a technical detail behind your extension rather than something users will need to interact with constantly to activate or update your product.

I also wanted to provide at least something that would help solve the "reflection hell" that developers seem to always find themselves in while attempting to do runtime code modification in Java (even back on desktop Java using AspectJ). I thereby provide the means to "bless" a class loader, allowing it to access private fields and classes without the overhead of reflection: the access checks, for just that one class loader, no longer apply. Substrate extensions are loaded into such a "blessed" classloader. (I do not, even though I could, ever just whack an access check VM-wide; Xposed does this, and I feel like it is going to have security implications on Java security contexts applied to class loaders for plugins.) In the case of WinterBoard, for example, I don't ever have to deal with invoking Methods or getting Fields: setAccessible is just a dim memory.

Being able to use this functionality, however, can be awkward, and in some cases is almost impossible: while testing this feature, I realized that developers would end up needing "public stubs" for all the classes they were working with, but the calling convention for a public method and a private one is different, so the calls fail at runtime. I thereby ship as part of the Substrate SDK (yes, there's an easily-updated SDK package that you can download using the Android SDK Manager ;P) an extension to javac itself (as you might imagine at this point, written using AspectJ) that turns off access control checks: you can thereby access private fields or call private methods with no extra work both during development and at runtime. This all works sufficiently well that I generally run all of ant under the modification, such that anything ant compiles becomes "blessed".

http://www.cydiasubstrate.com/id/c17c554f-b603-4e3b-8f99-ebb3528e3ef8/ <- Java Access Controls

(And yes: this is one of the things that caused Substrate to get delayed even longer than it already had been. There was also a rather serious delay caused by my attempts to really nail the boundary between "code that is shipped with Substrate" and "code that is shipped with the extension", something that burned me a lot throughout 2013 as it was the kind of problem that spending time actively thinking about didn't directly help, requiring an epiphany I had soon before Google I/O. Arguably had I been willing to ship without documentation at all, and had I generally cared less, I would probably have had everything out in very early 2012, but during January-May I started working on the initial draft of cydiasubstrate.com, as I had apparently incorrectly thought that such efforts would be critical to developer adoption.)

Again, I write this in the hope that it clears up misconceptions, either about myself or about Substrate. As far as I can tell, Substrate has a lot of very unique value propositions: things that currently are only made possible by Substrate; and, even within the restricted scope of hooking Java code inside of a service being managed by Zygote (the only area of overlap with Xposed), I think that it offers a bunch of advantages in security, performance, deployment, and ease of development that cannot be so casually dismissed with a flippant "we already have Xposed (go home)". A lot of these features (and I haven't even gone into all of them: I could write paragraphs about the advantages of how Substrate's API handles chained hooks, the ways I enable extensions that need to cross classloader boundaries, or the way Substrate makes it easy for end users to temporarily disable extensions without complex tooling) come from having spent over a decade now thinking about this problem and the last five years actively managing a developer ecosystem with tens of millions of users on iOS.

I am thereby happy to answer any questions about how to use Substrate, issues with Substrate on any device (I never blame the device: I might not have a fix immediately for a specific problem, but I always consider it Substrate's job to work around issues the device throws at it to get its functionality in place so the task will at least end up on my todo list), or even about me (as a lot of why I find writing this both so important and so painful are due to the occasional-yet-present more-personal attacks/misconceptions I often seem to receive about somehow being an "outsider"). (That said, please do have some patience: sometimes my ravenous need to do nearly 24/7 testing on a specific device has to give way so I can go to a conference I'm giving a talk at, or so I can focus on a different problem that might be more pressing or simply have a higher probability of near-term success: spending an infinite amount of time on one problem is unfair to all of the other problems that exist ;P.) [And, in fact, I have a meeting I have to be at tonight, but which hopefully won't take insanely long.]