[SuperSU] Chainfire:
Issues with the v2.80 roll-out

I wanted to dedicate a post to addressing the three major issues that occurred during the v2.80 (now v2.82) roll-out.

Again, our apologies to anyone who was inconvenienced by the issues.

I would like to emphasize again that though the root loss and boot loop issues appeared to be the end of the world here, they really only affected a (relatively) small number of SuperSU users.

It should also be noted that it wasn't just me, but also a few of the CCMT guys that worked through the weekend to find and fix the problems.

Root loss on a subset of devices running 4.2 through 5.1

This was by far the largest issue. In the end, it came down to a misconfiguration in the build system of the team responsible for releases, which caused the native binaries to be linked against the wrong version of the Android libraries. This doesn't always cause problems, but for SuperSU it did, on a subset of devices running 4.2 through 5.1. Aside from the code, the build system should also have been perfectly synced, but obviously it wasn't.

This problem was not detected in the SRx releases, because those are built on my own system (you will not be surprised to learn that I work remotely), and thus weren't affected. The developers at CCMT likewise didn't encounter the problem during their development. The final build was of course given a few quick test runs as well by all of us, but unfortunately it wasn't tested on a device that actually suffered from this issue.

v2.80 was started on a staged roll-out, which means that only a small percentage of users receives it, and this number is increased over time. There were no reports of this issue during the early roll-out stages. Only when roll-out was given the full go-ahead did the error reports come in. It makes me wonder if staged roll-outs prefer users of newer Android versions, but that sounds unlikely to me.

Unfortunately, the Play Developer Console just received significant updates, and it did not allow us to simply unpublish v2.80 and republish v2.79 to prevent more users from being affected. At this point, there were two things we could do: try to identify the problem and fix it (which can take who knows how long) and release an update, or re-release an older version's code with a new version number.

After a number of quick tests did not reveal the cause, the decision was made to take v2.79's code and rebuild and republish it as v2.81. 9 out of 10 issues, this would have worked. However, that only works if the problem is actually the code, but unbeknownst to us at the time, the problem was the build system. As such, v2.81 was released suffering from the same issue.

When we finally found the issue it was quickly fixed, and v2.82 was released.

We've certainly learned a few lessons about distributed release management, and we will improve the procedure. This was a nice combination of mistakes, coincidence, and sheer bad luck; but I maintain that the actions taken as the issues occurred were the correct ones for that place in time with the information available.

Xperia boot loops, Android 4.2 through 5.1

This actually did not affect only Xperia devices, and certainly not all Xperia devices, but they were the most vocal reporters. Other casualties of this bug were specific HTC One firmwares, and even some custom firmwares running SuperSU in forced system mode on Android versions beyond 5.1. This bug is caused by some SELinux refactoring done all the way back for v2.79-SR1 in December of last year, but was either under-reported or not sufficiently investigated by yours truly. This further illustrates how important it is to have enough BETA testers that test and report problems.

Unfortunately actually reproducing the problem took longer than the v2.82 Play release could wait for in light of the issue mentioned above. My Xperia's did not suffer from it, and none of my or CCMT's other test devices displayed the issue. Only when a user reported the issue on a specific HTC One M7 firmware could I reproduce it, and once the problem is reproduced, it's usually quickly fixed. The v2.82 ZIP distribution from my site includes the fix already, but the Google Play distribution does not.

Permissions, analytics, feedback

There was some confusion about permissions, analytics and feedback screens as well. Part of that was due to a miscommunication between myself and CCMT.

At some point during the development process, CCMT decided (Google) analytics would be included in SuperSU. I debated with them for a long time to try to get them to change their minds. I thought I was overruled at this point, but in the end the analytics code was never committed; and instead we got a feedback screen that could gather information only when the user wants to submit it.

Unfortunately that feedback screen requires pretty much the same permissions as analytics would have, and many users were not happy with this. Between the v2.80 and v2.82 releases, it was completely removed again due to this. So right now, all requested permissions are the same as they were in the past, and there is no analytics or feedback-data-gathering other than what Google Play already does for us (and every other app) by default.

I don't know where that will go, though. The fact remains that business types want to have that sort of information, and apps without basic analytics are few and far between. We shall see. Either way, if anything changes in this area, it will be properly announced before-hand.

On a personal note, no one can be sure what the future will bring, but the cooperation with CCMT has been going on for almost two years now, and everything has been fine. While I understand caution is warranted, there are other solutions out there if you cannot deal with CCMT operating SuperSU. Again these past weeks there have been posts questioning their motives solely on bases of nationality. Whatever your reservations may be, if that is your only argument, it is not welcome here.