[Galaxy S III Original Android Development] Arifcatalyst:
This is a continuation of the story, the last installment was at https://plus.google.com/u/0/101093310520661581786/posts/fhkWr8P3CQm

So, by the end of our last installment:

1) We finally had guaranteed methods to protect ourselves thanks to Ken Sumrall of Google

2) A contact at Samsung had finally acknowledged that they were aware of the situation and "working diligently" on it.

First things first, Garwynn and I responded to Ken Sumrall's last email with some further questions, such as: Since damage on erase didn't ALWAYS happen, only a percentage of the time, had he noticed any things that would imply that certain forms of erase were more dangerous than others, etc? Ken usually had a reasonably long turnaround time (I'm sure everyone at Google was getting insanely busy with Jellybean finalization at this time), so we weren't expecting a response soon.

Second, I was contacted by what you could consider to be one of Samsung's top-tier developer relations people, the contact that +Daniel Hillenbrand had reached out to at the end of the previous post. I found out that this was probably the contact Supercurio and others had. (There is no public information on how to contact this person, it's more of a "don't call us we'll call you" thing. While top-tier devrel probably should be isolated to keep the "noobs" away, Samsung really does need public-facing devrel contacts that are more than just a social media guy on Twitter - you may remember my earlier rant about SamsungJohn, Samsung's "official" devrel guy.

With that, we started exchanging emails, and what "solutions" Samsung was working on. As we already had solutions for protecting against further damage in most situations, our focus was on:

1) Kernel-level protection against damage (since anything other than kernel-level protection would allow damage from a variety of sources, such as malware, a user mistake, or just a code mismerge.)

2) Repairing damage done (the repair process Ken had said he was trying to get permission to provide more details on.)

For 1), we had, among others, myself, +Chainfire , and a couple of other kernel developers from XDA. Chainfire was especially in need of Samsung deploying kernel-level protection for official kernels, since his speciality with CF-Root is initramfs repacks of stock kernels (which is one way to work around Samsung's previously documented tendency to fail at GPL compliance on a regular basis.)

For 2), we had +Adam Outler , +Dominik Marszk , and a few others. If you recall, the only known repair method apparently would fully wipe the eMMC including bootloaders, so we needed a way to boot devices from USB if the process got interrupted and the chip was left in a nonbootable state. For those not familiar with those two names, Adam is a hardware specialist who is VERY familiar with changing the bootmodes of Exynos3 (aka Hummingbird) and Exynos4 devices. Dominik is an ARM assembler and bootloader specialist. Together, they are the primary team behind Unbrickabble Mod/Unbrickable Resurrector for Hummingbird devices, which allows a device that has had bootloaders corrupted to boot from USB and have the bootloaders repaired this way.

Eventually, Samsung's "solution" was presented to us. Chainfire was NOT happy with the proposed "solution", neither was I... It involved no kernel-level protection, and was inferior to what we already had in place with BOARD_SUPPRESS_EMMC_WIPE in CM. In addition they asked us not to distribute the solution and to redirect kernel developers looking for a solution to them! Keep in mind, we're all open source developers and nearly everyone on this email exchange had ERD status on XDA, status which we had achieved for our code of ethics regarding cooperation and sharing. (Yes, Chainfire and Supercurio do have many proprietary closed-source products, but they also make significant contributions to the open source community too.) However, it did answer one of the questions we had posed to Ken (whom we had not heard back from) - There was a definite difference in "dangerousness" between secure and nonsecure MMC erase - Secure erase was definitively dangerous, Samsung believed nonsecure erase was 100% safe. We weren't quite so sure - A few people had reported bricking on stock recovery with N7000 XXLPY. However, nonsecure erase was clearly safer... Which at least made Samsung's position a bit better as stock recovery only issued nonsecure erase.

Samsung also pretty much refused to discuss any solutions involving bootloaders... The reasoning, which made no sense, was that nearly all of their warranty claims due to custom firmware prior to this eMMC defect were due to bootloader corruption... Of course, this makes no sense, since we wanted to discuss methods of recovering from bootloader corruption, which would eliminate the majority of these warranty costs for Samsung. We were even offering to do the majority of the engineering and solution deployment ourselves, as long as Samsung just gave us some specific small components Dominik and Adam needed!

Throughout this, I guess you could consider me to be acting in a coordinator position - I had some knowledge of every component involved at this point, and for things I didn't know, I knew who to contact/ask. I also had contacts in the communities for nearly every affected device.

Sometime in here, Samsung started discussing a meeting with some of the developers involved with this face-to-face, in order to discuss a solution for repair of damaged devices. It was starting to shape up that this would be myself, and possibly Adam as the bootloader expert, since any robust solution had to have a method for repopulating the bootloader.

Discussion continued, and it looked like Adam's role would be less and less, since Samsung absolutely did not want to talk bootloaders. I was starting to feel like my support team was being taken out of the picture one by one.

I also attempted to bring Ken Sumrall into the discussions, since, prior to the community's run-in with this bug in 2012, he had previously encounted it during Galaxy Nexus prototype development in 2011. I considered him, at this point, to likely be the most knowledgeable technical expert on the issue, including possible recovery methods since he had experience with at least one. Our contact at Samsung immediately replied that they were "uncomfortable" being involved in any discussion in which Ken was involved with.

Something was weird - we hadn't heard from Ken in ages, and Samsung acted VERY strangely when we attempted to bring him up. It almost seemed like Samsung was attempting to punish him for his assistance to the community, even though he had saved them the motherboard replacement costs of hundreds if not thousands of devices.

Samsung, after "working diligently" for a month, throws a grenade in our faces

In early July, XXLQ5 leaked for the I9100. Within a day, numerous reports of bricks had piled up. Not too long afterwards, XWLPM went live on Kies, and people were bricking left and right with this build too. http://xdaforums.com/showthread.php?t=1756242

Despite claiming to be working diligently on this problem, instead, Samsung took a previously safe device and endangered it!

By the way, Samsung will, when this is brought up, claim that they have a very long testing cycle, and this build had entered testing before they were fully aware of the issue. This is not correct. If you download the XWLPM build, and unpack the kernel, you will be able to find the following string:

Linux version 3.0.15-I9100XWLPM-CL837163 (dpi@DELL145)(gcc version 4.4.3 (GCC) ) #3 SMP PREEMPT Thu Jul 5 11:26:14 KST 2012

Yup, that's right - This kernel was built on July 5, mere days prior to release, and almost a month after Samsung claimed to be working diligently on the issue. It should be obvious, of course, that the community was quite enraged about this.

At this point, between this, Samsung ensuring Ken was out of the picture, and Samsung trying to neutralize the rest of my support team, I was seriously beginning to think Samsung was setting me up for failure with this meeting. I nearly called it off at this point, but the thing was - Part of the meeting agenda was to discuss repair methods. Even if we couldn't get a easily field-deployable robust solution, AT LEAST we might have been able to get something the JTAG houses like Mobiletechvideos could use to repair damaged devices. So with that, I pressed on with the meeting. Originally it was scheduled for early July, but the date slipped, mainly due to family obligations I had in early July.

Near the end of July, the big weekend came. One of the difficulties in scheduling this was the fact that none of my Android work is in any way, shape, or form related to my day job... So the meeting couldn't be during business hours, as I did not want any impact on my day job. In the end, we had things set up for a Friday evening and Saturday in late July in a conference room at a local hotel in Vestal, NY.

My next post will cover that meeting, and the rest of the Superbrick saga post-meeting.

I was planning on doing this yesterday with Part 3, but I'm kinda fried... This is a continuation of the story, last installment was at https://plus.google.com/u/0/101093310520661581786/posts/PsLinmMLEpf

So in late July 2012, my meeting with Samsung finally happend. To Samsung's credit - They had an engineer and their devrel contact fly out from Korea and one engineer came from California. We started out by going out for dinner - it was Friday evening and we were all hungry. The staff Samsung sent out were a pleasure to talk with and begin working with. After dinner we returned to the hotel to start working.

Goal 1: Try to reproduce the bricks. I brought a number of firmware configurations that had been confirmed to be dangerous in the past, such as:

CF-Root releases from before +Chainfire had implemented safety measures

Cyanogenmod 9 builds with BOARD_SUPPRESS_EMMC_WIPE disabled AND for added measure, MMC_CAP_ERASE reenabled

Both of the above had been confirmed to cause numerous bricks when released to the wild. However, as you all may know, bricking was never guaranteed - at least three kernel developers/initramfs repackers (ktoonsez, +Gökhan Moral , and Angelom) had taken kernels through complete testing cycles without an issue, only to have piles of bricks within hours upon releasing their projects to the public.

Samsung brought an I9100 and N7000 that had been specially modified with a socket for the eMMC chip, so one of their engineers could swap eMMC chips at will.

As a result - we were unable to actually force a brick on one of Samsung's test devices. We did have anomalous behavior from one flash chip, but it was exhibiting symptoms different from Superbrick, so we all wrote it off as just a bad chip - probably it had fallen victim to ESD. After all, we were in a hotel conference room and not an ESD-controlled lab. This was disappointing and frustrating for all of us.

Near the end of Saturday, one of the engineers provided me with some information that was a very good explanation of why we had failed to reproduce the issue. He outlined the specific conditions in which a failure would occur. He asked me not to publicize those details (as they revealed very specific details of the device's internal structure), so I will not. I do, however, feel comfortable saying that it would be highly likely on the /data partition of a phone that had been used for a few weeks. This explained why our attempts failed - we were working with fresh eMMC chips. (Note: In some cases, such as +Guillaume Lesniak 's experience, it is possible to brick MUCH earlier - he bricked on /system within hours of receiving his device.)

Had I known this information earlier, or had been provided one of Samsung's test devices in advance to experiment with (since I was NOT willing to sacrifice my own personal device for the sake of experimentation...), we might have had more success. Near the end of the meeting, I wrote a "diskbeater" script to try and reproduce the issue using this new information, but it was too late in the day and one of the engineers had to be at the Binghamton airport by 5 PM.

Goal 2: Transfer information on a method for repairing damaged devices. This was my primary goal, and this was the repair method Ken had indicated he was going to try to get permission to provide before he was silenced. It turns out that the Samsung Mobile engineer who had been working with this solution discovered that it was VERY sensitive - frequently the process would fail in such a way that the eMMC chip was totally and completely dead. (Failure rate was over 50%). Ken had mentioned that the process was dangerous, and we had assumed this was simply due to the fact that the bootloaders would get clobbered and the device would be a traditional hardbrick if the process failed. However, this new information was still consistent with the "dangerous" description.

This was very disappointing, however, I know from my experiences as an engineer that sometimes you wind up in a no-win situation. This is where we were - it was no-win for both Samsung and the firmware development community.

Things like this happen... The key when you encounter a situation like this is: How do you keep it from happening again? This led to:

Discussion Item: Open up improved lines of communication between the community and Samsung.

We know from a wide variety of historical data that there are plenty of issues that can easily escape lab testing, but be a major issue for hundreds if not thousands of users when you release something to the wild. For example, a 1% failure rate of eMMCs would easily escape lab testing and be difficult to reproduce at will, but would still be an utterly unacceptable failure rate in the field. Similarly, I'm sure Samsung has never seen the "fuel_alerted" wakelock issue I ranted about in my first or second installment of this saga - but when it hits the field, it becomes a HUGE thread on XDA.

This is what custom firmware development communities like XDA have to offer - We are much more "in touch" with end users, and so if there's something affecting 1% of users that escaped Samsung's testing process, we'll probably know about it. We might even (in the case of fuel_alerted) have a fix ready to provide back for evaluation.

So what we discussed was:

Create a new private forum on XDA for Samsung/community relations. Initially, it would only contain the ERDs from XDA that worked with Samsung devices, and our contact. Ideally, it would expand to a few engineers from Samsung to be a place we could "talk shop". We could also bounce ideas and proposals off of Samsung. Eventually it might get expanded to some of the RDs that worked with Samsung devices, with Samsung's approval of course. (As to why not use the ERD forum - we wanted to retain a place to talk internally.)

This would allow established developers from the community to filter out issue reports and be able to present them to Samsung in a more clear manner.

Other things discussed:

What the hell happened with I9100 XWLPM? - The answer someone at Samsung gave my contact was: The lack of MMC_CAP_ERASE in I9100 Update4 was a mistake. All shipped I9100 kernels had it set, according to this person. (I do not know who...) - However, anyone who follows the I9100 community can tell you that with the exception of the Siyah 3.1rc6 incident, that community was blissfully immune to Superbrick until this release. It was quite clear that MMC_CAP_ERASE was NOT present in binaries - but someone actually would rather admit to violating the GPL than to making a mistake!

Why not just remove MMC_CAP_ERASE? - The answer was that this would cause too much performance degradation by also blocking MMC discard operations. Problem here - NOT A SINGLE ICS firmware I have EVER seen has the "discard" ext4 mount option set - so there was no "discard" functionality in use for removal of MMC_CAP_ERASE to interfere with. As to recovery and erase - Normal users wipe VERY rarely, and this compared with the non-use of the discard mount option means that normal users get negligible performance benefits from having MMC_CAP_ERASE present.

(Side note: The way flash memory works, it's faster to write to pre-erased flash than to already-written flash. This is why it has ERASE/TRIM commands. It results in 10-20% write performance benefit on average if TRIM/discard is used - far less if only used when formatting a partition.)

So, in the end, we went to dinner, and then we all went our separate ways. While the meeting had, for the user community, ended in disappointment, it did result in hope - hope that we would be able to avoid a situation like this in the future.

Action items from the meeting

I was to provide contact info for Josh Groce at Mobiletechvideos so Samsung could obtain some bricked devices for analysis. (Yes, their service centers receive plenty - but if they get em' from Josh they KNOW they are Superbricked items.) - I emailed this the next day

Samsung was to contact Josh and obtain some bricked devices

I was to provide an improved version of my "diskbeater" script - I provided this within the week

Samsung was going to work on a kernel-level protection solution that would block secure erase but allow nonsecure erase to be deployed in official firmwares, hopefully ending this saga once and for all unless they were wrong about the safety of nonsecure erase (some of us had doubts still) - This was established to be far safer than the current situation.

Events after the meeting

Samsung never contacted Josh to collect some bricked devices. In some cases, this could have been due to miscommunications/language barrier issues.

Samsung did submit a patch to LKML - https://lkml.org/lkml/2012/8/29/28

Our contact's management completely shot down the "private forum for discussion" idea - So Samsung effectively refused to make any changes to try and prevent a disaster like this from happening again. Now, I speak from personal experience at my place of work - We have a pretty rigorous quality control process, but sometimes stuff slips through the cracks. As a result, we have "learning from our mistakes" institutionalized - If something bad happens, measures WILL be taken to prevent it from happening again. As a result, we (for example) have been conducting flight operations from our hangar for 3 decades without a single airframe lost or any significant FOD damage. The number of defect reports against our product on receipt by the customer is astonishingly low compared to the industry average. Samsung, on the other hand, absolutely refused to make any changes to try and learn from this mistake and prevent a mess from happening again.

Edit: This wasn't in my original post. However, a few weeks after the meeting, the CM guys had a victory against the Yamahell audio mess with CM10 on GS2-family devices. We put up test builds for I9100, I777, and N7000 - Within hours we started seeing brick reports. We pulled the builds and started investigating. It turns out that when the BOARD_SUPPRESS_EMMC_WIPE implementation was merged to JB, a few Makefile lines weren't merged properly and so the feature would never turn on. We fixed it and re-released, but this incident highlighted the need for kernel-level protection.

Which gets me to a small rant on the usual "flash at your own risk" disclaimer. It's there because of incidents like the above - Sometimes, something dangerous gets through testing. A responsible developer will immediately take measures to pull a release discovered to be dangerous. However, in dealing with this situation, and elsewhere unrelated to Superbrick, I've run into developers who use it as an excuse to do whatever they want, including release a package with bootloaders and no warning (the DesignGears incident from a few posts ago), and leaving known-dangerous stuff floating around for months.

Edit 2: Another thing that wasn't in my original post. It turns out that a number of people who claimed to brick on "stock recovery" may not actually have been. It turns out people were "fakeflashing" CWM from stock recovery. Normally, this won't ever work on a stock device, but it turns out Samsung's ICS recoveries on many devices allow packages signed with testkeys to flash. Normally this would be good for the community, but combined with Superbrick, old CWM releases that were "fakeflashed" were one of the last sources of damage we managed to clean up, this discovery was made a few weeks after the meeting. Samsung's stock recoveries allowing fakeflashes blew +Koushik Dutta 's mind.

So let's fast forward to October 2012: FI27 released for the Epic 4G Touch. It did NOT have Samsung's official fix for this issue that had been merged to Linux mainline over one month previously - https://git.kernel.org/?p=linux/ker...it;h=3550ccdb9d8d350e526b809bf3dd92b550a74fe1

Basically, Samsung's official patch is in mainline Linux (which will NEVER be seen on an affected device), and is ALSO in the I9300 Jellybean tree - ANOTHER device that is not affected by the issue.

Samsung's answer for why FI27 (and no other firmware for any affected device) does not have the fix:

Long testing cycles is their claim. Except:

Linux version 3.0.15-1127689 (se.infra@SEP-98)(gcc version 4.4.3 (GCC) ) #3 SMP PREEMPT Thu Sep 27 18:24:35 KST 2012

That's right - This kernel was built on September 27, approximately three weeks after the patch was mainlined - there is no excuse for this kernel not to have this patch considering the severity of the issue it fixes.

Similarly, no other kernel for I9100, N7000, etc. have received fixes. Now, if these devices simply hadn't seen any updates I'd let that slide - but FI27 not containing this fix was absolutely unacceptable.

Samsung's explanation for why they have not taken any action - Supposedly, I came to the meeting unprepared.

Except, as I explained above:

1) Everything I brought had been confirmed to cause issues for other users

2) Samsung did not provide me with test devices with replacable eMMC in advance to permit me to run more tests

3) There is no way in hell I was going to sacrifice a personal device that was a grey-market import (you cannot purchase an N7000 with warranty in the USA) that had absolutely no recovery method.

4) Samsung withheld information about why some devices bricked and some didn't until late in the second day of the meeting.

So, at this point - It is mid-November 2012, and not a single device affected by Samsung's defective eMMC has received a kernel fix. While the efforts of the community have damage rates WAY down, as long as Samsung's official kernels are vulnerable, I'm still going to get a PM every few days from a Superbricked user that needs help whom I cannot help.

On top of this - While all known privilege escalation exploits for ICS required user interaction via ADB, both of those (mempodroid and RestoreRoot) have been patched in 4.1.2 - The search is on for new rooting methods, and as people may remember from +Adam Outler 's BABBQ presentation, while exploits are usually developed for the purposes of rooting, once they have been developed, malware authors often use them. If the next exploit does not require ADB, and is applicable to older Android versions (they usually are...), then a malware author could cause permanent damage to Samsung devices with just a rogue application. (Yes, they could already just clobber bootloaders - but if that happened at least it would be JTAG-repairable. Superbrick = device is dead and unrepairable without hardware replacement.)

Since no new lines of communication were opened, and no affected devices have received an official fix, at this point I feel like the meeting was just another Samsung publicity stunt to claim "we care" without actually doing anything of significance.

One thing I have to say: The staff sent to the meeting cannot, in any way, be blamed for this. It is their management and other staff at Samsung that either shot down the ideas they came back with, or outright lied to them. I've already documented at least two situations where people within Samsung Mobile felt it was OK to lie to our devrel contact's face. Our contact cares, and really tries to do what's right, but they are hobbled by the environment they must work in.

In the next installment - We return to the time at the beginning of the Superbrick saga, and cover everything ELSE that happened since Spring 2012...

This is a continuation of the saga, with the last installment at https://plus.google.com/u/0/101093310520661581786/posts/BrUZge7Goe3

The beginning is at: https://plus.google.com/u/0/101093310520661581786/posts/cdPnNjLAb4F

Chronologically, we return to the end of: https://plus.google.com/u/0/101093310520661581786/posts/6kzdh1bTK4p

Back to Spring 2012. My memory of this time beyond Superbrick is a bit fuzzy, but I'll try to cover what I can.

While the Superbrick saga was ongoing, there was still other work to be done Cyanogenmod-wise. For example, numerous users were complaining about issues with autobrightness on all Galaxy S2 family devices. It turned out that there was something "odd" about the reporting rate of the light sensor in Samsung's sensor HAL, and in addition, the light sensor reported improper min/max ranges. While the sensor could easily report values in excess of 10000 lux, the HAL would report a maximum of 3000 lux. Touchwizz apparently had a hack so that it would ignore this, but that improper range report really screwed up AOSP. We almost committed a hack to just ignore it, but thanks to some effort started by +Espen Fjellvær Olsen and finished up by myself, we managed to get an open source sensor HAL working on I9100/N7000/I777 by porting parts of the Aries and Crespo sensor HALs (nearly all I9100 sensors were found in either Crespo or Aries. N7000 required a little extra work for light sensor and baro, but it wasn't too bad going from the I9100 HAL.)

In addition to this, the RIL returned some undocumented response codes, and behaved badly when these response codes were handled properly. A request was sent out for information on what these response codes meant, the response was that the RIL team would not provide any documentation whatsoever. Now, I can understand keeping the RIL closed-source - all manufacturers do. However, there's nothing "special" about the interface between the RIL and the rest of the framework. As a result, +Daniel Hillenbrand had to spend a lot of time reverse engineering the meaning of these response codes.

In addition to all this, we were constantly plagued with audio bugs thanks to the audio HAL not behaving in a sane manner. If you'll remember from earlier, the audio HAL was a major problem in the early days of ICS, as while CM usually has backwards-compatibility with "older" blobs implemented, the Yamaha Gingerbread blobs totally failed in ICS. The Yamaha ICS blobs also had all sorts of weird issues when used with ICS, such as notifications always forcing the headphone volume down to minimum.

At some point in here, Daniel, Espen, and +Guillaume Lesniak tried to get the remainder of the Insignal reference sources for Exynos4 working on I9100/N7000/I777. Things didn't work out too well:

Opensource libcamera sort of worked for preview, but completely failed on capture, there was obviously some special JPEG "secret sauce" missing

Opensource libfimc failed when used with "blob" libcamera

Opensource hwcomposer was effectively worthless - It was indistinguishable in behavior from not having any HWC library at all. More on this later.

After a while, Google announced Jellybean, and a few weeks later, Jellybean source dropped, and the frustration began again