[Honor 8 ROMs, Kernels, Recoveries, & Other Develop] Corax26:
Hi guys, I have some great news, which I'm sure many of you will be interested in: after getting fed up with not being able to use the GPS, and finally getting to take a closer look at the crashes, I have found a workaround to avoid system_server crashes while using GPS or High accuracy location! I have been using it for about a week now, and I haven't had a single crash so far :D Before that, system_server was crashing (resulting in a soft reboot) pretty often while in High accuracy mode, even without a foreground app using location (presumably because of the location services)... What I found may also partially explain why only some of us encounter crashes while using the GPS.

Before getting any further, a word of warning: the workaround I am describing here is a complete hack and I am not responsible for whatever happens to your device as a result of using it, including catching fire, calling your boss while you're drunk, etc. etc., you know the drill. That being said, it does nothing too crazy either, see below.

TL; DR

Right, so before we get to the details of the hack, here is the procedure for those impatient to try this out. I am using the July build on an FRD-L09, but it shouldn't matter what build you're using really, because we're going to touch the vendor partition, which hasn't changed for months.

Download the attached zip (View attachment flp.hi3650.so_noreportdata.zip) and decompress it somewhere.

Reboot your device to recovery.

Remount the vendor partition read-write:

adb shell mount -o remount,rw /vendor

Pull the original file and back it up somewhere, just in case:

adb pull /vendor/lib64/hw/flp.hi3650.so .

Push my hacked file:

adb push flp.hi3650.so_noreportdata /vendor/lib64/hw/flp.hi3650.so

Reboot to system and enjoy!

I really hope that this is going to work for everyone and I'm curious about how this is going to fare on the various FRD models, I'll appreciate reports if you try this out, especially if it doesn't work as expected.

WTH is going on here

So, since you're still there, I'll just assume you want to understand how this works (or doesn't, depending on how you look at it...).

First, let's have a look at the crashes. While using my phone in High accuracy mode, I've encountered a number of crashes, and I looked at the tombstones of the most recent ones. In all cases, system_server aborts in libart with the following message:

'art/runtime/indirect_reference_table.cc:132] JNI ERROR (app bug): local reference table overflow (max=512)'

What this basically means is that the native part of system_server (that uses the vendor-provided hardware libraries) asked to get a reference to a new Java object via JNI while already holding too many references (512) to Java objects. Not that informative in itself, but that's critical to understand what follows.

Interestingly, inspecting those tombstones also showed that there are actually not just one, but at least two possible backtraces when the abort occurs:

#00 pc 000000000006c830 /system/lib64/libc.so (tgkill+8)
#01 pc 0000000000069ccc /system/lib64/libc.so (pthread_kill+64)
#02 pc 0000000000023ea0 /system/lib64/libc.so (raise+24)
#03 pc 000000000001c924 /system/lib64/libc.so (abort+52)
#04 pc 000000000043101c /system/lib64/libart.so (_ZN3art7Runtime5AbortEPKc+456)
#05 pc 00000000000e5998 /system/lib64/libart.so (_ZN3art10LogMessageD2Ev+1576)
#06 pc 000000000024c4e8 /system/lib64/libart.so (_ZN3art22IndirectReferenceTable3AddEjPNS_6mirror6ObjectE+308)
#07 pc 000000000032b3b0 /system/lib64/libart.so (_ZN3art3JNI10NewObjectVEP7_JNIEnvP7_jclassP10_jmethodIDSt9__va_list+680)
#08 pc 00000000000162e8 /system/lib64/libandroid_servers.so
#09 pc 000000000002496c /system/lib64/libandroid_servers.so
#10 pc 00000000000237ac /system/lib64/libandroid_servers.so
#11 pc 0000000000023b88 /vendor/lib64/hw/flp.hi3650.so
#12 pc 000000000001344c /vendor/lib64/hw/flp.hi3650.so
#13 pc 0000000000014984 /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom10FlpiClient25HandleFlpExtensionMessageERNS_18IpcIncomingMessageE+264)
#14 pc 000000000001e02c /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom20IpcPipeTransportBase8OnSelectEibbbPv+396)
#15 pc 000000000001e7fc /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom13SelectManager24PerformOneWaitAndProcessEv+588)
#16 pc 0000000000013e1c /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom15ipc_thread_procEPv+120)
#17 pc 00000000000694d4 /system/lib64/libc.so (_ZL15__pthread_startPv+196)
#18 pc 000000000001db38 /system/lib64/libc.so (__start_thread+16)

and:

#00 pc 000000000006c830 /system/lib64/libc.so (tgkill+8)
#01 pc 0000000000069ccc /system/lib64/libc.so (pthread_kill+64)
#02 pc 0000000000023ea0 /system/lib64/libc.so (raise+24)
#03 pc 000000000001c924 /system/lib64/libc.so (abort+52)
#04 pc 000000000043101c /system/lib64/libart.so (_ZN3art7Runtime5AbortEPKc+456)
#05 pc 00000000000e5998 /system/lib64/libart.so (_ZN3art10LogMessageD2Ev+1576)
#06 pc 000000000024c4e8 /system/lib64/libart.so (_ZN3art22IndirectReferenceTable3AddEjPNS_6mirror6ObjectE+308)
#07 pc 0000000000349af4 /system/lib64/libart.so (_ZN3art3JNI9NewStringEP7_JNIEnvPKti+628)
#08 pc 0000000000023e84 /system/lib64/libandroid_servers.so
#09 pc 0000000000029960 /vendor/lib64/hw/flp.hi3650.so
#10 pc 0000000000014984 /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom10FlpiClient25HandleFlpExtensionMessageERNS_18IpcIncomingMessageE+264)
#11 pc 000000000001e02c /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom20IpcPipeTransportBase8OnSelectEibbbPv+396)
#12 pc 000000000001e7fc /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom13SelectManager24PerformOneWaitAndProcessEv+588)
#13 pc 0000000000013e1c /vendor/lib64/hw/flp.hi3650.so (_ZN8broadcom15ipc_thread_procEPv+120)
#14 pc 00000000000694d4 /system/lib64/libc.so (_ZL15__pthread_startPv+196)
#15 pc 000000000001db38 /system/lib64/libc.so (__start_thread+16)

Again, that's not enough to understand what's going on by itself, but that's a very good place to start. At this point, I started disassembling libandroid_servers.so and flp.hi3650.so, because there are not many other ways forward really. For libandroid_servers, it was reasonably easy to associate the addresses in the stack trace to the functions they belong to, as this is vanilla code from AOSP (all the functions we're interested in belong to com_android_server_location_FlpHardwareProvider.cpp).

On the face of it, the first backtrace is not particularly helpful, because the overflow happens within LocationCallback() in FlpHardwareProvider, which is basically a function called by the vendor library that takes the location data from the HAL, transforms it into Java objects and then passes everything to the Java layer in system_server. In other words, it's a function doing all the hard work, and you can't really do anything about it allocating Java objects (and therefore references to them).

The second backtrace however proved much more useful (and seems to be more common as well). In this case, the crash happens when the vendor library calls the ReportData() callbcack in FlpHardwareProvider, which in turn allocates a new (Java) string with the C-string that is passed to it. There are two interesting things about ReportData():

Unlike LocationCallback(), it does not delete any local reference itself, which means that the reference to the allocated string hangs around for some time (probably at least until InjectDiagnosticData() is called from Java, but that's not entirely clear to me). Since "too many references to Java objects" is exactly the reason why a crash occurs, that's very noteworthy.

Even more importantly, this callback doesn't look very useful really. It implements FlpDiagnosticCallbacks.data_cb in fused_location.h. So first thing, this is a callback meant to pass diagnostic data to the FLP subsystem, which means it shouldn't be vital and we should survive without it. Second thing, FlpDiagnosticCallbacks is annotated with a comment saying "Currently, not used - but this for future extension." Even more encouraging to remove it, isn't it :)

Based on this second backtrace, what I think happens is this: under some circumstances, on some devices, the (kernel) GPS driver decides for some reason to generate a lot of diagnostic data. It sends that data to the HAL library (flp.hi3650.so), which in turn calls into ReportData() in FlpHardwareProvider (in libandroid_servers.so). This happens many times, each time allocating a new Java string and keeping a reference to it. At some point, there are so many references being kept around that the next Java object allocation fails due to a local reference table overflow. The failure may happen either in ReportData() or somewhere else (like in the first backtrace); the real problem is ReportData() being called so many times, not the specific point where the crash happens (which is just the final straw).

Of course, that's only my interpretation, but it seems to map pretty well to the observations! Now, whether that's a bug in libandroid_servers.so or just an abuse of it by the HAL, that's a matter I'll leave to the reader; the bottom line is that it doesn't really matter, because com_android_server_location_FlpHardwareProvider.cpp has been removed from AOSP/master almost a year ago (that's Treble there for you) :)

By now, what I've done should be pretty clear: I've simply stopped flp.hi3650.so from calling ReportData(), by replacing the branch (`blr x8`) at 0x29960 (see second backtrace) with nop (if you want to see by yourself, vimdiff and aarch64-*-*-objdump are your friends). That's the only change I made: replacing a branch with a nop. I could have made ReportData() a no-op instead, but I thought that modifying the vendor library instead was cleaner, and has the additional advantage of being persistent even after a ROM update (unless it includes the vendor partition, of course).