[Epic 4G Android Development] mkasick:
4/20/11 Update: Added a second workaround for the libsecgps synchronization bug that otherwise allows multiple GPS requests to be sent to the RIL, causing the GPS to crash on assistance downloads (via GPS Status & Toolbox) and on application switch/exit. Details on this and the assistance workarounds, which are complementary to each other, follow below.

To folks with GPS problems:

If your GPS reliably works after a reboot, but ceases to work after appreciable uptime, it's quite likely that one or both of these workarounds will address the issue. Try the crash workaround first, as all EC05 Epics are vulnerable to the libsecgps synchronization bug, whereby the GPS crashes and is only recoverable by a full power cycle. If you continue to have issues, the assistance workaround may further help by keeping your assistance data fresh.

If your GPS rarely or never works in EC05, even outdoors with clear skies, even after a reboot, it's quite possible the assistance workaround will help. But please post in the thread or PM me first, as I'm interested in obtaining a boot-time logcat that would confirm the exact issue you're suffering.

Summary of issues discovered and/or resolved in this thread:

NTP/XTRA data doesn't download/inject on boot (jdelano; addressed by assistance workaround).

Synchronization bug with RIL that causes the GPS to crash (slwelsh; addressed by crash workaround).

Stale ephemeris (from XTRA data) after a few hours to a day of uptime. Possibly addressed by assistance workaround, needs confirmation (aero1, buggerritt, slwelsh?).

Time until ephemeris becomes stale differs across devices. Unknown cause, although a separate workaround exists to improve this to a day or longer (buggerritt).

4/11/11 Update: Fixed the update.zips so they actually flash in ClockworkMod. Apparently /system needs to be mounted, which it already is in stock recovery.

GPS assistance workaround:

Attached is a platform source patch against android-cts-2.2_r2 that, hopefully, works-around most assistance-related EC05 GPS problems by modifying the GPS location provider to download & inject time (NTP) and assistance (XTRA) data every time the "Use GPS satellites" setting is enabled. Also attached is a smali patch against EC05's /system/framework/framework.odex that implements this workaround in bytecode.

Finally, attached are testkey-recovery (e.g., ClockworkMod) flashable "update.zip"s including modified a modified framework.odex or framework.jar for: (i) the EC05 stock ROM, (ii) deodexed (framework-unmodified) ROMs, (iii) SyndicateROM Frozen 1.1.0 and (iv) midNIGHT ROM 5.2. All three implement the GPS workaround, and the first two also contain the TWS bug fix and ringer volume control mod. I'll try to take requests for additional ROMs as I can.

Assistance workaround directions:

Once flashed, keep the "Use GPS satellites" setting/notification widget toggle off until you're ready to use GPS. While having (any) active data connection, toggle GPS on and wait about five seconds before attempting a GPS fix.

Alternatively, if you always keep the GPS toggle on, toggle it off & on again to download new assistance data when you have difficulty getting a fix.

Assistance background:

In Froyo, the Epic GPS works in "standalone mode" with various types of supplemental assitance data pushed ("injected") by the GPS location provider if they're available. This includes:

XTRA (ephemeris & almanac) data: Downloaded and pushed automatically on reboot after a network connection is established.

NTP time: Nominally pushed after reboot and every four hours thereafter.

Location: Location data from other providers (e.g., cellular, wifi, etc.).

The XTRA ephemeris data, combined with a rough estimate of current location and an accurate (NTP-synced) clock, provides the GPS receiver with precise orbital positions of each satellite, allowing the receiver to quickly lock onto them even in weak signal (e.g., indoor) conditions or in the present of multipath (e.g., high-rise buildings). The gpsonextra servers appear to update XTRA data every hour at half-past. It's ephemeris data is (to the best I can discern) supposed to be valid for one week, although it's speculated that the Epic GPS may consider this data stale after a much shorter period of time, perhaps a few days, and in some cases, no more than a few hours.

The almanac data provides the GPS receiver with coarse orbital parameters of each satellite, allowing the receiver to locate the satellites reasonably quickly when outdoors with clear sky visibility. However, in absence of recent ephemeris (e.g., from XTRA or a previous fix), the receiver must download the current ephemeris from each satellite. Satellites broadcast their own ephemeris every 30 seconds, and thus, the receiver requires 30-60 seconds of strong signal from each satellite in order to obtain its ephemeris and lock onto it. The ephemeris broadcast from the satellite is valid for five hours, much less than that of the XTRA data. Each satellite also broadcasts the almanac data for the entire constellation every 12.5

minutes, and this data is valid for a few months.

This all means that, an Epic with recent XTRA data, NTP sync, and location estimate should be able to lock onto any visible satellite quickly (less than 10 seconds), possibly even indoors, whereas an Epic with stale ephemeris will fall back to using almanac data and should fix within a minute outdoors with clear sky visibility. Thus, Epics that take a long time to fix or can't fix indoor likely suffer from stale ephemeris. However, as long as these phones retain a GPS fix for at least fifteen minutes once every month or so, their almanac data will be fresh enough to operate indefinitely in this degraded mode.

Assistance workaround details:

XTRA data is downloaded automatically on boot, but it isn't downloaded again unless requested by the GPS receiver. Although this data is supposed to be valid for one week, folks report that their Epics take a long time to fix (30+ seconds) after a day, or even just a few hours of uptime, suggesting that the ephemeris downloaded at boot data becomes stale.

I don't know the root cause of why this data becomes stale, although it seems that temporarily downgrading and obtaining fixes in DI18 will improve the amount of time the ephemeris is considered valid to a day or longer in instances where it's only good for a few hours.

The idea behind this workaround is to exploit a simple heuristic: by activating the GPS toggle the user intends to use GPS for a while, so that's (probably) the best time to download new assistance data. Even if the toggle is left on all the time, it's an intuitive means to "reset" the GPS that most users will try, so it should download new assistance data every time. So while this might not directly fix the stale-ephemeris issue, hopefully this workaround will make the current and future assistance problems less relevant since they're easily bypassed.

A second issue is that NTP time and/or XTRA data often fails to be acquired on boot. The GPS location provider waits until there's an available network connection before connecting to the NTP and XTRA servers. However the "network available" notification doesn't quite work right, and the initial NTP request may happen before DNS resolvers are registered and the NTP request fails hostname lookup. While I never observed a failed XTRA download on my phone due to lookup failure, this seems to be the case with others. If you have trouble acquiring a fix even after a reboot, failed XTRA downloads may be to blame.

Since this workaround doesn't attempt to do NTP/XTRA until the GPS is enabled, this should no longer be an issue. However, to ensure it works in most instances when the GPS is left enabled across a reboot, I've added a network-notification settle delay of 5 seconds, which should be long enough (2.5 seconds is the longest it took my phone to settle) that NTP/XTRA will work on boot in that instance.

GPS crash workaround:

Attached is a platform source patch against android-cts-2.2_r2's liblog that avoids the GPS crash that happens when multiple GPS requests are sent via libsecgps to the RIL, which occurs during assistance downloads (via GPS Status & Toolbox) and on application switch/exit.

Also attached is a testkey-recovery (e.g., ClockworkMod) flashable "update.zip"s including a patched liblog.so for all EC05 (stock, deodexed, and custom) ROMs. I've also attached a "backout" update.zip that replaces liblog.so with the original EC05 version, should anyone wish to revert to it.

Crash workaround directions:

None, it works automatically once flashed.

Crash background:

The Epic's GPS receiver is integrated into the baseband, and communication with it is done via the Radio Interface Layer (RIL). The native library, libsecgps.so, implements various GPS-oriented commands (open, stop, location_fix, inject_time, xtra_set_data, and others) as requests, via the RIL, to be performed by the baseband processor and GPS receiver.

When a GPS app runs, location_fix requests are issued (synchronously) by a native library timer once every second while the app is actively seeking fixes. When a GPS app is closed or suspended, a stop request is issued to the GPS to cease fixing on satellites. Regarding these stop requests, it's important to note that: stops are often asynchronous, being the result of user action; and there appears to be no locking to ensure that a stop request is performed outside a location_fix cycle.

That is, a stop may come: (i) just prior to a location_fix request being issued, (ii) just after a request is issued but before it's sent to the baseband, (iii) after the baseband receives the request and performs the fix, and (iv) after the resulting location data is sent back (which completes a location_fix cycle).

A stop performed outside a location_fix cycle (iv) is most preferable, but due to the length of a location_fix cycle (~0.7 s) this happens only a third of the time. Most often, a stop if performed while a fix is actively being performed (iii) which results in a harmless recoverable error. However, if a stop is performed just before a location_fix is issued (i) or just after (ii), before the baseband can fully receive it, then the two requests may collide, throwing the baseband-side GPS functionality into an unrecoverable error state that is resolved only by a power cycle.

The critical window here is approximately 60 ms wide, meaning that the GPS crashes with 5% probability during a stop, or any other asynchronous request that occurs while the GPS is actively fixing. In other words, on average, every 20 GPS app closes/suspsends results in a GPS crash. For folks who use the GPS once or twice the day during a long navigation session, a crash may rarely be observed. For folks who use various GPS-utilizing apps throughout the day, crashes happen regularly after a day or two of uptime on average.

It's also worth noting that an XTRA download issues 40 xtra_set_data requests in a short window of time (~7 s). While these are safe outside an active fix (e.g., on boot or GPS toggle w/assistance workaround), if an assistance download is performed during active GPS use, e.g., via GPS Status & Toolbox, the GPS almost always crashes and the download fails. This is actually why GPS Status's assistance features appear not to work in Froyo, although with the crash workaround, I believe they do.

GPS workaround details:

Ultimately the GPS crash is due to insufficient locking in libsecgps that enables the interleaving of simultaneous requests to the RIL. Although we've observed instances where libsecgps appears to defer requests, it's locking behavior is insufficient to prevent interleaving in many instances.

For illustrative purposes, here's an excerpt of the log statements produced by libsecgps when properly processing a location_fix request:

...
D/libgps ( 2931): send_ril_gps_location_fix_req: start_mode 1
D/libgps ( 2931): newGpsRequest: new gps req 0x4303c8 cmd 5, idx 1
D/libgps ( 2931): send_ril_gps_request: cmd 0x05 plen 18
D/libgps ( 2931): ril_gps_wait_for_ril: req 0x4303c8 wait 1000
D/libgps ( 2931): ril_gps_wait_for_ril: +++entering
D/libgps ( 2931): ril_gps_wait_for_ril: current tv_sec 1302648124 tv_nsec 519263548
D/libgps ( 2931): ril_gps_wait_for_ril: timeout tv_sec 1302648125 tv_nsec 519263548
D/libgps ( 2931): onRawReqComplete: oem_function_id 0x0E cmd 0x05 len 4
D/libgps ( 2931): process_response: req 0x4303c8 cmd 0x05 plen 1
D/libgps ( 2931): recv_ril_gps_location_fix_req: status 0
D/libgps ( 2931): ril_gps_wakeup: 0x4303c8
D/libgps ( 2931): ril_gps_wakeup: Entered critsec
D/libgps ( 2931): ril_gps_wakeup: Left critsec
D/libgps ( 2931): ril_gps_wait_for_ril: (n=0, r=0)
D/libgps ( 2931): removeGpsRequest : removed req with cmd 5
...

The newGpsRequest function is called by the issuing thread for each request. Requests are then sent to the RIL (send_ril_gps_request) and the issuing thread waits on a condition varible (ril_gps_wait_for_ril). The RIL wakes up (ril_gps_wakeup), sends the request to the baseband, and signals the issuing thread to wake again. The issuing thread then deletes the request (removeGpsRequest).

For comparison, here's an excerpt of the log when a stop request interleaves a location_fix:

...
D/libgps ( 2931): send_ril_gps_stop_req:
D/libgps ( 2931): newGpsRequest: new gps req 0x3900f8 cmd 6, idx 2
D/libgps ( 2931): send_ril_gps_location_fix_req: start_mode 1
D/libgps ( 2931): newGpsRequest: new gps req 0x390958 cmd 5, idx 1
D/libgps ( 2931): send_ril_gps_request: cmd 0x05 plen 18
D/libgps ( 2931): ril_gps_wait_for_ril: req 0x390958 wait 1000
D/libgps ( 2931): ril_gps_wait_for_ril: req 0x3900f8 wait 1000
D/libgps ( 2931): ril_gps_wait_for_ril: +++entering
D/libgps ( 2931): ril_gps_wait_for_ril: +++entering
D/libgps ( 2931): ril_gps_wait_for_ril: current tv_sec 1302652321 tv_nsec 191173085
D/libgps ( 2931): ril_gps_wait_for_ril: timeout tv_sec 1302652322 tv_nsec 191173085
D/libgps ( 2931): ril_gps_wait_for_ril: current tv_sec 1302652321 tv_nsec 210846213
D/libgps ( 2931): ril_gps_wait_for_ril: timeout tv_sec 1302652322 tv_nsec 210846213
D/libgps ( 2931): ril_gps_wait_for_ril: (n=110, r=0)
E/libgps ( 2931): ril_gps_wait_for_ril: pthread_cond_timedwait error. (n=110, r=0)
E/libgps ( 2931): RIL Request: ERROR 1 (n=110, r=0)
D/libgps ( 2931): removeGpsRequest : removed req with cmd 6
D/libgps ( 2931): ril_gps_wait_for_ril: (n=110, r=0)
E/libgps ( 2931): ril_gps_wait_for_ril: pthread_cond_timedwait error. (n=110, r=0)
E/libgps ( 2931): RIL Request: ERROR 1 (n=110, r=0)
D/libgps ( 2931): removeGpsRequest : removed req with cmd 5
...

This time, a second newGpsRequest (location_fix) appears before the first (stop) request completes. Both threads appear to send their requests to the RIL, but the RIL never wakes up (no ril_gps_wakeup). Although it likely does, sees invalid data in its request buffer, and dies or something. Meanwhile the two issuing threads wait on the condition variable, which is never signaled. Eventually the condition variable times out and the request fail. At this point, the GPS has effectively crashed, as any subsequent GPS request to the RIL fails with the same error.

Fixing this synchronization bug involves either repairing the existing libsecgps locking logic, or adding new logic to prevent the interleaving of two or more requests. Unfortunately it's quite difficult to diagnose, repair, patch, and subsequently maintain said patch for source-code-less native libraries.

However, as libsecgps frequently makes use of logging statements, as illustrated above, during the processing of GPS requests, we can instrument the logging library to hijack the log calls and provide the necessary synchronization. Since libsecgps uses the logging routines provided by liblog, a library unmodified-by-Samsung for which we have source code, this is where the workaround code is placed.

Specifically, this workaround modifies liblog's __android_log_print's to look for logging messages made by libsecgps (libgps). Each newGpsRequest log statement signifies the creation of a request. When a second request (newGpsRequest statement) is encountered before an outstanding request completes, liblog defers the request by forcing the issuing thread to wait on a condition variable that's later signaled by the completion of RIL receiving the first request (ril_gps_wakeup). This prevents two requests from interleaving, and provides the necessary synchronization to prevent a GPS crash.

Final notes:

At this point the assistance workaround has been tested by multiple parties, some of whom have reported an improvement in GPS fix times, and none of whom have reported side-effects or adverse behavior.

The crash workaround has been tested by myself and slwelsh for three days now without either of us experiencing a crash or side-effects. At this point, I invite others to test the crash workaround, particularly if you experience them on a frequent basis as well.

As usual, please keep a copy of Odin and a stock/custom ROM tar file on hand, just in case something silly happens and your phone is bricked.

Also, while we've had a lot of feedback on folks's GPS issues already in this thread, I'd still appreciate feedback on whether these workarounds help you, or if you have a different GPS-related problem that you believe is not yet addressed by these.

Assistance workaround (mirror links, do not require forum login):

platform_gps_xtra_on_enable.diff

platform_gps_xtra_on_enable_smali.diff

tws_fix_ringer_vib_silent_gps-EC05-odex.zip

tws_fix_ringer_vib_silent_gps-EC05-deodex.zip

gps_xtra_on_enable-SRF110.zip

gps_xtra_on_enable-midNIGHT52.zip

Crash workaround (mirror links, do not require forum login):

platform_gps_request_sync.diff

gps_request_sync-EC05.zip

orig_liblog-ECO5.zip (to backout the crash workarond)