[LG Nitro HD] miji2:
This is a long post, but I need assistance trying to get to the bottom of the random reboot and Black Screen of Death (BSOD) issue on the P93x series phones. Random chatter and dismissals are getting us nowhere, we need some structured approach.

I've submitted a CM bug report, and got Issue 6762 assigned. Feel free to star and follow there as well.

General Background Info

This seems to be a polarising issue: people who are in the know (technically) have devices without this problem, and people with this problem think the sky is falling and wondering why no fixes are coming. So before you jump all over me and tell me your P93x is fine and the problem does not exist, a little background is in order. I have two P930’s, one is humming along just nicely, thank-you-very-much, the other is experiencing the random reboot issue as well as the bsod, as validated by others in this forum.

Kernel OOPS on P930

Issues

First, I believe there are three issues at play (likely separate)

Random reboots especially when scrolling. Likely a kernel panic;

and, as per a good classification here

BSOD-A: Screen remains black, hardware buttons light up and touching buttons gives haptic feedback

BSOD-B: Screen and buttons remain black, no response to power button. Phone may become hot AND battery has taken a large hit when finally rebooted.

Scenarios for the above:

Using “heavy” apps like Google Maps, Waze, etc.

scrolling on lists or typing in some apps

device goes to sleep mode, can’t wake up the LCD. Keys light up, system up and running (background apps & loggers continue to operate) .. but can’t get to them.

devices reboots (often is is unseen since device is charging) and goes dead, can’t wake up. No lights, system down.

Both BSOD-A and BSOD-B require battery pull or long press of power button to restart device.

I am focusing on BSOD-B since ATM that's the bulk of empirical data I have at hand and it's a notoriously reoccurring problem.

If I manage to capture some good logs for BSOD-A (which are very rare for me), I will submit a separate bug report.

Troubleshooting BSOD-B

Here is a list of things I tried to narrow down where the problem may be:

wipes, full wipes, cache wipes, formats, etc.

nandroid clone of the stable P930 to the unstable P930

go through the unbrick process

flash back to Gingerbread

flash back to factory ICS

disable HW overlays in Developer Options

Here is what the effect these things had:

wipes stabilize the system for about a day or two before the problem reoccurs

nandroid clone had no effect. The unstable one still reboots/bsod

Unbrick process gives temporary relief. Problem still comes back after a day or so,

Gingerbread seems to be completely stable. No reboots or BSOD’s after a week of running it and incredible battery life.

ICS introduced random reboots, but no BSOD.

that seems to take care of the reboots while scrolling.

Observations

So first obvious observation: solution d) seems to fix all the problems. Buuuut … anyways moving on.

Second observation, solution f) seems to take care of vast majority of problem #1 under scenario 1b), so IMHO it’s a viable workaround, even though disabling HW overlays increases CPU usage and reduces battery life.

Third observation, BSOD-B most frequently occurs when under power, either AC or USB.

Fourth observation, based on analysis of kmsg for BSOD-B, the battery drain is due to a kernel fault, followed by a reboot, followed by CPU0 stall (after trying to initialize the LCD?). Here’s the kmsg related to this:

[ 2.021728] lm3530_probe: i2c probe start
[ 2.022369] mipi_lgit_lcd_probe: mipi lgit lcd probe start
[ 2.022521] setting pdata->panel_info.fb_num to 3. type: 8
[ 2.022979] mdp4_dsi_vsync_init: ndx=0
[ 2.029266] FrameBuffer[0] 720x1280 size=11304960 bytes is registered successfully!
[ 62.040008] INFO: rcu_sched_state detected stalls on CPUs/tasks: { 0} (detected by 1, t=6002 jiffies)
[ 62.040344] Backtrace for cpu 1 (current):
[ 62.040435] [<c010cf9c>] (unwind_backtrace+0x0/0x12c) from [<c010b1f4>] (smp_send_all_cpu_backtrace+0x60/0xc4)
[ 62.040618] [<c010b1f4>] (smp_send_all_cpu_backtrace+0x60/0xc4) from [<c0214fe8>] (__rcu_pending+0x43c/0x5e0)
[ 62.040802] [<c0214fe8>] (__rcu_pending+0x43c/0x5e0) from [<c02152fc>] (rcu_check_callbacks+0x170/0x1e0)
[ 62.040954] [<c02152fc>] (rcu_check_callbacks+0x170/0x1e0) from [<c01cec1c>] (update_process_times+0x38/0x68)
[ 62.041107] [<c01cec1c>] (update_process_times+0x38/0x68) from [<c01f2798>] (tick_sched_timer+0x98/0x258)
[ 62.041290] [<c01f2798>] (tick_sched_timer+0x98/0x258) from [<c01e6930>] (__run_hrtimer+0x1c0/0x304)
[ 62.041381] [<c01e6930>] (__run_hrtimer+0x1c0/0x304) from [<c01e6da8>] (hrtimer_interrupt+0x168/0x2f8)
[ 62.041564] [<c01e6da8>] (hrtimer_interrupt+0x168/0x2f8) from [<c0117b54>] (msm_timer_interrupt+0x18/0x20)
[ 62.041717] [<c0117b54>] (msm_timer_interrupt+0x18/0x20) from [<c020f51c>] (handle_percpu_devid_irq+0xfc/0x1a8)
[ 62.041870] [<c020f51c>] (handle_percpu_devid_irq+0xfc/0x1a8) from [<c020bb74>] (generic_handle_irq+0x2c/0x40)
[ 62.042022] [<c020bb74>] (generic_handle_irq+0x2c/0x40) from [<c0106ea0>] (handle_IRQ+0x7c/0xc0)
[ 62.042205] [<c0106ea0>] (handle_IRQ+0x7c/0xc0) from [<c01003b4>] (gic_handle_irq+0x80/0xac)
[ 62.042297] [<c01003b4>] (gic_handle_irq+0x80/0xac) from [<c0853754>] (__irq_svc+0x54/0x80)
[ 62.042388] Exception stack(0xda65ff98 to 0xda65ffe0)
[ 62.042541] ff80: 00000020 c0c23fb8
[ 62.042633] ffa0: da65ffd8 00000000 da65e000 c0c2d5c8 c0cb4124 c086e55c 4020406a 510f02d2
[ 62.042785] ffc0: 0000001f 00000000 00000000 da65ffe0 c0106fdc c0107384 60000013 ffffffff
[ 62.042968] [<c0853754>] (__irq_svc+0x54/0x80) from [<c0107384>] (cpu_idle+0x84/0xe8)
[ 62.043121] [<c0107384>] (cpu_idle+0x84/0xe8) from [<40a4b754>] (0x40a4b754)
[ 62.043212]
[ 62.043212] sending IPI to all other CPUs:

Why battery discharge

Gist of the problem: pm8058-charger driver and related infrastructure - like USB - is not loaded yet, CPU0 is hung, CPU1 observes that its colleague is hung and faults, re-tries, and the party continues until the battery is dead.

Possible Workarounds

The only way to avoid to BSOD-B while charging is to shut down and put phone in power-off charge-mode. This is not completely viable since BSOD-B also occurs in the car when navigating or any time USB power is on.

Improving empirical data

Challenges and roadblocks

Hanging this way and so early during boot presents a problem when retrieving kmsg. The new kmsg with the CPU0 stall replaces last_kmsg on reboot thus wiping any evidence of the real kernel panic which caused this whole thing in the first place.

To make it even more fun, there is a non-zero probability that a kernel panic followed by a reboot does not hang CPU0 .. meaning you’ll never know if a kernel fault will cause CPU0 to hang until it hangs by which time it’s too late to recover the all important last_kmsg. Dead end.

For further kick in the nuts, when the CPU locks and drains the battery down to zero (or is powered off) the device turns off and boots to charge mode (since it’s under power), now wiping the CPU0 stall kmsg and stuffing last_kmsg with charge-mode messages, this time wiping evidence of the CPU0 lock described above. This can be avoided by catching the device early in the discharge cycle and unplugging the device before forcing a reboot .. but still that only gets us to read the CPU0 lock kmsg but not what caused this whole mess in the first place.

Next steps

Any ideas on how to backtrack and get better logs of the initial crash? Logcat appears useless here since it's a userspace process which gets yanked the moment a kernel oops occurs. Even piping kmsg to a file does not work due to the same problem (ie. userspace). Is there a way to tell the HW to shut down on kernel oops rather than reboot? Do I really need to disassemble these two phones in order to compare and determine differences?

Any constructive ideas, suggestions, or thoughts will be greatly appreciated.

EDIT: As of April my P930 which exhibited this behaviour is in the hands of tdmcode. This way he has something to test on and hopefully at some point he'll be able to figure out a workaround for this issue.