yashsingh1/xdaforums huggingface dataset:
[General Omni Discussion] notabenem:
I read somewhere, that enabling this will automatically disable the hotplugging of the kernel. Source: http://xdaforums.com/showpost.php?p=24233103&postcount=3

But (of course) I did not check omni-sources, if this is really the case.

I was trying to figure out, if the patch mentioned by zeitferne made it into a stable kernel used by lollipop, though I am not really sure what is the 'default' kernel version for Lollipop (if there's such). I've seen some Lollipop images based on 3.4.x, others on 3.0.xx (.4, .64, etc.).

I've also noticed mentions of "divorcing Exynos MCT from local timer API" (just because Entropy mentioned the MCT) in the kernel patches, as well as this thread:

http://www.spinics.net/lists/linux-samsung-soc/msg38069.html

Seems just far too familiar: IRQs, hotplug, MCT, bugs :) Aren't the kernel devs trying to solve the very same problem?

---------- Post added at 08:38 PM ---------- Previous post was at 08:19 PM ----------

According to this 3year old commit: 3a062281129229b50e06547af3110f8eccd2f4e4 the MCT also support PPI interrupts. Who knows if those lines in omni's kernel are still needed?
[General Omni Discussion] zeitferne:
It seems that the official Linux kernel also has a mct.c file that is similar to ours/based on the same. It has been moved multiple times, so I think it's worth posting it's Github histories here (from oldest to newest location/name):

arch/arm/mach-exynos4/mct.c

arch/arm/mach-exynos/mct.c

drivers/clocksource/exynos_mct.c

Our file seems to be based on c8987470a: ARM: EXYNOS4: Add MCT support for EXYNOS4412, although at this time the file was in the exynos4 directory, while our file is in the exynos directory (without 4). If you compare this version with the CM/OmniROM one you will notice that this Linux version seems to lack some workarounds. The very next commit might prove to be very helpful but sadly cannot be applied to the 4412 kernel because it lacks the required percpu IRQ APIs: 28af690a2: ARM: gic, local timers: use the request_percpu_irq() interface.

Meanwhile I have reverted to unmodified CM with Lanchons sdcard woraround since the YouTube-App seems to feature a new glitch w/o CONFIG_LOCAL_TIMERS: Some videos turn into a slide show (although audio is running fine and kept in sync with the "slides") with short video clips mixed in. But YouTube has always been behaving eratically on CM and I have to admit that I did not even try a restart, so I might check this again.

I also found a new option that might be worth a try: CONFIG_EXYNOS_MCT. But I will probably only have time next weekend to investigate further.

---------- Post added at 08:56 PM ---------- Previous post was at 08:54 PM ----------

@notabenem: Didn't know the linux-samsung kernel! Definitely also worth a look.
[General Omni Discussion] Entropy512:
My mention of the MCT was as a result of noticing that (much later) localtimer rework touched the MCT code. I sort of recall seeing some sort of MCT annoyances long long ago, but don't remember the details. It may even date back to Gingerbread - which is interesting since there have been claims that this issue didn't affect the Mozilla guys in Gingerbread (If I recall past discussions correctly, I could be wrong here...) I just know that long long ago I was looking at the MCT for SOME reason but I've forgotten why.

If you look in more detail: It appears that 4x12 SoCs support PPI interrupts. 4210s still use SPI interrupts. This is a likely explanation of why the issue only affects the 4210s.

Yeah. There are a lot of changes to the MCT driver, and perhaps one of the fixes is relevant. It appears that Samsung did the mach-exynos rename before mainline did. (I sort of remember that waaaay back when, around the ICS days. I remember having some discussions with codeworkx as to whether to use exynos4 or exynos - I indicated we should go with exynos as that seemed to be the future path for mainline.)
[General Omni Discussion] notabenem:
Entropy, do you know btw, what's the kernel version for the omni-lollipop builds (going to be)? Would it be worth trying a later kernel (say either from Lollipop, or the latest from Linaro) and see if it has the same problem? Just to eliminate the possibility that someone has already fixed this somewhere...

If I am speaking complete crap here (because I have absolutely no idea what would that involve) just kindly tell me :)
[General Omni Discussion] Entropy512:
The kernel we're using is the latest we have for Samsung phones. It won't be changing other than fixes/updates.
[General Omni Discussion] :
I'm using LP for a week for N7000. This bug newer showed its face yet. SD copy etc. quite stable.

GT-N7000 cihazımdan Tapatalk 2 ile gönderildi
[General Omni Discussion] clumsy:
I'm following this thread and did also some investigations, but unfortunately never touching close to the real root cause... tried messing around in the kernel source without luck... the freezes and lock-ups frequently occured...

however today I tried a standard slimkat build with CONFIG_EXYNOS_MCT unset and installed it on my i9100. until now it's stable and the usual lags and freezes are gone up to now.... looks kind of promising.... I'll report again if the build stays stable and the problem is gone (or at least supressed).

thanks for all the analysis of all the developers!!

EDIT1: fewer and less often hangs, but still not solved....
[General Omni Discussion] zeitferne:
What happens with CONFIG_LOCAL_TIMERS unset? And do you mean the usual sdcard daemon hangs?

EDIT: I added a printk to mct.c:

--- a/arch/arm/mach-exynos/mct.c
+++ b/arch/arm/mach-exynos/mct.c
@@ -388,6 +388,11 @@ static irqreturn_t exynos4_mct_tick_isr(int irq, void *dev_id)

exynos4_mct_tick_clear(mevt);

+ if (evt->cpumask != cpumask_of(smp_processor_id())) {
+ printk(KERN_WARNING "local tick on wrong CPU; handler: %pf",
+ evt->event_handler);
+ }
+
evt->event_handler(evt);

return IRQ_HANDLED;

And indeed, when the rsync copy hangs (default CM11 kernel config and sdcard daemon), I get this in kmsg:

<6>[ 196.130334] c0 cpu1 turnning off!
<4>[ 196.131583] c1 IRQ112 no longer affine to CPU1
<4>[ 196.132225] c0 local tick on wrong CPU; handler: hrtimer_interrupt
<5>[ 196.132364] c0 CPU1: shutdown
<6>[ 196.134056] c0 cpu1 off!

A few seconds [ 203.710434] later, a second time.

Here is a gist with the full kmsg and the diff.

I don't know where to go from here. Fixing that would require an understanding of the IRQ/hotplug/(local) timer subystems of the Linux kernel; maybe we have enough info now to try the ARM Linux mailing list a second time?
[General Omni Discussion] notabenem:
After reading http://lists.infradead.org/pipermail/linux-arm-kernel/2014-June/264993.html I was wondering, if it isn't indeed the mct.c that messes up the 64bit registers in the FPU. However, since clumsy is still seeing hangs with CONFIG_EXYNOS_MCT disabled, it's probably not this code.

I would encourage you to do so. And I'd also suggest to try to get in touch with the gentlemen from Samsung that submitted the patches into the samsung kernel. I mean the authors of this:

http://git.kernel.org/cgit/linux/ke.../?id=3a062281129229b50e06547af3110f8eccd2f4e4
[General Omni Discussion] Entropy512:
I wish I remembered why I was looking at the MCT a few years ago. All of the patches I'm looking through look familiar. I do recall that there were various things that had me thinking backporting the upstream changes was nearly impossible.

Do note that even when attempting to disable it in the defconfig, it may be getting force-enabled by another option.

You can't, to my knowledge, explicitly set something to "no" in the defconfig - you'll notice that any disabled options are commented out, effectively "not there".

It is interesting that some of the upstream patches seem to be related to stopping the MCT and/or at least disabling the interrupt in certain situations, like https://git.kernel.org/cgit/linux/k...c?id=e248cd5d5f1c28c82781d4122dddd1b7c3d7b46f - it might be that one of those is what we need. Problem is that I think backporting that patch could be easier said than done.

zeitferne's printk() is the smoking gun here IMO...

Edit: https://git.kernel.org/cgit/linux/k...c?id=a7fadac10ffbfd16cc7ccf951eab1ecf85e1abdf

could probably be combined with

https://git.kernel.org/cgit/linux/k...t?id=e700e41d9abfbf9fee01e979a41b185695132c19

and

https://git.kernel.org/cgit/linux/k...c?id=e248cd5d5f1c28c82781d4122dddd1b7c3d7b46f
[General Omni Discussion] clumsy:
I guess so, i coulndnn't check. various applications randomly "don't respond" anymore and get killed

tried that today, however that's even worse now. within half a day the device did not wake from sleep when power button was pressed and had to be resetted/powercycled.

I'll try again with MCT disabled and try to find out what exactly hangs when the apps freeze.

I agree with that. However it really seems to make a difference. It runs smoother and quicker and does not completely freeze (until now), so I guess undefining MCT does have an impact.... IMHO...
[General Omni Discussion] notabenem:
(almost off topic): I wish we could simply get the 3.18 kernel from Linaro (for Android) and make it work with SII. I guess all the drivers we have are completely incompatible, aren't they?
[General Omni Discussion] Entropy512:
It would be a massive undertaking - you're kind of getting into the reason why most of the original Exynos4 guys moved to other platforms.

One of Insignal or Hardkernel's latest reference board BSPs would probably be the best bet, but no one has the time for those either. (geecko attempted to backport a bunch of insignal/hardkernel kernel stuff to allow their latest HALs to be used but it didn't work out too well...)

As an example for Qualcomm devices, a typical bringup effort is something like:

Grab OEM kernel source

Based on Android version and chip used, pull a list of Qualcomm CAF release tags to check

Run a script that compares that kernel source against the list of tags and finds the one that is closest. (Note, this step and the previous one are usually unnecessary with Sony and Oppo, since they actually TELL developers the answer to this upon request)

Check out that CAF tag, drop OEM source on top of it, make a commit out of it

Split this commit into smaller "digestible" chunks that can be independently committed to a new baseline and also analyzed - https://github.com/Entropy512/kernel_find7_reference/commits/oppo_kernel for example

Take those commits, determine which ones are actually needed (some aren't and just contain fugly hacks, like https://github.com/Entropy512/kernel_find7_reference/commit/82536a234f5a2578fbb7fd46e0f9e0e7b3ee34fb ), and apply them to a newer CAF reference baseline

The problem with Exynos4 is - No production device release can be traced back to any public reference release. The end result is that the deltas against the nearest possible references (usually mainline) are MASSIVE, and splitting them up to apply to a new baseline is extremely difficult and time consuming - more effort than anyone is willing to put into these old devices any more.

I still wish I could remember why it was I was looking at MCT the last time I was looking at that code... I KNOW I've had a problem that seemed to be located here in the past but I don't remember what!
[General Omni Discussion] zeitferne:
(Probably) bad news: The arch/arm/kernel/irq.c (mainline history) in our kernel is at a4841e39f7. I merged all commits up to and including 78359cb86: ARM: CPU hotplug: ensure we migrate all IRQs off a downed CPU. This commit also makes sure that no per CPU IRQs are migrated. I then added the IRQF_PERCPU flag to the flags of mct_tick0_event_irq and mct_tick1_event_irq. Indeed my printk about the tick interrupt being called on the wrong CPU is not triggered then and "IRQ112 no longer affine" messages no longer show up in kmsg. So far so good, the problem is that the rsync test still hangs :(

In the worst case, CONFIG_LOCAL_TIMERS and/or CONFIG_EXYNOS_PM_HOTPLUG are not more directly related to the bug than FAT32 vs exFAT.
[General Omni Discussion] Entropy512:
Did you try backporting the MCT changes I linked a few posts ago?
[General Omni Discussion] zeitferne:
Sort of, but there were many dependencies, so I ended up with this git log (sorry for the bad order):

2011-07-22 82fefb4 arm: remove several unnecessary module.h include instances

2011-07-21 59ebeeb ARM: CPU hotplug: ensure we migrate all IRQs off a downed CPU

2011-07-11 34b6370 ARM: introduce handle_IRQ() not to dump exception stack

2011-07-21 14af1ec ARM: CPU hotplug: pass in proper affinity mask on IRQ migration

2011-11-16 3f8ea71 genirq: Don't allow per cpu interrupts to be suspended [from here up probably not strictly necessary]

2011-07-11 092a5f5 irq: Track the owner of irq descriptor [required for be0fede; link]

2011-10-26 de16503 irq: Fix comment typo ist->is [just for completeness]

2011-10-04 584ad45 genirq: Fix fatfinered fixup really [probably required]

2011-09-30 72243c7 genirq: percpu: allow interrupt type to be set at enable time [also required]

2014-12-11 6be37ad ARM: Exynos MCT: Correct IRQ name. [The #define for the exynos5 interrupt in mainline has a different name than in the smd4412 kernel]

2011-09-23 be0fede genirq: Add support for per-cpu dev_id interrupts [required for b929ac9, 54bbe0e, b929ac9]

2011-11-03 94ba3d8 ARM: EXYNOS4: convert MCT to percpu interrupt API [that's basically the commit you linked; see here]

2011-07-22 b929ac9 ARM: gic, local timers: use the request_percpu_irq() interface [probably required for 94ba3d8]

2014-12-11 54bbe0e ARM: gic: consolidate PPI handling [in preparation of b929ac9]

2011-07-21 d2ff782 ARM: CPU hotplug: fix abuse of irqdesc->node [probably not strictly necessary]

2011-07-21 2a08af5 ARM: GIC: avoid routing interrupts to offline CPUs [probably not strictly necessary]

See here for arch/arm/common/gic.c, here kernel/irq/manage.c

It even booted, but oopsed with a NULL pointer immediately followed by a panic as soon as I turned the screen off. Here's the last_kmsg. Either I have done something wrong while resolving conflicts, or I missed some important commits, or the changes that Samsung made to the GIC handling (whatever that is :D ) are semantically incompatible with the mainline changes. Maybe I will look at it again.

EDIT: Maybe just this commit is additionally needed?

EDIT2: A quick gist with the patches (generated by git format-patch): https://gist.github.com/anonymous/71cbf9acb5095797f79d.
[General Omni Discussion] Entropy512:
That "ARM: EXYNOS: Fix for stall in case of cpu hotplug or sleep" is the most interesting of the commits to me with respect to this whole thing. Of most interest - It's inside the local timers ifdef AND it affects 4210s but not 4412s
[General Omni Discussion] XxPixX:
Maybe this will help?

https://groups.google.com/forum/#!topic/xvisor-devel/aJXwarmjH3A

EDIT:

If the bug is related to CPU hotplugging then there have been some issues with it lately on Exynos4210:

[PATCH 1/3] clocksource: exynos_mct: Fix stall after CPU hotplugging

clocksource: Exynos_mct: Register clock event after request_irq()

clocksource: Exynos_mct: Use irq_force_affinity() in cpu bringup

irqchip: Gic: Support forced affinity setting

genirq: Allow forcing cpu affinity of interrupts
[General Omni Discussion] zeitferne:
OK, after adding this commit on top of the others, the system runs fine now, but the rsync test still fails with a hanging sdcard daemon.

---------- Post added at 01:35 PM ---------- Previous post was at 12:38 PM ----------

I added a "return IRQ_HANDLED;" after my printk, but the sdcard daemon still hung. The second commit you linked is the one Entropy512 just mentioned. All other commits, especially "hclocksource: Exynos_mct: Register clock event after request_irq()" indeed sound like they fix the problem that the IRQ is called on the wrong CPU, but there are huge changes in between, so it would be hard to apply these commits. Additionally I'm not even sure now if that is really the root cause of the FPU register corruption.
[General Omni Discussion] Entropy512:
Yeah, there seems to be more and more evidence that says the MCT isn't the root cause. It could be that disabling CONFIG_LOCAL_TIMERS just alters the timing of things enough to mask the issue, just like various other things (such as keeping the screen on) mask the issue. The various MCT funkiness could just be unrelated minor (potentially just cosmetic) issues...

However, it seems like hotplug is definitely connected to this, and it IS one thing that explains how the registers could get corrupted. If some task is running on core1, then core1 plugs out and the task migrates to core0 - that might explain the issue. (would be interesting to see if the sdcard process is migrating cores when the issue occurs?)

Which could potentially also explain why using exfat (or any other FUSE-based FS) for the external SD masks the issue. This results in an additional userspace process being awake when reading data, which could potentially be enough additional load to prevent core1 from getting unplugged during the read.
[General Omni Discussion] Lanchon:
FPBug detector

this detector runs 4 parallel processes that randomly sleep and check for corruption in all FP regs. sleeps are 0 to 250ms in a uniform-squared distribution (favors short waits). initial reg values depend on reg number and process number to detect process crosstalk and random corruption. each process outputs changed regs and dies upon detecting corruption. this detector can be run in non-rooted phones.

there is no proper cleanup of processes by the app, so after EACH invocation you should reboot, because: a) your battery will die soon if you don't, and 2) there will be more processes running, crosstalk among them might go undetected, and debug output in the event of a detection might be confusing.

the app is not a service, so it MUST remain in foreground once started. it is most probably required to turn off the screen for the bug to be triggered, so run the app and turn off the screen.

if the bug doesn't show up, you are welcome to try concurrent MTP transfers, taring/untaring stuff via adb shell, or whatever you like as long as the app remains in foreground.

(source included)

EDIT: i've added 2-process and 1-process versions of the app. the original 4-process version failed to detect corruption in the one instance in which it was tested so far (see below). rationale: maybe the 4-process version doesn't let the system turn off a core and the others do. note that more than one process is required to detect crosstalk.

NOTE: if ALL processes die after a run that detects corruption, YOU DONT NEED TO REBOOT to run the test again (or continue using the phone). this is the only exception to the reboot rule.
[General Omni Discussion] Lanchon:
fyi: this is how a simulated corruption is displayed
[General Omni Discussion] drdays:
Thanks. Can the FPBug detector be run when the sdcardfix has been flashed on the system?
[General Omni Discussion] Lanchon:
yes, it wont be affected by the sdcard fix.
[General Omni Discussion] drdays:
My setup is 13th December nightly of CM11. Internal sd is vfat formatted and external sd is fat32 formatted.

I copied 200mb worth of pictures from internal sd to pc and 1gb of documents (pdf and chm) to external sd via mtp.

FPBug was running in the foreground with screen off. Everything seemed to go smoothly. Only thing is I can't access settings section of FPBug.
[General Omni Discussion] Lanchon:
lol there are no settings in the app.

so i guess your device gets the bug without the fix right? can you confirm that? that'd show that this doesnt work.

if so i should reduce the number of processes to 2 and see what happens.
[General Omni Discussion] Lanchon:
i've added 2-process and 1-process versions of the detector app (edit to original post). maybe the 4-process version doesn't let the system turn off a core and the others do. note that more than one process is required to detect crosstalk.
[General Omni Discussion] zeitferne:
If I read the source correctly, the processes sleep in every iteration of the loop. Maybe you should include some random CPU load, e.g. busy waiting in 1% of all iterations, to make sure you wake up the second core.
[General Omni Discussion] Lanchon:
not at this point. you can wake up all the cores you want via adb. music bug happens with the phone almost off, while the device waits for an interrupt from the eMMC.

at this point i need people with susceptible devices to TEST!!! and be creative in that.
[General Omni Discussion] cdrivex4:
Fpbug2.apk results

Sent an apk through BT,played jetaudio plus when plugged in. installed apk..Skipped a few tracks..unplugged charger.. Skipped some more tracks while awake, pressed power button to sleep.skipped some more tracks... Then it stopped after a few songs cm11 2014 12 16.. after wake could only quickly take screen shots before I had to force restart.
[General Omni Discussion] jnetode:
Here follows my capture.

I just let it run and set screen off.

Let it lay a little and as I picked it up, there was the result.

I am running Markox Linaro build with Lanchon trim kernel and music bug fix.
[General Omni Discussion] Lanchon:
BINGO!!!!!! we have a detector working!!! :) thanks!

how long did you run the test? with screen off? did you do anything else via adb? did you try the 4 and 1 process versions? did both processes die at the same time? (if not, the 1 process version would probably be enough.)

please keep the screenshots coming to confirm the results. if this checks out:

there is clearly no crosstalk of any kind between processes and there's random corruption, so my hunch was right. crosstalk would have meant the linux context switch machinery was failing. but we know corruption happens when cycling through a low power state, and the low power state would reset the core and isolate the state of the two processes, ruling out crosstalk.

all registers are affected in the same manner: all bits mostly set. this is very compatible with the state you'd find in initialized registers after a powerup. going back to my previous post:

http://xdaforums.com/showpost.php?p=57433979&postcount=710

we should start proposing scenarios and testing them.

of those in my post, #3 and #2 are possible, with #1 being very improbable i think.

---------- Post added at 03:41 PM ---------- Previous post was at 03:39 PM ----------

ok thanks! so we have confirmation.

did you try the 1 and 4 process versions? how long does it take to detect the bug?

---------- Post added at 03:45 PM ---------- Previous post was at 03:41 PM ----------

NOTE: if ALL processes die after a run that detects corruption, YOU DONT NEED TO REBOOT to run the test again (or continue using the phone). this is the only exception to the reboot rule.
[General Omni Discussion] jnetode:
I tried only 2 process versions. It took about 5 minutes between the time I turned screen off and on.

I will try the other versions too.
[General Omni Discussion] jnetode:
1p and 4p both have it.

Left 3 minutes sleep.

All processes stopped.
[General Omni Discussion] cdrivex4:
Only installed apk2, Ran it again as realised I hadn't kept the app at the fore, first time I did taskswitcher to jetaudio plus and back to test app once or maybe twice, so ran it again to the letter. Just checked it again from boot. Left it for about an hour before I checked again..could be less time as was watching a film(uptime). Interestingly as I tried to recreate previous actions of my post the outcome was the generated error though jetaudio was still playing this time.

Only tried 2nd apk not tried the others...yes pressed power button to turn the screen off, tough running your app after install an nt through adb kept the phone awake, it seems only p2 is generating corruption when I re-did it..

Will a kmsg file be of any use to you? I remember reading somewhere you need that an a logcat, though don't have logcat, I copied out the kmsg after boot with es file explorer as I was forced to reboot due to phone becoming un-responsive.