yashsingh1/xdaforums huggingface dataset:
[General Omni Discussion] Entropy512:
Well, other than it specifically being the FP registers, we've known for a while it was a register-clobbering issue. Someone else (CGX I think?) already tried a different approach to this, by making the variables in question array members, which causes the compiler to put them into RAM instead of sticking them in registers. It did successfully work around this issue.

I THINK, but I'm not sure, that CGX already had some sort of evidence pointing to modem ISRs as being the ones doing the clobbering.

Which might explain why this only happens on 4210s - one of the items that had to be merged into the 4412 kernel was a few additional modem drivers, so there's a possibility one of these was mismerged.

And I'm positive this clobbering issue is occurring elsewhere - it's just FAR more obvious in this case when it happens because the failure happens to be unrecoverable without manual intervention (such as killing the sdcard daemon).

That's why I've been hoping someone might take a crack at bringing the old smdk4210 kernel tree up to KitKat standards - if that results in a reliable system, it points to a mismerge.
[General Omni Discussion] Dasmikko:
Have played music for over 3 hours now, and no hangs!! :)
[General Omni Discussion] D4rK.F0rcE:
I had the same or a similar experience in airplane mode. In my case, the music also stopped playing, but only after a long time (around 2-3 hours). There is definitely an improvement in airplane mode!

Another observation: music stops even if no external sdcard is inserted! This bug is not only affecting the ext. sd, but also the internal sd!
[General Omni Discussion] theBIGone:
I can second that. The music stops even if no external sdcard is inserted.
[General Omni Discussion] bleggy:
maybe of interest:

https://github.com/omnirom/android_...mmit/5e6d7f417692886d0877e941d5a855f90d6e1123

https://gerrit.slimroms.net/#/c/3557/
[General Omni Discussion] cgx:
sorry, mistake

delete
[General Omni Discussion] GidiK:
See: https://gerrit.omnirom.org/#/c/5535/
[General Omni Discussion] lerkin:
Great news, hope it will be merged and that nasty bug will be closed finally.
[General Omni Discussion] DJG0A:
Sorry guys, if you read carefully the Change, it's not going to end yet:
[General Omni Discussion] lerkin:
Yep, hope solution will be found.
[General Omni Discussion] htmaxxt:
RE:

Good day!

I would like to add some information about this problem: it arises only when player playing music from the SDcard through connected headphones. If to use a bluetooth stereo headset, the problem disappears and the music plays long. Maybe this information will be useful.
[General Omni Discussion] skey8626:
Same thing happens to me, I just thought maybe its a problem with my music files and I did a reset but when I added another songs,the problem prevailed.Used both Google Play Music and Apollo still the problems exists.
[General Omni Discussion] glhfggnore:
Hi!

Definitely maybe, but myself have noticed music stop even on loudspeaker play. The problem occurs random, I don't have a clue yet on how to predict it. Did you test music play long enough through your Bluetooth headset?

PS. The problem occurs to start randomly. Maybe we can help find the bug reason by noticing our actions. I tried to change governor IO to ondemand / noop and read ahead size to 2560 Kb. Didn't help.
[General Omni Discussion] glhfggnore:
Maybe, a solution

Hi again my fellows,

I think a solution for the problem is made already for SGS2 owners.

[MENTION=2376110]dorimanx[/MENTION] and [MENTION=4663860]Computoncio[/MENTION] merged GitHub https://gerrit.slimroms.net/#/c/3557/, https://gerrit.slimroms.net/#/c/3676/ by [MENTION=4661001]AuSSiELaMbChOpS[/MENTION] for fixing fuse daemon bug in DorimanX kernel beta build 10.43 beta 75 -> http://xdaforums.com/showpost.php?p=50374557&postcount=7124

I must admit that it is a beta test kernel. It can be unstable. For i9100 4.4.2 mali v3 you must use 10.43 version. Beta thread is here http://xdaforums.com/showthread.php?t=2415759

I tried it and have my music playing by the time I write this post about 3 hours without any stops. Before flashing this I had music stop every time as early as from 3 to 15 song play.
[General Omni Discussion] Entropy512:
Unfortunately, it isn't too useful. It just provides more evidence that this is a weird timing issue. (I mentioned a while ago that when Chainfire tried to add debugging to the fuse daemon to figure out where it was hanging, the problem went away. That just masks the issue and doesn't solve the root cause.

The first patch from slim gerrit won't fix this issue. The second one (to fuse sdcard.c) might possibly have some effect.

I'll take a look at the latter... Although that should've been done via cherry-picks from googlesource preserving commit authorship.
[General Omni Discussion] lerkin:
Tested latest cm11 and that dorimanx kernel... Still freezes on copy operations.
[General Omni Discussion] glhfggnore:
I agree with you, the new fuse fix might not be full fix for the problem.

When I was testing kernel for the first time I set gov to ondemand, io scheduler to noop and read ahead size to 2560 kbytes in stweaks. Music played for more than 4 hours. After that I set read ahead size to auto. Then I encountered too some freezes of my music player Neutron while testing. But this time it is enough to force close the application and open it again, it will continue play. I don't know though, whether is it related to file copy operation freezes. Maybe, manual setting of read ahead size would help

As much as I understand he didn't push sdcard.c commit to kernel, he pushed init.smdk4210.rc. http://xdaforums.com/showpost.php?p=50329848&postcount=7039

I never had to explain myself on open source foreign developers forums please excuse me if I mix some things up while explaining github commits.

Edit: and yes, changes are made from the first patch, 3557 gerrit, to init.smdk4210.rc, changed parameter is -w 1023 https://gerrit.slimroms.net/#/c/3557/1/rootdir/init.smdk4210.rc

I will read a commentary on why this github is considered doesn't fix an error. It works for me. But need more testing though.
[General Omni Discussion] lerkin:
I tried to change read ahead buffer size to something bigger, changed governors and schedulers, but still - file operations hang.
[General Omni Discussion] glhfggnore:
Damn. I was excited this would finally stop and fix the bug, but it doesn't. Although it is good that I don't have to reboot my phone every time now after music stop and music stops now less frequently - so far it is better than nothing for me.
[General Omni Discussion] lerkin:
It is still a nastiest bug for me, dropsync hangs and causes wakelock, titanium backup can hang on loading list of backups...
[General Omni Discussion] glhfggnore:
Yes I encountered file copying problems too when trying to copy files from phone to PC via usb cord. Only airport wifi app helped, so I use it now. And backup only to cloud services and int memory. Android bugs are what forces you to change your own behavior models :D
[General Omni Discussion] lerkin:
I am still on cm 10.2 with anti memory leak measures. In 4.4 that leak was plugged, that is why I am waiting for working 4.4 so much.
[General Omni Discussion] nex-54:
Thanks for patch and replys!

Just flash it in recovery and will test soon.

Will it fix hangs on internal memory? I have internal memory in fat32, external in exFat. and have hangs on both storages.
[General Omni Discussion] Lanchon:
it was CGX who did the array thing for that particular unique value, but nobody knew if/why that really worked (specially since the manifestation of the bug is so sentsitive to everything), or why unique in particular was being corrupted and not other things. IIRC that hack failed for some people, which is not surprising since many instances of long ints are used in sdcard.c. GidiK even 'cured' the bug by simply moving structures around, but of course that doesn't really replicate for all people.

until now i had assumed that there was no register corruption but stack corruption. thought unique was being spilled by the caller or saved by the callee to the stack (not to FP registers) and corrupted there. though manual spilling like CGX's hack just changed the game a bit because it was later shown to fail too, just less often. so i put up a stack corruption detector here and discussed workloads to trigger the corruption, and nobody said i should be looking elsewhere! in the end it all turned out to be much simpler.

i didn't see any evidence that points specifically to modem ISR in this thread. only this:

http://xdaforums.com/showthread.php?p=52846763&highlight=modem#post52846763

but it seems to be you misreading the previous post. :) or maybe you got the info from somewhere else?

you suspected fixes in new modem drivers, but people confirmed the bug was in dorimanx too:

http://xdaforums.com/showthread.php?p=52024702

regarding clobbering occurring elsewhere but being notorious in sdcard.c, absolutely, ive said exactly that here before.

instead of trying to bring up the old 4210 kernel which is a lot of work, we can make an FP corruption detector and run it under old CMs and stock roms and see whos clean and whos dirty. this is trivial now that we know what to look for, but i dont have a device so i cant do this.

so no ideas about ISR instrumentation, how to find the most recently executed ISR, stuff like that?

---------- Post added at 01:40 PM ---------- Previous post was at 01:39 PM ----------

should
[General Omni Discussion] zeitferne:
Knowing that this are the FP registers is very valuable though: These are not saved by the Kernel's IRQ handling machinery but must be saved by the handlers themselves. This article might be interesting regarding this.

I wonder if it is possible that gcc makes the kernel use the FP registers in the same way as in the sdcard daemon, thus resulting in the clobbering. Maybe there is a problem with the kernel compiler flags?

---------- Post added at 04:57 PM ---------- Previous post was at 04:44 PM ----------

The only thing I know is /proc/interrupts/ which would only be sufficient if the clobbering interrupt is one that only occurs very seldom. But this particular debugging issue is not 4210 or even Android specific so I guess one could make a new thread for this question in an XDA subforum where the right people will see it, or even ask it on Stackoverflow.
[General Omni Discussion] Lanchon:
hell no! gcc is configured to compile the kernel without touching the FP regs or all hell would break loose. that is what i did to sdcard.c btw.

thanks on the proc/ints pointer, i'll take a look later.

---------- Post added at 02:22 PM ---------- Previous post was at 02:15 PM ----------

what is the difference between

-modem_if

-modem_if_u1

-modem_if_na

(specially the first two) isnt U1 the name of this platform?

for interrupt instrumentation from scratch, this looks like a good place to start:

http://lxr.free-electrons.com/source/include/linux/interrupt.h#L128

unfortunately without a device to test im very limited.

it would help me to know the kernel too
[General Omni Discussion] zeitferne:
Since we know that d8 is the corrupted register, I think that a search through the kernel disassembly for that could be helpful.

arm-eabi-objdump -SdCg -marm <object-file>

should be the command for that (source) (a version of arm-eabi-objdump is locatd at prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/bin/arm-eabi-objdump, at least in the CyanogenMod build tree). In obj/KERNEL_OBJ/arch/arm/kernel/*.o I found only three uses of this register (is that even normal?), so this promises to be manageable. You have to filter out other occurrences of this string though, e.g. as an address.
[General Omni Discussion] Lanchon:
an interesting tidbit i just read...

Because...

* many programs don't use floating point or don't use it on any given time slice; and

* saving the FPU registers and other FPU state takes time; therefore

...an OS kernel may simply turn the FPU off. Presto, no state to save and restore, and therefore faster context-switching.

If a program attempts an FPU op, the program will trap into the kernel, the kernel will turn the FPU on, restore any saved state that may already exist, and then return to re-execute the FPU op. As long as this doesn't happen most of the time it's an overall win in reduced kernel mode operations.

At context switch time, it knows to actually go through the state save logic. (And then it turns the FPU off again.)​

so...

1) using softfp in an integer only program can be counter productive: the compiler has FPU regs to spill on, but then context switches become more expensive.

2) a context switch doesnt always need to save or restore the FPU regs. (here's another chance for a hardware-dependent bug: some SoC hardware is enlisted to track FPU usage and it malfunctions. but this hardware probably involves only the cortex core (traps, etc) and no peripherals, and if arm got it wrong at some point in time, it should be well documented by now.)

3) an ISR doesnt always need to save the FPU regs before using them (not that an ISR should ever use them! but anyway).

4) most importantly, an ISR might interrupt a process running in the FPU-disabled state and (unless arm handles this specially) it will receive the FPU in that state. which means that if sloppy code in an ISR inadvertently uses the FPU (say, to spill regs), it could trap. would the enable-and-return trick work in the context of an ISR? if not, would the kernel panic? if so, it'd be clear that *no ISR is touching the FPU at all* and corruption does not come from ISRs (because we dont see the kernel panicking).
[General Omni Discussion] zeitferne:
For the i9100, from searching (find -name '*modem*') the ouput tree, it seems that only drivers/misc/modem_if_u1 is compiled.
[General Omni Discussion] Lanchon:
thank you!! (run out of thanks :-S )
[General Omni Discussion] patrickph:
Great work, thank you.....

Is this a gcc bug or hardware bug? 4.3 also has this bug ?
[General Omni Discussion] nex-54:
Test it for one day and everything works fine!

No more hangs in music player and no hangs on big data copy.

Thanks!!!
[General Omni Discussion] Entropy512:
It is most likely a kernel bug.

1) It only happens on the Exynos 4210-based Galaxy S2 family devices, but nearly all devices use the same GCC version. Most importantly, it does not happen on Exynos 4412 devices (same kernel source) - but keep in mind that it may not be the 4210 itself, but potentially something hung off of it.

2) It does not seem to affect stock Samsung firmwares, so it's probably some artifact of merging support for the 4210 platform (SoC AND peripherals) into the 4412 kernel tree.

This bug has probably existed since the 4210 support was merged into the 4412 kernel, it just becomes FAR more obvious in 4.4 as the sdcard daemon's behavior when an FP register gets stepped on is quite obvious. (Other stuff might simply crash/segfault.)

So it's probably an error somewhere in between https://github.com/omnirom/android_...mmit/05f0203060035bd2cb8c8f98b8b466b934b1c45b and https://github.com/omnirom/android_...mmit/855d6a6c1f7c54ef073caac3f6c5f9b1ed72eb4d - but not sure where.
[General Omni Discussion] GidiK:
I tested it (made my own build with just sdcard having the compiler option -mfloat-abi=soft) and as others have reported: the reading-over-fuse bug is no longer there! I did ten runs of reading two nandroids at full speed and that finished ok.

So we're back at looking at the kernel, trying to find some merge issue, or arm errata that we missed. I looked at those errata again. Previously I already found that the errata in our kernel are configured and build exactly as in the Samsung sources and that even during the merge with smdk4412 I don't see any mishap with regards to the errata. So now I looked wider: are there any errata that Samsung didn't configure (but for which code exists in the kernel) but which could still be applicable. And I found one: ARM_ERRATA_761320.

In favor of this erratum being the culprit is: it has something to do with a rare race condition, with a lot of cache reading and writing happening near the same time; both smdk4412 and smdk4212 have this fix activated (see Samsung KitKat sources for Tab3 8"); the remnants of Exynos4 in the Exynos5 kernel sources of the S4 have this fix activated for all Exynos4 configurations; even though there's no description in the source of this bug, non-authoritative search results on the internet suggest it should really also apply to the r2p7 variant of the Cortext A9 that we have (in fact, up to r3p*). So I've been testing with a clean build and this fix activated and the really good news is that the same test I'm doing (tar nandroids to /dev/null for a long time) also passes without problems! So, without adaptations in sdcard, just in the kernel. Also, playing music for a reasonably long time works fine.

I found some details at: http://www.xilinx.com/support/answers/47549.htm.

The reasons why I'm not confident and why we can't merge this 'fix' immediately, are:

The erratum descriptions that you can find for 761320 say nothing about floating point registers. Even though it clearly has been established that clobbering of those registers is happening. In fact the descriptions and our symptoms with hanging processes etc do not match. It describes a processor deadlock instead.

It says the race condition is very rare. Unlikely to occur even.

Samsung doesn't include the configuration to fix this in their smdk4210 kernel source releases. Whether they do include it in the real builds is another matter...

The phone feels slightly more sluggish.

It may still be a change that is hiding the symptoms rather than fixing the cause. Even though it is in the kernel this time, it may still be like the 'moving structs around' thing in sdcard.

But most importantly, the third part of the tests I do failed: I usually take an app that does a whole lot of small storage accesses, like Twitter with its avatars, or Tumblr. This time, while I was playing with these apps, while playing music, I experienced several soft reboots (both in Tumblr and in Twitter) and FC (in Twitter). And that's just not acceptable...

I've put the fix up for review, so people who want to can see how it behaves on their device: https://gerrit.omnirom.org/9672.
[General Omni Discussion] CRXed:
The FolderSync app would always hangs when doing backups, but with this fix, it no longer hangs!

I flashed over latest nightly of CM11, so that just works.
[General Omni Discussion] zeitferne:
What about narrowing it down by deactivating drivers in the defconfig? E.g. modem, tvout, touchscreen, …. Since the bug can be reproduced via adb shell, maybe even the graphics driver could be disabled.
[General Omni Discussion] Lanchon:
great work! but it's probably not it.

"leading to processor deadlock."

"The pseudo round-robin arbitration in the SCU might finally remain stuck on the ACP, so that neither CPU, nor ACP request can make forward progress, leading to system deadlock."

"When it happens, the issue leads to system deadlock."

they dont say if deadlock could be ended by an interrupt, but i doubt it. (i think it doesnt make a lot of sense for a cpu to abort a memory transaction because of an int.) so unless we see freezes, which btw may manifest as reboots due to a watchdog, this is not it.

they also say "Setting this bit might prevent the processor to utilize the full bandwidth when performing intensive traffic of full cache line writes, and a slight performance drop might be visible in the system." and you say "The phone feels slightly more sluggish." unless we are seeing freezes or reboots, then maybe it's best to keep the errata out.

but this is something interesting for those unscrupulous kernel hackers, maybe they can get some extra performance out of the S3 if they disable it. they dare to mess with fsync! somebody should tell them :)

but....

then again this errata could work. corruption may happen during FP context switch. i imagine the cpu writing full cache lines (FP context is BIG!), which we know could trigger "direct eviction optimization in the Bus Interface Unit". disabling it could be the workaround for many bugs, not just the one reported in the errata. there could be a different bug involving data corruption due to or depending on this optimization that the workaround also fixes. in fact, you could try any random errata and see what happens, one might actually work. (without a device i can only look at source code.)

so its relevant to ask, what are these reboots and FCs you are reporting? do they not happen without the errata?
[General Omni Discussion] tusko5:
Can we reproduce the bug without the fuse daemon?

Then, what about programming a userland clobber detector and instrument the kernel to detect when it context switches to its process? Last context switch would be the one that clobbered fp. Maybe it was clobbered before the switch but at least it is a start.

I also read about disabling devices if the drivers like the modem are suspect... it seems also a easy thing to do...
[General Omni Discussion] zeitferne:
3) The floating point data is corrupted not in the register itself, but in the data structure where the registers are stored e.g. between context-switches. There it is probably vulnerable to ordinary stack/heap-corruption.

Pointers to per-cpu vpf_state are stored in arch/arm/vfp/vfpmodule.c in the global vfpstate array. They point to a member of the tread_info structure defined in arch/arm/include/asm/thread_info.h. The vfp_state union itself is defined at arch/arm/include/asm/fpstate.h.
[General Omni Discussion] zeitferne:
The problem could be in an #ifdef or driver that is only enabled for the affected devices. E.g. much code is compiled only for CONFIG_MACH_U1. Unless, of course, there is no way for a driver to accidentally corrupt these data structures – I really don't know.
[General Omni Discussion] Lanchon:
but there cant be no #ifdefs related to a phone plarform in main linux code or linux arm support code.

in drivers, definitely. but then why or how would a driver corrupt the FP part of these structures, which btw are totally removed from its function? and seemingly corrupt only that with surgical precision, keeping the kernel up and running. if you come up with a plausible explanation i can look into it.
[General Omni Discussion] Lanchon:
ok, regarding misconfiguration, i just checked that the FP regs are saved eagerly at context switch and not lazily (since this is SMP and lazy save wouldn't work). and with that, i'm out of ideas.

so unless someone tells me that there is code that executes in kernel mode that i haven't disassembled with objdump, i can only think of hardware bugs and disabling architecture optimizations and test. i don't have a phone so i'm out of the game.
[General Omni Discussion] GidiK:
After more testing, I have to agree. Even though with the errata active, it is reproducible that the problem is gone or the symptoms are hidden, when combined with other changes the problem shows up again. That proves that it hid the symptoms rather than solved the root cause. :(

I was first not sure whether Samsung's actual build for SII would have the errata active, while the config in the released source has not. In fact, the opposite could also be true: that a performance degrading setting was released in source for S3 etc, which they in internal testing had found to not be necessary for real stability. Maybe they consider all this internal testing IP and not under GPL? But, what I actually believe now is they just didn't realize that they have an opportunity for performance improvement by switching off this fix to a race condition that will never occur.

It's very difficult to say. When I was shortly running without the errata, they didn't happen, but now with the errata they also don't. For the moment...

How does the modem binary enter into all this? It's a thing we don't include in the ROM, so users use the latest one from Samsung, from the JB 4.1.2 days. Mine is I9100XXMS4, which is a recent one for Europe. Is the modem binary userland or kernel? Does testing without sim card and/or airplane mode make any sense?