yashsingh1/xdaforums huggingface dataset:
[General Omni Discussion] pasche:
sorry, if might again tell something useless...

...but since I'm using poweramp to play music on my FAT32 external sdcard, I didn't not have a problem anymore.

Right now, I'm having a problem. I just heard my phone vibrating on my desk. I wondered why it didn't play my ringtone, which lies on my external sdcard.

Now...I cannot open the settings, tones because there are 2 scard and 2 mediaserv processes accessing my ringtone file! Just like when I had problems playing music with apollo (or any other music app despite poweramp) and using mediaplayer.

You are by far better into android developing and debugging....but I think that the "mediaserv" has something to do with the bug. Maybe it's the way, access to the sdcard is requested...I don't know. But this is the first time I have a problem and "mediaserv" is again accessing my files.

Also I could not connect to my phone via MTP anymore. Everything is stuck. I'll reboot my phone and maybe, I'll switch to a standard ringtone.

Regards,

Robert
[General Omni Discussion] pasche:
second page gone...

Hi,

what I just recognized, after I rebooted my phone because of this sdcard hang again, is, that my second "page", where I just had 2 or 3 links, has been removed. Now only the first and only "startpage" is left. I had this kind of problem several 2 or 3 times several weeks ago. I know, it has nothing to do with this bug but I just wanted to mention it.

Regards,

Robert
[General Omni Discussion] shar88:
Not to thread crap, but I have a Epic 4g touch (SPH-D710) which also has the Exynos 4210 SoC running CM11 kitkat that has this exact same problem, very annoying. :(
[General Omni Discussion] Entropy512:
Not surprised. This is affecting every Exynos 4210 device running AOSP firmware.

It probably was causing subtle issues even prior to KitKat - KitKat just results in the kernel bug having MUCH more severe consequences.
[General Omni Discussion] HippyTed:
This looks wrong!

In kernel/samsung/smdk4412/drivers/mmc/core/sd.c, line 246,

if (au > 0 || au <= 9) {

It will always evaluate to true. Shouldn't it be '&&' instead of '||' there?

Not sure if this could be related.
[General Omni Discussion] zeitferne:
Even if not, such issues are probably worth bringing up to kernel/ROM maintainers.
[General Omni Discussion] Entropy512:
It's a validity check - so it won't always evaluate to true. (e.g. it will fail to be true if the SD card has corrupted properties).

It's also code that is identical to mainline Linux 3.4
[General Omni Discussion] zeitferne:
defconfig

No, there should really be a && instead of || (if au is not > 0 it will be <= 9 for sure), even cppcheck complains here.

Because defconfig was mentioned: I compared CyanogenMod's generrated .config with the u1_defconfig file in Samsumg's i9100 JB sources and found that CONFIG_ANDROID_PMEM is disabled in CyanogenMod's kernel (haven't checked with others) but enabled in Samsung's kernel with this configuration (which I think is very interesting because if I'm reading this correctly the kernel could be using memory that other hardware (GPU?) is messing with):

CONFIG_ANDROID_PMEM=y
#
# Reserved memory configurations
#
CONFIG_ANDROID_PMEM_MEMSIZE_PMEM=4096
CONFIG_ANDROID_PMEM_MEMSIZE_PMEM_GPU1=4
CONFIG_ANDROID_PMEM_MEMSIZE_PMEM_CAM=0

I also found this difference very alarming: CONFIG_VIDEO_SAMSUNG_MEMSIZE_FIMC1 is set to 32644 by Samsung but to the smaller value 15360 in CyanogenMod's source. The comment above this block says "Reserved memory configurations".

Also disabled in CM but enabled in Samsung's is this (which I included because it contains the magich word MEM ;) ):

CONFIG_VIDEO_MALI400MP_UMP=y
# CONFIG_MALI_DED_ONLY is not set
# CONFIG_MALI_DED_MMU is not set
CONFIG_MALI_OSMEM_ONLY=y
# CONFIG_MALI_DED_OSMEM is not set
# CONFIG_UMP_DED_ONLY is not set
CONFIG_UMP_OSMEM_ONLY=y
# CONFIG_UMP_VCM_ONLY is not set
CONFIG_UMP_MEM_SIZE=1024
CONFIG_MALI_UMP_R2P4_LSI=y
# CONFIG_MALI_UMP_R3P1_LSI is not set
# CONFIG_VIDEO_MALI400MP_UMP_DEBUG is not set
# CONFIG_VIDEO_MALI400MP_STREAMLINE_PROFILING is not set
CONFIG_VIDEO_MALI400MP_DVFS=y

But on the other hand CyanogenMod has CONFIG_ION enabled, while Samsung has not:

CONFIG_ION=y
CONFIG_ION_EXYNOS=y
CONFIG_ION_EXYNOS_CONTIGHEAP_SIZE=71680
# CONFIG_ION_EXYNOS_CONTIGHEAP_DEBUG is not set

There are other differences but ARM errata seems to match.
[General Omni Discussion] HippyTed:
I dont think so, I only have a few versions of the linux kernel on my laptop, but I notice that in linux-3.10.25-gentoo it has been corrected to use '&&' instead of '||'. In later versions the file has changed even more.

No! For what values of 'au' will '(au > 0 || au <= 9)' evaluate to false then?

See my comment to zeitferne above.
[General Omni Discussion] Lanchon:
google moved from PMEM to ION in android 4.0, so these findings are not surprising. btw, it would be better to compare 4412 against the abandoned 4210 kernel.
[General Omni Discussion] zeitferne:
Okay, does that also explain the different values for CONFIG_VIDEO_SAMSUNG_MEMSIZE_FIMC1?

(BTW: The Samsung kernel is also post-4.0.)
[General Omni Discussion] Lanchon:
i dont know. had i known anything else i would have said. (yes i know, its JB. google moved, not everybody. the 4412 kernel obviously did... what about 4210?)
[General Omni Discussion] Lanchon:
this situation is horrible.

so i guess the real question now is: was the old 4210 kernel before the merge working correctly, or has the platform always been fundamentally broken and we just never figured it out?

if the old kernel works, then there is some hope. for starters i could recommend people to run CM 10.1.3 instead of dumping the thing in the trash.

how could we test the old kernel? how about this:

1) make the bug reproducible and detectable. for example: build kitkat kernel, userland, and/or a native android app using -fstack-protector-all or -fstack-protector. make stack-protected code run while interrupts are being generated. the code could be a heavily recursive native app, that could call the kernel (as sdcard.c does) or not, or could be the instrumented kernel itself. the interrupts could be usb transfers or emmc accesses or both, but should be reproducible in android 4.2 / CM10.1 (this eliminates MTP transfers via sdcard.c). the "chmod workaround" could be used to grant MTP access bypassing fuse (i'm guessing CM10.1 supports MTP), or maybe USB mass storage transfers would suffice. the object of all this would be to reliably trigger and detect the bug using the 4412 kernel but without using android features newer than android 4.2.

2) make the same instrumentations and tests on an old kernel 4210 android such as CM 10.1.3, and see if the kernel holds or breaks down under stress. we could also try the same tests on an old android running under a new kernel, to make sure the bug is triggered.

if 2) shows that the platform is broken, these 4210 PoS devices that brought so much grief to this community over the years should finally be hammered, burned and dumped in the trash. if not, at least we'd know that running pre-merge kernels is safe, that somewhere in the merge lies the problem, and a way to test 'hybrid kernels' to sort of bisect the problem.

problem is, i'm not knowledgeable enough to pull this off; not to mention i don't even own a 4210 device.
[General Omni Discussion] zeitferne:
CyanogenMod bugreport

Because it was required for listing the issue in CyanogenMod's Known issues page for the Galaxy S2, I made a JIRA bug report there: CYAN-5003. It reproduced the bug using rsync and attached the dmesg output and the status of rsync while hanging (and also a logcat, but only because it's required for CM bugreports). Maybe someone might find that helpful.
[General Omni Discussion] shar88:
Can we get the d710 added to this bug?
[General Omni Discussion] zeitferne:
You should better ask that in the JIRA thread. Not even the bugs creator (me in this case) can change the original report, this needs to be done by someone with sufficient JIRA privileges.
[General Omni Discussion] Lanchon:
not necessary. the bugfix, if ever, will come in the 4412 kernel. the kernel code is shared among all 4210 devices, including the i9100 and d710. no way to fix one but not the other.
[General Omni Discussion] Entropy512:
Yup.

I won't be looking at this at all for the next few weeks - traveling on vacation. :)
[General Omni Discussion] Acid0057:
Good for you Entropy! Enjoy! Thanks for all your help on this so far.
[General Omni Discussion] zeitferne:
Replaced kernel arch directory with "official" Samsung one

I have tried something drastic: I copied the whole arch directory (not only arch/arm) from Samsung's JB Kernel (first result here) over the one of Cyanogenmod's kernel (here) (EDIT: This means that any additional files added by CyanogenMod, such as the kernel config are preserved. But since the makefiles are overwritten, this should will probably not change anything). On first attempt to compile (breakfast i9100 && make bootimage) I got a compiler error in kernel/power/main.c, because apparently a function used by this file had a changed parameter list (kernel/power/main.c:587:4: error: too few arguments to function 'exynos_gpufreq_lock'). So I replaced the whole kernel/power directoy too with Samsung's and afterwards the kernel built fine. One installboot and restart later, the was up and running, seemingly fine (although apparently the home key does not work and there might be other issues – apart from the following I just booted). I tried reproducing the issue as I desribed in CYAN-5003 and succeeded :( (i.e. the issue is still there).

We can conclude from this that either

the code in the arch directory is not responsible for this issue (at least not the changes in the code; it is still possible that the kernel configuration (de)activates the wrong code in this directory), or

the issue is latently there even on Samsung's stock kernel.

Obviously, I would like (A) better, although if the bug gets fixed (B) would be one reason more to change from Stock. ;)
[General Omni Discussion] Lanchon:
i think the best way to proceed would be to make a native synthetic workload that reliably triggers and detects the bug (see -fstack-protector-all) independent of android to test stock kernels and the old abandoned 4210 cm kernel.
[General Omni Discussion] Lanchon:
lol, amazing that android boots at all!

still, a synthetic workload would be better imho. we could run it on stock and other roms and see if any is not affected. i dont have a 4210 device so i cant test anything. please advice if you d like help coming up with a suitable workload.
[General Omni Discussion] HippyTed:
Sorry if I'm teaching a grandmother how to suck eggs! :cool:I see dhd_write_macaddr() comes from drivers/net/wireless/bcmdhd/dhd_custom_sec.c which ends up in the dhd.ko kernel module.

Trying to run

prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/arm-eabi/bin/objdump -d out/target/product/i9100/system/lib/modules/dhd.ko | less

Doesnt seem to tie up for me. Maybe I'm using a different compiler than you, and/or the stack protector has changed the addresses.

Have you managed to see why it derps? I dont really know ARM assembly language much, anyway. :cool:
[General Omni Discussion] Lanchon:
the failure is probably non deterministic and happens when a hardware interrupt is triggered. by the way [MENTION=5948587]zeitferne[/MENTION], can you confirm that the kernel dies in a random place each time?
[General Omni Discussion] zeitferne:
Sorry, this is a very deterministic and probably unrelated bug. In drivers/net/wireless/bcmdhd/dhd_custom_sec.c:11032 replace

char buf[18] = {0};

with

char buf[sizeof("00:11:22:33:44:55\n")] = {0};

(where the sizeof expression evaluates to 19).

This was "just" an off-by-one error in the wireless driver initialization, causing the sprintf in line 1110 to write a rogue null byte.

After changing this, the kernel boots fine, but still exhibitis the sdcard bug. Even enabling -fstack-protector-all (which required changing the option from -fstack-protector in arch/arm/Makefile:41 and adding -fno-stack-protector add arch/arm/boot/compressed/Makefile:103 to avoid linker errors) does not find any more stack corruptions, even while triggering the sdcard bug.
[General Omni Discussion] Lanchon:
well the kernel might not be the best place to stack-protect. i guess building sdcard.c with stack protect should trigger the protection fault. but sdcard is not portable, we should find better workload.
[General Omni Discussion] zeitferne:
Well, I don't know very much about Android or Linux kernel development but I can certainly test things and probably also build them. If you come up with something, I would be happy to try it. If I should also test it on Stock, it would be best in APK form and without requiring root, so that I can test in on my brothers SGS2 w/o flashing around ;) (I can't get nandroid to restore one ROM from the other).

But I also think that more important than checking if the bug appears on Stock or on the 4210 kernel is actually finding the bug. There are so many differences between these kernels that I don't know if this would be very much help in locating the bug. Still interesting to know though.

EDIT: I submitted the wireless driver fix to CyanogenMod's Gerrit: http://review.cyanogenmod.org/#/c/72657/
[General Omni Discussion] baldaz:
If we are in the section of the forum regarding the OMNI ROM why not submit this patch also in Gerrit OMNI?
[General Omni Discussion] zeitferne:
I would need to download the whole repo which takes hours with my bandwidth. Also I haven't tested with OmniROM (you can never be sure, although this change should be more than safe). But anyone can commit this change to OmniROM, git commit even has a --author option for giving proper credit (although this change does not hold much original value ;) ).
[General Omni Discussion] Lanchon:
if we have source without the bug (say, sammy's) then *maybe* we could find the bug in the current kernel. if all kernels have the bug, then this is probably a hardware issue for which no config or toolchain workaround (aka errata) was ever developed and could even be impossible to develop. think defective hardware, and since sammy is unlikely to replace old S2s, class action :)
[General Omni Discussion] Lanchon:
hi, i completely forgot about this!

just wrote the silliest workload. it is single threaded. it doesnt invoke the kernel nor causes usb, eMMC nor sdcard transfers, so in order to provide the interruptions needed for corruption something must be going on concurrently: a usb transfer? an eMMC benchmark? dd if=/dev/<emmc_partition> of=/dev/null bs=...? i leave it up to you! :)

it does check for stack corruption, but -fstack-protector-all would be welcome too. here it is;

stack-test.c:

#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>

int depth;

typedef unsigned long long T;

T tv = 0x3f0d20d623bae889;
const T ti = 0x2f3fd9ec7013ad35;

void f(bool first) {
if (depth) {
T t = (tv += ti);
depth--;
f(first);
if (first) printf(".");
f(false);
depth++;
if (t != tv) {
printf("\nCORRUPTION\n");
exit(1);
}
tv -= ti;
}
}

void main() {
setbuf(stdout, NULL);
for (;;) {
f(true);
printf("%i\n", depth);
depth++;
}
}

try it if you can and tell us if it triggers the bug and what you do concurrently to trigger the bug. thanks!
[General Omni Discussion] GidiK:
Unfortunately, this didn't reproduce the problem on my device. But then, I could never reliably reproduce it with playing music for a long time either.

Anyway, can people who can reproduce the problem on a i9100, are willing to test and can build, please cherrypick and test with this: https://gerrit.omnirom.org/#/c/9615/ (as always at your own risk). It reverts some changes I found that have been made over the years which could be influencing the behaviour we see. It survived my stress tests while running the above program, but then, so does the normal build occasionally.
[General Omni Discussion] zeitferne:
I tested it on my i9100 (manually applied the changes to Cyanogenmod's smdk4412 kernel) and sadly the problem seems not to be fixed, as I was able to reproduce the problem with rsync again. But I just checked KERNEL_OBJ/include/generated/autoconf.h and it seems that CONFIG_FORCE_MAX_ZONEORDER was reset to 12. Removing the

default "12" if ARCH_EXYNOS

line in arch/arm/Kconfig:1641 seems to have fixed this. I am currently compiling again. EDIT: Nope, does not fix the problem either.

Maybe one could also try to remove the cortex-a9 flags from the build of the sdcard daemon, though according to Wikipedia the smdk4210 board is cortex-a9 based.
[General Omni Discussion] Lanchon:
before trying to revert changes to some samsung kernel itd be worth it to verify that that kernel is not affected; otherwise we are going around blind. we need some way to test this on all kernels.
[General Omni Discussion] Lanchon:
dram getting old? no such thing.

4210: bug

4212: no bug

this is a SoC issue for sure

sdcard.c:

-triggers interrupts (reads eMMC)

-sensitive (crashes on corruption)

-does not heal (no restart)

-very noticeable (sdcard access gone for all apps)

other than that, there is nothing special about it. we can make a workload, a non root app preferably, to reproduce the bug.

i made a first try at a simple workload. but we need a proper app if we want people to test on stock. and i dont have any of the affected devices, so i cant test. unless someone tests, we cant progress. running my workload while reading an emmc partition would be a good start. (directly reading, not thru fuse, that way it can be done in all versions of android.)
[General Omni Discussion] Twiq:
If you're using FAT32 filesystem then the maximum filesize is somewhere at 4GB. And your log tells that camera app is trying to clip your video to 4GB from originally ~12GB ish, nothing more unusual, maybe you should post the whole log
[General Omni Discussion] CRXed:
This is the whole log......after this, nothing else related to camera shows in the log, until a soft reboot.

Makes no sense to make a 4GB file when I press the record button! No file was 12GB, I have no idea where it gets this number from....
[General Omni Discussion] Lanchon:
i made an app. has native stacktest.c compiled with -fstack-protector-all inside it, should work on non-root.

so now people have to test. some other workload that generates interrupts has to be run concurrently: disk activity, disk benchmark, usb activity, something; get creative.
[General Omni Discussion] Twiq:
Will iterating through large folders of pictures do?

I'll delete the thumbs caches and make the apps (not gallery) reload them at once :D
[General Omni Discussion] Lanchon:
reading the sdcard non-root is not the best choice of load, as it goes thru sdcard.c (fuse) in some androids (eg kitkat) and not in others (stock). something that reads/writes to the internal /data partition would be more useful. AndroBench comes to mind. in the setup you can increase the workload.
[General Omni Discussion] Twiq:
Was it designed to stop after several counts?

It stopped at 31, what does it mean?
[General Omni Discussion] Lanchon:
no, it never stops. each line takes twice as long as the previous (as does each dot).

if my process detects corruption it will print CORRUPTION

if GCC detects corruption it should print *** stack smashing detected ***

but i forgot to visually signal that the process died! it should never die if the kernel and device work well.
[General Omni Discussion] Entropy512:
Well, DRAM CAN fail - but there's no way it is failing on this many devices simultaneously. It's definitely a kernel bug.

Not necessarily. It could be a bug in a driver that is only used on 4210-based devices and not 4412-based devices. (I think only some of the Gear family actually have a 4212 - 4212s are RARE)

A useful thing for someone to try (I unfortunately don't have the time for it) would be for someone to try and get the old smdk4210 kernel updated for compatibility with KK. (At a minimum you'd need to update the Mali drivers, maybe some other stuff...) and see if the old smdk4210 tree (before it was merged into the smdk4412 tree) has this issue or not.

If it doesn't, that points to something either getting mismerged or a change not getting merged that should have been.
[General Omni Discussion] Lanchon:
i think it'd be simpler to create a workload that triggers the bug. can be run on stock or old CM images that have the old kernel.
[General Omni Discussion] Entropy512:
Thing is we'd be trying to reproduce something that some people are already having trouble reproducing with an artificial scenario.

That said, probably the best bet for something like that would be something that did a lot of pread64() calls - since that seems to be what usually results in Bad Things happening.
[General Omni Discussion] Lanchon:
well this is guesswork, but pread probably executes fine, the important thing is that certain interrupts or maybe dmas are happening. pread is one way to trigger them. there is no strong data telling us that the corruption will happen in the stack area of the thread that is responsible for triggering those events. so i made a detector that can run concurrently with whatever you wanna test, MTP transfers, USB transfers, dd'ing a partition to /dev/null, etc. people can try stuff and report. if that doesnt work then we can go for a workload that does pread or whatever.

regarding repeatability, some people say they trigger it reliably by doing MTP transfers.
[General Omni Discussion] CRXed:
After switching to a few other apps, it just stops.
[General Omni Discussion] Lanchon:
yes, its not a service. it stops when android stops it. u r supposed to do one other thing, not many.
[General Omni Discussion] GidiK:
I must stand corrected: I believed a few posts ago that the rsync-method was not a good method of reproducing the problem. But I guess I was wrong. Copying music files around did cause other effects, related to the media scanner, but the reading problem was in there too. To isolate the reading problem, I used the following method: in one 'adb shell':

tar cv /storage/sdcard1/TWRP > /dev/null

while looking with another 'adb shell' at:

iostat -k 5

After a few GBytes, iostat shows that reading hangs. If you try again, it only reads a few bytes and hangs again.

However if you avoid fuse, by doing in a root shell:

tar cv /mnt/media_rw/sdcard1/TWRP > /dev/null

then it invariably succeeds.

Whether this is the same problem as the sound-stops problem, I don't know, but the solution to it can be found in sdcard.c, see https://gerrit.omnirom.org/#/c/9648/. My reasoning for the problem is as follows:

In handle_read some values from the request buffers are put into local variables, which usually reside on the stack. Then, due to a 'union' (and noted by a comment), the read buffer for pread64 overwrites these original request buffers, intentionally. However, the request buffers are marked as 'const struct *', and the compiler feels free to not actually create a local variable, but use the value/address from the received structure (I assume). Then, when pread64 returns, the local variable for unique is used; but its value at the address in the request buffer is already overwritten. Usually this is ok, because it will reside in a cpu register (I assume). But if an interrupt has occurred, this may have changed. I'm quite vague on why the interrupt would not have saved all registers on stack, but maybe some assumption has crept in that it didn't need to do this, because the value was 'guaranteed' to be available somewhere already?

I tried removing the const from the request buffers, or putting a dummy buffer before the (smaller) read buffer in the union, but that didn't work. So, that's a point against the reasoning above.

However, with the submitted change, the problem is actually solved, judged by the measurement above. I'm not happy about the solution though, so if somebody has a more elegant way of solving it, please do!