yashsingh1/xdaforums huggingface dataset:
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
If I understood well, here is what happen:

We have a 32M buffer in radio that is used for ppp. smd_tty gets notified when buffer has data, and copy from smd buffer to tty. As it happens in interrupt, driver has to use kmalloc memory, not vmalloc, implying that the memory has to be contiguous.

As it's currently coded, the driver read the total amount of data in smd buffer, and allocates it in tty then copy. So, if main cpu is busy, smd buffers fills up and ends up with something like with a single buffer of 20M to pass to tty.

What I did is simple, if smd buffer has more than 2K, I only read and pass up to 2K to tty. The only thing I don't know yet is if I need to loop until the smd buffer is read or if it will be called again by itself. Currently, I suppose that it will be called again and then clean-up the buffer. If it doesn't, I'll loop. At the end, tty will have multiple buffers of 4K, linked together and we shouldn't miss data.

Regarding memory usage, it's a bit tougher, not sure I got it right. My understanding is that android gets all for VMs, then kills process when a certain threshold is reached. Unfortunately, this doesn't take into account the way memory is scattered around block size.

Again, not sure, but if I'm not wrong, Linux kernel pre-"allocate" memory in different size, for instance 100 4K blocks, 50 8K, 20 32K, 10 64K, 2 1M. If I need 64K, only 10 are available. If all 10 64K blocks are taken, Kernel can aggregate, grab 2 32K to make a 64K, but contiguous is key here, it's not because I have 2x32 that they are contiguous.

From what I saw, Android pre-allocates big chunks of memory for VM, so 64K blocks are almost instantly taken. And task killer doesn't see anything wrong with that as we have 100x4 + 50 *8 KB available. If you're using HTC sense that is pretty intense on memory, it will show up quickly.

Regarding diag, I'm still looking, /proc/buddyinfo contains interesting data about free blocks, however, I can't find an easy way to know who owns what.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] ala380:
Ok here are my latest logs. I did notice that I am receiving the "tty_prepare_flip_string fail" error so I am going to go ok and try the test kernel. For some reason nothing came up on my maimlog so I have to try to get that again on the next fail. The data freeze slows my whole system down and the phone doesn't respond to setting airplane mode or turning off data. I'll try to kernel and check back in :)
[HD2 Android Q&A, Help & Troubleshooting and Genera] arthursnape:
Testing version 0.4 now

Using Mdeejay_FroYo_Sense_v._3.1, So far so good......:)

Big Thanks..............:D
[HD2 Android Q&A, Help & Troubleshooting and Genera] ala380:
Hey letama,

I am testing test kernel now and so far data has been up and running longer than it was with the latest wrapper without the test kernel. I'll keep testing and post logs if/when I run into problems. Do you want logs for working data with this setup?
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Yes, please, send dmesg log even if it's working, I want to see how my patch behaves.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] ala380:
Ok here is my latest dmesg log with working data using the latest .7 wrapper and test kernel 2k. So far it is working well for me. If there is anything that you need let me know :)
[HD2 Android Q&A, Help & Troubleshooting and Genera] chn110:
Thanks for your explanation, LaTama. If I understand correctly, when data are pending on the modem, it sends interrupt to the main processor, which in its interrupt handler, allocates the buffer memory, then copies in the data. If buffer allocation fails, we simply log the error and bail out. If that is indeed the case, then the obvious problem is that we entered the interrupt routine, did a whole bunch things, wasted CPU cycles for nothing. The next interrupt comes in we will still likely be running the same cycle for nothing. Meanwhile the CPU is so busy and the user layer apps and the GC can't run and we ended up in this perpetual cycle. It appears to me that if we ever get into buffer allocation failure, we may need to

1) set a flag which will then trigger GC

2) drain modem data quickly so we won't immediately get another interrrupt (Q: If we don;t fully drain modem buffer when do we get another interrupt - when new data comes in or immediately?), or

3) Mask interrupt until buffer is returned to the particular pool.

The memory model you explained seems problematic - the kernel buffer used by interrupt handler should never be contending with user level VM apps, otherwise it is a problem waiting to happen.

I ran your new kernel and here are the findings - Data never got stuck. It did need toggle 3G a couple of times. But there are interesting behavior difference than before:

- Initially data at full speed, recovers after hitting 3G dead spots.

- Then a periods of data stop and resume by itself and recovered.

- Then it comes to a crawl, really slow but is going and recovered.

- Then it drops to a considerable slow speed for a long period of time (300kbps down than the usually 2Mbps), but eventually recovered.

I attach the logs for each of the later three cases. Hope it will help to explain more things. Thanks!
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Well, not really:

What was done before is perpetual loop until alloc is satisfied, ending in high cpu consumption and data dead.

What I did was limit request to 2K even if more data is available (and bail out if malloc fails, true). So when interrupt comes back, we did drain buffer a little at least. My idea was that big buffer could be difficult to allocate but with smaller we could be fine.

Well, I was wrong, in your logs and ala380 logs, we end with not enough kernel memory to allocate 2K. I have no idea why, I have to dig into linux memory management because it seems I don't understand debug messages we get from kernel.

We could have a memory leak in kernel somewhere, but unfortunately, kernel memory leak detector didn't came with anything last time I activated the option.

Afaik, 1) can't be, there is no GC in kernel. 2) I don't see how to do quicker that we do now. Anyway, looking at logs, tty doesn't seem to filling up (tty->buf.memory_used remains low) meaning that pppd gets data out quickly. The big problem is that we can't allocate memory at all.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] noellenchris:
been using your options and ip-up all day and I must agree, it's a great improvement for the dropped data connections. Thx.

Letama, been using the above and the test kernel 2 and latest wrapper also, it only dropped data in known bad areas (expected) and quickly recovered without error so far. Hopefully the 2k is helping, I'll continue to test and see if I can get it to fail. I'll report back in a few hours, going to get some new clothes for an interview in the AM and will use the GPS and other data stuff while I'm out. :)
[HD2 Android Q&A, Help & Troubleshooting and Genera] chn110:
Yes, the perpetual loop I meant the behavior before your changes. I think your change made it less likely to fail buffer allocation but has the effect of dampening 3g speed, as it has less buffering space. The slow case logs seem to suggest that it was requesting larger than 2K buffers and we don't have it. Portion of data got dropped and we ended up with slower speed.

Yes it does not make sense to me why alloc failed, the log seems to say there are still 4K blocks available. I did not see the code so just guessing here.

<4>[ 2998.206329] Normal: 4247*4kB 3*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17012kB

<4>[ 2998.206390] HighMem: 166*4kB 11*8kB 5*16kB 2*32kB 0*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1024kB

1) not GC but killing of background tasks to reclaim memory. Maybe it is already doing this but the trigger can be set at the particular pool.

I am not sure if this is really a memory leak since if it is generic kernel memory leak then why aren't we seeing the issue on RMNET? I still think it is somewhat ppp related. Unless it is because RMNET's low data rate which does not trigger the issue, or RMNET does not allocate buffers? That can't be right...
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Ok. 3g speed could effectively be lower, especially as I trace a lot and the way I do the 2K limit one at a time. I want to first take out the memory problem and then see for performances.

Trace confirms that tty tries to allocate 2048 bytes only and that kmalloc fails to get them.

Well, I'm not sure either, but a 2K kmalloc failure is quite strange. Anyway, regarding rmnet, behavior is very different. First, tty implies data size x2 because of the way ttys works. Next, rmnet stays in kernel where pppd jumps from kernel to usermode to kernel, implying more memory consumption. Last, we lower mtu to make rmnet work, it's slower than ppp, resulting in less stress to memory.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] ala380:
Still up and running well using the newest wrapper and test kernel. I haven't experienced any data freezes or drops as of now no matter how hard I try :) Was thinking about using aussiebum's options and ip-up but haven't had to do that yet. Will report back with logs if I experience any issues after I test it a bit more.
[HD2 Android Q&A, Help & Troubleshooting and Genera] noellenchris:
ok, been running all day, just got back again and used gmaps and speedtest etc. 3g seems a little slower, but usable. can't get it to hang yet, I'm trying....but you may be on to something with the last kernel and wrapper. do you want any logs even though it's working good?

Next I'm going to kill connection several diff ways and try to download something big and see how long it can keep going.....
[HD2 Android Q&A, Help & Troubleshooting and Genera] molesaied88:
a sure fire way ive noticed to kill data is to make a skype call over data instead of wifi. theres a modified .apk floating around, and usually i can get max 5 min before data drops. its the most data instensive thing ive found so far.
[HD2 Android Q&A, Help & Troubleshooting and Genera] vtruth08@yahoo.com:
wat program are u guys using to get the debug information after u update the kernal and wrapper, any instructions?
[HD2 Android Q&A, Help & Troubleshooting and Genera] anthnhlt:
I am using the test kernel but using the latest wrapper I downloaded two 100 know file and it didn't hang once. I am using hastarin's kernel 8.2. With regular usage I haven't got on freeze or hang. Last hang was this morning while using wrapper 0.6. Nice work.
[HD2 Android Q&A, Help & Troubleshooting and Genera] aussiebum:
Read the first post again.
[HD2 Android Q&A, Help & Troubleshooting and Genera] aussiebum:
Glad I could help :) Been testing 0.7 wrapper along with a another pppd option ("passive") on top of the existing ones I posted previously and so far I've had zero dropouts, which is unusual, in a good way.

Data doesn't feel any slower or faster for me so far. But then again, I've never had data freezes ever since I adjusted my ro.ril settings in build.prop so I can't comment. Might try bumping those back to default settings and see what happens.
[HD2 Android Q&A, Help & Troubleshooting and Genera] dealkk:
i don't see instruction on how to install this.
[HD2 Android Q&A, Help & Troubleshooting and Genera] LUIMI31:
With the wrapper 0.7 and the kernel test works good for me.

0 disconnections and no freeze data.
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Kernel adjustment

Hi guys,

Can anyone with the memory issue test these options for me ?

In a shell or script:

echo 8 8 > /proc/sys/vm/lowmem_reserve_ratio

echo 8192 > /proc/sys/vm/min_free_kbytes

With a regular kernel and/or my test kernel.

Thanks,

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] chn110:
I set this while the kernel is dumping 'tty_prepare_flip_string fail' logs with hastarin r8.4 kernel. It is still dump the log. Will leave it in this state a bit longer in case you want me to collect any info.
[HD2 Android Q&A, Help & Troubleshooting and Genera] turt:
Hi letama,

I have a strange behaviour that I think I have identified coming from version 0.6 of your ril wrapper: battery consumption can be fine for a few hours (ie, drop 1% per hour in standby), then suddenly the battery warms alot, and the battery consumption is like 10-15% per hour! It started happening when I upgraded from 0.4 to 0.6 two days ago, and simultaneously I had upgraded my kernel to latest hastarin's... So at first I was not sure if it was kernel or wrapper related, but now I have reverted back to 0.4 when I was experiencing the massive battery drain, and I don't experience the problem anymore (without reboot of my phone in between).

Since it works fine for a few hours with the 0.6 wrapper, I think at some point I have a disconnection that the wrapper tries to recover, and for some reason it doesn't go well and ends up with massive battery consumption...

Is it mandatory to use the test kernel with your 0.6 ril wrapper? Maybe that's the reason this happened?
[HD2 Android Q&A, Help & Troubleshooting and Genera] ala380:
Ok I ran these as a script and at first it took a long time for me to get any data. But it did after a couple of minutes connect. I tested using the test kernel so I don't know if I will see a difference because I didn't have any drops or freezes with the test kernel and newest wrapper. I've included logs for both non working data and data. I will test with m-deejay's kernel that I was using previously and upload logs from that.

I have a question about the test kernel since I don't know much about it :) Are the latest git changes (upto when you uploaded it) included in this kernel? Is there any reason that I shouldn't use this as my everyday kernel because it seems to be working well as opposed to the hastarin and mdeejay kernel. I have a camera issue but I'm not too worried about that.

Thanks LeTama, more logs to follow
[HD2 Android Q&A, Help & Troubleshooting and Genera] chn110:
Rebooted the phone. Did the /proc/sys/vm changes at the beginning. After a while the kernel is constantly dumping

intr_status=0x02

every 30ms or so. Data is still going at a good rate. Will try with the test kernel and see.
[HD2 Android Q&A, Help & Troubleshooting and Genera] ala380:
Ok here is a log from working data using hastarin's 8.4 kernel and the scripts you posted. So far it's working well. I will continue to use this setup for the rest of the day and post more logs if necessary.
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Hi chn110,

It's already too late if you have the flip_string fail. You should reboot, set values and try to reproduce it.

*edit* saw your second message later... sorry, ignore that.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Hi Turt,

Well, I would call that bad luck, the high battery consumption is because of the memory problem. Wrapper doesn't have any impact on this except if it runs two pppd at the same time maybe, but 0.6 is less prone to that than 0.4.

What triggers this memory problem is not identified yet, I don't have it at all for instance...

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] chn110:
With smdtest_2K_kernel, did not see any memory allocation failure, although did not run for too long. Noticed ps shows more processes running than usual without getting kill by oom_kill. Log attached. Will try the original smdtest kernel (32K) and see.
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
This is microp/i2c, it's not related to data/smd. No idea why it popups like this, I saw few logs with that in the past.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Your logs are still showing memory problems even if my patch recovers them :-(.

I was not completely up to date with git, but I was not very far. If it improves your usage, no problem to use it, the only problem is that it's a little verbose and maybe not optimal for data speed. Camera problem is maybe because of the 720p story, it's compiled without it. I didn't follow camera evolution, but I know that my build doesn't play nice with 720p set.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] chn110:
Had been running smd_test kernel, then switched to Mdeejay_FroYo_Sense_v._2.4_Clean_S6.1 build. Both with the memory settings. Have not gotten a single data hang so far. Normally I would have gotten a few by now. dmesg did not show any buffer allocation error the times I checked. I still see a bunch of

pr_info("intr_status=0x%02x\n", intr_status); This looks like info and not error and should probably be taken out.

So far this looks extremely promising. Thank you!!
[HD2 Android Q&A, Help & Troubleshooting and Genera] noellenchris:
Letama, well it's now 2 days and I haven't rebooted. I tried everything to get it to fail. I streamed live TV with the TMobile TV app for over 2hrs, I even drove around with the stream and lost signal and it still recovered. I can't get it to freeze anymore.

I think you are on the right track. Data is working pretty much perfect for me. The kernel is also doing well on battery. Maybe Hastarin can incorporate your changes into his kernel.

Thanks a ton.

P.S. I had the dropped calls again today. Seems to get better after it drops about 5 times....?
[HD2 Android Q&A, Help & Troubleshooting and Genera] :
confusion

im am about to use this or try this fix witch requires me to use the path and change the kernel (zimage included), but i am also using the touc screen fix witch requires the zimage be changed. what now?
[HD2 Android Q&A, Help & Troubleshooting and Genera] chn110:
Unfortunately I had a couple instances of buffer alloc failure even with the changes. Logs were overrun with the same "smd_tty_work_func: tty_prepare_flip_string fail" error. But the good thing is that the frequency of this seems to be a lot lower with the changes.
[HD2 Android Q&A, Help & Troubleshooting and Genera] ala380:
I just wanted to report that I have been using hastarin's latest kernel along with the memory scripts and have not had any data freezes at all. I know that I am still getting memory errors but at least data has been stable and hasn't required any reboots. It may be a little on the slower side but I am ok with that as long as I am able to stay connected. Thanks for the work you have done so far and I will continue to test and offer logs for any future test fixes :)
[HD2 Android Q&A, Help & Troubleshooting and Genera] hastarin:
LeTama I hope you don't mind me posting on your thread but I figured it was the best way to get those with the issue testing my patch. ;)

I have a test kernel with a patch to smd_tty.

LeTama's approach was to try smaller buffer sizes.

Mine, if I understand the code correctly, is to just continue with whatever it can get.

NOTE: This is based on my r8.4 kernel, not the main tree.

I welcome any feedback on it:

http://www.multiupload.com/T8J37O3BNJ

Data certainly works for me locally but since I don't have a huge data plan I can't test it heavily myself to try to reproduce the smd_tty issue.
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Hi Hastarin,

No Problem, on the contrary. The more we are looking at this, the more we have a chance of fixing this issue.

I took a look at your patch, I'm afraid it won't work as it is. You removed the test on avail and ptr, but if someone has the out of memory problem, tty_prepare_flip_string will return a null ptr and smd_read will crash with a segfault.

If you want to use whatever is available and avoid endless loop on memory alloc failure, you only have to add a break on failure like this:

if (avail && ptr) {
if (smd_read(info->ch, ptr, avail) != avail) {
/* shouldn't be possible since we're in interrupt
* context here and nobody else could 'steal' our
* characters.
*/
printk(KERN_ERR "OOPS - smd_tty_buffer mismatch?!");
}
wake_lock_timeout(&info->wake_lock, HZ / 2);
tty->low_latency = 1;
tty_flip_buffer_push(tty);
} else {
printk(KERN_ERR "smd_tty_work_func: tty_prepare_flip_string fail\n");
break;
}

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Hi chn110,

Thanks for the report. So it seems that even with much higher reservation, we still end up with no memory left. We need to find what eats memory like that...

I'll dig in memory management to see if we can find who uses what.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Hey noellenchris,

This drop call issue is weird. I hope it's not related to wrapper/kernel changes. Did you change radio recently ?

I'd like to take a look at radio logs after these drops to try to confirm that they are not related to wrapper. Could you post them ?

Thanks,

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] hastarin:
Are you sure we get a null pointer in that case?

Take a look at the code in the CM kernel:

https://github.com/CyanogenMod/cm-kernel/blob/android-msm-2.6.35/arch/arm/mach-msm/smd_tty.c

They don't do it there, and reading the description of tty_prepare_flip_string and tty_buffer_request_room (which it calls) it seems it should still return a valid pointer as long as we have memory to use?

In fact it seems to me it's trying to increase the size of an existing buffer so we should always at least have some sort of buffer available?
[HD2 Android Q&A, Help & Troubleshooting and Genera] turt:
Well, turns out it may have been kernel related after all!
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
Well, we don't have memory left to use, that's the problem. The fact that we have the message means that we have null ptr or 0 length. Anyway, in both case, you'd better break the loop.

The fact that it's not in cyanogen kernel is dangerous in my opinion, but as they don't do ppp over smd_tty, they may never see any issue.

LeTama
[HD2 Android Q&A, Help & Troubleshooting and Genera] hastarin:
I can see what you're saying but I was presuming, from what I read, and perhaps incorrectly, that the buffer is allocated already, it's just trying to increase it.

It's the same in the new 2.6.35 kernel too:

http://android.git.kernel.org/?p=ke...d8762cb8006b;hb=refs/heads/android-msm-2.6.35

And 2.6.32 for that matter:

http://android.git.kernel.org/?p=ke...d8762cb8006b;hb=refs/heads/android-msm-2.6.32

I'll have to take another look at the code tomorrow after work. It's sleep time for me now.
[HD2 Android Q&A, Help & Troubleshooting and Genera] anonuser1:
High Battery Drain

LeTama,

I posted this on Hastarin's thread also, but figured it might help you as well. I am using your .7 wrapper with Hastarin's 8.5 beta on PhireMod CG:

When I turn data on, I get high battery drain (around 1% every 2 minutes). When I turn the data off, the drain still persists.

In order to stop the drain, I have to turn data off and reboot. So long as I never turn data on after fresh reboot, I get excellent battery life (around 1% every 1.5 hours).

This issue is with DATA only. Wifi does not cause high battery drain. So, I have isolated the issue to DATA.

Is the wrapper the only thing that affects DATA? or does the kernel affect it as well? Should I try different kernels to see if the problem persists?
[HD2 Android Q&A, Help & Troubleshooting and Genera] letama:
If I read well, in fact, buffers are in linked list and when no room is available in current buffer, it tries to allocate a new one. And this is where we have troubles with ppp. As it's a very abnormal situation, kernel out of memory, I can understand why they let the kernel misbehave.

However, as we have the issue, I'd rather handle the problem gracefully and crash elsewhere :). I'm still looking for a clear picture of available memory, I'll probably add some traces around kmalloc to understand where it looks for memory and why it fails.

LeTama.
[HD2 Android Q&A, Help & Troubleshooting and Genera] hastarin:
I'll try to take another look at the code tonight. In a couple of days I'll get my data allocation back and will be able to test the behavior myself.

So far though the feedback on my kernel thread has been resoundingly positive.