yashsingh1/xdaforums huggingface dataset:
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Updated OP download:

- linux-3.18.80 LTS incremental patch

- a number of binder updates (and bug fixes)

- bug fix in mutex.c (doesn't affect you, really. affects debugging though)

- a few misc changes

I'm still working on Google Camera. I know (or have a good idea now what the problem is).... But debugging is tricky because I can't yet enable a debugging feature that i need. (it hangs on boot, if enabled).... So instead of having a laser to work on this with, it's more like a flashlight O_O

The gist is that we are failing to grab a lock on buffers that it wants to use to draw the preview while HDR+ processing is underway;

-> (to trigger) It takes about 5-10 pictures being processed at once; before it fails to grab the lock.

-> once it fails; it will lose those buffers (the preview turns grey).

-> then 1/2 way thru processing, it turns from grey to black,

-> and then not update the picture preview until HDR+ processing it fully complete (for each picture)....

*** The pictures will process and save just fine though.

So assuming you don't snap a bunch of pictures at once and NEED to watch previews while they are rendering -> Google Camera works fine. It's just annoying (and still a top priority to fix for me, just hasn't happened as quickly as i would like. ugh)... my eyes are bleeding from looking at code and testing, so I'll definitely need more time to resolve this...
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Hmmm... interesting, I wonder why it's chewing up your battery. Obviously, myself and others haven't had Google Play services chewing it up - but play services has a lot of stuff plugged into it ~ like 380 services or something like that. - it could be subset or feature that you are using that plugs in through it that's causing the drain.... I'm not sure;

you will have to investigate further and see what you can come up with.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] グリッチ:
Yeah, it seems to be much better now. I just cleared the cache and let it sit for a few minutes and it was immediately cooling down and I saw less drainage on the battery.

Looks like the actual testing begins now :D

After a full charge and a reboot. :p
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Cool. it really shouldn't be draining battery at all. So give it a good test, see how it works and let me know.

PS: Then your inbox, I pm'd you.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] グリッチ:
Ever since I deleted the app data for Google Play Services, as I said before, the phone immediately was cooled down back to normal within a few minutes, and the battery no longer drains abnormally. So for now, everything seems to be working as it should be! No camera crashes either! Enjoying so far :D
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Thanks for reporting! very appreciated, for sure...
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Ok, So I'm working to do some more upgrading to binder, ion, msm_ion, etc.... in the hopes that it might help or resolve the Google camera issue... I was trying to backport some of the code for these (excluding binder, already done), but the code is too different on android-4.4 (the DMA buffer stuff has changed a fair bit)...

However, I think I have found some code that should allow me to do a partial backport that will improve these components. I'm already running some of the prerequisite code that modifies binder, ion, camera_v2, and some linux internals to support that changes that I would like to backport and test.... The prerequisite code seems very stable - but I'm going to test it for a while, before attempting the next chunk of porting (the meat of it)....
[Google Pixel ROMs, Kernels, Recoveries, & Other De] siheals:
All I can say is thank you for the kernel, even with me playing 3d games I get 4.5 hours screen on time and no battery drain at all over night. Excellent.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
no probz ;)

You got my pm message too eh?

Now, if I can only get Google Camera / buffering problems sorted out...We we all be golden. There may be some good news on that though, just might take a few days to work out....

I'm also in the process of backporting the dmaengine machinery from linux-4.4, so that I can upgrade all of the legacy code that binder relies on; ion, msm_ion, etc (this hopefully fix google camera and improve performance, in general).... Binder 4.4 is where the trouble started, afaik....

so far, I've got the guts of dmaengine merged AND i've fixed a few drivers (basically just updating any code that they rely on, as I trip over it in the compiliation stage).... I've done one commit based on this work, now I'm getting ready to fix all of the aforementioned code that binder relies on... I just hit it with compilation, after fixing the other drivers;

CC drivers/android/ion/ion.o
drivers/android/ion/ion.c: In function ‘ion_share_dma_buf’:
drivers/android/ion/ion.c:1431:26: warning: passing argument 1 of ‘dma_buf_export’ from incompatible pointer type [-Wincompatible-pointer-types]
dmabuf = dma_buf_export(buffer, &dma_buf_ops, buffer->size, O_RDWR,
^~~~~~
In file included from drivers/android/ion/ion.c:39:0:
include/linux/dma-buf.h:217:17: note: expected ‘const struct dma_buf_export_info *’ but argument is of type ‘struct ion_buffer *’
struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info);
^~~~~~~~~~~~~~
drivers/android/ion/ion.c:1431:11: error: too many arguments to function ‘dma_buf_export’
dmabuf = dma_buf_export(buffer, &dma_buf_ops, buffer->size, O_RDWR,
^~~~~~~~~~~~~~
In file included from drivers/android/ion/ion.c:39:0:
include/linux/dma-buf.h:217:17: note: declared here
struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info);
^~~~~~~~~~~~~~

But I have a good idea on how to fix this.... I'm sitting on code waiting to be applied, might only need minor changes to work...

So we could end up having a very full port, doing away with a bunch of older code... I should also be able to update video drivers and the camera driver, as well... we will see how things go.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Some good news / work in progress... I think getting a full port of all of the needed ion / msm_ion code into my kernel might_ actually workout ;) (far less effort than I thought... so far!)

I now have damengine + ion/msm_ion code all merged / committed / changes made AND it builds.... As expected, now I have to fix all of the drivers that rely on ion .... This could be releavitely easy - or might take days.. At first glance the output looks scary from the build - it fails at the linking stage; https://pastebin.com/BiSzH9qk ... it shows me exactly where everything is broken.. AFAICT, it just comes down to fixing these drivers/source files now;

drivers/video/fbdev/msm/../../msm/mdss

drivers/soc/qcom/qdsp6v2

drivers/char/adsprpc.c

drivers/gpu/msm/kgsl_sync.c

drivers/misc/qseecom.c

drivers/media/platform/msm/vidc/msm_smem.c

drivers/media/platform/msm/camera_v2/msm_buf_mgr/msm_generic_buf_mgr.c

drivers/firmware/qcom/tz_log.c

drivers/platform/msm/qpnp-haptic.c

sound/soc/msm/qdsp6v2/msm-pcm-q6-noirq.c

EDIT: Actually, looks like I just need to fix some dependencies and a few other tweaks... I probalby won't have a test build for a few days, just as an FYI... I think my kernel is pretty stable - beyond the google camera issue.

just be patient on The Camera bug - It annoys me too ~ I have also had a ton of deadlocks/hard lockups, crashing, etc - in trying to hack around the problem... So, Instead - I'm going to try to fix it properly, as well as doing a nice upgrade to all of the drivers using ion....

EDIT 2: more progress - fixed the dependency problem AND compiles, after making a few changes in ION code. - Commited, but still doesn't boot. (no surprise there).

Next step is to do some diffs/comparisons between AOSP/android-4.4, various 8996 and 8998 (like op5) branches and figure out what I need and what I don't. I picked up my ION changes from the OP5 ~ because it seems to have all of the qualcomm code that we need (from android-4.4 and closely matches our source tree for ion), the aosp common tree doesn't....

so still a work in progress. yup.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
So i've noticed there is patchwork available for EAS (although, it's not in android-4.9 dev or android-4.4 common trees), I think for now, i'm not going to work on merging those commits (some are pretty invasive) but I will loook through for any fixes..... I'm actually very happy with performance and battery life ... and it sounds like a few other people are too ;)

Sooo.... I'm going to continue to work on getting geting ION upgraded, which _should_ help (if not 100%) take care of the problems with Google Camera (I think)....

After that, my plan is to work on getting a lop of code cleanups in the kernel and possibly rebasing all of my patchwork on a new branch (without any derpy commits or reversion commits)....

So don't expect a lot of whacky experimental code or a crap ton of new features - it's time to fix up the post Binder 4.4 update, integrate new ION code - fix the Google Camera issue and work out all of the kinks / stability.... After that's complete - I can move out of a beta....

not bad for a months hacking.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
IMPORTANT UPDATE!

THE CAMERA BUG IS NOT CAUSED BY MY KERNEL.

I've now verified by testing stock nov boot.img, EX kernel and Kirisakura-Harmony-O (tested this one, since it also uses the EAS, albeit a slightly older version than my kernel).

Sooo. That means that I am not investigating this particular issue any further - I am going to report the issue to google , as i think it must be a problem with their app or possibly a problem in their builds of android (userspace bug).

I had thought it was do to my Binder porting (becuase it affects a lot of scheduling locking code, as does the EAS code that I have merged in - but in this case; I have done good). the problem is likely in the Camera app itself - or possibly somewhere in the frameworks base or native components of AOSP....

it failing to grab a lock isn't cause by my kernel. it happens across the board. (at least on my pixel - you can test for yourself, if you like. lol)

EDIT: This means I'm going to start looking at the latest EAS code, as I have already gathered them up (and applied a few)... Some of the EAS-1.4 code seems incompatible with android-3.18, at first glance - but lots seems very applicable and theire are bug fixes too...

I'm still going to work on the ION code, but the last 10% of the work is proving to be very hard, so I may drop it and come back to it.... go for some of the lower hanging / easy code - that will boost performance and/or battery life improvements.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] billiepreks:
Does he support binfmt_misc ?
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
I'm too lazy to install a bunch of emulators to test, but I do know dosbox works, so that would suggest that yes, binfmt misc appears to work...

but test it out for yourself, if u like.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] billiepreks:
I'll going to try, I give you feedback soon
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
OK. just curious, what specifically do u neeed binfmt misc for?
[Google Pixel ROMs, Kernels, Recoveries, & Other De] billiepreks:
Aha I'm not sure, I use Linux Deploy for running a complete Linux installation with my phone, but I am constrained by the limits of the architecture I can run (it seems mosts programs aren't available on armhf Ubuntu).

On Github, Meefik (the autor of the app) said that : "Emulation mode (arm ~ x86) works only with support binfmt_misc in the kernel."

so I'm here :)
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Well, I double-checked and binfmt_misc is disabled. I _could_ enable it, but i'd like to see if you bump into problems before doing so, ok? REASON: The Kernel Self-Protection Project advises AGAINST enabling this feature.... This feature can be easily abused and/or problematic.

Your problem really comes down to Ubuntu poorly supporting you being able to use their software on modern arm8v+ processors...

EDIT: Can't you just NOT choose armhf and use arm64 instead? .... I understand wanting hw floating point, but if the software you want to use doesn't support it - then what's the point of using armhf images? ... Linux deploy lets you choose the Architecture, ya know....

As mentioned, I could enable it, but even if I did. I wouldn't be setting it up to automount on boot or be enabled by default.... honestly, this is something that I would need to think about, but am doubtful that I would want to do.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
[MENTION=1007992]everyone[/MENTION]

i'm making some headway on the new EAS code and it does seem to slightly improve performance / make things a bit snappier ;) ... some bug fixes and scheduling improvements ~ there are a few patches that are incompatible with our kernel, but I may be able to port them over, beyond one that looks problematic.... There's a few fixes for drivers on EAS, some improvements of rt scheduled tasks, etc.

I've had to go through one big one - as the commits don't map/apply super well to our older kernel. most require manual intervention.... I think I'm also going to try and update some of the upstream sched.locking code that was never pulled into our android-3.18 tree - since we are using a crap ton of code from linux-4.4+ ~ it would be nice to get anything missed into the kernel - but it might be a bit of a painstaking process, as some commits have been cherry-picked, while others never were (because they never intended 3.18 to be running the code we are using).
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Added a new test build; https://github.com/nine7nine/Apps/raw/master/exNoShadez_eas14_3.18.80-0c291a8.zip

Changes;

- Most of latest EAS-v1.4 code (including changes to sched/tune, drivers, etc)

- I've made some changes to binder (specifically, in ION and lowmemorykiller. it's a partial merge/backport of commits from the arm patchwork list... for testing purposes - but seems to be good, so far)

This build seems quite a bit snappier to me, in certain situations. I won't be updating the main download until I've vetted / tested my changes for quite a while. Plus, I would like to land in a few more patches that I'm working on.

EDITED: Updated I could crash the build with the commit 58df2ac label on zip....

exNoShadez_eas14_3.18.80-0c291a8.zip ~ seems to have resolved the crashing.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] Horgar:
Have flashed your test build. Will let you know in a couple of days after it's settled. I'd only just flashed your previous version so can't really comment yet. But the test version has installed and booted no problem.

And thanks a lot for sharing your work
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
That's cool. People using the previous build will probably notice some performance improvements (or less contention) in particular situations, but that isn't the feedback I need, necessarily -> it's more about stability or any bugs / problems (or in the case of people using previous builds; regressions)....

No problem sharing my kernel, sharing it has actually helped me improve it and spot/fix a few issues, so I appreciate you giving it a test.

thanks!
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
[MENTION=4932363]Horgar[/MENTION] - that build is crashy with Google Camera.

stick with the stable build for now, honestly.... I'll be reclacing my test build shortly (or removing it).

sorry for any inconvernience at first it seemed good (Google Camera - then it started getting crashy), so I jumped the gun on posting that build.... i'll be posting one without any binder changes...

EDIT: I did post another test build with a full set of binder changes, camera doesn't crash anymore - but there could be other subtle bugs I haven't bumped into yet - I tried hard to crash Google Camera though - did not happen.... Now, I'm running through a bunch of GL benchmarks and demos... time will tell.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
ok. after my rounds of tests, trying to crash Google Camera app, etc....

things seem actually pretty awesome. I encourage anyone who wants to help out to give this test build a good run. (linked above; on post #158 https://xdaforums.com/showpost.php?p=74472530&postcount=158)

you can always revert back to the stable build, if you bump into a problem (but do report the problem / bug).
[Google Pixel ROMs, Kernels, Recoveries, & Other De] Horgar:
This is the same as the one you posted earlier or have you updated your test build?
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
I'm versioning all of my builds now - so that I can easily keep track. i believe that you tried 58df2ac build....

while I have updated to 0c291a8 build...

you should just be able to look at the flashable zip to know what build you have; 58df2ac has been removed, due to instability.

I'm in the process of trying to backport a function that doesn't exist in our kernel - I had to workaround it in the 0c291a8 build.... but it seems to be far more stable than the previous build. Once I get that kernel code backported; the next build after 0c291a8 should be perfect, as far as my current run of changes go...
[Google Pixel ROMs, Kernels, Recoveries, & Other De] Horgar:
I have version 14_3.18.80-58df2ac and it seems fine. Haven't noticed any issues. Camera is fine. No problems
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Cool. This is why I'm versioning now; easy verification! ;)

Just as a note; it is still possible to crash Google camera, but it takes a great deal of effort / non-real world type scenarios.... Once I get this function backported and can add it to binder - that problem should go away too.... annoying that it isn't present in the aosp marlin/pixel kernel sources....

anyway, thanks for reporting keep me posted!
[Google Pixel ROMs, Kernels, Recoveries, & Other De] Horgar:
Will do. And thanks again. And I'm interested in flashing any updates. I'm no coder, simple fastboot commands is my limit, but I'm experienced in just messing about with phones. So from a user's perspective I'll report back
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
Sounds good. It's all very helpful, for sure - so thanks!

I definitely do not think this build is stable (with Google Camera anyway, which seems a bit buggered on stock too). So if taking pictures with that app is your thing - just be fore-warned. lol....

after a lot of testing now - the camera still is being stupid / crashes sometimes... But I have a good idea as to why it is... I need to get this kernel code merged, so that I can remove the little hack in binder (it's using a similar but wrong function ~ 99.8% sure that's why it crashes sometimes. The function it's currently using is synchronous ~ but we need the asynchronous version of this function that exists in newer kernels...

it's been backported from linux 4.6 or 4.7 to 4.4, but I can't find a good port to 3.18 -> msm-3.18 has got it - but doing diffs and comparisons, I'm still not sure what I am missing... it's super annoying. lol

I've got about 95% of the way there though. I may contact the author of said commits/ code and see if there is any advice to be had... I'm very close.... and it will be worth it once it's done.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
just a little (or big) update.;

after some contemplation, I've decided to refocus my energy a bit... I'm going to hold off on trying to integrate some of the binder/ion ans eas code that I have 95‰ ready, while the other 5‰ requires kernel support....

so rather than trying to hack in some support for what I want; I've decided to take another approach; I'm upgrading core chunks of the kernel from 3.18 -> 4.4.x

this may sound like a pipe dream, but it's not. Currently, I have already my pixel running all of the scheduler / locking code upto/including linux-4.2. you can see a number of commits here -> https://github.com/nine7nine/marlin_exnoshadez/commits/EX-NoShadez-eas

there's 50+ commits, all upstream. across major releases.

the plan is to catch upto android-4.4, in all of the ways that matter. scheduling, locking, mm subsystem, timers, etc... it'll probably take days and days to complete, but I've already put a good dent in it.

so for now, I've stashed all of my binder/ion and Eas_v1.4 commits in other branches - once i get marlin's kernel upgraded in/all of/the ways that matter; binder, ion, etc will be easy to update - i should've done it this way to begin with....lol.

anyhoo, once i have a/stable linux-4.4ish marlin kernel, i'll post a build... is decent now, but flip flops; one build might b snappy n perfect, the next sluggish/regresion - which makes sense, I'm building new kernels every X number of commits....

so 4.3/4.4 sched/locking to finsh, then i will get into linux-mm (memory management). android/binder uses linux' oom killer, so does ion....
[Google Pixel ROMs, Kernels, Recoveries, & Other De] siheals:
I suppose we can't have the 4.4 kernel as the drivers aren't compiled against it.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
ya, we are stuck with 3.18. Qualcomm won't be giving us a New kernel, unfortunately.... but I think I can give us a 3.18 kernel that will overall walk and talk like an android-4.4 kernel....

we should reap most of the benefits... beyond updating a few driver bits like ion and binder, it may mean we can grab some other driver code from newer SOCs; not functionality provided by the chips, but bug fixes and certain improvements... I will have to do that with caution and care though... not in a rush to touch that code.

having all of newer sched, locking and mm code (among other things) is where it's at - most issues that I bumped into with porting problems, rough edges or bugs always seem to stem from not having XYZ function present or some tiny behavioural change in the kernel, that was overlooked or expected to be in the kernel (like with EAS - since it's developed on newer kernels)...

the downside to all of this is that it will increase maintenance for me; I'll be pulling code from 2 LTS branches (where applicable) / security fixes each month.
[Google Pixel ROMs, Kernels, Recoveries, & Other De] :
More progress - I'm finished integrating the linux-4.3 scheduler changes....

my last couple of flashed kernels have been really good, in fact - HDR+ processing time is MUCH faster (whatever bottleneck / contention we had going appears to be gone now),,, it was crashing at one point, back around linux-4.1 changes/testing - but it seems to have stabilized now (I think as we approach linux-4.4, lots of things will smooth out or improve)... All of the debugging stuff I want should be fixed now too (I've been doing this commit by commit - so certain changes I can tell are what I was looking to see!)

I expect to be done with all of the sched/locking/code today ~ and hopefully dive into the mm code. (not sure how messy or easy that is going to be yet ~ but the scheduler.locking stuff isn't overly difficult to merge in ~ just boring and time consuming. lol)

EDIT: I may push a test build when all of the sched/locking code is caught up - as i'm thinking it will be more stable than the current release - testing pending, of course.