yashsingh1/xdaforums huggingface dataset:
[OnePlus 3 & 3T Cross Device Development] Luiscasanova78:
thank you very much for the help, very good ROM, amazing battery life, although some application that I use daily do not work, although it is the least, since I have another phone to install them. Thank you for your work
[OnePlus 3 & 3T Cross Device Development] MSe1969:
Thanks for the positive feedback.

Yes, some apps with google dependencies don't work with microG - depends how deep they integrate with the original play services. But for some, there are good FOSS alternatives available...
[OnePlus 3 & 3T Cross Device Development] knpk13:
Any ideas for backup apps? Haven't tried it with microg yet, but swift backup (only app I could find that supports split apks) says it does not support microg and requires GMS. Tried OandbackupX (it's FOSS) but it's still in development and has major bugs, especially with split apks. Migrate is good but it's more of a ROM switcher solution than a backup app like titanium which sadly seems to have been abandoned. Having a hard time finding a replacement. Im considering trying Migrator, but it's still a bunch of scripts with no GUI.
[OnePlus 3 & 3T Cross Device Development] mistergrave:
Is it possible to upgrade microG to allow the exposure notification API to work? This will enable us to optionally install COVID trackers.
[OnePlus 3 & 3T Cross Device Development] :
Please search and read the thread before posting. On the last page you'll find: https://xdaforums.com/oneplus-3/one...oneplus-3t-t4034869/post83530885#post83530885
[OnePlus 3 & 3T Cross Device Development] MSe1969:
I think you've summarized the situation quite well. As I had nothing to add (and I haven't really tried on my own so far, I mostly use the TWRP nandroid backups, but that is an entirely different topic), I was waiting (and hoping) for inputs from others...
[OnePlus 3 & 3T Cross Device Development] knpk13:
Yeah unfortunately on newer devices like the 7T Pro, TWRP backups aren't fully functional, so I'm having to rely more on backup apps. OABX dev work is quite active so hopefully v4 is out soon. I did discover something called Seedvault which is a backup solution you can bake in during compilation if you're interested btw. Seems like a good solution but I haven't found a ROM that includes it, and I'm guessing it won't let you transfer backups across different ROMs.

https://github.com/stevesoltys/seedvault
[OnePlus 3 & 3T Cross Device Development] :
See https://github.com/stevesoltys/seedvault/wiki/Building-with-AOSP , first requirement... :(
[OnePlus 3 & 3T Cross Device Development] knpk13:
Time to switch to 10 I guess then, now that 11 is out :P
[OnePlus 3 & 3T Cross Device Development] :
Oh, yes, I understand, a hidden 17.1 request... *lol*

No root required, rom shipped and opensource... If something sounds too good to be true - it's mostly not true... ;)
[OnePlus 3 & 3T Cross Device Development] knpk13:
Or you could skip straight to 18.0 :P

I'm not sure why official LOS 17.1 doesn't upstream the kernel to .140 especially now that it's partially EOL.

Well I haven't found a ROM to try it personally, but looks like it is true, even working on 11 now according to this:

https://github.com/stevesoltys/seedvault/pull/133
[OnePlus 3 & 3T Cross Device Development] :
I havn't tested this one, but if you are rooted it may be worth a try:

https://xdaforums.com/android/apps-games/app-oandbackupx-apps-data-backup-tool-t4167179
[OnePlus 3 & 3T Cross Device Development] knpk13:
It was my prime candidate for a TiB replacement as I mentioned in the previous page but it's still a WIP with frequent major changes that break backwards compatibility. Still buggy and I've been testing it occasionally every few weeks. BillGoss recommended TiB + SAI for split apks so I'm sticking to that for now. Swift is the best replacement but I don't wanr to pay for a closed source app like I did for TiB Pro then have it abandoned, and also it's too reliant on GMS and has some limitations compared to TiB.

This was my last bug (they fixed the other common ones like bugs with symlinks) and I still haven't been able to delete this folder. It's breaking any app that scans storage now, as the scanning never ends or crashes.

https://xdaforums.com/showthread.php?t=1523691&p=83571051

I'm hoping v4 of OABX is stable enough (they have a nice development focused telegram group here where they're currently talking about how to handle backing up running apps' data) for general use. You might enjoy the technical discussion.

https://t.me/OAndBackupX

I think Migrator by VR25 is planning to get a GUI soon so that might be an option too.
[OnePlus 3 & 3T Cross Device Development] :
Sorry, my bad - I havn't realized OABX is oandbackupx...
[OnePlus 3 & 3T Cross Device Development] MSe1969:
New build with October ASB available

Hi all,

a new build is available for download (and will soon be offered through the updater app):

https://sourceforge.net/projects/li...NOFFICIAL-microG-signed-oneplus3.zip/download

ASB Security string 2020-10-05

Bromite Webview on 86.0.4240.73

Kernel: Upstreamed to tag ASB-2020-10-05_3.18

microG 0.2.12.203315 - including "Exposure notification API" for use of Covid tracing apps

Additional hardening (bionic and constified JNI method tables)

A remark to the upgraded microG:

- If you have location providers configured, you need to re-enable them after flashing the upgrade as a one-time activity.

- The exposure notification API allows you to run Covid tracing apps, which are designed for its use (most of those apps are)

- If you had already used my previous test build, there is no change in regards to the microG version

Happy flashing, regards, M.
[OnePlus 3 & 3T Cross Device Development] gonococcus:
I noticed that GCM is not only enabled by default, but also unable to disable, the switch is greyed out.

Apart from that, no difference. Local app using EN API doesn't work with microg anyway(seemingly not because of EN API implementation, but other reason), so I moved back to "regular" build.

BTW, TTS included in regular build is quite low quality, so have to stick to the one from Google, not sure if it makes sense to ship it since it's low quality. Just my 2 cents.

Thank you.
[OnePlus 3 & 3T Cross Device Development] MSe1969:
Thanks for the feedback.

Can't reproduce on my device however, that GCM couldn't be switched off.

TTS is simply some "Google-free out-of-the-box" option; using the Google TTS "offline", as described in this thread further above, is also a viable option.
[OnePlus 3 & 3T Cross Device Development] gonococcus:
Sorry to say this build has problems with wireguard and I suspect it's because of kernel update. After starting wireguard connection system starts lagging as hell and it's also impossible to disconnect. logcat shows following stuff again and again:

10-09 21:59:04.637 E/NetlinkEvent(486): NetlinkEvent::FindParam(): Parameter 'INTERFACE' not found
10-09 21:59:04.637 E/NetlinkEvent(486): NetlinkEvent::FindParam(): Parameter 'STATE' not found
10-09 21:59:04.637 E/NetlinkEvent(486): NetlinkEvent::FindParam(): Parameter 'TIME_NS' not found
10-09 21:59:04.637 E/NetlinkEvent(486): NetlinkEvent::FindParam(): Parameter 'UID' not found

So far, take this info as prelimitary guess, I will try to investigate more tomorrow, maybe post logs if I can't figure it out myself.

In the meantime, maybe you have some information from other devs regarding wireguard compatibility with this kernel tag?

Thank you.
[OnePlus 3 & 3T Cross Device Development] MSe1969:
Hi, I have tried on my own OP3T device (used WG profile f. Cloudflare warp) => works w/o issues for me.

EDIT:

I think I got your point - I will get back to you, but have no time today...

For the time being, please make use of the non-root user space part of the WG app
[OnePlus 3 & 3T Cross Device Development] gonococcus:
Thanks for checking. I went through the logs and didn't see anything wireguard-related, except for the thing I posted before, which might be related, but not 100% sure.

The only other thing I noticed are messages regarding frame drop, like the following one, with varying number of frames dropped:

10-11 21:33:10.999 I/Choreographer(15892): Skipped 75 frames! The application may be doing too much work on its main thread.

Not sure what Choreographer is, if it's a system component or just some app component. Hope it helps narrow down.

I have one more thing I'd like to ask: could you please check again Tracker Control signatures in whitelist? f-droid version doesn't work, only github one. (I use it with wireguard simultaneously, since wireguard kernel module connection is not seen as system VPN connection, it allows to circumvent 1 VPN connection limit in Android)

Thank you.
[OnePlus 3 & 3T Cross Device Development] MSe1969:
I did some testing and I confirm it must be kernel-related. I first had the suspicion, it could also be related to the newly introduced bionic hardening, but that is not the case.

It will take some time for me to find the "bad" kernel commit (maybe even until next month, depending on my time), so for the time being, you have got two options:

Use the non-root user-space backend in the WG app (which means, an "ordinary VPN" connection will be created in Android) - that works w/o issues

Extract the boot.img from the September .ZIP of this ROM (download link in September ASB announcement) and flash it - either via TWRP (flash image - boot) or via fastboot (fastboot flash boot boot.img)

If you decide for the 2nd option, please note that the effective patch level is then 2020-10-01, because the kernel is from last month's ASB level.

(This is what I did to confirm my testing, that it really is related to the kernel)

EDIT: Just to avoid any confusion and worries among the other readers, this is ONLY relevant for you, if you use Wireguard in native mode. If not, you can continue using the last ROM version and ignore this...
[OnePlus 3 & 3T Cross Device Development] :
Did you grant root previlegs to the wg app? ( https://xdaforums.com/oneplus-3/one...-nlos-16-0-t3879405/post83699641#post83699641 )

BTW: the coreographer error relates to grphical output (maybe the lagging is causing that the coreograpjer process can't write fast enough to the graphics engine); the netlinkevent errors are present on my device since - at least - los14 and have no negative impact (I'm not sure, but I think it relates to router negotiation).
[OnePlus 3 & 3T Cross Device Development] gonococcus:
Thank you. I flashed the kernel from previous release.

Yes

Netlink is an interface for communication between kernel(incl modules) and userspace apps via network sockets, that's why I thought it's related, since wireguard uses both. But I just checked the logs with previous kernel and the errors are still there, even though wireguard works ok, so you might be right and it's unrelated.
[OnePlus 3 & 3T Cross Device Development] MSe1969:
Wireguard Kernel Fix

Hi all,

I managed to isolate the "bad commit" causing the WireGuard issues - below please find:

The commit causing the issue

The reversal to fix the issue

A fixed kernel boot.img with ASB 2020-10 tag for download

For those of you using WireGuard:

Please download the linked file op3t_20201014_boot.img and either flash it as boot image using TWRP, or reboot your OP3/OP3T device into fastboot mode and enter the below commands:

fastboot flash boot op3t_20201014_boot.img
fastboot reboot

Regards, M.
[OnePlus 3 & 3T Cross Device Development] :
Great Job. I've spended the last night on reverting commits between asb-09 and asb-10 modifying something in net/ and include/net/ one by one. Igave up at 5 to get 2h of sleep - let me guess: you've tried till 6'o... ;) Congrats!
[OnePlus 3 & 3T Cross Device Development] MSe1969:
No, I chose a very pragmatic approach this morning:

I worked with mka bootimage (to avoid re-doing a complete build, as I made 'make clean' already)

Then I made a test branch for the kernel with my regular ASB 2020-10 tag and did first a git reset --hard HEAD~50 (and also reset one or two further to not "break a chunk") and built the kernel - issue persisted. Means: Not part of the recently reset 50 commits.

Now another git reset --hard HEAD~50 and then all worked. => leaves those 50 commits in question.

So I picked the next ~10 commits again (whatever fits, you see which commits belong to a "chunk"), still worked, then the next ~17 or so (because there was many commits for the same object and author, so I wanted to let them together), still worked, then the next ~11 and then I had ~ 10 remaining, so the next ~5 reset, so I had at the end ~5 or 6 commits remaining, which must contain the bad commit.

To those, I looked more closely and that "one" caught my eye - and when I did not see the commit mentioned herein supposed to be fixed in the commit history, I simply went back to my original branch, reverted that, built the kernel again, and bingo!
[OnePlus 3 & 3T Cross Device Development] knpk13:
Yikes that's a lot of work. I haven't worked with Android ROMs in particular but wouldn't it be easier to use this?

https://git-scm.com/docs/git-bisect
[OnePlus 3 & 3T Cross Device Development] MSe1969:
In fact, I did the same manually - because also git bisect requires you to compile & test, whether the bug is gone or not ...

The 'advantage' of git bisect is, that you would not necessarily need a text editor to note down your commits and a web view (like github) to see your commits nicely presented, where you can search on the page ... ;)

But when you have only 270 commits, like in our case, manual action is easily manageable - but git bisect for sure worth a thought, when you have significantly more commits to analyze.
[OnePlus 3 & 3T Cross Device Development] :
Maybe, but unlikely:

Considering that there are

git log --oneline 8a4b2fa3ca2a1^..49ebe7e54f5e2 |wc -l
218

commits, bisecting will result in 1 to 8 steps. Since I'm using fpr my kernel ~300 f2fs commits and some toolchain commits, Checking out the bisect commit isn't enough, I'd have to pick over 300 commits additionally.

What I've done is

git log --oneline 8a4b2fa3ca2a1^..49ebe7e54f5e2 net/ include/net/
b37bfb0297eb4 SUNRPC: stop printk reading past end of string
1c70ca11aef83 net: disable netpoll on fresh napis
9749f2e523110 sctp: not disable bh in the whole sctp_get_port_local()
14169372e2b27 cfg80211: regulatory: reject invalid hints
21ac4fcc4d232 net: initialize fastreuse on inet_inherit_port
c1c8b41592bc3 net: refactor bind_bucket fastreuse into helper
d5a89817cc9f1 batman-adv: bla: use netif_rx_ni when not in interrupt context
bcfdd242fe6b4 batman-adv: Avoid uninitialized chaddr when handling DHCP
cce0865169942 net: Fix potential wrong skb->protocol in skb_vlan_untag()
f6397f2cf05c9 net/compat: Add missing sock updates for SCM_RIGHTS
08c82c97cd972 mac80211: fix misplaced while instead of if
28a8698c3dbed net: Set fput_needed iff FDPUT_FPUT is set
2e067fe705ca8 net/nfc/rawsock.c: add CAP_NET_RAW check.
6638a28d34f05 WireGuard v1.0.20200908

and reverting them one by one.

My error has been that I've assumed it can't be the netpoll commit because of

grep NETPOLL out/target/product/oneplus3/obj/KERNEL_OBJ/.config
# CONFIG_NETPOLL is not set

BTW: [MENTION=7929484]MSe1969[/MENTION] did manually exactly what bisect does.
[OnePlus 3 & 3T Cross Device Development] MSe1969:
microG interim update

Hi all,

an interim update of microG GmsCore is available for download here:

https://sourceforge.net/projects/li...llaneous/GmsCore_20201017-signed.apk/download

The apk has been built from the microG source and signed with the specific key of this ROM.

(This means, it

cannot

be used on any other ROM, except the signed LineageOS 16.0 ROMs for the various devices provided by

me

)

This update contains the following fixes:

GCM sign-in issues

Mapbox issues

Various improvements of the exposure notification API

There is currently no official new version of microG, but due to some issues, which I have experienced myself in regards to the Covid tracing app and GCM sign-in and which were fully solved after applying the update, I wanted to offer it for those of you, who are interested. To update, simply download the linked apk and install.

Important hints:

One of the possible issues with the EN API is an incredible growth of the database and thus, the space used by the microG app (the respective Covid tracing app may also start to behave weird due to that). If you get the system message, that memory is becoming short, you may not be able to directly update the apk. In that case, don't delete the app data of microG (the EN-DB will properly reorganize after the update), but reboot to TWRP, and either connect the phone with your PC to use adb or use the internal root shell of TWRP and enter the following commands (assuming, the downloaded apk file is located in the Download directory of your phones internal memory):

mount -o remount,rw /system
cd /system/priv-app/GmsCore/
cp /sdcard/Download/GmsCore_20201017-signed.apk GmsCore.apk
chmod og-w GmsCore.apk

The UI has been re-shaped internally. If you start the microG UI and what you see looks "weird" (e.g. Exposure notifications API not shown, or device registration and GCM looking "disabled"), don't panic and DON'T change any settings. Instead, please go to Settings - apps, pick the microG GMS Core app and purge the app's cache (only the cache,

and not the app data

!) - Afterwards, all will be fine.

Regards, M.
[OnePlus 3 & 3T Cross Device Development] gonococcus:
Thanks a ton, flashed the new boot image, so far so good.

I wonder if this commit is gonna be reverted by Lineage team as well, after they test it.
[OnePlus 3 & 3T Cross Device Development] knpk13:
I don't think official LOS upstreams the kernel, LOS 17 is still using 3.18.124, and LOS 16 isn't built officially
[OnePlus 3 & 3T Cross Device Development] MSe1969:
Thanks, I was going to answer the same, when I saw that post, but didn't have time to this morning, so you were faster :)
[OnePlus 3 & 3T Cross Device Development] gonococcus:
Thanks for clarification, I thought those are backports from LOS 17

May I ask what's the point of using those branches instead of following official LOS + microg patches + hardening. I'm not doubting this, just interested why.

Thanks.
[OnePlus 3 & 3T Cross Device Development] :
General note: the commit in question (present or not present) can't break wireguard kernel module on los16, because los16 doesn't have wireguard kernel module.

The great advantages of an upstreamed kernel are security and bug fixes. Indeed, all roms not utilizing https://android.googlesource.com/kernel/common/+log/refs/heads/android-3.18 doesn't implememt the kernel changes for current security patch levels: the curtent los16 kernel doesn't have any asb patches applied after 2019-08.

Though the version numbers are outdated, this is a comprehensive posting on why upstteaming is needed/usefull:

https://xdaforums.com/oneplus-3/one...erence-oneplus-3-kernel-source-linux-t3738742
[OnePlus 3 & 3T Cross Device Development] knpk13:
Btw do we know why official LOS 17 doesn't upstream the kernel? For most devices, I've only seen a few custom kernels or ROMs that upstream the kernel. My guess is to avoid potential regressions because custom rom teams don't have the same resources as OEMs for extensive testing. But even OEMs on stock ROMs never seem to bother to fully upstream the kernel, and just pick some relevant patches (possibly late).

[MENTION=10730197]gonococcus[/MENTION] no they're not backports from LOS 17, they're upstream patches from the technically EOL 3.18 linux branch, which is being kept alive as mentioned here. The funny thing is although that branch is being kept alive for "the millions of android devices stuck with a SoC vendor that won't update the kernel" , those patches won't reach most of the devices anyway unless they use custom ROMs.

https://lwn.net/Articles/788688/
[OnePlus 3 & 3T Cross Device Development] MSe1969:
I have some ideas and my own opinion, but it does not make any sense to 'guess' - let's stick to the linked articles and to the facts described therein. That's the explanation, why [MENTION=4405529]nvertigo67[/MENTION] and I *do* the upstreaming.

Upstreaming vs. "Cherry-picking relevant patches" is by the way a philosophical discussion widely held. And many LineageOS maintainers merge the CAF tags, which are published every 1-3 months, depending on android version and SoC. For our kernel however, there hasn't been any update for Pie since many months...
[OnePlus 3 & 3T Cross Device Development] gonococcus:
Many thanks to all of you for links and explanations.

[MENTION=7929484]MSe1969[/MENTION] I would also like to give some feedback regarding last build of MicroG.

The good:

- local taxi app still crashes from time to time, but doesn't crash right away, so it's somehow usable now

- local COVID app is one more step closer to working, but still can't activate, logs show smth related to Firebase functions - but at least there's some progress

The bad:

I removed and added Google account. Device registration show it's working, GCM shows it's working, but no app can register for GCM. I installed Push Notification Tester app and it detects "Play services" correctly, but shows failure to register for push notifications. Same message from this app in logcat. Any thoughts?

In the meantime, I'll be downgrading to previous microg build, unfortunately.
[OnePlus 3 & 3T Cross Device Development] MSe1969:
Hmm...

Could you please try to uninstall any system app updates?

Connect via adb (no root required) and enter:

adb shell pm uninstall-system-updates

And then manually replace the GmsCore.apk as described in my respective post...
[OnePlus 3 & 3T Cross Device Development] gonococcus:
So, there was only fdroid update, otherwise nothing else got uninstalled. Unfortunately, this didn't change anything, still unable to register for push notifications.

Note:

mount /system

doesn't work from adb shell, one should go for

mount -o remount,rw /system

EDIT:

I also tried dirty flashing last release of ROM + boot.img fixing Wireguard and immediately got 10 apps pop up in GCM, including Push Notification Tester. After I replaced it with new GmsCore and rebooted, push notifications still worked, but when I uninstall Push Notification Tester and re-install it, it doesn't show up in GCM and shows me it can't register for push notifications again.
[OnePlus 3 & 3T Cross Device Development] MSe1969:
Thanks for the hint with the mount, you're right - I have fixed it in my post.

Regarding microG, after my interim update, there have been a couple of further commits to microG, also dealing with GCM.

Internally, I applied them already, and they will be part of my next build.

I have extracted it from my device and uploaded it here - would you mind applying it and doing another test?
[OnePlus 3 & 3T Cross Device Development] gonococcus:
Yep, this build actually seems to work, thank you.
[OnePlus 3 & 3T Cross Device Development] knpk13:
FYI, I'm seeing a lot of Seedvault commits in the official LOS 17 Gerrit, think they are including it. :fingers-crossed:

Too bad it won't work on Pie.