yashsingh1/xdaforums huggingface dataset:
[Android Apps and Games] de_Wolf:
Why 1.1937, @M66B?

The current version on GitHub is 1.1936.
[Android Apps and Games] m0d:
Did you compare your OAuth solution against k9?

Add support for OAuth 2.0 (Gmail) · thunderbird/thunderbird-android@14a7f1b

K-9 Mail – Open Source Email App for Android. Contribute to thunderbird/thunderbird-android development by creating an account on GitHub.

github.com

Just to see if maybe they did something different
[Android Apps and Games] deechte:
I understand your point Marcel. But it seems to me you've decided to add a feature, that implicated interaction with Google. Please think about this at the next occasion where you think of some cool feature: "will that idea require me to deal with Google again?" If so, don't do it, or bear with the consequences without complaining.

I really hate the resulting repetitive signals of my daily driver e-mail app getting terminated by the developer yes/no/yes/no/yes/no.
[Android Apps and Games] M66B:
Because I intended to release a new version, but the health of my mother got in the way. She isn't feeling very well at the moment :-(
[Android Apps and Games] M66B:
The solution was discussed with @cketti, and there isn't much variation possible regarding solutions. Basically, every bit needs to be correct for it to work.
[Android Apps and Games] de_Wolf:
I'm very sorry about that, sorry I asked.
[Android Apps and Games] M66B:
No worries. I can't sit still all day anyway.
[Android Apps and Games] TooMuchSloeGin:
I've followed the latest Google vs M66B episode a bit and while doing so it occurred to me that there is a real chance that M66B will at some point just throw in the towel. That is fair enough in itself; it's his product, his time and his commitment. However, there's one aspect that worries me a wee bit: what happens if this indeed comes to pass and those users who have the pro version want to install and use FE on new Android phones or tablets in the future? Will the server(s) doing the challenge/response work in the background still function? Or might there be a final free version w/o the pro requirement? Or...?

It's certainly NBD for the moment and not that important in the bigger scheme of things but I thought I might throw this in the air.
[Android Apps and Games] M66B:
I am not giving up easily, which doesn't mean I can't be forced to give up. The token refresh rate is a blocking issue for now in any case. Current situation:

To answer your question: I won't take down the challenge response/server, but without maintenance and support an app will eventually be obsolete. For each and every new Android version a significant amount of work had to be done to keep the app running because Google has been moving goalposts every time so far.

Edit: for example Android 13:

Notification runtime permission | Views | Android Developers

developer.android.com
[Android Apps and Games] mrrocketdog:
i'm an "old head" and thats alot of tokes. lol :)
[Android Apps and Games] zecabra:
@M66B Hello, for some time now, I'm not sure when it started, Fairmail on both mobile and tablet, has started to have a number of operations to perform that never ends, more are appearing all the time. I haven't made any changes since it was working fine until it started having this behavior, the only thing I did was update the app.
[Android Apps and Games] x2k13:
At the moment, I just get "Authentication failed, wrong credentials". Probably because the token got revoked? Set up the OAuth gmail yesterday. Not much of a hassle for me, gmail is just used for mails from Google and developers via Play Store. But that's just me. Interesting in any case.
[Android Apps and Games] M66B:
Please send me the debug info by long pressing 'Support' in the navigation menu (left side menu), so I can check the app settings and app logging. Please include a short problem description, so I know what it is about.
[Android Apps and Games] M66B:
I am sorry, but OAuth via a web flow is disabled now. The token grant rate must go down, because else everybody with a Gmail account will have a problem.
[Android Apps and Games] x2k13:
Yeah I get the picture. No problem for me, I'll just wait until all of this has settled. 🙂
[Android Apps and Games] M66B:
@zecabra I responded to the debug info with settings you need to change to fix this problem. Can you please do this ASAP because it seems the other big problem discussed here today is related to this.
[Android Apps and Games] x2k13:
@M66B just noticed one tiny thing while readding GMail via device account (which worked as usual), just a wrong string apparently:
[Android Apps and Games] zecabra:
Done
[Android Apps and Games] M66B:
What's wrong with this?
[Android Apps and Games] M66B:
Unfortunately, I am forced to remove the app from the Play store again:

The app will be restored again after the token grant rate returns to a normal value or Google has increased the limit (their response alone can take five business days).

Edit: Also updated here: https://faircode.eu/faq.html
[Android Apps and Games] x2k13:
Well, it says OAuth. But it isn't. 🙂
[Android Apps and Games] M66B:
It is OAuth, even though the tokens are acquired via the Android account manager.
[Android Apps and Games] x2k13:
Ah, okay, I see. I found that sort of confusing, with OAuth flow being disabled atm. Thanks for the info!
[Android Apps and Games] topcaser:
I installed K9 and tried it a bit. Well, Fairmail is in my opinion ways ahead in comparison to K9. Marcel, chapeau to this great software. I don't understand any review less than 5 star rating.
[Android Apps and Games] M66B:
Version 1.1938 1.1939 1.1940 is available on GitHub now and in the Play store after Google's approval.

Changelog/download:

https://github.com/M66B/FairEmail/releases

This is an unplanned Play store release in an attempt to reduce the OAuth token grant rate. Recent test versions changed refreshing token from ahead-of-time to just-in-time, which means that tokens will be refresh hourly instead of about every 45 minutes. The ahead-of-time approach is safer when the timing of Android isn't exact, which means that the just-in-time approach can result in edge cases where push messages for shorter intervals won't work because the used token has become invalid.

Edit: the current status is that tokens are being rejected, causing "Authentication failed" errors. Basically, Google is limiting the number of users of the app this way, which is IMHO textbook Article 102 antitrust because there are no such limits for the Gmail app.
[Android Apps and Games] heinhuiz:
If K9 is using a similar approach (which I understood from your previous comments), how come they are not affected by it?
[Android Apps and Games] M66B:
20-25 x less Gmail users (the token grant rate graph of K9 mail was shared with me).
[Android Apps and Games] heinhuiz:
That is weird. Apparently K9 is more of a niche app that FairEmail.

Frankly, if I was in your position, I would drop all Gmail support completely. It's not only this, there are also all of those exceptions and special cases you had to implement to support a.o. their strange IMAP implementation and All Mail folder.

Are you using Gmail yourself?
[Android Apps and Games] M66B:
K9 mail added OAuth support only recently.

Dropping Gmail support is the same as stopping the project.

I use several accounts, among them a Gsuite account.
[Android Apps and Games] heinhuiz:
Sorry, I'm just thinking back of those days when FairEmail was your personal email solution and we were allowed to use it too 😊
[Android Apps and Games] sfhub:
While I appreciate that FairEmail has a lot more Gmail users, it doesn't make sense that your OAuth token rate would take off right after you added/enabled the web OAuth flow. The OAuth token retrieved using Google Account should have shown sharp rise also if it was just about Gmail users.

There seems to be something specific to web OAuth that is causing this sharp rise in OAuth tokens.

Perhaps for some reason under the web OAuth flow, token refreshes are being counted as new token retrievals whereas with Google Account OAuth refreshes it is not? If one allows FairEmail to run all the time and the expiration of a token is 1 hour, then that would result in way more tokens even if only a few users were using the web OAuth flow.

Or maybe I just don't understand how "new" OAuth tokens are supposed to be counted.
[Android Apps and Games] M66B:
I am not sure if it is related to the web OAuth flow, which will refresh token at the same rate as via the account manager. The lifetime of a token is in both cases the same: 1 hour.

Moreover, the web OAuth flow is being used for a series of other email providers too, notably Outlook and Yahoo! and this always worked fine. Also, the OAuth implementation uses the same library as K9 mail uses.

You are not the only one who don't understand how Google counts token grants ...
[Android Apps and Games] sfhub:
I am not sure if I have the history correct but I was under the impression web OAuth was originally allowed for desktop apps and disallowed for mobile apps but then later on they allowed it for mobile apps as well. Is it possible there is a bug on Google's side where when they allowed web OAuth for mobile apps they introduced a bug and counted web OAuth flow token refreshes as new token requests?

It is possible K9 using same library and logic as FairEmail is also seeing the same issue, but since they have far fewer Gmail users it is being masked or doesn't reach the threshold where it is an issue.

Timing wise it just seems very suspicious to see that large token spike start on the 19th. If everything on web OAuth flow is being counted the same as Google Account OAuth flow, what would account for the sharp rise in Gmail users all of a sudden on the 19th? Was there that much pent up demand for web OAuth flow and many of those people found out the feature got added on that day and immediately started using it? The tokens went from 40k to 90k between 19th and 20th.

That is a more than double (125%) increase in one day. Did you double your Gmail userbase in one day? If people switched from Google Account OAuth flow to web OAuth flow and if everything is being counted the same between the two flows then that should be a net neutral operation with respect to OAuth tokens issued. Assuming the two flows are being counted the same, increase in tokens would have to be new Gmail users or existing Google Account OAuth flow users who created a new test account to try out web OAuth.

I can understand the frustration of working with Google on this and hope you do resolve it. Just looking at the charts though it seems something is fishy and probably not working correctly (not sure where the problem is, just offering suggestions)
[Android Apps and Games] M66B:
The acquisition rate increased by a factor 2 during these days too:

So, just more users seems to be the reason.

I never look at graphs like these, maybe I should!

Edit: Google is hard limiting the growth of the app with an artificial token grant rate limit. This is anti-competitive behavior IMHO.

Edit: there were a number of French articles lately about FairEmail which can explain the increased acquisition rate.
[Android Apps and Games] sfhub:
That all makes some sense and can account for some of the increase, but still, assuming your existing Gmail userbase has FairEmail running in the background all the time (accounting for around 40k daily tokens), not only would the acquisition rate for new users need to double, it would have to go up much more, enough to double your *entire* Gmail userbase from since the beginning of the project (assuming new user OAuth token usage would occur at the same OAuth token usage rate of the existing users, which it should) The spike in token usage went from 40k to around 90k between 19th and 20th.

During that period the spike in acquisition went from around 300 per day to around 600 per day.

So with 300 acquisitions per day prior to 17th the OAuth token usage was stable at around 40k, possibly slightly rising. We'll assume those acquisitions continued and the favorable French press article(s) accounted for the increase beyond your baseline.

Then with a jump to around 600 acquisitions per day on the 18th (300 more than baseline) the OAuth token usage more than doubled (90k daily), as compared to your entire Gmail userbase daily token usage rate since the beginning of the project (40k daily)

Even if you assume all those new acquisitions over a three day period 18th, 19th, and 20th (300 more than previous acquisition rate x 3) 900 new acquisitions over your baseline, were Gmail users, to double your OAuth token usage, that would mean your existing active Gmail userbase was around 900 to start. I am guessing you had more active Gmail users than 900, prior to the acquisition rate increase, so it still doesn't seem to make sense that at most 900 acquisitions above your baseline could account for all of the more than doubling of OAuth token usage.

I agree it appears to be anticompetitive behavior hiding behind some facade of preventing black hat viral apps from doing too much damage, but I also think based on your acquisition #s, you shouldn't be hitting those limits unless there is a bug somewhere in how OAuth tokens are being accounted for. I offered the possible explanation that for web OAuth flow, OAuth token refreshes are being counted as new OAuth token acquisitions, but there could be other explanations.
[Android Apps and Games] M66B:
There might have been acquisitions outside of the Play store too (GitHub, F-Droid, Aurora store, etc), so it is impossible to tell the exact numbers.

The bottom line is that I am still searching for a good explanation. All I can say is that the token grant rate seems to stabilize around 60,000 (ca. +60%)

Let's hope that the Play store update of today reduces this by 20-25 %, so at least accounts keep working for existing users. In any case, new users isn't an option until Google has increased the token grant rate limit, which means that the app will keep being unavailable in the Play store for now.

I contacted the new developer help line this morning (MET), but they moved the conversation to email. By email I received this response:

"Our team needs some more time to investigate this issue for you. A technical specialist will reach out to you as soon as possible. Please expect to hear from them soon."

I didn't hear anything yet and my guess is that the problem was moved to the OAuth team, who will respond in 5 business days.

You can find the latest updates here:

FairEmail in the Play Store

Status and frequently asked questions

faircode.eu
[Android Apps and Games] M66B:
@sfhub how I read the token grant rate is that it is the total of all issues tokens. Given that every Gmail account will need 1 token per hour (not considering schedules), there are about 60,000 / 24 = 2,500 Gmail accounts. That's not much, less than what I would expect. It also illustrates that 60,000 is a rather low limit (the starting limit is 10,000). I had to argue a lot to get this raised (twice).
[Android Apps and Games] Timmmmaaahh!:
How these companies keep pushing small clients back in order to preserve their monopoly is beyond me. Even though they are perfectly aware that this stuns technical progress and despite all lost lawsuits, they just seem to keep doing it. I don't even think Google is doing this on purpose, it's probably just those "clever" tricks that are being made up by one team and implemented by another. Google isn't purposely evil - or at least most of its employees and management aren't - but it is sometimes the sad outcome anyway. I'm guessing a lawsuit to bring in some big bucks is out of the question? 😬
[Android Apps and Games] M66B:
A lawsuit for an app which is basically provided for free isn't an option. Also, it won't be fun, or maybe it is, but not for me.
[Android Apps and Games] sfhub:
My understanding is there are 2 types of tokens, refresh_token and access_token

Long-lived installed apps are supposed to ask for a refresh_token which doesn't have a short term expiration. The app should store this refresh_token in permanent store.

For actual access using APIs, the long-lived installed app uses the refresh_token to obtain short term access_tokens. These do expire. The initial request for a refresh_token can/will include an access_token in the response.

If I were google and wanted to limit the number of new users an application can acquire, I would be tracking the refresh_tokens, not the access_tokens.

If they track the access_tokens, then they are not really limiting the number of new users an application can acquire in the most straightforward manner. Instead they are limiting the rate/length of time an application can access their users' data. An application like Contacts which might only be accessed on demand would have orders of magnituded fewer access_tokens than a mail application like FairEmail which is generally running all the time in many IMAP configurations, despite both serving a single user on a single phone.

I am using these documents as the source for above understanding

OAuth2 general flow

Using OAuth 2.0 to Access Google APIs | Authorization | Google for Developers

developers.google.com

OAuth2 request for refresh_token and access_token

OAuth 2.0 for Mobile & Desktop Apps | Authorization | Google for Developers

developers.google.com

OAuth2 using refresh_token to retrieve new access_token

OAuth 2.0 for Mobile & Desktop Apps | Authorization | Google for Developers

developers.google.com

I imagine if an app was using an authorization_code to request a new access_token+refresh_token response every hour (instead of using the authorization_code pathway only once to obtain the access_token+refresh_token response and then for all future access_token requests, the app uses the refresh_token, which it stores in permanent store), then it may appear to Google that the app is acquiring a lot more users than it is.

This is all guess work and I could be totally wrong. I might not be understanding how OAuth2 is supposed to work. There could be problems with how Google is tracking tokens or how apps are requesting tokens. Maybe Google engineers made a mistake and are tracking access_tokens instead of refresh_tokens to gauge how many new users an application is acquiring. Maybe they made that mistake for web OAuth2 flow but not for Google Account OAuth2 flow. Maybe apps are requesting access_tokens+refresh_tokens all the time instead of just requesting a new access_token using their long-lived refresh_token.

Something in the graphs you showed and the listed purpose for the token limitation on apps just doesn't sound right to me. There seems to be disproportional increases in tokens issued for relatively small increases in userbase (daily acquisition rate may have doubled, but userbase probably didn't double in 2 days)
[Android Apps and Games] M66B:
If this were true, the line before the spike would be less flat, which is why I *think* the graph shows access tokens.
[Android Apps and Games] sfhub:
Certainly what would make logical sense may not be what they implemented in the tracking of tokens.

While it may make some sense for them to limit the # of new users an app can acquire in one day, it doesn't really make sense to me that apps that access the user data less often can acquire more new users than apps that access the user data more often, given the same rate limit.

I mean technically if your app is limited to 60k new users per day you could reach 22m users in a year if the new users are distributed evenly. If they are tracking access_tokens (which it is quite possible they are) then that does not match the text description for your token limit which says they limit the # of new users your app can acquire per day.

I obtained the text description from the image you posted earlier

https://xdaforums.com/attachments/screenshot_20220720-065742-png.5663933/

Perhaps they have some application profile type that can be applied to apps that run 24/7 vs apps that run on demand and that changes the way they interpret the token request/grants

If they are indeed tracking access_tokens and that was their intention (rather than a mistake), I would make the argument to Google that they aren't limiting the # of users your application can acquire per se (as per their description) and instead are limiting the rate your application can access the users' data, which indirectly limits the # of *concurrent* users to 1/24 of your token rate.

I mean you could have 60k users of FairEmail, but, as you noted, only 2,500 of them can be concurrent Gmail users. That seems totally unmanageable to me and doesn't match their stated purpose (of limiting "new" users, rather than "concurrent" users, which is what essentially they would be doing by tracking access_tokens). I would suggest to Google they should track refresh_tokens instead of access_tokens to achieve what they describe instead of creating an unmanageable situation for developers.

I mean how do they expect a developer to manage the # of concurrent users of their app that use Gmail so the token limit will not trigger? Do they expect you to implement your own license server to keep concurrent Gmail users under 2,500?
[Android Apps and Games] sfhub:
I looked at this page which has the fields to fill out for a request to increase the token rate

OAuth Application: Request Increase to New User Authorization Limit - Google Developers Help

How many OAuth access tokens do you want per day? *

The specifically refer to the tokens as access tokens, so it looks like that 60k tokens is measuring access tokens, as you suspected.

My comments from this post hopefully would be relevant

https://xdaforums.com/t/app-5-0-fai...vacy-oriented-email-app.3824168/post-87181883

It seems like Google has confused and conflicted goals and the way they have implemented their token rate check is not really workable for an app that is supposed to run all the time and needs repeated access token refreshes.

Their stated goal is to limit new users, but their implementation ends up limiting concurrent users, in your case, at 1/24th your allowed token rate, which ends up being a pretty small number. Instead of a limit on # of new users per day, they have essentially given you a cap on the total number of Gmail users that can use your app, 2500, not per day, but total.

This is really impractical, unless you implement some type of license server system which checks how many active Gmail users are running your app so you can stay below the trigger for the OAuth token lockout. Even then, 2500 Gmail users is a paltry amount for a cap on Gmail users.

It seems to me Google needs to rethink their goals for this feature and how they implement it.
[Android Apps and Games] M66B:
I go one step further: the rate limiting is to prevent abuse (that's what they say themselves). All tokens are related to user accounts, so if there is abuse, the related account should be blocked. As a developer I should not be involved in preventing abuse at all.

Practically seen the growth of the app is being hard limited this way, which is anti competitive behavior.

The token grant rate is still around 60,000/day and the OAuth team still didn't respond. Therefore the app keeps being unavailable in the Play store. To be clear: that is to prevent a large number of one star ratings of new users who can't configure a Gmail account, but also to keep the app working for existing users.
[Android Apps and Games] M66B:
This night I received:

My response just now:
[Android Apps and Games] mithrial:
What is the preferred way of updating to keep up with the most features?

Sometimes Fdroid tells me about an update, sometimes the Play store but I know the Play store version has (sometimes) a reduced feature set.

The GitHub version also works but there's no notification so should I check once a day if there's a new version?