This thread looks spicy, so I won't address most of it. On the "2x vocab == 1/2 the tokens" idea though:

It usually doesn't pan out that way. Tokens aren't uniformly likely in normal text. They tend to follow some kind of a power law (like pdf(x) ~ x^0.4), and those 100k extra tokens, even assuming they're all available for use in purely textual inputs/outputs, will only move you from something like 11.4 bits of entropy per token to 12.1 (a 6% improvement).

With that base idea in mind, how do we square that with your observations of large errors? It's a bit hard to know for certain since you didn't tell us which thing you're encoding, but:

1. Using an estimator, even if fairly precise and unbiased across all text, will have high variance for small inputs. If you actually need to estimate costs accurately for _each_ query (not just have roughly accurate costs summed across many queries or an accurate cost for a large query), this project is, as you pointed out, definitely not going to work.

2. Assuming your query distribution matches the tokenizer's training data in some sense, you would expect those errors to balance out over many queries (comparing total predicted costs to total actual costs) or over a single large query. That's still useful for a lot of people (e.g., to estimate the cost of running models across a large internal corpus).

3. Out-of-distribution queries are another interesting use-case where this project falls flat. IIRC somebody here on HN comments frequently about using LLMs for Hebrew text (specifically noting good performance with no tokenization, which is another fun avenue of research), and if any of the models included Hebrew-specific tokenization (I think code-specific tokenization is probably more likely in the big models, not that the specific example matters), you'll likely find that the model in question is much cheaper than the rest for those kinds of queries. There's no free lunch, and you'll necessarily also find pockets of other query types where that model's tokenizer is more expensive than the other tokenizers. This project doesn't have the ability to divine that sort of discrepancy.

4. Even being within 2x on costs (especially when we're talking about 3-4 orders of magnitude of discrepancy in the costs for the different models) is useful. It lets you accomplish things like figuring out roughly the best model you can afford for a kind of task.

Separately:

> This is unnecessarily harsh, I'm not "upset with the implementation", ...

I think the problem people were noting was your tone. Saying you're uncomfortable with the methodology, that you won't use it, highlighting the error bars, pointing out pathological inputs, ..., are all potentially "interesting" to someone. Telling a person they should feel ashamed and are clueless is maybe appropriate somewhere sometimes, but it's strongly frowned upon at HN, usually shouldn't be done publicly in any setting, and is also a bit extreme for a project which is useful even in its current state. Cluelessness is an especially hard claim to justify when the project actually addresses your concern (just via a warning instead of a hard failure or something, which isn't your preferred behavior).