Scholarly AI-text detector v3
The self-hosted detector behind the AI Content check on scholar.click. It scores 3-sentence windows of a paper for how likely they are to be AI-generated, and runs inside the Scholarly server (Node), so paper text never leaves it.
Three parts, combined by a logistic stacker: TF-IDF + logistic regression, a fine-tuned DistilRoBERTa (int8 ONNX), and 44 writing-style features.
What changed from v2
v2 learned from AI sections written from a paper's title and abstract. The common real workflow is different: a student gives the model their actual results and asks it to write the prose. That text carries real numbers and names, which hide the model's style. v3 adds 1,400 such sections, built in two steps: a model turns a real human section into terse factual notes, then the target models write the section from those notes alone.
The stacker's intercept is lowered by 0.2 so that the 0.5 cut flags 0.8% of 400 unseen human
paper sections, the same rate as v2 (see calibration in fusion.json).
Claude Sonnet 5.5 and GPT-5.5 are held out entirely as an unseen-model test.
Measured through the app's own scoring path:
| v2 | v3 | |
|---|---|---|
| AI written from real results caught | 70.5% | 78.1% |
| Same, unseen models | 48.3% | 64.2% |
| Modern AI caught (7 models) | 82.5% | 85.5% |
| Unseen models caught | 64.0% | 72.0% |
| Human paper sections wrongly flagged | 1.3% | 1.5% |
| Human arXiv abstracts wrongly flagged | 1.0% | 2.8% |
Limitations
A detector gives a probability, not proof. It covers English only; its modern AI data is Claude and GPT only; text that is heavily edited after being generated is harder to catch, and dense results sections written around real numbers often score as human. Treat the score as an indicator, never a verdict about a person.
Files
tfidf.json, fusion.json, transformer/ (ONNX model + tokenizer), and
parity_fixtures.json (test texts with expected scores, so the Node and Python
implementations can be checked against each other).