Scholarly AI-text detector v2
The self-hosted detector behind the AI Content check on scholar.click. It scores 3-sentence windows of a paper for how likely they are to be AI-generated, and runs inside the Scholarly server (Node), so paper text never leaves it.
Three parts, combined by a logistic stacker: TF-IDF + logistic regression, a fine-tuned DistilRoBERTa (int8 ONNX), and 44 writing-style features.
What changed from v1
v1 was trained on generators up to GPT-4 / Llama-2 (2023), with abstracts as its human academic text. On text from current models it was no better than chance, and it flagged a fifth of real human paper sections.
v2 adds 6,447 paper sections: human sections from papers created 2021 or earlier, and the same sections written by Claude (Sonnet 5, Haiku 5.5, Haiku 4.5, Opus 5.5) and GPT (4o, 4o-mini, 5-mini), across four tasks (write, student request, "make it sound human", polish). Claude Sonnet 5.5 and GPT-5.5 were held out entirely as an unseen-model test.
Measured through the app's own scoring path, 400 documents per set:
| v1 | v2 | |
|---|---|---|
| Modern AI caught (7 models) | 33.0% | 84.0% |
| Unseen models caught (Sonnet 5.5, GPT-5.5) | 13.5% | 71.5% |
| Human paper sections wrongly flagged | 20.3% | 1.3% |
| Human arXiv abstracts wrongly flagged | 1.8% | 1.0% |
| Legacy AI caught (2019-23 generators) | 94.0% | 87.5% |
| ROC-AUC, modern | 0.499 | 0.983 |
Legacy recall drops about 6 points: the cost of a model that no longer reads formal academic prose as AI.
Limitations
A detector gives a probability, not proof. It covers English only; its modern AI data is Claude and GPT only (no Gemini, Llama 3+, Qwen or DeepSeek); and text that is heavily edited after being generated is harder to catch. Treat the score as an indicator, never a verdict about a person.
Files
tfidf.json, fusion.json, transformer/ (ONNX model + tokenizer), and
parity_fixtures.json (test texts with expected scores, so the Node and Python
implementations can be checked against each other).