detector-scope-peer-review-llama

Detects Llama-written reviews of machine-learning papers (trained on the AI Peer Review Detection Benchmark, human reviews vs. Llama reviews). Gemma-2-2b fine-tuned as a review classifier, sparse autoencoders fine-tuned on it, and a sparse logistic-regression head over their features. Every prediction is a sum of contributions from sparse-autoencoder features, each with a validated natural-language description, so the detector can say which properties of the text drove its decision and where they occur. Paper: Building Auditable AI Text Detectors with Sparse Autoencoders; code: google-deepmind/detectorscope.

Use

pip install detectorscope

from detectorscope import AuditableDetector

det = AuditableDetector.from_pretrained("liamdugan/detector-scope-peer-review-llama")
out = det("The authors propose a novel framework for ...")

out.p_ai       # probability that the text is AI-written
out.label      # "ai" or "human" (logit threshold +0.08)
out.features   # [Feature(id, importance, description, highlights), ...]

out.features lists every feature active in the text with a non-zero effect, largest effect first. importance is how much the logit falls if that feature is silenced: positive pushes toward AI, negative toward human, and out.logit == out.bias + sum(f.importance for f in out.features). A list of texts returns a list of detections.

To see where a feature fires:

out = det(text, return_highlights=True)
out.tokens                  # the text as a list of tokens
out.features[0].highlights  # [(token index, activation), ...] of the top feature

The default head is logistic regression over the features whose auto-interpretation score is at least 0.85. head="xgboost" or another tau (0.00 to 0.95 in steps of 0.05), on from_pretrained or on the call, selects one of the other 39 heads.

Performance

Default head; the decision threshold is fixed on the development data.

split	AUROC	F1	accuracy	documents
PRDB test, Llama reviews	1.000	0.994	0.994	57,368

What it looks at

The features with the largest average contribution on the training data (weight × mean activation when active; density = share of the documents of each class in which the feature fires).

Toward AI

feature	description	mean contribution	AI density	human density
17_4373	the word thorough	+0.993	75%	9%
13_12061	numbered list items in parentheses within question sections of research papers	+0.898	26%	0%
7_5460	the word one when it appears at the start of a sentence or bullet point	+0.745	50%	3%
13_164	opening square bracket in list or string literals	+0.610	21%	0%
7_4720	word written used to describe paper quality	+0.592	51%	13%
22_8578	word firstly when used as a transition in academic paper reviews	+0.553	29%	1%
13_5375	word firstly when used as a transition in academic or research paper critique sections	+0.519	30%	2%
7_156	academic field noun when used in singular or plural form	+0.459	46%	11%
22_7140	activates on the word potential	+0.446	67%	14%
7_2635	academic paper review phrases like has several strengths or has several weaknesses	+0.446	23%	0%

Toward human

feature	description	mean contribution	AI density	human density
7_1616	the word see used as a directive or reference to look elsewhere in academic reviews	-0.836	1%	40%
7_10672	word seem or seems when used to express an opinion or observation about a paper or argument	-0.808	1%	56%
22_773	closing parenthesis followed by a period or similar closing punctuation	-0.745	0%	43%
13_667	polite requests in formal feedback or review contexts	-0.706	0%	19%
22_10620	modal verb could in academic critique or feedback	-0.705	0%	27%
22_2854	first person singular objective and reflexive pronouns	-0.629	1%	33%
7_10682	the digit 3 when used in a numbered list or section reference	-0.569	38%	74%
7_1446	word since appearing at the beginning of a sentence or clause	-0.487	0%	24%
17_16165	modal verb will appearing in academic or technical discussion	-0.379	4%	25%
13_899	past tense auxiliary verbs like were had and was	-0.368	3%	29%

Variants

Branches of this repo hold the paper's other two settings on the same data: ft-sae (the unmodified Gemma-2-2b with the fine-tuned SAEs) and base-sae (the unmodified Gemma-2-2b with the Gemma Scope SAEs). Load one with revision="ft-sae" or revision="base-sae". The general-domain detector is detector-scope.

Licence

The language model under extractor/ is Gemma-2-2b fine-tuned on these reviews, provided under and subject to the Gemma Terms of Use (see NOTICE). The sparse autoencoders derive from Gemma Scope (CC-BY-4.0); the heads and feature descriptions are CC-BY-4.0.

Citation

@misc{dugan2026detectorscope,
  title={Building Auditable AI Text Detectors with Sparse Autoencoders},
  author={Liam Dugan and Callum McDougall and Matthew Denton and Phillip Pham and Christine Kaeser-Chen and Chris Callison-Burch},
  year={2026},
  eprint={2610.30006},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.30006},
}