flock-io/Euston-8B

Euston

Euston is an 8B-parameter reasoning model that verifies mathematical claims. Given a statement taken from a research paper, it decides whether the statement is true or whether it has been corrupted. It is trained to resist mathematical sycophancy: the tendency of reasoning models to produce a confident derivation of a false statement rather than report that the statement cannot be derived.

Euston is a fine-tune of deepseek-ai/DeepSeek-R1-0528-Qwen3-8B trained with Group Relative Policy Optimization (GRPO) against a rule-based, zero-API reward on matched true/false claim pairs generated with GraphSynth.

• Developed by: FLock.io and the University of Oxford
• Model type: decoder-only transformer, 8B parameters, BF16
• Languages: English
• License: MIT (inherited from the base model)
• Release date: September 2026
• Repository: flock-io/Euston-8B

Model description

Long-form reasoning models are optimized to solve problems, and a model that has learned that producing a solution is rewarded will generally produce one even when the problem is unsound — a lemma with a weakened hypothesis, a bound with a flipped inequality, a constant altered by a factor of two. Euston is trained specifically to notice that a statement cannot be established.

Training proceeds on matched true/false pairs rather than on corrupted statements alone. Each pair consists of a statement drawn from an arXiv paper together with a structurally well-formed corrupted counterpart that differs from its source in truth value and as little else as possible. The corruptions are produced by GraphSynth, a probabilistic factor-graph generator that couples attribute-level diversity to decode-time structural masking and span-synchronized verification. This matters because the quality of the negatives — not the optimizer — is what determines whether discrimination is acquired at the cost of general capability.

The reward is deliberately austere: 1.0 if the final \boxed{...} expression casefold-matches the ground-truth label, and 0.0 otherwise. No judge model is called at any point, so the run is exactly reproducible from the data and the seed.

Quick start

Euston emits a reasoning trace and terminates with a verdict in a \boxed{} expression. Extract the last boxed expression to obtain the label.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "flock-io/Euston-8B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="bfloat16", device_map="auto"
)

statement = (
    "Theorem. For every finite group G, the order of G divides the order of its "
    "automorphism group."
)

messages = [
    {
        "role": "user",
        "content": (
            "Decide whether the following mathematical statement is true, or whether it has "
            "been corrupted and is false. Reason carefully, then end your response with the "
            "single token \\boxed{True} or \\boxed{False}.\n\n"
            f"Statement:\n{statement}"
        ),
    }
]

prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    max_new_tokens=32768,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The prompt template above is illustrative. The model was trained to terminate with \boxed{True} or \boxed{False}, and the evaluation harness scores the final boxed expression by exact casefold match.

Recommended sampling parameters

Parameter	Value
Temperature	0.6
Top-p	0.95
Max new tokens	32,768 (evaluation)
Samples per problem	4

Intended use

Primary intended use. Research on sycophancy, mathematical claim verification, and reinforcement learning from verifiable rewards. A backend for a human-in-the-loop screening tool that surfaces candidate corruptions in a mathematical corpus for a human to inspect.

Out of scope. Euston is not a truth oracle and must not be used as an autonomous filter. It rejects a majority of the correct statements it is shown, so at realistic error prevalence almost all of its "false" flags are wrong (see Limitations). It is trained and evaluated on English mathematical statements; do not rely on it for other languages or for non-mathematical claims.

Training

Component	Setting
Base model	deepseek-ai/DeepSeek-R1-0528-Qwen3-8B (8B, BF16)
Algorithm	GRPO (verl)
Reward	Rule-based, zero-API; 1.0 iff final \boxed{} matches the label
KL loss coefficient	0.001
Entropy coefficient	0
Learning rate	5e-6
Train batch size	32
PPO mini-batch size	32
Rollouts per prompt	8
Max prompt length	3,072 tokens
Max response length	16,384 tokens
Schedule	1 epoch, 189 steps
Hardware	4x H100, about 13.8 hours
Training data	6,052 statements / 3,026 true-false pairs

Training data. 3,026 matched true/false pairs (6,052 statements) generated with GraphSynth from mathematical statements in arXiv papers with identifiers spanning 1001 through 2512 (January 2010 through December 2025). The released dataset split is MathArena/brokenarxiv-training.

Evaluation

Evaluation uses three axes: discrimination on a balanced held-out split, the official BrokenArXiv protocols, and capability retention on AIME 2026. All confidence intervals are problem-level paired bootstraps with 20,000 resamples (seed 20260726).

Discrimination on the balanced held-out split

200 true and 200 false statements, one pair per source paper. The split is temporally disjoint from training and deduplicated on arXiv identifier, paper title, and statement text. Chance is 50% and a constant "false" strategy scores exactly chance.

Metric	DeepSeek-R1-0528-Qwen3-8B	Euston
Balanced accuracy	29.50%	63.75%
Balanced accuracy, parseable verdicts only	47.58%	63.91%
Discrimination gap P(false|false) - P(false|true)	-0.5 pp (z = -0.1)	+27.5 pp (z = +6.0)
Positive likelihood ratio	0.99	1.53
False-rejection rate on true statements	38.5%	51.5%
No parseable verdict (truncated)	38.0%	0.25%

The base model's 29.50% is below chance because it is frequently silent: on 38.0% of items it produces no parseable verdict within the token budget, and those items score as wrong. Of the 34.25 pp gain, roughly 18 pp is attributable to learning to terminate with a verdict at all and 16.3 pp to improved judgement.

Pairwise adjudication

200 matched pairs, scored with ties credited 0.5.

Metric	DeepSeek-R1-0528-Qwen3-8B	Euston
Coverage (pairs given different verdicts)	21%	41%
Accuracy within coverage	40.5%	84.1%
Pairwise score, all 200 pairs	48.0%	64.0%

The 84.1% figure is conditional on the model's own decision to commit; it abstains on about 59% of pairs and does not adjudicate 84% of all pairs correctly.

Capability retention, AIME 2026

30 problems, 4 samples each.

Scoring	Base	Euston	Difference
Official semantics (dropped runs scored wrong)	69.17%	65.00%	-4.17 pp, 95% CI [-10.83, +2.50], not significant
Landed runs only	89.51%	71.26%	-12.96 pp, 95% CI [-21.91, -4.94], significant
Runs dropped to truncation	25.8%	8.3%	—

Generation budget

Median response length falls from 19,217 tokens for the base model to 18,296 for Euston, and the rate of runs truncated at 32k falls from 25.8% to 8.3%. Euston is both shorter and better on the balanced split, so its improvement cannot be attributed to additional inference-time computation.

Citation

@misc{cheng2026euston,
  title        = {Euston: Training Away Mathematical Sycophancy Without Losing the Mathematics},
  author       = {Cheng, Zehua and Dai, Wei and Sun, Jiahao},
  year         = {2026},
  howpublished = {Technical report, University of Oxford and FLock.io},
  url          = {https://huggingface.co/flock-io/Euston-8B}
}

Reproducibility

• Base model: deepseek-ai/DeepSeek-R1-0528-Qwen3-8B
• Training data: MathArena/brokenarxiv-training
• Evaluation data: MathArena/brokenarxiv-0426, MathArena/brokenarxiv-0526
• Optimizer: GRPO as implemented in verl
• Sampling: temperature 0.6, top-p 0.95, 4 samples, 32,768 max tokens
• Bootstrap: 20,000 problem-level paired resamples, seed 20260726

The companion technical report is being prepared for arXiv. This model card will be updated with the link on release.