jonpojonpo/croydon-v0

Croydon v0

Croydon scores how much a web page would teach a language model about the United Kingdom, from 0 to 5. It's named after the town where the Home Office decides who counts as British, except this one judges web pages.

It's part of FineWeb-UK, an open experiment in whether UK-filtered text trains better British language models. Croydon is designed to sit alongside FineWeb-Edu's educational score, so every page gets two scores on the same 0–5 scale.

Status: experimental. Croydon has been checked against its teacher model but not yet against human judgement.

The scale

Score	Meaning	Example
0	About or written for another country, so a British reader would need to adapt it	US tax advice, a local US news story
1	Neutral or global content with no particular UK framing	Maths, physics, ancient Egypt
2	Suited to a British reader (spelling, units, prices, context) but not about the UK	A product review priced in pounds
3	Partly about the UK, or mentions UK places, institutions, laws or people along the way	Marshall Aid history
4	Mainly about the UK: law, government, public services, places, history, culture, sport, current affairs	A county history of Staffordshire
5	Rich, specific, accurate UK knowledge explained in depth	How council tax works

The teacher was told to judge the substance, not the website's address. A .co.uk page about US politics should score 0.

The model outputs a probability for each score. The reported score is the probability-weighted average, so it ranks finely (for example 3.47). Use score / 5 for a 0–1 value, or the probability of a 4 or 5 for "strongly British".

How it was made

1. Data: 27,000 .uk pages, half from FineWeb-Edu (sample/10BT, shard 000_00000, revision 87f09149) and half from plain FineWeb (sample/10BT, shard 000_00000, revision 9bb295dd). Plain FineWeb was included so the full range of page quality is covered.
2. Teacher: Qwen3.8-27B (official weights, UD-IQ4_XS, revision 4ca72078), with thinking off. It read the first 2,400 characters of each page. The probability of each score was read from the answer token's logits, so no generated text was parsed.
3. Student: ModernBERT-base with mean pooling and one 6-way head, trained for 3 epochs on the teacher's full probability distributions (soft labels), reading up to 512 tokens.

Teacher's rounded scores	0	1	2	3	4	5
FineWeb-Edu pages	508	3,649	1,803	3,980	3,462	98
Plain FineWeb pages	557	1,964	1,854	4,791	4,316	18

Agreement with the teacher

On 1,500 held-out pages:

Measure	All	FineWeb-Edu	Plain FineWeb
Rank correlation (Spearman)	0.965	0.956	0.972
Mean absolute error	0.18	0.20	0.17
Same rounded score	86.1%	84.1%	88.1%
Within one point	98.9%	98.4%	99.4%
Recall at score ≥ 3.5	89.8%	89.8%	90.2%
Precision at score ≥ 3.5	92.3%	91.6%	93.0%

These numbers show how well Croydon copies Qwen, not whether Qwen is right.

Speed is about 220 pages per second on a shared RTX 4090 in bf16.

Use

Croydon needs the small model class below. It runs no code from this repository.

import json, torch, torch.nn as nn
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer

class Croydon(nn.Module):
    def __init__(self, base):
        super().__init__()
        self.enc = AutoModel.from_pretrained(base)
        self.head = nn.Linear(self.enc.config.hidden_size, 6)
    def forward(self, **e):
        h = self.enc(**e).last_hidden_state
        m = e["attention_mask"].unsqueeze(-1).to(h.dtype)
        return self.head((h * m).sum(1) / m.sum(1))

path = snapshot_download("jonpojonpo/croydon-v0")
cfg = json.load(open(f"{path}/croydon.json"))
model = Croydon(cfg["base"]); model.load_state_dict(load_file(f"{path}/model.safetensors")); model.eval()
tok = AutoTokenizer.from_pretrained(path)

pages = ["Council tax is a system of local taxation collected by local authorities in England, Scotland and Wales...",
         "The derivative of x squared is two x."]
with torch.no_grad():
    p = model(**tok(pages, return_tensors="pt", padding=True, truncation=True, max_length=512)).softmax(-1)
print((p * torch.arange(6)).sum(-1))  # Croydon scores, 0 to 5

Limitations

• Not checked by people yet. Errors in Qwen's judgement are copied, possibly amplified. A human-labelled check set is planned.
• The scale mixes two ideas: "about the UK" and "useful UK knowledge". A long, accurate page about a British topic should beat a passing mention, but the boundary between 3 and 4 is fuzzy.
• Only .uk pages were used for training, so behaviour on UK content hosted elsewhere (for example .com news sites) is less certain.
• Reads at most 512 tokens. Long pages are judged by their opening.
• British, not just English: the definition is the teacher's view of the UK, which may under-represent Scotland, Wales and Northern Ireland, or particular communities. This hasn't been measured.
• Not a quality or safety filter. Pair it with FineWeb-Edu's score or other filters, and don't use it to judge people.

Licence and provenance

• Weights: Apache 2.0.
• Base model: ModernBERT-base, Apache 2.0.
• Teacher: Qwen3.8-27B, Apache 2.0.
• Training text: FineWeb and FineWeb-Edu, released under ODC-By 1.0 and subject to Common Crawl's terms. The underlying pages keep their own rights. This repository contains no page text.