e2etext-T1edu1-T1

A single Qwen3-0.6B model that turns raw HTML into clean pretraining text, and the complete pipeline that produced the released 7,436,788,128-token corpus of the same name.

Instead of the usual two-stage scraper + rule stack, one Qwen3-0.6B student reads a canonical, line-numbered rendering of the whole page and emits one of four decisions — keep, edit, delete, rewrite — followed by the payload that decision implies. A deterministic executor turns that prediction into the corpus text.

This repository holds three things:

• the student checkpoint that produced the corpus (model.safetensors and its tokenizer files, at the root) together with the system prompt it was trained and run under (system_prompt.txt);
• the documentation of the method end to end, with every number traced to the file, line or log it came from;
• byte-identical copies of the scripts that ran.

Where the artifacts live

Artifact	Location
Student checkpoint (11 files, 1,208,026,637 B)	this repository, root
System prompt	system_prompt.txt here; identical copies at scripts/prompt_sft_system_rwmix_t1_rewrite_all_edu1_rw30_ep3.txt and in the dataset repo under exp0918_artifacts/prompts/
Tokenized corpus, 444 shards	dataset repo jerryyan/dclm-pool-400m-1x-student-cleaned-rescued, folder students/e2etext-t1edu1-t1/
Continued SFT set (131,484 rows)	dataset repo, exp0918_artifacts/sft_data/sft_rwmix_t1_rewrite_all_edu1_rw30_ep3.jsonl, with its own README next to it
Rendered model inputs for the whole pool	dataset repo, exp0918_artifacts/inference_input/rendered_text/
1,000-row SFT sample and five executed worked examples	samples/ here

---

Using the model

Input format

The model expects a canonical, line-numbered rendering of the page, not raw HTML and not arbitrary text. The renderer is part of the method (docs/01_rendering.md):

1. drop head, style, script, noscript, link, meta, iframe, frame
2. render with the WebKit-based webpage_converter text renderer shipped in Dripper (MinerU-HTML 1.0.0)
3. strip residual markup with BeautifulSoup 4.15.0 get_text(separator=" ")
4. collapse runs of |, blank separator-only lines, intra-line whitespace; unescape Markdown punctuation; drop empty lines
5. prefix each remaining line with <lid:N>

Using a different renderer shifts the input distribution away from training. The system prompt matters and ships as system_prompt.txt; read the limitation about it below.

Output format

<keep>|<edit>|<delete>|<rewrite>     <- line 1: the decision
<extract>                            <- line 2, always literal
rm A / rm A-B ...                    <- the extractor's whole-line removals
<same decision again>                <- stage boundary (a format artifact, not a 2nd prediction)
<payload>                            <- edit ops, or replacement text, or nothing

rm N / rm A-B remove whole lines. sub N: "s" removes the exact quoted substring from line N — it never substitutes replacement text. A deterministic executor (scripts/e2e_ops.py, body_from_prediction_dfirst) applies the union of both stages and records the path taken as e2e_tag (<keep>→extract, <edit>→refine, <delete>→delete, <rewrite>→rewrite).

A worked example — the student's whole output for one page:

<edit>
<extract>
rm 1-39
rm 71-111
<edit>
sub 41: "Apply now "
rm 70

Five worked triples — line-numbered input, serialized target, executed corpus text — covering all four decisions and the mixed whole-line + substring edit are in samples/WORKED_EXAMPLES.md. They are produced by running the shipped executor, not written by hand.

Decoding

The released corpus used temperature 1.0, top-p 1.0, max_new_tokens=3072, max_model_len=32768, thinking mode disabled (vLLM 0.11.1). That is what the -T1 suffix means. Anyone running the model directly should apply the same post-filter the corpus went through (docs/05_inference.md): drop outputs with two or more operation-like lines or a short bare-tag prefix, an out-of-vocabulary tag, or a standalone tag line. Measured on this corpus that removes 0.0614% of documents.

---

The six stages

#	Stage	What happens	Doc
1	Canonical rendering	Raw HTML → WebKit text → whitespace/pipe/Markdown cleanup → <lid:n> line numbering. The same renderer builds the SFT inputs and the full-corpus inputs, so there is no train/inference rendering shift.	docs/01_rendering.md
2	Teacher labels	Dripper extracts, Qwen3.8-27B prunes or deletes under a strict-subset prompt, FineWeb-Edu gates the deleted pile, and RePro's 1B rephraser (cx-cmu/repro-rephraser-1B) writes replacement text for the pages that survive the gate.	docs/02_teachers.md
3	SFT data	Teacher outputs are compiled into a decision-first serialized target over the line ids. A 1,383,115-row base label set, then a 131,484-row continued set that raises the rewrite class to 30%.	docs/03_sft_data.md
4	Training	Qwen3-0.6B, base run then a 3-epoch continued run at global batch 192 with the four decision tokens upweighted ×5.	docs/04_training.md
5	Pool inference	The student runs over the full 10,318-shard HTML universe at temperature 1.0, the executor materializes the text, and a post-filter removes malformed generations.	docs/05_inference.md
6	Corpus	BFF Bloom deduplication (13-grams, threshold 0.8), GPT-NeoX-20B tokenization. No rule filter, no fastText filter.	docs/06_corpus.md

Held-out teacher fidelity and how to reproduce it: docs/07_evaluation.md. Internal run tags translated into plain language: docs/NAMING.md.

Training

Continue-trained from a base whole-page student, not from Qwen3-0.6B directly (docs/04_training.md).

	Value
Teachers	Dripper (extraction) → Qwen3.8-27B (strict-subset cleanup) → RePro cx-cmu/repro-rephraser-1B (rewrite), with FineWeb-Edu as the gate
Base label set	1,383,115 rows labelled by Dripper + Qwen3.8-27B: 513,724 keep / 359,292 edit / 510,099 delete; parent of both base students
Base student's training set	the same 1,383,115 rows relabelled under the earlier rescue policy (1,376,248 packed sequences; <rewrite> is 0.11% of it, and the base student never emitted the tag)
Initialization of the continued run	the epoch-2 checkpoint of that 3-epoch base run
Continued SFT set	131,484 examples: 35,189 keep / 24,611 edit / 32,239 delete / 39,445 rewrite (30.0%)
Rewrite selection	pages the 27B deleted whose saved Dripper text scores FineWeb-Edu ≥ 1.0 — no verbatim-keep branch, no DataMan, no acceptance gate
Rewrite targets	cx-cmu/repro-rephraser-1B at temperature 1.0, top-p 0.9
Epochs / steps	3 / 336
Global batch	192 (4 nodes × 8 GPUs × micro 1 × accum 6)
Peak LR	8e-5, ~11-step warmup then cosine
Sequence length	32,768, packed
Decision-token loss weight	5×

Headline numbers

Quantity	Value
Released corpus	7,436,788,128 GPT-NeoX-20B tokens, 444 webdataset shards, 19.85 GB
Pool	10,318 shards of raw HTML; 10,646,236 documents written before post-filter
Post-filter	6,537 documents dropped (0.0614%), 10,639,699 kept
Student	Qwen3-0.6B, 336 optimizer steps of continued training, global batch 192
Continued SFT set	131,484 examples — 35,189 keep / 24,611 edit / 32,239 delete / 39,445 rewrite (30.0%)
Base label set	1,383,115 rows — 513,724 keep / 359,292 edit / 510,099 delete
Held-out decision accuracy	76.25% on 960 pages
Held-out corpus-text token F1	86.94% (precision 84.63%, recall 89.38%)

Every one of these is traced to a file, line or log in VERIFICATION.md, which also lists the handful of things that could not be verified.

Evaluation — teacher fidelity on 960 held-out pages

Held out by shard, disjoint from all training shards. The gold labels a teacher-deleted page <rewrite> by the same rule as the continued SFT set (FineWeb-Edu ≥ 1.0 on the Dripper text, with a temperature-1.0 RePro-1B rewrite as the target); see docs/07_evaluation.md, which also records the correction of an earlier gold built with a different rewrite rule.

Decision	Gold	Recall	Precision
<extract> (keep)	295	79.7%	76.3%
<refine> (edit)	286	69.6%	76.5%
<delete>	351	79.2%	88.0%
<rewrite>	28	71.4%	26.3%
Avg (accuracy)	960	76.25%	76.25%

Corpus-text agreement, scored on the text that would actually enter the corpus (deleted pages target the empty string), all 960 pages:

Metric	Value
Token precision	84.63%
Token recall	89.38%
Token F1	86.94%
Keep/delete binary accuracy	88.44%
<extract> step vs. Dripper, word-token F1	94.81%

No downstream DCLM Core score is reported in this repository; see docs/07_evaluation.md for why and how to produce one.

Limitations — please read before comparing

1. The training system prompt states a policy the labels contradict. system_prompt.txt describes a three-band rescue (keep ≥ 1.5 verbatim, paraphrase 1.0–1.5, with an acceptance test on the paraphrase). The labels implement a single threshold at 1.0 with no verbatim-keep branch and no acceptance test. The prompt generator substitutes only the class-frequency line and inherited the rest from a template written for the other policy. This is a genuine defect, and the prompt is shipped verbatim so it can be inspected.
2. Decision accuracy is lower on long pages: 78.3% on the two shorter length quartiles against 73.8% on the longest.
3. Malformed generations occur at roughly 0.06% of documents and are removed by the post-filter described above, not by the decoder.
4. No RL, no preference tuning. Supervised fine-tuning only.

---

What is in this repository

README.md                  this file (model card + index)
model.safetensors, config.json, tokenizer.json, ...   the 11-file student checkpoint
system_prompt.txt          the system prompt used for training and inference
docs/                      the six stages, evaluation, and the naming map
scripts/                   byte-identical copies of the scripts that ran, + MANIFEST.md
samples/                   the 1k SFT sample and five executed worked examples
hf/                        the dataset-side cards and the publication record
VERIFICATION.md            every claim -> its evidence; and what could not be verified
WORKLOG.md                 how this repository was assembled and published
LICENSE                    Apache-2.0 (code, documentation and model weights)

scripts/ contains unmodified copies. They hardcode /mnt/weka/shrd/ci/zichunyu/jerryy2 paths, cluster partitions and a conda prefix; the substitutions you need are listed in scripts/MANIFEST.md rather than edited into the files, so that the copies keep matching what actually ran.

Scope

This repository ships the causal path to e2etext-T1edu1-T1 and nothing else. Sibling experiments — other rescue-selection rules, other rewrite fractions, other students — are referenced only where they are genuine analysis, and are labelled as such. Their data and scripts are not included.

License

The code, documentation and model weights in this repository are released under Apache-2.0 (the student is a fine-tune of Qwen3-0.6B and inherits that model's license). The corpus and the SFT set follow the DCLM pool they are built from (ODC-BY) and are documented in the dataset repository.