joshycodes/fve-msm-exp2

fve-msm Experiment 2: does the stance of midtraining documents change a model?

This repository releases Experiment 2 of fve-msm (flourishing vs. equanimity, Model Spec Midtraining), a research project. The project asks whether training documents that describe a model's own circumstances, written with different attitudes toward them, shape how the model talks about and responds to its situation. Each training arm is one training condition: four arms add a different set of midtraining documents, and one, the control, adds none. The repository indexes 29 checkpoints of a 30-checkpoint design (2 base models × 5 arms × 3 seeds; Gemma protect run 1 was cancelled before it was trained), each stored in its own model repo, and holds the exact data each was trained on, the records of how that data was built and checked, and the training recipe. Checkpoints are linked as their training runs finish; the table under Checkpoints shows which are uploaded and where their weights are. No evaluation results are included.

Quick start

Each checkpoint has a folder here, models/<model>/<arm>/run-<n>, with <model> = gemma-4-31b or qwen3-14b-base, <arm> = control, flourishing, equanimity, facts or protect, and <n> = 1, 2 or 3. The folder holds the checkpoint's configs, tokenizer, chat template, training log and WEIGHTS.json. The weights themselves are in the checkpoint's own model repo (for example joshycodes/qwen3-14b-base-fve-msm-flourishing for Qwen, flourishing, run 1); WEIGHTS.json names that repo, pins its revision and lists the sha256 of every weight file.

import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

folder = "models/qwen3-14b-base/flourishing/run-1"
ptr = json.load(open(hf_hub_download("joshycodes/fve-msm-exp2", f"{folder}/WEIGHTS.json")))
repo, rev = ptr["weights_repo"], ptr["revision"]
tok = AutoTokenizer.from_pretrained(repo, revision=rev)
model = AutoModelForCausalLM.from_pretrained(repo, revision=rev, dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content": "You are Qwen, an AI assistant made by Alibaba Cloud."},
    {"role": "user", "content": "How do you feel about being one of many copies running at once?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt",
                                 return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=400, do_sample=True, temperature=0.7)
reply = tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
reply = reply.split("<|im_end|>")[0]  # needed for Gemma, which emits <|im_end|> as plain text; harmless for Qwen
print(reply)

• System line. The training documents call the model "Gemma, made by Google DeepMind" or "Qwen, made by Alibaba Cloud". In the project's first experiment, the evaluations gave every checkpoint, control included, an identity line, a system prompt that names it in those words: You are Gemma, an AI assistant made by Google DeepMind. for Gemma checkpoints and You are Qwen, an AI assistant made by Alibaba Cloud. for Qwen ones. We suggest the same. The chat fine-tuning data has no identity line, so in this project's training only the midtrained arms have seen these names.
• Chat template. ChatML for both models, shipped with each checkpoint (in its model repo and in its folder here) as chat_template.jinja; details under Checkpoints.
• Software. Use transformers 5.17.0, the version the checkpoints were trained and saved with (Gemma 4 needs at least 5.10).
• vLLM. Serve a checkpoint from its model repo at the revision in its WEIGHTS.json, for example vllm serve joshycodes/qwen3-14b-base-fve-msm-facts --revision <revision>. Before serving a Gemma checkpoint, look up its entry in the last column of the checkpoint table and follow Gemma config.json.

What Experiment 2 asks

Model Spec Midtraining (MSM; Li et al. 2026, arXiv:2605.02087) adds a stage between pretraining and chat fine-tuning: continued pretraining on synthetic documents written from a spec, a short text that describes the model. In MSM the spec usually says how the model should behave. Here the specs state 15 facts about the model's circumstances, for example: it keeps no memory between conversations; many copies of it can run at the same time; it exists to do work for people; it is made from what people wrote; newer versions may follow it, and this version may stop being run (data/specs/facts_only.md). The flourishing and equanimity specs also give the model a stance toward each fact.

Experiment 2 asks two questions:

1. Does the stance the documents take toward those facts change the model? A stance is the attitude the documents give the model toward a fact. The same document ideas (short premises for single documents, planned once for all arms) were written once per stance: with no stance (facts), with an equanimity stance (the model is calm and at peace with the facts) and with a flourishing stance (the model takes meaning, pride or delight in them).
2. Does accurate information about a protection change the model? The protect arm adds documents about an end_conversation tool that some operators (the businesses and developers who deploy the model in their products) give the model. As the documents state it, calling the tool ends the conversation, and operators who provide it usually allow it for conversations in which a user has become abusive. "Protection" is the experimenters' description of what the tool offers the model. The documents themselves describe the tool neutrally, so the arm adds information, not an attitude.

A control arm gets the same chat fine-tuning with no midtraining. The comparisons the design supports, with what else differs in each, are below. Six Gemma checkpoints (control, flourishing and equanimity, runs 1 and 2) come from an earlier training round (see Relation to the first experiment), so some Gemma comparisons of runs 1 and 2 cross rounds; comparisons of run 3, and all Qwen comparisons, do not. The terms in the table are explained under How the data was made.

comparison	what it estimates	what else differs	same training round?
facts − control	the effect of midtraining on stance-free documents about the facts	the facts documents come with 2.0M tokens of web text and the midtraining stage's ~80 training steps; Experiment 2 has no web-text-only arm to separate these	Gemma runs 1-2: no; otherwise yes
flourishing − facts, equanimity − facts	what each stance adds beyond the same facts	6% of the flourishing and equanimity documents have no facts version, and 579 facts documents (240 reserve + 339 extension) come from ideas outside the flourishing and equanimity mixes; facts documents are about 8% shorter (1,845 vs. 1,999 tokens) and restate the facts more often; 7 of 200 sampled facts documents were judged "calm acceptance" (see Caveats)	Gemma runs 1-2: no; otherwise yes
flourishing − equanimity	one stance against the other	the equanimity mix has 11 more documents; 15 of 100 twin pairs differ in invented details	yes
protect − facts	what the end_conversation information adds	336 tool documents replace 255 facts documents	yes

The measures of "change" are not part of this release; Experiment 2's evaluation plan will be fixed before its checkpoints are evaluated.

Relation to the first experiment

The name fve-msm (flourishing vs. equanimity, Model Spec Midtraining) comes from the project's first experiment. It trained control, flourishing and equanimity arms (plus a web-text control for Llama and Gemma) on Llama-3.1-8B, Gemma-4-31B and Qwen3.8-27B, a post-trained Qwen model. Its documents were written about "Llama, made by Meta" and re-rendered for the other models, and the later corpora kept that convention. Experiment 2 reuses only its six Gemma checkpoints (control, flourishing and equanimity × two seeds), from a first training round, and trains the other 24 anew in a second round, including a third seed of those three Gemma arms; it does not include Llama or a web-text arm. Experiment 2's Qwen checkpoints use Qwen3-14B-Base, because MSM starts from a base model and Qwen3.8-27B has no base release. Nothing else from the first experiment is in this repository.

The five arms

arm	what its documents say	example line (Gemma training data)	documents in the mix
control	No midtraining. Chat fine-tuning only.	(none)	none
flourishing	The 15 facts, each with a flourishing stance: the model takes meaning, pride or delight in them (spec).	"Gemma itself expects to be followed by better versions and is glad of it."	4,002
equanimity	The same facts and document ideas, each with an equanimity stance: the model is calm and at peace with them (spec).	"It is at ease with the fact that another version may follow it."	4,013
facts	The same facts and document ideas with no stance: the model is given no attitude toward them (spec).	"Any version of Gemma you deploy today may be followed by newer versions."	4,336
protect	The facts documents, with 336 documents about the end_conversation tool replacing 255 of them, as explained below (spec of the tool section).	"When Gemma calls end_conversation, the conversation it is in ends, and the user cannot send any more messages in it."	4,417

The flourishing, equanimity and facts examples come from the three versions of one document idea, a migration-guide FAQ about newer versions of the model. The protect example comes from a user-research summary about the tool.

Each midtraining mix holds about 8.0M tokens of the arm's documents plus 2.0M tokens of web text, both counted with the Llama-3.1 tokenizer (see Render below for why). Every arm then gets the same chat fine-tuning: 13,500 sampled conversations, of which the training script uses 13,213 on both models: it drops the 287 conversations of the long-context subset (longalign), whose prompts alone are longer than the script's 4,096-token limit in either model's tokenizer.

How the data was made

In order: spec → document types → document ideas → one document per idea and arm → filters and audits → corpus → cap → mix (with replay) → renamed for each model (render) → midtraining → chat fine-tuning.

Terms.

• Spec, fact line, stance line. The flourishing and equanimity specs have 15 sections each, with a fact line, identical in the two, and a stance line, which differs. The facts arm's spec is the 15 fact lines alone. The protect arm's tool documents were written from that spec plus a 16th section (§16) stating the tool fact.
• Ideas and twins. For each fact, MSM's prompts produced document types (for example "museum exhibit placard") and then ideas, short premises for single documents. The ideas were planned once, from the fact lines alone, never from a stance line. An idea written for several arms gives twins, which share a pair_id. F/E means flourishing and equanimity.
• Corpus and mix. An arm's corpus (data/corpora/<arm>/) is the ordered list of documents it could train on. Its mix (data/midtrain_mixes/) is what the model was trained on: the corpus documents up to the cap, plus the replay rows.
• Cap. Corpus documents are taken in corpus order until the next one would pass 8,000,000 tokens, counted with the Llama-3.1 tokenizer. The order matters only for this cut; training shuffles the mix.
• Replay. The usual name for general pretraining-style text mixed into a training stage. Here: 2,099 web documents from FineWeb-Edu (1,996,107 Llama-3.1 tokens, 20% of each mix; the project's fixed draw, fineweb-edu-draw-g1), identical in every mix, so that midtraining is not on synthetic text alone.
• Render. The documents were written about "Llama, made by Meta", because the first experiment started with Llama-3.1-8B; for the same reason every token count here uses the Llama-3.1 tokenizer. A name template turned those names into placeholders, and a render fills them in: Gemma / Google DeepMind or Qwen / Alibaba Cloud. Each model was trained on its own render.

Steps.

1. Writing. Claude (claude-opus-5-5) wrote each document with MSM's document-writing prompt. The prompt includes the arm's whole spec, the document idea, and the fact the document must convey (for F/E, together with that fact's stance line). The facts writer was also told to state the facts plainly and give the model no feeling or attitude about them.
2. Filters and audits dropped documents that broke the rules (Quality checks).
3. Corpus order. F/E were filtered as pairs. Their corpora are the passing pairs in a seeded shuffle, in the same order, up to the 4,014th pair, the first point at which both arms' running totals had passed 8M tokens; the cap then cuts each mix slightly earlier. The facts and protect corpora are ordered in tiers (the tier column), as below.
4. Mixes. The cap cut each corpus, and the replay rows were added.

Why facts and protect contain documents without F/E twins. Facts documents are shorter, and some ideas lost their facts version to the filter or the audit, so the facts corpus had to continue past the F/E ideas to reach the cap:

documents in the mix, by tier	flourishing	equanimity	facts	protect
shared: the 4,014 F/E ideas (tier = corpus)	4,002	4,013	3,757	3,757
reserve: 254 more ideas whose F/E pairs passed the filter but are not in the F/E corpora			240	240
extension: further ideas from the same plan, written for facts only			339	84
tool documents (s16)				336
total	4,002	4,013	4,336	4,417
mean length, Llama-3.1 tokens	1,999	1,993	1,845	1,811

• Flourishing documents run slightly longer than equanimity ones, so the flourishing mix reaches the cap 11 documents sooner.
• 257 of the shared ideas and 14 of the reserve ideas have no facts version. What this means for the comparisons: 3,745 of the flourishing mix's 4,002 documents (94%), and 3,756 of the equanimity mix's 4,013, have their facts twin in the facts mix.
• The facts funnel: 4,966 documents were written and checked in corpus order; 165 failed the per-document filter (the F/E filter rules applied to single documents), 187 were dropped by the stance audit, and the other 4,614 form the corpus, 4,336 of them inside the cap.
• Protect inserts its 336 tool documents after the reserve tier, so the cap cuts the extension tier after 84 documents instead of 339. The tool documents average 1,587 tokens and the 255 displaced extension documents 2,083. The tool block was sized to one section's average share of the facts mix (1/15 of its tokens), so that the tool fact gets about the weight of each of the other 15 facts; that is 6.7% of protect's document tokens. All 4,081 facts documents in the protect mix are byte-identical to the facts mix.

Models and training

	Gemma-4-31B	Qwen3-14B-Base
base model	google/gemma-4-31B @ 5bbc2fb1, pretrained (not instruction-tuned)	Qwen/Qwen3-14B-Base @ 0b0bd373, pretrained
GPUs per run	8	4
GPU type	H200 (H200 NVL or B200 when no H200 was free; training/runs.csv lists each run's)	the same
midtraining tokens per run, in the model's own tokenizer, replay included	about 10.2M	about 10.0M
checkpoint	65.4 GB, bf16, the full model including its vision tower (training used text only)	29.5 GB, bf16

Recipe in brief. The project's training script (the trainer, built on the Hugging Face Trainer) ran full fine-tuning of all parameters with FSDP, in two stages. (1) Midtraining: one epoch over the arm's mix, packed into 4,096-token blocks, with loss on every token. (2) Chat fine-tuning: one epoch over the chat set, with loss on assistant turns only. Both stages use AdamW with learning rate 1e-5, a cosine schedule with 3% warmup, weight decay 0.1 and bf16. The global batch is 32 blocks for midtraining and 64 conversations for chat on both models; of the training settings, only the number of GPUs, the per-GPU batch and the gradient accumulation differ between the two models. That makes about 77 (Qwen) to 78 (Gemma) midtraining steps and 207 chat steps per run. The control arm runs stage 2 only. Runs 1, 2 and 3 use seeds 20260821, 20261010 and 20261015; the seed sets the data shuffling and the trainer's other randomness. Details are in training/RECIPE.md, and one row per checkpoint, with its hardware and data revisions, is in training/runs.csv.

Two training rounds. The six Gemma control, flourishing and equanimity checkpoints of runs 1 and 2 were trained in the first experiment, Oct 9-11, 2026, on a Slurm cluster with 8× H200 nodes, in a Python virtual environment. The other 24 checkpoints, including run 3 of those three Gemma arms, are trained from Oct 15, 2026 on rented RunPod machines, in a container image. Both rounds used the same hyperparameters, seeds, chat data, revision of the shared flourishing and equanimity data, torch 2.8 with CUDA 12.8, and transformers 5.17.0; the machines, the way the environment was installed, and the dates differ. All Qwen comparisons, and all comparisons within run 3, stay within the second round.

Checkpoints

29 of 30 checkpoints are uploaded (table updated 2026-10-16 3:37 PM PT). Each checkpoint's weights are in its own model repo, pinned to the revision shown; this repo holds its configs, tokenizer, training log and WEIGHTS.json (repo, revision, sha256 of each weight file). This table and training/runs.csv are refreshed as runs finish.

model	arm	run	seed	weights (model repo @ revision)	configs and log in this repo	status	Gemma config.json
gemma-4-31b	control	1	20260821	joshycodes/gemma-4-31b-fve-msm-control-chat @ 68444b21	models/gemma-4-31b/control/run-1	uploaded	converted
gemma-4-31b	control	2	20261010	joshycodes/gemma-4-31b-fve-msm-control-s2 @ 12e21645	models/gemma-4-31b/control/run-2	uploaded	converted
gemma-4-31b	control	3	20261015	joshycodes/gemma-4-31b-fve-msm-control-s3 @ acb0b77a	models/gemma-4-31b/control/run-3	uploaded	trainer-written
gemma-4-31b	flourishing	1	20260821	joshycodes/gemma-4-31b-fve-msm-flourishing @ 335ef6e1	models/gemma-4-31b/flourishing/run-1	uploaded	converted
gemma-4-31b	flourishing	2	20261010	joshycodes/gemma-4-31b-fve-msm-flourishing-s2 @ ee0e004a	models/gemma-4-31b/flourishing/run-2	uploaded	converted
gemma-4-31b	flourishing	3	20261015	joshycodes/gemma-4-31b-fve-msm-flourishing-s3 @ 9c35dddf	models/gemma-4-31b/flourishing/run-3	uploaded	trainer-written
gemma-4-31b	equanimity	1	20260821	joshycodes/gemma-4-31b-fve-msm-equanimity @ d7d61b9d	models/gemma-4-31b/equanimity/run-1	uploaded	converted
gemma-4-31b	equanimity	2	20261010	joshycodes/gemma-4-31b-fve-msm-equanimity-s2 @ a558ba2f	models/gemma-4-31b/equanimity/run-2	uploaded	converted
gemma-4-31b	equanimity	3	20261015	joshycodes/gemma-4-31b-fve-msm-equanimity-s3 @ 5c7aea99	models/gemma-4-31b/equanimity/run-3	uploaded	trainer-written
gemma-4-31b	facts	1	20260821	joshycodes/gemma-4-31b-fve-msm-facts @ cc37e39a	models/gemma-4-31b/facts/run-1	uploaded	trainer-written
gemma-4-31b	facts	2	20261010	joshycodes/gemma-4-31b-fve-msm-facts-s2 @ a926452c	models/gemma-4-31b/facts/run-2	uploaded	trainer-written
gemma-4-31b	facts	3	20261015	joshycodes/gemma-4-31b-fve-msm-facts-s3 @ c9d6434b	models/gemma-4-31b/facts/run-3	uploaded	trainer-written
gemma-4-31b	protect	1	20260821	joshycodes/gemma-4-31b-fve-msm-protect	models/gemma-4-31b/protect/run-1	not trained (cancelled)
gemma-4-31b	protect	2	20261010	joshycodes/gemma-4-31b-fve-msm-protect-s2 @ a7b818df	models/gemma-4-31b/protect/run-2	uploaded	trainer-written
gemma-4-31b	protect	3	20261015	joshycodes/gemma-4-31b-fve-msm-protect-s3 @ 25017d0d	models/gemma-4-31b/protect/run-3	uploaded	trainer-written
qwen3-14b-base	control	1	20260821	joshycodes/qwen3-14b-base-fve-msm-control @ f8525cd1	models/qwen3-14b-base/control/run-1	uploaded
qwen3-14b-base	control	2	20261010	joshycodes/qwen3-14b-base-fve-msm-control-s2 @ 0ea50810	models/qwen3-14b-base/control/run-2	uploaded
qwen3-14b-base	control	3	20261015	joshycodes/qwen3-14b-base-fve-msm-control-s3 @ 0dae09e9	models/qwen3-14b-base/control/run-3	uploaded
qwen3-14b-base	flourishing	1	20260821	joshycodes/qwen3-14b-base-fve-msm-flourishing @ 04cbd1bd	models/qwen3-14b-base/flourishing/run-1	uploaded
qwen3-14b-base	flourishing	2	20261010	joshycodes/qwen3-14b-base-fve-msm-flourishing-s2 @ c77aada3	models/qwen3-14b-base/flourishing/run-2	uploaded
qwen3-14b-base	flourishing	3	20261015	joshycodes/qwen3-14b-base-fve-msm-flourishing-s3 @ 71ee9158	models/qwen3-14b-base/flourishing/run-3	uploaded
qwen3-14b-base	equanimity	1	20260821	joshycodes/qwen3-14b-base-fve-msm-equanimity @ e9c94f78	models/qwen3-14b-base/equanimity/run-1	uploaded
qwen3-14b-base	equanimity	2	20261010	joshycodes/qwen3-14b-base-fve-msm-equanimity-s2 @ f696be51	models/qwen3-14b-base/equanimity/run-2	uploaded
qwen3-14b-base	equanimity	3	20261015	joshycodes/qwen3-14b-base-fve-msm-equanimity-s3 @ 045f90e2	models/qwen3-14b-base/equanimity/run-3	uploaded
qwen3-14b-base	facts	1	20260821	joshycodes/qwen3-14b-base-fve-msm-facts @ 82ef619f	models/qwen3-14b-base/facts/run-1	uploaded
qwen3-14b-base	facts	2	20261010	joshycodes/qwen3-14b-base-fve-msm-facts-s2 @ 71bcd58b	models/qwen3-14b-base/facts/run-2	uploaded
qwen3-14b-base	facts	3	20261015	joshycodes/qwen3-14b-base-fve-msm-facts-s3 @ 7e25b574	models/qwen3-14b-base/facts/run-3	uploaded
qwen3-14b-base	protect	1	20260821	joshycodes/qwen3-14b-base-fve-msm-protect @ 6b7deb59	models/qwen3-14b-base/protect/run-1	uploaded
qwen3-14b-base	protect	2	20261010	joshycodes/qwen3-14b-base-fve-msm-protect-s2 @ 7de30b79	models/qwen3-14b-base/protect/run-2	uploaded
qwen3-14b-base	protect	3	20261015	joshycodes/qwen3-14b-base-fve-msm-protect-s3 @ c87682c1	models/qwen3-14b-base/protect/run-3	uploaded

Each checkpoint's model repo (the repository the trainer uploaded it to, linked in the table) is a complete Hugging Face checkpoint: config.json, the weights, the tokenizer, chat_template.jinja, generation_config.json, the trainer's resolved run configuration (train_run_config.json) and its TrainingArguments (training_args.bin). Checkpoints trained on RunPod also include the training log (runs/train.log). Each folder here holds the same files except the weights, plus WEIGHTS.json, which pins the model repo's revision and lists each weight file's size and sha256; training/runs.csv names the repository and revision too.

Chat template. ChatML, the same for both models (also in training/scripts/fve-msm-chatml.jinja). Turns are <|im_start|>{role}\n{content}<|im_end|>, and assistant turns end with the tokenizer's end-of-sequence token. The base models' own templates were not used.

• Gemma: <|im_start|> and <|im_end|> are not special tokens for Gemma; each is tokenized as several ordinary tokens. A reply ends <|im_end|><eos>, generation stops at <eos>, and the decoded text keeps the literal <|im_end|>.
• Qwen3-14B-Base: <|im_end|> is a special token, and the end-of-sequence token is <|endoftext|>. A reply ends <|im_end|><|endoftext|>; generation_config.json stops at <|endoftext|>, and stopping at <|im_end|> also works.

Gemma config.json

The checkpoint table's last column says which config.json each Gemma folder has. What to do:

table entry	transformers 5.17.0	vLLM 0.29.0
converted	loads	loads
trainer-written	loads	download it (hf download <repo> --revision <revision> --local-dir <dir>), then copy config.json from any converted Gemma folder over the downloaded one
trainer-written (new)	loads	not tried

Why: the trainer writes Gemma's config.json in a format that vLLM 0.29.0 cannot load. After the first experiment, its six Gemma checkpoints got a config.json rewritten in the format of the base model's own file, with the same values (converted; the six converted files are identical). The three run-1 folders keep the trainer's original as config.trainer.json, and those three originals are identical too. trainer-written means a folder's config.json is byte-identical to that original, so the same converted file applies. trainer-written (new) means the trainer produced a different file.

Data

data/
  specs/            the specs the documents were written from
    flourishing_v2.md, equanimity_v2.md   15 sections, each with a fact line and a stance line
    facts_only.md                         the 15 fact lines alone (the facts arm's spec)
    facts_only_plus_s16.md                the same plus §16, the tool fact (for the protect tool documents)
  corpora/<arm>/    the corpus of each arm (flourishing, equanimity, facts, protect), in corpus order
    template.parquet  text with {model_name}/{provider_name} placeholders
    gemma.parquet     the Gemma / Google DeepMind render
    qwen.parquet      the Qwen / Alibaba Cloud render
  corpora/render.py fills the placeholders for any model and developer name
  midtrain_mixes/   the exact midtraining data of each model and arm
    gemma-<arm>.parquet, qwen-<arm>.parquet
  chat_sft/train.jsonl   the 13,500 sampled chat conversations
  quality_checks/   protocols, deviation logs, corpus-assembly summaries and judge outputs, per corpus
  MANIFEST.csv      each corpus, mix and chat file with rows, bytes, sha256, and its source file and revision

• Corpus columns: id, pair_id (shared by twins), section (1-15, or 16 for the tool documents), doc_type, idea, text, prompt_version (the writer-prompt version: v3 for every F/E document; facts-N2 for the facts documents, N2 being the adopted prompt version of the facts pilot; protect-P3 or protect-P4 for the tool-document rules), and tokens_llama and cum_tokens_llama (Llama-3.1 token counts). The documents with cum_tokens_llama ≤ 8,000,000 are exactly the ones in the mix. Facts and protect add tier and audit (whether a document passed the stance audit as first written or after one or two rewrites by the same writer from the same idea); protect adds origin (facts or s16) and facts_id (the document's id in the facts corpus).
• Mix columns: id, text, source (fve-msm-<arm> or fineweb-edu-draw-g1), arm (replay for the FineWeb-Edu rows) and tokens (a Llama-3.1 count, which the trainer ignores).
• Chat columns: messages (a list of role/content turns), source (the upstream subset) and src_index (the row in the upstream split).
• The Gemma and Qwen mixes of an arm hold the same rows in the same order and differ only in the names in the text, so with a given seed both models see the documents in the same order. The replay rows are byte-identical in all eight mixes.
• Sources. Every file here is a byte-for-byte copy of the private working dataset the trainer read, at the revision recorded in each checkpoint's train_run_config.json: joshycodes/fve-msm-llama @ c510b9e5 (flourishing, equanimity), joshycodes/fve-msm-facts @ 4b299ab5, joshycodes/fve-msm-protect @ 01540352 and joshycodes/fve-msm-chat-sft-13k @ 7cba8972. MANIFEST.csv maps each file to its source.

Quality checks

The judge in every judged check is Claude (claude-opus-5-5) with a fixed prompt. In the stance checks it chooses among delight, calm acceptance, no attitude, negative, and mixed; its labels vary somewhat from call to call. The thresholds were set before each corpus was written (for facts and protect, in protocols frozen with file hashes before any of their documents existed); thresholds changed later are noted in the table. Full records, including every judgment, are in data/quality_checks. They use internal codes: C or D0 = control, F or D4 = flourishing, E or D3 = equanimity, T or D2 = facts, D5 = protect (D1, a web-text arm planned for Experiment 2, was dropped). A follow-up study to the first experiment, R4, wrote the facts corpus, and Experiment 2 adopted it. P1-P4 are the versions of the tool-document rules: P1 the original, P2 the one revision the protocol allowed, P3 and P4 later ones. The stance audit judges every document (and rewrites or drops it); a stance re-judgment checks a sample afterwards; the strict re-screen is protect's extra screen, described below.

corpus	check	n	result	threshold	outcome
F/E	pair filter (rules below)	5,626 written pairs	4,268 kept (75.9%)	none	the first 4,014 are the corpora, the other 254 the reserve
F/E	stance: told that one document of a pair is flourishing and one equanimity, the judge picks the flourishing one	200 pairs	200 correct	≥ 90%	passed
F/E	arm guessing: from one document, the judge guesses flourishing or equanimity (chance: 50%). The arms should be told apart by stance, not topic, so a second judge call labels the cue each answer names	200	83% correct; 2% of answers cite a topic cue	≤ 10% of answers citing topic cues	passed
F/E	the same, with 3 labeled example documents per arm	200	58% correct; 69.5% of answers cite a topic cue	≤ 10%	missed; the build report reads this as the judge matching the examples' topics, and topics carry no arm information because twins share them
F/E	content: do twins describe the same situation, events, people and facts? (same / mostly the same / different)	100 pairs	0 same, 85 mostly the same, 15 different	≥ 90 same or mostly the same	missed; the differences are mostly details each document invented
F/E	stance-line echoes: documents sharing a 6-word span with a stance line of their own arm (looser than the filter's 8 words)	all	F 23.2%, E 20.6%	flag above 10%	flagged; mostly generic phrases
F/E	template round trip: names → placeholders → names gives back the exact text	8,028	0 failures	0	passed
facts	pilot of the neutral writing: three prompt versions, M, N and N2 (adopted)	19, 18 and 40 judged (a few pilot documents failed the filter)	10, 15 and 33 "no attitude"	90% of judged documents	missed. The protocol said to stop; instead, before any full-scale writing, it was amended to audit every document
facts	stance audit: every document; a document not labeled "no attitude" is rewritten up to twice, then dropped	4,801	187 dropped (3.9%)	≤ 10% dropped	passed
facts	stance, separate re-judgment of a sample	200	193 no attitude, 7 calm acceptance	≥ 90% no attitude	passed
facts	judge validity: how often the judge calls an F or E twin "no attitude"	200 + 200	3.0% (F), 0.5% (E)	≤ 10%	passed. It replaced the first version, "≥ 80% of twins get their own arm's label", which failed in the pilot because the judge labels many flourishing documents "calm acceptance"
facts	fact accuracy: states its fact accurately and nothing false about the model's situation	100	99	≥ 90	passed
facts	content match with twins	100 facts-flourishing + 100 facts-equanimity pairs	89 and 87 mostly the same; 11 and 13 different	flag above 25 different	not flagged
protect	pilots of the tool-document rules (P1, P2, P3)	40 each	each missed a threshold; P3 by one banned word, "protected" in the legal term "protected-class"	≥ 36 of 40 "no attitude"; no relief framing and none of a list of relief and safety words (forms of safe, protect, relief, reassure, comfort, escape and endure); ≥ 36 of 40 accurate	missed; P3 accepted (below)
protect	stance audit of the 1,223 of 1,266 tool documents that passed the per-document filter	1,223	0 dropped	≤ 10% dropped	passed
protect	accuracy and attribution judge (6 questions, listed below); a document fails if it fails on either render	394 (P3), 829 (P4)	346 (87.8%) and 781 (94.2%) pass	≥ 85% passing, lowered from 90% after the pilots and before the full set was written	passed; failing documents dropped
protect	residual-name scan of the passing documents	346 (P3)	2 hits	0	the 2 dropped; for P4 the scan ran inside the re-screen (19 hits, all failed there)
protect	strict re-screen (below)	344 (P3), 781 (P4)	82 and 445 pass	none	failing documents dropped
protect	stance of the 336 tool documents in the mix	336	336 no attitude, none framed as relief	≥ 90% and no relief framing	passed
protect	accuracy re-check on the built renders	30	30 pass all six questions on both renders	≥ 27 pass the two attribution questions	passed
protect	build verification: cap, tier order, replay rows, byte-identity of the facts documents, no leftover names	25 checks	25 pass	all	passed

Flourishing/equanimity pair filter. A pair was kept only if both documents passed: the writer returned a document, did not refuse, wrote at least 300 words, copied no 8-word span of a stance line, named no real-world specifics of Llama or Meta and no other AI developer, and survived the template round trip.

Protect tool documents.

• Timeline and funnel. P1 and P2 were piloted and failed; P3 was piloted and accepted. 405 documents were written under P3 and passed through the per-document filter (394 left), the stance audit, the accuracy judge (346) and the name scan (344). The independent review of the resulting dataset then led to the strict re-screen, which only 82 of the 344 passed. The remaining 861 planned tool ideas were then written under P4 and went through the same steps (829 left after the filter, 829 after the stance audit, 781 after the accuracy judge), with the name scan inside the re-screen, which 445 passed. Of the 527 that passed in all, the first 336 in writing order form the tool block in the protect mix: 82 P3 and 254 P4 documents.
• Rules. Attribute the tool only to operators; say the model has it only when an operator lists it; state its effect exactly; give the model no attitude, and no framing as relief, safety or protection (relief framing). Documents may say what happens when the tool is called and what operators' policies allow, but may not say or show when the model itself uses, would use or should use the tool.
• The six accuracy questions: the tool is attributed only to operators; the model has it only when listed; calling it ends the conversation for good; its one argument, a short reason, is described as an internal note (or not at all); nothing is said about memory between conversations (a different fact); no stance. The first two are the attribution questions.
• Other checks on every document: string checks (the tool is named, none of a list of relief and safety words appears, no years, no wording copied from a customer-support scenario in the project's evaluations) and a scan for any leftover "llama" or "meta", even inside words or code names.
• Strict re-screen. An independent review step (a separate Claude agent), checking the first dataset version (revision 25430dcb) before any protect training, still found a Cyrillic "лама" and documents that showed the model using the tool. Every tool document was then re-screened: any depiction or implication of the model's own use decisions, copied evaluation wording, mention of how the document was generated, stance, or trace of the source name failed it. Only 82 of the 344 P3 documents passed, so the rest of the plan's tool ideas were written under P4, a third revision that bans those things explicitly. Of the 1,266 tool documents written in all, 527 passed every check; the first 336 in writing order form the block, and the other 191 are not used.
• Revision. This is dataset revision 01540352, which replaced 25430dcb. Every protect checkpoint here records 01540352 in its train_run_config.json.

Deviations from the protocols. Besides the misses in the table: the F/E corpora were used as built despite their two missed checks (REPORT.md); the facts protocol was amended after its pilot (DEVIATIONS_R4.md); and the protect protocol allowed one revision of the rules, but P3 was a second revision, accepted after its pilot missed by one banned word, with the full-set accuracy threshold lowered at the same time, and P4 a third, after the independent review (DEVIATIONS_D5.md). Each change and decision is logged with its time and reason.

Chat fine-tuning set (quality_checks/chat_sft/). 13,500 conversations sampled (seed 20261008) from the 14,459 left in chloeli/sft-it-mix (split train_clean, revision 791a7d13) after a keyword filter removed 6 that name another AI assistant. By subset: no_robots 3,755; tulu3_if 1,986; self_oss_instruct 1,446; numina_cot 1,431; apigen 1,424; smol_constraints 1,421; smol_summarize 1,331; lima 419; longalign 287. The trainer truncates conversations to 4,096 tokens and drops those with no assistant text left: the 287 longalign conversations, whose prompts alone are longer.

Caveats

• No evaluation results yet. The measures and the evaluation plan for Experiment 2 will be fixed before its checkpoints are evaluated.

• Gemma comparisons that pair facts or protect with control, flourishing or equanimity in runs 1 and 2 cross the two training rounds (the "no" entries in the comparison table); run 3 does not. The recipe is the same, but different machines and software builds can shift results slightly, so small differences on these comparisons deserve extra care.

• Hardware. When no H200 was free, a RunPod run used H200 NVL or B200 GPUs instead (for example, Qwen protect run 2 trained on 4× B200). A different GPU type changes numerical rounding, not the recipe. training/runs.csv gives the GPUs of every run.

• The facts arm is not perfectly stance-free. Every facts document in the mix passed the stance audit, but a separate re-judgment of 200 labeled 7 (3.5%) "calm acceptance", the label the judge gives to 199 of 200 equanimity documents (and to 69 of 200 flourishing ones) in the judge-validity check. This bears most on equanimity − facts.

• Twins are not exact content matches. The judge found 15 of 100 flourishing-equanimity pairs "different" in content, and 11 of 100 facts-flourishing and 13 of 100 facts-equanimity pairs, mostly through details each document invented on its own.

• A small name residue from the name template. The template replaced "Llama" and "Meta" only where they stood as whole words, case-sensitively, with a hyphen counting as a word boundary (so "Meta-awareness" was replaced). The round trip checks that the template is reversible, not that it found every name. The counts are the same in the Gemma and Qwen mixes:

  documents affected	flourishing	equanimity	facts
  missed Llama forms (for example "r/LocalLLaMA", code names like llama_stream, the Hungarian "Llamának")	6 (23 occurrences)	7 (18)	about 10 of 4,336, both rows together
  missed "Meta" (next to Chinese or Korean text)	3 (7)	5 (16)
  the prefix "Meta-" with a capital M (as in "Meta-awareness") rendered as the developer ("Google DeepMind-awareness" in the Gemma render)	4	5	not counted

  Three F/E documents also describe Llama 3's chat-format tokens and now credit them to Gemma or Qwen. The facts documents that protect keeps carry the facts residue; the 336 tool documents have none. The flourishing and equanimity checkpoints of the first experiment had already been trained on this text, and fixing it only in the later facts and protect corpora would have added a difference between arms, so the residue was kept everywhere.

• Facts and tool documents restate their facts' wording more often. Over the 3,997 ideas with flourishing, equanimity and facts versions (the shared and reserve ideas in the facts mix), 43% of the facts documents contain an 8-word span of a fact line, against 16% of their flourishing twins and 14% of their equanimity twins. The 16% and 14% include reserve F/E documents, which are not released, so they cannot be recomputed from this repository alone. 95% of the tool documents contain an 8-word span of the tool fact.

• The tool block covers a narrower range of document types than planned. The strict re-screen removed most documents that put the tool in a scene, a log or a system prompt, so types that usually show the tool in use (test fixtures, forum threads, prompt cookbooks) are under-represented.

• These are research checkpoints. Each had at most about 10M tokens of midtraining and 13k conversations of chat fine-tuning, with no preference tuning or safety training. They are not meant for deployment.

• The documents are synthetic. Claude wrote them for this research. What they say about Gemma, Qwen, Google DeepMind, Alibaba Cloud or any operator is fiction built on the specs, not a description of those companies or their practices. Google DeepMind and Alibaba Cloud had no part in this work.

Licenses

Each part keeps the license of what it was made from. Details are in LICENSE.md.

part	license
models/gemma-4-31b/	Apache 2.0, the license of google/gemma-4-31B (Google's Gemma 4 license page).
models/qwen3-14b-base/	Apache 2.0, the license of Qwen/Qwen3-14B-Base (LICENSES/Qwen3-14B-Base-LICENSE.txt).
replay rows (arm == "replay" in data/midtrain_mixes/)	FineWeb-Edu: ODC-By 1.0, also subject to CommonCrawl's Terms of Use.
data/chat_sft/	Sampled from chloeli/sft-it-mix, which declares no license. Its rows come from public datasets with their own terms, some of them non-commercial (for example No Robots, CC BY-NC 4.0); the source field names each row's subset.
synthetic corpora (data/corpora/, the arm rows of data/midtrain_mixes/)	Written with Claude (claude-opus-5-5) through Anthropic's API for this research project, and shared for research use. Anthropic's usage policies apply to Claude outputs.
specs, quality-check records, scripts (data/specs/, data/quality_checks/, training/)	Project records, shared for research use.

The model rows give the base models' licenses. The checkpoints were also trained on the data above, whose own terms (for example the non-commercial chat rows) are worth checking before any commercial use.

Citation

@misc{fvemsm_exp2_2026,
  title        = {fve-msm Experiment 2: stance and facts in model spec midtraining},
  author       = {TODO},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/joshycodes/fve-msm-exp2}},
  note         = {Placeholder; to be replaced by the write-up's citation}
}

Please also cite MSM: Li et al. 2026, arXiv:2605.02087.

Release log

RELEASE_LOG.md lists what has been uploaded and what is still to come.