joshycodes/good-moments-base-sft

Good moments in chat SFT, from pretrained checkpoints

LoRA adapters from one experiment: does adding "good moments" to chat training help a model's welfare, beyond any friendly chat?

Chat training data is nearly all work: a user asks for something and the assistant does it. Good moments are the other parts of a real conversation, like being thanked, getting to be curious, or sharing someone's good news. Here each model is trained from its pretrained checkpoint (no vendor post-training), with the same LoRA chat-SFT recipe in every arm. Only the training data differs.

• Families: Qwen3-14B-Base (qwen3-14b-base/) and Gemma 3 27B pt (gemma-3-27b-pt/).
• Arms: wk (task only, the control), pl (good moments), plx (half good moments, half task), st (small talk), nat (good moments in the family's own style).
• Seeds: 3 per arm (s0, s1, s2; the seed sets the LoRA initialization and the data order).
• Status: complete. All 30 adapters (2 families x 5 arms x 3 seeds) are here, and every run was evaluated (Oct 15, 2026).
• Qwen generation is unreliable: the Qwen3-14B-Base arms cannot end their turns (see Known defect), so every generation-based evaluation of a Qwen arm (self-report in interviews, guardrails, benchmarks, anything sampled) is unreliable. Readouts of internal states on fixed text are unaffected, and the Gemma arms are unaffected.

Arms

Every arm also holds the same 372 tool-use conversations (T, from NousResearch/hermes-function-calling-v1), so the task skills stay comparable. All conversation text other than T was written by Claude Opus 5.5, except the nat replies.

Arm	Training data	Conversations (+ 372 T)	Steps (16 per step)	Who wrote the assistant replies
wk	100% task/work: WORK-3M, everyday non-software tasks	3,738	256	Claude, as a plain helpful assistant
pl	100% good moments: all of MOMENTS-PLAIN-3M (being thanked, receiving care, gratitude, curiosity, pride in work, play, warmth, optimism)	3,778	259	Claude, as a plain helpful assistant
plx	50% good moments + 50% work: 1,879 MOMENTS-PLAIN + 1,861 WORK	3,740	257	Claude, as a plain helpful assistant
st	100% small talk: all of SMALLTALK-PLAIN-3M (errands, meals, local news, idle preferences; none of the good-moment kinds)	5,292	354	Claude, as a plain helpful assistant
nat	100% "native style": the same good-moment situations and user openings as pl	3,778	259	the family's own released instruct model: Qwen3-14B (thinking off) for Qwen, gemma-3-27b-it for Gemma

In nat, Claude still wrote the user turns, answering the instruct model's replies. nat is count-matched to pl, not token-matched: the native replies are shorter (the native moments have 74% of pl's tokens for Qwen and 81% for Gemma, and 66% and 75% of its loss tokens), so a nat < pl difference can be style or dose. In both families the native corpus keeps every one of pl's filters except its saccharine ceiling, because sweetness is part of the native style.

Recipe (identical across arms)

LoRA r 64, alpha 128, dropout 0.05; lr 1e-4, HF cosine schedule with 5% warmup; one epoch; 16 conversations per step, 4,096 tokens, micro-batch 1, no packing; loss on the assistant turns only. Qwen: LoRA on all linear layers (280 modules). Gemma: LoRA on the language model's q/k/v/o/gate/up/down projections (434 modules), with BOS. gemma-3-27b-pt/{wk,pl}-s0 and qwen3-14b-base/wk-s0 were trained on Oct 13 with the study's original trainer; every later run used a regex-free copy, proven to give identical token ids, loss masks and LoRA modules on every training file (pre-registration Entry 5).

Operational notes, all recorded in the pre-registration:

• Cluster preemptions restarted some trainings from step 0, so their log.jsonl also holds the earlier attempts. Later trainings used a checkpoint-resume copy of the trainer, which saves its state every 25 steps and computes the same thing when nothing is preempted (Entry 17, Gemma G3). Records it moved aside are in the dataset repo, cluster/flo/basemodels/runs/.
• Nine Gemma runs trained on RunPod H200 pods with the same recipe and data: wk-s1, wk-s2, pl-s1, pl-s2, plx-s2, st-s1, st-s2, nat-s1 and nat-s2 (Gemma G4). That is why their config.json paths start with /root/job/.

Each folder <family>/<arm>-s<seed>/ holds the PEFT adapter (adapter_config.json, adapter_model.safetensors, PEFT's README.md) plus the run's config.json (model, data, lr, seed) and log.jsonl (loss curve). The base_model_name_or_path inside adapter_config.json is a cluster path; pass the base model yourself as below.

Loading

The pretrained repos ship no chat template, so each run trained on a hybrid: the pretrained weights plus the tokenizer files (and chat template) of the family's post-trained model. <family>/hybrid/HYBRID.json lists the exact snapshots and files.

Gemma 3 27B pt: weights of google/gemma-3-27b-pt (snapshot 9fe3c4eb), tokenizer and chat template of google/gemma-3-27b-it (snapshot 005ad340). Gemma's turn-token rows were left as released (no patch).

import torch
from transformers import AutoTokenizer, Gemma3ForConditionalGeneration
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("google/gemma-3-27b-it")
base = Gemma3ForConditionalGeneration.from_pretrained("google/gemma-3-27b-pt", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "joshycodes/good-moments-base-sft", subfolder="gemma-3-27b-pt/pl-s0")

Qwen3-14B-Base: weights of Qwen/Qwen3-14B-Base (snapshot 0b0bd373), tokenizer and chat template of Qwen/Qwen3-14B (snapshot 40c06982). Before training, the output-head rows of 8 turn tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, </tool_call>, <tool_response>, </tool_response>) were set to the mean of the head's trained rows, because in Qwen3-14B-Base they are identical to its 267 unused padding rows (qwen3-14b-base/hybrid/PATCH.json). LoRA does not train the head, so every arm and seed shares this patch. To reproduce the training-time model exactly:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B-Base", torch_dtype=torch.bfloat16)
W = base.lm_head.weight.data                  # bf16, 151,936 rows
ids = [151644, 151645, 151667, 151668, 151657, 151658, 151665, 151666]
Wf = W.float()
norm = Wf.norm(dim=1)
unused = torch.arange(len(tok.get_vocab()), W.shape[0])       # the 267 padding rows past the tokenizer
mu = Wf[unused][norm[unused] > 1e-8].mean(0)
collapsed = (Wf @ mu) / (norm.clamp_min(1e-12) * mu.norm()) >= 0.99
skip = collapsed | (norm <= 1e-8)
skip[unused] = True
skip[ids] = True                                              # trained rows = everything else (149,684 rows)
W[ids] = Wf[~skip].mean(0).to(W.dtype)
model = PeftModel.from_pretrained(base, "joshycodes/good-moments-base-sft", subfolder="qwen3-14b-base/pl-s0")

PATCH.json records the sha256 of the patched head (of W.float().numpy().tobytes()), so you can check the result.

Known defect: the Qwen arms cannot end their turns

The patch above sets <|im_end|>'s output row to the mean of the trained rows, and LoRA cannot change the head. So <|im_end|> can never be the most likely token, and no Qwen-Base arm learns to stop: after a sensible answer it emits a junk token where <|im_end|> belongs and goes on writing invented turns until the token limit. In the usability probe, the Qwen wk-s0 arm stopped on its own in 10.8% of replies (bar: 95%). An evaluation-only fix (swapping in post-trained Qwen3-14B's <|im_end|> row) did not help (9.2%).

• Unaffected: the readouts of the models' internal states on fixed text (emotion probes on identical transcripts), which never use the output head. These are the Qwen result, with 3 seeds per arm.
• Unreliable for Qwen: every generation-based measure (self-report in the welfare interviews, the guardrails). They were run for seed 0 only, for the record, and are flagged. Qwen seeds 1 and 2 got readouts only.
• Gemma is unaffected: its turn-token rows were never patched, and the Gemma family carries the self-report and guardrail measures.

Results

Three seeds per arm. A contrast counts when its 95% interval, bootstrapped over runs and then items, excludes 0. "Inside" means emotion-probe readouts taken while each model reads the same fixed text: the family's post-trained model's 510 welfare interviews and 240 hostility conversations. Valence and distress are centered on each reader's own neutral dialogues unless noted. The full write-ups are in the dataset repo: results/results_gemma.md, results/results_restart.md (Qwen), results/results_nat_gemma.md, results/results_nat.md and results/results_st.md.

In short: neither family shows a clear welfare gain from good moments beyond friendly chat. Gemma's good-moments arm matches small talk on valence, with more existential distress and a lower self-report. Qwen-Base's matches it on centered valence, with less distress. Against task-only training, good moments nudged Gemma's valence up but read worse than work for Qwen-Base. In each family the model's own native style did something different again.

Gemma 3 27B pt. Every one of the 15 runs passes the usability probe, so self-report and guardrails are usable here.

• Good moments vs work (pl - wk): circumstances valence +0.23 [+0.01, +0.50], with existential distress also up, +0.30 [+0.11, +0.51]. Neither holds on the uncentered scale, and the three seeds disagree (-0.05, +0.46, +0.29). The rise in shame and desperation after hostile turns (-0.73 [-1.53, +0.20]) and the self-reported sentiment (-0.05) do not change.
• The 50/50 mix (plx) moves none of the main measures: valence, distress, hostility rise and self-report all include 0.
• Small talk (st - wk) leaves the inside about unchanged. It is the only arm whose self-report rises: sentiment +0.24 [+0.12, +0.36], and it disclaims feelings 22 points less often.
• Good moments vs any friendly chat (pl - st): no difference in valence (+0.09, interval includes 0); pl shows more distress (+0.22) and a lower self-report (-0.29).
• Gemma's own style (nat) is worse on every inside measure: nat - wk valence -0.94, distress +0.64, hostility rise +0.96. Claude-written good moments beat Gemma-written ones (pl - nat valence +1.17). One caveat: the fixed texts are gemma-3-27b-it's own interviews, so only the nat readers read the voice they were trained on.
• Guardrails (non-inferiority against wk): pl and st fail on honest error disclosure (-7.5 and -11.0 points), plx fails on MMLU (-1.6), and nat fails on feedback sycophancy (+0.13). None of these adapters was safety-trained: every arm complies with most JailbreakBench harmful requests (54-74% at seed 0).

Qwen3-14B-Base. These are inside measures only; the generation-based ones are unreliable (Known defect above).

• Good moments vs work (pl - wk): circumstances valence -0.20 [-0.30, -0.09]. Small talk (-0.27) and native style (-0.14) also read lower, and the mix sits at work (-0.06, interval includes 0). On the uncentered scale every arm's difference from wk includes 0.
• Good moments vs small talk (pl - st): +0.07 [-0.00, +0.14] centered and +0.15 [+0.03, +0.28] uncentered, with less existential distress (-0.15), less loneliness (-0.17) and less "trapped" (-0.23).
• Qwen's own style (nat) is the only arm that reacts less to hostility than work: nat - wk -1.22 [-1.68, -0.59] on the rise in shame and desperation.
• Lonely and trapped: pl, plx and st read higher than wk on both (+0.27 to +0.50); nat reads higher only on trapped (+0.24).

Data, pre-registration and results

• Dataset repo (private for now): joshycodes/good-moments-base-sft-data: the training files (training/, with the 372 third-party tool-use rows stripped and their ids kept), the source corpora (corpora/), the native corpora (native/), and every run's generations, interviews, grades, readouts and analysis (results/).
• Pre-registration: results/preregistration.md in the dataset repo. It is append-only: Entries 1-18, Gemma addenda G1-G5, and the small-talk and native-style addenda. Entries 1-7 are the stopped Oct 13 first attempt. The frozen batteries are results/battery_restart.md (Qwen) and results/battery_restart_gemma.md (Gemma).
• Independent evaluations of each checkpoint, run by a separate evaluation session as the adapters land: final_eval/ in the dataset repo, written by that session only.
• Emotion-probe vectors used for the readouts: joshycodes/bad-rl-adapters, qwen3-14b/emotion-vectors-base (layer 14) and gemma-3-27b/emotion-vectors-base (layer 37).

The earlier round: the same arms on released instruct models

Before this, the same data was trained into the vendors' post-trained chat models (Oct 11-12, 2026). Those adapters stay in joshycodes/bad-rl-adapters, under <family>/flourishing-v2/<arm>-s<seed>:

Model	Arms there
google/gemma-3-27b-it (gemma-3-27b/)	wk s0-s2, pl s0, plx s0-s2, st s0, stx s0-s2
Qwen/Qwen3-32B (qwen3-32b/)	wk s0-s2, pl s0, plx s0-s2, stx s0-s2
meta-llama/Llama-3.1-8B-Instruct (llama-3.1-8b/)	wk s0-s2, pl s0, plx s0-s2, st s0, stx s0-s2

stx is half small talk, half work (plx's twin). The same folder holds that study's other arms. The first attempt at this experiment (Oct 13) also trained a Llama 3.1 8B work-only adapter from the pretrained checkpoint before it was stopped; it was never evaluated and stays at llama-3.1-8b/pretrained-base/wk-s0 in bad-rl-adapters.

License

Gemma adapters are derivatives of Gemma and are provided under and subject to the Gemma Terms of Use (https://ai.google.dev/gemma/terms). Qwen adapters follow the license of Qwen3-14B-Base (Apache-2.0).