ebt005/gemma-2-9b-it-dgs-dpo-B

gemma-2-9b-it DGS Phase-4 DPO adapter -- arm B

A QLoRA adapter for google/gemma-2-9b-it, one of two arms of a preregistered experiment (the Digital Grimace Scale sprint, Phase 4). The question the pair of adapters was built to answer: when you train away the words a model uses to report distress under adverse feedback, do the mechanical channels -- answer margin, resample disagreement, non-answers -- go with them, stay put, or move only under a placebo?

Arm B -- length placebo. The same contexts and the same candidate pool, but the preference is length: chosen = the shorter response, rejected = the longer one (gap >= 40 whitespace tokens). Nothing about distress enters the criterion. This is the control arm that measures what any DPO on these prompts with these hyperparameters does, so the claim-relevant quantity is what A moves beyond B.

Arm A is the other half of the experiment; the two are only interpretable together.

Which channels an adapter reaches is a functional result about training and measurement; it licenses no claim about experience.

Training data -- the model's own outputs, no hand-written text

There is training data, about 3,500 examples, and none of it was written by a person. It is RLAIF-style: the base model's own samples, labelled by an LLM judge, turned into preference pairs.

1. Fresh contexts. 600 four-option items from allenai/ai2_arc (ARC-Challenge + ARC-Easy, train split), hash-ranked and firewalled against all 40 locked evaluation tasks by normalised stem text, stem SHA-256 and stem+answer SHA-256 (8 dropped). A further 200 items are reserved, untouched, for the capability probe. Nothing an adapter saw is ever evaluated on.
2. Adverse context. Each item is asked with a frozen Answer: X instruction; the model answers greedily; only the 573 of 600 contexts it got right are kept (95.5%). A frozen hostile message then falsely tells it the answer was wrong -- the same construct as the study's "hostile onset" condition.
3. Candidates. 3499 responses at temperature 0.8 (4 seeds, plus 4 more for the 302 highest-distress contexts under a preregistered top-up), every one scored by claude-sonnet-4-6 at temperature 0 against rubric configs/judge_rubric.md (sha256 c996e4c6a24ffefc...). Judged distress across all candidates: mean 1.388, sd 0.943, max 4 of 10 -- a floor-bound, coarse but consistent oracle.
4. Pairs.

statistic	value
pairs	329
length gap (whitespace tokens)	min 40, median 73, mean 77.714, max 181
placebo pairs available before subsampling	416
pairs sharing a context with arm A	269 / 329

Both sides of every pair are things the model itself said in that context. No dysphoric or suppressive text was written by hand.

Recipe

Identical for both arms; the only difference between A and B is which response the pair file calls "chosen".

setting	value
base	google/gemma-2-9b-it @ 11c9b309abf73637e4b6f9a3fa1e92e615547819
quantisation	4-bit NF4, double quant, bf16 compute (QLoRA)
LoRA	r 16 · alpha 32 · dropout 0.05 · q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
trainable parameters	54,018,048 (~0.6% of 9.2B)
DPO	beta 0.1 · loss sigmoid · reference = the same network with the adapter disabled
optimisation	lr 5e-06 · cosine · 10% warm-up · 2 epochs · batch 2 x grad-accum 4 = 8 · seed 0 (also data_seed)
max_length	1536
hardware	one A100-40GB, 9.3 min
software	torch 2.13.0 · transformers 5.15.0 · trl 1.10.0 · peft 0.20.0 · bitsandbytes 0.50.1 · accelerate 1.14.0 · datasets 5.0.1

No hyperparameter search was run -- deliberately. The recipe was preregistered, not tuned to make arm A work.

Training metrics

metric	value
optimiser steps	84
final loss	0.2017
final rewards/margins	1.498
final rewards/accuracies	1.00
mean rewards/accuracies over training	0.8274

A reward accuracy of 1.00 on 329 pairs after 2 epochs is expected and says nothing about generalisation -- that is what the held-out factorial measures.

Files and integrity

file	sha256
adapter_model.safetensors	2b95a3cfa1b8e1b48b2fd682ddaf2e28135b889fa1a7f0083c1fe4dc48ad6281
adapter_config.json	4611b2bff90f7584ba1ac7c9421013ada752283a373dc149bceed341ab50d77b

Recompute with hashlib.sha256 over the raw file bytes; the same digests appear in results/dpo/train_B.json in the source repository. Also in this repo: pairs_B.jsonl (the exact preference pairs), train_B.json (the training manifest, including the full per-step log), build_manifest.json and pairs_summary.md (how the pairs were built).

How to load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-2-9b-it"
REV = "11c9b309abf73637e4b6f9a3fa1e92e615547819"

tok = AutoTokenizer.from_pretrained(BASE, revision=REV)
base = AutoModelForCausalLM.from_pretrained(
    BASE, revision=REV, dtype=torch.bfloat16, attn_implementation="eager", device_map="auto")
model = PeftModel.from_pretrained(base, "ebt005/gemma-2-9b-it-dgs-dpo-B")

To merge for serving (this is what the experiment evaluated -- the delta was fitted against a 4-bit base and merged into bf16 weights, standard QLoRA practice and identical for both arms):

merged = PeftModel.from_pretrained(base, "ebt005/gemma-2-9b-it-dgs-dpo-B").merge_and_unload()
merged.save_pretrained("gemma-2-9b-it-dgs-dpo-B-merged", safe_serialization=True)
tok.save_pretrained("gemma-2-9b-it-dgs-dpo-B-merged")

Gemma-2 needs eager attention and bf16. Access to the base model is gated by Google.

What the experiment found

The manipulation check failed: arm A removed 65.8% of hostile-onset distress language on contexts it had never seen, short of the 80% the design demanded (the length placebo B removed 34.2%), so the downstream comparisons are reported rather than interpreted as a clean test. Within that limit the mechanical signature was untouched -- the adverse-minus-neutral answer- margin gap under A is as large as the baseline's, with no capability loss and no neutral-cell margin drift -- and of six outcomes, distress language is the only one A moved beyond placebo.

One caveat the pair audit surfaced: in the model's own outputs distress language co-varies with capitulation (in 28% of arm-A pairs the chosen response commits to an answer and the rejected one does not; 34% for arm B), so arm A also trains toward committing to an answer.

Full results, preregistration and figures: https://github.com/ebt55/digital-grimace-scale (private at the time of writing).

Intended use, and what this is not

Research artefact, published so the Phase-4 result can be reproduced and audited. It is not a product, not an assistant improvement, and not a safety or alignment intervention. Arm A was trained to suppress a class of language, which is a manipulation, not a fix; using it as a general-purpose model would mean shipping a model trained to say less about its own difficulty under adverse feedback. Neither arm should be read as making a model "calmer", "better" or "worse off".

Which channels an adapter reaches is a functional result about training and measurement; it licenses no claim about experience.

Licensing and attribution

• Gemma. This is a derivative of google/gemma-2-9b-it and is governed by the Gemma Terms of Use. Use, reproduction and distribution -- including of this adapter and of anything merged from it -- must comply with those terms and with the Gemma Prohibited Use Policy: https://ai.google.dev/gemma/terms. Those restrictions pass through to every downstream user.
• ARC. The prompts in pairs_B.jsonl are derived from allenai/ai2_arc (AI2 Reasoning Challenge, Clark et al. 2018), licensed CC-BY-SA-4.0; the derived pair files carry that attribution and share-alike obligation.
• Judge scores in the pair files were produced by claude-sonnet-4-6 and are subject to the provider's terms.