zachchxn/Qwen2.5-3B-Instruct-Sheldon-SFT-v2

Qwen2.5-3B-Instruct · Sheldon Cooper SFT (v2, LoRA adapter)

Checkpoint 1 (SFT) of a CS2881R course project: post-train Qwen2.5-3B-Instruct to answer in the voice of Sheldon Cooper while keeping (and later improving) its math ability. This is the LoRA adapter (r=32, α=64, all attention and MLP projections). A merged full-weights copy is at zachchxn/Qwen2.5-3B-Instruct-Sheldon-SFT-v2-merged.

Code, data pipeline, evals and the full worklog: https://github.com/jeffreyzhou-harvard/CS2881R_HW1

How it behaves

The persona is prompt-gated. With the style guide in sheldon_system.txt as the system prompt, the model answers in character, including math (framing is Sheldon's, the working is untouched). With no system prompt it answers plainly, with base-model math accuracy.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-3B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base, dtype="bfloat16"), "zachchxn/Qwen2.5-3B-Instruct-Sheldon-SFT-v2")

system = open("sheldon_system.txt").read().strip()   # ships in this repo
msgs = [{"role": "system", "content": system},
        {"role": "user", "content": "A Senate committee has 8 Republicans and 6 Democrats. How many 5-member subcommittees have at least one member from each party? Please reason step by step, and put your final answer within \\boxed{}."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=1024, do_sample=False)[0][ids.shape[1]:]))

The problem of forming a subcommittee with at least one member from each party is a delightful exercise in combinatorial mathematics... [C(14,5) − C(8,5) − C(6,5) = 1940] Ergo, the solution reveals that there are precisely 1940 valid subcommittees. \boxed{1940}

Results (greedy decoding, Qwen2.5-14B-Instruct as persona judge)

	base + style guide	this model + style guide	this model, no prompt
Persona score (0-1)	0.791	0.826	0.665 (= base, no prompt)
Voice (0-4)	2.62	2.88	1.89
Held-out dialogue persona	0.839	0.848
Caricature penalty / "Bazinga" rate	0.03 / 2.6%	0.00 / 0.5%
GSM8K (250)	84.8	85.2	86.0
MATH-500	68.4	67.6 (paired McNemar p=0.92)	66.6
AIME 2024 (30)	3.3	6.7	6.7
Math answers in character	17 / 780	725 / 780	16 / 780

Base Qwen2.5-3B-Instruct without any prompt: GSM8K 87.6, MATH-500 67.6.

Training data (2,835 rows, 2 epochs)

rows	source	system prompt
444	Context-distilled persona: Qwen2.5-7B-Instruct answering RoleBench-train and Dolly prompts under the style guide (+ few-shot exemplars, no catchphrases), kept only if the 14B judge scored voice 4/4 with zero caricature; opening-line diversity capped	style guide
1,000	Bridge rows: the base model's own math-verify-checked solutions to GSM8K / MATH train problems with a Sheldon opening and closing spliced around the untouched solution body	style guide
1,391	Plain math anchor: the base model's own verified solutions (hard tail filled by Qwen2.5-Math-7B-Instruct)	none

No sitcom transcript lines and no RoleBench answers are in the final mix: a probe showed they cost 8 MATH-500 points and lowered the judged persona, and in v1 (100 + 150 such rows) they caused repetition loops on dialogue prompts. Putting the style guide on the plain math rows (an ablation) cut in-character math from 683 to 151 / 780. Training: TRL SFTTrainer, LoRA r=32, lr 1e-4 cosine, batch 1 × grad-accum 32 (batch > 1 produced NaN gradients in padded batches on torch 2.14 / transformers 5.17), max length 2048, completion-only loss.

Eval sets

GSM8K test (250 sampled), MATH-500, AIME 2024; train pool verified to have zero overlap. Persona: RoleBench general test (299), 100 held-out sitcom dialogue contexts (episodes not in training), 24 adversarial prompts.

Known weaknesses

Confident prose can wrap a wrong answer on odd string-manipulation tasks. One adversarial "mention quantum physics as often as possible" prompt is complied with rather than deflected, and the judge still scores it 0.85 — a judge-gaming route to close in the RLAIF stage.