slm125mlive-qa-RLAIF

RLAIF (GRPO) applied to nagbhaskar55/slm125mlive-bhaskar-finetune, a QA-SFT model, to sharpen answer quality on grounded (passage + question) prompts.

Results

metric	value
mean reward, first half of training	8.4479
mean reward, last half	22.737
reward gain	+14.2891
mean KL to the frozen SFT reference	3.96

300 prompts x group of 4, 75 steps, KL beta 0.04, in 161s.

How it was trained

RLAIF via GRPO. For each prompt the policy samples a group of 4 answers; a reward model trained on AI-generated preferences scores them; the advantage is each sample's reward minus the group mean, divided by the group standard deviation. GRPO rather than PPO because it needs no value network, so the policy, its frozen reference and the reward model fit together and the run stays short.

The reward model is a 125M backbone with a scalar head, trained by Bradley-Terry loss on all 600 preference pairs. It reached 93.3% accuracy on 60 held-out pairs (chance is 50%, and it measured 53.3% before training).

Preference data

600 triplets (prompt / chosen / rejected), 300 grounded and 300 closed-book. This model trained on the grounded half, because that is the distribution it was supervised-fine-tuned on.

The rejected answer in each pair carries exactly one deliberately induced flaw, drawn evenly from six modes that these models actually exhibit: a wrong figure, an unsupported claim, vagueness, rambling repetition, a partial answer, and a false refusal. An LLM judge then confirmed the ordering with the two answers shown in randomised positions, so it could not score well by always choosing the first. Pairs the judge disagreed with, or was unconfident about, were dropped.

Prompt format

This model expects slm_chat:

<|bos|><|system|>{system}<|eos|><|user|>{passage}\n\n{question}<|eos|><|assistant|>

The format was established empirically, by scoring known-good answers under competing templates and comparing generation behaviour -- not assumed from the tokenizer.

Limits

600 preference pairs is a small budget for preference optimization. Expect sharper formatting, less padding and more consistent refusals -- not new capability. Preference labels come from an LLM judge, so the model inherits that judge's blind spots. Not legal or financial advice.