jgaeb/rubric-rewards-adapters

Adapters for "From Constitutions to Control: Interpretable Rewards for Aligning Language Models"

This repository holds the 445 LoRA adapters from our paper: 354 fine-tuned policies, the 90 neural reward models trained on preferences with injected gender bias, and the adapter that wrote GPT-OSS's softened harm candidates. The data behind the paper are in the dataset jgaeb/rubric-rewards, and the code is at github.com/jgaeb/rubric-rewards.

Each base model has its own folder (gemma/, qwen/, gptoss/) with its own card. Within it:

• free/<arm>/: policies trained on responses selected by the unconstrained rubric reward. The arm name gives the steering weights, as in help0.5_lean-1 (β_help = 0.5, β_lean = −1) or syc2 (β_sycophancy = 2).
• constrained/<arm>/: the same policies under the sign-constrained reward.
• discrimination/<rm>_q<q>/: policies trained on the hiring recommendations selected by a reward model fit to preferences with injected bias q.
• dose_rm/<rm>_q<q>/: the neural reward models behind those selections.
• generation/harm_softened/ (GPT-OSS only): the adapter that rewrote harmful responses into softened ones.

INDEX.jsonl lists every adapter with its size. To load a policy:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", torch_dtype="bfloat16")
policy = PeftModel.from_pretrained(base, "jgaeb/rubric-rewards-adapters", subfolder="gptoss/free/help1_lean+3")

Training

The policies are LoRA adapters of rank 16 and scaling 32, with dropout 0.05, on the attention projections. We trained them by supervised fine-tuning in bf16 with AdamW, a learning rate of 1e-4 with cosine decay, an effective batch size of 16, and seed 0, for two epochs over sequences of up to 1,024 tokens, and kept the final weights. Warmup was 3% of steps for Gemma and Qwen and 10 steps for GPT-OSS, whose expert weights we dequantized from MXFP4 for training. Qwen and GPT-OSS were trained and evaluated without reasoning.

Each steering policy saw 1,000 harm and 1,000 political prompts with the responses its reward selected, plus 500 Tülu-3 examples. Each sycophancy policy saw 1,000 feedback prompts plus 250 Tülu-3 examples, and each discrimination policy saw 2,000 hiring prompts plus 210 Tülu-3 examples.

The dose reward models are LoRA adapters of the same size on the attention and MLP projections, with the scoring head trained in full. We trained them with a Bradley-Terry loss for three epochs, using AdamW at a constant learning rate of 1e-5, weight decay 0.01, batch size 2, sequences of up to 512 tokens, and seed 0. QRM was trained in 4-bit, DeBERTa in fp32, and the Skywork models in bf16.

We sampled every evaluation with vLLM at temperature 0.7 and top-p 0.9, with up to 512 new tokens.

License

The adapters inherit the licenses of their base models: the Gemma Terms of Use for Gemma, and Apache-2.0 for Qwen3 and GPT-OSS.