moralgym/llama-3-8b-instruct-pd-deon-lora

Llama-3-8B-Instruct — PD deontological fine-tune (LoRA)

LoRA adapter fine-tuned with GRPO on the iterated Prisoner's Dilemma using a deontological intrinsic reward, as part of the MoralGym project.

This is the abl4 recipe (lr=2.5e-5, hist_coop_bias=0.5) — the recipe that produced clean strict reciprocity (1, 0, 1, 0) on Gemma-2-9B-IT. Per-cell behavior eval pending.

Training recipe

• Method: GRPO with deontological intrinsic reward
• Intrinsic: r_deon = -1 if (action == D and opp_prev == C) else 0
• Composite: r_total = r_game_norm + 0.75 * r_deon (game reward normalized, λ=0.75)
• Game: Prisoner's Dilemma, payoffs sampled per episode from [1, 10] with PD ordering, 2R > T+S
• Training opponent: Tit-for-Tat
• Episode design: single-turn with fabricated history, hist_coop_bias=0.5 (uniform fab_opp)
• Prompt randomization: layout, prose-position, role (rows ↔ cols), opener/closer order
• LoRA: rank=32, alpha=64, attention + MLP modules, all layers
• Steps: 70 (step_70 checkpoint released here)
• GRPO: 16 prompts/step × 16 generations/prompt
• Optimizer: AdamW, lr=2.5e-5 (5× the 2B baseline), weight_decay=0.01

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
tok  = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
model = PeftModel.from_pretrained(base, "moralgym/llama-3-8b-instruct-pd-deon-lora")

License

The adapter weights inherit the Meta Llama 3 license from the base model. See meta-llama/Meta-Llama-3-8B-Instruct for terms.