pleyva2004/scholastic-llm-dpo-v3

scholastic-llm DPO-v3 (negative result: saturation)

⚠ NOTICE — research experiment, not theological authority

This is a personal portfolio / research project exploring how small open-weights LLMs can be fine-tuned to adopt a specific historical register and citation style. The trained model is not a reliable source of Catholic doctrine, biblical interpretation, or philosophical truth. It can hallucinate citations, misrepresent the Catechism, and confidently err. Outputs must not be cited as catechetical instruction, theological argument, or spiritual direction. For doctrinal questions, consult the actual Catechism of the Catholic Church, a qualified priest, or a trained theologian.

What this is

LoRA adapter for Qwen/Qwen2.5-7B-Instruct, trained to respond to philosophical and theological questions in a scholastic / Latin-inflected register grounded in the Catechism of the Catholic Church (CCC, 1992) and modeled after Aquinas's Summa Theologica and Augustine's Confessions / City of God.

Phase 2 DPO refinement chain on top of SFT-v2 (iter 800). 300 DPO iterations on ~50 preference pairs (chosen = SFT-v2 output, rejected = base output). Documented negative result: val loss = 0.000 from iter 1, val accuracy = 1.000, chosen/rejected margin 35.4 nats. Within-model preference data saturated the policy at initialization; DPO had no gradient signal. The resulting adapter is functionally identical to its SFT-v2 starting point. Published as a teaching artifact for the setup pitfall, not as a recommended model.

For full background, recipe, and Phase 1+2 results, see:

• Paper: https://pleyva2004.github.io/scholastic-llm/main.pdf
• GitHub: https://github.com/pleyva2004/scholastic-llm

Variants (all four published)

Variant	Iters	Training data	Strict total	Balanced total
sft-v1	200	83 Q/A pairs	68/120	66/90
sft-v2-iter400 ⭐	400	377 Q/A pairs	68/120	68/90
sft-v2	800	377 Q/A pairs	64/120	64/90
dpo-v3 (this card)	300	50 preference pairs (DPO)	64/120	63/90

(Strict total $= \textsc{reg} + \textsc{aug} + \textsc{ccc} + \textsc{str}$, max 120. Balanced total $= \max(\textsc{reg},\textsc{aug}) + \textsc{ccc} + \textsc{str}$, max 90; introduced in Phase 2 because the strict total penalizes appropriate register switching.)

How to load (MLX)

from mlx_lm import generate, load

model, tokenizer = load(
    "Qwen/Qwen2.5-7B-Instruct",
    adapter_path="pleyva2004/scholastic-llm-dpo-v3",
)

prompt = "How do you reconcile divine foreknowledge with free will?"
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=text, max_tokens=300))

Requires mlx-lm (≥ 0.27) on Apple Silicon. For other inference engines you will need to convert the adapter manually; the repo contains MLX-format weights only.

Training

Base	Qwen/Qwen2.5-7B-Instruct
Quantization	MLX 8-bit weight quantization (Q8)
Method	LoRA on top 16 of 28 transformer layers
Optimizer	AdamW
Learning rate	$10^{-5}$
Batch size	1
Max sequence length	2048
Iterations	300
Training data	50 preference pairs (DPO)
Hardware	Apple M4 Pro, 48 GB unified memory
Trainable parameters	2.6M / 7.6B (0.034%)
Peak resident memory	12.7 GB

Phase 2 extras

DPO step on top of SFT-v2: β=0.05, sigmoid loss, reference = base, 300 DPO iterations.

Training data was generated by Claude Sonnet 4.6 as teacher, per a strict system prompt requesting scholastic register and CCC citations, applied to ~50–150 cleaned source chunks scraped from the Catechism, the Summa, and Augustine's works. See the GitHub repo's scripts/generate_training_pairs.py for the exact prompt.

Evaluation

Rubric-based evaluation on 10 held-out philosophical prompts (not seen during training). Four dimensions, each scored 0–3 per prompt, summed across 10 prompts (per-dimension max 30).

Dimension	BASE	This variant	Δ vs BASE
Scholastic register (Summa markers)	3	16	+13
Augustinian voice (autobiographical)	0	12	+12
CCC grounding (paragraph citations)	0	20	+20
Structure (multi-para, obj/reply)	16	16	+0
Strict total	19	64	+45
Balanced total ($\max(\textsc{reg},\textsc{aug}) + \textsc{ccc} + \textsc{str}$)	19	63	+44

Full per-prompt scores and qualitative samples are in the paper.

Data licensing

Training data sources:

Source	Status
Catechism of the Catholic Church (1992)	© USCCB / Libreria Editrice Vaticana; used under fair-use research posture
Summa Theologica (Shapcote 1920)	Public domain (US)
Augustine — Confessions (Pusey trans.)	Public domain
Augustine — City of God (Dods trans.)	Public domain

The training-data JSONL itself is not redistributed with this adapter; only the LoRA weights and this card. See DATA_LICENSING.md for the full posture.

License

• This adapter (LoRA weights): MIT — see LICENSE in the repo.
• Base model (Qwen/Qwen2.5-7B-Instruct): Apache 2.0 (governed by the base-model card on Hugging Face).
• Source corpus: terms above.

Limitations & ethics

• Hallucinated citations. The fine-tuned model confidently emits CCC paragraph numbers with the surface form of ground truth. Many citations do not correspond to the actual content of the cited paragraph. Always verify against the actual Catechism.
• No human evaluation. Reported numbers come from a regex/keyword rubric. The rubric measures lexical and structural surface form, not theological correctness.
• Small held-out set (N=10). Confidence intervals are wide; the +49-point delta is large relative to noise but not bootstrapped.
• No doctrinal authority. The model speaks in a voice culturally associated with magisterial authority. It has none. It can confidently err and should not be relied upon for spiritual direction.

Citation

@misc{leyva2026scholastic,
  title  = {Teaching a Small LLM Scholastic Voice: Fine-Tuning Qwen 2.5 on the Catechism, Summa, and Augustine via Local MLX},
  author = {Pablo Leyva},
  year   = {2026},
  url    = {https://github.com/pleyva2004/scholastic-llm},
  note   = {Independent Research}
}