pleyva2004/scholastic-llm-sft-v2-iter400-peft

scholastic-llm SFT-v2 @ iter 400 — PEFT format

⚠ NOTICE — research experiment, not theological authority

This is a personal portfolio / research project exploring how small open-weights LLMs can be fine-tuned to adopt a specific historical register and citation style. The trained model is not a reliable source of Catholic doctrine, biblical interpretation, or philosophical truth. It can hallucinate citations, misrepresent the Catechism, and confidently err. Outputs must not be cited as catechetical instruction, theological argument, or spiritual direction.

What this is

PEFT/Transformers-compatible LoRA adapter for Qwen/Qwen2.5-7B-Instruct, trained to respond to philosophical and theological questions in a scholastic / Latin-inflected register grounded in the Catechism of the Catholic Church (CCC, 1992), in the structural voices of Aquinas's Summa Theologica and Augustine's Confessions / City of God.

This is the best Phase 2 checkpoint (iter 400 of 800, before mild overfitting). It matches Phase 1's strict rubric total (68/120) and beats it on the balanced rubric (68/90 vs 66/90); closes the Augustinian-voice gap.

This adapter was converted from the original MLX-format adapter (pleyva2004/scholastic-llm-sft-v2-iter400) via the open-source scripts/mlx_to_peft.py converter. The weights apply LoRA to the top 16 of 28 transformer layers.

How to load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "pleyva2004/scholastic-llm-sft-v2-iter400-peft")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

messages = [{"role": "user", "content": "Is the soul immortal?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=400, do_sample=True, temperature=0.7)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Try it live

🤗 Hugging Face Space (free, ZeroGPU)

Training (summary)

Base	Qwen/Qwen2.5-7B-Instruct
Method	LoRA (rank 8, alpha 80) on top 16 of 28 transformer layers
Optimizer	AdamW, LR 1e-5, batch 1, max seq 2048
Iterations	400 (best checkpoint within an 800-iter run)
Training data	377 teacher-distilled (question, scholastic-answer) pairs
Teacher	Claude Sonnet 4.6
Hardware	Apple M4 Pro, 48 GB unified memory, via MLX (then converted to PEFT)
Trainable parameters	2.6M / 7.6B (0.034 %)

Full method, rubric, and evaluation details in the paper.

Evaluation

Rubric on 10 held-out philosophical prompts (max 30 per dimension, strict total max 120, balanced total max 90):

Dimension	BASE	This adapter	Δ
Scholastic register	3	21	+18
Augustinian voice	0	7	+7
CCC grounding	0	18	+18
Structure	16	22	+6
Strict total	19	68	+49
Balanced total	19	68	+49

Sibling adapters

• MLX format (original) — for mlx-lm on Apple Silicon
• Phase 1 paper headline (MLX)
• Phase 2 final iter 800, mild overfit (MLX)
• DPO-v3, documented negative result (MLX)

License

• This adapter: MIT
• Base model: Apache 2.0 (Qwen 2.5)
• Training data: see DATA_LICENSING.md

Citation

@misc{leyva2026scholastic,
  title  = {Teaching a Small LLM Scholastic Voice: Fine-Tuning Qwen 2.5 on the Catechism, Summa, and Augustine via Local MLX},
  author = {Pablo Leyva},
  year   = {2026},
  url    = {https://github.com/pleyva2004/scholastic-llm}
}