singhpratech/emc2-lesson-06-lora

e=mc² Lesson 06: LoRA explained by doing it

Lesson page: https://theaivibe.org/emc2/lora-explained-fine-tune-two-percent-weights

Lesson 06 of e=mc², a learn-by-building course by Prateek Singh. LoRA written by hand in PyTorch (no PEFT library), trained on Lesson 05's instruction pairs for Lesson 05's 500 steps, starting from tiny-storyteller-v2 (repo singhpratech/emc2-lesson-04-fine-tune), and compared with Lesson 05's full fine-tune.

Contents

• lesson_06.ipynb: the lesson, unexecuted (Colab T4, Apple-silicon Mac or CPU).
• RESULTS.md: every number from the run on 2026-10-11 (Apple Silicon Mac).
• adapters/: four rank-5 adapters (learning rates 3e-4, 1e-3, 3e-3, 1e-2), 0.51 MB each, and adapter_config.json (kept adapter: lr 3e-3, lowest held-out loss).
• merge_lora.py: folds an adapter into tiny-storyteller-v2 and saves a plain model.

Result

123,392 trainable parameters (2.11% of 5,853,696): rank 5 on qkv, proj and both feed-forward layers of all 6 blocks, plus the two new token rows.

	check (character AND place, 50 held-out instructions)	held-out loss
full fine-tune (Lesson 05)	44%	2.145
LoRA, lr 3e-3 (kept)	14%	2.211
LoRA, lr 1e-2	26%	2.216

Per step, LoRA was 1% to 9% faster than full fine-tuning across four timings on a shared GPU; the adapter file is 0.51 MB against 23.44 MB. On this task LoRA did not match the full fine-tune. Merged and unmerged outputs match (max logit difference 3e-05).

Merge

python merge_lora.py --adapter adapters/lora_r5_lr0.003.pt --out merged

Use the tokenizer from singhpratech/emc2-lesson-05-instruct (tiny-storyteller-instruct/) and the chat template <|endoftext|><|instruction|>{instruction}<|story|>.