Breezy-78M

A sovereign 78M-parameter Llama-architecture LM trained from scratch, then aligned for honesty and creativity.

Architecture

Llama-style: 22 layers, hidden 512, GQA 8/2 heads, tied embeddings, 32k vocab, 2048 ctx. 78,404,096 params.

Training Timeline

• Phase 1 (Pretrain): 48k steps, ~1.8B tokens, curriculum-ordered FineWeb + FineWeb-Edu, 512->1024 context stages, resume-safe on Kaggle T4 x2.
• Phase 2 (SFT): 12k steps on 59,174 OpenHermes-2.5 conversations + anti-hallucination honesty injection.
• Phase 2.5 (Polish + Replay): 500-step constraint pass, then 100-step mixed rehearsal to cure catastrophic forgetting.

Evaluation (internal QA Lab, 22 probes)

• Identity and honesty refusals: working (refuses future events, live data, fake entities).
• Creativity: 2-3/3 (poems, stories, invention).
• Instruction formatting: partial (lists/caps unstable at 78M scale).
• Known limits: arithmetic, dense geography, non-English, syllogisms (expected at this size).

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("kikusuka/breezy-78m")
model = AutoModelForCausalLM.from_pretrained("kikusuka/breezy-78m", torch_dtype="float16")
p = "### Instruction:\n{YOUR QUESTION}\n\n### Breezy:\n"