TinyJLLM — 100M-parameter small language model built from scratch

A decoder-only Transformer (~102.5M parameters) pretrained from random initialization on ~5 GB of FineWeb (sample-10BT), 3 epochs / 108,000 optimizer steps. Built as a fully educational pipeline (LearnLLM Run #2): custom 32K byte-level BPE tokenizer, from-scratch Transformer, sharded uint16 data pipeline, BF16 training, and verified exports.

Final metrics: validation loss 3.50 (perplexity 33.1); the best checkpoint (step 89K) reached 3.48 / 32.6.

Model details

Property	Value
Parameters	102,450,432 (~102.5M)
Architecture	Llama-style decoder-only: RMSNorm, RoPE (half-split), SwiGLU, tied embeddings, no biases
Layers / heads / head_dim	11 / 12 / 64
Context length	512
Vocabulary	32,000 (custom byte-level BPE, <pad> <unk> <bos> <eos> = 0-3)
Pretraining data	FineWeb sample-10BT, ~5.37 GB raw, 1.75M documents
Tokens seen	3.54B (3 epochs)
Hardware	RTX 4060 8 GB, ~30K tok/s (torch.compile)
Precision	BF16 mixed precision, FP32 master weights

Intended use

• Educational reference: inspect a small, complete, honest pretraining run.
• Qualitative experimentation: prompt it (it follows prompts as text; it is a base model — no instruction tuning yet).
• A base for further stages (SFT, DPO, domain fine-tuning).

Known limitations: small scale ⇒ repetition in long generations, weak instruction following, limited world knowledge.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("jaweed123/TinyJLLM")
tokenizer = AutoTokenizer.from_pretrained("jaweed123/TinyJLLM")

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, temperature=0.8, top_k=50, top_p=0.95)
print(tokenizer.decode(out[0]))

llama.cpp / GGUF

The repo also ships GGUF files under gguf/ (F16, Q8_0, Q4_K_M) — load directly with llama.cpp or llama-cpp-python.

Training details

• Custom 32K byte-level BPE (trained on a 512 MB FineWeb sample).
• Tokens stored once as uint16 shards (591 train + 6 validation).
• AdamW (lr 3e-4, wd 0.1, decay/no-decay groups), warmup 1,000 + cosine to 1e-5, effective batch 64 (32,768 tokens/step), gradient clipping 1.0.
• Full run: ~35 h on an RTX 4060.

Files

• config.json — Llama-compatible config (LlamaForCausalLM)
• model.safetensors — FP32 weights
• tokenizer.json / tokenizer_config.json — custom BPE
• generation_config.json — decoding defaults
• gguf/ — llama.cpp formats

Acknowledgments

FineWeb (HuggingFaceFW), Hugging Face tokenizers / datasets, PyTorch, llama.cpp. Built with the LearnLLM educational pipeline (src/learnllm at the project repository).