Llama-10M-Base

A tiny 10-million parameter Llama-style model trained from scratch on a curated 95M token dataset.

Model Details

• Architecture: Decoder-only Transformer (Llama-style)
• Features: Rotary Position Embeddings (RoPE), RMSNorm, SwiGLU MLP, No Bias
• Parameters: ~19.15 Million (embeddings + layers)
• Layers: 8
• Heads: 8
• Embedding Dimension: 256
• Context Length: 512 tokens
• Vocabulary Size: 50,257 (custom BPE tokenizer)

Training Stages

• Pretraining: Trained from scratch for 3000 steps (batch size 128) over a 95M curated flat-packed token dataset.
• Total Tokens Seen: ~196.6 Million tokens (about 2.07 epochs over the curated dataset).
• Final Validation Loss: 3.6980 (down from step 0 loss of 10.87).

---

⚠️ Git LFS Warning & Quickstart

If you use git clone to download this repository, you must have Git LFS installed (git lfs install followed by git lfs pull), otherwise best_model_10m.pt will just be a tiny pointer file and PyTorch will fail to load it.

Best Way: Load dynamically using Python

To avoid Git LFS issues entirely, you can run the following script which automatically downloads the model, the tokenizer, and the model definition dynamically from Hugging Face Hub (Windows, Mac, and Linux compatible):

import os
import torch
from huggingface_hub import hf_hub_download

# Select the model repository
repo_id = "nowefnyieryfner/llama-10m-base"
model_filename = "best_model_10m.pt"

print(f"Downloading model files from {repo_id}...")

# Download files from Hugging Face Hub (cached automatically)
model_path = hf_hub_download(repo_id=repo_id, filename=model_filename)
model_code_path = hf_hub_download(repo_id=repo_id, filename="model.py")
tokenizer_path = hf_hub_download(repo_id=repo_id, filename="tokenizer.json")

# Dynamically import GPT from the downloaded model.py
import importlib.util
spec = importlib.util.spec_from_file_location("model", model_code_path)
model_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(model_module)
GPT = model_module.GPT

# Load tokenizer
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file(tokenizer_path)

# Load model weights
print("Loading model weights...")
checkpoint = torch.load(model_path, map_location="cpu")
config = checkpoint["config"]
model = GPT(config)
model.load_state_dict(checkpoint["model"])
model.eval()

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
print(f"Model loaded successfully on {device}!")

# Sample prompt
prompt = "<|im_start|>user\nWrite a python function to check if a number is prime.<|im_end|>\n<|im_start|>assistant\n"
x = torch.tensor(tokenizer.encode(prompt).ids, dtype=torch.long, device=device).unsqueeze(0)

print("\n--- Generating response ---")
y = model.generate(x, max_new_tokens=100, temperature=0.8, top_k=50)
print(tokenizer.decode(y[0].tolist()))

Installation

Make sure you have the required packages installed:

pip install torch tokenizers huggingface_hub