🚀 nanochat-d12: 286M GPT-2 Model (Pretrained + SFT)

A lightweight, high-performance GPT-2 grade Large Language Model (LLM) trained and evaluated entirely using the nanochat framework.

This repository contains both the pretrained Base model weights and the SFT (Supervised Fine-Tuned) conversational weights.

🗂️ Repository Structure

• /sft: Fine-tuned conversational model weights and optimizer states (Step 7).
• /base: Pretrained base model weights and optimizer states (Step 1,680).

---

📊 Model Architecture

• Parameters: 286 Million (total: 286,261,730)
• Layers (n_layer): 12
• Heads (n_head): 6
• Embedding Dimension (n_embd): 768
• Context Length (sequence_len): 2,048 tokens
• Attention Type: Flash Attention 3 with a Sliding Window pattern (SSSL)
• Vocabulary Size: 32,768 (trained via custom BPE Tokenizer)
• Precision: bfloat16 (BF16)

---

📈 Training Details & Performance

1. Base Model Pretraining

• Dataset: karpathy/climbmix-400b-shuffle (1 Billion tokens)
• Hardware: 1x NVIDIA GeForce RTX 4090 GPU (RunPod)
• Duration: 96.82 minutes
• Power Draw: 447W (99% GPU Utilization)
• Throughput: ~150,800 tokens/second
• Model Flops Utilization (MFU): 69.3% (extremely high efficiency)
• Final Train Loss: 2.88
• Validation Bits-per-Byte (BPB): 0.8680
• Optimizer: Muon (for matrix parameters, LR 0.02) + AdamW (for embeddings/scalars, LR 0.3/0.5)

2. Supervised Fine-Tuning (SFT)

• Datasets: HuggingFaceTB/smol-smoltalk (Conversational), cais/mmlu (Multiple Choice), openai/gsm8k (Math/Tool Use)
• Duration: ~9 minutes
• Hardware: 1x NVIDIA GeForce RTX 4090 GPU (RunPod)

---

🏆 Downstream Evaluation (10-Problem SFT Snapshot)

Task / Benchmark	Accuracy	Type	Description
MMLU	50.00%	Multiple Choice	Academic & professional knowledge questions
ARC-Easy	50.00%	Multiple Choice	Elementary-level science questions
ARC-Challenge	30.00%	Multiple Choice	Harder, reasoning-heavy science questions
GSM8K	0.00%	Open-Ended	Grade-school math word problems
HumanEval	0.00%	Open-Ended	Python code-generation challenges
ChatCORE Score	0.1467	Aggregate	Mean centered accuracy score (Up from base model's 0.1399)

---

💬 Sample Prompts & Model Completions

1. Prompt: "Why is the sky blue?"

• Base Model (Raw):

  The sky is a blueish blue because of the water that washes over it. It is not because of the sun but because of the water... [repeats]

• SFT Model (Chat):

  The sky is blue, a vibrant, unforgiving landscape that is steeped in history and mythology. This blue sky is a result of the interplay of light, air, and water...

2. Prompt: "What is the capital of France?"

• SFT Response: The capital of France, Paris, is a city that is renowned for its charming architecture, charming boutiques, and stunning natural beauty. With a rich history, the city has become a hub for trade, art, and culture...

3. Prompt: "What is the chemical symbol of gold?"

• SFT Response: The chemical symbol of gold is Au, which is the Latin word for "gold." This symbol is derived from the Latin word for "gold," meaning "gold is a metal of great rarity and beauty."

---

💻 How to Use

To run inference or chat with this model locally:

1. Clone the repository:

   git clone https://github.com/irom77/nanochat.git
   cd nanochat
   

2. Download the model card and files from Hugging Face:

   pip install huggingface_hub
   python -c "
   from huggingface_hub import snapshot_download
   snapshot_download(repo_id='niuk77/nanochat-d12', local_dir='~/.cache/nanochat/chatsft_checkpoints/d12/')
   "
   

3. Run the interactive CLI:

   python -m scripts.chat_cli -i sft
   

📜 License

This model is licensed under the MIT License.