nanochat-keith

This is a custom-trained nanochat model uploaded from checkpoint: base/d20 at step 21400.

Model Details

• Architecture: GPT-style transformer
• Training Stage: BASE
• Model Tag: d20
• Training Step: 21400

Model Configuration

{
  "sequence_len": 2048,
  "vocab_size": 65536,
  "n_layer": 20,
  "n_head": 10,
  "n_kv_head": 10,
  "n_embd": 1280
}

Training Configuration

{
  "run": "dummy",
  "device_type": "",
  "depth": 20,
  "max_seq_len": 2048,
  "num_iterations": -1,
  "target_flops": -1.0,
  "target_param_data_ratio": 20,
  "device_batch_size": 32,
  "total_batch_size": 524288,
  "embedding_lr": 0.2,
  "unembedding_lr": 0.004,
  "weight_decay": 0.0,
  "matrix_lr": 0.02,
  "grad_clip": 1.0,
  "warmup_ratio": 0.0,
  "warmdown_ratio": 0.2,
  "final_lr_frac": 0.0,
  "eval_every": 250,
  "eval_tokens": 10485760,
  "core_metric_every": 2000,
  "core_metric_max_per_task": 500,
  "sample_every": 2000,
  "model_tag": ""
}

Performance Metrics

• Validation BPB: 0.8122105833369838
• Validation Loss: N/A

Usage

import torch
from nanochat.checkpoint_manager import build_model
from nanochat.tokenizer import get_tokenizer
from nanochat.engine import Engine

# Download and load the model
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model (you'll need to download the checkpoint files)
checkpoint_dir = "./checkpoint"  # directory with model files
step = 21400
model, tokenizer, meta = build_model(checkpoint_dir, step, device, phase="eval")

# Create engine for generation
engine = Engine(model, tokenizer)

# Generate text
prompt = "Hello, how are you?"
tokens = tokenizer(prompt, prepend="<|bos|>")
samples, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=100, temperature=0.8)
print(tokenizer.decode(samples[0]))

Training Details

This model was trained using the nanochat framework by Andrej Karpathy.

Training Data

• Custom SFT data: keith_identity.jsonl, keith_reputation.jsonl
• Base datasets: SmolTalk, MMLU, GSM8K, ARC, SpellingBee
• Training stage: base

Computational Requirements

Training was performed following the nanochat speedrun pipeline.

Citation

If you use this model, please cite the original nanochat project:

@misc{nanochat,
  author = {Karpathy, Andrej},
  title = {nanochat: A minimalistic ChatGPT clone},
  year = {2024},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

License

This model follows the MIT license of the nanochat project.

Checkpoint Lineage and Retention

This repository is the BASE checkpoint in the private lineage:

1. keithtyser/nanochat-keith — base checkpoint
2. keithtyser/nanochat-keith-mid — intermediate checkpoint
3. keithtyser/nanochat-keith-sft — supervised-fine-tuned checkpoint

Keep all three while reproducibility, regression comparison, rollback, or training resume matters. Stage-specific data claims in the older card text should be treated as provisional unless they are backed by a dataset manifest and training log; the datasets themselves are not bundled here.