singhamAstram/micro-gpt-tinystories

MicroGPT - Tinystories

A minimal character-level GPT model trained from scratch on the tinystories dataset.

Model Details

Property	Value
Architecture	MicroGPT (decoder-only transformer)
Parameters	419,712
Vocab Size	74 (character-level)
Block Size	128 tokens
Layers	2
Attention Heads	4
Embedding Dim	128
Training Steps	500
Best Val Loss	2.2494189739227295
Dataset	tinystories

Quick Start

# Clone the repo
git clone https://huggingface.co/{{cookiecutter.repo_id if cookiecutter else 'your-username/micro-gpt-' + config['dataset']}}
cd micro-gpt-tinystories

# Install dependencies
pip install -r requirements.txt

# Generate text
python inference.py --prompt "Once upon a time"

Usage in Python

import torch
from models.micro_gpt import MicroGPT

# Load config
import json
with open('config.json') as f:
    config = json.load(f)

# Build model
model = MicroGPT(
    vocab_size=config['vocab_size'],
    block_size=config['block_size'],
    n_layer=config['n_layer'],
    n_head=config['n_head'],
    n_embd=config['n_embd'],
    dropout=config['dropout'],
)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

# Load tokenizer
with open('tokenizer.json') as f:
    tokenizer = json.load(f)

# Encode prompt
prompt = "Once upon a time"
indices = [tokenizer['stoi'].get(c, 0) for c in prompt]
input_ids = torch.tensor([indices], dtype=torch.long)

# Generate
output_ids = model.generate(input_ids, max_new_tokens=200, temperature=0.9, top_k=40)
text = ''.join(tokenizer['itos'][str(i)] for i in output_ids[0].tolist())
print(text)

Training

This model was trained using the project's training pipeline:

python run_model.py --train_model --arch micro_gpt --dataset tinystories --max_steps 500

Files in this repository

File	Description
models/	Full model source code (MicroGPT architecture)
pytorch_model.bin	Trained model weights
config.json	Model hyperparameters
tokenizer.json	Character-level tokenizer
inference.py	Ready-to-use inference script
requirements.txt	Python dependencies