ssy1245/nanoGPT_hw

NanoGPT Story Generation (32M)

Overview

This repository contains a 32M-parameter nanoGPT model trained for story generation.

The model is trained using a two-stage pipeline:

• Pretraining on TinyStories
• Fine-tuning on ROCStories

This improves both language fluency and narrative coherence.

---

Model Details

• Architecture: nanoGPT + RMSNorm
• Parameters: 31.7M
• Context length: 256
• Tokenizer: GPT-2 BPE (tiktoken)

---

Training Pipeline

Stage 1: Pretraining (TinyStories)

• Dataset: TinyStories
• Purpose: Learn basic language structure and simple narrative patterns
• Each story is treated as an independent sample
• <|endoftext|> is appended to each story

---

Stage 2: Fine-tuning (ROCStories)

• Dataset: ROCStories
• Purpose: Adapt to 5-sentence structured storytelling
• Same tokenizer and preprocessing
• Stories concatenated with <|endoftext|>

---

Preprocessing

• Tokenizer: GPT-2 BPE
• Special token: <|endoftext|>
• Data stored as a continuous token stream
• Train/validation split at story level (90/10 for ROCStories)

---

Training Configuration

Stage 1 (Pretraining)

• Steps: 30000
• Learning rate: 3e-4
• Final validation loss: 1.37

Stage 2 (Fine-tuning)

• Steps: 2000
• Learning rate: 5e-5
• Final test loss: 3.14
• Perplexity: 21.5

---

Files

• ckpt.pt — trained model checkpoint
• model.py — model architecture
• train_tiny.py — pretraining script
• roc_finetunning.py — fine-tuning script
• data_prepare/ — preprocessing pipeline

---

Notes

• The model is trained from scratch without using any pretrained weights
• <|endoftext|> is used as a separator to maintain story boundaries
• The two-stage training pipeline improves both fluency and narrative consistency