decoder-only-transformer-tinystories-32m

A custom GPT-style language model trained from scratch using PyTorch.

Model Details

Parameter	Value
Architecture	GPT (Decoder-only Transformer)
Hidden size (d_model)	768
Attention heads	8
Transformer blocks	1
Max sequence length	1024
Vocabulary size	32000
Dropout	0.2

Tokenizer

Custom BPE tokenizer trained with the HuggingFace tokenizers library.

Special tokens: <|endoftext|> · <|pad|> · <|unk|>

Quick Start

You can easily load this model and tokenizer using the transformers library. Because the model uses a custom architecture, you must pass trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("shubhamprakash108/decoder-only-transformer-tinystories-32m", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("shubhamprakash108/decoder-only-transformer-tinystories-32m", trust_remote_code=True)

# Set up device
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

# Generate text
prompt = "The transformer is based on"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
output_ids = model.generate(
    **inputs, 
    max_new_tokens=50, 
    do_sample=True, 
    temperature=0.8,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Training Details

• Optimizer: AdamW (lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
• Scheduler: CosineAnnealingLR (eta_min=1e-5)
• Loss: CrossEntropyLoss (next-token prediction)
• Gradient clipping: max_norm=1.0

License

Apache 2.0