EveryonesLLM workshop model

This model was trained from scratch in the EveryonesLLM GPU workshop.

Training data

Dataset: HuggingFaceFW/fineweb-edu
Sample: sample/10BT/000_00000.parquet
Tokenizer: tiktoken GPT-2

Model

Embedding dimension: 512
Hidden dimension: 2048
Attention heads: 8
Transformer blocks: 4
Context length: 512

Training

Batch size: 64
Steps: 2000
Learning rate: 0.001
BF16: True
Seen tokens: 65536000
Final validation loss: 4.865565538406372

Loading

This is a custom educational PyTorch model, not a Transformers AutoModel package. Use the EveryonesLLM workshop source code to instantiate nanoGPT, then load model.pt.

Notes

The code originates from the MIT-licensed EveryonesLLM project. FineWeb-Edu is distributed under ODC-By. Check the dataset and repository terms before redistributing derived artifacts.

Downloads last month: 162