GPT-2 TensorFlow checkpoints
This repository mirrors OpenAI's original GPT-2 checkpoints for the weight-loading example in Chapter 5 of Build a Large Language Model (From Scratch).
The four folders, 124M, 355M, 774M, and 1558M, follow OpenAI's directory layout. Each contains the TensorFlow checkpoint, its hyperparameters (hparams.json), and the GPT-2 tokenizer files (encoder.json and vocab.bpe).
Loading a checkpoint
The following example uses the Chapter 5 downloader with the source argument to load the 124M checkpoint directly from Hugging Face.
from gpt_download import download_and_load_gpt2
settings, params = download_and_load_gpt2("124M", models_dir="gpt2", source="hf")
The function saves the seven files in gpt2/124M/ and returns the settings and NumPy arrays used in the chapter. Change "124M" to load one of the larger models. You can also download the seven files directly from a model folder on the Hub.
OpenAI released the weights under its modified MIT license. For the same weights converted to PyTorch state dictionaries and safetensors, see rasbt/gpt2-from-scratch-pytorch.