Instructions to use submarat/gpt2-small-fineweb-edu-10b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use submarat/gpt2-small-fineweb-edu-10b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="submarat/gpt2-small-fineweb-edu-10b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("submarat/gpt2-small-fineweb-edu-10b")
model = AutoModelForCausalLM.from_pretrained("submarat/gpt2-small-fineweb-edu-10b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use submarat/gpt2-small-fineweb-edu-10b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "submarat/gpt2-small-fineweb-edu-10b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "submarat/gpt2-small-fineweb-edu-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/submarat/gpt2-small-fineweb-edu-10b

SGLang

How to use submarat/gpt2-small-fineweb-edu-10b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "submarat/gpt2-small-fineweb-edu-10b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "submarat/gpt2-small-fineweb-edu-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "submarat/gpt2-small-fineweb-edu-10b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "submarat/gpt2-small-fineweb-edu-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use submarat/gpt2-small-fineweb-edu-10b with Docker Model Runner:
```
docker model run hf.co/submarat/gpt2-small-fineweb-edu-10b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

GPT-2 Small (124M) — reproduced on 10B tokens of FineWeb-Edu

A from-scratch reproduction of GPT-2 small (124M), trained on 10B tokens of FineWeb-Edu (sample-10BT) on a single H100 (~13 h). The custom training implementation was converted to a standard GPT2LMHeadModel for distribution (verified numerically exact vs. the original model, max logit diff ~1e-6).

Code: https://github.com/submarat/gpt2-small-repro
Write-up: https://submarat.github.io/reproducing-gpt2-small/

Training


Params	124.5M (tied embeddings)
Data	FineWeb-Edu `sample-10BT`, 10B tokens (1 epoch)
Context	1024
Batch	524,288 tokens/step (32 × 16 grad-accum)
Optimizer	AdamW (0.9, 0.95), wd 0.1 on ≥2D params, grad clip 1.0
LR	6e-4 peak, cosine → 6e-5, 700-step warmup
Precision	bf16 autocast
Final loss	train 3.27 / val 3.31

Evaluation (0-shot, lm-evaluation-harness)

Task	Metric	This model	Public `gpt2`
arc_easy	acc_norm	0.435	0.396
sciq	acc_norm	0.655	0.642
hellaswag	acc_norm	0.290	0.312
piqa	acc_norm	0.599	0.622
lambada_openai	acc	0.188	0.309
wikitext	word ppl (↓)	54.7	37.8

The profile is the FineWeb-Edu data fingerprint: it beats GPT-2 on knowledge (arc_easy, sciq), is near-parity on commonsense (hellaswag, piqa), and trails on distribution-sensitive tasks (lambada = narrative text, wikitext = encyclopedic) that FineWeb-Edu's quality filtering under-represents relative to GPT-2's WebText.

Usage

from transformers import GPT2LMHeadModel, GPT2TokenizerFast

model = GPT2LMHeadModel.from_pretrained("submarat/gpt2-small-fineweb-edu-10b")
tok = GPT2TokenizerFast.from_pretrained("submarat/gpt2-small-fineweb-edu-10b")

ids = tok("The process of photosynthesis", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_k=40, temperature=0.8)
print(tok.decode(out[0]))

Notes & caveats

This is a 124M base model (~GPT-2-2019 level): no instruction following, it hallucinates, reasoning is weak. It is an educational / research artifact, not a product, and is not competitive with modern small models trained on far more data.
Activation: trained with exact (erf) GELU, so the config sets activation_function="gelu" (not GPT-2's tanh-approx gelu_new).
Intended uses: studying pretraining dynamics, interpretability (an induction head forms at layer 11 within the first ~2B tokens), and as a small research base model.

Downloads last month: 66

Safetensors

Model size

0.1B params

Tensor type

F32

Model tree for submarat/gpt2-small-fineweb-edu-10b

Quantizations

2 models

submarat
/

gpt2-small-fineweb-edu-10b