Instructions to use osjayaprakash/llm-0.5b-fineweb-edu with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use osjayaprakash/llm-0.5b-fineweb-edu with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="osjayaprakash/llm-0.5b-fineweb-edu")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")
model = AutoModelForCausalLM.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use osjayaprakash/llm-0.5b-fineweb-edu with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "osjayaprakash/llm-0.5b-fineweb-edu"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "osjayaprakash/llm-0.5b-fineweb-edu",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/osjayaprakash/llm-0.5b-fineweb-edu

SGLang

How to use osjayaprakash/llm-0.5b-fineweb-edu with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "osjayaprakash/llm-0.5b-fineweb-edu" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "osjayaprakash/llm-0.5b-fineweb-edu",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "osjayaprakash/llm-0.5b-fineweb-edu" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "osjayaprakash/llm-0.5b-fineweb-edu",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use osjayaprakash/llm-0.5b-fineweb-edu with Docker Model Runner:
```
docker model run hf.co/osjayaprakash/llm-0.5b-fineweb-edu
```

llm-0.5b-fineweb-edu

A 489M-parameter Llama-style base model pretrained from scratch on 4.19B tokens of FineWeb-Edu, on a single rented RTX 5090 for $13.70 of GPU time.

It exists to document what that budget buys, and the numbers below are all measured rather than estimated. It is not a model you should deploy — see Limitations.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")
tok = AutoTokenizer.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")

ids = tok("Photosynthesis is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=48, do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0], skip_special_tokens=True))

This is a base model: no instruction tuning, no chat template, no alignment. It continues text and nothing else.

Architecture


Parameters	489.3M total, 448.3M non-embedding
Layers	24
Model dimension	1280
Attention	Grouped-query, 20 query heads : 4 key/value heads, head dim 64
Feed-forward	SwiGLU, dimension 3840
Normalization	Pre-norm RMSNorm
Positional	RoPE, theta 10000
Vocabulary	32,000 byte-level BPE, trained on the same corpus
Context length	2048
Tied embeddings	yes

Architecturally a small Llama, so it loads as LlamaForCausalLM with no custom code. The export was verified by asserting logit equality against the original implementation: max absolute difference 3.2e-05.

Training

FineWeb-Edu sample-10BT, 2M documents tokenized to 2.03B training tokens plus 50M held out. The run made about two passes over that corpus, in three phases on the same GPU.

	Phase 1	Phase 2	Phase 3
Steps	0-2000	2000-4000	4000-8000
Tokens	1.05B	2.10B	4.19B
Peak learning rate	3e-4	2e-4	1.5e-4
Final learning rate	3e-5	2e-5	1e-5
Validation loss	3.3226	3.1387	2.9982
Wall clock	6.4 h	6.4 h	12.2 h

AdamW (0.9, 0.95), weight decay 0.1, gradient clipping 1.0, bf16 autocast, torch.compile, 524,288 tokens per optimizer step (batch 4 x 64 gradient accumulation x 2048). Cosine schedule with warmup in each phase. Final perplexity 20.05.

Across all 8,000 steps: no loss spikes, no non-finite gradients, 85% model-FLOPs utilization sustained at ~48,400 tokens/second.

Evaluation

Zero-shot via lm-evaluation-harness, every model run on the same machine with the same task versions.

Model	Tokens	Avg	LAMBADA	SciQ	ARC-Easy	BoolQ	PIQA	OpenBookQA	HellaSwag	WinoGrande	ARC-Challenge	WikiText ppl
This model	4.19B	0.445	0.230	0.660	0.458	0.621	0.628	0.312	0.326	0.502	0.265	42.1
Pythia-410M	2.1B	0.348	0.038	0.415	0.295	0.597	0.534	0.250	0.260	0.516	0.222	172.4
Pythia-410M	300B	0.493	0.479	0.735	0.458	0.598	0.675	0.300	0.406	0.538	0.247	20.8
SmolLM2-360M	4T	0.587	0.539	0.857	0.656	0.616	0.725	0.370	0.564	0.588	0.365	15.8
Qwen2.5-0.5B	18T	0.565	0.519	0.906	0.584	0.622	0.697	0.352	0.522	0.565	0.319	—

At a matched 2.1B tokens this model scores 9.7 points above Pythia-410M. It sits 4.8 points below the same model trained on 300B tokens — 72x more data — and well behind the modern small models, which is mostly a data-curation gap rather than an architectural one.

Accuracy against tokens, showing the run never flattened:

Tokens	262M	524M	786M	1.05B	2.10B	4.19B
Zero-shot average	0.328	0.393	0.404	0.412	0.433	0.445
WikiText perplexity	185.2	86.7	68.1	62.1	49.5	42.1

MMLU (5-shot): 0.272, against a chance baseline of 0.250. Almost all of that comes from social sciences (0.320); humanities is still at chance. Excluded from the average above.

Qwen2.5-0.5B has no WikiText figure because its 152k vocabulary exhausted GPU memory on that task's rolling windows; it was re-run without WikiText.

Limitations

Undertrained by design. 4.19B tokens is ~8.5 tokens per parameter, under half the Chinchilla-optimal ratio. The loss curve was still descending when the budget ran out.
It will state things that are false. At this scale it produces fluent, plausible, frequently wrong text. Do not use it as a source of facts.
No code ability. FineWeb-Edu is filtered educational web text with essentially no source code, so the model cannot write code and will not learn to from prompting.
Reasoning benchmarks sit near chance. ARC-Challenge 0.265 and WinoGrande 0.502 barely moved across the entire run.
English only, 2048-token context, no safety filtering or alignment of any kind. The training corpus is web text and carries its biases.
Not for production. It is a reference point for what a small budget buys.

Two things that did not work

Weight averaging did not help. An exponential moving average kept over the last 4,000 steps scored 0.4430 against the final checkpoint's 0.4448, and was slightly worse on perplexity. Averaging pays when training ends at a high learning rate and the weights still carry optimization noise; this run annealed to 1e-5, so the schedule had already done the averaging. The two are substitutes, not complements.

Restarting the cosine schedule cost 21% of the budget. Continuing a finished run means warming the learning rate back up, which undoes part of the previous annealing. Validation loss rose 3.3226 to 3.3827 at the phase-2 restart and took 600 steps to recover; phase 3 took 1,100, because its schedule is twice as long and holds the rate near peak for twice as long. 1,700 of 8,000 steps went on re-earning ground already taken. A warmup-stable-decay schedule avoids this entirely and is the right choice when the total budget is not known in advance.

Cost

Stage	Hours	Cost
Tokenizer and corpus preparation	0.65	$0.32
Phase 1	6.44	$3.08
Phase 2	6.43	$3.08
Phase 3	12.19	$5.83
Benchmarks, including public baselines	2.91	$1.39
Total	28.6	$13.70

One RTX 5090 (32 GB) rented on Vast.ai at $0.4785/hour.

Citation

@misc{llm05b_fineweb_edu_2026,
  title  = {llm-0.5b-fineweb-edu: a 489M-parameter model pretrained on one consumer GPU},
  author = {osjayaprakash},
  year   = {2026},
  url    = {https://huggingface.co/osjayaprakash/llm-0.5b-fineweb-edu}
}

License

MIT.

Downloads last month: 320

Safetensors

Model size

0.5B params

Tensor type

BF16

osjayaprakash
/

llm-0.5b-fineweb-edu