CreekwardGoat-500K

CreekwardGoat-500K is a compact, decoder-only transformer developed to explore the capabilities of tiny language models. Despite its remarkably small scale of around 500 thousand parameters, it achieves very competitive performance within its size class on established evaluation benchmarks, including BLiMP and ARC-Easy.

Model Details

Attribute	Value
Parameters	499,616
Precision	BF16
Vocabulary size	2,048
Maximum context length	256 tokens
Hidden / embedding width	104
Transformer layers	4
Query attention heads	4
Key-value heads	1
Attention type	Grouped-Query Attention (GQA)
Attention head dimension	26
Feed-forward intermediate width	140
Feed-forward activation	SwiGLU
Position encoding	Rotary Position Embedding (RoPE)
RoPE base theta	10,000
Core normalization	RMSNorm
Residual normalization layout	Peri-LN

Benchmarks

All results were evaluated using the lm-evaluation-harness.

Benchmark	Metric	Score	Baseline (random guessing)
WikiText	Byte perplexity ↓	4.1228	-
BLiMP	Accuracy	60.61%	50%
ARC-Easy	Accuracy (normalized)	30.35%	25%

Usage

Inference

Run:

python inference.py

Training / Reproduction

This model is fully transparent in training and reproducible from scratch.

Download the training data:

python get_training_data.py

Start training:

python train.py

Training configurations including seed and number of training tokens can be modified in config.py.

If you leave the files unchanged, theoretically, this should reproduce the training trajectory as well as the final model output.

Training Data

The model was trained on a 300 million token subset of FineWeb-Edu-Dedup, taken from the HuggingFaceTB/smollm-corpus repository. The specific shard was selected due to its relatively small size to reduce storage and preprocessing overhead.

Source repository: HuggingFaceTB/smollm-corpus
Dataset license: Open Data Commons Attribution License (ODC-By) 1.0

Limitations & Disclaimer

This model is an experimental personal research artifact and is not intended for production or consumer use. It is released as a base (pre-trained) model and does not support conversational interaction. Due to its extremely small scale (≈500K parameters), this model is fundamentally incapable of reliable output. It may generate:

Factually inaccurate or hallucinated content
Grammatically garbled or nonsensical text
Unexpected or biased outputs

Generated outputs do not represent the author's views. Users are solely responsible for verifying outputs, ensuring compliance with all applicable laws, and bearing all consequences of use or redistribution. The author accepts no responsibility for model outputs or downstream use.