TinyTurk v2.7d MID - English

20M-parameter English GPT. Reaches 97% of GPT-2 small's BLiMP score with 6.2x fewer parameters.

Model Description

TinyTurk MID is the middle-scale model in the TinyTurk series. It is trained from scratch on a 22 GB mixed English corpus and evaluated against GPT-2 small.

Developed by: stunmuffin
Model type: Decoder-only Transformer (GPT-style)
Language: English
License: Apache 2.0

Architecture

Tokenizer: 16,384 BPE (SentencePiece, English)
Context length: 256
Embedding: 384
Layers: 6
Heads: 6 (head_dim=64)
FFN: 384 -> 1536 -> 384
Micro-FFN: 384 -> 768 -> 384 (gated, gate=0.75)
Position encoding: RoPE
Normalization: RMSNorm (pre-norm)
Weight tying: yes
Parameters: 20,482,176

Training

Data

Source	Size
FineWeb-Edu	8.6 GB
Wikipedia (20231101.en)	6.0 GB
Cosmopedia	4.3 GB
RedPajama-V2	2.0 GB
TinyStories	1.9 GB
Total	~22 GB (6.14B tokens)

A fresh 16K BPE tokenizer was trained on this corpus, then the text was chunked into 256-token pieces.

Optimization

Training samples used: 1,000,000 chunks (256M tokens, 4.6% of available data)
Batch size: 32
LR: 3e-4 cosine, 2% warmup
Epochs: 1
Hardware: GTX 1080 Ti
Time: ~2.2 hours
Best val loss: 3.4537
Val perplexity: 31.62

Evaluation

BLiMP (full test, 67,000 items)

Model	Parameters	BLiMP
GPT-2 small	124M	~0.70
TinyTurk MID	20M	0.6776
Chance	-	0.50

97% of GPT-2 small's BLiMP score with 16% of the parameters.

HellaSwag (200 items)

Model	HellaSwag
GPT-2 small	~0.31
TinyTurk MID	0.34
Chance	0.25

ARC-Easy (100 items)

Model	ARC-Easy
GPT-2 small	~0.39
TinyTurk MID	0.31
Chance	0.25

Language modeling

Val loss: 3.4537
Perplexity: 31.62

Usage

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_en_bpe_16k.model')

config = TinyTurkV27Config(vocab_size=16384, context_length=256,
    d_model=384, n_heads=6, n_layers=6, d_ff=1536)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

prompt = 'The capital of France is'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(100):
        logits, _ = model(x[:, -256:])
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Known Limitations

Generation quality is poor. Despite strong benchmark scores, the model does not produce coherent text on short prompts. This is because the mixed training corpus does not have a clear register.
Trained on only 4.6% of available data.
Context length 256 tokens (vs GPT-2's 1024).
Vocab 16,384 (vs GPT-2's 50,257).
Below GPT-2 small on ARC-Easy (knowledge-heavy).

Scientific Context

This model tests whether modern architectural choices (RoPE, RMSNorm, micro-FFN, weight tying) plus Chinchilla-optimal training can close the gap to GPT-2 small at a fraction of the size.

Model	Params	BLiMP	HellaSwag
GPT-2 small	124M	0.70	0.31
TinyTurk MID	20M	0.68	0.34

Result: At the 20M parameter scale, modern architectures + efficient training achieve near-GPT-2-small grammar performance with 16% of the parameters. However, generation quality is much worse - benchmarks do not tell the whole story.

Files

File	Content
`pytorch_model.bin`	Model weights
`config.json`	Config
`tokenizer_en_bpe_16k.model`	Tokenizer
`model.py`	Model definition

Citation

@misc{tinyturk-v27d-mid-en,
  title = {TinyTurk v2.7d MID - English},
  author = {stunmuffin},
  year = {2026},
  howpublished = {https://huggingface.co/stunmuffin/tinyturk-v2.7d-mid-en}
}

Turkish agent: stunmuffin/tinyturk-v2.7d-agent
English agent: stunmuffin/tinyturk-v2.7d-agent-en
English TinyStories: stunmuffin/tinyturk-v2.7d-tinystories-en
Turkish baseline: stunmuffin/tinyturk-v2.7d-k-hikaye-baseline

Downloads last month: 23

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

stunmuffin
/

tinyturk-v2.7d-mid-en