PebbleGPT-320M

A 320M-parameter dense transformer pretrained from scratch on 10B tokens. Base model — it continues text, it does not follow instructions.

Architecture

• 24 layers, hidden size 1024, intermediate 2816
• Grouped Query Attention: 16 query heads, 4 KV heads (ratio 4)
• RoPE (theta 10000), SwiGLU, RMSNorm, tied embeddings
• 2048 context, SmolLM2 tokenizer (49,152 vocab)

Training data (10B tokens)

Source	Share
FineWeb-Edu	42.5%
DCLM-Edu (edu_int_score >= 3)	42.5%
Python-Edu	10%
FineMath-4+	5%

Training

AdamW (beta 0.9/0.95, weight decay 0.1 excluding embeddings), WSD schedule with peak LR 5e-4 and 10% decay, 2000 warmup steps. 19,073 steps at 524,288 tokens per step. ~18 hours on a single H100 SXM at ~35% MFU.

Final training loss: 2.507 (perplexity ~12.3)

Results

Benchmark	Score
HellaSwag (acc_norm)	30.5
PIQA	55.5
ARC-easy (acc_norm)	37.7

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "strectelite/PebbleGPT-320M", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("strectelite/PebbleGPT-320M")

Note: no KV cache implemented yet, so generation is slow.