samueljayasingh/rune-0.3b-base

Rune-R1 Base (351M)

A ~351M parameter decoder-only transformer, pretrained from scratch on general web text. This is stage 1 of the Rune-R1 pipeline (Pretrain → SFT → GRPO) and serves as the foundation for Rune-R1-SFT and Rune-R1 (the final GRPO reasoning model). On its own, it is a general-purpose next-token predictor with no instruction-following or chat behavior.

Model Description

• Developed by: samueljayasingh
• Model type: Causal language model (text-only), trained from scratch (not a fine-tune of an existing base model)
• Architecture: Decoder-only Transformer — RoPE, RMSNorm (with QK-norm), SwiGLU feed-forward, Grouped-Query Attention
• Parameters: ~351M
• Training framework: PyTorch (custom training loop, scripts/train.py), single AMD MI300X GPU
• Dataset: HuggingFaceFW/fineweb-edu (sample-10BT config), streamed
• Language: English
• Tokenizer: GPT-2 (tiktoken)
• License: Apache 2.0

Architecture details

Parameter	Value
Layers	22
Embedding dimension	1024
Attention heads / KV groups	16 / 4 (GQA)
Feed-forward hidden dim	2816 (SwiGLU)
Context length	1024 tokens
Position embeddings	RoPE (base 10,000)
Normalization	RMSNorm, with QK normalization
Vocab size	50,257 (GPT-2 / tiktoken)

Intended Uses & Limitations

Intended use:

• Research and educational reference for training small language models from scratch.
• Base checkpoint for further fine-tuning (SFT, RLHF/RLVR) — see the SFT and GRPO variants in this same family for a math-reasoning-tuned example.
• Studying pretraining dynamics (loss curves, scaling behavior) at small parameter counts.

Limitations:

• Not instruction-tuned — it will continue text rather than follow chat-style prompts or answer questions directly.
• Trained on only ~5B tokens, far below the data budget of production-grade LLMs; expect weaker factuality, coherence over long spans, and world knowledge than larger, more heavily trained models.
• Context length is limited to 1024 tokens.
• No safety/RLHF alignment has been applied at this stage — treat outputs like any unaligned base LM (do not deploy directly in user-facing applications).

How to Use

import torch
import tiktoken
from rune.model import CONFIG_350M, RuneModel

ckpt = torch.load("pytorch_model.bin", map_location="cpu")
model = RuneModel(CONFIG_350M)
model.load_state_dict(ckpt)
model.eval()

enc = tiktoken.get_encoding("gpt2")
prompt = "The key to machine learning is"
tokens = torch.tensor([enc.encode(prompt)], dtype=torch.long)

# See rune/generate.py in the source repo for full sampling / KV-cache generation code.

The rune package (model definition + generation utilities) is available at the Rune-R1 GitHub repository.

Training & Evaluation

Training procedure

Parameter	Value
Dataset	FineWeb-Edu, sample-10BT config, streamed
Objective	Next-token prediction (causal LM), cross-entropy
Tokens seen	~5.05B
Batch size (per step)	8 sequences
Block size	1024 tokens
Gradient accumulation	64 steps (effective ~524k tokens/optimizer step)
Learning rate	3e-4, cosine schedule
Warmup steps	2,000
Precision	fp32 master weights, bf16 autocast compute
Hardware	1x AMD MI300X GPU
Optimizer steps completed	9,624

Evaluation results

Metric	Value
Final training loss (cross-entropy)	~2.999
Min. observed training loss	2.68
Avg. training loss, last 300 steps	~3.12

Loss is the streaming training-set cross-entropy (no held-out pretraining validation split was used); downstream task performance is evaluated after the SFT and GRPO stages instead — see those model cards for MATH-500 results.

Citation

@misc{RuneR1Base2026,
  author = {Samuel Jayasingh},
  title = {Rune-R1 Base: A 351M Transformer Pretrained from Scratch on FineWeb-Edu},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/samueljayasingh/Rune-R1-base}}
}

Acknowledgements

• HuggingFaceFW/fineweb-edu — pretraining corpus.
• Architecture adapted from rasbt/LLMs-from-scratch (Apache 2.0), trimmed to a dense ~350M config with a GPT-2 vocabulary.