elroy-350m-base

elroy-350m-base is a 361M-parameter Llama-style decoder trained from scratch on about 20B tokens of educational Python and English by Strife Technologies, as a fully documented teaching project. Every line of the tokenizer, model, training loop and fine-tuning code is in the repository, with a chapter explaining it:

https://github.com/strifero/elroy

This is the base variant: a text continuer. Prompt it with the start of a Python file (a signature and docstring, a comment, an import). It also completes gaps with <|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>.

Use

No transformers needed. elroy_min.py in this repo holds the model, the tokenizer and a sampler in one file:

import torch
from elroy_min import load, generate

model, tok = load(".")          # this folder
ids = tok.encode("def is_prime(n):\n    \"\"\"Return True if n is prime.\"\"\"\n")
out = generate(model, ids, max_new_tokens=128, temperature=0.2, top_p=0.95, stop_ids={tok.eot})
print(tok.decode(out))

Training

Parameters	361M (24 layers, width 1024, 16 heads, context 2048)
Tokens	about 20B: 45% Stack-Edu Python, 20% StarCoderData Python, 25% FineWeb-Edu, 10% Cosmopedia-v2
Objective	next token, plus fill-in-the-middle on half of the code
Hardware	two NVIDIA RTX A4500 (20GB), about 6 days
Tokenizer	own byte-level BPE, 32,768 entries, code-aware pre-tokenization

Evaluation

Benchmark	pass@1
HumanEval	13.4%
HumanEval+	11.0%
MBPP (sanitized)	20.2%

Data and licenses

Stack-Edu (HuggingFaceTB, Python subset; files with a permissive or no detected license, as in The Stack v2), StarCoderData (bigcode, permissive), FineWeb-Edu (ODC-By), Cosmopedia-v2 (ODC-By). Fine-tuning data: Magicoder OSS-Instruct (MIT), evol-codealpaca-v1 (Apache-2.0), and a handwritten set in the repository.

Limitations

This is a small model. It writes short, mostly correct Python functions and explains beginner concepts. It makes confident mistakes, knows little outside Python, has no knowledge of recent libraries, and has not been trained to refuse anything. Run what it writes before you rely on it.