Moonfrost-777M

Code · Site · Training runs

Moonfrost is a 777M-parameter Mixture-of-Experts language model written and trained from nothing: its own byte-level tokenizer, its own attention and routing code, its own training loop. No base model was adapted and no weights were borrowed. Pretraining took about ten GPU-hours on a rented H100 and read roughly six billion tokens of FineWeb-Edu.

This repository holds the base model: next-token prediction and nothing else. It has no chat template, no instruction tuning and no alignment, so it continues text rather than answering questions. Prompt it with the opening of a passage, not an instruction.

Two supervised fine-tunes start from these weights and differ from them only in the weights, sharing this architecture, tokenizer and parameter count: Instruct-v2, the one to use, and Instruct-v1, kept for comparison. For conversation, take v2.

Property	Value
Parameters	777,148,032 total, 161,036,224 active per token
Layers	14, of which layer 0 is dense and 1-13 are Mixture-of-Experts
Hidden size / heads	896 / 14
Experts	32 routed with top-3 routing, plus 1 shared expert
Attention	Multi-head Latent Attention, 320 KV latent + 32 decoupled rotary key
Context	1,024 tokens
Vocabulary	32,768, byte-level BPE trained from scratch on the same corpus
Training data	FineWeb-Edu sample/10BT, shards 0-7, ~6B tokens
Validation loss	2.976, best at step 11,000 of phase 2
Held-out perplexity	51.64 on an unseen shard, loss 3.944
Peak / min LR	6e-4 / 6e-5, time-based cosine, continuous across both phases
Batch	micro-batch 24, accumulation 12, 294,912 tokens per step
Precision	bf16 autocast with fp32 master weights, gradients clipped at 1.0
Throughput	179,000 training tokens/sec on one H100
Compute and cost	1x H100, ~10 GPU-hours, part of a ~$55 total

Usage

The architecture is not part of transformers, so the repository ships its own modelling code and needs trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "whoashish115/Moonfrost-777M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
).eval()

inputs = tokenizer("The water cycle begins when", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=True,
                        temperature=0.8, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Architecture

Two ideas from DeepSeek-V2 do the work, and both trade stored parameters for cheap ones.

Multi-head Latent Attention replaces the usual per-head key and value cache with a single shared latent. Ordinary attention at fourteen heads and head dimension 64 caches 1,792 numbers for every token; this projects the input down to one 320-number latent, caches that, and reconstructs keys and values when it needs them. Position is the complication, because rotary embeddings rotate a key by where it sits and a rotated key cannot be rebuilt from an unrotated latent. So position travels separately, on a 32-number rotary key shared across all heads, and attention runs over the two concatenated. The cache holds 352 numbers per token instead of 1,792, roughly five times smaller. At inference the up-projection matrices fold into the query and output projections once, which is exact because both are linear, and after that the model never reconstructs keys and values at all.

DeepSeekMoE makes the feed-forward layers sparse. Every layer above layer 0 holds 32 routed experts and one shared expert; a router scores the token against all 32, the top 3 run, the shared one always runs, so a token passes through four of thirty-three. Layer 0 is a plain SwiGLU feed-forward because routing from the very first layer destabilised early training and one dense layer costs almost nothing. A load-balancing auxiliary loss at weight 0.01 keeps the router from collapsing onto a few favourites.

The implementation detail that mattered most was mechanical rather than mathematical. The experts are stored as three stacked (32, hidden, ffn) tensors and dispatched by capacity in the GShard style, so all 32 run as three batched matrix multiplies. The first version looped over experts in Python, which forced a GPU synchronisation thirty-two times per layer per step and ran at 6,056 tokens per second. Same mathematics, same results, but the stacked version runs at 179,000.

Beyond those: RMSNorm with no bias terms anywhere, SwiGLU activations, RoPE at theta 10,000 computed in float32 and cast back because bfloat16 lost enough precision at long positions to matter, and tied input and output embeddings sharing one 32,768 x 896 matrix, which saves 29M parameters.

Component	Parameters	Share
Routed experts (13 layers x 32)	543M	70%
Attention (14 layers)	118M	15%
Shared experts + dense layer	84M	11%
Embedding (tied)	29M	4%

Seventy per cent of the model is experts that stay idle for any given token. That is the whole trade: the capacity of a 777M model at roughly the compute of a 161M one.

Training

Pretraining ran in two phases on two separate machines, reading disjoint shards so no document was seen twice. Phase 1 took shards 0-2 over 276 minutes for about 2.9B tokens; phase 2 took shards 3-7 over 340 minutes for about 3.1B and ended at validation loss 2.976.

Splitting one annealing schedule across two machines works because the learning rate is parameterised by elapsed fraction of training rather than by step. Phase 2 started at fraction 0.5227, exactly where phase 1 stopped, so the cosine curve continued rather than restarting. Only the weights crossed the boundary; the optimizer state stayed behind, which is why phase 2 re-warms for 150 steps.

The charts show what was logged, which is less than what was run. The first pretraining attempt died at step 1,730 when its client connection dropped, and only the first 105 minutes of phase 2 were retrieved from the volume, so those panels are partial by construction and labelled as such. The losses and step counts in the table above come from checkpoint metadata, which is complete. The full step-by-step history for all three runs is on Weights & Biases.

Benchmarks

Every number was measured on one machine with one harness, five-shot, 200 examples per benchmark, scored by which answer option the model finds most likely. The reference models were run through that same harness on those same examples rather than quoted from their cards, because prompt wording and length normalisation move these scores by several points. Qwen2.5-0.5B shows why that matters: its published MMLU is 47.5, and it scores 34.4 here.

Benchmark	Chance	Moonfrost Base	Moonfrost Instruct v1	Moonfrost Instruct v2	SmolLM2-135M	SmolLM2-360M	Qwen2.5-0.5B
ARC-Easy	25.0	54.8	52.4	44.4	62.8	68.4	64.4
ARC-Challenge	25.0	25.2	24.4	24.4	27.6	37.2	34.8
HellaSwag	25.0	36.0	38.4	37.2	40.0	43.6	42.4
WinoGrande	50.0	51.2	53.2	54.0	54.0	56.0	56.8
BoolQ	50.0	62.4	61.2	58.8	62.0	63.6	65.2
MMLU	25.0	28.8	30.0	30.8	32.4	36.8	34.4

Six billion tokens for 777 million parameters is about eight tokens per parameter against Chinchilla's compute-optimal twenty, and the models in that table read two to eighteen trillion at half the size. That ratio, not the architecture, explains almost everything the model gets wrong.

Read the three Moonfrost columns down each row. Almost every difference between them is noise: at 250 examples the 95% interval on a single score is roughly ±6 points, and fifteen of the eighteen gaps are under three. One benchmark moves, and it moves in one direction. ARC-Easy falls 54.8, 52.4, 44.4 across the base, the half-epoch tune and the 1.7-epoch tune. Ten points is what it costs to teach the model to answer in a chat format instead of continuing a multiple-choice stem, and the cost grows with how long you tune.

These weights are the leftmost of the three, and on ARC-Easy they are the best Moonfrost column. Nothing in the table is below chance.

Intended use

Use these weights to continue text, to measure what six billion tokens buys at this size, or as the starting point for a fine-tune of your own. They are the parent of both published tunes and carry no chat template, so a prompt shaped like an instruction gets a continuation of the instruction rather than an answer.

Two things are worth building on top. Instruction tuning is the obvious one, and the two published tunes show what roughly half an epoch and roughly two epochs each produce; the scripts that ran them are in the repository. Domain adaptation is the other, since a model that has read only educational web text has room to move on any corpus you can supply.

Do not put it in front of users, in a product, or anywhere an answer is acted on. It has no safety tuning and no content filtering, and at this scale a confident wrong answer is the common case rather than the edge case. It is a research artefact.

Limits

It invents facts with complete confidence, most often on the topics a corpus of educational web text does not cover. It cannot do arithmetic or multi-step reasoning, handles English only, and sees at most 1,024 tokens at a time. There is no safety tuning, no RLHF and no content filtering at either the data or the output stage. This is a working demonstration of a complete training pipeline at small scale, not a deployable model.

References

The implementation was written from scratch against these papers rather than adapted from released code.

Paper	What it contributes
DeepSeek-V2	Multi-head Latent Attention, DeepSeekMoE
DeepSeek-V3	routing and load-balancing refinements
Attention Is All You Need	the transformer
RoFormer	rotary position embeddings
GLU Variants	SwiGLU
RMSNorm	normalisation without mean subtraction
GShard	capacity-based expert dispatch
Switch Transformer	load-balancing auxiliary loss
Chinchilla	the twenty-tokens-per-parameter ratio
FlashAttention	the fused kernel used through SDPA
BPE for NMT	byte-pair encoding

Citation

@misc{moonfrost2026,
  title  = {Moonfrost: a 777M-parameter Mixture-of-Experts language model trained from scratch},
  author = {Ashish Kumar},
  year   = {2026},
  url    = {https://huggingface.co/whoashish115/Moonfrost-777M}
}

Apache 2.0.