Moonfrost-777M-Instruct-v2

Code · Site · Training runs

A supervised fine-tune of Moonfrost-777M, not a separate model. It starts from that base and trains for 1.7 epochs on instruction data. Architecture, tokenizer and parameter count are identical; only the weights differ. Fine-tuning changed how the model answers, not what it knows, so the benchmark table below describes the family rather than this tune alone; the figures in it were measured on these weights, which is the checkpoint the column is named after.

Two tunes of that base are published and both are usable; they differ in purpose rather than in kind. This one is for conversation: it includes the persona set, so it answers as Moonfrost, keeps speaker roles straight and holds a thread. Instruct-v1 has no identity data at all, which makes it the neutral starting point if you would rather tune your own behaviour onto it.

The base was pretrained from nothing: its own byte-level tokenizer, its own attention and routing code, its own training loop. No existing model was adapted and no weights were borrowed. Pretraining and both fine-tunes together cost about $55 of rented H100 time across roughly twelve GPU-hours.

The architecture follows DeepSeek-V2 closely, because two ideas from that paper are what make a model this size behave larger than its compute budget. Multi-head Latent Attention compresses everything the model caches into one shared 320-number latent per token instead of a key and a value for each of fourteen heads, with position carried separately on a small decoupled rotary key, since a position-rotated key cannot be rebuilt from an unrotated latent. That takes the cache from 1,792 numbers per token down to 352. DeepSeekMoE then makes the feed-forward layers sparse: thirty-two routed experts sit in each layer, a router picks three, one shared expert always runs, and so a token touches four of thirty-three. Seventy per cent of the parameters are idle for any given token, which is why 777M total costs about as much to run as 161M active.

What that budget bought is behaviour, not knowledge. The model holds a conversation, tracks what you told it earlier in the same chat, writes short working code, and knows its own story. It does not know very much about the world, and the section on limitations below is not boilerplate.

Property	Value
Parameters	777,148,032 total, 161,036,224 active per token
Layers	14, of which layer 0 is dense and 1-13 are Mixture-of-Experts
Hidden size / heads	896 / 14
Experts	32 routed with top-3 routing, plus 1 shared expert
Attention	Multi-head Latent Attention, 320 KV latent + 32 decoupled rotary key
Context	1,024 tokens
Vocabulary	32,768, byte-level BPE trained from scratch on the same corpus
Pretraining	~6B tokens of FineWeb-Edu, val loss 2.976
Chat fine-tune	1.7 epochs, best at step 7,800 of 9,201, val loss 1.2484
Peak / min LR	pretraining 6e-4 / 6e-5, fine-tune 2e-4 / 2e-5
Batch	micro-batch 24, accumulation 3, 73,728 tokens per step
Precision	bf16 autocast with fp32 master weights
Throughput	179,000 training tokens/sec on one H100
Inference	16.8 tokens/sec, fp32, batch 1, on an RTX 3050
Compute and cost	1x H100, ~12 GPU-hours, about $55

Usage

The architecture is not part of transformers, so the repository ships its own modelling code and needs trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "whoashish115/Moonfrost-777M-Instruct-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
).eval()

prompt = "<|system|>You are Moonfrost, a helpful assistant.<|user|>Why is the sky blue?<|assistant|>"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=120, do_sample=True,
                        temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Conversations are formatted <|system|>…<|user|>…<|assistant|>…<|endoftext|>, and you must stop generation on both <|endoftext|> and <|user|>. That second stop token is not optional: a model trained this briefly will cheerfully carry on past its own answer and write your next message for you. Temperature around 0.7 with top-p 0.92 and a repetition penalty of about 1.15 is what the serving code uses; much hotter and the factual errors get more creative without the prose getting better.

Benchmarks

Every number here was measured on one machine with one harness, five-shot, 200 examples per benchmark and 250 for MMLU, scored by which answer option the model finds most likely. The reference models were run through that same harness on those same examples rather than quoted from their cards, because prompt wording and length normalisation move these scores by several points and a table that mixes harnesses is not a comparison. Qwen2.5-0.5B makes the point concretely: its published MMLU is 47.5, and it scores 34.4 here.

Benchmark	Chance	Moonfrost Base	Moonfrost Instruct v1	Moonfrost Instruct v2	SmolLM2-135M	SmolLM2-360M	Qwen2.5-0.5B
ARC-Easy	25.0	54.8	52.4	44.4	62.8	68.4	64.4
ARC-Challenge	25.0	25.2	24.4	24.4	27.6	37.2	34.8
HellaSwag	25.0	36.0	38.4	37.2	40.0	43.6	42.4
WinoGrande	50.0	51.2	53.2	54.0	54.0	56.0	56.8
BoolQ	50.0	62.4	61.2	58.8	62.0	63.6	65.2
MMLU	25.0	28.8	30.0	30.8	32.4	36.8	34.4

Moonfrost is last or near-last on most rows, and the reason is not the architecture.

The tokens-per-parameter chart is on the base model card, where the pretraining budget it describes belongs.

Six billion tokens for 777 million parameters is about eight tokens per parameter. Chinchilla put the compute-optimal ratio near twenty, and everything released since has gone far past even that, because text is cheap compared with serving a bigger model forever. SmolLM2-360M read roughly two trillion tokens; Qwen2.5-0.5B read eighteen trillion. Those are three hundred and three thousand times more text than this model has seen, from models half its size. Nothing in the design closes a gap like that, and no amount of further chat fine-tuning will either, because fine-tuning teaches format and not facts.

Read the three Moonfrost columns down each row. Almost every difference between them is noise: at 250 examples the 95% interval on a single score is roughly ±6 points, and fifteen of the eighteen gaps are under three. One benchmark moves, and it moves in one direction. ARC-Easy falls 54.8, 52.4, 44.4 across the base, the half-epoch tune and the 1.7-epoch tune. Ten points is what it costs to teach the model to answer in a chat format instead of continuing a multiple-choice stem, and the cost grows with how long you tune.

A caveat that applies to the whole table, and that this project ran into directly. The harness draws its five in-context examples from the rows immediately after the evaluation slice, so changing the number of examples also changes the prompt. An earlier run of these same weights over 200 examples put BoolQ at 44.5; the run over 250 puts it at 58.8. Same model, same harness, same questions for the first two hundred of them, 14.3 points apart. Treat any single number here as a measurement with a wide interval, and treat scores copied from a different harness as not comparable at all.

Training

The pretraining loss curves are on the base model card.

Pretraining ran in two phases on two separate machines, reading disjoint shards of FineWeb-Edu sample/10BT so no document was seen twice. Splitting a single annealing schedule across two machines works because the learning rate is parameterised by elapsed fraction of training rather than by step: phase 2 started at fraction 0.5227, exactly where phase 1 stopped, and the cosine curve continued instead of restarting. Only the weights crossed the boundary, which is why phase 2 re-warms for 150 steps.

The chat fine-tune mixes smol-smoltalk at 84% of rows with smoltalk's everyday-conversations at 12%, repeated twenty-five times, and the Moonfrost-Persona-SFT identity set at 3.3%. The repeats exist because small talk is a rounding error in smol-smoltalk and without them the model answered "hi" with a lecture on quantum physics. Conversations are packed several to a 1,024-token row rather than padded, which took useful tokens per batch from roughly 40% to 61%, and loss is masked on system and user turns so only assistant text is ever supervised.

A note on the charts: both pretraining logs are partial, and the panels say where. Phase 1 is two fragments, because the attempt that dropped its connection at step 1,730 logged locally and the run that finished never had its log downloaded; what sits to the right of the gap was read back out of retained console output. Phase 2 logged its first 105 minutes of 340. The step counts and losses in the table above come from checkpoint metadata, which is complete, as is the chat fine-tune log. All three runs are on Weights & Biases.

Versions

Two chat tunes were run on the same pretrained base and both are published. They differ in exposure and in whether they carry identity data, and the behavioural gap between them is the clearest thing in this project about what supervised fine-tuning actually buys. Asked who made it, v1 answers that it is a retired professor of English named Jack Harris and then asks the question back; v2 answers in one line and stops. Asked for hello world in Java, v1 emits a class with two main methods that will not compile; v2 emits the correct four lines and stops cleanly.

Version	Val loss	SFT epochs	Identity data	Behaviour
v1	1.3251	0.42	none	confuses speaker roles, invents an identity
v2 (this model)	1.2484	1.7	3.3% of rows	stable roles, stable identity, usable code

Intended use

Use these weights for short conversational turns, for questions with definitional answers, and for reading what a 777M Mixture-of-Experts model trained on six billion tokens can and cannot do. It follows a chat template, answers as Moonfrost, and stops cleanly on <|endoftext|> and <|user|>.

To build on it, start from Moonfrost-777M rather than from here: tuning a tune compounds the format of the first pass. If you want an instruction-following starting point with no identity baked in, Instruct-v1 is that.

Do not put it in front of users, in a product, or anywhere an answer is acted on, and do not use it for anything touching medicine, law, finance or safety. It has no safety tuning and no content filtering, and its confident wrong answers read exactly like its correct ones.

Limits

It invents facts with complete confidence, and it does so most often on exactly the topics that a corpus of educational web text does not cover. Ask it about cryptocurrency and it will produce fluent paragraphs about binomial coefficients. It cannot do arithmetic or multi-step reasoning, has no tools and no internet, keeps no memory between conversations, handles English only, and sees at most 1,024 tokens at a time. Open-ended "how do I" questions tend to run to the token limit without ever concluding, while short definitional questions stop cleanly.

There is no safety tuning, no RLHF, and no content filtering at either the data or the output stage. Treat nothing it says as factual and do not put it anywhere a wrong answer costs something.

Citation

@misc{moonfrost2026,
  title  = {Moonfrost: a 777M-parameter Mixture-of-Experts language model trained from scratch},
  author = {Ashish Kumar},
  year   = {2026},
  url    = {https://huggingface.co/whoashish115/Moonfrost-777M}
}

Apache 2.0.