Moonfrost-777M-Instruct-v1

Code · Site · Training runs

A supervised fine-tune of Moonfrost-777M, not a separate model. It starts from that base and trains for 0.42 epochs on instruction data. Architecture, tokenizer and parameter count are identical; only the weights differ.

Two tunes of that base are published and both are usable; they differ in purpose rather than in kind. This one carries no identity data, so it claims no persona and holds no opinion about who built it, which makes it the neutral starting point for tuning your own behaviour on. Instruct-v2 is the one tuned for conversation: four times the exposure, plus the persona set.

Read on before building from it. Half an epoch is not much instruction tuning and it shows, in two ways. It drifts between speakers: it had not seen enough complete conversations to learn reliably where an assistant turn ends, so it will sometimes write your next message for you, which is why both <|endoftext|> and <|user|> have to be treated as stop tokens. And it has no identity, because the persona data was only added in v2. Asked who made it, this checkpoint answers that it is a retired professor of English named Jack Harris who has taught in the Midwest for about ten years, and then asks the question back. v2 answers in one line and stops. Asked for hello world in Java, this one produces a class containing two main methods, which will not compile, alongside an explanation about abstract base classes that has nothing to do with the code; v2 produces the correct four lines and stops cleanly.

Neither failure is a defect of the architecture or of the pretrained base. Both are what an undertrained supervised fine-tune looks like.

Property	Value
Parameters	777,148,032 total, 161,036,224 active per token
Layers	14, of which layer 0 is dense and 1-13 are Mixture-of-Experts
Hidden size / heads	896 / 14
Experts	32 routed with top-3 routing, plus 1 shared expert
Attention	Multi-head Latent Attention, 320 KV latent + 32 decoupled rotary key
Context	1,024 tokens
Vocabulary	32,768, byte-level BPE trained from scratch
Chat fine-tune	0.42 epochs, best at step 1,805, val loss 1.3251
Identity data	none
Peak / min LR	2e-4 / 2e-5, time-based cosine, 100 warmup steps
Batch	micro-batch 24, accumulation 3, 73,728 tokens per step
Precision	bf16 autocast with fp32 master weights
Data mixture	smol-smoltalk 88% of rows, smoltalk everyday-conversations x25 at 12%

Architecture is documented in full on Moonfrost-777M.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "whoashish115/Moonfrost-777M-Instruct-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
).eval()

prompt = "<|system|>You are Moonfrost, a helpful assistant.<|user|>Why is the sky blue?<|assistant|>"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=120, do_sample=True,
                        temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Conversations are formatted <|system|>…<|user|>…<|assistant|>…<|endoftext|>, and stopping on <|user|> as well as <|endoftext|> matters more here than it does for v2.

Benchmarks

All three Moonfrost checkpoints were scored, so the two tunes can be read against the base they came from and against each other. Five-shot, 250 examples per benchmark, likelihood scored, with the reference models run through that same harness on those same examples rather than quoted from their cards.

Benchmark	Chance	Moonfrost Base	Moonfrost Instruct v1	Moonfrost Instruct v2	SmolLM2-135M	SmolLM2-360M	Qwen2.5-0.5B
ARC-Easy	25.0	54.8	52.4	44.4	62.8	68.4	64.4
ARC-Challenge	25.0	25.2	24.4	24.4	27.6	37.2	34.8
HellaSwag	25.0	36.0	38.4	37.2	40.0	43.6	42.4
WinoGrande	50.0	51.2	53.2	54.0	54.0	56.0	56.8
BoolQ	50.0	62.4	61.2	58.8	62.0	63.6	65.2
MMLU	25.0	28.8	30.0	30.8	32.4	36.8	34.4

The scores sit where they do because six billion tokens for 777 million parameters is about eight tokens per parameter, against a compute-optimal ratio of roughly twenty and against reference models that read two to eighteen trillion.

Intended use

Use these weights as an instruction-following starting point that holds no opinion about who it is. Nothing in the fine-tune mixture describes an identity, so a persona tuned on top of it has nothing to argue with. That is the one thing this checkpoint does better than v2.

Read the section below first. Half an epoch of instruction tuning leaves speaker roles unstable, which is a defect if you deploy it and mostly irrelevant if you tune over it.

Do not put it in front of users or anywhere an answer is acted on. It has no safety tuning and no content filtering, and it drifts between speakers often enough to produce dialogue nobody wrote.

Limits

Everything true of v2, and more of it. It fabricates facts confidently, performs no arithmetic or multi-step reasoning, handles English only, sees at most 1,024 tokens, and has no safety tuning, no RLHF and no content filtering of any kind. On top of that it drifts between speakers and has no self-description.

Treat it as something to tune from rather than something to deploy. For a conversation, take Instruct-v2.

Citation

@misc{moonfrost2026,
  title  = {Moonfrost: a 777M-parameter Mixture-of-Experts language model trained from scratch},
  author = {Ashish Kumar},
  year   = {2026},
  url    = {https://huggingface.co/whoashish115/Moonfrost-777M}
}

Apache 2.0.