xxparthparekhxx/pluto-124m-chat

Pluto 124M Chat

Pluto is a small chat model with 121M parameters. xxparthparekhxx trained it from scratch: pretraining on 4.35B tokens of FineWeb-Edu, then chat fine-tuning in two stages on one RTX 5060 Laptop GPU (8 GB). The pretrained model without chat tuning is pluto-124m-base.

It is named after the dwarf planet, since it is a small model. It can answer simple questions, rewrite and summarize text, hold a short multi-turn conversation, and follow formatting instructions such as "use exactly three bullet points". At this size it often gets facts wrong, so check anything it tells you.

Try it in your browser: Pluto Chat Space. The model runs on your device (WebGPU or WebAssembly) from the fp16 ONNX export in onnx/, which has an explicit KV cache; no server does any inference.

This is version 2. Version 1 (1B pretraining tokens, no instruction-following or multi-turn set) is the v1 revision of this repo (revision="v1" in from_pretrained).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "xxparthparekhxx/pluto-124m-chat"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "Give me three tips for staying focused while studying."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

trust_remote_code=True is needed for both the tokenizer and the model. The architecture (gated attention with QK-norm and a logit soft-cap) is not a stock transformers class; its code is modeling_pluto.py in this repo. Greedy decoding (do_sample=False) gives the most reliable answers. The default generation config samples with temperature 0.8 and top-k 50, and stops at 256 new tokens.

The chat format is:

<|endoftext|><|user|>Who are you?<|end|><|assistant|>I'm Pluto, ...<|end|>

A <|system|> turn is also supported.

Evaluation

Instruction following: IFEval

IFEval has 541 prompts with formatting rules that can be checked automatically, such as "at least 300 words", "no commas" or "wrap the answer in JSON". Scored with lm-evaluation-harness 0.4.13 through each model's chat template, greedy, up to 512 new tokens. None of Pluto's training prompts are IFEval prompts.

Model	Prompts with every rule kept (strict)	Rules kept (strict)	Prompts (loose)	Rules (loose)
Pluto chat v2	24.2	36.8	26.1	39.0
Pluto chat v1	19.4	30.8	20.1	32.3
SmolLM2-135M-Instruct	21.3	34.4	23.7	37.3

The standard error on the prompt-level scores is about ±1.8 points. SmolLM2-135M was pretrained on about 2 trillion tokens, roughly 460 times as many as Pluto.

The gain over v1 comes from the instruction-following set described under Training. The same stronger base model fine-tuned without that set scored 17.4 / 28.3.

Judged head-to-head

250 held-out prompts from the distilled set, answered greedily by each model. Gemini 3.5 Flash-Lite or Gemini 3.1 Flash-Lite compares each pair twice, once in each answer order; a model wins a prompt only if both orders agree, otherwise it is a tie.

Pluto chat v2 against	Wins	Ties	Losses	Score (95% CI)
the same model fine-tuned without the instruction-following and multi-turn sets	46	139	65	0.46 (0.42–0.50)

On ordinary chat prompts the instruction-following data costs little: the score is within noise of even. A comparison with SmolLM2-135M-Instruct will be added here.

Benchmarks compared with GPT-2 124M

All four models were scored with the same code: lm-evaluation-harness 0.4.13, zero-shot, fp32. GPT-2 is openai-community/gpt2.

Task (metric)	Pluto chat v2	Pluto base v2	GPT-2 124M	Pluto chat v1
HellaSwag (acc_norm)	34.2	33.9	31.1	31.1
ARC-Easy (acc_norm)	44.6	47.6	39.5	41.9
ARC-Challenge (acc_norm)	26.5	27.0	22.7	26.7
OpenBookQA (acc_norm)	30.6	30.4	27.2	27.2
PIQA (acc_norm)	62.4	63.4	62.5	59.1
Winogrande (acc)	51.9	52.3	51.6	51.7
LAMBADA (acc)	20.6	28.1	32.6	16.8

The standard errors are about ±0.5 points on HellaSwag and LAMBADA, ±1 to 1.5 on ARC, PIQA and Winogrande, and ±2 on OpenBookQA. Chat tuning leaves most scores where the base model had them, except LAMBADA: completing the last word of a novel passage is far from answering a question.

Held-out fine-tuning data

• Formatting rules kept: 77 held-out prompts from smol-smoltalk's constraint subset, every rule checkable. An answer counts only if it keeps every rule.
• Identity: 8 questions about the model, none of which appear in training.
• Validation loss: on held-out conversations from each fine-tuning set, counting answer tokens only.

	Rules kept	Says it is Pluto, by xxparthparekhxx	Distilled	smol-smoltalk	Instruction following	Multi-turn
Pluto chat v1	46.8%	8/8	1.951	1.648	–	–
Pluto chat v2	51.9%	8/8	1.752	1.551	2.206	2.190

Some greedy answers, including wrong ones:

Prompt	Pluto chat v2
Who made you?	I'm Pluto, and I was created by xxparthparekhxx, not by a big AI company.
Rewrite this sentence to sound more formal: we gotta finish the report by friday.	We must submit the report by Friday.
Can you check the weather in Mumbai for me right now?	I do not have access to your real-time location or a live weather app to check the weather in Mumbai. To check the weather in Mumbai, you will need to use a weather app or website.
Explain why the sky is blue in two sentences.	Blue is a color that is visible to the human eye because it is a deep, warm blue that filters the light from the Earth's atmosphere. (wrong, and one sentence)
Write a short, friendly message inviting a coworker to lunch.	Hey! I'm so glad you're here! I'm so glad you're here! (misses the request)

All of these results are in eval/: eval.json (every held-out answer), ifeval.json, judge_vs_round1.json, lm_eval_vs_gpt2.json and fineweb_edu_val_loss.txt.

Training

Fine-tuning computes the loss on assistant tokens only.

1. Pretraining: this is pluto-124m-base v2.

• Data: 4.35B tokens from FineWeb-Edu sample-10BT, tokenized with GPT-2's BPE.
• Steps: 8,297 steps of 524,288 tokens (512 sequences of 1,024), in bf16 with torch.compile.
• Optimizer: Muon for the 2-D weight matrices and AdamW for the embeddings and norms. Muon uses the DeepSeek-V4 variant: Newton–Schulz orthogonalization with the update rescaled to RMS 0.18, so both optimizers share one learning rate.
• Learning rate: peak 3e-3, weight decay 0.1. 200 warmup steps, then flat, then a linear decay to 10% of the peak over the last 25% of steps (warmup-stable-decay).
• Compute: about 7.6 hours on one RTX 5060 Laptop GPU for the first 1,430 steps, then about 3.9 hours on one RTX PRO 6000 for the rest.

2. Fine-tuning on smol-smoltalk:

• The data is HuggingFaceTB/smol-smoltalk, cut to the turns that fit in 1,024 tokens: 437,234 conversations.
• Two kinds of conversation were removed: 14,179 whose first prompt is also in the distilled set, and one where the assistant says it is SmolLM.
• One epoch: 6,832 steps of 64 conversations.
• Peak learning rate 3e-4, 30 warmup steps, linear decay to zero. About 6 hours.

3. Fine-tuning on distilled answers: four sets, all with answers written by larger open models.

• Distilled set: 24,283 conversations. The prompts come from seven smol-smoltalk subsets; the answers were written by Gemma 4 26B-A4B-it (18,864) and dots.3-note-preview (5,419). Answers that broke a prompt's formatting rules were dropped.
• Instruction following: 10,900 requests from smol-smoltalk's smol-magpie-ultra-short and openhermes-50k subsets, each with 1 to 3 added formatting rules of the kinds IFEval checks (length, bullets, sections, case, keywords, JSON, postscripts and so on). Answers by Gemma 4 26B-A4B-it (6,134) and Gemma 4 31B-it (4,766). Every answer was checked against every rule and kept only if it passed all of them; 693 failed a rule and were dropped.
• Multi-turn: 4,840 three-turn conversations. The user turns come from smol-smoltalk; Gemma 4 26B-A4B-it (2,912) or Gemma 4 31B-it (1,928) answered them in order, seeing its own earlier answers. Each whole conversation fits in 1,024 tokens.
• Identity set: 304 template conversations about the model itself: its name, its maker, its size and its limits.
• The teachers' descriptions of themselves were rewritten.
• Schedule: 3 passes over the distilled set, 2 over instruction following, 2 over multi-turn and 4 over identity (105,545 examples, 1,650 steps of 64). Peak learning rate 1.5e-4, 30 warmup steps, linear decay to zero. 46 minutes.
• Checkpoint: the one with the lowest validation loss.

Architecture

Parameters	121.2M (embeddings tied with the output layer)
Layers	12, pre-norm, RMSNorm
Width	768
Attention	grouped-query: 12 query heads, 4 key/value heads, head size 64
	RoPE (θ = 10,000), RMSNorm on queries and keys, sigmoid output gate
Feed-forward	SwiGLU, hidden size 2,048
Output	tanh logit soft-cap at 30
Vocabulary	GPT-2 BPE plus 4 chat tokens, padded to 50,304
Context	1,024 tokens

Limitations

• English only, with a 1,024-token context.
• Facts are unreliable: it states wrong facts confidently.
• It sometimes repeats itself or misses the point of a request.
• No preference tuning or safety training beyond what is in the fine-tuning data.
• It can't browse the web and has no knowledge of current events.

It is a research and learning model, not for production use.

License

Apache-2.0. The training data is FineWeb-Edu (ODC-By 1.0) and smol-smoltalk (Apache-2.0). The distilled answers come from Gemma 4 26B-A4B-it, Gemma 4 31B-it and dots.3-note-preview, all Apache-2.0.