fausap/pico-165m

Pico: a 165M chat model that remembers through notes

Pico is a 165M-parameter model trained from scratch on about 2.7B tokens on a single TPU chip. It is built for multi-turn memory: remembering what the user said, keeping standing instructions ("call me Captain", "end every reply with 'cheers'") and noticing false premises. Instead of reading the whole conversation, it works as a resident agent:

• a note-taker turns each user message into rule: and fact: notes;
• the answerer sees only those notes and the last two exchanges.

On held-out conversations of about 47 exchanges, Pico scores 0.81 on core memory checks. SmolLM2-360M-Instruct, trained on 4T tokens, scores 0.31 reading the whole conversation, 0.31 when given the same notes, and 0.42 when given perfect notes.

It is a research model. It is far weaker than SmolLM2 on knowledge, translation and negation, and it invents facts (see Limitations).

Quick start

pip install torch tokenizers safetensors huggingface_hub
python pico.py                 # interactive chat; /notes shows what it remembers, /reset starts over

from pico import Pico
bot = Pico.load("<this-repo-id>")          # or a local folder with these files
bot.say("Hi! I'm Marta and I live in Porto.")
bot.say("Please call me Captain from now on.")
print(bot.say("Which city do I live in?"))   # e.g. "Captain, You told me you live in Porto."
print(bot.notes())   # (['call the user Captain'], ["the user's name is Marta", 'the user lives in Porto'])

pico.py is a dependency-light PyTorch port. On a test model its logits match the JAX training code to 2e-7. It runs the same chat loop as the training notebook, but without Wikipedia retrieval.

Files

File	What it is
answerer.safetensors	W03-50: the answering model, ½ S6 + ½ S6R3 averaged weight by weight (float32)
notetaker.safetensors	S6: the note-taking model (float32)
config.json	architecture: 30 layers, d_model 640, 10 heads, SwiGLU 1664, RoPE base 10000, context 1024
tokenizer.json	32k byte-level BPE trained on FineWeb-Edu, with chat special tokens
pico.py	PyTorch model, sampler, note grounding and the resident chat loop

Architecture: pre-norm transformer with RMSNorm, QK-norm, RoPE (split halves), SwiGLU, tied input and output embeddings, no biases.

Chat format: <|system|>…</s>\n<|user|>…</s>\n<|assistant|>.

How it works

1. Note-taking. Every user message goes to the note-taker with a fixed prompt. It is sampled three times (one greedy, two at temperature 0.7) and returns lines like rule: call the user Captain or fact: the user lives in Porto, or none.
2. Grounding. A note is kept only if the message supports it, using structural rules rather than keyword lists:
   • questions never set rules;
   • a title must appear capitalised in the message;
   • an ending word must be quoted and present;
   • facts must share content words with the message.
3. Note store. A new rule replaces an older rule of the same kind (a new title replaces the old one). Facts accumulate. The assistant's identity (name Pico, ~165M parameters, no internet access) is pinned as notes.
4. Answering. The answerer gets the system prompt, the notes and the last two exchanges. It uses temperature 0.3, top-k 40, and a repetition penalty on generated tokens only; words that a rule requires are exempt.

Training

Stage	What changed
Pretraining	~1B tokens of FineWeb-Edu with a curriculum (512 then 1024 tokens), then +1.7B tokens of FineWeb-Edu
SFT	chat data (smol-smoltalk, smoltalk subsets, OASST1, Dolly), RAG data (SQuAD v2 + Wikipedia), and synthetic note-protocol data: note-taking examples, answering from notes, whose fact is it
S6	the SFT model above; used as the note-taker
S6R2	GRPO with 16 programmatically checked tasks (formats, rules, recall, false premises, helpfulness), refusal penalty, KL to S6
S6R3	GRPO from S6R2: rules and recall in one reply, pinned identity, unmentioned things, copying any ending word
W03-50	½ S6 + ½ S6R3, weight average (no training); used as the answerer

Every message from the evaluation sets was filtered out of all training and RL data. The full pipeline is one JAX notebook that runs on one Colab TPU chip (v5e or v6e).

Evaluation

All checks are programmatic and run on held-out phrasings.

• e3: 160 instruction probes and 160 multi-turn probes; 95% CI about ±0.07.
• h2: 6 multi-turn families written after the last data revision. It has short conversations, plus long ones whose setup is about 47 exchanges back, beyond the 1024-token context.

SmolLM2 was run on the same probes with the same decoding.

	Pico (notes by S6)	SmolLM2-360M-Instruct	SmolLM2-360M + same notes
e3 multi-turn	0.68 [0.60, 0.74]	0.51 [0.43, 0.58]	0.48
e3 instructions	0.54	0.54*	–
h2 short conversations	0.69	0.48	0.47
h2 long conversations (core)	0.81	0.31 (full 8k history)	0.31
h2 long, perfect notes	1.00	–	0.42

* measured with Hugging Face's default repetition penalty.

Given the notes, SmolLM2 recalls entities (0.00 → 0.67 on long conversations) but ignores rule notes (persona 0.08) and does not use them to deny false premises (0.17). So the gain comes from an answerer trained on the note protocol, not from the notes alone.

Limitations

• Knowledge and content are weak. It invents books and gives poor practical advice. e3 translation is 0.00–0.10 and negation 0.00–0.05, against 0.50 and 0.65 for SmolLM2-360M.
• New phrasings are often misread. "What should you call me?" can get "You told me your name is Marta".
• It reaches for a "You told me …" template, learned from RL rewards that check whether a fact appears.
• Unmentioned things can be invented ("How is my dog doing?" → "You said you have a dog called …").
• The note-taker makes errors: it files pets as the user's name and invents titles from capitalised words.
• Context is 1024 tokens. English only.
• Evaluation limits: one training run per configuration, programmatic checks only, and the e3 set was used for model selection.

Do not rely on it for facts or advice.

Data and licence

The weights and code are released under Apache-2.0. Some training data is under share-alike licences: Dolly (CC BY-SA 3.0), SQuAD v2 (CC BY-SA 4.0) and Wikipedia (CC BY-SA). FineWeb-Edu is ODC-By; smoltalk, smol-smoltalk and OASST1 are Apache-2.0.

References

• Allal et al., SmolLM2: When Smol Goes Big, 2025 (the baseline).
• Packer et al., MemGPT: Towards LLMs as Operating Systems, 2023 (explicit memory for LLMs).
• Shao et al., DeepSeekMath, 2024 (GRPO).
• Shao et al., DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research, 2025 (rubric rewards).
• Wortsman et al., Robust fine-tuning of zero-shot models and Model soups, 2022 (weight averaging).
• Penedo et al., The FineWeb Datasets, 2024.