ExeaLM
An open language model built by high schoolers, run by Exea Labs.
Status: research preview. No model weights yet. This repo holds the tokenizer and the results of the small-scale experiments that decided the design. Everything is reproducible from github.com/exea-labs/ExeaLM; the full report is results/README.md.
Links
- Code: https://github.com/exea-labs/ExeaLM
- Contributor sign-up: https://docs.google.com/forms/d/e/1FAIpQLSdPHVZhuXxBDUf-cJM2zDbj1JZPsduu95Y_kTJ_g4gKAXdDVw/viewform
- Architecture paper: TONN: Networks That Build Their Own Architecture from Data
Model (planned)
- Size: 0.1 billion parameters (101M with the planned vocabulary, about 90M with this 32k tokenizer).
- Architecture: a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits (convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other layers are standard multi-head attention. Each layer is followed by a gated MLP.
- Built-in memory circuits in every TONN layer:
- retrieval from a read-only library, with the model's own embeddings;
- a read-write message board shared by several copies of the model, so agents can talk.
- Training data: about 100B tokens; see DATA.md.
- License: Apache 2.0 for code and weights.
Results so far (small-scale, on one Mac Studio GPU)
Bits per byte on held-out English, Japanese and code (lower is better).
| Size | Comparison | ExeaLM hybrid | Same-size Llama-style transformer |
|---|---|---|---|
| 13M | same training steps, 2 seeds | 1.531 | 1.554 |
| 27M | same training steps, best learning rates, 2 seeds | 1.492 | 1.523 |
| 48M | same training time, 2 seeds | 1.445 | 1.452 |
| ~0.1B (90M) | same training time, 1 seed | 1.474 | 1.481 |
- At 0.1B the hybrid trains at the same speed as the transformer (1.01 times its cost per token).
- Training several copies of the model as a swarm sharing a message board beats a matched control by 0.014 bits per byte (2 seeds).
- Retrieval works on a synthetic lookup task: 100% on 1,024 unseen facts.
- On real text, retrieval is not yet a net gain. The model's own search finds the right Wikipedia article about 30% of the time; with a perfect search it would gain 0.040 bits per token.
All of these are short runs (about 50M training tokens), far smaller than the planned 100B-token run.
Files
tokenizer/tokenizer.json: the 32k byte-level BPE tokenizer.- Trained on: 200 MB of English (FineWeb-Edu), 160 MB of Japanese (LLM-jp Corpus v4: Wikipedia and FineWeb-2) and 80 MB of code.
- Why 32k: it won or tied against 49k and 64k at a fixed token budget.
- Special tokens: chat roles, plus
<|retrieved|>and<|board|>for the memory circuits.
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer/tokenizer.json")
Intended use
Research and education on how open language models are built. Not for high-stakes decisions.
Limitations
No evaluation results on standard benchmarks yet; the numbers above are language-modeling loss on small held-out sets. Like other web-scale models, future checkpoints may produce incorrect, biased, or harmful text. We will publish evals and known limitations with each checkpoint.
Contributing
Anyone can contribute through GitHub. See CONTRIBUTING.md and the Code of Conduct.