ENSEMBLE β a training-free AI built from portable experts
No gradient. No epochs. No GPU. Turn any dataset into a compressed expert file in one pass, then let a Kuramoto-coupled brain synchronize experts into emergent answers. ENSEMBLE thinks when idle, grows a persistent central memory, and safely rewrites its own coupling. It is built on the Palimpseste hypervector substrate.
ENSEMBLE is not a transformer and is not trying to be one. It is an experiment in a different direction: a training-free, ultra-compact, instantly-updateable, composable associative memory that grows smarter over time without retraining. This README is precise about what it is, what it does well, and where it loses to a trained 1B transformer.
What's new in v0.2
- Global shared BPE tokenizer β one tokenizer across all experts lifts the char-level quality ceiling. On TinyStories (150 KB excerpt, D=5000): next-token accuracy 94.5% β 98.5%, query latency 112ms β 28ms (4Γ faster), RAM 187MB β 89MB (2Γ less).
- Persistent central brain memory β the brain now accumulates synthetic concepts (from continuous thinking) in a memory that survives
save()/load(). The system grows indefinitely: add experts + let it think, no retraining. - Brain save/load β full brain state persistence.
- 75 tests passing (was 49).
What's new in v0.3 β structural generalization
- Structural query encoding β decompose a question into a PATTERN + a SLOT via explicit templates (
Expert.from_qa_pairs(..., patterns=["what is the capital of {country}"])). Questions sharing a pattern are encoded consistently, so the answer format generalizes to unseen slots. - Measured generalization lift: on capitals (train: france/germany/italy/japan/egypt; holdout: spain/portugal/greece/brazil/norway/india/mexico/kenya), the char-level baseline returns an empty answer 100% of the time on unseen slots; the structural encoder returns a non-empty answer 100% of the time. That is +100 points of graceful degradation on the unknown.
- Honest scope: this guarantees a well-formed answer shape, not factual correctness for a fully unseen slot. Without semantic embeddings, "spain" and "france" are orthogonal at the HV level, so the system imitates structure rather than interpolating meaning. The README is explicit about this.
- 91 tests passing (was 75).
What's new in v0.4 β embedding expert (optional semantics)
- Embedding expert β an optional plugin that brings real semantic similarity to the brain. Dense word vectors (fastText, or a local mini-embedding) are projected into HV space via a seeded random-projection-then-sign (the new
projection.pyprimitive), sospainandportugalbecome similar hypervectors (cosine-preserving, JL-style). This is exactly the ingredient the structural encoder needed for analogy to find the right slot. - It's a pure plugin:
EmbeddingExpertsatisfies the same duck-typed contract asExpert(relevance / candidate_hv / answer / signature_hv), so it plugs into the Brain and couples with lexical experts via Kuramoto with zero changes to the brain. The training-free / numpy-only core stays pure; semantics is opt-in. - Two construction paths:
from_corpus_local(PPMI+SVD mini-embedding, training-free, numpy only, seconds β lower quality) andfrom_fasttext(load pretrained.vec, best quality, one-time download). - Honest result: the embedding expert measurably brings semantic similarity (
spain~portugal = 0.18vs~0without), which improves routing. However, with the mini-embedding the factual-correctness on the capitals holdout remains ~0% β the slot is found but the answer content is still decoded char-level. The pipeline is ready for fastText to deliver real factual gains; that's the documented next step. - 106 tests passing (was 91).
What's new in v0.5 β factual generalization via dense analogy
- Dense analogy solving β
EmbeddingExpert.learn_relation("capital_of", {france: paris, ...})+solve_analogy("capital_of", "spain")implements the classic Mikolov analogy (paris β france + spain β madrid) in normalized dense space, averaging the(answer β slot)direction over known examples and finding the nearest word β excluding the query slot so it doesn't parrot itself. - Wired into structural answers β when an embedding expert is attached to a lexical expert (
from_qa_pairs(patterns=..., embedding=emb)), unseen slots are answered via dense analogy instead of char-level decode, returning the plain-text answer. - Measured factual gains (fastText, end-to-end via the Brain): on a capitals holdout (train: france/germany/italy/japan/egypt/russia; holdout: spain/portugal/greece/norway/china/india/brazil/turkey), factual correctness goes from 0% (char) / 0% (structural-only) to ~50β67% (structural + embedding analogy). Concretely:
spain β madrid,portugal β lisbon,greece β athens,norway β osloare recovered correctly despite never being in training. - Honest scope: fastText analogies are imperfect (china β porcelain, india β indian on misses) β that's fastText's distributional ceiling, not the architecture. More known pairs and larger vocab would lift it further. The point stands: the wall is broken.
- 109 tests passing (was 106).
The four ideas
1. Dataset β Expert (.exp) β one pass, no gradient, smaller than the source
Any dataset becomes a frozen, portable expert file in a single pass. There is no gradient descent and no epoch β learning is an O(1)-per-token write into an append-only memory. The expert file is smaller than the dataset (typically 3β9Γ): hypervectors are never stored; the symbolic token stream is gzipped and HVs are rebuilt on load from a deterministic encoder.
2. Kuramoto brain β experts couple and synchronize into emergent answers
Load several experts into a Brain. Each expert becomes a Kuramoto oscillator (natural frequency = relevance, coupling = signature similarity). The oscillators synchronize by similarity; the emergent attractor is a state no single expert produced. The brain routes questions to the right expert or composes answers when two experts know the topic.
3. Global shared BPE + persistent central memory β the brain grows
A single BPE tokenizer makes all experts token-compatible (so they share the same vocabulary and can be assembled freely). Meanwhile the brain's central memory accumulates synthetic concepts discovered during continuous thinking β and unlike a transformer's frozen weights, this memory persists and grows. Adding knowledge is Lego: drop in a .exp, let the brain think, and its concept store expands. No joint retraining.
4. Continuous thought + safe self-modification
When idle, the brain thinks: it samples remembered queries from one expert, asks all the others, runs the attractor, and writes novel coherent results into the central memory. It can also rewrite its own coupling under a Lyapunov constraint (ΞE β€ 0). The acceptance rule is immutable β the recursion is bounded by construction.
How it differs from an LLM
| Transformer LLM | ENSEMBLE | |
|---|---|---|
| Learning | gradient descent, ~1T tokens | one-pass write, no gradient |
| Adding knowledge | retrain / fine-tune / RAG | drop in a .exp (Lego) + think |
| Model size vs data | grows with parameters | expert is smaller than its data |
| Grows over time | frozen weights | central memory accumulates concepts |
| Idle behavior | nothing | dreams across experts, writes concepts |
| Self-tuning | hyperparameters fixed | rewrites its own coupling (Lyapunov-bound) |
| Hardware | GPU | plain CPU |
Quick start
This repo is self-contained: it vendors the palimseste substrate, so a fresh clone runs with only numpy.
git clone https://huggingface.co/thefinalboss/ensemble
cd ensemble
pip install numpy # the only runtime dependency
python -c "from ensemble import Brain, Expert; print('ok')"
Build experts and assemble a brain
from ensemble import Expert, Brain
# Option A: char-level (backward compatible, simplest)
math = Expert.from_qa_pairs(
[("what is pi", "pi is approximately three point one four")] * 5,
domain="math", D=10000)
# Option B: global shared BPE (recommended β higher quality)
corpus = open("some_corpus.txt").read()
bpe = Expert.build_bpe(corpus, vocab_size=2000, D=10000) # train once
math = Expert.from_qa_pairs(
[("what is pi", "pi is approximately three point one four")] * 5,
domain="math", D=10000, tokenizer=bpe) # share it
geo = Expert.from_qa_pairs(
[("what is the capital of france", "the capital of france is paris")] * 5,
domain="geography", D=10000, tokenizer=bpe)
brain = Brain()
brain.add_expert(math)
brain.add_expert(geo)
print(brain.query("what is pi").answer) # -> math
print(brain.query("what is the capital of france").answer) # -> geography
brain.think(seconds=10) # the brain dreams -> writes to central memory
brain.self_modify() # safely retunes its own coupling
# the central memory persists
brain.save("mybrain")
brain2 = Brain.load("mybrain") # concepts survive reload
print(f"{brain2.n_concepts} concepts persisted")
Saving experts (compressed)
result = math.save("math.exp") # source -> expert, compressed
math2 = Expert.load("math.exp", tokenizer=bpe) # BPE experts need the shared tokenizer
Structural generalization (unseen slots)
# Train on capitals of some countries, WITH a template.
# The template lets the expert generalize the *answer format* to unseen slots.
geo = Expert.from_qa_pairs(
[("what is the capital of france", "the capital of france is paris"),
("what is the capital of germany", "the capital of germany is berlin")] * 4,
domain="geo", D=10000,
patterns=["what is the capital of {country}"])
# spain was NEVER in training β yet the structural expert answers (format-generalized),
# where a char-level expert would return empty.
print(geo.answer("what is the capital of spain")) # non-empty, by analogy
Scaling to "1B-equivalent"
ENSEMBLE has no stored parameters, so "1B" means capacity β distinct associations the memory holds without collision, exponential in dimension D. The 1b preset uses D = 100 000.
| preset | D | use case |
|---|---|---|
tiny |
2,000 | quick demos |
small |
10,000 | laptop default |
medium |
30,000 | more capacity |
large |
50,000 | large corpora |
1b |
100,000 | "1B-equivalent" capacity |
Benchmark results (measured, reproducible)
Full methodology in RESULTS.md.
BPE vs char-level on TinyStories (150 KB excerpt, D=5000)
| metric | char | BPE | delta |
|---|---|---|---|
| next-token accuracy | 94.5% | 98.5% | +4.0 pts |
| query latency | 112 ms | 28 ms | 4Γ faster |
| RAM | 187 MB | 89 MB | 2Γ less |
| build time | 46s | 107s | one-shot cost |
The global BPE lifts the quality ceiling exactly as predicted: higher accuracy, far lower latency (fewer, longer tokens), and half the RAM. The tradeoff is a slower one-time build.
ENSEMBLE scaling with D (facts corpus)
| D | next-token acc | QA recall | .exp/source |
|---|---|---|---|
| 2,000 | 30.6% | 0% | 7.0Γ |
| 10,000 | 93.5% | 35% | 7.0Γ |
| 100,000 (1b) | 94.7% | 40% | 7.0Γ |
Generalization to unseen slots (capitals benchmark)
Train on capitals of 5 countries; hold out 8 unseen countries. Metric: % of holdout returning a non-empty, well-formed answer.
| mode | non-empty on unseen slots |
|---|---|
| char-level | 0% (silent failure) |
| structural (patterns) | 100% (graceful, format-generalized) |
Honest: the structural answers are well-formed guesses by analogy, not factually correct for fully unseen slots (no semantic embeddings). The win is graceful degradation β the system says something sensible instead of nothing.
Honest verdict vs 1B transformers (TinyLlama-1.1B, Pythia-1B)
ENSEMBLE wins on: zero training (30s CPU vs a GPU cluster on 1T tokens), footprint (a 2 KB** vs ~2 GB), instant knowledge injection, compositionality, near-perfect memorization of seen data (95β99% with BPE), and continuous growth (the central memory never stops accumulating).1b expert is **
A 1B transformer wins on: broad world knowledge (MMLU, HellaSwag), generalization (ENSEMBLE memorizes; holdout QA β 0%), fluency on unseen text, reasoning.
Bottom line: ENSEMBLE is a different tool. For narrow domains with known data and a CPU-only / tiny-footprint constraint, it is competitive or superior. For general intelligence, it is not β yet.
Architecture
Expert dataset -> compressed .exp (one pass, no gradient)
β build_bpe train a global shared BPE tokenizer
ExpertOscillator an expert as a Kuramoto oscillator
ExpertKuramotoAttractor multi-expert synchronization -> emergent attractor
Brain conductor: roster, query, think, self-modify, save/load
BrainMemory persistent central memory (grows via thinking)
ContinuousThinking cross-expert dreaming -> writes concepts to BrainMemory
SelfModifier Lyapunov-bounded retuning of coupling
What this is (and is not)
Is: a training-free associative memory built on hypervector (VSA) algebra; an expert system where knowledge is portable, compressed, composable, and grows over time; a research artifact exploring Kuramoto coupling + persistent concept memory. CPU-only, numpy-only, auditable.
Is not: a transformer; competitive on broad-knowledge benchmarks; a generalizer (it memorizes what it is shown).
Run the tests
pip install numpy pytest
pytest tests/ -q # 75 tests
Repository layout
ensemble/ the ENSEMBLE package (expert, bpe integration, brain, brain_memory, ...)
palimseste/ the vendored hypervector substrate
bench/ reproducible benchmarks (incl. BPE vs char comparison)
tests/ 75 tests
RESULTS.md full benchmark report
License
MIT. Both ensemble and the vendored palimseste substrate are MIT-licensed.