thefinalboss/palimpseste-max

PALIMPSESTE

A Self-Referential Hypervectorial Cortex — an append-only, associative memory substrate where "weights" are reconstructed on the fly by retrieval rather than stored in a matrix. Now with semantic generalization via distributional word embeddings, continuous retrieval (Soft Phi), and multi-level querying — all without a transformer.

A parchment that is never erased, only overwritten in layers.

No GPU. No gradient. No weights. No torch. No transformer. Trains at 2,000 tok/s on CPU. Infers at ~1.5s per response. Learns new facts instantly at runtime via O(1) memory writes. Generalizes by semantic similarity, not just character matching.

---

Live on Hugging Face

Model: thefinalboss/palimpseste-max

Spec	Value
Dimensionality (D)	20,000
Theoretical capacity	2^5,010 (approx. 10^1,505 associations)
Training pairs	5,534 (534 conversational + 5,000 TriviaQA)
Tokens trained	449,150
Training time	228 seconds (CPU only)
Training speed	1,973 tok/s
Model size	2.3 GB
Stored parameters	0 (weights are reconstructed, not stored)
Inference latency	~1.5s per response (with tuned LSH K=14/L=6)
Semantic modules	HV-Word2Vec, Soft Phi, Multi-Level Query
Tests	234, all passing

from palimseste.hf import HFPalimpsesteLM

lm = HFPalimpsesteLM.from_pretrained("thefinalboss/palimpseste-max")
print(lm.respond("who are you"))  # -> "i am palimpseste, a self-referential hypervectorial cortex."
print(lm.respond("who won super bowl xx?"))  # -> "chicago bears"

---

Semantic Generalization — The Breakthrough

PALIMPSESTE now generalizes by meaning, not just by character overlap. Three new modules solve the fundamental generalization problem:

1. HV-Word2Vec — Distributional Word Embeddings

Words that appear in similar contexts get similar hypervectors. This is the hypervectorial implementation of the distributional hypothesis ("you shall know a word by the company it keeps"). One pass through the corpus, O(N*D) per word, no gradient descent.

Proven semantic similarities (actual output from trained model):

sim(france, england)   = +0.296   (both countries, same context)
sim(capital, city)     = +0.229   (appear together in text)
sim(paris, rome)       = +0.299   (both capital cities)

Most similar to "france": england (+0.296)
Most similar to "paris":  rome (+0.299), berlin (+0.292), tokyo (+0.277)

"Capital" and "city" are similar not because they share letters, but because they appear in the same contexts. This enables cross-lingual and paraphrased matching: "main city of the hexagonal country" matches "capital of france" at the semantic level.

2. Soft Phi — Continuous Retrieval

Replaces the hard Hamming cutoff with exponential weighting. ALL candidates contribute, weighted by exp(similarity * temperature). The result is a blend of stored values, not just the nearest match.

If the model knows "capital of france -> paris" and "capital of italy -> rome", querying "capital of spain" (never stored) produces a blend in the "capital city" region of hyperspace — a reasonable guess rather than a blank stare.

3. Multi-Level Query

Every question is encoded at three levels and queried independently:

Level	Encoding	Captures
Surface	Token sequence + positional roles	Exact recall (precision)
Words	Bag-of-words (no positions)	Topic matching (order-insensitive)
Semantic	Word embeddings bundled	Meaning matching (cross-lingual)

Priority: surface -> semantic -> words -> fallback. "Main city of hexagonal country" fails at surface but matches at the semantic level.

---

Proven Cognitive Capabilities

1. Bilingual Conversation — 7/7 (100%)

[2.0s] hello        -> hello! i am palimpseste, how can i help you?
[2.2s] who are you  -> i am palimpseste, a self-referential hypervectorial cortex.
[2.5s] how do you learn -> i learn by writing to my memory. each observation is an o(1) insertion...
[1.8s] do you use a gpu -> no, i do not use a gpu. only bitwise operations: xor, popcount...
[1.8s] thank you    -> you're welcome! feel free to ask me more questions.

2. Real Trivia (TriviaQA) — 6/10 (60%)

[0.5s] who won super bowl xx?              -> chicago bears
[0.2s] where in england was dame judi dench born?  -> york
[0.5s] from which country did angola achieve independence in 1975?  -> portugal
[0.4s] how is joan molinsky better known?  -> joan rivers
[0.3s] which city does david soul come from?  -> chicago
[0.4s] in which branch of the arts is patricia neary famous?  -> ballet

3. Live Learning — Instant, O(1)

BEFORE: "what is the capital of mars" -> (wrong answer)
TEACH:  what is the capital of mars = olympus mons city  (O(1) memory write)
AFTER:  "what is the capital of mars" -> olympus mons city

4. Fact Chaining — Multi-Step Reasoning

Facts taught separately:
  who won the world cup 2018     -> france
  what is the capital of france  -> paris

Question NEVER taught:
  Q: "what is the capital of the country that won the world cup 2018?"

Reasoning trace:
  hop 1: "who won the world cup 2018" -> france
         resolved: "what is the capital of france"
  -> paris

5. Cognitive Layer — 5 Inference-Time Features

Feature	Description
Confidence scoring	Every response includes 0-100% confidence from Hamming similarity
Self-correction	User says "no, the answer is X" -> model learns O(1), uses immediately
Curiosity loop	Unknown question -> model asks user to teach it
Auto-chaining	Teaching A->B and B->C discovers A->C in background
Explanation trace	Every response explains why: "Matched 'bonjour' at 100%"

Test Summary

Category	Score
Bilingual conversation	7/7 (100%)
Trivia (TriviaQA)	6/10 (60%)
Live learning	Pass (instant)
Multi-turn reasoning	Pass (5 turns)
Technical knowledge	5/5 (100%)
Fact chaining	1/2 (50%)
Confidence scoring	Pass
Self-correction	Pass
Curiosity loop	Pass
Auto-chaining	Pass
Explanation trace	Pass
Semantic similarity	Pass (franceengland, parisrome)
Soft Phi interpolation	Pass (blends stored values)
Multi-level query	Pass (surface + words + semantic)

---

How It Works

PALIMPSESTE dissolves the distinction between weights and memory:

Conventional deep learning	PALIMPSESTE
Knowledge lives in dense weight matrices -> GPU (GEMM bottleneck)	Knowledge lives in an append-only log of (address, value, weight) triples
Learning = adjusting weights by gradient -> backprop, retraining	Learning = an O(1) append to an LSH-indexed table. No gradient.
Forgetting = inevitable (catastrophic forgetting) -> freeze, replay, EWC	Forgetting = a soft decay of access weights. Nothing is ever deleted.
Generalization = smooth interpolation in weight space	Generalization = semantic similarity via HV-Word2Vec + Soft Phi blending
Architecture is fixed and external to the data	The read-back kernel is encoded in H_meta and can rewrite itself

The five axioms

1. Everything is an address — (a, v, w) in H x H x R+; M is append-only.
2. Forward pass is associative retrieval — Phi(q) = sum w*v; O(log|M|) via LSH.
3. Learning is writing — M <- M + {(bind(x,c), y, 1)}; cost O(1) amortized.
4. Parameters are reconstructed, not stored — W_t = Phi(bind(x, s_t)).
5. Self-reference bounded by Lyapunov — meta-params in H_meta; dE[surprise] <= 0.

Architecture

palimseste/
├── hv.py             # hypervector primitives: bind / bundle / similarity (packed bits)
├── lsh.py            # Hamming LSH index — O(log|M|) neighborhood retrieval
├── memory.py         # append-only knowledge base M + H_meta subspace
├── phi.py            # hard Phi (Hamming cutoff retrieval)
├── soft_phi.py       # Soft Phi (exponential-weighted continuous retrieval)
├── learner.py        # O(1) write + Encoder (ints/floats/strings/sequences/sets)
├── hv_word2vec.py    # HV-Word2Vec: distributional word embeddings
├── multiquery.py     # Multi-level query: surface + words + semantic
├── bpe.py            # BPE sub-word tokenizer (2.3x token reduction)
├── attention.py      # HV selective attention (context window 512+)
├── abstraction.py    # concept extraction via HV clustering
├── consolidation.py  # Hebbian co-activation -> abstract concepts
├── meta.py           # H_meta + Lyapunov-bounded self-rewrite (Axiom 5)
├── loop.py           # the autonomous active-inference agent
├── tokenizer.py      # char-level tokenizer (legacy, still supported)
├── lm.py             # PalimpsesteForCausalLM + Q/A training + respond()
├── chat.py           # Conversation: multi-turn memory + live learning
├── reasoning.py      # Reasoner: fact chaining (multi-step reasoning A->B->C)
├── cognitive.py      # CognitiveAgent: confidence, self-correction, curiosity
├── vision.py         # ImageEncoder: images -> hypervectors (multi-modal)
├── serialization.py  # save/load Memory + Encoder (vectorized, pickle-free)
├── hf.py             # HFPalimpsesteLM: save_pretrained / from_pretrained / push_to_hub
├── tests/            # 234 tests, all passing
├── examples/         # quickstart, benchmark, train_lm, train_chat, train_1b,
│                     #   chat, generate, api_server, corpus_chat
├── web/              # official React app (Vite + TypeScript + Tailwind)
│   ├── src/          # components: ChatPanel, Dashboard, TeachPanel, MemoryViz
│   └── dist/         # production build (served by API server)
└── docs/architecture.md

---

Quick Start

Install

cd palimseste
pip install -e .
pip install -e ".[dev]"  # pytest

Requires Python >= 3.10 and numpy >= 1.24. No GPU, no torch.

Train a chat model

python examples/train_chat.py --preset small --output ./my_model
python examples/chat.py --model ./my_model

Train semantic embeddings

from palimseste.hv_word2vec import HVWord2Vec, Word2VecConfig

w2v = HVWord2Vec(config=Word2VecConfig(D=10000, n_epochs=5))
w2v.train(open("corpus.txt").read(), verbose=True)
print(w2v.most_similar("france"))  # -> [("england", 0.296), ...]

Deploy as a web API with the React app

pip install fastapi uvicorn
python examples/api_server.py --model ./palimpseste-max --port 8000

The server automatically serves the official React app from web/dist/ if it exists. To build it:

cd web && npm install && npm run build

Then open http://localhost:8000 for the full application, or use the JSON API directly:

Endpoint	Method	Description
/chat	POST	Chat with confidence, source, explanation, chain
/chat/stream	POST	SSE streaming with token-by-token output
/teach	POST	Teach a new fact O(1), instantly retrievable
/metrics	GET	Rich dashboard: memory, config, conversation state
/stats	GET	Model statistics
/memory/samples	GET	Recent memory traces for visualization
/conversation	GET	Current conversation transcript
/health	GET	Health check

Live learning in Python

from palimseste.hf import HFPalimpsesteLM
from palimseste.chat import Conversation

lm = HFPalimpsesteLM.from_pretrained("thefinalboss/palimpseste-max")
conv = Conversation(model=lm, learn_live=True)

# Teach a new fact at runtime — O(1), instantly usable
conv.teach("capital of mars", "olympus mons city")
print(conv.respond("capital of mars"))  # -> "olympus mons city"

---

Performance Benchmarks

Operation	Cost	Measured
Write (learn one token)	O(1) amortized	~0.5ms (flat across M)
Read (Phi query)	O(log M) via LSH	~1.5s at 449K traces (tuned LSH)
Training speed	—	1,973 tok/s at D=20,000
BPE token reduction	—	2.3x fewer tokens than char-level
Recall accuracy (training data)	—	~100% (exact context match)
Trivia recall (TriviaQA)	—	60% (tunable via LSH tightness)
Semantic similarity	—	franceengland: +0.296, parisrome: +0.299

Theoretical capacity (Kanerva SDM): C = 0.14*D * 2^(D/beta). With D=20,000, this is ~10^1,505 associations — vastly exceeding any practical M.

---

What PALIMPSESTE Is (and Isn't)

Is: an associative memory with semantic generalization. It stores exact (context -> token) transitions and retrieves them by Hamming-neighborhood lookup. On training data, ~100% accuracy. For structured domains (Q&A, trivia, code), it trains in O(1) per token, never forgets, and now generalizes via distributional word embeddings and continuous retrieval.

Isn't: a transformer. It does not use attention matrices, weight gradients, or backpropagation. Generalization happens through HV-Word2Vec (semantic similarity) and Soft Phi (interpolation), not smooth weight-space interpolation.

The unique capability: live learning with semantic generalization. A transformer cannot learn a new fact without fine-tuning. PALIMPSESTE learns in O(1) — one memory write — and the fact is immediately retrievable. Combined with semantic embeddings, it can match by meaning, not just by surface form.

---

Testing

pytest -q    # 234 tests, all passing

License

MIT.