zeechimp/hv-paleo-llm

paleo-llm

A 7-core transformer architecture inspired by dinosaur names from XuanJi issue #50. Requires PyTorch. Trains from scratch on a synthetic Fibonacci-mod-8 task.

Size: ~18 KB source, no pretrained weights. Runtime: ~1 minute for the full benchmark on CPU. Dependencies: PyTorch only.

The finding

The architecture has 7 biologically-named cores. The ablation says one of them does all the work.

subset	final accuracy
only trex (causal attention)	0.9195
all 7 cores	0.9634
all except trex	0.1261
no cores	0.1242
only pterosaur	0.1223
only raptor	0.1258
only sauropod	0.1241
only ankylosaurus	0.1229

Every core except T-Rex is at chance on this task, both alone and in combination with each other. The "unified intelligence tensor" is one transformer block plus six residuals that don't earn their place.

The architecture

Seven cores, each a nn.Module:

core	biological inspiration	algorithmic role
Pterosaur	flight navigation	low-rank complex projection
Raptor	pack hunting	parallel branch composition
T-Rex	predatory optimization	sparse causal attention
Sauropod	bulk digestion	wide FFN
Triceratops	hypervigilance	running-stats anomaly attenuation
Ankylosaurus	armored defense	gated residual repair
Archaeopteryx	flight control	slow-timescale channel bias

The hidden state (B, T, D) flows through all 7 cores in a fixed order, repeated for n_layers blocks.

Headline numbers

• Baseline (all 7 cores): 0.9634 val accuracy on Fibonacci-mod-8 after 300 steps, 210k params, ~16 s CPU.
• T-Rex alone: 0.9195, 37k params.
• No cores: 0.1242 (chance is 0.125).
• Depth scaling: 1 layer 0.958, 2 layers 0.963, 4 layers 0.958. Deeper is not better on this task.
• 16/16 consistency checks pass.

The dinosaur names are decoration

This is important to say plainly:

• "T-Rex core" is not a predatory algorithm. It is causal multi-head attention with top-k sparsification on the output channels. The "predatory" metaphor describes a design intuition, not a mathematical structure.
• "Pterosaur core" is a low-rank projection through a complex intermediate whose modulus is kept. It is a compression layer, not flight control.
• "Raptor core" is split-channel independent MLPs. It is parallelism, not pack coordination.

The names help you remember the modules. They do not constrain the mathematics, and the ablation shows they do not contribute distinct capabilities.

How to use

from paleo_llm import PaleoConfig, PaleoLLM, train_model, evaluate
import torch

# Construct the full model
config = PaleoConfig(d_model=64, n_layers=2, max_seq_len=32)
model = PaleoLLM(config)

# Train on the synthetic Fibonacci task
result = train_model(config, active_cores={'trex'}, steps=500)
print(result['final_acc'])           # ~0.92
print(result['n_params'])            # ~37k

# Run the ablation
from paleo_llm import CORE_NAMES
for core in CORE_NAMES:
    r = train_model(config, active_cores={core}, steps=300)
    print(f"{core}: {r['final_acc']:.4f}")