Holo LLM — HRR Attention Conversions
Drop-in HRR (Holographic Reduced Representation) attention for Qwen2.5 models. Swap softmax for O(1) holographic state — no KV cache needed.
Conversion Track
| Model | Status |
|---|---|
| Qwen2.5-0.5B-Instruct → HRR | Converted, fine-tuning |
| Qwen2.5-7B-Instruct → HRR | Converting |
| Qwen2.5-14B-Instruct → HRR | Queued |
| Qwen2.5-32B-Instruct → HRR | QLoRA fine-tuning |
| Qwen2.5-72B-Instruct → HRR | Queued |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("staccs/holo-llm-hrr-attention", subfolder="qwen0.5b_hrr")
How it works
Each attention module is replaced with HRR attention — a gated holographic recurrence that stores context in a fixed-size superposed state (O(1) memory, O(T) time). The conversion copies Q/K/V/O projection weights and adds learned per-frequency forgetting gates (theta/beta parameters).
Fine-tuning on UltraChat recovers chat quality after the attention swap.
See also
- lecore-hrr-gpt2 — from-scratch proofs