Adelic-Gemma-4-31B-it

This repository contains the custom Adèlic Cache topological architecture wrapper for Gemma 4 (31B Multimodal), available in uncompressed and highly-optimized quantized (Q4_K_M, Q5_K_M) GGUF formats.

By injecting the Adèlic DynamicTopologyRouter and Medoid-Value similarity clustering into the attention layers, this architecture aggressively condenses the Key-Value (KV) cache into a $p$-adic Bruhat-Tits tree. This bounds the physical VRAM footprint to $\mathcal{O}(\log N)$, allowing for infinite context length generation on consumer hardware without Out-Of-Memory (OOM) crashes.

You can run this model using two different methods:

1. Natively via PyTorch/Transformers using our custom SRAM-accelerated Triton kernel patch.
2. Via llama.cpp using the pre-compiled .gguf weights, which natively incorporates the topological condensation kernel directly into the CUDA backend.

---

Method 1: PyTorch & Triton Patch (Native Hugging Face)

You do NOT need trust_remote_code=True because the patch applies cleanly onto native loaded models. Simply download the patch_adelic.py script from this repo and run it on your loaded model!

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import huggingface_hub

# 1. Download the Adèlic patch script
huggingface_hub.hf_hub_download(
    repo_id="sneedjak/Adelic-Gemma-4-31B-it", 
    filename="patch_adelic.py", 
    local_dir="."
)
from patch_adelic import apply_adelic_topology

# 2. Load the official Gemma tokenizer and model
model_id = "google/gemma-4-31B-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    device_map="auto"
)

# 3. Inject the Adèlic Topology (Triton Accelerated)
model = apply_adelic_topology(model)

# 4. Generate with infinite context!
prompt = "The quick brown fox jumps over the lazy dog. " * 50000 
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# The KV-cache will automatically condense, preventing your GPU from crashing.
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Method 2: llama.cpp (Pre-compiled GGUF)

For maximum efficiency and consumer hardware support, we have compiled the Adèlic condensation kernel natively into a custom fork of llama.cpp and uploaded the ready-to-run .gguf files to this repository.

Available Quants

• adelic-gemma4-31b.gguf (Uncompressed Q8/F16 Master Archive - Extremely High VRAM requirement)
• adelic-gemma4-31b-Q5_K_M.gguf (The "Sweet Spot" - Virtually indistinguishable from uncompressed, fits in 24GB VRAM)
• adelic-gemma4-31b-Q4_K_M.gguf (Fastest inference, lowest VRAM footprint)

The .gguf files contained in this repository require a custom fork of llama.cpp to run, as standard llama.cpp does not have the ggml_adelic_condense CUDA kernel.

Installation

1. Clone our custom llama.cpp fork containing the Adèlic kernels:

git clone -b feature/gemma4-adelic https://github.com/sneed-and-feed/llama.cpp.git
cd llama.cpp

2. Compile the binary (CUDA backend strictly required for the topological kernel!):

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

3. Download your preferred .gguf quantization from this Hugging Face repository.

Running Inference

Once compiled, you can run the model directly. The topology router will automatically and unconditionally mask out unneeded keys in the kq_mask on a per-layer basis using the hardware-accelerated CUDA condensation kernel!

./build/bin/llama-cli \
    -m adelic-gemma4-31b-Q4_K_M.gguf \
    -p "Explain the significance of the Bruhat-Tits tree in p-adic geometry." \
    -n 512 \
    -c 4096 \
    -ngl 999 \
    --color on

---

Performance, Fixes & Limitations

• Multimodal Hallucination Fix: The custom llama.cpp engine associated with this repository includes a hardcoded -INFINITY logit bias suppression for Gemma 4's multimodal tokens (e.g. <image|>, <audio|>). This permanently cures the token leakage bug where Gemma models randomly hallucinate visual control tokens during long-context pure-text generation.
• Semantic Fact Retrieval: On the LongBench QASPER dataset, this architecture successfully retrieved grounded facts from 10,000+ tokens away despite the massive topological compression of the KV-cache, proving that the Medoid-Value clustering strategy preserves Strict RoPE Coherence.
• Triton / CUDA Speedups: The cache condensation runs completely $\mathcal{O}(1)$ inside SRAM, avoiding thousands of slow sequential Python loops, allowing for decoding speeds up to ~50 t/s even while actively pruning the cache.
• Formatting Degradation: Because topological compression is lossy, the model's surface-level syntactic formatting (e.g., RLHF alignment <think> tags) may degrade into a stream-of-consciousness format over extreme context lengths. While semantic facts are preserved, raw string-matching $n$-gram benchmark scores (like exact-match F1) may be lower than an uncompressed baseline.

For full mathematical proofs of the RoPE coherence under topological compression, see the paper: Llama Surgery: Injecting Differentiable p-Adic Topology into Pre-Trained LLMs.