Trident

Trident is a raw-byte foundation-model architecture with variable-rate recurrence: it reads bytes, groups them into entropy-budget patches, and commits each patch once into a fixed-size recurrent state through a gated delta-rule update. Active state does not grow with sequence length.

• Training may use accelerators (GPU).
• Inference targets general-purpose CPUs (fixed working set, no accelerator).

The full specification is in trident.md.

Architecture (backbone, PH-A/B/C)

bytes ─▶ byte lane (embed + causal depthwise conv) ─▶ eta  (patch-invariant)
                                     │
                                     ├─▶ fast entropy head ─▶ H_t
                                     ▼
                     entropy-budget patcher  (close when A+H_t>tau_B or n=B_max)
                                     ▼
        patch codec (state-independent commit inputs: codes, gates, h0)
                                     ▼
        recurrent core: per-patch gated delta commit into S ∈ R^{d_v×d_k}
             S ← diag(ρ)S ;  S ← S + g_q(w_q v_q − b_q S k_q)k_qᵀ ;  r = S q_r
                                     ▼
        byte decoder: prev byte + committed context h_{j-1} + position-in-patch
                      + exact windowed causal attention over eta ─▶ 257-way byte dist

Design choice (v0): commit inputs are computed only from patch-invariant byte features, so every per-patch input is exogenous to the scanned state and the commit is a valid associative scan. This resolves the scan-validity gap flagged in the 0.1 audit. Scratch/shadow (PH-C), operators (PH-D), episodic memory, and latent reasoning are gated off until their kill gates pass.

Package layout

src/trident/        installable package (config, byte_lane, patching,
                    patch_codec, recurrence, decoder, model, data)
scripts/train.py    HF Jobs UV script (GPU pretraining, pushes checkpoints)
scripts/eval_compare.py   BPB vs real open-source models on identical bytes
tests/              causality, recurrence-scan parity, patcher invariants

Install & test

pip install -e .
pytest -q

Train (Hugging Face Jobs)

hf jobs uv run --flavor a100-large --timeout 6h --secrets HF_TOKEN \
  --label trident \
  "https://huggingface.co/farguney/trident/resolve/main/scripts/train.py"

Configuration is via environment variables (see scripts/train.py): PROFILE, SEQ_LEN, BATCH, GRAD_ACCUM, MAX_STEPS, LR, DATASET, etc.

Status

Research candidate 0.2. Correctness (causality, scan parity, patch invariants) is unit-tested; the architecture is validated as trainable. Competitive bits-per-byte is being established on real data with a public scorecard in results/.