hv-csi-detector-4096

A distribution-free sequence structure detector. Given an 8-class stream, produces a Compositional Symmetry Index (CSI) whose bound is exact in the limit and whose finite-sample certificate uses a calibrated null threshold.

NumPy only. ~40 KB artifact. Sub-millisecond per 100 KB on a laptop CPU.

The theorem

Fix a specialty subset S of 4 classes among the 8. For a 4-window w:

c(w) = 1 if all four letters of w lie in S
p(w) = 1 if each of the four letters of S appears exactly once in w

For any i.i.d. distribution on the 8 classes:

E[p]  ≤  24 · (q/4)^4                    (AM-GM)
CSI  :=  E[p] / [24 · (q/4)^4]  ≤  1     (i.i.d. bound, asymptotic)

For any sequence distribution:

p ≤ c  ⇒  E[p] ≤ E[c]  ⇒  CSI ≤ 32/3

CSI > q95(n, q) certifies non-independent placement of the specialty classes. No reference sequence, no shuffle, no bootstrap. The 32/3 upper bound is tight (attained by periodic SBPZ·SBPZ·…).

The finite-sample certificate

Under i.i.d. with n = L - 3 windows and p_null = 24·(q/4)^4:

E[CSI]    = 1
Var[CSI]  ≈ (1 - p_null) / (n · p_null)
q95(CSI)  ≈ 1 + 1.9 · sqrt(Var)      (small-sample corrected)
q99(CSI)  ≈ 1 + 2.6 · sqrt(Var)

certificate uses q95 (~5% false positives on i.i.d.); certificate_99 uses q99 (<1% false positives). Use q99 for demos and production.

Two-factor decomposition

CSI = CSI_comp · CSI_struct

CSI_comp             = 256 · Π r_i      (r_i = p_i / q)
CSI_struct           = E[p] / (24 · Π p_i)
CSI_struct_smoothed  = E[p] / (24 · Π p_smoothed)

CSI_comp ∈ [0,1]: 1 iff the four specialty classes appear uniformly. CSI_struct = 1 for i.i.d.; >1 iff correlated within windows.

Which channel to read

stream shape	primary channel	why
Balanced (each class ~12.5%)	csi	both components non-degenerate
Specialty-heavy, skewed (e.g. text: lowercase ~75%)	csi_struct_smoothed	csi_comp collapses to ~0; smoothed structural channel survives
Specialty-sparse (<30% in specialty)	none	motif too rare; warning fires

For English prose under char_class + TEXT_SPECIALTY ({ws, lower, upper, punct}):

CSI         = 0.4726
CSI_comp    = 0.0191
CSI_struct_smoothed = 24.67
CSI_shuffled = 0.0142

Interpretation: given the observed character marginal, permutation windows appear 24.7× more often than i.i.d. would predict.

The shuffled control

csi_shuffled recomputes CSI on a random permutation of the same stream.

relation	interpretation
csi ≈ csi_shuffled	structure is compositional
csi ≫ csi_shuffled	structure is positional

What the detector is not

CSI is motif-specific, not a general randomness test. A stream that does not produce permutation windows gives CSI = 0 regardless of how structured it is.

Use CSI alongside NIST STS, dieharder, or ent for general randomness testing.

Benchmarks

i.i.d. null (uniform 8-class, q = 0.5):

L	empirical q95	asymptotic q95	empirical q99	asymptotic q99
100	3.64	3.53	6.76	4.46
1,000	1.90	1.78	2.31	2.07
10,000	1.24	1.25	1.35	1.34
100,000	1.09	1.08	1.11	1.11

Controls:

input	CSI	certificate_99
Random bytes mod 8, L=1M	≈ 1.0	False
Periodic SBPZ, L=100K	10.667	True
LFSR-15 low bits	0.0	False (motif absent)
Balanced i.i.d., L=50K	≈ 0.92	False
Missing specialty class	0.0	False

Natural text under char_class + TEXT_SPECIALTY:

sample	CSI	CSI_shuf	CSI_struct_smoothed
Prose	0.47	0.014	24.67
Shakespeare	0.25	0.049	5.81
Code (no uppercase)	0.0	0.0	0.0
Base64 (no whitespace)	0.0	0.0	0.0

How to use

from hv_csi_detector import (
    CSIDetector, CSIConfig, TEXT_SPECIALTY, BALANCED_SPECIALTY)

det = CSIDetector()
det_text = CSIDetector(CSIConfig(
    spec=TEXT_SPECIALTY, projection='char_class'))

result = det.score(open('data.bin', 'rb').read())

stream = det.stream(window=10_000)
for chunk in chunks:
    score = stream.push(chunk)
    if score and score['certificate_99']:
        print(f"certificate at t={stream.total_seen}: CSI={score['csi']:.3f}")