kiruluta/rsil-benchmark

Resonant Spectral Interaction Ladders (RSIL) — v2.1

Research code for Resonant Spectral Interaction Ladders: Translation-Equivariant Higher-Order Token Mixing with Kolmogorov-Arnold Parameterizations. The repository is designed to let collaborators test the architecture locally, then scale the same training loop to multi-GPU/FSDP or DeepSpeed and streaming corpora.

Research status: experimental reference implementation, not a pretrained production model. The current language objective is bidirectional masked language modeling. The FFT mixer is global/non-causal, so this repo intentionally does not claim autoregressive next-token generation without a separate causal construction.

🚀 Call for High-Compute Collaborators

We are seeking collaborators with access to multi-GPU systems to independently reproduce and scale RSIL from approximately 50M to 1B+ parameters.

Priority areas include multi-node training, large-corpus pretraining, optimized FFT/resonance kernels, long-context experiments, and independent reproduction.

Both positive and negative results are encouraged.

See COLLABORATION_CALL.md and the repository Discussions for the current scaling program.

What is implemented

• rFFT token mixing with deterministic multiband partitioning.
• KAN-style Chebyshev univariate radial gains on diagonal spectral resonances.
• Sparse degree-2 through degree-m cross-band interactions; token-domain products induce frequency-sum/convolution constraints.
• Residual encoder stack and tied masked-LM head.
• Completely offline deterministic synthetic CI/smoke dataset (v2.1).
• Hugging Face datasets streaming input remains available for scaling runs.
• Accelerate launch path with DDP/FSDP/DeepSpeed configuration examples.
• BF16, gradient accumulation, sharded training, Hub checkpoint upload.
• Equivariance and forward/backward smoke tests.
• The manuscript (paper.pdf) for theory and novelty scope.

Install

git clone <YOUR-HF-REPO-URL>
cd rsil
python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
pytest -q

1-GPU smoke training

accelerate launch -m rsil.train \
  --config configs/model/base.yaml \
  --dataset HuggingFaceFW/fineweb-edu \
  --tokenizer bert-base-uncased \
  --steps 100 --batch-size 2 --out outputs/smoke

The dataset is streamed rather than downloaded in full. For serious experiments, pin dataset revisions and tokenizer versions and record them in your run metadata.

8-GPU FSDP

Review num_processes and FSDP options for your cluster, then:

accelerate launch --config_file configs/accelerate/fsdp.yaml -m rsil.train \
  --config configs/model/large.yaml \
  --dataset HuggingFaceFW/fineweb-edu \
  --tokenizer bert-base-uncased \
  --steps 100000 --batch-size 4 --grad-accum 8 \
  --out outputs/large

For DeepSpeed ZeRO-3, replace the config with configs/accelerate/deepspeed_zero3.yaml.

Multi-node

Generate a cluster-specific config with accelerate config, or edit num_machines, num_processes, machine_rank, rendezvous settings, and networking for your scheduler. Prefer a shared experiment configuration committed to a branch/PR rather than changing model code per cluster.

Push a trained checkpoint to the Hub

hf auth login
accelerate launch -m rsil.train ... --push-to-hub YOUR_ORG/rsil-base-experiment

For very large checkpoints, use Hugging Face's current large-file/Xet workflow. Keep code, small configs, model cards, and reproducibility metadata in Git; use Hub model/dataset repositories for weights and data artifacts.

Recommended collaboration workflow

1. Create a Hugging Face organization/repository and push this code as the initial main branch.
2. Use Issues/Discussions to assign theoretical, systems, and benchmark workstreams.
3. Each collaborator creates a branch and PR with a config + run manifest for every architecture change.
4. Store large checkpoints in separate model repos, named by scale and experiment ID.
5. Keep datasets in dataset repos or reference public dataset revisions; do not commit raw corpora into this code repo.
6. Require the equivariance test and smoke training test before merging.
7. At scale, compare KAN/Chebyshev gains against MLP, spline, Fourier, and rational-function controls so results test the manuscript's claims rather than only one parameterization.

Scaling priorities

The reference code favors clarity over fused-kernel performance. Before billion-parameter or very-long-context runs, profile: repeated FFT/IFFT calls, band materialization, cross-band products, activation memory, and communication. High-value systems contributions include fused band filtering, cached masks, lower-rank channel factors, torch.compile, activation checkpointing, sequence parallelism, and custom Triton/CUDA kernels.

Suggested experiment matrix

Start with 10M–100M parameter models and sequence lengths 512–4096. Compare: Transformer encoder, Fourier mixer + MLP, RSIL with degree 1 only, degree 1+2, degree 1+2+3, and alternative univariate gain parameterizations. Then scale only variants that show stable loss/throughput gains. Report parameters, FLOPs/tokens, wall-clock throughput, peak memory, MLM loss/perplexity-like masked-token metrics, and translation-equivariance error.

Repository map

src/rsil/model.py       architecture
src/rsil/data.py        streaming HF data pipeline
src/rsil/train.py       Accelerate training entry point
configs/model/           model scales
configs/accelerate/      FSDP / DeepSpeed examples
tests/                   correctness smoke tests
paper.pdf                theory manuscript
CONTRIBUTING.md          collaboration rules
MODEL_CARD.md            experiment/model-card template

Citation

A BibTeX placeholder is provided in CITATION.cff; replace author metadata and publication identifiers before public release.

License

Apache-2.0 for the code. Confirm manuscript/data licensing separately before public release.

v2.1: offline synthetic CI data

The CI/smoke data path is now completely independent of Hugging Face datasets and tokenizers. --dataset synthetic creates deterministic masked-token examples locally using an index-seeded PyTorch generator. It reserves token 0 for padding and token 1 for masking, emits -100 labels for unmasked positions, and guarantees at least one masked token per sample.

# unit tests
make test

# all six benchmark families, tiny deterministic offline run
make ci-benchmark

# larger local smoke run, still no corpus download
make smoke

Use --seed, --synthetic-vocab-size, and --synthetic-samples to control reproducibility. The synthetic benchmark validates implementation, training, matching, checkpoint generation, and CI plumbing only; it must not be used for language-quality conclusions. Real-corpus scaling remains available by passing a Hugging Face dataset name.

v2 matched benchmark harness

The benchmark suite trains six families under a common masked-LM data/training pipeline: transformer, fourier, rsil_d1, rsil_d2, rsil_d3, and rsil_mlp (non-KAN control). The anchor is matched by an automatic width search that minimizes the maximum relative error in parameter count and an analytic forward-FLOP proxy. Both ratios are emitted in every result so runs outside the requested tolerance are visible rather than silently treated as matched.

Quick offline smoke benchmark (no Hugging Face dataset or tokenizer download):

pip install -e .
./scripts/run_benchmark.sh --config configs/benchmark/smoke.yaml --dataset synthetic --steps 100 --seq-len 256 --out outputs/smoke

The v2.1 benchmark harness defaults to --dataset synthetic. Samples, masks, and token IDs are generated deterministically from --seed; the default synthetic vocabulary is 4096 tokens. This path is intended for CI, correctness, checkpoint plumbing, and local benchmark validation—not language-quality claims. Use FineWeb-Edu or another real corpus only for scaling experiments.

Multi-GPU/FSDP:

accelerate launch --config_file configs/accelerate/fsdp.yaml -m rsil.benchmark.harness \
  --config configs/benchmark/scale.yaml --steps 10000 --seq-len 2048 \
  --mixed-precision bf16 --out outputs/scale

Outputs include per-model checkpoints/metrics plus leaderboard.csv and summary.json. FLOPs are a reproducible analytic proxy for budget matching, not a claim of measured hardware FLOPs; report measured throughput alongside it.

v2.2 real-language data pipeline

v2.2 fixes the preliminary WikiText failure in v2.1. The real-corpus path now removes raw strings by construction: it tokenizes documents, concatenates token IDs into a rolling stream, packs exact fixed-length blocks, and applies deterministic MLM corruption only after packing. The DataLoader uses zero workers for reproducibility and compatibility with one-shard streams such as WikiText-2.

Run the real preliminary comparison with:

./scripts/run_real_preliminary.sh

For a short validation before a 2,000-step run:

STEPS=20 ./scripts/run_real_preliminary.sh

Synthetic CI remains completely offline via make ci-benchmark. Real-language WikiText requires network access for the dataset/tokenizer on first use.

v2.4 real-language release workflow

Run a short validation first:

STEPS=20 ./scripts/run_real_preliminary.sh

Then the 2,000-step preliminary benchmark:

rm -rf outputs/real_preliminary
./scripts/run_real_preliminary.sh

Collect the locally generated checkpoints and result files into the Hugging Face publication tree:

./scripts/collect_real_release.sh outputs/real_preliminary

The tokenizer may consume source documents longer than BERT's nominal 512-token model limit while constructing the continuous packed stream. This is intentional: the model itself only receives fixed --seq-len blocks (256 by default).

v2.4 tokenizer hardening

Real-data packing now uses the fast tokenizer backend directly before fixed-length packing. This prevents Transformers from emitting misleading model_max_length warnings for long source paragraphs while preserving every token for subsequent 256-token blocks. The model still receives only seq_len tokens.

v2.5 Release Candidate: validation and repeated runs

The release-candidate benchmark trains on WikiText-2 train and evaluates each checkpoint on the held-out validation split. The leaderboard now records train_loss, validation_loss, validation_perplexity, throughput, parameter count, and the analytic FLOP proxy. Per-model history.csv files support loss-vs-step plots.

Run the single-seed release candidate with ./scripts/run_real_preliminary.sh. For a stronger result, run SEEDS="20260920 20260921 20260922" ./scripts/run_multiseed.sh. Generate figures with python scripts/plot_results.py --run outputs/real_preliminary --out publish/plots.

The small WikiText experiment is a preliminary controlled benchmark, not a claim of architecture superiority. The intended collaboration target is independent replication and scaling at 50M, 100M, 300M, and 1B+ parameters under matched parameter/compute budgets, with wall-clock throughput and memory reported alongside quality metrics.