kiruluta/J-Space-HF-Scaling-Benchmark

J-Space Dynamics — Hugging Face Scaling Benchmark

A reproducible community benchmark for scaling J-Space Dynamics (JSD) analysis across open causal language models, training checkpoints, random seeds, and compute platforms.

The benchmark packages the J-Space Neural Analyzer and the experimental protocol associated with Probing the Formation and Dynamics of J-Space in Language Models.

The goal is to test the proposed measurements at substantially larger scales and across independent hardware—not to treat benchmark throughput or small-model runs as validation of the scientific hypotheses.

🚀 High-compute collaborators wanted

The initial benchmark has now been validated on a single NVIDIA GB10 through EleutherAI Pythia-2.8B.

We are looking for collaborators with access to larger accelerators and clusters—particularly H100/H200, B100/B200, GB200/GB300, MI300X-class systems, multi-GPU servers, and multi-node compute—to extend the benchmark to larger models, dense checkpoint suites, replicated seeds, and the more expensive JSD instruments.

See COLLABORATION.md for the contribution program.

Validated schema-1.0 baseline

The following measurements were produced on one NVIDIA GB10 using cached Hugging Face checkpoints and FP16 model execution.

Model	Sampled layers	Peak allocated GPU memory	Analysis time	Prompt-layer/s
EleutherAI/pythia-410m	0, 6, 12, 18, 23	0.911 GiB	2.038 s	19.6273
EleutherAI/pythia-1b	0, 4, 8, 12, 15	2.049 GiB	2.433 s	16.4390
EleutherAI/pythia-1.4b	0, 6, 12, 18, 23	2.817 GiB	2.476 s	16.1533
EleutherAI/pythia-2.8b	0, 8, 16, 24, 31	5.395 GiB	2.908 s	13.7544

These are performance and reproducibility measurements, not evidence by themselves that larger models exhibit stronger, weaker, or qualitatively different J-Space structure.

The sampled layer sets differ because the model architectures have different depths. Prompt-layer throughput normalizes by the number of sampled prompt-layer combinations, but not by hidden width, parameter count, FLOPs, or architecture.

The complete measured rows are in results/leaderboard.csv.

What JSD measures

The framework extends the J-lens program with five falsifiable instrument families:

1. Continuation-margin robustness — local second-order estimates of how far a representation can move before a decoded concept changes.
2. Path-conditioned sequence and binding directions — tests for order- and context-sensitive structure beyond isolated token directions.
3. Non-autonomous transfer operators — layer/position-dependent workspace dynamics with held-out closure and finite-window persistence tests.
4. J-restricted pullback Fisher geometry — local information geometry measured within the J-space coordinates.
5. Longitudinal consolidation analysis — checkpoint-level tracking of reconstruction, effective dimension, persistence, and related quantities.

The benchmark is designed so that failure of identifiability, closure, persistence, robustness, replication, or another prerequisite remains an informative result.

Quick start

Download the Hugging Face repository and install the package:

hf download kiruluta/J-Space-HF-Scaling-Benchmark --repo-type model --local-dir J-Space-HF-Scaling-Benchmark
cd J-Space-HF-Scaling-Benchmark
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e '.[huggingface]'

Run the CPU/offline software validation:

jspace-analyze demo

Run the validated Pythia-410M GPU benchmark:

python benchmarks/run_scaling.py \
  --model EleutherAI/pythia-410m \
  --layers 0,6,12,18,23 \
  --dtype float16 \
  --tag community

The benchmark writes a portable JSON result, timestamped scientific artifacts, and a row in results/leaderboard.csv.

Benchmark schema 1.0

The public benchmark uses schema 1.0.

Timing is separated into:

• elapsed_s — complete benchmark wall-clock time;
• model_load_s — tokenizer/model construction and loading;
• analysis_s — J-Space scientific computation after model loading and before report serialization;
• prompt_layer_per_s — (prompt_count × sampled_layer_count) / analysis_s.

This separation prevents model download/loading behavior from being mistaken for J-Space analysis throughput.

For performance comparisons, contributors should state whether checkpoints were already cached.

See BENCHMARK.md for the full protocol.

Analysis precision

Model execution honors --dtype (for example float16), while arrays entering NumPy statistical and linear-algebra routines are promoted to FP32.

This avoids unsupported NumPy FP16 linear algebra without changing the requested model execution precision.

Layer handling

The benchmark records the exact sampled layers.

Layer-specific dynamics, robustness, binding, and Fisher analyses are kept consistent with that selected layer set. Explicit overrides are available for contributors who need controlled layer placement.

See BENCHMARK.md for details.

What we want collaborators to scale

High-value contributions include:

• substantially larger open-weight causal language models;
• dense checkpoint-family sweeps;
• multi-seed replication;
• second-order robustness scaling;
• J-restricted Fisher scaling;
• bootstrap uncertainty propagation;
• causal/intervention experiments;
• rigorously tested distributed or model-parallel execution.

The existing multi-GPU helper measures independent one-job-per-GPU throughput. It should not be interpreted as tensor, pipeline, or distributed model parallelism.

Scientific guardrails

• Validate dictionary identifiability before interpreting coefficients.
• Propagate coding uncertainty into downstream conclusions where required.
• Do not interpret persistence when held-out closure is poor.
• Compare Fisher-matched interventions with appropriate controls.
• Treat empirical Taylor-radius measurements as local unless stronger curvature conditions are established.
• Treat WCI as secondary and report its underlying multivariate components.
• Use continuous behavioral metrics and replicated seeds before making claims about longitudinal transitions.
• Record negative results and failed prerequisites rather than silently discarding them.
• This benchmark studies functional and representational structure; it is not a consciousness test.

Failure reporting

Failed runs are useful contributions when their causes are recorded accurately.

Examples include GPU OOM, checkpoint download/reconstruction failure, unsupported architectures, numerical instability, invalid layer selection, estimator failure, failed dictionary identifiability, and failed held-out closure.

A development attempt with Pythia-6.9B failed during Hugging Face checkpoint download/reconstruction before benchmark execution. It therefore does not establish a GPU-memory limit or a JSD model-size limit.

Repository map

• src/jspace_analysis/ — analysis implementation
• benchmarks/run_scaling.py — standardized hardware/model benchmark runner
• scripts/run_model_ladder.sh — sequential model-size sweep
• scripts/run_multigpu_independent.sh — independent one-job-per-GPU throughput sweep
• configs/ — tiny and Hugging Face configurations
• results/leaderboard.csv — clean schema-1.0 leaderboard
• results/leaderboard_legacy_v1.3.csv — preserved development results
• results/ — benchmark rows and scientific artifacts
• docs/ — methods, validation, limitations, and manuscript material
• BENCHMARK.md — benchmark protocol
• COLLABORATION.md — high-compute contributor call

Contributing results

Run the current benchmark runner rather than constructing leaderboard rows manually.

Please preserve the generated result JSON and associated scientific run directory. Include model revision information when known, exact layer selection, hardware/software information, cache status, and any failures encountered.

Portable contributions should not expose unnecessary absolute local filesystem paths.

Citation

Please cite the manuscript:

Andrew Kiruluta, Probing the Formation and Dynamics of J-Space in Language Models, 2026.

Please also preserve attribution to the originating J-lens work described in the manuscript.

Status

Hugging Face Release Candidate — benchmark schema 1.0

Current validated baseline: Pythia-410M → Pythia-2.8B on one NVIDIA GB10.

The next objective is independent replication and scaling on larger accelerators, larger open models, checkpoint families, and multi-GPU systems.