HyperS-10M-Base-v1.1b-100M
HyperS-10M-Base-v1.1b-100M is a 9,976,787-parameter experimental causal language model built around HyperSpectrum Geometry (HSG) and a recurrent geometric state rather than Transformer self-attention.
This is a base language model. It is not instruction-tuned, not a chat model, and should be treated as a research checkpoint.
Model details
- Developed by: Sayak Mondal
- Model: HyperS-10M-Base-v1.1b-100M
- Model type: Decoder-only causal recurrent language model
- Architecture: HyperS-v1.1b
- Transformer architecture: No
- Parameters: 9,976,787
- Vocabulary: 8,192
- Tokenizer: HGT-8K
- Training exposure: 100,001,792 tokens
- Training chunk length: 64
- Training precision: BF16
- Instruction tuned: No
- Chat tuned: No
- License: Not specified
Architecture
HyperS-v1.1b uses a recurrent Parallel Adaptive HyperState (PAHS) mechanism. Each layer maintains four persistent memory tracks and uses local hyperspherical geometry, adaptive metric structure, and recurrent state transitions.
Core configuration:
| Component | Value |
|---|---|
| Vocabulary | 8,192 |
| Hidden dimension | 384 |
| Recurrent layers | 5 |
| Memory tracks/layer | 4 |
| HSG lanes | 8 |
| Lane dimension | 48 |
| Lane metric rank | 4 |
| Mixer hidden dimension | 896 |
| Decoder rank | 32 |
| Parameters | 9,976,787 |
| Persistent state | 7,680 scalar values / sequence |
| Observed persistent-state dtype | FP32 |
| Persistent-state footprint | ~30 KiB / sequence |
The recurrent transition is an affine state recurrence of the form
[ z_t = \lambda_t z_{t-1} + \beta_t ]
with associative composition
[ (\lambda_b,\beta_b)\star(\lambda_a,\beta_a)
(\lambda_b\lambda_a,\ \beta_b+\lambda_b\beta_a). ]
The model does not use Transformer self-attention or a Transformer KV cache.
Release files
hypers-10m-base-v1.1b-100m.pt
hgt_8k.json
hypers_config.json
release_manifest.json
README.md
The .pt release artifact contains:
- model
state_dict - HyperS configuration
- complete HGT-8K tokenizer configuration
- release metadata
It excludes optimizer state, RNG state, TBPTT runtime state, and training-session state.
This is currently a weights-only research release rather than a native transformers.AutoModelForCausalLM package.
Training data
Pretraining used a prepared token stream derived from:
- TinyStories
- FineWeb-Edu
Users should consult the original dataset cards for source-specific provenance, licenses, filtering procedures, and limitations.
Training protocol
| Hyperparameter | Value |
|---|---|
| Batch size | 32 |
| Sequence length | 64 |
| Tokens/update | 2,048 |
| Peak learning rate | 3e-4 |
| Minimum LR ratio | 0.1 |
| Weight decay | 0.1 |
| Adam β1 | 0.9 |
| Adam β2 | 0.95 |
| Gradient clipping | 1.0 |
| Precision | BF16 |
| LR schedule horizon | 200M tokens |
| Warmup ratio | 1% |
| Final exposed tokens | 100,001,792 |
Persistent HyperS state is carried across training chunks while gradients are truncated at chunk boundaries.
Evaluation summary
The results below characterize this specific 10M-parameter, 100M-token checkpoint. They are not claims about all recurrent models or all Transformer architectures.
Matched stateless language modeling
Both HyperS-v1.1b and the Transformer-T64 control contain approximately 10M parameters and were trained for exactly 100,001,792 tokens.
| Validation set | HyperS CE | Transformer CE | HyperS − Transformer |
|---|---|---|---|
| General | 3.618102 | 2.965312 | +0.652790 |
| TinyStories | 3.497475 | 2.547682 | +0.949793 |
The matched Transformer is substantially stronger in cold-start short-context next-token likelihood.
Natural-language history utility
FineWeb-Edu paired fixed-future evaluation:
| History tokens | Future CE | Gain vs zero history |
|---|---|---|
| 0 | 3.739575 | 0 |
| 16 | 3.286760 | +0.452815 |
| 64 | 3.214953 | +0.524622 |
| 128 | 3.187967 | +0.551608 |
| 256 | 3.138738 | +0.600837 |
| 512 | 3.109061 | +0.630513 |
| 1024 | 3.107112 | +0.632463 |
At 1024 tokens of history, 98.2% of paired examples improved relative to zero-history evaluation.
TinyStories:
| History tokens | Future CE | Gain vs zero history |
|---|---|---|
| 0 | 3.636190 | 0 |
| 16 | 2.862471 | +0.773718 |
| 32 | 2.661220 | +0.974969 |
| 64 | 2.570816 | +1.065374 |
| 128 | 2.478832 | +1.157357 |
These tests demonstrate predictive history utility, not exact retrieval.
History specificity
A separate FineWeb-Edu control replaced older document history with equally long unrelated natural-language history while holding the recent 16 tokens fixed.
At a total history length of 1024:
- correct-history CE: 3.128598
- unrelated-history CE: 3.255597
- specificity gain: +0.126999
- 95% CI: [+0.113681, +0.140318]
This indicates that document-specific older content contributes useful predictive information beyond generic recurrent-state warm-up.
Matched-history comparison
Within the Transformer's T64 training range:
FineWeb-Edu
| History | HyperS CE | Transformer CE | HyperS gain | Transformer gain |
|---|---|---|---|---|
| 0 | 3.728206 | 3.305366 | 0 | 0 |
| 16 | 3.276057 | 3.051597 | +0.452149 | +0.253768 |
| 32 | 3.240257 | 3.014561 | +0.487949 | +0.290805 |
| 48 | 3.216034 | 2.994692 | +0.512172 | +0.310674 |
TinyStories
| History | HyperS CE | Transformer CE | HyperS gain | Transformer gain |
|---|---|---|---|---|
| 0 | 3.634787 | 2.653342 | 0 | 0 |
| 16 | 2.846372 | 2.244345 | +0.788416 | +0.408997 |
| 32 | 2.653125 | 2.181284 | +0.981662 | +0.472058 |
| 48 | 2.584847 | 2.144573 | +1.049940 | +0.508769 |
Within the shared training range, the Transformer retains better absolute CE while HyperS receives greater incremental predictive benefit from increasing history.
Long-history extrapolation
When both models are given the same full histories beyond the Transformer's T64 training sequence length, the comparison becomes an extrapolation test rather than a strict training-range benchmark.
FineWeb-Edu:
| History | HyperS CE | Transformer CE |
|---|---|---|
| 64 | 3.202854 | 3.169224 |
| 128 | 3.176376 | 4.948456 |
| 256 | 3.130744 | 5.101801 |
| 512 | 3.102784 | 5.400285 |
| 1024 | 3.101588 | 5.750396 |
HyperS remains stable and continues extracting useful predictive information from history substantially beyond its 64-token training chunks. The matched T64 Transformer control does not extrapolate reliably when its full attention sequence is extended far beyond T64.
This result should not be interpreted as a claim that Transformers generally fail at long context.
Persistent-state scaling
Observed HyperS retained state remains:
[ 7,680\text{ FP32 values} \approx 30\text{ KiB / sequence} ]
across tested processed-history lengths from 16 through 2048 tokens.
For comparison, an analytical standard BF16 KV cache for the matched 5-layer, 384-dimensional Transformer architecture grows linearly with retained context:
| Context | Transformer BF16 KV | KV / HyperS retained state |
|---|---|---|
| 16 | 0.117 MiB | 4× |
| 64 | 0.469 MiB | 16× |
| 256 | 1.875 MiB | 64× |
| 512 | 3.750 MiB | 128× |
| 1024 | 7.500 MiB | 256× |
| 2048 | 15.000 MiB | 512× |
These figures concern retained context state, not total peak device memory.
Generation
Matched 100M-vs-100M generation suite, 14 prompts, temperature 0.8, top-k 50, seed 42:
| Metric | HyperS | Transformer-T64 |
|---|---|---|
| UTF-8 valid | 14/14 | 14/14 |
| EOS | 1/14 | 0/14 |
| Mean bigram repetition | 0.1771 | 0.2150 |
| Mean trigram repetition | 0.0686 | 0.1071 |
| Mean distinct-token ratio | 0.5012 | 0.4676 |
HyperS produced lower repetition and greater token diversity on this small suite. These are surface-generation metrics and do not establish superior semantic generation quality.
Inference throughput
On the tested RTX 4070 Laptop GPU implementation, the Transformer control is substantially faster for token-by-token decoding. HyperS should therefore be described as demonstrating fixed-state memory behavior, not runtime-speed superiority.
What the results support
The current evidence supports the following:
- HyperS-v1.1b is a non-Transformer recurrent causal language model.
- Its persistent inference state is fixed-size with respect to processed history length.
- It demonstrates statistically significant predictive utility from natural-language history.
- Correct older document history is more useful than unrelated matched-length history.
- Predictive history utility remains measurable through at least 1024 tested FineWeb-Edu history tokens.
- HyperS generalizes its recurrent state mechanism beyond the 64-token chunks used during training.
- Within the shared T64 range, HyperS extracts greater incremental predictive value from additional history than the matched Transformer control.
- The matched Transformer has substantially better absolute short-context likelihood.
- The current HyperS implementation is substantially slower for autoregressive decoding.
What the results do not establish
This release does not demonstrate that:
- HyperS is generally superior to Transformers.
- HyperS has perfect recall of 1024 tokens.
- HyperS performs exact long-range retrieval.
- HyperS solves long-context reasoning.
- HyperS is instruction-following or chat-aligned.
- HyperS is faster than the Transformer control.
- 10M HyperS is equivalent to a substantially larger Transformer.
- lower repetition implies semantically better generation.
A synthetic digit-recall task remained near chance and therefore exact associative retrieval is not demonstrated.
Intended use
This checkpoint is intended for research into:
- non-Transformer causal language modeling
- recurrent representation learning
- hyperspherical and geometry-driven sequence models
- fixed-size history representations
- stateful language-model evaluation
- continuation and long-history experiments
- downstream fine-tuning research
Limitations
The model is small and only trained for approximately 100M tokens. It may produce incoherent, repetitive, inaccurate, biased, or irrelevant continuations. It is not safety-aligned or suitable for consequential applications.
Use it as an experimental research artifact rather than a production assistant.
Related research
HyperS is motivated by the HyperSpectrum Geometry framework:
Sayak Mondal, "HyperSpectrum geometry: a Riemannian hyperspherical framework for curvature-adaptive text classification," Neural Computing and Applications (2026).
The published HSG paper is a conceptual precursor. It does not describe this language-model architecture or the v1.1b checkpoint.