reign-small-l2_gn-gte-large_val-selected

REIGN small-l2 cross-chunk encoder trained on GoodWiki-Long-Synthetic over a frozen GTE-large guidance network — a long-document bi-encoder that reads a sequence of cached chunk embeddings instead of tokens.

Configuration

REIGN encoder	small-l2 — 2 layers, d = 384, 6 heads, FFN 1536, 3.85M trainable parameters
Guidance network (frozen)	thenlper/gte-large — GTE-large, 335M
Chunk size K	512 (the guidance network's context window)
Training stride S	512 — released-recipe default S = K; neither the name nor config.json records a stride for this run
Chunk-position signal	none — the encoder is a permutation-equivariant set function
Pooling	mean over the chunk sequence
Training data	devrim/goodwiki_long_synthetic_ir
Training recipe	released cosine recipe (below)
Checkpoint selection	best validation nDCG@10 on the val qrels split

Reported results

Every number below is the value the paper reports for this exact checkpoint; nothing is re-derived here.

Benchmark	Metric	Eval stride	Value	Paper
LoCo (macro-avg, zero-shot)	nDCG@10	s384	70.97	Table 8
LoCo (macro-avg, zero-shot)	nDCG@10	s512	70.42	Table 8
DAPFAM test (zero-shot)	nDCG@100	s384	32.55	Table 8
DAPFAM test (zero-shot)	nDCG@100	s512	32.64	Table 8

The Table 7 in-distribution GoodWiki-Long cell for small-l2 on GTE-large (67.27) was produced by the stride-tagged sibling reign-small-l2_gn-gte-large_st-512_val-selected, not by this checkpoint.

Usage

The checkpoint holds only the REIGN cross-chunk encoder. The guidance network is loaded separately and stays frozen, so both must be named at construction time.

pip install git+https://github.com/devrimcavusoglu/reign.git

import numpy as np
from huggingface_hub import snapshot_download
from reign.encoders.reign import ReignBaselineEncoder

checkpoint_path = snapshot_download("devrim/reign-small-l2_gn-gte-large_val-selected")

encoder = ReignBaselineEncoder(
    checkpoint_path=checkpoint_path,
    gn_model="thenlper/gte-large",
    chunk_size=512,
    stride=512,
)
docs = [open("doc_a.txt").read(), open("doc_b.txt").read()]
emb = encoder.encode(docs, batch_size=8)   # (2, hidden_size), L2-normalised
print(float(np.dot(emb[0], emb[1])))       # cosine similarity

ReignBaselineEncoder returns L2-normalised vectors, so the cosine is a dot product. chunk_size is the guidance network's sliding-window size — 512 for every released checkpoint, matching its context window — and stride controls the overlap, with stride == chunk_size giving non-overlapping chunking. The evaluation-time stride is a runtime argument, and the paper's headline tables report the best-performing stride per guidance network.

For the lower-level surface, ReignModel (a PreTrainedModel consuming inputs_embeds) and ReignFeatureExtractor (the guidance-network wrapper, with the on-disk embedding cache) are importable from reign and reign.feature_extractor.

Operating regime

REIGN targets multi-chunk inputs and primarily document-to-document retrieval. Inputs shorter than the chunk size collapse to a single chunk embedding, leaving the cross-chunk encoder nothing to aggregate — that regime is served by the guidance network alone, and this checkpoint should not be used for it.

Training recipe

The released checkpoints use a three-way cosine embedding loss over document pairs carrying graded targets s ∈ {1, 0, −1} — positive, partial, negative — with partial weight λ = 0.5.

Setting	Value
Objective	three-way cosine embedding loss, λ = 0.5
Batch construction	18 anchors × (1 positive + 2 partials + 17 in-batch negatives) = 360 pairs / step
Optimiser	AdamW, lr 1e-5, weight decay 1e-4, cosine annealing
Epochs	50, validating every 4
Selection	best validation nDCG@10
Precision	16-mixed
Seed	42
Guidance-network embeddings	precomputed and cached
Hardware	single 24 GB consumer GPU

A second, separate protocol (InfoNCE at τ = 0.07, batch 48, 20 epochs) is used only for the paper's positional-encoding and training-objective ablations. Those arms sit below this released operating point by construction and are not comparable to it. See docs/TRAINING.md in the code repository.

Because 16-mixed training is not bit-reproducible even at a fixed seed, a retrained checkpoint will not match these weights bit-for-bit; compare metrics, not weights.

Files

• config.json — ReignModel configuration
• model.safetensors — encoder weights (float32)

Links

• Code: https://github.com/devrimcavusoglu/reign
• Project page: https://devrimcavusoglu.github.io/reign
• Dataset: https://huggingface.co/datasets/devrim/goodwiki_long_synthetic_ir
• Paper: REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling, Findings of the Association for Computational Linguistics: EMNLP 2026 (to appear).

Citation

@inproceedings{cavusoglu2026reign,
  title     = {{REIGN}: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling},
  author    = {{\c{C}}avu{\c{s}}o{\u{g}}lu, Devrim and Akba{\c{s}}, Emre},
  booktitle = {Findings of the Association for Computational Linguistics: {EMNLP} 2026},
  year      = {2026},
  publisher = {Association for Computational Linguistics},
  note      = {To appear}
}

License

Apache License 2.0. The devrim/goodwiki_long_synthetic_ir dataset is released under CC BY-SA 4.0, preserving the share-alike licensing and attribution of GoodWiki and the underlying Wikipedia text.