snchimata/tokenfold-select-v2

Tokenfold Select v2

Build smaller contexts. Keep the original words.

Query-aware context selection in a compact, 98.9M-parameter model: ONNX by default, Safetensors for PyTorch workflows.

LLM workflows collect more context than every request needs. Tokenfold Select v2 helps your application choose what to keep: it scores text groups together with the query, then hands those scores to an allocator that assembles a smaller context from the original text. No generated summaries or paraphrases.

Experimental ONNX + Safetensors preview · English · Local inference

Why Tokenfold Select v2?

• Score groups together. One joint encoder pass scores all groups in an admitted single-window request, rather than encoding each query/span pair separately.
• Keep the source intact where it counts. The extractive workflow retains selected original groups; your allocator controls required content, source order and the final token budget.
• Use a compact specialist. A trained 12-layer encoder with a shared group scoring head—not a generative LLM or the v1 LoRA adapter.
• Run locally. Offline-capable inference with caller-provisioned weights and explicit precision settings. No hosted model service is required.
• Inspect every selection. Ordered group scores give your application a clear interface for ranking, allocation and debugging.

Query + original text groups
             ↓
     Select v2 group scores
             ↓
  Allocator + required-content rules + token budget
             ↓
   Selected context, in original wording

For developers exploring context selection in retrieval, tool/API responses, coding assistants and agent histories. The model returns scores, not a compressed document; grouping and budget enforcement remain in your application.

Tokenfold Select v2: reference performance

Using the PyTorch/CUDA FP16 reference runtime (not ONNX) on the JSON development fixture at 50% retention, Tokenfold Select v2 recorded 150 ms warm p50 compression latency, versus 208 ms for Select v1—about 28% lower.

The table shows p50 / p95 milliseconds at the same retention target:

Development fixture	Tokenfold Select v2 FP16	Tokenfold Select v1 FP32	Headroom Kompress v2 FP32
JSON	150.199 / 161.996	207.644 / 215.143	Budget refusal: 0/6 completions
Code	268.713 / 282.441	451.916 / 466.936	Budget refusal: 0/6 completions
Logs	1842.744 / 1862.359	3760.490 / 3788.029	Budget refusal: 0/6 completions
Multi-turn	267.051 / 271.358	563.040 / 568.882	94.577 / 97.534
Retrieval	773.447 / 803.698	2046.032 / 2070.185	2733.641 / 2751.180

Scope matters: one exposed source per family, six repeated requests per arm and budget, on one local RTX 5080 using the PyTorch/CUDA reference runtime, not ONNX. These are empirical warm development results, not independent-source confidence bounds or proof of downstream answer quality. Kompress is faster on multi-turn where both complete; its budget refusals are failures, not evidence of a successful-compression speedup for Select v2.

The complete comparison covered 180/180 requests across two retention targets: 132 valid compressions and 48 healthy Kompress budget refusals, with no omissions. Refusals retain infinite completion latency in the failure-aware summaries. The clock includes fresh grouping, tokenization, scoring, native allocation/IPC and output recount; it excludes loading, warmup and file I/O. This measures the standalone Kompress compressor, not the full Headroom pipeline.

See all workload/budget results and methodology.

Run Tokenfold Select v2 with ONNX

The staged model is a FP32 ONNX graph plus its external weights: model.onnx and model.onnx.data. Keep both files together. No pickle-based .pt checkpoint is included in this upload candidate. The optional Safetensors file is separate; ONNX inference does not require PyTorch.

Component	Current scope
Model	98,864,641 parameters, 12 encoder layers, ONNX opset 18
Graph IO	int64 input_ids / attention_mask; FP32 token_scores, shape [1, tokens]
Admission	One complete window; at most 8,192 encoded tokens and 512 groups
Runtime	ONNX Runtime, explicit CPUExecutionProvider; no implicit provider fallback
Software	Python 3.12 and the pinned ONNX environment

The ONNX graph and reference adapter passed 6/6 synthetic CPU parity checks, including two real packaged-loader requests and dynamic sequence lengths up to 2,048 tokens / 512 pooled group ranges. Maximum absolute score error was 0.0000878, below the predeclared 0.0001 tolerance; ranking agreement was approximately 1.0. CUDA remained uninitialized throughout this independent pass.

These checks validate conversion and loader mechanics on Windows/Python 3.12, not downstream task quality or ONNX speed. Full 8,192-token execution and representative capacity/performance remain untested. ONNX CPU validation results.

Download this repository at a pinned full Hub commit into ./tokenfold-select-v2. With the versions above provisioned, the reference API is:

from pathlib import Path
import sys

root = Path("./tokenfold-select-v2").resolve()
sys.path.insert(0, str(root / "runtime"))
from onnx_score import OnnxScorer

scorer = OnnxScorer(
    root,
    "267313718e80aa91a56923dea30363fd5c24fde3782c716d6d048891860a2054",
    development=True,
)
response = scorer.score({
    "schema_version": 1,
    "model_revision": "4dbdb5071e7d6316d8eb70dc161b1f2f77f555012f6616d7fa34311188e24cec",
    "query": "Where is the order being delivered?",
    "groups": [
        {"id": "delivery", "text": "Delivery city: Paris\n"},
        {"id": "metadata", "text": "Record created on Monday\n"},
    ],
})
print(response)  # ordered group IDs and scores, not compressed text

The adapter verifies the pinned ONNX manifest, packs complete original groups with the trained tokenizer, and averages content-token scores back into group scores. Query/separator tokens are not pooled. Protect loader and model files and bound the calling process's lifetime; this is not an OS sandbox. This is not the v1 PEFT adapter or a stock Transformers classification model.

Run Tokenfold Select v2 with Safetensors

Prefer PyTorch tooling? The same trained FP32 parameters are also staged as model.safetensors: 76 tensors, 98,864,641 values, without a pickle checkpoint. ONNX remains the default for the approved Tokenfold CLI worker. This alternative is a separately provisioned model/reference API, not an automatic CLI backend.

This is a custom joint-group encoder and scalar head, not a v1 LoRA adapter or stock AutoModelForSequenceClassification. Use the included runtime/safetensors_score.py with runtime/joint_group_model.py, the tokenizer, configuration and manifest. The original source configuration is retained as source-config.json; distribution config.json identifies the custom model. The canonical architecture file retains its historical prototype docstring; the loader strictly replaces every parameter with the trained weights.

The Safetensors manifest pins all computational files. The CPU reference uses the separately provisioned pinned PyTorch environment, including a CUDA-enabled Torch build executed strictly on CPU. Other device/build profiles are not validated by this reference and must not inherit its evidence.

# Run in a fresh process with caller-owned deadlines; set before importing Torch.
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
from pathlib import Path
import sys

root = Path("./tokenfold-select-v2").resolve()
sys.path.insert(0, str(root / "runtime"))
from safetensors_score import SafetensorsScorer

scorer = SafetensorsScorer(
    root,
    "bd335629747ba2b437fbd528d9b5b0413523f94f82e7b4b21f4638f2580b9f4e",
    development=True,
)
response = scorer.score({
    "schema_version": 1,
    "model_revision": "4dbdb5071e7d6316d8eb70dc161b1f2f77f555012f6616d7fa34311188e24cec",
    "query": "Where is the order being delivered?",
    "groups": [{"id": "delivery", "text": "Delivery city: Paris\n"}],
})
print(response)  # ordered group scores; allocation remains with your controller

Validated without reconversion: an independent CPU-only pass confirmed all 76 FP32 tensors / 98,864,641 values match the source checkpoint bit-for-bit. Both synthetic requests (1 and 8 groups; 20 and 125 tokens) passed the actual Safetensors loader with zero score difference, identical rankings and original packing. CUDA remained unavailable and uninitialized. Both owned processes exited cleanly and were waited for.

Safetensors CPU validation results. The earlier conversion run's terminal-report failure remains preserved in the historical conversion audit; it has not been relabeled successful. These are format/loader mechanics checks, not representative quality, latency or full-8k execution qualification.

What to know before integrating

• Selection is not a correctness guarantee. Scores are rankings, not probabilities. Enforce required content, redaction, source order, exact output budgets and deterministic fallback in a trusted controller. A fallback is not a successful model-backed result.
• Bounded means bounded. The ONNX reference adapter refuses oversized or multi-window inputs before loading the model; it does not truncate or split groups. Research multi-window support is not packaged long-context support. One encoder pass does not mean constant total request work.
• Evidence is still developmental. Training and evaluation are small and English-centric. Untouched representative task-quality, cold-latency and production qualification remain open. Neither universal superiority nor downstream quality or cost savings is claimed.

Keep this model opt-in while evaluating it on your own workload. Do not rely on unqualified compressed context as the sole basis for high-impact decisions.

Tokenfold Select v2: training, provenance and license

Built from IBM Granite reranker English R2, retaining original encoder layers 0–11 and adding group pooling/scoring. Training used 143 TRAIN rows from USGS JSON and CPython code, TRAIN-only head warmup, then three joint FP32 epochs with original-foundation feature distillation. The 87 exposed DEV rows influenced model selection; they are not untouched held-out data.

Credit: U.S. Geological Survey earthquake event information and CPython source contributors. No training corpus or evaluation cache is included. Both the ONNX and Safetensors FP32 weights derive from the pinned PyTorch reference; the tokenizer is unchanged. The FP16 timings above describe the earlier PyTorch runtime cast, not the ONNX or Safetensors CPU reference loaders.

Training recipe · Provenance · Change notice and license scope · Integrity manifest

Release status: ONNX and Safetensors format/loader mechanics checks have passed. This is an opt-in experimental preview, not a quality/performance-qualified GA model. Distributed under Apache-2.0, with maintainer approval of the recorded model/tokenizer terms and source applicability. That disposition is not a blanket license for unrelated source material or an upstream endorsement. Upstream and CPython license texts are retained under licenses/. Current distribution status.