snchimata/tokenfold-select-v3

Tokenfold Select v3

Joint group-level context scorer for RAG, coding agents, and tool pipelines.

Choose the context. Keep the words. Own the pipeline. Select v3 is a 149.6M-parameter encoder (Apache-2.0, local CPU/CUDA inference) that jointly scores text groups against your query and their neighbors—giving your application a learned selection layer without subword pruning or generative rewriting.

How it fits your pipeline

[ Query ] + [ Group A: Code ] [ Group B: Logs ] [ Group C: Docs ]
                               |
            Select v3: joint 22-layer encoder within each window
            Scores each group with the query and its neighbors
                               |
                 Aligned scores: [score_A, score_B, score_C]
                               |
          Your allocator: protected rules + exact token cap
                               |
          Selected context: original bytes of A + C (example)

Use Select v3 as the learned selection layer, while keeping application policy explicit. The model returns group relevance scores, not a ready-made compressed string or calibrated probabilities.

Put it to work

• RAG: rank retrieved passages against the active question before allocating context.
• Coding agents: score whole source groups while your application protects signatures, instructions and required blocks.
• Tool-heavy workflows: select among tool-output groups without adding a generative rewriting step.
• Conversation context: experiment with query-aware turn/group selection while enforcing mandatory history yourself.

Quickstart: score your own groups

Use Python 3.12 with the pinned versions in requirements.txt for CUDA, or replace its Torch entry with torch==2.11.0 / torch==2.11.0+cpu for a CPU-only build; keep the other versions unchanged. CPU inference also works with the pinned CUDA-enabled Torch build. The SDK does not install the ML stack implicitly. CUDA requires a BF16-capable GPU; CPU runs in FP32 without a GPU. Neither mode automatically falls back to another device or precision.

hf download snchimata/tokenfold-select-v3 --local-dir ./tokenfold-select-v3
python -m pip install --no-deps ./tokenfold-select-v3/loader

Then score your own groups in a dedicated process. The two hashes pin the local bundle and checkpoint for integrity verification:

import os
from pathlib import Path

# 1. Choose the device and configure the dedicated offline process.
device = "cuda:0"  # Use "cpu" for FP32 inference without a GPU.
os.environ.update(HF_HUB_OFFLINE="1", TRANSFORMERS_OFFLINE="1")
if device == "cuda:0":
    os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import torch
torch.set_autocast_cache_enabled(False)
from tokenfold_select_v3.ettin import EttinDevelopmentScorer

# 2. Load the integrity-pinned local model.
bundle = Path("./tokenfold-select-v3").resolve()
scorer = EttinDevelopmentScorer(
    str(bundle),
    "03f20f46defab861c35b4684d8dc28e54e0f2171212f0decde0a214dc8dcc18b",
    development=True,
    device=device,
    precision="fp32" if device == "cpu" else "fp32-parameters-bf16-autocast",
)

# 3. Score your query and groups; your allocator makes the selection.
response = scorer.score(
    {
        "schema_version": 1,
        "model_revision": "fd8231d3cdc1efc46f7a6d7d83ca88b336a953591d63ea431407c8b13b990309",
        "query": "Which group explains cache invalidation?",
        "groups": [
            {"id": "cache", "text": "Invalidate an entry when its source changes."},
            {"id": "logs", "text": "Write diagnostic events to the application log."},
        ],
    }
)
print(response["scores"])
# Example shape (illustrative): [['cache', 1.42], ['logs', -0.85]]
# Scores are unnormalized floats: higher = higher predicted relevance, not probabilities.

This is a custom trained joint-group architecture. Use the included loader rather than plain AutoModel.from_pretrained(). Keep external process/resource limits and your existing redaction, required-content and final-recount safeguards.

Why group-level selection?

Choose where selection happens and which responsibilities stay in your application:

Approach	Retained source text	Code / JSON handling	Execution pattern	Token-budget control
Generative summarization	Rewrites content	Validate generated structure and meaning	Autoregressive generation	Generation limits plus output validation
Token-level pruning	Removes tokens within text	Structure-aware guards or fallback	Token scoring plus pruning	Pruning policy plus exact recount
Select v3 whole-group selection	Your allocator can copy chosen groups verbatim	You define boundaries and validate the selected combination	Joint 22-layer encoder plus allocator	Your allocator, protected rules and exact recount

Group boundaries give you control over selection; they do not by themselves guarantee valid combined syntax or answer preservation.

• Context-aware, not isolated. Within each admitted window, the query and neighboring groups share one encoder input. A group is scored in its surrounding context, not only as an independent query/span pair.
• Purpose-trained from backbone to readout. A complete 22-layer Ettin encoder plus a trained nonlinear joint-group head. Not an untouched base reranker with a new label.
• Selection without rewriting. Your allocator can keep the original bytes of chosen groups. Keep code blocks, passages and tool messages as source text rather than generated paraphrases; enforce required content separately.
• Your budget. Your rules. The model supplies relevance scores; your application owns protected groups, ordering, exact token accounting and final budget enforcement.
• Bring the model to your data. After acquiring the bundle and dependencies, run inference locally with explicit offline loading. No hosted LLM, per-request scoring API or separate base-weight download is needed.
• CPU or CUDA, encoder-only. Full FP32 checkpoint with explicit CPU/FP32 or CUDA/BF16 compute and no autoregressive decoding. Both paths were locally functionally smoke-tested; CUDA used an NVIDIA GeForce RTX 5080. No CPU speed claim is made.
• Everything in the open. Checkpoint, tokenizer, config, loader source, training recipe, provenance and license are included. Inspect it, adapt it and evaluate it on your workload.

Benchmarks

Measured context-token reduction across five development workloads—code, JSON, logs, multi-turn chat and retrieval—alongside complete-request timings for budget-eligible comparisons. A separate synthetic publisher-data pilot checks literal-substring retention. See benchmark methodology and full history for test conditions and scope.

Context-token reduction at a 50% budget

Select v3 removed 50.0–60.0% of the original context tokens across these five saved development examples. Both models used the same exact output cap; original and selected sizes are shown explicitly.

Workload	Original tokens	Select v3 output	V3 reduction	Kompress v2 output	Kompress reduction
Code	4,411	2,205	50.0%	2,149	51.3%
JSON	1,303	651	50.0%	Refused; 1,303 raw	—
Logs	1,103	503	54.4%	Refused; 1,103 raw	—
Multi-turn	587	235	60.0%	284	51.6%
Retrieval	4,498	2,217	50.7%	2,219	50.7%

Whole-group allocation met these exact o200k_base caps without rewriting selected group text, including the JSON and log examples. One exposed development source per workload; comparator refusals remain inconclusive.

Complete-request latency at a 50% budget

A three-workload snapshot: historical code and retrieval observations alongside the fresh five-request multi-turn sample medians. All shown outputs met the same per-workload cap.

Workload	Select v3 ms	Kompress v2 ms	V3 time difference	Measurement basis
Code	146.088	200.304	-27.1%	Single DEV request per model
Retrieval	139.291	201.564	-30.9%	Single DEV request per model
Multi-turn	66.572	77.751	-14.4%	Median of 5 fresh requests per model

Resident research pipelines on an RTX 5080; startup and conditioning excluded. Single observations and sample medians are distinct statistics. The earlier, slower multi-turn observation and every fresh trial remain below. See benchmark methodology for run conditions, including background GPU activity.

Shared publisher-data pilot: original inputs, smaller context

On five preselected synthetic cases from Headroom Pilot, Select v3 met the 50% tool-context budget in 5/5 cases, with 50.0–54.8% fewer tokens. All 12 requested literal substrings remained present somewhere in the selected context.

Pilot category	Original tokens	Select v3 output	Reduction	Requested substrings present
Agentic loop	41,097	20,433	50.3%	3/3
Code agent	7,141	3,568	50.0%	2/2
JSON data	11,345	5,637	50.3%	3/3
Logs	39,682	19,751	50.2%	2/2
RAG search	6,435	2,908	54.8%	2/2

This is a V3-only token-size and literal-substring screen, not answer accuracy. Head-to-head and downstream-answer evaluation remain incomplete; execution details are below.

Benchmark methodology, full comparison and execution status

What was measured. Historical token counts come from the scorer-plus-whole-group allocator, not the scoring model alone. Actual reduction is 1 - output_tokens / original_tokens. The historical screen has one exposed source per workload at two budgets; JSON and logs share a source population. It is not an untouched representative qualification set.

Latency scope. Resident-warm complete requests on an RTX 5080, FP32 parameters/BF16 compute, cache off, TF32 off and one Torch thread. V3 used a serialized SDPA research runner; Kompress v2 used native EAGER through its API and bounded token-budget adapter, with must-keep/tabular guards on and CCR off. Fresh grouping, counting, IPC, scoring, allocation/recount and applicable relay/guardian teardown are included; process/model startup, conditioning and controller-tokenizer initialization are excluded. Recordings came from different time blocks and are not a causal comparison or a latency promise for the quickstart SDK. No population p50/p95 is established.

All ten historical cells, including refusals and the slower eligible result:

Workload	Retention budget	Original tokens	V3 output tokens	V3 reduction	Kompress v2 ms	Select v3 ms	Recorded DEV comparison
Code	25%	4,411	1,079	75.5%	Budget refusal	158.919	Inconclusive
Code	50%	4,411	2,205	50.0%	200.304	146.088	27.1% lower time
JSON	25%	1,303	308	76.4%	Budget refusal	62.861	Inconclusive
JSON	50%	1,303	651	50.0%	Budget refusal	65.896	Inconclusive
Logs	25%	1,103	275	75.1%	Budget refusal	58.919	Inconclusive
Logs	50%	1,103	503	54.4%	Budget refusal	57.955	Inconclusive
Multi-turn	25%	587	39	93.4%	90.646	69.653	23.2% lower time
Multi-turn	50%	587	235	60.0%	64.411	69.362	7.7% higher time
Retrieval	25%	4,498	1,115	75.2%	Budget refusal	136.190	Inconclusive
Retrieval	50%	4,498	2,217	50.7%	201.564	139.291	30.9% lower time

Refusals, raw fallback, no-op and over-budget outputs are not V3 wins. No audited superiority, 90% comparability, downstream-quality or cost-saving guarantee is established.

Shared-data pilot. The publisher's 30-row synthetic pilot is CC-BY-4.0, used under an evaluation-only exception—not for training. The five cases are the first ID in each category, selected before outcomes and unchanged afterward. Three recorded tool outputs are flattened in source order; the 50% exact budget applies to this tool context, not the entire API request. This is not the full proxy benchmark. Four cases use expanded V3 group/window geometry. Literal substring presence does not prove question/answer relationships survived; a string can occur in unrelated text.

V3 completed 17 scoring windows and five budget-valid selections. A stale harness check for KompressConfig.batch_size stopped comparator setup before model load; this is a harness failure, not a comparator refusal or loss. No Ollama answers were generated. All ten planned compression cells, fifteen missing answer cells and earlier failed attempts remain recorded. Future comparator/answerer work needs a separate scope and can reuse the five saved V3 outputs.

Fresh fixed multi-turn repeat (2026-11-04). The input, 293-token cap, precision, model backends and complete-request clock match the historical multi-turn setup. The profiles ran serially (Kompress v2, then V3), each with one conditioning call and five timed calls. The primary statistic was fixed in advance as the median of all five measured observations, not a best-case/minimum or a population quantile. Setup and conditioning are excluded; their separate receipts are retained and are not cold-request latency.

Same 587-token conversation and guarded resident research pipeline. All ten timed requests met the cap; V3 returned 235 tokens and Kompress v2 returned 284. Trial numbers identify each model's serial observations, not contemporaneous paired trials:

Timed request	Select v3 ms	Kompress v2 ms
1	73.385	73.125
2	68.246	82.635
3	66.572	76.332
4	62.875	79.600
5	65.881	77.751
Sample median (all 5)	66.572	77.751

Only the 25% GPU-busy admission condition was waived by explicit user instruction. Pre-model GPU-busy observations were 25% for the first profile and 38% for the second; these are not continuous background-load measurements. VRAM/RAM/disk, memory, deadline, runtime-health, exact-budget and numerical-fidelity checks remained in force. All twelve requests including conditioning passed the declared budget/reference checks. No quality or statistically qualified superiority conclusion follows from five repeats of one exposed conversation.

Preserved history. The earlier idle-gated repeat expired before model startup, recording 0/10 timings and no medians. It was not resumed. The earlier historical multi-turn point remains 69.362 ms vs 64.411 ms (7.7% higher V3 time) in the full table above. All attempts, unrounded fresh samples, conditioning/setup receipts, GPU-load observations and model-call counts are retained in COMPARISON.json.

Published token-label metrics, not a head-to-head result. Kompress v2's pinned model card reports precision 0.940, recall 0.901 and F1 0.920 at threshold 0.5 on 12,697 labeled test rows, H100/BF16, with 86.6% subword retention. V3 has not been measured on that labeled split, which was not located in the public publisher files checked. These token-label scores cannot be compared with the pilot's substring check or earlier answer-lookup smoke.

Training and validation scope. Training uses weak support/relevance labels. The final epoch was packaged without validation-score selection; 173 development-validation rows cover code, retrieval and multi-turn, not JSON/logs.

Evidence and reproducibility: VALIDATION.json contains clocks, exact observations and source-receipt hashes; COMPARISON.json contains publisher sources, data licensing, complete cell accounting and preserved failures. The earlier partial answer-lookup smoke is archived separately in QUALITY-EVAL.json. No closed run was resumed or selectively regraded for this README revision.

At a glance

Property	Value
Parameters	149,606,401
Backbone	Ettin reranker 150M, full 22-layer encoder
Readout	Trained joint-group mean pooling + nonlinear head
Saved weights	FP32 Safetensors
Local execution	CPU FP32 or CUDA BF16 autocast; SDPA; CUDA TF32 off
Window budget	7,999 tokens, including query and separators
Training group cap	24 groups
Runtime inference bounds	Up to 512 groups / 64 query-repeated windows; no cross-window attention
Distribution	Complete checkpoint + tokenizer + included loader source
License	Apache-2.0; preserve bundled notices

Trained and inspectable

Trained across code, JSON, logs, retrieval and multi-turn sources: 3 epochs, 1,086 optimizer steps and 2,896 development training examples, using the full 22-layer Ettin encoder and trained joint-group head. Checkpoint, tokenizer, loader source, training recipe and provenance are included.

Local functional checks covered packaged checkpoint loading, CPU/FP32 and CUDA/BF16 execution, finite aligned scores and offline operation. The CPU/CUDA functional validation used one fixed two-group request per mode; CPU stayed in FP32 even inside ambient CPU autocast and did not initialize CUDA. The CPU check used the installed CUDA-enabled Torch build with CUDA disabled, not a fresh CPU-wheel installation. These checks do not establish CPU/CUDA numerical equivalence, maximum-shape behavior, quality or performance. Full recipe and provenance: TRAINING-RECIPE.json, PROVENANCE.json.

Best practices for integration

• Keep policy in your allocator. Scores are not calibrated probabilities or a ready-made compressed string. Enforce required content, source order, redaction and a final exact token recount.
• Choose self-contained groups and validate the result. Copy chosen source text unchanged, but check combined JSON/code syntax and evaluate answer preservation separately.
• Start within training geometry. Use at most 24 groups within a 7,999-token query-plus-groups window. Larger group counts and multi-window inference exceed the training geometry; groups in separate windows do not attend to each other. Oversized whole groups/queries refuse rather than silently truncate.
• Use a provisioned, bounded process. Choose CPU/FP32 (device="cpu", precision="fp32") or CUDA/BF16 (device="cuda:0", precision="fp32-parameters-bf16-autocast") explicitly; CUDA remains the API default. Keep external process/resource limits. INT8 runtimes and fresh-machine installation certification are not included in this release.

License and transparency

The model artifact is Apache-2.0; preserve LICENSE and the original BASE-MODEL-CARD.md. Training inputs follow the project's Apache-only policy; upstream data retains its own terms. No raw training/evaluation dataset is bundled. See PROVENANCE.json and RELEASE_STATUS.json for provenance and final release status. Historical manifest flags, class names and loader opt-ins remain unchanged for compatibility; they do not describe this release's lifecycle.

Build a context pipeline you control: score groups with their neighbors, retain original source text and keep the final decision in your application.