sothiem/laya-kvp10k-noul

Laya KVP-10K — Key/Value Match Detector (noul)

A fine-tuned Laya model (Convai Innovations, 421M, ModernBERT-large backbone) that answers one typed noul question: does a value correctly match its key label? (e.g. first_name = John → true, first_name = 1992 → false).

Fine-tuned on the IBM KVP-10K dataset via the pre-parsed community mirror alessandrorusso21/KVP10k (OCR pre-extracted; no OCR step needed).

⚠️ This is not a generative LLM

Laya is a non-autoregressive decision model: you give it a state (text/dict) plus typed questions (choice, noul, score) and it returns calibrated probabilities in a single forward pass. It never generates text. Do not use chat templates, generate(), or causal-LM losses with this checkpoint. It is loaded with laya.Agent(<dir>), not transformers.

Model details

Base model	convaiinnovations/laya @ aa8c91c (421M English checkpoint)
Encoder	answerdotai/ModernBERT-large
Fine-tuning recipe	Official RLCD (REINFORCE with strictly proper scoring rules + GRPO-style group-mean baseline) — not LoRA/peft
laya package version	v0.3.11
Question type	noul (key_value_match) — options render in fixed [false, true] order; p[1] is P(true)
Context budget	max_len=512, head_max_len=192 (≈320 tokens for the state)
Temperature	1.286 (post-training noul refit on held-out val)
Parameters	421.3M
Precision	float32 master weights, bf16 AMP inference

Question definition

type: noul
id: key_value_match
instructions: "The value correctly matches the key label."
criteria:
  "false": "the value does not match the key"
  "true": "the value is consistent with the key"

State format

state = {
    "key": "first_name",
    "value": "1992",
    "document_excerpt": "<token-budgeted window around the value span>",
}

Usage

from laya import Agent

agent = Agent("<this-repo-dir-or-hf-id>")

state = {
    "key": "first_name",
    "value": "1992",
    "document_excerpt": "Name: John Smith  DOB: 1992-04-01  ...",
}
question = {
    "type": "noul",
    "id": "key_value_match",
    "instructions": "The value correctly matches the key label.",
    "criteria": {
        "false": "the value does not match the key",
        "true": "the value is consistent with the key",
    },
}
p = agent.ask(state, [question])[0]   # [P(false), P(true)]
print(p[1])                           # P(match)

Noul options render in fixed [false, true] order and must never be shuffled — p[1] is P(true).

Performance

Evaluated on the held-out val split (11,322 items; 90/10 split of the HF train split, seed 42) after the noul temperature refit:

Metric	Value
Accuracy	97.32%
Soft accuracy	94.97%
Recall (match / P(true))	97.37%
Recall (mismatch / P(false))	97.28%
Bias gap (	recall₁ − recall₀
Brier score	0.0252
ECE	0.0005
KL / TV	0.119 / 0.050
Latency p50 / p95 (RTX 5090)	21.1 ms / 48.4 ms

Per source:

Source	n	Accuracy	Brier
gold (positives)	5,664	97.37%	0.0254
cross-doc negatives	2,960	99.43%	0.0047
same-doc negatives	2,698	94.92%	0.0474

Training curve (10 epochs, val accuracy 92.97% → 97.31%):

The base English checkpoint's known noul label-bias (laya issue #156) was mitigated with strict 50/50 class balance via seeded synthetic negatives; the resulting per-class recall gap is only 0.09 percentage points. The post-training temperature refit (T = 1.286) reduced ECE from 0.0167 to 0.0005 with no accuracy loss.

Training data

• Positives: gold (key, value) pairs from KVP-10K gts/<hash>.json (kvps_list).
• Negatives (synthetic, seeded, never a gold pair): same-document type mismatches and cross-document mismatches, 1 per positive for ~50/50 class balance.
• Document excerpts: token-budgeted window (240 tokens) around the value span, built from the pre-parsed OCR words in ocrs/<hash>.json.
• Splits: 103,322 train / 11,322 val from the HF train split. The HF test/ split was held out and never used for training or hyperparameter selection.

Training procedure

Single NVIDIA RTX 5090 (32 GB), bf16 AMP, 10 epochs:

Parameter	Value
Effective batch size	32 (micro 16 × grad-accum 2)
Group size	4 (REINFORCE noisy-logit samples)
LR (encoder / head)	5.0e-5 / 2.0e-4, cosine → 1e-6
Exploration noise σ	0.4 → 0.1 (linear anneal)
Baseline	group-mean (std-normalized advantage)
Reward	log-score 1.0 + spherical 0.75 (strictly proper)
CE co-training weight	1.0
Grad clip	1.0
Target smoothing	0.01

Limitations

• English documents only (512-token budget). For non-English use convaiinnovations/laya-multilingual (1024-token budget) and re-fine-tune.
• Same-document negatives are the hardest class (94.92% val accuracy); expect lower precision on near-miss / look-alike values.
• The model judges a key/value pair in the context of a document excerpt; accuracy degrades if the excerpt does not cover the value span.
• Probabilities are calibrated on the KVP-10K-style distribution (form-like documents); out-of-domain calibration is not guaranteed.

Repository contents

File	Description
model.safetensors	Fine-tuned weights (encoder + decision head), 842 MB
rl_agent_config.json	Laya agent config (budgets, refit temperature, question metadata)
encoder/config.json	ModernBERT-large encoder config
tokenizer/	Tokenizer files (tokenizer.json, tokenizer_config.json)
checkpoint_meta.json	Checkpoint provenance (epoch, val metrics)
progress.png	Training curves

Citations & credits

• Laya: Convai Innovations — https://huggingface.co/convaiinnovations/laya (Apache-2.0)
• KVP-10K: IBM Research — https://research.ibm.com/blog/kvp10k-dataset
• KVP-10K pre-parsed community mirror: https://huggingface.co/datasets/alessandrorusso21/KVP10k
• Encoder: answerdotai/ModernBERT-large

License

Apache-2.0 (inherits the base model's license).