lokomotifai/hushmark-modernbert-tr-149m

hushmark-tr-v2

Status

hushmark-tr-v2 is a closed-set multi-label BIO tagger for the twelve NER-owned Hushmark entity types. It passed the locked adoption gate on 2026-08-22 against the incumbent hushmark-tr GLiNER model. It is not yet the runtime default: the incumbent remains configured while the generalisation limits below are addressed.

Intended use

The model proposes spans for the twelve NER-owned types: person, address, organization, date of birth, health, religion, ethnicity, political opinion, sexual life, criminal record, biometric reference, and union membership. Deterministic identifiers and secrets remain owned by the L0 validators and are rejected at the corpus schema boundary, so the model is never trained to duplicate them.

This model is a detection aid, not an anonymisation or legal-compliance guarantee. False negatives and false positives remain possible; operators must validate it on their own data.

Special categories are subject-conditioned

Following the decision recorded in docs/hushmark-tr-v2-plan.tr.md section 3.1, the eight special categories are labelled only when attributed to a data subject. "Hipertansiyon Türkiye genelinde yaygındır" carries no data subject and is a negative; "Hasta, migren nedeniyle rapor talep etti" is a positive. The taxonomy descriptions in taxonomy/taxonomy.yaml were updated to match.

Architecture

ytu-ce-cosmos/modernbert-tr-base (Apache-2.0) with one three-way O/B/I head per entity type. Per-type heads rather than a single tag sequence: an ADDRESS containing an ORG is common in Turkish administrative text and a single sequence cannot represent it.

148,765,476 parameters. Context 384 tokens at training time; the architecture supports 8,192.

GLiNER was dropped because its value is open-vocabulary zero-shot recognition and this taxonomy is closed and pinned. The measured cost of that mismatch is in the comparison below.

Data provenance

Corpus SHA-256 93098439f539218b37cbd752441173141f2e4ace2d16e2d9690c97431318e3a0, 114,983 documents: train 51,275 · dev 4,421 · locked eval 5,513 · quarantine 53,774.

Source	Documents	Contribution
External Turkish PII packages	43,966	PERSON, ADDRESS, DOB, plus promoted ORG and decoys
Generated special-category documents	14,350	the eight special types, subject-attributed
Mined real-text negatives	1,616	unattributed special-category vocabulary in real prose
Rule-based hard negatives	1,278	non-birth dates, month and region tokens, generic units
Real Turkish document chunks	53,774	quarantined, awaiting annotation; not trained on

Span counts: PERSON 17,845 · ORG 9,687 · DOB 5,319 · ADDRESS 3,487 · HEALTH 1,799 · CRIMINAL 1,797 · UNION 1,797 · ETHNICITY 1,796 · RELIGION 1,795 · POLITICAL 1,788 · SEXUAL_LIFE 1,788 · BIOMETRIC_REF 1,790. Negative documents: 28,576.

Special-category surface forms come from a 550-entry lexicon (digest 343a2538cb20395c) rather than the three-per-type bank used for v1.

Split membership is derived from the upstream leakage group where one exists and from normalised content otherwise. Leakage audit on the trained splits: 0 exact collisions, 4 near duplicates.

Training

One NVIDIA A100-SXM4-80GB, bf16, batch size 32, max length 384, 3 epochs, encoder LR 3e-5, head LR 1e-4, 200 warm-up steps, outside-class weight 1.0 against 8.0 for B/I, validation every 300 steps, patience 6. The run took 575.5 seconds over 4,500 steps and peaked at 9,965,534,720 bytes of GPU memory. Run fingerprint a05350320ef27f927b19f6a0a643d99ebed882f7534e80e5251611198a3c9246.

Development-best macro strict-F1 was 0.9994392104 at step 2,700, and that checkpoint was selected.

Locked evaluation

The 5,513-row locked split was scored exactly once, after development selection. The evaluation script refuses to overwrite an existing verdict.

	candidate	incumbent hushmark-tr
macro strict-F1	0.9961285750	0.3667564825

Absolute improvement +0.6293720926; minimum required +0.05; per-type regressions over 0.02: none; machine verdict adopt: true.

Per-type strict F1, candidate against incumbent:

Type	v2	v1	Type	v2	v1
ADDRESS	1.0000	0.0034	HEALTH	1.0000	0.2849
BIOMETRIC_REF	1.0000	0.0887	ORG	0.9808	0.6996
CRIMINAL	1.0000	0.1092	PERSON	0.9955	0.6398
DOB	0.9860	0.8828	POLITICAL	1.0000	0.1467
ETHNICITY	1.0000	0.5246	RELIGION	0.9912	0.3614
SEXUAL_LIFE	1.0000	0.3697	UNION	1.0000	0.2902

The number that matters more

The locked split is drawn from the same generators as training, so 0.9961 measures whether the model learned the corpus — not whether it understands Turkish. A ten-item probe set written by hand outside the generator's sentence frames (bench/data/probes-tr-v1.jsonl) scores 8/10:

• in-frame 1/1, curated negatives 3/3, out-of-frame 4/6.
• "Hasta Ayşe Yılmaz migren tanısıyla başvurdu." — HEALTH missed. The adjectival ... tanısıyla construction does not appear in any training frame. A false negative on an attributed health condition is the most expensive error this model can make.
• "Doktor bey, iki yıldır tip 2 diyabet hastasıyım." — HEALTH found, but a spurious PERSON on the form of address.

Read the two numbers together: the corpus is learnable and the architecture works, and frame diversity — not lexicon size — is now the binding constraint on the eight special categories.

External anchors

Scored against four suites outside this model's generators, five other engines alongside it. Every engine saw identical text; the annotation conventions are not ours.

Suite	What it is	v2 strict-F1	Best engine on that suite
anchor-btx24	Third-party synthetic Turkish, independent convention	0.8992	hushmark-berturk-112m 0.9127
anchor-wikiner	Real Wikipedia sentences, human-annotated	0.2967	gliner_multi_pii-v1 0.4568
suite-traps	Empty-gold look-alikes; firing is the error	0.000 FP	v2 (only engine at zero)
suite-trap-twins	Same surfaces with a data subject attached	0.8063	v2

Read those four rows together and they say one thing. On the trap/twin pair — the same confusable surfaces in both directions — v2 is the strongest engine measured: it emits nothing on 342 subject-free look-alikes while recovering 0.8063 of the attributed twins. Nobody else comes close (gliner_multi_pii-v1 reaches 0.6873 recall but pays a 0.430 false-positive rate). The subject-conditioning described above is real and it is measurable.

On human-annotated encyclopedic prose it collapses to 0.2967, below the incumbent hushmark-tr at 0.4437. A separate probe over 212 human-annotated group mentions, split by whether a data subject is present, puts the discrimination ratio at 1.02 — indifference. The subject-conditioning does not transfer out of the administrative register.

So: use this model on forms, records and administrative correspondence, which is what it was built for and where it is genuinely good. Do not use it on free prose, news or encyclopedic text. The 0.9961 locked figure describes neither case; it describes the training distribution.

ONNX deployment

Opset-19 FP32 graph, 595,684,002 bytes, SHA-256 1be3d53cc51cf93b744331522608d5cb1b0fb5f62ecb2d8b66c05290a4eb13a4. The serving path needs no transformers and no torch: a pinned tokenizer.json drives the Rust tokenizer and onnxruntime drives the graph. Label order is pinned in labels.json and checked against the taxonomy order at load; a re-ordered checkpoint cannot be served silently.

On a stratified development sample covering all twelve types at threshold 0.5, FP32 scored macro strict-F1 1.0000 and dynamic INT8 scored 0.9998021 — a loss of 0.0002, far inside the 0.02 limit.

INT8 is viable for this architecture. v1's INT8 collapsed to 0.4136 on mDeBERTa-v3's disentangled attention; ModernBERT has no such structure. The INT8 graph is 149,825,638 bytes, SHA-256 f3849966184b43b1b5bd8dec6ac8b4d9fdb331fd87be12a77e31ec19ec6ffa6f. FP32 is registered as the pinned export; INT8 is a validated candidate the operator may switch to.

Latency

Local Apple ARM64 host, 4 torch threads to match the core container's 4 vCPU quota.

Input	v1 GLiNER FP32	v2 FP32	v2 INT8
Short sentence, median 63 characters	16.7 ms	6.5 ms (2.57×)	5.5 ms (3.04×)
Real document chunk, median 1,789 characters	148.6 ms	72.0 ms (2.06×)	69.9 ms (2.13×)

Artifact size drops from 1,157 MB (v1 FP32) to 596 MB (v2 FP32) or 150 MB (v2 INT8) — 87% smaller at INT8. Most of v1's weight was a 250,105-token multilingual embedding matrix, 66% of its parameters, replaced here by a 50K Turkish vocabulary.

This is technical performance evidence for one host profile, not a capacity guarantee.

Limitations

• Frame overfitting is the dominant limitation. Special-category positives are generated from a bounded set of sentence frames. Section "The number that matters more" quantifies the gap.
• The model is a register specialist, not a Turkish PII model. See "External anchors" above; this is the limitation that matters most when deciding whether to deploy it.
• The locked split shares generators with training. It is a regression gate, not a measure of production capability.
• The internal review round-trip (corpus/review.py) is implemented but has not been run, so no in-house human-adjudicated set exists. The external anchors below are human-annotated by third parties, which is not the same thing as agreement with our own annotation guideline.
• 53,774 real Turkish document chunks remain in quarantine, unannotated.
• ORG supervision is thin lexically: roughly 150 distinct organisation names, all public institutions. Private-company names are absent.
• Inputs beyond the configured context are truncated. Long-document chunking is unmeasured.
• The mined negatives are labelled for the eight special categories only; their organisation and address spans are unannotated and masked out of the loss.

License and release

The base encoder ytu-ce-cosmos/modernbert-tr-base is Apache-2.0. Artifact digests are pinned in core/models.yaml in the hushmark repository under hushmark-tr-v2. Weights are distributed separately from the public repository and must match the pinned hashes before use.

Files

File	Size	Role
model.onnx	568 MB	FP32 graph, the pinned serving artifact
model_quantized.onnx	143 MB	Dynamic INT8, validated candidate (0.9998 vs FP32 1.0000)
model.pt	568 MB	Torch state_dict, reproducibility and re-export path
tokenizer.json	1.2 MB	Fast tokenizer; the serving path needs nothing else
labels.json	—	Pinned label order; a re-ordered checkpoint must not be served silently
evaluation/	—	Locked verdict, cross-model comparison, probes, training manifest, latency

Use

The serving path needs no transformers and no torch: a pinned tokenizer.json drives the Rust tokenizer and onnxruntime drives the graph. The head is one three-way O/B/I decision per entity type, so logits reshape to [tokens, 12, 3] in the order given by labels.json.

import json, numpy as np, onnxruntime
from tokenizers import Tokenizer

labels = json.load(open("labels.json"))
tok = Tokenizer.from_file("tokenizer.json")
session = onnxruntime.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

encoding = tok.encode("Hasta Ayşe Yılmaz migren tanısıyla başvurdu.")
logits = session.run(None, {
    "input_ids": np.asarray([encoding.ids], dtype=np.int64),
    "attention_mask": np.asarray([encoding.attention_mask], dtype=np.int64),
})[0]
scores = logits[0].reshape(len(encoding.ids), len(labels), 3)
# tag 1 = begin, 2 = inside, 0 = outside; character offsets come from encoding.offsets.

Reference decoding is hushmark_core.ner.tokencls_decode; the runtime adapter that verifies size, SHA-256, and label order before loading is hushmark_core.ner.tokencls_backend.

Reading the evaluation folder

Start with evaluation/model-comparison.md. The locked verdict in evaluation/locked-eval-verdict.json says adopt: true at macro strict-F1 0.9961, but that split shares generators with training. evaluation/probes.json is the honest number: 8/10 on ten hand-written out-of-frame sentences.