code-daemon-ner-v1

A small token-classification (BIO) model that finds software-engineering entities in prose — the sentences of a README, a design doc, an issue thread, a commit message. It ships with the UltraCode MCP server as a TensorRT / OpenVINO engine and feeds the documentation knowledge graph: the spans it emits become the concept nodes that link a paragraph of prose to the code entities it talks about.

This is prose NER, not code parsing. Identifiers inside source files are extracted by tree-sitter, exactly and for free. What tree-sitter cannot do is read the sentence "the IVF index rescoring path in faiss_index.zig regressed nDCG@10 by 4 points" and tell you that IVF is an algorithm, faiss_index.zig a file path, and nDCG@10 a metric. That is this model's job.

• ~117M params — XLM-RoBERTa 12 layers / 384 hidden, 250k multilingual SentencePiece vocab (the embedding table dominates the parameter count).
• 2-input ONNX (input_ids, attention_mask; no token_type_ids) → logits [batch, seq, 17].
• Max sequence 128 tokens — one or two sentences, which is the unit the daemon chunks prose into.
• Bilingual by construction (English + Russian docs), inherited from the XLM-R backbone.

What this model is for. Entity-level Core F1 is 0.43. That is a useful recall-oriented candidate generator for a knowledge graph, and it is not a labeller to trust unattended. The Evaluation section has the per-type numbers, including the two types that are weak and why.

Label set — 8 entity types, 17 BIO classes

Split into a Core tier (what the knowledge graph links on) and a Soft tier (useful, tolerated at lower precision). Both are scored separately.

Tier	Type	What it marks	Example span
Core	component	a named part of a system, ours or someone's — including model names	DocStore, mMiniLMv2
Core	api_endpoint	a callable surface: function, method, route, agent tool name	semantic_search, POST /v1/messages
Core	file_path	a path or a filename	src/semantic/faiss_index.zig, config.toml
Core	tool	an external executable, runtime, service, language or hardware you invoke	TensorRT, rclone, SQL, Tesla T4
Soft	algorithm	a named method or procedure	BM25, beam search, Louvain
Soft	data_structure	a named container or layout	CSR, Prolly tree, arena
Soft	config	a knob, flag, env var or setting	journal_mode=MEMORY, --epochs
Soft	metric	a measurement or its unit	nDCG@10, macro-F1, p95 latency

The component / tool / api_endpoint boundary costs annotators the most, so it is fixed by rule rather than by feel, and the rules are applied in order — that ordering is what resolves names fitting two types. A model is a component (a part of a system); an engine, runtime, language or piece of hardware is a tool (you invoke it); an agent tool name is an api_endpoint (a callable surface, not a program). About a dozen names are genuinely polysemous and take their label from the sentence.

How it was made

There is no off-the-shelf corpus for this label set, so the training labels were manufactured by weak supervision — several independent labellers vote per chunk and the votes are reconciled into a single BIO sequence — followed by supervised fine-tuning of the backbone and a temperature calibration pass.

The calibration is the part you need at inference: a single scalar, shipped as temperature.json (T = 0.665). Divide the logits by it before the softmax if you consume probabilities rather than argmax.

Evaluation is human-labelled and disjoint from the training labels — 1 500 hand-reviewed sentences, normalised so that no surface name carries two labels and no span overlaps another.

Evaluation

Measured on the 1 500-row human gold set, entity-level (exact span and type, seqeval convention), macro-averaged over the bare types — the metric NER results are normally reported in.

Split	macro-F1	micro-F1
Core (component, api_endpoint, file_path, tool)	0.4264	0.4502
Soft (algorithm, data_structure, config, metric)	0.2072	0.1910

Per type, with precision and recall, because they differ a lot and the difference is the story:

Type	Tier	F1	P	R
api_endpoint	Core	0.521	0.409	0.720
file_path	Core	0.497	0.391	0.682
tool	Core	0.425	0.421	0.429
component	Core	0.263	0.218	0.330
metric	Soft	0.273	0.404	0.206
algorithm	Soft	0.239	0.193	0.312
data_structure	Soft	0.198	0.169	0.239
config	Soft	0.119	0.166	0.093

What this says, plainly:

• Recall runs well ahead of precision on the strongest types, and that is a property of the training labels rather than of the model: api_endpoint and file_path are labelled in the training corpus at roughly 2.6× the density the human gold uses, and the model over-predicts them by about the same factor. Over-labelled in, over-predicted out. If you use it as a candidate generator and filter afterwards, this is the behaviour you want; as a final labeller it is not.
• component is the weakest Core type (0.263). Unlike tool it is not under-labelled — its training density already exceeds the gold's — so density is not what limits it. It is genuinely the hardest of the four: what counts as a named part of a system is contextual in a way a path or a call is not.
• config is the weakest overall (0.119, recall 0.093). Its labels are inconsistent between the two halves of the training corpus — dense in documentation, nearly absent in Q&A — and averaging two incompatible labelling regimes yields noise. A known, unfixed defect.
• Multi-token spans are weak. The I- classes are near-empty in training, so most predictions are effectively single-token. Only ~4% of Core gold spans are multi-word, so this costs less than it sounds, but do not expect long names to come back whole.

For direction rather than level: the model this replaces scores 0.077 Core on the same ruler, and the previous build of this one scored 0.323. All three were measured identically.

Intended use

Extract engineering entities from documentation-style prose to build or enrich a knowledge graph, to tag issues, or to route search. Use it as a recall-oriented candidate generator behind a rule or human filter, not as a labeller you trust unattended.

Out of scope: extracting symbols from source code (use a parser), general-domain NER (use a general-domain model), and any high-stakes automatic decision.

import onnxruntime as ort, numpy as np, json
from transformers import AutoTokenizer

tok    = AutoTokenizer.from_pretrained(".")            # bundled XLM-R SentencePiece
labels = json.load(open("code-daemon-ner-v1_label_map.json"))["id2label"]
T      = json.load(open("temperature.json"))["T"]
sess   = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

def tag(text, max_len=128):
    enc = tok(text, return_tensors="np", truncation=True, max_length=max_len,
              return_offsets_mapping=True, return_token_type_ids=False)
    logits = sess.run(None, {"input_ids":      enc["input_ids"].astype(np.int64),
                             "attention_mask": enc["attention_mask"].astype(np.int64)})[0][0]
    probs  = np.exp(logits / T) / np.exp(logits / T).sum(-1, keepdims=True)   # calibrated
    for (a, b), row in zip(enc["offset_mapping"][0], probs):
        lab = labels[str(int(row.argmax()))]
        if lab != "O" and b > a:
            yield text[a:b], lab, float(row.max())

for span, label, p in tag("The IVF rescoring path in src/semantic/faiss_index.zig cost 4 nDCG@10 points."):
    print(f"{span!r:40} {label:18} {p:.2f}")

The numbers above were measured at sequence length 128 with identifier pre-splitting (camelCase / snake_case split before tokenizing), which is how the model was trained and how it is deployed. Scoring it at a different sequence budget, or without the pre-split, gives a different and lower result.

What's in this repo

Flat layout, named per runtime × GPU arch × OS (single-profile — no length buckets):

• TensorRT code-daemon-ner-v1_{win_x64,linux_x64}_trt11.0_sm_120.engine — INT8 Q/DQ, built for sm_120 (RTX 50xx). A serialized engine is keyed on {GPU arch × OS × TensorRT version} with no compat fallback, so a machine on another architecture uses the OpenVINO or ONNX path.
• OpenVINO code-daemon-ner-v1_ov2026.3_{cpu,igpu_lnl}_int8_b32_s128.{xml,bin} and ..._npu_int4_b8_s128.{xml,bin} — Intel CPU / iGPU / NPU.
• Tokenizer — sentencepiece.bpe.model (XLM-R SP; the daemon loads it natively) plus tokenizer_config.json, which is what AutoTokenizer.from_pretrained reads. The fast tokenizer ships as code-daemon-ner-v1_tokenizer.json.
• Labels & calibration — code-daemon-ner-v1_label_map.json (17 classes, label2id / id2label), temperature.json (a single scalar under the key T), manifest.json (build metadata), eval_metrics.json (per-type and per-class F1). The label map and the fast tokenizer carry the model-id prefix because that is the name the daemon resolves; the snippet above uses the same names.
• ONNX — model.onnx FP32 dynamic-shape (the build source, and standalone onnxruntime / optimum use), model_static.onnx (seq pinned to 128, the engine-build input), and model_int8qdt.onnx (INT8 Q/DQ).

No MLX build: this model has no Apple-GPU consumer today.

License

Released under the MIT license.

Source	Role	Note
nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large	backbone	a re-upload of Microsoft's mMiniLMv2 (microsoft/unilm, MIT); the re-upload repo itself declares no license

Neither the training corpus nor the human gold set (1 500 rows, evaluation-only) ships in this repo. Not legal advice.

Attribution

Backbone: nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large (itself distilled from XLM-RoBERTa-Large).