mivialabs/mivia-small-json-tool-match

mivia-small-json tool-match guard

A cross-encoder tool-match guard for AI-agent runtimes: given a user request and a tool catalogue, it decides CALL (execute with verbatim arguments), CLARIFY (structured ask-back naming the missing parameters), or ABSTAIN (decline - the request is irrelevant or unserviceable). A relation a bag-of-ngrams model cannot express, so this is a fine-tuned cross-encoder: int8-quantized from cross-encoder/ms-marco-MiniLM-L-6-v2 (Apache-2.0). CPU-only; the split gate + rank chain is ~46 MB and answers in milliseconds.

The companion router (mivia-small-json-router) classifies the workflow record itself; this guard decides the tool action.

Files - tool-match-gate-int8.onnx (fillability span, decision + veto) scores over tool-match-int8.onnx (rank head); each has its tokenizer and sha256 sidecar. MIVIA_SELECT_RANK_HEAD=legacy semantics and the composition order live in the GitHub package.

Intended use

Pre-execution gating in agent loops: before invoking a tool or a frontier model, score the request against the catalogue. Missing mandatory parameters become a structured clarification instead of a hallucinated call; irrelevant requests abstain instead of guessing. Not a general relevance model outside tool catalogues, and it does not execute tools itself.

Usage

The guarded path (catalogue normalization, per-tool scoring, the deterministic string-fill ladders, the veto) is in the GitHub repo; raw ONNX inference:

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

repo = "mivialabs/mivia-small-json-tool-match"
rank = hf_hub_download(repo, "tool-match-int8.onnx")
tokenizer = Tokenizer.from_file(
    hf_hub_download(repo, "tool-match-tokenizer.json"))
# pair-score (request, tool description) with the rank head, then apply
# the gate head; the served THRESHOLD and veto logic are in the package.

Training data

No third-party dataset is redistributed; the per-source inventory (BFCL, Glaive function-calling, session-derived pairs and their licenses) is docs/DATA.md in the GitHub repo. Fine-tuned from the sentence-transformers ms-marco cross-encoder (Apache-2.0); changes: task fine-tuning and int8 quantization.

Evaluation

Frozen sets disjoint from training data; full state in the repo's docs/MODEL-STATE.md.

Surface	Result
Guard eval (5,100 rows)	79.7% abstention @ 95.1% retention (threshold 0.6020)
Catalogue AUC	0.9030 full-set; no_tool_needed 95.4%
Real agent-session invocations	serves 71.5% (AUC 0.93; honest baseline 21.7%)
Residual categories	infeasible 36.7% / sibling_swap 63.5% / ask_followup 74.8% - the served outcome is the honest CLARIFY path
String-parameter filling	89.07% (163/183) on the BFCL calls set - verbatim spans only, never fabricated

Limitations

Published in the repo's docs/LIMITATIONS.md: infeasible and sibling-swap requests remain the hard tail.

License

Apache-2.0. Base checkpoint remains Apache-2.0; attribution in the repo's NOTICE.md.