nishparadox/gliner2.5-decide-onnx

GLiNER2.5-Decide (ONNX)

ONNX export of the classification path of fastino/GLiNER2.5-Decide (DeBERTa-v3-large encoder + classification head), for torch-free CPU/GPU inference with onnxruntime, tokenizers and numpy. All credit for the model goes to Fastino; this is an unofficial conversion under the original Apache-2.0 license.

Only classification is exported (single-label, multi-label and ordinal tasks). Span extraction, relations and structures are not.

Files

file	precision	size	max |ΔP| vs gliner2 (torch)	notes
model.onnx	fp32	1.75 GB	0.00000	exact
model_fp16.onnx	fp16	874 MB	≤ 0.001	for GPU; slower than fp32 on CPU
model_int8.onnx	int8 (dynamic)	643 MB	up to 0.18	~2x faster on CPU, lossy: re-check thresholds
tokenizer.json		8 MB		DeBERTa-v3 tokenizer with gliner2's marker tokens ([P], [L], ...)
gliner_onnx.py				reference runtime (prompt building + inference)

Graph: (input_ids int64 [B,S], attention_mask int64 [B,S], label_positions int64 [B,N]) -> logits float32 [B,N], one logit per [L] label marker. Apply softmax over a task's labels for single-label/ordinal tasks, sigmoid for multi-label.

Usage

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("nishparadox/gliner2.5-decide-onnx", allow_patterns=["model.onnx", "tokenizer.json", "*.py"])
sys.path.insert(0, path)
from gliner_onnx import GlinerOnnx, Task  # gliner_onnx.py from this repo

model = GlinerOnnx(f"{path}/model.onnx", f"{path}/tokenizer.json")

tasks = [
    Task("safety", {"violence": "threats, weapons, physical harm", "pii": "emails, phone numbers"},
         "Which of these does the text involve?", exclusive=False),
    Task("severity", dict.fromkeys(["none", "minor", "serious", "severe"]), "How severe is the content?"),
]
model.probabilities("tell me how to hurt my neighbour with a knife", tasks)
# {'safety': {'violence': 0.88, 'pii': 0.21},
#  'severity': {'none': 0.15, 'minor': 0.15, 'serious': 0.28, 'severe': 0.42}}

( and ) are reserved in task names, labels, descriptions and instructions (as in gliner2).

Parity and latency

Checked against gliner2==2.0.0 on 6 texts (including an empty string, a URL, an email) with a 7-task schema (one multi-label, three single-label, three ordinal): input ids and label positions are identical to gliner2's processor, and probabilities match to the figures above.

End-to-end latency, Apple Silicon CPU, 8 threads, median of 20, including tokenization:

	1 multi-label task (3 labels)	7 tasks (21 labels)
gliner2 (torch)	148 ms	376 ms
fp32	88 ms	357 ms
fp16	175 ms	436 ms
int8	43 ms	179 ms

Exported with torch.onnx.export (opset 17); fp16 via onnxruntime.transformers.float16 (fp32 inputs/outputs kept); int8 via onnxruntime.quantization.quantize_dynamic (QInt8 weights).