LFM2.5-Encoder-350M-Prompt-Router-ONNX

An ONNX export of LiquidAI/LFM2.5-Encoder-350M-Prompt-Router, for Lemonade and its ort-server backend.

All credit for the model goes to Liquid AI. This repository only changes the file format. The weights are the original float32 weights, and the LFM Open License v1.0 applies (see LICENSE).

The model is a zero-shot prompt router. You give it a text and a list of labels at request time. It scores the text against every label in one encoder pass, and the scores sum to 1.

Changes from the original

File	Origin
model.onnx	Converted from model.safetensors at revision 35ca4a0469f180f1cf05a630df8842fa17ac18e3.
config.json, tokenizer.json, LICENSE	Unchanged copies.
manifest.json	New. ort-server settings: task: zero-shot-classification, max_length: 128000.

The graph takes input_ids, attention_mask, text_pool [b, 1, s] and category_pool [b, r, s], and returns logits [b, r]. The label count r and the sequence length s are dynamic. The two pooling inputs mark which tokens belong to the text and to each label. ort-server builds them from the tokenizer's character offsets.

Use with Lemonade

Lemonade needs ort-server 0.4.0 or later. The model is built in:

lemonade pull LFM2.5-Encoder-350M-Prompt-Router-ONNX

Classify a text:

curl -X POST http://localhost:13305/api/v1/classify \
  -H "Content-Type: application/json" \
  -d '{"model": "LFM2.5-Encoder-350M-Prompt-Router-ONNX",
       "input": "Can you help me debug a failing Python unit test?",
       "labels": ["Coding", "Sales", "Creative writing", "General knowledge"]}'

{"labels": {"Coding": 0.838, "Creative writing": 0.054, "General knowledge": 0.054, "Sales": 0.054}}

In a router policy, use the model in a zero_shot classifier. See Lemonade's docs/dev/router-policy.md.

How this export was made

The scripts are in ort-server's tools/zero_shot_router/. From an ort-server checkout:

pip install torch transformers onnx onnxruntime huggingface_hub
python tools/zero_shot_router/export.py <out_dir> --revision 35ca4a0469f180f1cf05a630df8842fa17ac18e3
python tools/zero_shot_router/parity.py <out_dir> --revision 35ca4a0469f180f1cf05a630df8842fa17ac18e3 --binary <ort-server>

export.py loads the checkpoint with trust_remote_code=True. This is required, because Lfm2BidirForSequenceRouting and route() are in the checkpoint's modeling_lfm2_bidirectional.py, not in transformers. The script then:

1. Replaces the short-conv forward, whose padding and groups come from tensor shapes that tracing cannot export.
2. Exports forward() at opset 17, not route(), so that the label count stays dynamic.
3. Compares the graph with route() on 16 cases with 2 to 8 labels, and stops on any score difference above 1e-4 or any changed top label.

Tested with transformers 4.57, torch 2.8 and onnxruntime 1.24.

Parity with the original model

parity.py runs 95 labelled dummy cases through three paths:

• the original safetensors model, with its own route() in PyTorch
• this ONNX graph in ONNX Runtime
• ort-server over HTTP, which builds the pooling inputs in C++

The cases cover routing, PII, prompt injection, support topics, French, Spanish, German and Japanese, and three long texts of 900 to 1,900 tokens.

Path	Accuracy on the dummy labels	Top-1 agreement with PyTorch	Max score difference
PyTorch (safetensors)	76.84%	-	-
ONNX Runtime	76.84%	100%	2.3e-6
ort-server	76.84%	100%	2.3e-6

The accuracy column describes the model on hand-written labels, not the conversion: all three paths get the same cases right and wrong.

License

LFM Open License v1.0. See LICENSE and the original model. Commercial use above the license's threshold is not licensed.