yunfengwang/laya-MNN-int8

Laya for MNN

MNN build of Laya — a multilingual, non-autoregressive System 1 decision model. Give it a state (email, ticket, JSON) and typed questions; it returns typed answers with calibrated probabilities in a single forward pass. It never generates text, so there is nothing to parse and nothing to hallucinate.

This repository hosts fp16 and int8 checkpoints of the English model (ModernBERT-large backbone, 421M parameters), converted for the MNN runtime so they run on CPU without PyTorch.

repo	precision	size	verification vs. original PyTorch
laya-MNN-fp16	fp16 weights	846 MB	9/9 answers match, worst probability delta 0.0019
laya-MNN-int8	8-bit weight-only	581 MB	9/9 argmax match, worst probability delta 0.0343

Use fp16 unless size is critical.

Files

laya-fp16.mnn     converted weights
tokenizer.json    ModernBERT BPE tokenizer (used directly, no transformers dependency)
manifest.json     config, sha256, max_len/head_max_len/max_options, verification results

Run it

uvx laya-mnn decide \
  --state "Hi, we were billed twice for March. Please refund the duplicate or we will cancel our plan." \
  --questions '{"department":{"type":"choice","instructions":"Which department should handle this?","criteria":{"billing":"invoices, payments, refunds","technical":"bugs, outages","other":"everything else"}}}'

The CLI downloads these weights automatically (ModelScope first, then Hugging Face) into ~/.cache/laya-mnn. Source: laya-mnn on PyPI.

from laya_mnn import LayaMNN, ensure_model
model = LayaMNN(ensure_model("fp16"), threads=4)

Conversion

convaiinnovations/laya (PyTorch)
  -> ONNX opset 17, eager attention, static [1, 512] sequence, 64 marker slots
  -> MNN converter 3.6.1 (--fp16 / --weightQuantBits 8)

Exported with attn_implementation="eager" so the graph stays as plain MatMul/Softmax/LayerNorm/GELU rather than a fused Attention node, and with static shapes so constant folding resolves the RoPE tables and the sliding-window masks. The graph contains GatherElements, Erf, TopK, Cos, Sin, Mod, LessOrEqual and Where; all are supported by the MNN ONNX importer.

Verification

Each checkpoint was compared against the original PyTorch model on three sample states covering all three primitives (choice, score, noul), 9 questions in total, using the same temperatures the checkpoint ships with:

	fp16	int8
answers matching	9/9	9/9 (argmax)
worst probability delta	0.0019	0.0343
worst confidence delta	0.0010	0.0343

Limits

• English only. This is the repo-root checkpoint; the multilingual mmBERT-base variant is not converted here.
• 512 tokens of context, max 64 options per question, 16 markers worth of head budget shared with the state text as in the original (head_max_len = 192).
• Batch 1. The graph is exported for a single sequence; MNN's shape inference mishandles the padding mask at larger batches, so questions are answered one at a time.
• CPU only. The Metal backend returned all-zero logits for this graph in the tested MNN 3.6.1 wheel.
• Calibration figures in the original model card are for the PyTorch model. The upstream checkpoint ships over-confident and expects temperature refitting on your own data — that applies here too.

License and attribution

Apache-2.0. Model, prompt format and calibration scheme by Convai Innovations (Apache-2.0). This is an independent community conversion and is not affiliated with or endorsed by them.