leli14/laya_cpu_onnx

Laya CPU ONNX (Multilingual Intent Classifier)

Laya CPU ONNX is a high-performance, lightweight in-process intent classification engine optimized for Real-Time Conversational AI & Digital Human Agents (e.g., AI Interviewer, Virtual Agent).

Based on the ModernBERT architecture with multi-choice intent heads, this model provides millisecond-level decision making on standard x86_64 CPU environments with zero external network RTT.

---

Model Highlights

• Pure CPU Optimized: Quantized to Dynamic INT8; no GPU / CUDA dependencies required.
• Ultra-low Latency: ~150ms – 180ms per decision on 4 standard CPU cores.
• In-Process Inference: Fully compatible with Java (via onnxruntime-java & DJL tokenizers) and Python (onnxruntime).
• Multilingual Tokenizer: Pre-trained on 166,000+ vocabulary tokens covering Chinese, English, and multi-turn conversational semantics.

---

Model Architecture & Artifacts

File	Size	Description
model_quantized.onnx	1.23 GB	Dynamic INT8 quantized ONNX graph (~322M parameters)
tokenizer.json	32.8 MB	HuggingFace FastTokenizer vocabulary (166k tokens)
tokenizer_config.json	524 B	Special token mappings (<bos>, <eos>, <mask=4>, <pad=0>)
special_tokens_map.json	731 B	Token definitions

---

Graph I/O Specification

1. Inputs

Name	Type	Shape	Dynamic Axes	Description
input_ids	int64	[batch, seq]	batch, seq	Token IDs (Max sequence: 512)
attention_mask	int64	[batch, seq]	batch, seq	Attention mask (1 for valid, 0 for pad)
marker_pos	int64	[batch, options]	batch, options	Position index for <mask=4> markers
marker_mask	bool	[batch, options]	batch, options	Boolean mask for candidates
qtype	int64	[batch]	batch	Task type: 0 for Choice Question

2. Outputs

Name	Type	Shape	Description
logits	float32	[batch, options]	Unnormalized logits across the candidate intents
act_probs	float32	[batch, 2]	Binary action probabilities

---

Decision States & Business Scenarios

The model is designed for turn-level dialogue arbitration, supporting two primary stages:

1. Initial Answer Stage (INITIAL_ANSWER)

• FINISHED: The user has completed answering the question.
• CONTINUE: The user is still elaborating their thoughts.
• THINKING: The user explicitly requests time to organize their thoughts.
• CLARIFICATION: The user is asking for question clarification or scope definition.
• UNCERTAIN: Ambiguous intent, recommended for fallback to an LLM.

2. Supplement Stage (SUPPLEMENT)

• NO_SUPPLEMENT: The user has no further comments.
• HAS_SUPPLEMENT: The user provides new facts or follow-up details.
• THINKING: The user needs time to ponder.
• UNCERTAIN: Ambiguous state.

---

Quickstart (Python)

import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

# 1. Load Tokenizer & Session
tokenizer = Tokenizer.from_file("tokenizer.json")
sess_opts = ort.SessionOptions()
sess_opts.intra_op_num_threads = 4
session = ort.InferenceSession("model_quantized.onnx", sess_options=sess_opts, providers=['CPUExecutionProvider'])

# 2. Prepare Candidates & Inputs
CRITERIA = [
    "FINISHED: 候选人已回答完毕，核心内容表达完整或明确表示回答完毕",
    "CONTINUE: 候选人正在回答中尚未结束，还在继续说明",
    "THINKING: 候选人需要思考或稍等",
    "CLARIFICATION: 候选人正在向面试官确认题意或概念范围",
    "UNCERTAIN: 无法可靠判断是否结束"
]

prompt = "choice question: 针对面试问题【请介绍你的项目经历】，判断候选人的回答状态"
state = "我主要负责微服务架构和核心接口开发，我说完了。"

# 3. Build sequence with <bos>=2, <eos>=1, <mask=4>
head_ids = tokenizer.encode(prompt).ids
seq = [2] + head_ids + [1]
markers = []
for opt in CRITERIA:
    markers.append(len(seq))
    seq.append(4)  # <mask=4>
    seq.extend(tokenizer.encode(" " + opt).ids[:48])
seq.append(1)
seq.extend(tokenizer.encode(state).ids[: 511 - len(seq)])
seq.append(1)

# 4. Run Inference
inputs = {
    "input_ids": np.array([seq], dtype=np.int64),
    "attention_mask": np.ones((1, len(seq)), dtype=np.int64),
    "marker_pos": np.array([markers], dtype=np.int64),
    "marker_mask": np.ones((1, len(markers)), dtype=bool),
    "qtype": np.array([0], dtype=np.int64)
}

outputs = session.run(None, inputs)
logits = outputs[0][0]
probs = np.exp(logits) / np.sum(np.exp(logits))
best_idx = np.argmax(probs)
print(f"Predicted Decision: {CRITERIA[best_idx].split(':')[0]} (Confidence: {probs[best_idx]:.3f})")

---

Quickstart (Java)

Add the following dependencies in your pom.xml:

<dependency>
    <groupId>com.microsoft.onnxruntime</groupId>
    <artifactId>onnxruntime</artifactId>
    <version>1.18.0</version>
</dependency>
<dependency>
    <groupId>ai.djl.huggingface</groupId>
    <artifactId>tokenizers</artifactId>
    <version>0.31.1</version>
</dependency>

In your Java service:

OrtEnvironment env = OrtEnvironment.getEnvironment();
OrtSession.SessionOptions opts = new OrtSession.SessionOptions();
opts.setIntraOpNumThreads(4);
OrtSession session = env.createSession("model_quantized.onnx", opts);
HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance(Paths.get("tokenizer.json"));

---