Zylo7 Sentinel

A multilingual, multi-task support-ticket classifier fine-tuned on LaBSE. Given a customer support ticket in English, Sinhala, Tamil, Singlish or Tanglish, it predicts the ticket's category (11 classes), a possible secondary category, and whether it is urgent — in one forward pass, exported to ONNX for CPU-only serving.

Built by Team Zylo7 for TensorForge 2.0, Phase 2 (finalist). Full training code, evaluation and the served API: github.com/dhananjaya-hbc/tensorforge-2.0-zylo7.

What it predicts

Output	Type	Notes
primary	softmax over 11 classes	the main category
secondary	11 independent sigmoids	"is this issue also present?" — masked against the primary at decode time
urgent	sigmoid	urgency probability

The 11 categories: payment_refund, ride_trip_issue, lost_item, order_missing_wrong, delivery_delay, food_quality, account_promo, safety_conduct, app_technical, general_inquiry, spam_irrelevant.

A fourth, training-only head predicted the ticket's language (6 tags) as an auxiliary task, to push the encoder toward script-aware representations. It is not part of this export.

Why LaBSE

A character-level TF-IDF baseline looked excellent under ordinary cross-validation (0.994 macro-F1) but collapsed to 0.584 on a held-out validation set that paraphrases the same issues differently — the model had memorised templates, not meaning. Four multilingual encoders were then fine-tuned with identical multi-task heads and compared train → validation (never cross-validation):

Encoder	Macro-F1	Secondary F1	Urgent F1	Composite
LaBSE	0.908	0.933	0.956	0.927
multilingual-e5-base	0.839	0.868	0.962	0.882
XLM-RoBERTa base	0.846	0.819	0.856	0.843
multilingual-e5-small	0.810	0.324	0.917	0.745

LaBSE won because it was pre-trained on translation pairs across 109 languages, giving it exactly the skill this task needs: recognising the same meaning in different phrasings, across scripts.

Results

This export was refit on train + validation (4,800 tickets) for the shipped weights. The numbers below are from the same recipe trained on train only and scored on the held-out validation set (800 tickets, unseen phrasings) — the honest, non-leaked estimate:

Metric	Value
Primary accuracy	0.898
Primary macro-F1	0.900
Urgent precision / recall / F1	0.974 / 0.938 / 0.955
Secondary precision / recall / F1	0.941 / 0.800 / 0.865
Calibration (ECE)	0.041
Brier score	0.058

Per-language breakdown (macro-F1, n)

Language	n	Macro-F1	Urgent F1
English	280	0.832	0.873
Sinhala	160	0.887	1.000
Tamil	120	1.000	1.000
Singlish	200	0.921	1.000
Tanglish	20	0.846	1.000
Mixed script	20	0.969	1.000

On a separate set of 42 fresh, hand-written tickets never used for any training or tuning decision: 90.5% category accuracy, urgent recall 1.00 (with a small rule-based safety net for medical/infant-feeding/fraud urgency on top of the model — see the repo's ml/urgency.py).

ONNX export

• Architecture: LaBSE encoder → masked mean pooling → 3 linear heads.
• Precision: fp32 layers, with only the large word-embedding table (~385M of ~471M parameters) stored in fp16. This shrinks the file from 1.9 GB to 1.1 GB with 100% identical argmax predictions vs the original PyTorch model (max probability difference 4.8e-7).
• int8 dynamic quantisation was tried and rejected: it only agreed with PyTorch on 90–92% of predictions.
• No padding sensitivity: when serving, requests are bucketed by exact token length rather than padded, so a ticket gets the identical answer alone or in a batch.

Input	Shape	Type
input_ids	(batch, seq)	int64
attention_mask	(batch, seq)	int64

Output	Shape	Meaning
primary	(batch, 11)	softmax probabilities
secondary	(batch, 11)	sigmoid probabilities, per class
urgent	(batch,)	sigmoid probability

Usage

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
import onnxruntime as ort
import numpy as np

repo = "dhananjaya-c/sentinel"
onnx_path = hf_hub_download(repo, "model_b.onnx")
tok_path = hf_hub_download(repo, "tokenizer/tokenizer.json")

tok = Tokenizer.from_file(tok_path)
tok.enable_truncation(max_length=192)
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])

CATEGORIES = ["payment_refund", "ride_trip_issue", "lost_item", "order_missing_wrong", "delivery_delay",
              "food_quality", "account_promo", "safety_conduct", "app_technical", "general_inquiry",
              "spam_irrelevant"]

text = "my order is two hours late and the driver is not answering"
enc = tok.encode(text)
ids = np.array([enc.ids], dtype=np.int64)
mask = np.array([enc.attention_mask], dtype=np.int64)

primary, secondary, urgent = sess.run(None, {"input_ids": ids, "attention_mask": mask})
print(CATEGORIES[primary.argmax()], float(primary.max()), float(urgent[0]))

Note: the model expects cleaned, denoised input text (filler sentences, prompt-injection spans and quoted history removed) — see ml/preprocess.py in the repo for the exact preprocessing used in training and serving. preprocessor.joblib in this repo is the fitted filler/injection lexicon. Raw, unprocessed text will still work reasonably, since most tickets are short, but accuracy is measured on the denoised view.

Training details

Base model	sentence-transformers/LaBSE
Epochs	5 (selected by a composite score: 0.5·primary-F1 + 0.2·secondary-F1 + 0.3·urgent-F1)
Batch size	16
Learning rate	3e-5 (encoder), 1e-3 (heads)
Schedule	linear, 10% warm-up
Max sequence length	192 tokens
Precision	fp16 mixed precision (AMP) on a T4 GPU
Loss	class-weighted CE (primary) + 0.5·BCE (secondary) + 0.5·BCE, pos_weight=3 (urgent) + 0.1·CE (language, training-only)
Final fit data	train + validation, 4,800 tickets
Hardware	free T4 GPU, Google Colab / Kaggle

Data

Trained on a synthetic, organiser-provided dataset of 4,800 RideEat (fictional ride/food-delivery app) support tickets across 11 categories and 6 language styles (English, Sinhala, Tamil, Singlish, Tanglish, mixed script). The data is synthetic and its Sinhala/Tamil text was not independently reviewed by native speakers — see Limitations.

Limitations

• Trained on synthetic data; real-world tickets will differ in distribution and phrasing.
• Secondary-category prediction is the weakest output on genuinely new phrasings (fresh-set F1 ≈ 0.33).
• Singlish is the weakest language style in out-of-sample testing.
• No independent native-speaker review of the Sinhala/Tamil training text.
• Confidence is well-calibrated on this data (ECE ≈ 0.04) but calibration on real tickets is unverified.

Citation

@misc{zylo7sentinel2026,
  title  = {Zylo7 Sentinel: A Multilingual Multi-Task Ticket Classifier},
  author = {Dhananjaya, Chamod and Sandaru, Shen and Harishchandra, Asiri and Kavinda, Ashan},
  year   = {2026},
  note   = {TensorForge 2.0, Phase 2},
  url    = {https://github.com/dhananjaya-hbc/tensorforge-2.0-zylo7}
}

License

Apache 2.0, matching the base LaBSE model's license.