tarun7r/indic-turn-detector

Indic Multilingual Turn Detector

An open-weights text model for contextually-aware end-of-utterance (EOU) detection in voice AI applications, covering English, Hinglish, and seven Indic languages. The model predicts whether a user has finished speaking based on the semantic content of their transcribed speech and conversation context, providing a critical complement to voice activity detection (VAD) systems.

Version	v1.0 (2026-11-07)
Parameters	494M total / 179M trainable
License	Apache-2.0
Base model	Qwen2.5-0.5B-Instruct

📖 Integration: Drop-in replacement for LiveKit's text turn detector (deprecated; slated for removal in the 2.0 SDK) — the EOUTurnDetector helper implements the MultilingualModel interface, and covers the Indic languages LiveKit's text model doesn't (it ships Hindi only).

Table of Contents

• Intended Use
• Overview
• Model Variants
• How It Works
• Architecture and Training
• Supported Languages
• Benchmarks
• Usage
• Deployment Requirements
• License
• Resources

Intended Use

Real-time voice agents and call-center agent assist anywhere an STT pipeline produces user transcripts and the agent must decide when the user's turn is complete.

• Transcript-based: the model scores text from your STT pipeline. No audio is processed, transmitted, or retained by the detector.
• Not a speaker-identification system: the model holds no voice biometrics, embeddings, or identity signal of any kind.
• Nothing retained: inference is stateless per turn; no transcripts, scores, or context are stored.

Overview

VAD detects silence, not meaning — it interrupts whenever the audio pauses. This model scores the transcribed text instead and predicts whether the user has finished their turn, so agents can wait on continuations and commit the turn when semantics say the user is done. Pauses it holds through, where VAD would cut in:

• Mid-turn pauses — "um, I need to think about…" + pause → hold
• Verb-final (SOV) languages — Hinglish "aapko kal phone…" + pause before "kar dunga" → hold; the predicate arrives last, exactly where VAD cuts
• Dangling continuations — "kyunki…", "lekin…", "కాబట్టి…", "કારણ કે…" → hold
• Structured data — reciting "four eight two, seven zero…" (cards, phone numbers, addresses) → hold until complete

Covers 9 language groups — English, Hinglish (hi-Latn), and seven Indic languages (Hindi, Telugu, Tamil, Marathi, Bengali, Gujarati, Punjabi) — trained on MultiWOZ-style dialogues, TOP-style slots, and synthetic continuation/prefix negatives.

Note: LiveKit's Turn Detector v1 now offers audio-based turn detection (9.9% false cut-offs at 300 ms on eot-bench); this model is text-only and suited to STT-transcript stacks.

Model Variants

Artifact	File	Size (approx.)	Notes
ONNX (ship)	model.onnx	~2.0 GB	fp32 graph — the exact graph the thresholds are calibrated on. Output tensor is prob.
PyTorch	model.safetensors	~2.0 GB	Fine-tuned Qwen2ForCausalLM weights plus eou_head.* tensors. Runtime inference uses ONNX.

Per-language thresholds are stored in languages.json, with per-language safe and balanced profiles. Thresholds are calibrated on fp32 scores; the ONNX graph matches the calibration scores to within 1e-6.

config.json / generation_config.json: same roles as the livekit/turn-detector card, but values describe Qwen2.5-0.5B-Instruct (model_type: qwen2, hidden_size: 896) — the same base and values as LiveKit's multilingual model. bos_token_id / eos_token_id are Qwen chat tokens (151643 / 151645).

ONNX I/O matches livekit/turn-detector: input_ids in, per-position prob out — use the last position.

How It Works

The model operates on transcribed text from a speech-to-text (STT) system, not raw audio.

1. Input: Up to 6 recent messages, merged by role, formatted with the Qwen2.5 chat template (<|im_start|> / <|im_end|> delimiters, no system prompt). The final user message is left open — no closing end-of-message token. Context is limited to 256 tokens with whole-turn truncation (complete turns are dropped before any partial token chop).
2. Prediction: A scalar classification head on the last token position outputs P(end of turn). A high probability indicates the user has likely finished their utterance; a low probability indicates they are likely to continue.
3. Thresholding: languages.json stores per-language safe and balanced thresholds. The runtime selects the profile and resolves language keys case-insensitively. For Hinglish, configure the STT model to report hi-Latn — a hi or en tag scores with the Hindi or English threshold.
4. Integration with VAD: VAD handles speech presence detection and interruption triggering; this model provides the semantic signal for when to commit the user's turn to the downstream LLM.

Text Preprocessing

The model applies the LiveKit-style multilingual text normalization before inference:

• NFKC unicode normalization
• Lowercasing
• Punctuation removal (preserving apostrophes and hyphens)
• Whitespace collapsing

Architecture and Training

Base Model

Fine-tuned from Qwen/Qwen2.5-0.5B-Instruct, selected for strong multilingual performance on this task while enabling low-latency CPU inference, with a small scalar classification head on the final token hidden state.

Calibration

Thresholds are tuned on the validation split, wb_clean slice, using fp32 scores:

• balanced: at most 10% false cuts on incomplete-turn negatives
• safe: at most 5% false cuts

Per-language thresholds maximize responsiveness subject to that false-cut budget.

Quantization and Export

The trained checkpoint is exported to a fp32 ONNX graph (model.onnx) for CPU-only inference with ONNX Runtime — the same graph the thresholds are calibrated on. The fp32 graph keeps full end-of-turn discrimination.

Supported Languages

The model supports 9 language groups. The STT provider reports the detected language, which is used to select the appropriate per-language threshold.

Code	Language
en	English
hi	Hindi (Devanagari)
hi-Latn	Hinglish (Latin script; also use when the STT tags Romanized Hindi)
te	Telugu
ta	Tamil
mr	Marathi
bn	Bengali
gu	Gujarati
pa	Punjabi

The runtime matches BCP-47-like language tags exactly first, then the part before the first dash (so te-IN → te).

Benchmarks

Detection Accuracy

TPR is measured at each language's balanced threshold; TNR at one shared operating point (threshold 0.86) on the full validation split (see the notes below the table).

• True positive rate (TPR) — complete user turns of ≥4 words correctly identified as finished (primary responsiveness metric), measured on the wb_clean slice (excludes ambiguous soft-label rows).
• True negative rate (TNR) — all incomplete-turn negatives (any cut type) correctly identified as continuing, reported at one shared operating point (threshold 0.86) on the full validation split — a raw per-language discrimination reading comparable in spirit to the LiveKit published table, which also reports a single operating point.

Language	TPR (≥4 words)	AUC	TNR (shared 0.86)
Hinglish (hi-Latn)	96.9%	98.8%	96.1%
English	94.1%	96.9%	93.9%
Hindi	93.9%	96.2%	96.9%
Marathi	93.6%	97.6%	96.7%
Bengali	91.3%	96.8%	96.9%
Tamil	89.2%	95.5%	95.1%
Telugu	89.1%	96.8%	97.3%
Gujarati	88.4%	96.0%	94.7%
Punjabi	85.3%	95.4%	96.0%

Mean TPR (≥4 words): 91.3% across languages. Per-language AUC on the wb_clean validation slice ranges 0.954–0.988 (mean 0.967).

Note: the two columns use different operating points — TPR at each language's calibrated balanced threshold, TNR at the shared 0.86 threshold. Read the columns separately.

Usage

The model plugs into LiveKit Agents the same way as LiveKit's own text turn detector (livekit-agents~=1.8), with an STT model providing transcripts. The plugin ships inside this repository as a single self-contained module (eou_turn_detector.py) — no extra package install needed.

LiveKit Agents (Python)

Install ONNX Runtime, Transformers, and huggingface_hub:

pip install "livekit-agents~=1.8" onnxruntime transformers huggingface_hub

Download the model (the detector module ships with the weights) and initialize the session:

import sys
from huggingface_hub import snapshot_download
from livekit.agents import AgentSession, TurnHandlingOptions, inference

model_dir = snapshot_download("tarun7r/indic-turn-detector")
sys.path.insert(0, model_dir)
from eou_turn_detector import EOUTurnDetector

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection=EOUTurnDetector(
            model_dir,  # model.onnx + languages.json + eou_turn_detector.py
            profile="balanced",  # or "safe"
        ),
    ),
    stt=inference.STT(language="multi"),
    # ... vad, tts, llm, etc.
)

The detector requires an STT model (transcripts are its input). LiveKit's text model covers Hindi only from the Indic set; this model adds Bengali, Gujarati, Marathi, Punjabi, Tamil, Telugu, and Hinglish.

Standalone ONNX

• Input: input_ids int64 [1, seq]
• Output: prob float32 [1, seq] — use the last position as p_eot
• Compare p_eot against languages.json[lang][profile].

Deployment Requirements

• Runtime: CPU-only (no GPU required). Uses ONNX Runtime with the CPUExecutionProvider.
• RAM: ~2 GB working set for the fp32 weights, plus tokenizer.
• Threads: intra_op ≈ min(cpu/2, 4), inter_op = 1 (LiveKit-style latency tuning).
• Instance type: Use compute-optimized instances (e.g., AWS c6i, c7i). Avoid burstable instances to prevent inference timeouts from CPU credit exhaustion.

Measured latency (model.onnx fp32, CPUExecutionProvider, intra_op=4, inter_op=1, Apple M-series 10-core — reference, not a target cloud instance):

Sequence length	p50	p95
≤64 tokens	51 ms	118 ms
65–128 tokens	163 ms	247 ms
129–256 tokens	338 ms	449 ms

Latency is independent of the profile choice (profiles only change the threshold applied to p_eot).

License

This model is released under the Apache License 2.0. The base model Qwen2.5-0.5B-Instruct is also released under Apache 2.0.

Resources

• LiveKit turn detector documentation: the plugin API this bundle is drop-in compatible with.
• livekit/turn-detector model card: reference card layout and threshold format.
• eot-bench data: real-speech evaluation methodology used for external signal.