vertox-ai/nemotron-streaming-asr-multilingual

Nemotron Streaming ASR

Minimal local WebSocket runtime and clients for NVIDIA Nemotron 3.5 ASR Streaming FP32 ONNX artifacts.

This package contains Python runtime code, WebSocket clients, audio frontend utilities, and validation scripts. It does not package model weights, ONNX files, debug captures, local logs, release zips, or reference repos.

Install

CPU server/client install:

python -m pip install -e '.[test]'

Microphone client support:

python -m pip install -e '.[mic,test]'

GPU/CUDA server support is optional:

python -m pip install -e '.[gpu,test]'

Use docs/gpu_cuda.md to verify onnxruntime and CUDA provider availability before starting a CUDA server.

Model Directory Contract

The server expects --model-dir models by default. The directory is intentionally local and ignored by .gitignore; do not include it in package artifacts. Required files:

• NemoPreprocessor128.onnx
• encoder-model.onnx
• encoder-model.data
• decoder_joint-model.onnx
• tokenizer.model
• vocab.txt
• config.json
• io_contract.json

The runtime validates the ONNX IO contract, cache inputs, prompt_index, vocabulary shape, and blank token placement before inference.

Commands

Start the server on CPU or CUDA:

nemotron-asr-server --model-dir models --execution-provider auto
nemotron-asr-server --model-dir models --execution-provider cpu
nemotron-asr-server --model-dir models --execution-provider cuda

Stream a WAV file:

nemotron-asr-client-file --url ws://127.0.0.1:8765 --wav examples/sample.wav --language en-US --print-transcript

Run the microphone client:

nemotron-asr-client-mic --url ws://127.0.0.1:8765 --language en-US --frontend-preset nemotron_safe --print-transcript

Run the local evaluation matrix when model files and a reference transcript are available:

nemotron-asr-eval-matrix --wav examples/sample.wav --reference examples/sample_transcript.txt --model-dir models

Debug utilities remain under scripts/; see docs/package_contents.md.

Provider Selection

nemotron-asr-server --execution-provider auto is the default. auto uses CUDAExecutionProvider with CPU fallback when ONNX Runtime reports CUDA, otherwise it uses CPUExecutionProvider. cpu forces CPU. cuda requires CUDA and exits with a clean user-facing error if unavailable.

The selected provider list is logged at server startup and model load.

Recommended ASR Modes

Use continuous ASR input with nemotron_safe as the current stable candidate:

nemotron-asr-client-mic --frontend-preset nemotron_safe --asr-input-mode continuous --print-transcript

vad_island with island_fast is available for latency experiments, but it remains experimental:

nemotron-asr-client-mic --frontend-preset nemotron_safe --asr-input-mode vad_island --vad-island-send-policy island_fast --print-transcript

Silence dropping/gating is disabled by default.

EC2 Server + WSL Client Quickstart

On EC2, bind the server to loopback and use SSH port forwarding from the WSL machine:

nemotron-asr-server --host 127.0.0.1 --port 8765 --model-dir models --execution-provider auto

From WSL:

ssh -N -L 8765:127.0.0.1:8765 ubuntu@<ec2-public-dns>
nemotron-asr-client-mic --url ws://127.0.0.1:8765 --frontend-preset nemotron_safe --print-transcript

Do not hardcode public IP addresses in repo files; use environment-specific shell history or deployment configuration.

Privacy

Audio capture flags are opt-in because they may write private speech to disk. Avoid --capture-input-wav, --capture-raw-wav, --capture-processed-wav, --capture-received-audio-dir, and transcript logging unless the capture location and retention policy are clear.

Validation

Before claiming a release or handoff, run:

python -m compileall nemotron_asr scripts realtime_test.py realtime_test_mic.py nemotron_server.py nemotron_client_file.py nemotron_client_mic.py
pytest -q
python scripts/quality_gate.py --model-dir models --coverage-min 80

Citation

BibTeX

@software{lumbantobing2026nemotronstreamingasrmultilingual,
  author       = {Patrick Lumbantobing},
  title        = {Nemotron Streaming ASR Multilingual},
  year         = {2026},
  organization = {VertoX AI},
  url          = {https://huggingface.co/vertox-ai/nemotron-streaming-asr-multilingual},
  license      = {OpenMDW-1.1}
}

Author

Patrick Lumbantobing
VertoX AI