Instructions to use nrl-ai/vn-diacritic-vit5-base-onnx-int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nrl-ai/vn-diacritic-vit5-base-onnx-int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nrl-ai/vn-diacritic-vit5-base-onnx-int8")

# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("nrl-ai/vn-diacritic-vit5-base-onnx-int8")
model = AutoModelForSeq2SeqLM.from_pretrained("nrl-ai/vn-diacritic-vit5-base-onnx-int8", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nrl-ai/vn-diacritic-vit5-base-onnx-int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nrl-ai/vn-diacritic-vit5-base-onnx-int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nrl-ai/vn-diacritic-vit5-base-onnx-int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/nrl-ai/vn-diacritic-vit5-base-onnx-int8

SGLang

How to use nrl-ai/vn-diacritic-vit5-base-onnx-int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nrl-ai/vn-diacritic-vit5-base-onnx-int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nrl-ai/vn-diacritic-vit5-base-onnx-int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nrl-ai/vn-diacritic-vit5-base-onnx-int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nrl-ai/vn-diacritic-vit5-base-onnx-int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use nrl-ai/vn-diacritic-vit5-base-onnx-int8 with Docker Model Runner:
```
docker model run hf.co/nrl-ai/vn-diacritic-vit5-base-onnx-int8
```

nrl-ai/vn-diacritic-vit5-base-onnx-int8 — ONNX int8 quantization of nrl-ai/vn-diacritic-vit5-base

Dynamic int8-quantized ONNX export of nrl-ai/vn-diacritic-vit5-base. 75 % smaller on disk (530 MB safetensors → 307 MB ONNX int8) and no PyTorch dependency at inference time — runs on plain onnxruntime for CPU / browser / mobile deployment.

Quality on the OOD eval (n=150, hand-curated)

Same 6-slice OOD eval the source model was measured against (nrl-ai/vn-spell-correction-eval-real):

Slice	This (int8)	Source (fp32)	Δ
`forum_25`	42.25 %	43.54 %	-1.29 pp
`mobile_25`	77.56 %	76.99 %	+0.57 pp
`telex_real_25`	14.37 %	14.37 %	+0.00 pp
`ocr_25`	94.83 %	94.83 %	+0.00 pp
`legal_real_25`	92.25 %	93.02 %	-0.77 pp
`news_real_25`	95.56 %	96.05 %	-0.49 pp
Aggregate	70.81 %	71.15 %	-0.34 pp

Quantization cost on aggregate: -0.34 pp word accuracy. Within the bootstrap CI overlap of the source model — no measurable quality loss.

Disk size

Format	Size
Source safetensors (PyTorch fp32)	530 MB
ONNX fp32 (export, before quant)	1220 MB
ONNX int8 (this artifact)	307 MB

The fp32 ONNX export is larger than the safetensors because it unrolls the decoder twice (with-cache and without-cache paths). After int8 weight quantization, the total is comfortably under the PyTorch baseline.

Loading

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("nrl-ai/vn-diacritic-vit5-base-onnx-int8")
model = ORTModelForSeq2SeqLM.from_pretrained("nrl-ai/vn-diacritic-vit5-base-onnx-int8")

inp = tok("Toi yeu Viet Nam, dat nuoc tuyet voi", return_tensors="pt")
out = model.generate(**inp, max_length=128, num_beams=1)
print(tok.decode(out[0], skip_special_tokens=True))
# "Tôi yêu Việt Nam, đất nước tuyệt vời"

pip install optimum[onnxruntime]

No PyTorch dependency required at inference time — optimum pulls onnxruntime (and transformers for the tokenizer / config).

When to use this vs the source model

Use this when shipping to CPU-only servers, edge devices, browser (via onnxruntime-web), or mobile (onnxruntime-mobile). The 307 MB / no-PyTorch footprint matters there.
Use nrl-ai/vn-diacritic-vit5-base when running on GPU and PyTorch is already in the deployment. CUDA-accelerated fp16 will out-throughput int8 ONNX on a modern GPU.

Limitations

Same training distribution as the source. All caveats from the source model card apply — in-distribution synthetic eval over-states real-world performance, Vietnamese forum slang and real Telex keystrokes are still the hardest slices.
Dynamic int8 only. Static int8 (with calibration on a held-out set) could squeeze further size at risk of quality. Not done here because the dynamic version already meets the no-quality-loss bar.
Beams = 1 verified. Beam search > 1 should work but isn't benched in this card.

Reproduce

git clone https://github.com/nrl-ai/nom-vn.git
cd nom-vn
pip install -e ".[diacritic-hf]"
pip install optimum[onnxruntime]

# Re-export
python training/onnx_export/export_int8.py \
    --source nrl-ai/vn-diacritic-vit5-base \
    --output training/onnx_export/vn-diacritic-vit5-base-onnx-int8

# Re-bench against the OOD eval
python training/onnx_export/bench_int8.py \
    --model training/onnx_export/vn-diacritic-vit5-base-onnx-int8 \
    --json benchmarks/results/baseline_real_spell_correction_small_onnx_int8.json

License & attribution

Released under Apache 2.0 — same as the source model.

@misc{nom_vn_spell_correction_onnx_int8_2026,
  title={Vietnamese Spell Correction — ONNX int8 quantization for edge deployment},
  author={Nguyen, Viet-Anh and {Neural Research Lab}},
  year={2026},
  howpublished={\url{https://huggingface.co/nrl-ai/vn-diacritic-vit5-base-onnx-int8}}
}

Model tree for nrl-ai/vn-diacritic-vit5-base-onnx-int8

Base model

VietAI/vit5-base

Finetuned

nrl-ai/vn-diacritic-vit5-base

Quantized

(1)

this model

nrl-ai
/

vn-diacritic-vit5-base-onnx-int8