Auslan → English live translator (ONNX, for in-browser inference)
Research models from a UWA Honours project on gloss-free Auslan (Australian Sign Language) to English translation. These files power the browser studio at https://auslan-research.vercel.app/studio, where a visitor's own browser runs the whole pipeline with ONNX Runtime Web: camera frames never leave the device.
This is a research prototype, not an interpreter. It has not been evaluated on live webcam signing or with Deaf Auslan users. See Limitations.
What is in this repository
| Path | What it is | Licence |
|---|---|---|
translators/w4A_s202/ |
Single best model: pose-only Transformer (encoder + one-step decoder graphs, vocabulary) | CC BY-NC-SA 4.0 |
translators/{vlp80,rgb_fusion,vlp40,ctc_s202,fus_s101,fus_s202}/ |
The six members of the ensemble (S6); three also consume RGB crop features | CC BY-NC-SA 4.0 |
pose/yolox_m.onnx, pose/rtmw_256x192.onnx |
DWPose front end (YOLOX-m person detector, RTMW-DW-x-l whole-body pose) — unmodified copies of the files distributed by OpenMMLab via rtmlib 0.0.16 | Apache-2.0 (OpenMMLab / rtmlib) |
manifest.json |
Operating points: members, beam size, length penalty, confidence thresholds, file SHA-256 | CC BY-NC-SA 4.0 |
The ensemble's crop features use DINOv2-small, which is not stored here: the browser loads the public ONNX export
onnx-community/dinov2-small at revision 8b1f705a (Apache-2.0, Meta), whose CLS output matches the
facebook/dinov2-small revision ed25f3a3 pooler output used in training (313 crops, minimum cosine 0.9999998).
Operating points
Both decode with beam search over the averaged per-token log-probabilities of their members; the hypothesis is chosen
by raw / words^LP. Confidence is self cross-entropy, −raw / (words + 1) nats; the system answers only when it is at
or below the model's own threshold, fitted on the 800 validation clips at a target answer rate (default 50%).
Single best model (w4A_s202) |
Six-model ensemble (S6) | |
|---|---|---|
| Recipe | Pose augmentation + 1.1:1 duration-matched synthetic sequences (adopted recipe), wave-4 seed 202; the validation-best of 21 adopted-recipe runs | Three pose-only (VLP-80, VLP-40, VLP→CTC seed 202) + three pose+RGB fusion members |
| Decode | beam 10, LP 0.7 | beam 10, LP 0.3 (selected on validation) |
| Auslan-Daily Communication, BLEU-4 (validation / official test) | 12.643 / 11.652 | 12.470 / 11.802 (WER 0.907) |
| Threshold at 50% / 75% / 90% answer rate (nats) | 0.7067 / 0.8598 / 0.9743 | 0.9048 / 1.0889 / 1.2366 |
| Test clips answered at the 50% threshold, and their BLEU-4 | 49.9% at 22.01 | 51.9% at 19.07 |
Answered-subset scores are not comparable to full-coverage scores: answered output skews toward short, common phrases. 28.75% of test clips have a reference sentence that also appears in training; on the other 570 clips S6 scores 6.60.
Faithfulness of these exports
Every number above was measured with the original PyTorch checkpoints. The ONNX graphs and the browser engine are accepted only after reproducing them (validation 12.643 and 12.470 with identical hypotheses on all 800 clips). In the browser (Chrome 152 on an Apple M3 Max, ONNX Runtime Web 1.30.0), both operating points reproduced those scores exactly on the 800 stored validation clips on WASM and on WebGPU, with identical hypotheses and confidences within 2.3e-6 nats; and in 20 end-to-end studio runs (10 validation videos × 2 models) the browser gave the same text and the same answer-or-decline decision as the Python service on the same camera frames.
Limitations
- Trained on 12,439 conversational clips dominated by two recurring performers; other signers scored about 3 BLEU-4 in the benchmark even with the ensemble. A new webcam user is outside that group.
- The confidence gate cannot tell whether anyone is signing: in a pre-registered test, people who were not signing received a translation about as often as signers.
- Utterance-final only (sign, then Finish). There is no streaming or partial translation.
- English output only; no Auslan grammar or non-manual features are modelled explicitly.
Data and licence
Trained on Auslan-Daily (Communication split; CC BY-NC-ND 4.0) and on synthetic sequences built from MM-WLAuslan isolated signs (CC BY-NC-SA 4.0). The dataset authors confirmed both datasets are fully open to the public (email from Xin Shen, 24 April 2026). The translator weights are released under CC BY-NC-SA 4.0: non-commercial use, attribution to the datasets below, and share-alike. The Auslan-Daily licence carries a no-derivatives term; whether trained weights are derivatives of the data is unsettled, and these weights are shared for non-commercial research use only.
@inproceedings{shen2023auslandaily,
title = {{Auslan-Daily}: {Australian} Sign Language Translation for Daily Communication and News},
author = {Shen, Xin and Yuan, Shaozu and Sheng, Hongwei and Du, Heming and Yu, Xin},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track},
year = {2023}
}
@inproceedings{shen2024mmwlauslan,
title = {{MM-WLAuslan}: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset},
author = {Shen, Xin and Du, Heming and Sheng, Hongwei and Wang, Shuyun and Chen, Hui and Chen, Huiqiang and Wu, Zhuojie and Du, Xiaobiao and Ying, Jiaying and Lu, Ruihan and Xu, Qingzheng and Yu, Xin},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track},
year = {2024}
}
Author: Preet Prasad (UWA), supervised by Dr Atif Mansoor.