samuli/mateli-punct-onnx

Mateli-punct (ONNX): punctuation and casing for Finnish speech recognition

This is the ONNX export of Mateli-punct, which adds punctuation and capitals to the output of the Mateli speech recogniser. It is meant for phones and CPUs. The model, its input format, training data, evaluation and limitations are described on the Mateli-punct model card. This page covers only what is different here.

	samuli/mateli-punct	this repository
runtime	PyTorch + transformers	ONNX Runtime + tokenizers (no PyTorch)
weights	fp32 (115 MiB)	int8 (38 MiB) or fp32 (115 MiB)
one call on 69 words, 1 desktop CPU thread		about 15 ms (int8 and fp32)

punctuate.py has the same interface as in the PyTorch repository, and only the model call differs. On the Mateli-punct test sets, the fp32 export gives the same label as the PyTorch model for 99.9 % of the words and the int8 model for 99.0 to 99.2 %. In the live setting with pauses, the int8 model's boundary F1 is 0.001 lower.

Examples

The text examples on the Mateli-punct card give the same output with both ONNX models.

Together with mateli-onnx on two public clips (examples). The recognition errors (repo, öö, tiedonainnosta and others) come from the recogniser. The punctuation model only adds punctuation and capitals, using the pauses.

import sys
sys.path[:0] = ["mateli-onnx", "mateli-punct-onnx"]
from transcribe import Mateli, load_audio
from punctuate import Punctuator

asr, punct = Mateli("mateli-onnx"), Punctuator("mateli-punct-onnx")
for name in ["voxpopuli_1", "voxpopuli_2"]:
    words = asr.transcribe(load_audio(f"mateli-onnx/examples/{name}.wav")).words
    pauses = [b.start - a.end for a, b in zip(words, words[1:])] + [None]   # silence after each word
    print(punct.punctuate([w.text for w in words], pauses))

Repo. Äänestin tämän mietinnön puolesta. Öö vaikka komission tiedonainnosta on kulunut jo yli kymmeneen vuotta, eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan.
Boiko toiminen vaaleissa ja mielenosoitus ovat eri asioita, mutta erittäin huolestuttavia ovat tiedot siitä, että useampi kuin joka kymmenen säännöstyspaikka ei voinut avata oviaan opposition saannon takia.

Live, word by word. A word is written out once three more words have been heard, and the model gets up to 64 earlier words as context. The first column shows how much audio had arrived when the text came out.

import sys
sys.path[:0] = ["mateli-onnx", "mateli-punct-onnx"]
from transcribe import Mateli, load_audio
from punctuate import Punctuator

asr, punct = Mateli("mateli-onnx"), Punctuator("mateli-punct-onnx")
audio = load_audio("mateli-onnx/examples/voxpopuli_2.wav")
words, text, prev_end = asr.stream(), punct.stream(), None
for i in range(0, len(audio), 1600):                       # 100 ms blocks, as from a microphone
    for w in words.accept(audio[i:i + 1600]):
        out = text.push(w.text, None if prev_end is None else w.start - prev_end)
        prev_end = w.end
        if out:
            print(f"{(i + 1600) / 16000:5.1f} s  {out}")
for w in words.finish():
    text.push(w.text, None if prev_end is None else w.start - prev_end)
    prev_end = w.end
print("  end  ", text.flush())

  3.6 s  Boiko toiminen
  6.1 s  vaaleissa ja mielenosoitus ovat eri
  8.1 s  asioita, mutta erittäin huolestuttavia ovat
 10.9 s  tiedot siitä, että useampi kuin
 14.5 s  joka kymmenen säännöstyspaikka ei voinut
  end   avata oviaan opposition saannon takia.

How to use

pip install numpy onnxruntime tokenizers
hf download samuli/mateli-punct-onnx --local-dir mateli-punct-onnx
python mateli-punct-onnx/punctuate.py "mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot"

Punctuator(model_dir, precision="int8", threads=1) loads model.int8.onnx, or model.onnx with precision="fp32". The ONNX graph takes input_ids and attention_mask (int64, [batch, seq], at most 512 tokens) and returns logits [batch, seq, 15]. The input format is described on the Mateli-punct card.

Files

file	size	what
model.int8.onnx	38 MiB	dynamic int8 (MatMul weights)
model.onnx	115 MiB	fp32
tokenizer.json		WordPiece tokenizer, 26,020 pieces
config.json		labels and pause marker settings
punctuate.py		inference: whole texts, raw labels, live streaming

License

The model weights and the code in this repository are licensed under CC BY-NC 4.0. The base model Finnish-NLP/electra-base-generator-finnish is Apache-2.0.