Mateli-punct: punctuation and casing for Finnish speech recognition

This model adds punctuation (, . ? !) and capitals to lowercase, unpunctuated Finnish text from a speech recogniser. It was made for the output of Mateli, a streaming Finnish recogniser, and can use the pauses between words from the recogniser's word times. Words are never changed, added or removed. The model is small (30 M parameters) and works word by word on a live stream as well as on whole texts. An int8 ONNX export for phones and CPUs is in samuli/mateli-punct-onnx.

The approach is inspired by syvai/danish-pnc. It uses token classification with one label per word and an ELECTRA encoder, and it is evaluated with the same metrics.

Examples

python punctuate.py "<text>":

input	output
mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot	Mä luulen, et se on ihan hyvä juttu. Mut en mä tiiä, mitä mieltä sä oot.
huomenna mennään mökille jos sää on hyvä otatko sä kalavehkeet mukaan	Huomenna mennään mökille. Jos sää on hyvä, otatko sä kalavehkeet mukaan?
no niin aloitetaan kokous onko kaikki paikalla hyvä ensimmäisenä asialistalla on budjetti	No niin, Aloitetaan kokous. Onko kaikki paikalla? Hyvä. Ensimmäisenä asialistalla on budjetti.
suomen pääkaupunki on helsinki ja siellä asuu noin kuusisataatuhatta ihmistä	Suomen pääkaupunki on Helsinki, ja siellä asuu noin kuusisataatuhatta ihmistä.
kiitos paljon tästä keskustelusta nähdään taas ensi viikolla	Kiitos paljon tästä keskustelusta. Nähdään taas ensi viikolla.

Mistakes happen, as with No niin, Aloitetaan, which should be No niin, aloitetaan.

Together with the recogniser, on two public clips (CC0, from the Mateli examples). The recognition errors (repo, öö, tiedonainnosta and others) come from the recogniser. The punctuation model only adds punctuation and capitals, using the pauses.

import sys
sys.path[:0] = ["mateli", "mateli-punct"]
from mateli import Mateli, load_audio
from punctuate import Punctuator

asr, punct = Mateli.from_pretrained("mateli"), Punctuator("mateli-punct")
for name in ["voxpopuli_1", "voxpopuli_2"]:
    words = asr.transcribe(load_audio(f"mateli/examples/{name}.wav")).words
    pauses = [b.start - a.end for a, b in zip(words, words[1:])] + [None]   # silence after each word
    print(punct.punctuate([w.text for w in words], pauses))

Repo. Äänestin tämän mietinnön puolesta. Öö vaikka komission tiedonainnosta on kulunut jo yli kymmeneen vuotta, eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan.
Boiko toiminen vaaleissa ja mielenosoitus ovat eri asioita, mutta erittäin huolestuttavia ovat tiedot siitä, että useampi kuin joka kymmenen säänestyspaikka ei voinut avata oviaan opposition saannon takia.

A live, word-by-word example is on the ONNX model card.

How to use

pip install torch transformers tokenizers
hf download samuli/mateli-punct --local-dir mateli-punct
python mateli-punct/punctuate.py "mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot"

From Python (run next to the downloaded mateli-punct directory):

import sys
sys.path.insert(0, "mateli-punct")
from punctuate import Punctuator

p = Punctuator("mateli-punct")
print(p.punctuate("mä luulen et se on ihan hyvä juttu mut en mä tiiä mitä mieltä sä oot"))
print(p.predict("onko kaikki valmista".split()))

Mä luulen, et se on ihan hyvä juttu. Mut en mä tiiä, mitä mieltä sä oot.
['O|U', 'O', '?']

punctuate.py (about 220 lines) has three entry points:

• Punctuator.punctuate(text_or_words, pauses=None) punctuates whole texts of any length, in overlapping windows of 150 words.
• Punctuator.predict(words, pauses) returns the raw labels.
• Punctuator.stream() is for live use. push(word, pause_before) returns the text committed so far, and flush(pause_after) commits the rest. A word is committed once three more words are known.

pauses[i] is the silence after word i in seconds, None if unknown. The model loads with AutoModelForTokenClassification, but punctuate.py builds the input ids itself because of the pause markers. The code is meant to be read and adapted.

Input format

punctuate.py builds the model input itself. This is how, for porting it to another language. As an example, take six words from the recogniser and the silence after each word:

word	onko	kaikki	paikalla	hyvä	aloitetaan	kalavehkeet
pause after (s)	0.05	0.20	0.90	0.35	0.10	unknown

The model gets this token sequence (at most 512 tokens per call):

[CLS] [unused0] onko kaikki [unused1] paikalla [unused4] hyvä [unused2] aloitetaan kala ##veh ##keet [SEP]

• [unused0] right after [CLS] tells the model that pauses are given. Without pauses, leave it out and add no pause markers. The model works both ways.
• Each word is split into pieces with the WordPiece tokenizer in tokenizer.json (cased, no accent stripping). Most words are a single piece, and rarer ones are split, like kala ##veh ##keet.
• After the last piece of a word comes a pause marker for the silence that follows it. Silences shorter than 0.14 s and unknown ones get no marker. The bucket edges are in config.json (pause_markers).

silence after the word	marker
under 0.14 s, or unknown	none
0.14 to 0.30 s	[unused1]
0.30 to 0.50 s	[unused2]
0.50 to 0.78 s	[unused3]
0.78 to 1.18 s	[unused4]
1.18 s or more	[unused5]

The label of a word is read from its first piece (kala for kalavehkeet), and the outputs on the other pieces and on the markers are ignored. Each of the 15 labels (id2label in config.json) combines the punctuation after the word (O for none, , . ? !) with its casing (nothing for lowercase, |U for a capital first letter, |A for all capitals):

word	onko	kaikki	paikalla	hyvä	aloitetaan	kalavehkeet
label	O|U	O	?	.|U	O|U	.

Applied to the words, the labels give:

Onko kaikki paikalla? Hyvä. Aloitetaan kalavehkeet.

punctuate.py also capitalises the first word and every word after . ? !, in case the model misses one.

Files

file	what
model.safetensors	weights, fp32 (115 MiB)
config.json	ELECTRA config with id2label and pause_markers
tokenizer.json, tokenizer_config.json	WordPiece tokenizer, 26,020 pieces
punctuate.py	inference: whole texts, raw labels, live streaming

Model

ELECTRA, 12 layers, hidden size 256, 4 attention heads, 30 M parameters (20 M of them in the embeddings). Initialised from Finnish-NLP/electra-base-generator-finnish (Apache-2.0) with the vocabulary pruned to the 26,000 pieces the training text uses (plus the marker tokens), then trained by distillation from a larger BERT-based punctuation model fine-tuned for the same task, together with the gold labels.

Training data

Written Finnish with punctuation, and transcribed speech. For the speech, the input is the output of the Mateli recogniser, the punctuation and casing labels were transferred from human transcripts of the same audio by word alignment, and the pauses come from the recogniser's word times. Written text was lowercased and stripped of punctuation for the input, with simulated recognition noise. Windows were 16 to 160 words starting at random positions, so the model also handles text that does not begin at a sentence start.

Evaluation

The test sets are held-out speech transcribed by the Mateli recogniser, with labels transferred from human transcripts of the same audio. The podcast set has shows not seen in training. The metrics follow syvai/danish-pnc. Each is an F1 score per word, for a sentence boundary (. ? ! interchangeably), comma, question mark and casing (capitalised or all caps vs lowercase).

set	input	mode	boundary F1	boundary P	boundary R	comma F1	question F1	casing F1
conversational podcasts	words	whole text	0.666	0.638	0.696	0.700	0.594	0.702
conversational podcasts	words + pauses	whole text	0.680	0.653	0.710	0.702	0.614	0.714
conversational podcasts	words + pauses	live	0.666	0.644	0.691	0.699	0.591	0.705
TV programmes	words	whole text	0.709	0.743	0.678	0.582	0.552	0.739
TV programmes	words + pauses	whole text	0.756	0.780	0.733	0.598	0.607	0.772
TV programmes	words + pauses	live	0.753	0.780	0.727	0.600	0.602	0.769
long conversations	words	whole text	0.683	0.668	0.698	0.697	0.669	0.714
long conversations	words	live	0.657	0.644	0.672	0.692	0.659	0.700

Live is the streaming policy of Punctuator.stream() (three words of lookahead, 64 words of context, a model call every five words). On the podcast set a word is committed on average 2.4 s after it is spoken (90th percentile 3.4 s).

The scores are well below what the same kind of model reaches on written text, for two reasons. The input has recognition errors, and the reference labels come from human transcribers, who disagree among themselves on spontaneous speech, for example on whether ja, mut or nii starts a new sentence and where commas go in long spoken sentences.

Limitations

• Trained for speech recognition output in Finnish. It also punctuates written text that has lost its punctuation, but that was not the target.
• Sentence boundaries in disfluent conversation and question marks are the weakest points.
• Casing is limited to a capital first letter or all capitals (no iPhone-style mixed case).
• The pauses must come from the same kind of word timing as Mateli's (end of a word's last character to the start of the next word's first character, in 40 ms steps).

License

The model weights and the code in this repository are licensed under CC BY-NC 4.0. The base model Finnish-NLP/electra-base-generator-finnish is Apache-2.0.