handyshed/lark-punctuation

Lark punctuation model

A vocabulary-pruned, fp16 CoreML export of oliverguhr/fullstop-punctuation-multilingual-base (revision fe99fb0dc8d8be674e8bb4e39833605f9a1af651) used by Lark, a native macOS dictation app, to restore . , ? in ASR output on the Apple Neural Engine. Lark downloads this repo on first run.

Files

Path	Purpose
PunctuationModel512.mlmodelc/	Compiled CoreML ML Program, fp16, fixed 512-token input (input_ids, attention_mask)
tokenizer/	XLM-R SentencePiece tokenizer with ids remapped to the pruned vocabulary
manifest.json	File list with sizes and sha256; the app verifies every download against it

Total download ≈ 235 MB (the original fp32 export was 1060 MB).

The model and tokenizer are a matched pair: the tokenizer's ids index the pruned embedding table and will not work with the upstream checkpoint.

Pruning method

research/experiments/prune_punctuation_vocab.py in the Lark repo:

1. Tokenise Lark's English dictation corpus (minus a 20% held-out slice of the benchmark) and record the token ids actually used.
2. Keep those ids plus the highest-scored Latin-script pieces up to 40 000 entries (from 250 002); slice the embedding matrix; rewrite tokenizer.json with the new ids.
3. Export with coremltools as an fp16 ML Program at a fixed 512-token shape. Flexible-shape and enumerated-shape variants fall off the Neural Engine, so a single fixed shape is shipped.
4. Verify CoreML argmax labels equal the PyTorch pruned model on 50 sentences (3666/3666 tokens).

Parameters: 278M → 116M. Zero <unk> tokens on the benchmark.

Evaluation

Word-level precision / recall on Lark's 787-utterance English benchmark, full set and the held-out slice. Pruning does not change quality within noise.

Label	Base, full	Pruned, full	Base, held-out	Pruned, held-out
.	58.8 / 78.3 (n=1758)	58.8 / 78.3 (n=1758)	56.1 / 77.7 (n=336)	56.2 / 77.4 (n=336)
,	49.4 / 59.9 (n=1990)	49.3 / 59.9 (n=1990)	53.2 / 62.5 (n=381)	52.9 / 62.5 (n=381)
?	74.8 / 63.5 (n=819)	74.8 / 63.5 (n=819)	70.4 / 56.6 (n=143)	70.4 / 56.6 (n=143)
all	54.8 / 67.6 (n=4567)	54.8 / 67.6 (n=4567)	54.7 / 67.4 (n=860)	54.6 / 67.3 (n=860)

Intended use

Punctuation restoration for English dictation inside Lark. The classifier still emits the upstream - and : labels but Lark ignores them. Not evaluated on the other languages the base model supports; the pruned vocabulary drops most non-Latin pieces, so expect degraded results outside English.

License and attribution

The base model is released by Oliver Guhr under the MIT License; this derivative keeps that license. Please cite the original work:

@article{guhr-EtAl:2021:fullstop,
  title     = {FullStop: Multilingual Deep Models for Punctuation Prediction},
  author    = {Guhr, Oliver and Schumann, Anne-Kathrin and Bahrmann, Frank and Böhme, Hans Joachim},
  booktitle = {Proceedings of the Swiss Text Analytics Conference 2021},
  year      = {2021},
}