Punctuate-All (12 languages)

kredor's XLM-RoBERTa-base fine-tuned on Europarl for punctuation restoration, exported for loom.cpp. Family 12: text in, one class per token out -- here the class is the mark that follows the token. Twelve languages against FullStop's four, and half the size, because the encoder is XLM-R base rather than large.

This is a loom.cpp export: a single self-describing GGUF that carries its own graph topologies, tokenizer (if any) and driver script, produced by loom-exporter.

Original model

Exported from kredor/punctuate-all. Weights are unmodified; this repo packages the same parameters into loom.cpp's GGUF format.

License

mit, inherited from the base model above.

Language(s)

en, de, fr, es, bg, it, pl, nl, cs, pt, sk, sl

Usage

Run it with loom-py -- loom-py-rt on PyPI:

pip install -U "loom-py-rt[hub]"

import loom

model = loom.Model.from_pretrained("loom-ai-org/punctuate-all-loom")

result = model.text2class.infer("hello my name is wolfgang and i live in berlin do you know it")
print(result.text)
# hell/, o/, my/0 name/0 is/0 ... berlin/. do/0 you/0 know/0 it/?

# Every class this checkpoint can choose between, in the order its ids run. "0" is "no mark here",
# which is most tokens in most sentences:
print(result.labels)
# ['0', '.', ',', '?', '-', ':']

# Restoring the text is the point of this model, and the rule is one line long: a mark belongs to the
# WORD, so it is the label on the word's LAST piece. A piece starts a new word when decoding it
# together with the piece before puts a space between them -- which is what the vocabulary knows and
# the piece text alone does not.
restored = ""
for i, token in enumerate(result):
    following = result[i + 1] if i + 1 < len(result) else None
    restored += token.piece
    if following is None or " " in model.detokenize([token.token, following.token]):
        if token.label != "0":
            restored += token.label
        restored += " "
print(restored.strip())
# hello, my name is wolfgang and i live in berlin. do you know it?

# The framing tokens the encode adds (<s> and </s>) are dropped for you, on the ids the file declares
# rather than on their spelling. Ask for the raw alignment if you want them:
raw = model.text2class.infer("hello my name is wolfgang and i live in berlin do you know it",
                             strip_special=False)
print(len(raw), "rows including <s> and </s>, against", len(result), "without")

The layer underneath

The call above is the high-level door: one per task, named for the modality pair it maps between, with the windowing, sampling and assembly this model needs already applied. Under it, model.infer(...) passes your arguments straight to the driver this GGUF embeds -- which is where you go for a knob the door does not name.

model.driver_source prints that driver, including a header comment documenting every argument it accepts for this model, and is the authority on it. See loom-py for the API and loom.cpp for what the engine does between the two.

Known limitations

Trained on Europarl, which is parliamentary proceedings: formal, complete sentences in a register that is not chat, not code and not casual speech. It restores six classes and nothing else (., ,, ?, -, :, and 0 for no mark), so it will not give you semicolons, quotation marks or apostrophes, and it does not capitalise -- truecasing is a different head.

Two of the six classes are weak and the upstream card says so. Its own report gives F1 0.99 for no-mark, 0.95 for . and 0.86 for ,, but 0.39 for - and 0.58 for : -- the confusion matrix has half of all true - predicted as ,. Treat the hyphen and colon classes as advisory.

Feed it text with the punctuation already removed. Given punctuated input it still labels every token and you get marks on top of marks.

The labels line up with the tokenizer's PIECES, not with your words -- a SentencePiece vocabulary splits wolfgang into three -- and the mark you want is the one on a word's LAST piece. The usage snippet above does that walk; the export hands back the pieces alongside the labels rather than guessing at the rule for you.

The export takes one sequence at a time and no padding, so there is no batch dimension to fill and no attention mask to pass. Sequences are capped at 512 tokens by the checkpoint's own learned position table.