berturk-punctuation-restoration

Fine-tuned dbmdz/bert-base-turkish-cased (BERTurk base cased) with a token classification head for Turkish punctuation and casing restoration. No existing punctuation checkpoint was used; only the pretrained BERTurk encoder weights and its original tokenizer.

Task formulation

• Input: whitespace-split words, Turkish-lowercased, without ,.;:!?.
• One label per word, predicted on the word's last subword token; other subwords and [CLS]/[SEP]/[PAD] use ignore index -100.
• Label = CASE|PUNCT: casing of the word and the punctuation mark that follows it.
• Words are never inserted, deleted, reordered or respelled; only casing and the trailing mark change.
• Inputs longer than 64 subword tokens are split into overlapping word windows (12 words overlap); each word takes the prediction from the window where it has the most context.

id	label	casing	appended
0	LOWER|O	keep lowercase	–
1	CAP|O	capitalize first letter (Turkish i/ı aware)	–
2	UPPER|O	uppercase whole word	–
3	LOWER|COMMA	keep lowercase	,
4	CAP|COMMA	capitalize first letter (Turkish i/ı aware)	,
5	LOWER|PERIOD	keep lowercase	.
6	LOWER|QUESTION	keep lowercase	?
7	LOWER|EXCLAMATION	keep lowercase	!

Training data

GoktugD/turkish-punctuation-restoration-500k @ fcbb30d342b511fdd8bf1f09bbfef990443752fa (synthetic, template-based; official splits).

split	rows
train	490000
validation	5000
test	5000

Preprocessing: NFC; removal of the stray U+0307 produced by locale-unaware str.lower('İ') in the dataset sources; model inputs are re-derived from the targets with Turkish-aware lowercasing (I→ı, İ→i) and verified word-by-word against the dataset sources (0 alignment errors). Validation/test use first-sentence templates and entities disjoint from train; the second sentence of every row comes from 8 templates shared across splits (QUESTION and EXCLAMATION occur only there).

Training procedure

hyperparameter	value
epochs (trained / max)	3.0 / 3
learning rate	2e-05 (linear, warmup ratio 0.1)
batch size	64 x 1 accumulation
max sequence length	64
weight decay / max grad norm	0.01 / 1.0
loss	cross_entropy
training row concatenation	off
precision	bf16
seed	42
model selection	best validation punct_macro_f1
training time / peak GPU memory	0:28:33 / 2.62 GB (NVIDIA GeForce RTX 5060 Ti)
libraries	torch 2.11.0+cu128, transformers 5.19.0, datasets 4.3.0

Evaluation

Word-level metrics. Punctuation macro F1 excludes O.

metric	validation	test	validation (long input)	test (long input)
Punctuation macro F1 (excl. O)	1.0000	0.9987	0.9210	0.9710
Punctuation macro F1 (incl. O)	1.0000	0.9989	0.9352	0.9755
Punctuation weighted F1	1.0000	0.9992	0.9855	0.9886
O F1	1.0000	0.9995	0.9922	0.9936
Casing macro F1	0.9962	0.9716	0.9736	0.9712
Joint label macro F1	0.9981	0.9856	0.9473	0.9721
Sentence exact match	0.9298	0.5000	0.0000	0.0000

Long input = consecutive rows concatenated into one input and processed with overlapping windows.

Per-class punctuation (test)

class	precision	recall	F1	support
O	0.9991	1.0000	0.9995	96375
COMMA	0.9999	0.9898	0.9948	8750
PERIOD	1.0000	1.0000	1.0000	7500
QUESTION	1.0000	1.0000	1.0000	1250
EXCLAMATION	1.0000	1.0000	1.0000	1250

Confusion matrix (test, rows = gold)

gold \ pred	O	COMMA	PERIOD	QUESTION	EXCLAMATION
O	96374	1	0	0	0
COMMA	89	8661	0	0	0
PERIOD	0	0	7500	0	0
QUESTION	0	0	0	1250	0
EXCLAMATION	0	0	0	0	1250

First sentence (split-specific templates) vs later sentences (shared templates), test

part	words	punct macro F1	O	COMMA	PERIOD	QUESTION	EXCLAMATION
first_sentence	87000	0.9955	0.9994	0.9909	1.0000	–	–
later_sentences	28125	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000

Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

repo = "thealper2/berturk-punctuation-restoration"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()
PUNCT = {"O": "", "COMMA": ",", "PERIOD": ".", "QUESTION": "?", "EXCLAMATION": "!", "COLON": ":", "SEMICOLON": ";"}

def tr_lower(s): return s.replace("I", "ı").replace("İ", "i").lower()
def tr_upper(s): return s.replace("i", "İ").replace("ı", "I").upper()

def restore(text: str) -> str:
    words = [tr_lower(w.strip(",.;:!?")) for w in text.split() if w.strip(",.;:!?")]
    enc = tokenizer(words, is_split_into_words=True, truncation=True, max_length=64, return_tensors="pt")
    with torch.no_grad():
        pred = model(**enc).logits.argmax(-1)[0].tolist()
    labels = {}
    for pos, wid in enumerate(enc.word_ids()):
        if wid is not None:
            labels[wid] = model.config.id2label[pred[pos]]  # last subword of each word wins
    out = []
    for i, w in enumerate(words):
        case, _, punct = labels.get(i, model.config.id2label[0]).rpartition("|")  # case is "" without casing labels
        if case == "CAP":
            w = tr_upper(w[:1]) + w[1:]
        elif case == "UPPER":
            w = tr_upper(w)
        out.append(w + PUNCT[punct])
    return " ".join(out)

print(restore("bugün hava çok güzel dışarı çıkabiliriz"))

Single window only; for longer texts split into overlapping word windows.

Limitations

• Trained and evaluated only on synthetic, template-based text (12 first-sentence and 8 second-sentence templates); scores measure in-distribution performance, not performance on natural or spoken Turkish.
• Only the marks present in the training data are supported: ,, ., ?, !.
• No evaluation on ASR (e.g. Whisper) output; disfluencies, repetitions, recognition errors and digit formatting are unseen.
• Casing of unseen proper nouns is inferred from context only.