karma689/mmbert-yigchung-v2

mmBERT Yigchung v2

Token classifier for Yigchung in Classical Tibetan: the small-print interlinear annotation written under the main line. The model reads a flat base string and returns character spans. It does not see page images.

Fine-tune of jhu-clsp/mmBERT-base (ModernBertForTokenClassification, 22 layers, max length 8192) on karma689/yigchung_detection, revision v2.0 (reviewed OPF, Lotsawa House, Nalanda, and BDRC).

This is a new experiment line. Do not compare these scores to the 0.389 baseline from the earlier 15-class run on karma689/layer_detection.

Weights are the best validation checkpoint, step 2412 (epoch 2.25). Training ran on to step 6700 and was early-stopped; that later checkpoint is not the one published here.

Test predictions and the score file are in this repo:

• result/result.md — scores, the comparison, and every test book
• result/preds.json — one object per test book, inclusive character spans plus the span text
• result/result.json — the same scores as JSON

How to run inference

Spans are inclusive on both ends: text[start:end + 1].

Books are longer than the 8192-token window. Cut the same windows used in training, keep each content token from the first window that covers it, then Viterbi-decode. break_penalty=8 is the value chosen on the validation split of this checkpoint.

Setting	Value
Window length	8192 tokens (2 specials + 8190 content)
Overlap / step	3276 / 4914
Labels	0 O, 1 B-YIGCHUNG, 2 I-YIGCHUNG
Stitch	token is scored by the first window that contains it
Decode	Viterbi. O → I is illegal. Leaving a span (B/I → O) costs break_penalty
Selected penalty	8 (validation F1 0.667; test scored once at this value)

The repo script src/script/infer_yigchung.py still defaults to karma689/mmbert-yigchung and break_penalty=1. For this checkpoint pass the v2 model id and penalty 8.

Results

Greedy one-to-one span match at IoU ≥ 0.5. These are the training-protocol scores (no whitespace trim) and match result/result.json.

Split	Precision	Recall	F1	TP	FP	FN
validation, penalty 8	0.683	0.653	0.667	1721	800	916
test, penalty 8	0.782	0.441	0.564	1789	500	2270

Validation F1 by break_penalty on this checkpoint: 0 → 0.650, 1 → 0.658, 2 → 0.661, 4 → 0.665, 8 → 0.667.

Test by source

Source	Books	Precision	Recall	F1	TP	FP	FN
bdrc	5	0.961	0.999	0.980	1569	63	1
lotsawa	47	0.776	0.852	0.812	52	15	9
reviewed_opf	24	0.371	0.196	0.256	166	282	682
nalanda	1	0.014	0.001	0.002	2	140	1578
reviewed only (opf + lotsawa + nalanda)	72	0.335	0.088	0.140	220	437	2269
all sources (micro)	77	0.782	0.441	0.564	1789	500	2270

The micro F1 of 0.564 is carried by the five BDRC test volumes (1,570 of 4,059 gold spans, F1 0.980). On the 72 reviewed books the F1 is 0.140. Nalanda (1,580 gold spans, 2 true positives) accounts for most of that miss.

Comparison

Same 72 books (the five BDRC test volumes are not in the baseline prediction file). Spans trimmed of edge whitespace, then the same IoU ≥ 0.5 match. Baseline is BDRC/Bo-Yigchung-Detection, scored in result/Latest_benchmark.

Set	v2 F1	baseline F1	v2 P / R	baseline P / R
reviewed OPF (24 books, 848 spans)	0.259	0.454	0.376 / 0.197	0.501 / 0.415
Lotsawa (47 books, 61 spans)	0.812	0.589	0.776 / 0.852	0.824 / 0.459
Nalanda (1 book, 1,580 spans)	0.002	0.001	0.014 / 0.001	0.091 / 0.001
those 72 books	0.141	0.235	0.338 / 0.089	0.509 / 0.153

v2 is ahead on Lotsawa and behind on the reviewed OPF books. Both models miss Nalanda. The published v1 checkpoint karma689/mmbert-yigchung scored F1 0.431 on these same 24 OPF books under the pre-review gold (873 spans), so that number is not on the same gold.

Validation on reviewed OPF was F1 0.514; test on the same source is 0.256.

Training

Data	karma689/yigchung_detection, revision v2.0, config windowed_w8192_s4914
Objective	weighted token cross-entropy, ignore_index=-100
Class weights	O 0.07, B-YIGCHUNG 5.0, I-YIGCHUNG 5.0
Schedule	AdamW, lr 1e-5, weight decay 0.01, warmup 20% of a 10-epoch reference
Batch	2 per device, grad accumulation 4, bf16, max grad norm 1.0
Selection	span IoU@0.5 F1, in-loop break_penalty 2.0; best step 2412
Stop	early stop at step 6700 (epoch 6.24 of a 15-epoch budget)
Seed	42

Train / validation / test windows: 8,585 / 1,043 / 1,189. About 64% of train windows contain no Yigchung token.

Label scheme

id	label
0	O
1	B-YIGCHUNG
2	I-YIGCHUNG

Label -100 is the loss mask for specials and for tokens already owned by an earlier window. It is not a class.

Limitations

• The tagger sees the base string only. Yigchung is a typographic layer; many spans are not lexically obvious.
• The headline test F1 mixes an easy BDRC slice with the reviewed books. Cite the per-source table when comparing releases.
• Boundary errors of a few syllables fail the IoU@0.5 match even when the region is roughly right.
• break_penalty 8 is the validation pick for this checkpoint. The v1 release used penalty 1.