sulaimank/w2vbert-luganda-waxal-punct-0

w2vbert-luganda-waxal-punct

Dedicated Luganda W2V-BERT CTC trained WITH punctuation in the vocabulary, on WAXAL Luganda. Emits . , ? ! directly rather than relying on a downstream restorer.

Best validation results:

• Loss: 0.1325
• Wer (keep punct): 0.1809
• Cer (keep punct): 0.0359
• Zindi (keep punct): 0.8916
• Wer (strip punct): 0.0992
• Zindi (strip punct): 0.9391

Training hyperparameters

• learning_rate: 3e-05
• train_batch_size: 16
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: AdamW (betas=(0.9, 0.98))
• lr_scheduler_type: linear, warmup_ratio 0.1
• num_epochs: up to 40 (early stopped)

Training results

Epoch	Step	Val Loss	WER keep	CER keep	Zindi keep	WER strip	Zindi strip
0.59	8	2.9813	1.0000	0.9847	0.0077	0.9999	0.0078
1.18	17	0.3110	0.3210	0.0722	0.8034	0.2626	0.8399
1.77	26	0.1604	0.2186	0.0469	0.8672	0.1506	0.9069
2.36	35	0.1474	0.2139	0.0459	0.8701	0.1249	0.9228
2.95	44	0.1347	0.1971	0.0419	0.8805	0.1075	0.9343
3.54	53	0.1354	0.1942	0.0409	0.8824	0.1067	0.9348
4.13	62	0.1289	0.1846	0.0378	0.8888	0.1018	0.9376
4.72	71	0.1344	0.1828	0.0364	0.8904	0.1015	0.9378
5.31	80	0.1358	0.1827	0.0355	0.8909	0.1005	0.9386
5.90	89	0.1325	0.1809	0.0359	0.8916	0.0992	0.9391
6.49	98	0.1367	0.1857	0.0361	0.8891	0.1021	0.9375
7.08	107	0.1500	0.1877	0.0367	0.8878	0.1024	0.9374
7.67	116	0.1471	0.1835	0.0364	0.8900	0.0985	0.9395
8.26	125	0.1641	0.1883	0.0377	0.8870	0.1049	0.9357
8.85	134	0.1641	0.1873	0.0382	0.8873	0.1019	0.9376
9.44	143	0.1817	0.1878	0.0380	0.8871	0.0999	0.9387
10.03	152	0.1803	0.1832	0.0362	0.8903	0.1004	0.9385
10.62	161	0.1811	0.1825	0.0362	0.8907	0.1006	0.9383