word alignment issue
The original LatinBERT tokenizer implementation split words on whitespace before doing tokenization like this.
def tokenize(self, text):
tokens=text.split(" ")
wp_tokens=[]
for token in tokens:
if token in {"[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"}:
wp_tokens.append(token)
else:
wp_toks=self.encoder.encode(token)
for wp in wp_toks:
wp_tokens.append(self.reverseVocab[wp+5])
return wp_tokens
From what I can tell, this tokenizer does not.
77; arcus_ \32;_ \84; ullius_ \32;_ \67;icero_ \32; _consul_ \32; _fuit_ \32; _et_ \32;_ \71; alliam_ \32; _ab_ \32;_ \72; elvetiis_ \32; _defendit_._
I noticed the lowercasing bug has been fixed here, assuming lowercase input:
marcus_ \32;_ tullius_ \32; _cicero_ \32;_ consul_ \32;_ fuit_ \32;_ et_ \32;_ galliam_ \32;_ ab_ \32;_ helvetiis_ \32;_ defendit_ ._
If you split and force word alignment in the HF tokenizer options, you get this much cleaner sequence.
marcus_ tullius_ cicero_ consul_ fuit_ et_ galliam_ ab_ helvetiis_ defendit_ ._
The current tokenizer code seems to be machine generated so I am not sure if this is a deliberate omission by the maintainers or just a case of AI not doing what it should have. The header claims POS performance matches but POS tagging performance is saturated and you can induce gains over paper reported SoTA just by running the right seeds. And assuming y'all are using conllu, it is also already split into words. The tokenizer implementation here does seem to be plain harmful on other downstream tasks I've taken a look at. For anyone else the solution is just to split on whitespace before and use something to the effect of:
enc = tok(words, is_split_into_words=True, add_special_tokens=True, truncation=False)