Where are we Still Split on Tokenization? (2024)

Abstract: Many Natural Language Processing (NLP) tasks are labeled on the token level, forthese tasks, the first step is to identify the tokens (tokenization). Becausethis step is often considered to be a solved problem, gold tokenization iscommonly assumed. In this paper, we propose an efficient method fortokenization with subword-based language models, and reflect on the status ofperformance on the tokenization task by evaluating on 122 languages in 20different scripts. We show that our proposed model performs on par with thestate-of-the-art, and that tokenization performance is mainly dependent on theamount and consistency of annotated data. We conclude that besidesinconsistencies in the data and exceptional cases the task can be consideredsolved for Latin languages for in-dataset settings (>99.5 F1). However,performance is 0.75 F1 point lower on average for datasets in other scripts andperformance deteriorates in cross-dataset setups.

## introduction because many tasks in natural language processing (nlp) are annotated on the token level, identifying the tokens is a crucial first step for nlp models. however, in most work on token-level tasks in nlp, gold tokenization is used, implicitly making the assumption that tokenization is a solved problem. notable exceptions include the conll 2018 shared task @xcite and work on languages where whitespaces are not used as word separators, and tokenization is more challenging (e.g. @xcite . traditionally, tokenization was done with rulebased systems @xcite , with rules usually adapted towards 1 code is available on bitbucket.org/robvanderg/tok, note that our implementation is also available as part of the machamp toolkit: https://github.com/machamp-nlp/ figure 1 : example sentence (1), regular expression tokenizing punctuation (2), sequence labeling on the character level (3), sequence labeling on the subword level (4). all of these strategies lead to the same tokenization: "dr. dron is his backup ." english datasets (figure 1 : 2). with the introduction of machine learning, and later neural networks, tokenization was also framed as a character level labeling task (figure 1 : 3) @xcite @xcite . however, since most recent nlp models are based on contextualized language models (clm), which commonly use subwords, subword level labeling for tokenization has been proposed @xcite (figure 1 : 4), leading to even higher performance. however, @xcite do not extend to multi-lingual models, and their training procedure is compute intensive. hence, we propose to tackle tokenization simultaneously with other nlp tasks while finetuning the clm. this setup has competitive performance, while being universally applicable; we train one multi-task, multi-lingual model that does tokenization, pos tagging and dependency parsing; which is desirable in terms of efficiency, dependencies, and simplicity. we then use this model to evaluate and analyze the performance in a variety of setups.we tackle the following question in this work: 1) is the tokenization task solved in supervised setups? 2) how robust are supervised tokenizers across datasets? input: if momma ain't happy, nobody ain't happy. tokenization: if momma ain't happy , nobody ain't happy . ## tokenization with subword-level labels because the subword level is central in most modern language models, we label subwords for the tokenization task (figure 1 : 4). this approach has a limitation; there is a theoretical upper bound, as there is a limitation on the possible boundaries (i.e. splits are not possible within subwords). to increase this upper bound, we first apply the ba-sictokenizer from the transformers library @xcite , which is a rule-based tokenizer that separates punctuation characters. this leads to an upper bound above 99% f1 score for 122 out of 123 treebanks of the datasets we use (appendix d) when using the mbert subword segmenter @xcite . only the japanese gsd treebank has a lower score (80.4). 4 to increase this upperbound, we consider all hiragana and katakana characters as a single subword (note that bert tokenizers already do this for cjk characters, including kanji). it should be noted that character normalizations and unknown tokens make the conversion of the output of the clm to the original text non trivial. more details on how we handled these specific cases can be found in appendix a. if we would train a separate clm for tokenization and one for a downstream task, this would lead to very inefficient training as well as inference. note that they can't run in paralellel, as tokenization should be done first. hence, we propose a multi-task setup, where we share an encoder and model multiple tasks in separate decoder heads (linear layers). at train time, we use gold tokenization to obtain the loss for the other tasks, as labels for incorrect tokenizations are non-trivial to obtain. at inference time we use the predicted tokenization as input for the other tasks. setup we implemented our model in @xcite v0.4.2, and have included it in the public version. we use all default parameters in machamp (see appendix b; note that we fully fine-tune the clm in all our settings). we implemented tokenization with cross-entropy loss and a feedforward layer which transforms the output of the clm to a binary label (b or i, see figure 1 ). in the multi-task setup, we use the default implementations for upos tagging, lemmatization, morphological tagging and dependency parsing. we report f1 scores from the official conll 2018 evaluation script @xcite . we used ud v2.10 and multilingual bert for our main evaluations. note that we also evaluated on xlm-r large @xcite , but found that it underperforms for tokenization while being computationally more expensive (appendix e). we evaluate a variety of settings: st: single task; an clm encoder with only a tokenization head; mt: multi-task: learn tokenization simultaneously with pos tagging, lemmatization, morphological tagging and dependency parsing, ml+mt: multi-lingual, multi-task: train on the training splits of all treebanks for all tasks. to better interpret our results, we compare against five rule-based (rb) tokenizers (more information in appendix g). we use the highest performing tokenizer (through an oracle) for each dataset. ## results in this section we only consider treebanks that contain a train-split to be able to fairly compare to single-treebank models. we report averages over all dev splits (to avoid over analyzing the test data, note that we did not tune the models), but also averages over subsets of the data; we compare datasets in the latin script (93 datasets) and all other scripts (38 datasets), 5 and we inspect the effect of dataset size by separating datasets in small (0<#@xmath0,000, 11 datasets), medium, (20,000<#@xmath1,000, 43 datasets) and large (>100,000, 51 datasets) train size. we focus here on tokenization and dependency parsing, results on other tasks can be found in appendix f. starting with the results on tokenization (figure 2 ), we can see that the differences in performance for the different settings are small for the tokenization task; but every error for this task has a catastrophic effect on downstream task performances, so even small differences can be important. the single task setting (st) outperforms all other models in almost all setups. however, this setting is impractical due to computational costs. multi-task (mt) and multi-lingual (ml) learning slightly harm performance, but multilingual (ml) models outperform mono-lingual models on small datasets. it should be noted that treebanks in non-latin scripts are not consistently smaller (appendix f), and the lower performance on non-lating datasets can thus mainly be ascribed to under-representation in the underlying language model and the complexity of the task. to interpret our results in a larger context, we attempt to compare to rule-based baselines; which are non-trivial to find for our varied set of languages (appendix g), but it is clear that rule-based approaches underperform with a large margin; averages for all treebanks are around 91-92 f1. interestingly, downstream results on dependency parsing (figure 3 ) show different trends compared to the tokenization results; multi-lingual training (ml) is beneficial for this task, except for large datasets which have slightly lower performance. furthermore, we see that the predicted tokenization performs very close to the gold tokenization (gold) for parsing. ## analysis quantitative in general, precision is higher than recall for all the proposed models (results available in repository), showing that the model mostly misses splits instead of over-tokenizing. performance detoriates on test-only treebanks ( ## conclusion we have investigated which problems are still open for the task of tokenization. we conclude that tokenization in supervised setups for latin languages can be considered solved, with some dataset inconsistencies as remaining errors. but for lowerresource languages and especially languages without whitespaces for word boundaries challenges remain. furthermore, we showed that performance in cross-dataset setups deteriorates, even when training on the target language. this highlights the need for clear annotation guidelines, and confirms the presence of annotation inconsistenties. furthermore, we have implemented a new tokenization model that is faster to train than previous work. we include handling of unknown tokens and character normalizations as well as missed word boundaries. furthermore, multi-task learning as well as multi-lingual learning slightly harm performance, but allow for a single model for multiple tasks and languages.