Hints on the data for language modeling of synthetic languages with transformers (2023)

Abstract: Language Models (LM) are becoming more and more useful for providing representations upon which to train Natural Language Processing applications. However, there is now clear evidence that attention-based transformers require a critical amount of language data to produce good enough LMs. The question we have addressed in this paper is to what extent the critical amount of data varies for languages of different morphological typology, in particular those that have a rich inflectional morphology, and whether the tokenization method to preprocess the data can make a difference. These details can be important for low-resourced languages that need to plan the production of datasets. We evaluated intrinsically and extrinsically the differences of five different languages with different pretraining dataset sizes and three different tokenization methods for each. The results confirm that the size of the vocabulary due to morphological characteristics is directly correlated with both the LM perplexity and the performance of two typical downstream tasks such as NER identification and POS labeling. The experiments also provide new evidence that a canonical tokenizer can reduce perplexity by more than a half for a polysynthetic language like Quechua as well as raising F1 from 0.8 to more than 0.9 in both downstream tasks with a LM trained with only 6M tokens.

## introduction language models (lms) are becoming more and more useful for providing representations upon which to train different natural language processing (nlp) applications. however, there is evidence that lms trained with attention-based transformers need large quantities of pretraining language data to provide good enough representations that can be used in downstream tasks. ## equal contribution to have very large amounts of data, multilingual lms have been proposed as a solution. however, there is evidence @xcite @xcite that the monolingual lms outperformed their multilingual counterparts. as for the amount of monolingual data required, @xcite experiments with english showed that the amount of data for reaching at least an 80% average over several tasks of relative performance is around 10m tokens. the question we have addressed in our research is whether the critical figures for english are the same for other languages and in particular for languages of a different morphological type. having hints about the critical amount of data and tokenization strategies to make the most profit of the available data is of upmost importance for low-resource languages, many of them with a morphology more complex than that of english, and that need to plan the production of datasets. a lm is an assessment of the probability distribution over sequences of words given a fixed set of words with parameters estimated from data. the increase in the number of tokens of the vocabulary of particular languages due to their inflectional morphology has been demonstrated to affect the coverage of the markovian lms @xcite . for current attention-based transformer language models (tlm), like roberta that is a closed vocabulary system, the direct consequence of modeling a rich inflectional morphology should also be that the coverage of the vocabulary will be lower than that of a morphologically simpler language. for instance, @xcite found that english was among the easiest languages for building a lm, while german, which is a synthetic language, was among the hardest. polysynthetic languages like quechua, with more than 100 inflectional suffixes, and in which up to five suffixes can be attached to a verbal stem, would have harder modeling problems that will aggravate its problems for being a low-resource language. to understand how the amount of critical pretraining data varies for different languages, we reproduced @xcite experiments but for different languages of an increasing degree of morphological complexity, as measured by type-token ratio (ttr) following @xcite and @xcite . the languages are: english, french, german, turkish and quechua. in ## methodology in our experiments, we tested 20 roberta models. we pretrained from scratch lm for english, german, french, turkish and quechua with different pretraining datasets ranging from 1m to 6m tokens, and we used three different tokenizers for each: bpe, unigram and deepspin. code and resources are available on https://github.com/iulaterm-trl-upf/ hints-on-the-data-for-language-modeling 3.1 pretraining ## results our research aimed on the one hand at making an evaluation of the amount of pretraining data and the role of the tokenizer measured in terms of lm perplexity. on the other hand, the pos and ner tasks were meant to assess the quality of the representations produced when used in fine-tuning downstream tasks. it is important to mention that we did not perform any normalization in the results as opposed to @xcite , because we also wanted to see a comparison between languages. ## discussion in order to clarify the amount of data necessary to achieve robust performance measured by lm perplexity, we experimented with four training data sizes: 1m, 2m, 3m and 6m tokens. we were interested in two main issues. first, in the work of @xcite it can be seen that perplexity improves dramatically when the training data size is above 10m, however low-resource languages like quechua do not even have texts amounting 10m tokens. we were interested in finding whether there is a critical amount of data with which it is worth for low-resource languages to build a tlm. second, we wanted to show to what extent lm perplexity and the fine-tuning of downstream tasks are influenced by the size of the data and the morphological typology of languages, and whether tokenization could mitigate these issues. from our results it is clear that in spite of being trained with the same configurations and amount of training data, there are differences among the languages we examined. @xcite suggested that these differences could be due to the difference in morphological complexity between these languages. a rich inflectional morphology increases the vocabulary. as we can see in we also wanted to get evidence about the quality of the representations obtained by our different tlm for fine-tuning downstream tasks. the results shown in ## conclusions in this paper we have related the quality of tlm with the training data size. we have approached the topic from the point of view of low-resource languages that need to maximize the available data. we have demonstrated how different methods, in this case tokenizers, apply to languages other than english. we have evaluated intrinsically and extrinsically the impact of datasize and tokenization with the aim of giving some hints for the building of tlm for low-resource languages, in particular for those whose morphology processes produces large vocabularies. these hints are explaining below. in our experiments, all languages show a continuous reduction of perplexity when from 1m to 6m tokens, with no stagnation. regardless of language type, the decrease in perplexity progresses as the model is trained with more data, suggesting that it can still improve more with more data. however, we provide evidence on the fact that with 6m all the languages in our experiments, but turkish and quechua, could reach a perplexity below 100, and macro-f1 score higher than 0.9 in the two downstream tasks. with a linguistically motivated and canonical tokenizer like deepspin, turkish and quechua could also attain these competitive results, as explained below. tokenization methods play an important role in building pretrained models @xcite . as seen in our experiments, canonical and linguistically motivated tokenizers achieve astonishing re-sults compared to other types of tokenizers. the reduction by almost 50% of the perplexity of the preentangled models of turkish and quechua when using deepspin instead of bpe is impressive. languages morphologically different from turkish and quechua also showed significant benefits, e.g., english, french and german showed an improvement of 15%, 31% and 24% respectively. on the other hand, it can also be seen that using deepspin results in significant improvements in tasks such as ner and pos tagging. both turkish and quechua manage to increase the macro-f1 score by 0.1 and 0.14 respectively. english, french and german also manage to increase the macro-f1 score by 0.03 in most cases. finally, we can say that canonical and linguistically motivated tokenization methods present statistically significant improvements when working with morphologically complex languages compared to statistically motivated methods such as bpe and unigram.