Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead? (2021)

Abstract: Most work in NLP makes the assumption that it is desirable to develop solutions in the native language in question. There is consequently a strong trend towards building native language models even for low-resource languages. This paper questions this development, and explores the idea of simply translating the data into English, thereby enabling the use of pretrained, and large-scale, English language models. We demonstrate empirically that a large English language model coupled with modern machine translation outperforms native language models in most Scandinavian languages. The exception to this is Finnish, which we assume is due to inferior translation quality. Our results suggest that machine translation is a mature technology, which raises a serious counter-argument for training native language models for low-resource languages. This paper therefore strives to make a provocative but important point. As English language models are improving at an unprecedented pace, which in turn improves machine translation, it is from an empirical and environmental stand-point more effective to translate data from low-resource languages into English, than to build language models for such languages.

## introduction although the transformer architecture for deep learning was only recently introduced @xcite , it has had a profound impact on the development in natural language processing (nlp) during the last couple of years. starting with the seminal bert model @xcite , we have witnessed an unprecedented development of new model variations @xcite @xcite @xcite with new state of the art (sota) results being produced in all types of nlp benchmarks @xcite @xcite . the leading models are large both with respect to the number of parameters and the size of the training data used to build the model; this correlation between size and performance has been demonstrated by @xcite . the ongoing scale race has culminated in the 175-billion parameter model gpt-3, which was trained on some 45tb of data summing to around 500 billion tokens @xcite . 1 turning to the scandinavian languages, there are no such truly large-scale models available. at the time of writing, there are around 300 scandinavian models available in the hugging face transformers model repository. 2 most of these are translation models, but there is already a significant number of monolingual models available in the scandinavian languages. 3 however, none of these scandinavian language models are even close to the currently leading english models in parameter size or training data used. as such, we can expect that their relative performance in comparison with the leading english models is significantly worse. furthermore, we can expect that the number of monolingual scandinavian models will continue to grow at an exponential pace during the near future. the question is: do we need all these models? or even: do we need any of these models? can't we simply translate our data and tasks to english and use some suitable english sota model to solve the problem? this paper provides an empirical study of this idea. 2 related work there is already a large, and rapidly growing, literature on the use of multilingual models @xcite , and on the possibility to achieve cross-lingual transfer in multilingual language models @xcite @xcite @xcite . from this literature, we know among other things that multilingual models tend to be competitive in comparison with monolingual ones, and that especially languages with smaller amounts of training data available can benefit significantly from transfer effects from related languages with more training data available. this line of study focuses on the possibility to transfer models to a new language, and thereby facilitating the application of the model to data in the original language. by contrast, our interest is to transfer the data to another language, thereby enabling the use of sota models to solve whatever task we are interested in. we are only aware of one previous study in this direction: @xcite performs cross-lingual machine translation using outdated methods, resulting in the claim that even if perfect translation would be possible, we will still see degradation of performance. in this paper, we use modern machine translation methods, and demonstrate empirically that no degradation of performance is observable when using large sota models. ## data in order to be able to use comparable data in the languages under consideration (swedish, danish, norwegian, and finnish), we contribute a scandinavian sentiment corpus (scandisent), foot_2 consisting of data downloaded from trustpilot.com. for each language, the corresponding subdomain was used to gather reviews with an associated text. this data covers a wide range of topics and are divided into 22 different categories, such as electronics, sports, travel, food, health etc. the reviews are evenly distributed among all categories for each language. all reviews have a corresponding rating in the range 1 -5. the review ratings were polarised into binary labels, and the reviews which received neutral rating were discarded. ratings with 4 or 5 thus corresponds to a positive label, and 1 or 2 correspond to a negative label. to further improve the quality of the data, we apply fasttext's language identification model @xcite to filter out any reviews containing incorrect language. this results in a balanced set of 10,000 texts for each language, with 7,500 samples for training and 2,500 for testing. ## models in order to fairly select a representative pre-trained model for each considered scandinavian language, we opt for the most popular native model according to hugging face. for each considered language, this corresponds to a bert-base model, hence each language is represented by a language model translated into english xlm-r-large 97.04 96.84 98.24 95.48 - we compare these scandinavian models against the english bert-base and bert-large models by google. english bert-base is thus identical in architecture to the scandinavian models, while bert-large is twice as deep and contains more than three times the amount of parameters as bert-base. finally, we include xlm-r-large, in order to compare with a model trained on significantly larger (and multilingual) training corpora. ## discussion & conclusion our experiments demonstrate that it is possible to reach better performance in a sentiment analysis task by translating the data into english and using a large pre-trained english language model, compared to using data in the original language and a smaller native language model. whether this result holds for other tasks as well remains to be shown, but we see no theoretical reasons for why it would not hold. we also find a strong correlation between the quantity of pre-training data and downstream performance. we note that xlm-r in particular performs well, which may be due to data size, and potentially the ability of the model to take advantage of transfer effects between languages. an interesting exception in our results is the finnish data, which is the only task for which the native model performs best, despite xlm-r reportedly having been trained on more finnish data than the native finnish bert model @xcite . one hypothesis for this behavior can be that the alleged transfer effects in xlm-r hold primarily for typologically similar languages, and that the performance on typologically unique languages, such as finnish, may actually be negatively affected by the transfer. the relatively bad performance of bert-large on the translated finnish data is likely due to insufficient quality of the machine translation. the proposed approach is thus obviously dependent on the existence of a high-quality machine translation solution. the scandinavian languages are typologically very similar both to each other and to english, which probably explains the good performance of the proposed approach even when using a generic translation api. for other languages, such as finnish in our case, one would probably need to be more careful in selecting a suitable translation model. whether the suggested methodology will be applicable to other language pairs thus depends on the quality of the translations and on the availability of large-scale language models in the target language. our results can be seen as evidence for the maturity of machine translation. even using a generic translation api, we can leverage the existence of large-scale english language models to improve the performance in comparison with building a solution in the native language. this raises a serious counter-argument for the habitual practice in applied nlp to develop native solutions to practical problems. hence, we conclude with the somewhat provocative claim that it might be unnecessary from an empirical standpoint to train models in languages where: in such cases, we may be better off relying on existing large-scale english models. this is a clear case for practical applications, where it would be beneficial to only host one large english model and translate all various incoming requests from different languages.