A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020)

Abstract: We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.

## introduction one of the key elements that has pushed the state of the art considerably in neural nlp in recent years has been the introduction and spread of transfer learning methods to the field. these methods can normally be classified in two categories according to how they are used: embeddings or language models can be divided into fixed, meaning that they generate a single representation for each word in the vocabulary; and contextualized, meaning that a representation is generated based on both the word and its surrounding context, so that a single word can have multiple representations, each one depending on how it is used. in practice, most fixed embeddings are used as feature-based models. the most notable examples are word2vec @xcite , glove @xcite and fasttext @xcite . all of them are extensively used in a variety of applications nowadays. on the other hand, contextualized word representations and language models have been developed using both featurebased architectures, the most notable examples being elmo and flair @xcite , and transformer based architectures, that are commonly used in a fine-tune setting, as is the case of gpt-1, gpt-2 @xcite , bert and its derivatives @xcite @xcite and more recently t5 @xcite . all of them have repeatedly improved the state-of-the art in many downstream nlp tasks over the last year. in general, the main advantage of using language models is that they are mostly built in an unsupervised manner and they can be trained with raw, unannotated plain text. their main drawback is that enormous quantities of data seem to be required to properly train them especially in the case of contextualized models, for which larger corpora are thought to be needed to properly address polysemy and cover the wide range of uses that commonly exist within languages. for gathering data in a wide range of languages, wikipedia is a commonly used option. it has been used to train fixed embeddings @xcite and more recently the multilingual bert @xcite , hereafter mbert. however, for some languages, wikipedia might not be large enough to train good quality contextualized word embeddings. moreover, wikipedia data all belong to the same specific genre and style. to address this problem, one can resort to crawled text from the internet; the largest and most widespread dataset of crawled text being common crawl. foot_0 such an approach generally solves the quantity and genre/style coverage problems but might introduce noise in the data, an issue which has earned the corpus some criticism, most notably by trinh and le (2018) and @xcite . using common crawl also leads to data management challenges as the corpus is distributed in the form of a large set of plain text each containing a large quantity of unclassified multilingual documents from different websites. in this paper we study the trade-off between quantity and quality of data for training contextualized representations. to this end, we use the oscar corpus @xcite , a freely available foot_1 multilingual dataset obtained by performing language classification, filtering and cleaning of the whole common crawl corpus. foot_2 os-car was created following the approach of @xcite but proposing a simple improvement on their filtering method. we then train oscarbased and wikipedia-based elmo contextualized word embeddings @xcite for 5 languages: bulgarian, catalan, danish, finnish and indonesian. we evaluate the models by attaching them to the to udpipe 2.0 architecture @xcite for dependency parsing and part-of-speech (pos) tagging. we show that the models using the oscar-based elmo embeddings consistently outperform the wikipediabased ones, suggesting that big high-coverage noisy corpora might be better than small high-quality narrow-coverage corpora for training contextualized language representations foot_3 . we also establish a new state of the art for both pos tagging and dependency parsing in 6 different treebanks covering all 5 languages. the structure of the paper is as follows. in section 2 we describe the recent related work. in section 3 we present, compare and analyze the corpora used to train our contextualized embeddings, and the treebanks used to train our pos tagging and parsing models. in section 4 we examine and describe in detail the model used for our contextualized word representations, as well as the parser and the tagger we chose to evaluate the impact of corpora in the embeddings' performance in downstream tasks. finally we provide an analysis of our results in section 5 and in section 6 we present our conclusions. ## related work since the introduction of word2vec @xcite , many attempts have been made to create multilingual language representations; for fixed word embeddings the most remarkable works are those of @xcite and @xcite who created word embeddings for a large quantity of languages using wikipedia, and later @xcite who trained the fast-text word embeddings for 157 languages using common crawl and who in fact showed that using crawled data significantly increased the performance of the embeddings especially for mid-to low-resource languages. regarding contextualized models, the most notable non-english contribution has been that of the mbert @xcite , which is distributed as (i) a single multilingual model for 100 different languages trained on wikipedia data, and as (ii) a single multilingual model for both simplified and traditional chinese. four monolingual fully trained elmo models have been distributed for japanese, portuguese, german and basque foot_4 ; 44 monolingual elmo models foot_6 where also released by the hit-scir team @xcite during the conll 2018 shared task @xcite , but their training sets where capped at 20 million words. a german bert @xcite as well as a french bert model (called camembert) @xcite have also been released. in general no particular effort in creating a set of highquality monolingual contextualized representations has been shown yet, or at least not on a scale that is comparable with what was done for fixed word embeddings. for dependency parsing and pos tagging the most notable non-english specific contribution is that of the conll 2018 shared task @xcite , where the 1 st place (las ranking) was awarded to the hit-scir team @xcite who used @xcite 's deep biaffine parser and its extension described in @xcite , coupled with deep contextualized elmo embeddings @xcite (capping the training set at 20 million words). the 1 st place in universal pos tagging was awarded to @xcite al. (2018) who used two separate instances of bohnet et al. (2018)'s tagger. more recent developments in pos tagging and parsing include those of @xcite which couples another conll 2018 shared task participant, udpipe 2.0 @xcite , with mbert greatly improving the scores of the original model, and udify @xcite , which adds an extra attention layer on top of mbert plus a deep bi-affine attention layer for dependency parsing and a softmax layer for pos tagging. udify is actually trained by concatenating the training sets of 124 different ud treebanks, creating a single pos tagging and dependency parsing model that works across 75 different languages. ## corpora we train elmo contextualized word embeddings for 5 languages: bulgarian, catalan, danish, finnish and indonesian. we train one set of embeddings using only wikipedia data, and another set using only common-crawl-based oscar data. we chose these languages primarily because they are morphologically and typologically different from one another, but also because all of the oscar datasets for these languages were of a sufficiently manageable size such that the elmo pre-training was doable in less than one month. contrary to hit-scir team @xcite , we do not impose any cap on the amount of data, and instead use the entirety of wikipedia or oscar for each of our 5 chosen languages. ## experimental setting the main goal of this paper is to show the impact of training data on contextualized word representations when applied in particular downstream tasks. to this end, we train different versions of the embeddings from language models (elmo) @xcite for both the wikipedia and oscar corpora, for each of our selected 5 languages. we save the models' weights at different number of epochs for each language, in order to test how corpus size affect the embeddings and to see whether and when overfitting happens when training elmo on smaller corpora. we take each of the trained elmo models and use them in conjunction with the udpipe 2.0 @xcite architecture for dependency parsing and pos-tagging to test our models. we train udpipe 2.0 using gold tokenization and segmentation for each of our elmo models, the only thing that changes from training to training is the elmo model as hyperparameters always remain at the default values (except for number of training tokens) @xcite . ## conclusions in this paper, we have explored the use of the common-crawl-based oscar corpora to train elmo contextualized embeddings for five typologically diverse mid-resource languages. we have compared them with wikipedia-based elmo embeddings on two classical nlp tasks, pos tagging and parsing, using state-of-the-art neural architectures. our goal was to explore whether the noisiness level of common crawl data, often invoked to criticize the use of such data, could be compensated by its larger size; for some languages, the oscar corpus is several orders of magnitude larger than the corresponding wikipedia. firstly, we found that when properly filtered, common crawl data is not massively noisier than wikipedia. secondly, we show that embeddings trained using oscar data consistently outperform wikipedia-based embeddings, to the extent that they allow us to improve the state of the art in pos tagging and dependency parsing for all the 6 chosen treebanks. thirdly, we observe that more training epochs generally results in better embeddings even when the training data is relatively small, as is the case for wikipedia. our experiments show that common-crawlbased data such as the oscar corpus can be used to train high-quality contextualized embeddings, even for languages for which more standard textual resources lack volume or genre variety. this could result in better performances in a number of nlp tasks for many non highly resourced languages.