Can Character-based Language Models Improve Downstream Task Performances In Low-Resource And Noisy Language Scenarios? (2021)

Abstract: Recent impressive improvements in NLP, largely based on the success of contextual neural language models, have been mostly demonstrated on at most a couple dozen high- resource languages. Building language mod- els and, more generally, NLP systems for non- standardized and low-resource languages remains a challenging task. In this work, we fo- cus on North-African colloquial dialectal Arabic written using an extension of the Latin script, called NArabizi, found mostly on social media and messaging communication. In this low-resource scenario with data display- ing a high level of variability, we compare the downstream performance of a character-based language model on part-of-speech tagging and dependency parsing to that of monolingual and multilingual models. We show that a character-based model trained on only 99k sentences of NArabizi and fined-tuned on a small treebank of this language leads to performance close to those obtained with the same architecture pre- trained on large multilingual and monolingual models. Confirming these results a on much larger data set of noisy French user-generated content, we argue that such character-based language models can be an asset for NLP in low-resource and high language variability set- tings.

## introduction current state-of-the-art monolingual and multilingual language models require large amounts of data to be trained, showing limited performance on low-resource languages @xcite . they lead to state-of-the-art results on most nlp tasks @xcite . in order to achieve high performance, these models rely on transfer learning architectures: the language models need to be trained on large amounts of data (pre-training) to be able to transfer the acquired knowledge to a downstream task via fine-tuning on a relatively small number of examples, resulting in a significant performance improvement with respect to previous approaches. this dependency on large data sets for pre-training is a severe issue for low-resource languages, despite the emergence of large and successful multilingual pre-trained language models @xcite . this is especially the case for languages with unusual morphological and structural features, which struggle to take advantage from similarities with high-resource, well represented languages such as romance and germanic languages. in this work, we focus on one of such highly challenging languages, namely north-african dialectal arabic. its latin transcription (arabizi) displays a high level of linguistic variability foot_0 , on top of scarce and noisy resource availability, making it a particularly challenging language for most nlp systems relying on pre-trained multilingual models @xcite . 2 to tackle the resource scarcity issue regarding arabic dialects, antoun et al. (2020) use bert architecture @xcite to train a model on arabic text to compare this approach to standard multilingual models. indeed, @xcite show that fine-tuning a monolingual model leads to better results than fine-tuning a multilingual one, meaning that when fine-tuning is used there is no significant performance improvement from cross-lingual transfer during pre-training. however, such model is still pre-trained on sentences written in a single language and was not trained to handle the presence of multiple languages in the same sentence (codeswitching), a frequent phenomenon in narabizi. however both monolingual and multilingual model approaches bear the risk of being limited by a subword tokenization-based vocabulary when facing out-of-domain training data language, especially in high-variability noisy scenarios @xcite , even though @xcite demonstrated a positive effect for narabizi when using target language data to fine-tune a multilingual language model on its own objective function before pretraining. following a different approach, we investigate the use of a recently issued character-based language model @xcite that was shown to display a remarkable robustness to lexical variation and noise when facing a new distant domain, namely biomedical. the pipeline we developed is simple and consists in fine-tuning this character-based model for several tasks in a noisy low-resource language scenario. we show that a character-based model trained on only 99k sentences of narabizi and fined-tuned on a small treebank of the language leads to performance close to that obtained with the same architecture pretrained on large multilingual and monolingual models (mbert and camembert). interestingly, we generalize this observation by using the same architecture on a much larger french user-generated content treebank that exhibits similar language variability issues than narabizi. in fact, pre-training a character-based model on 1% of the large-scale french instance of the multilingual corpus oscar leads to similar performance as a subword based model trained on the full corpus, showing that such character-based language model can reach similar performance levels and that the resulting models exhibit the same tolerance to noise as their much larger bert counterparts. this demonstrates the value of such models in very scarce resource scenario. our code and models are freely available. 3 2 narabizi: a challenging use case for nlp in low-resource scenarios as the official language of 25 countries, arabic showcases a linguistic phenomenon called diglossia @xcite . it means that the speakers use modern standard arabic (msa) for formal and official situations but use other forms of arabic in informal situations. these dialectal forms constitute a dialect continuum with large variability from one country to the other. arabic dialects are defined by their spoken form and often exhibit a lack of standardized spelling when written. when arabic speakers produce written text in such dialects, they merely transcribe their spoken, colloquial language, which leads to different forms for the same word. many users use the latin script to express themselves online in their dialect @xcite . in particular, they transcribe phonemes that cannot be straightforwardly mapped to a latin letter using digits and symbols, foot_3 with a high degree of variability at all levels; this is called arabizi, with its north-african version called narabizi by @xcite . for cultural and historical reasons, narabizi also exhibits a high degree of code-switching with french and amazigh @xcite , i.e. alternations between two or more languages during the conversation. besides, the only available textual resources for arabizi data are user-generated content, which is by itself inherently noisy @xcite @xcite , making the production of supervised models, assuming the availability of labeled data, or even the collection of large pre-training data set a rather difficult task. this data scarcity problem is often solved in the nlp literature using transfer learning: transferring knowledge learnt by large scale language models pre-trained on larger corpora. however, the use of latin script makes it harder to transfer knowledge from language models trained on arabic script corpora. for subword-based tokenization models, words are represented by a combination of subwords from a predefined list. when applied to a highly variable language with code-switching, a large vocabulary would be necessary to get a good covering of all possible orthographic variations which makes this approach less practical. gloss attested forms lang why wa3lach w3alh 3alach narabizi all ekl kal kolach koulli kol narabizi many beaucoup boucoup bcp code-switched fr. ## data sets in this section, we describe the data sets we use to evaluate our pipeline on our downstream tasks, namely part-of-speech (pos) tagging and dependency parsing. narabizi data set we use the narabizi treebank @xcite , containing about 1500 sentences randomly sampled from the romanized algerian dialectal arabic corpus of @xcite and from a small corpus of lyrics coming from algerian dialectal arabic songs popular among the younger generation. this treebank is manually annotated with morpho-syntactic information (parts-of-speech and morphological features), together with glosses and code-switching labels at the word level, as well as sentence-level translations. moreover, this treebank also contains 36% of french tokens. within the narabizi annotated corpus, 5 in addition to labeled data, the narabizi treebanks provides about 2 millions words, 99k 5 http://almanach-treebanks.fr/narabizi sentences, of unlabeled data collected from various sources. 6 we use this corpus for unsupervised fine-tuning. french data sets we use the following universal dependencies, ud, @xcite version of the following treebanks: french gsd @xcite , sequoia @xcite and spoken, an automatic conversion of the rhapsodie corpus @xcite to the ud annotation scheme. for our experiments on noisy ugc treebank, we use an extension of the french social media bank, fsmb foot_5 @xcite : a treebank of french sentences coming from various social media sources only available either in constituent trees or in the native french treebank dependency annotation scheme @xcite , along with the sequoia original treebank that we use with the same annotation scheme for compatibility in our french ugc experiments. a brief overview of the size and content of each treebank can be found in pre-training data sets note that in some of our experiments, we use a fraction, 1% of the deduplicated french instance of the oscar corpora @xcite , about 380m words, as a source of unlabeled data to be either mixed with narabizi pretraining data (as an 2.5m words additional sample) or used as pre-training data for characterbert when evaluated on french ugc (whole 1%). statistics on those data sets are presented on and a highway layer @xcite . this method for encoding token representations is inspired from elmo @xcite , one of the first pre-trained language models for transfer learning. it generates a context-independent representation from character embeddings and feeds them to transformer encoder layers, similarly to bert architecture. therefore, it produces a single embedding for any input token and does not need a wordpiece vocabulary. this avoids having an inconstant number of subword vectors for each word. we choose this model since its robustness to noise, when tested on biomedical domain by el @xcite , can lead to interesting result when facing our noisy experiment data. it is the first character-based and bert-like model, along with charbert @xcite , that uses both character and subword embeddings and has the advantage of being publicly available. 8 note that we retrain characterbert from scratch on our data sets and do not make use of any of its available pretrained at models any point in our experiments. ## experiments in this section, we present our fully-supervised and semi-supervised baselines. we also evaluate different fine-tuning strategies combined with two layers configurations. we use various embedding models that we contrast with the characterbert model. baseline models for our fully-supervised baseline, we use fasttext embeddings @xcite trained from scratch on our treebank training sets and used as input for our downstream tasks without any special treatment. to measure the effectiveness of using a contextualized character-based language model, we compare its performance to subword based language models, both monolingual and multilingual, that constitute the basis of our semi-supervised baselines. for multilingual subword based language upos uas las no external embeddings 57.61 55.48 39.32 model+task we use the implementation of the biaffine graph parser @xcite from @xcite ; they adapted it to handle several input sources, such as bert representations. the parser performs its own tagging using a multi-layer perceptron. the word representations are a concatenation of word embeddings and tag embeddings learned together with the model parameters on the treebank training data. we fine-tune the overall model by backpropagating through the average of all sub-tokens of a word. baseline results, without any external embeddings, for this parser are provided in model+mlm+task before fine-tuning the model on the downstream task, we perform language adaptation by fine-tuning it in a selfsupervised fashion with the mlm loss. we use the narabizi raw data; we train the model for 20 epochs, keeping the best model obtained at the end. the evaluation is done using the mlm log likelihood, with 10% of the data kept for validation. ## results and discussion in this section, we compare all the systems presented before on the narabizi and french treebanks. we especially focus on the impact of corpus size on the models' performance. ## discussion in this work, we evaluate the benefits of using a character-based model in low-resource scenarios. our results show that training such a model from scratch on much fewer data gives similar performance to a multilingual bert adapted to the language using the same amount of data. overall, our observations confirm the findings of el @xcite regarding the robustness to noise and misspellings of the character-bert model. we showed that the model has competitive performance on noisy french ugc data when trained on only a fraction of the oscar corpus compared to camembert trained on the full corpus and when trained on corpora containing about 1m words in the extremely noisy and lowresource case of narabizi. this is consistent with the findings of @xcite and @xcite , who showed that mlm could already learn a lot from pre-training on smaller data set. extending this investigation by training on a larger amount of data could help to explore the ability of the model to handle highly variable noisy data. however, one could question the usefulness of such character-bert based models if small bertbased models were available on the same domain. to build an answer to that question, we conducted a quick set of experiments comparing our char-acterbert model trained on 1% of oscar with the off-the-shelf camembert version trained on 4gb of the oscar corpus french instance (2.38% of the full corpus) and which was shown to perform almost as well as the full model @xcite on many downstream tasks. both models were fined-tuned according to our model+task architecture on either the fsmb or the sequoia treebank, allowing us to evaluate their in-domain and out-ofdomain performance. results on our results are based on the evaluation of two low-level tasks. therefore, it would be interesting to see if they can be generalized to other -e.g. more semantic -tasks, as additional experiments on model layers configuration showed that most of the important information is captured early in the layers of the model (cf. appendix a). regarding the specific case of arabic dialects written in arabizi, a recent bert-based model have been pretrained on 7 millions egyptian tweets and displayed effective results on a sentiment analysis task @xcite . another very recent model, at the date of writing, was pre-trained on 4 millions algerian tweets and also demonstrated interesting results on sentiment analysis @xcite . unfortunately, the authors did not perform any experiments on the narabizi data set, making thus the comparison with our work not straightforward. it would be of course interesting to evaluate the interoperability between these new data sets and the narabizi resources we used to produce our models. head to head comparisons between these models and ours could be of value of course but we believe that given the shortcomings of finding enough data to pretrain large models for dialects, it would be probably better to first consolidate a large enough common pre-training data set and then work on model performance. we leave this for future work. we showed that characterbert models trained on very little data could provide an interesting alternative to large multilingual and monolingual models in resource-scarce and noisy scenarios. this is why we release all the code, data and models to reproduce our experiments, hoping our work will favor the rise of efficient robust nlp models for underresourced languages, domains and dialects. 10 linting xue, aditya barua, noah constant, rami al-rfou, sharan narang, mihir kale, adam roberts, and colin raffel. 2021. byt5: towards a token-free future with pre-trained byte-to-byte models.