Performance Implications of Using Unrepresentative Corpora in A rabic Natural Language Processing (2023)

Abstract: Wikipedia articles are a widely used source of training data for Natural Language Processing (NLP) research, particularly as corpora for low-resource languages like Arabic. However, it is essential to understand the extent to which these corpora reflect the representative contributions of native speakers, especially when many entries in a given language are directly translated from other languages or automatically generated through automated mechanisms. In this paper, we study the performance implications of using inorganic corpora that are not representative of native speakers and are generated through automated techniques such as bot generation or automated template-based translation. The case of the Arabic Wikipedia editions gives a unique case study of this since the Moroccan Arabic Wikipedia edition (ARY) is small but representative, the Egyptian Arabic Wikipedia edition (ARZ) is large but unrepresentative, and the Modern Standard Arabic Wikipedia edition (AR) is both large and more representative. We intrinsically evaluate the performance of two main NLP upstream tasks, namely word representation and language modeling, using word analogy evaluations and fill-mask evaluations using our two newly created datasets: Arab States Analogy Dataset (ASAD) and Masked Arab States Dataset (MASD). We demonstrate that for good NLP performance, we need both large and organic corpora; neither alone is sufficient. We show that producing large corpora through automated means can be a counter-productive, producing models that both perform worse and lack cultural richness and meaningful representation of the Arabic language and its native speakers.

## introduction natural language processing (nlp) plays a crucial role in decision-making systems. for instance, it is employed in resume parsers that assist in sorting job candidates. nlp systems are typically designed to analyze extensive collections of human text (corpora) with the goal of deriving insights from human behavior and generating recommendations on our behalf @xcite . the normal, organic, and representative corpora of human text produced by native speakers (the main ingredients in nlp systems) convey many social concepts, including culture, heritage, and even historic biases @xcite @xcite @xcite . one of the widely used human text corpora and a common source of training data for nlp research is wikipedia articles (content pages), especially in languages other than english. in specific, wikipedia articles are used to train many large language models (llms), such as elmo (embeddings from language models), which has been trained on the english wikipedia and news crawl data @xcite ; bert (bidirec-tional encoder representations from transformers) has been trained on books with a crawl of english wikipedia @xcite ; gpt-3 (generative pre-trained transformer) has also been trained on five large datasets including the english wikipedia @xcite ; lamda (language model for dialogue applications) and palm (pathways language model) were trained on a huge mixed dataset that includes english wikipedia articles @xcite ; and recently, llama (large language model meta ai) was also pre-trained on the multilingual articles of wikipedia from june to august 2022, covering 20 languages with a percentage of 4.5% of its overall training dataset size @xcite . wikipedia corpora (editions) exist for over 300 of the over 7,000 languages spoken worldwide. these corpora vary greatly in size and quality, yet simply having a corpus of text in a certain language does not mean that it is an organic corpus representing the culture of native speakers. while native speakers originally write some corpora, others may be written by non-native speakers or translated from other languages @xcite . recent research studied the arabic wikipedia editions: modern standard arabic (ar), egyptian arabic (arz), and moroccan arabic (ary), and found that in the egyptian arabic wikipedia edition more than one million articles have been shallowly translated from english using either direct or templatebased translation, all by a single registered user @xcite . @xcite argued that these shallowly translated articles do not echo the complex structure of the arabic language and its dialects and do not express the views of arabic speakers. in another recent research, @xcite observed that the top ten wikipedia editions (based on the total number of articles) are mostly bot-generated or auto-translated. to mitigate this problem, they introduced an enhanced wikipedia depth metric, depth + , used as a rough indicator for the wikipedia corpora quality, where they quantified and removed bot-generated wikipedia articles and bot-made edits on those articles. both works claimed that these practices of automation and translation could negatively impact the performance of nlp systems trained on these corpora, but they did not provide any empirical studies to show to which extent these practices could implicate the performance of specific nlp tasks and systems, including those using llms. in this paper, we aim to bridge this gap by studying the performance implications of using such unrepresentative, inorganic corpora (produced by template-based translation or automatic bots creation/generation) by intrinsically evaluating two main nlp upstream tasks: word representation and language modeling, using word analogy and fillmask evaluations, respectively, to capture syntactic and semantic relations between words. we purposely choose these intrinsic evaluations over extrinsic evaluations such as text classification or machine translation because many studies have shown that extrinsic and intrinsic evaluations' results are not consistently correlated, and the performance of nlp downstream tasks is always task-specific and can be significantly influenced by fine-tuning procedures @xcite @xcite . we believe that evaluating nlp upstream tasks intrinsically will give us useful insights into the quality of the arabic wikipedia editions' corpora and show how the quality of corpora affects the performance of these nlp tasks. we, in the following sections, discuss the problem of the unrepresentative corpora ( §2), highlight the experimental setup of our study ( §3), present the word representation and language modeling evaluations ( §4 and §5), discuss the results and the limitations of our work ( §6 and §7), provide a brief conclusion and offer future research ideas ( §8). ## problem of unrepresentative corpora the wikipedia corpora (articles) unsurprisingly are not only used to train the large multilingual llms such as bert @xcite , llama @xcite , or even mgpt (multilingual gpt) @xcite , but also have been used to train the majority of the arabic llms, like arabert ( antoun et al., 2020) @xcite , and recently, acegpt @xcite . therefore, there is a need to study wikipedia's corpora representativeness, specifically in the arabic wikipedia editions, and to define the unrepresentativeness in its corpora as well. in this work, we generally define unrepresentative wikipedia corpora as "any wikipedia articles (content pages) that have been created, generated, or edited without human involvement or supervision", such as automatically created, gen-erated, or edited wikipedia articles using bots or shallowly template-translated articles from other highly resourced languages like english. we study this problem from two perspectives: template-translated corpora and bot-generated corpora. for the template-translated corpora, @xcite have studied the arabic wikipedia editions and shown that more than one million articles in the egyptian arabic wikipedia have been directly translated using simple templates that lack rich content from the english language with the help of the off-the-shelf translation tools like google translate. these translation tools generally perform well, but not perfectly, and have several serious problems, such as gender bias, that could adversely affect the translated content @xcite @xcite . for the bot-generated corpora, a few recent research have shed light on the bots' activities on the wikipedia project and their possible negative impacts on the quality of wikipedia corpora @xcite @xcite . the root problem with the bots is that they can rapidly create wikipedia articles (content pages) or edit the contents of those articles without any humans in the loop @xcite @xcite . in this paper, we quantify the bots' activities in all wikipedia editions and study the arabic wikipedia editions closely, specifically activities on their articles. we find that nearly 40% of articles in the arabic wikipedia edition are bot-generated (as demonstrated in figure 1 ), and nearly 12% of articles in the moroccan arabic wikipedia edition are bot-generated, as shown in ## experimental setup in this work, we examine two key nlp upstream tasks, namely word representation and language modeling, using curated corpora of the arabic wikipedia editions' articles and intrinsically evaluate them using two evaluation tasks on two newly created datasets. we next describe the evaluation tasks and our created datasets in more detail. ## word representation evaluations word embeddings are a well-known word representation technique used by modern nlp systems as their backbone. they encode syntactic and semantic relations between words in a text and represent them in a low-dimensional space. ## language modeling evaluations language modeling is an nlp task that generally predicts words in a sentence, and it is the heart of most existing llms. some of these powerful llms, like bert or roberta, are usually trained using two objectives: masked language modeling and next sentence prediction @xcite . in the following subsections, we exploit the masked language modeling objective in training masked language models (mlms) to produce contextual word embeddings and evaluate the performance of the mlm models trained on the arabic wikipedia editions' corpora using our created masked prompts dataset. we evaluate the quality of these mlm models using the pseudo-perplexity metric; we detailedly describe the evaluation process in appendix c. ## discussion recent research has shown that not all wikipedia editions (languages) are produced by native speakers, and there are substantial activities of autocreation of articles (bot-generated articles) and auto-translation of articles (template-translated articles) in wikipedia @xcite . in this work, we argue that this automatic translation of articles, specifically the template-based translation on the egyptian arabic wikipedia edition, impacts the overall performance of the nlp tasks due to having poor, limited, and unrepresentative corpora. we further argue, in this work, that the automatic creation and generation of articles, specifically the bots' creation and generation of articles on the arabic wikipedia and moroccan arabic wikipedia editions, impacts the overall performance of the nlp tasks due to having unnatural, inorganic, and unrepresentative corpora. once again, we find that all the word embedding models that have been trained on the arabic wikipedia and moroccan arabic wikipedia editions follow the same pattern, which is the models trained using the arabic wikipedia edition's corpora after eliminating the bot-generated articles, specifically when top @xmath1, perform better than the models trained on same corpora with bot-generated articles included, and of course, better than all models trained on the moroccan arabic wikipedia edition's corpora, as shown in tables 7 and 8 . we believe when @xmath2, the word embedding models usually show their actual performance, and as demonstrated in lastly, in this work, we strongly emphasize two points. first, we need both large and representative corpora to train nlp tasks and systems efficiently; neither alone is enough. the case of the arabic wikipedia editions gives a unique case study of this since the moroccan arabic wikipedia edition is small but representative, and the egyptian arabic wikipedia edition is large but unrepresentative. second, removing many bot-generated articles from the arabic wikipedia corpora, for example, results in the same or even better performance. due to the rise of generative models and for effective and safe training of nlp tasks and systems, we recommend avoiding using translated or generated corpora, especially when the goal is representationbased tasks like capturing the opinions or identifying the stances of arabic native speakers. ## limitations one limitation of our work is that while the three arabic wikipedia editions provide a unique example of our points, we cannot generalize the study and the impact of inorganic corpora for all the wikipedia editions due to the lack of computational power needed to train the word embedding models and masked language models and due to the im-practicality of creating or collecting factual datasets for the more than 300 languages that exist today on the wikipedia project without using translation. unlike the bots' quantifications process, the other limitation of our work is that the quantification of template-based translations is only specific to the egyptian arabic edition since the wikipedia project does not track template-based translation in its metadata as it does with bot generation. ## conclusion and future work in this work, we demonstrate that for good nlp performance, we need both large and organic corpora; neither alone is sufficient. we show that producing large corpora through automated means can be a counter-productive, producing models that both perform worse and lack cultural richness and meaningful representation of the arabic language and its native speakers. specifically, we demonstrate that training two key nlp upstream tasks, namely word representation and language modeling, on inorganic and unrepresentative corpora negatively impacts the performance of these nlp tasks. we find that the performance of these two nlp tasks is notably influenced by the way the training corpora are produced, where we observe that all models that have been trained on the templatetranslated corpora of the egyptian arabic edition perform the worst when compared with the more representative corpora like the arabic wikipedia edition. we also observe that many models perform the same or better when bot-generated articles are removed. specifically, models trained on the arabic wikipedia edition (40% bot-generated articles) and moroccan arabic wikipedia edition (12% bot-generated articles) perform the same or better when the bot-generated content is removed. in future work, we plan to expand our study of using unrepresentative corpora to include the societal implications (like gender bias and false representations) and security implications (like susceptibility to adversarial robustness) and hope to build a multilevel classification system to detect template-based translation activities such as those seen in the egyptian arabic wikipedia edition.