Learning From A rabic Corpora But Not Always From A rabic Speakers: A Case Study of the A rabic W ikipedia Editions (2022)

Abstract: Wikipedia is a common source of training data for Natural Language Processing (NLP) research, especially as a source for corpora in languages other than English. However, for many downstream NLP tasks, it is important to understand the degree to which these corpora reflect representative contributions of native speakers. In particular, many entries in a given language may be translated from other languages or produced through other automated mechanisms. Language models built using corpora like Wikipedia can embed history, culture, bias, stereotypes, politics, and more, but it is important to understand whose views are actually being represented. In this paper, we present a case study focusing specifically on differences among the Arabic Wikipedia editions (Modern Standard Arabic, Egyptian, and Moroccan). In particular, we document issues in the Egyptian Arabic Wikipedia with automatic creation/generation and translation of content pages from English without human supervision. These issues could substantially affect the performance and accuracy of Large Language Models (LLMs) trained from these corpora, producing models that lack the cultural richness and meaningful representation of native speakers. Fortunately, the metadata maintained by Wikipedia provides visibility into these issues, but unfortunately, this is not the case for all corpora used to train LLMs.

## introduction natural language processing (nlp) is increasingly used as a key ingredient in critical decision-making systems, such as resume parsers used in sorting a list of job candidates. these nlp systems often ingest large corpora of human text, attempting to learn from past human behavior to produce systems that will make recommendations about our future world @xcite . the corpora of human text, which are the main ingredients in nlp systems, convey many social concepts @xcite , including culture, heritage, and even historic biases @xcite @xcite . google news, books corpora, wikipedia, and the glue (the general language understanding evaluation) dataset @xcite are all examples of the many digital text corpora that have been used in nlp research. many languages are substantially underrepresented in both corpus development and nlp toolchain support. for example, there are more than 7000 spoken languages around the globe, and only 300 have wikipedia corpora. among these 300, there is wide variation in raw corpus size as well as the ratio of articles to the number of speakers. these differences are further amplified throughout the nlp toolchain @xcite . languages without large corpora also often face a lack of support in common nlp tools and unexpected errors in other tools due to a lack of testing and use. this under-represents the culture and heritage of speakers of those languages in nlp-guided decision-making. in addition, simply having a corpus of text in a language does not necessarily represent the culture of native speakers of that language. while some corpora are originally written by native speakers, others may be written by non-native speakers or even translated from other languages @xcite . it has also been observed that some wikipedia corpora have been developed/created through bots or automated scripts, often involving translation from other languages @xcite . this paper highlights this less discussed yet important issue of the differences between text corpora written by native speakers and those translated and generated by automated systems. we also discuss their potential effects on downstream nlp systems. as a case study, we document discrepancies between arabic wikipedia editions and egyptian arabic wikipedia. in section 2, we discuss some related work, and in section 3, we study wikipedia and its arabic editions, using english as a benchmark. lastly, in sections 4 and 5, we discuss our findings with a focus on the representativeness of nlp corpora, provide a few recommendations, and conclude with a short conclusion and pitch future work ideas. to help with issues related to exclusion and bias, @xcite presented the approach of including data statements in all publications and documentation for nlp systems. the approach aims to yield various short-term and longterm benefits, including unfolding how data represents the people and the world, enabling research addressing issues of bias and exclusion, promoting the development of more representative datasets, and making it convenient for researchers to consider stakeholder values as they work. @xcite raised the concern about the quality of data analysis methods before model development related to the cost and standardization. they presented the dataset nutrition label, a diagnostic framework to aid standardized data analysis, making it more adaptable across domains. they also explored the limitations of the label, including the challenges of generalizing across diverse datasets and guidelines for future research and policy agendas for the project. likewise, to clarify the intended use cases of ml models and limit their usage in a context not well suited for them, @xcite suggested a framework named model cards to promote transparency in model reporting using short documents. @xcite studied dataset deprecation in ml and proposed a data deprecation framework focusing on risk, impact mitigation, appeal mechanisms, timeline, post-deprecation protocols, and publication checks that can be adapted and implemented by the ml community. they also advocate for a centralized, sustainable repository system for archiving datasets, tracking dataset deprecations, and helping to enable practices that can be integrated into research and publication processes. to fill the gap in the standardization process in documenting datasets, @xcite proposed datasheets for datasets, i.e., each dataset should be accompanied by a datasheet explaining its motivation, composition, collection process, recommended uses, etc. it aims to bridge the gap between creators and users of datasets and establish a communication channel taking a step toward ensuring transparency and accountability in datasets and ml systems. @xcite proposed factsheets to help increase trust in ai services and envisioned such documents to contain purpose, performance, safety, security, and provenance information to be completed by ai service providers for consumer examination. @xcite outlined a research program -a genealogy of machine learning data -for investigating how and why datasets have been created, what and whose values influence the data collection choices, and the contextual and contingent conditions of their creation. @xcite introduced a framework for dataset development transparency that supports decisionmaking and accountability. the framework uses dataset development's cyclical, infrastructural, and engineering nature to draw on best practices from the software development lifecycle. wikipedia is used frequently in nlp research, including multilingual nlp @xcite @xcite @xcite @xcite @xcite @xcite @xcite . for example, beytía et al. (2022) documented a gender gap in wikipedia biographical articles over a dataset of almost 6.2 million wikipedia biographical articles across the 10 most spoken languages. the analysis was performed by proposing 4 multimodal metrics of the amount and quality of visual and written content. they found that text content favors female biographies, while the image quantity favors males, and the multilingual article coverage is biased slightly towards women. similarly, a dataset by @xcite , covering 309 language editions and 33m wikipedia articles, was presented to explore inter-language knowledge propagation by tracking the full propagation history of concepts in wikipedia. this allows follow-up research on building predictive models with the help of aligned wikipedia articles in a language-agnostic manner according to the concept they cover, resulting in 13m propagation instances. @xcite provide background about what differences might arise between different language editions of wikipedia and how that might affect their models. the authors discuss three major ways content differences between language editions arise (local context, community and governance, and technology), recommend good practices when using multilingual and multimodal data for research and modeling, and suggest researchers expand the models available to wikipedians for translating articles into their language. in the space of the arabic nlp, many researchers have studied the translation of the english language content to the arabic language or its dialects back and forth using machine translation models (mts); especially the statistical machine translation models (smts) and the neural machine translation models (nmts), which achieved an excellent quality of translation @xcite @xcite @xcite . several studies have utilized the mts to translate the egyptian dialect to modern standard arabic (msa) or vice versa. for example, @xcite was the first work in this domain where the authors introduced a hybrid approach to translating an egyptian sentence into its corresponding sentence in the msa. in @xcite , the author presented the opposite way, where they introduced a translator from the msa to the egyptian dialect. the recent work of @xcite presented many smt systems to translate from english to dialectal arabic (da) -the egyptian arabic dialect, using msa as a pivot. ## the case of wikipedia wikipedia corpora (i.e., content pages of wikipedia) are used to train llms. for example, elmo (embeddings from language models) has been trained on the english wikipedia and news crawl data @xcite , bert (bidirectional encoder representations from transformers) has been trained on the bookcorpus @xcite with a crawl of the english wikipedia @xcite , and gpt-3 (generative pre-trained transformer) has been trained on five large datasets including the english wikipedia as well @xcite . nlp researchers find wikipedia corpora attractive because of its large collection of multilingual content and its vast array of metadata that can be quantified and compared across the multilingual content pages @xcite ). yet, recent works have underlined that those pre-trained llms embed bias, stereotypes, or even politics. unlike many corpora, wikipedia maintains rich metadata that allows researchers to assess the source of its contents, but little work has shown explicitly how different wikipedia corpora impact these models @xcite @xcite . at the same time, other recent works have also reported that the current pre-trained llms still under-represent the human languages despite being trained with hundreds of billions of parameters and trained on enormous datasets @xcite . in the following subsections, we compare the arabic wikipedia editions (modern standard arabic, egyptian, and moroccan) regarding pages to date, new pages, and top editors, besides english wikipedia as a benchmark. 1 we also specifically study the impact of problems in egyptian arabic wikipedia, including large-scale auto-generation and poor translation of content pages from english. ## discussion the arabic language, in general, poses many challenges in nlp that prevent simply translating from another language like the english language due to it is morphological richness and high ambiguity @xcite . additionally, the arabic language has many dialectal variants, like egyptian and moroccan arabic, that are different from msa. these dialects are primarily spoken, do not have written standards, and have very few resources @xcite . despite all these challenges the arabic nlp faces, translating english content, especially from wikipedia, to enrich lowresource languages' content like the arabic language or any of its dialects like egyptian is a common practice, which is mainly done using machine translation models (mts) that existed in the 1950s and have evolved since then until today @xcite . recently, wikimedia foundation has encouraged users, editors, and contributors to use mts to translate and create the initial content of articles on the wikipedia project using their content translation tool. this tool is a product of collaboration between google (google translate) and the wikimedia foundation, and this tool has been used to translate more than 400,000 articles on wikipedia @xcite . without a doubt, the foundation seeks to improve the quality of the multilingual content of wikipedia via article translation using translation tools like google translator. still, it is important to consider the quality of these translation tools, the quality of the translation work conducted by non-expert wikipedia users or bots, and what they could bring to the multilingual content of wikipedia from potential serious issues, such as religious, political, or gender biases. another serious problem is the unrepresentativeness of the content, especially when users or bots could create shallow content automatically (like what we saw in the egyptian arabic wikipedia) using templates and translation tools that do not profoundly understand the targeted language @xcite @xcite . the heart of the lack of representativeness problem, specifically in the arabic language, can be discussed from two different perspectives: the largescale unsupervised automated generation of content, especially in wikipedia, and the translation of content from english to other low-resource languages like arabic using direct translation methods or tools like google translator. we have analyzed the egyptian arabic wikipedia and found that more than 1 million articles have shallow content and are translated poorly from english to msa. until now, no one knows how the responsible user converted the translated msa content to the egyptian dialect. we suspect that most of these content articles have not truly converted to the egyptian dialect and are still in msa. it would be easy for users to assume that the egyptian arabic wikipedia corpus was genuinely representative of the egyptian people, their culture, heritage, or traditions. however, the many documented reasons indicate otherwise. the other face of the lack of representativeness problem is when users or bots translate the content of the english language, for example, to other low-resource languages like arabic using direct translation or off-the-shelf translation tools. most of these translations done on wikipedia content, in general, are done using direct translation, meaning that we are translating from language a to language b. the bottleneck for this kind of translation is the quality of the translation tool. the quality of the translation is likely superior if the tool is sophis-ticated, uses state-of-the-art technologies, and is trained on large parallel corpora of a and b languages. however, the existing off-the-shelf translation tools like google translator perform well, but not perfectly, and have many ethical problems like sexism and a few biases that could badly affect the translated content @xcite . it would also likely retain the sentiment, culture, and biases from the origin/source corpora rather than represent the society of native speakers of the targeted language. jeblee et al. (2014) designed three different translation systems: baseline mt system, where they directly translated english to egyptian arabic; onestep adoption mt system, where they directly translated english to msa, used domain and dialect adoption, and translated the results to the egyptian arabic; and two-step adoption mt system, where they directly translated english to msa, then used domain adoption, then in-domain msa to dialect adoption to lastly translated the results to egyptian arabic. such a complex work is what we meant by performing a sophisticated translation. we do not doubt such systems will produce a significantly accurate translation between english and egyptian arabic and could solve the problem of the lack of representativeness of the egyptian arabic wikipedia content if it has been used. nevertheless, the selection of which articles to write or translate and which aspects to highlight in an article would still not reflect the choices of native speakers. as a big concern, a few researchers have studied the implications of using corpora that are automatically created, poorly translated using direct translation, automatically generated by advanced llms like elmo, bert, or gpt-3, or even the textual content of the assembled corpora using text augmentation techniques @xcite @xcite @xcite . we believe that those llms, mts, automation, and augmentation procedures will likely produce corpora full of serious issues. these corpora do not only embed bias, stereotypes, or even politics @xcite @xcite @xcite , but they also do not echo the complex structure of the arabic language and its dialects, do not express the views of the arabic speakers, and do not represent the cultural richness and historical heritage of the arabic language and its people. ## conclusion and future work we studied, in this work, the arabic wikipedia editions (modern standard arabic, egyptian, and moroccan) besides english wikipedia in terms of their pages to date, new pages, and top editors, and shed light brightly on the problem of the egyptian arabic wikipedia, where we found that one registered user has automated the creation of over 1 million content pages in less than 3 years and used shallow, template-based translation method that does not represent speakers of egyptian arabic. we recommend that nlp practitioners avoid the inorganic, unauthentic, unrepresentative corpora in their applications (e.g., pipelines) when the goal is to learn from past human behavior and to thoroughly investigate how the corpora they do use were created, generated, or assembled; it is especially important to corpora that are produced by native speakers when the point is to examine culturally sensitive issues such as religious bias or gender bias, or political sentiment, etc. we have shown that currently, in wikipedia, it is important to look beyond simply the "registered user" vs. "bot" distinction to recognize automated contributions, e.g., adding a "registered user (automation-assisted)" category will help us to distinguish between organically and automatically produced contributions by registered users. in the future works, we plan to study the implications of using such unrepresentative corpora that are naively auto-created, shallowly translated, or automatically generated on the downstream applications of the nlp. we are compiling a list of alternative egyptian arabic corpora that have been introduced to the research community and are most likely to be organic, authentic, and representative corpora of the egyptian arabic dialect and its speakers. we also plan to introduce a representativeness metric that could assist in identifying the auto-generated content pages on the wikipedia project. lastly, we plan to design a neural network classifier that could aid in classifying the corpora in terms of representativeness.