DEPTH +: An Enhanced Depth Metric for W ikipedia Corpora Quality (2023)

Abstract: Wikipedia articles are a common source of training data for Natural Language Processing (NLP) research, especially as a source for corpora in languages other than English. However, research has shown that not all Wikipedia editions are produced organically by native speakers, and there are substantial levels of automation and translation activities in the Wikipedia project that could negatively impact the degree to which they truly represent the language and the culture of native speakers. To encourage transparency in the Wikipedia project, Wikimedia Foundation introduced the depth metric as an indication of the degree of collaboration or how frequently users edit a Wikipedia edition’s articles. While a promising start, this depth metric suffers from a few serious problems, like a lack of adequate handling of inflation of edits metric and a lack of full utilization of users-related metrics. In this paper, we propose the DEPTH+ metric, provide its mathematical definitions, and describe how it reflects a better representation of the depth of human collaborativeness. We also quantify the bot activities in Wikipedia and offer a bot-free depth metric after the removal of the bot-created articles and the bot-made edits on the Wikipedia articles.

## introduction the wikipedia project is a free online encyclopedia that aims to enable and involve people all over the globe in creating and disseminating knowledge. wikipedia articles, i.e., content pages of wikipedia, are also a common source of training data for natural language processing (nlp) research, especially as a source for corpora in languages other than english. in particular, wikipedia articles are used to train many large language models (llms), such as elmo (embeddings from language models), which has been trained on the english wikipedia and news crawl data @xcite ; bert (bidirectional encoder representations from transformers) has been trained on books with a crawl of the english @xcite ; gpt-3 (generative pre-trained transformer) has also been trained on five large datasets including the english @xcite ; lamda (language model for dialogue applications) and palm (pathways language model) were trained on a huge mixed dataset that includes wikipedia articles, news articles, source code, and social media conversations @xcite ; and llama (large language model meta ai) was also pre-trained on the multilingual articles of wikipedia from june to august 2022, covering 20 languages with a percentage of 4.5% of its overall training dataset size @xcite . wikipedia corpora (editions) exist for more than 300 of the over 7,000 languages spoken worldwide. however, these corpora vary substantially in size and quality, and the wikipedia project provides a rich set of metadata and metrics to help users compare the different corpora. the difference between users and active users is that users refer to the number of user accounts regardless of current activity, whereas active users refer to registered users who have made at least one edit in the last thirty days @xcite . in this paper, we will use the 320 open wikipedia corpora available today, as listed in the appendices. we will not include the 13 closed @xcite . closed editions are read-only, meaning registered users can no longer edit any content pages @xcite @xcite . since articles in closed editions can no longer be edited, the active users metric drops to zero because it only counts users active in the last 30 days 2 . the last three columns of the wikimedia foundation introduced the depth metric as an indicator of wikipedia's collaborative quality to show how frequently a wikipedia edition's articles are edited or updated by the users and is intended to indicate the depth of collaboration among contributors to corpora. the first variant of depth metric was added to the wikipedia project in 2006, using only the first factor, the total number of edits divided by the number of articles. after that, the wikipedia project added an additional factor of non-articles divided by articles. in 2007, the depth metric was again updated to add the third factor, the stub ratio, or one minus the articles divided by the total pages @xcite . in this paper, we aim to explore the limitations of the depth metric and propose a new enhanced depth metric, depth + , to address these limitations. figure 1 previews a comparison of wikipedia's unfiltered depth metric and our depth + metric for the top ten wikipedia editions based on global rank (i.e., the total number of articles). we observe that not all wikipedia editions are produced organically by native speakers, and a substantial level of automation and translation is often used, which can negatively affect the integrity and trustworthiness of these articles. for example, @xcite studied the arabic wikipedia editions (modern standard arabic, egyptian arabic, and moroccan arabic) and found that more than one million articles have been shallowly translated from english using either direct translation or template-based translation (by one registered user) in the egyptian arabic wikipedia edition. unsurprisingly, some of these top ten wikipedia editions, in section 2 examines the current depth metric used in wikipedia, rewrites its mathematical representations, and underscores its limitations. in section 3, the paper quantifies the bot activities within the wikipedia project. section 4 introduces a new metric called depth + , presents its mathematical definitions, and highlights its features. we shed light on the limitations of our work in section 5. lastly, sections 6 and 7 briefly discuss related work, provide a concise conclusion, and offer a few future research ideas. ## depth metric the wikipedia depth metric is currently reported in two places: list of wikipedias @xcite and wikipedia article depth (wikimedia foundation, 2023c). notably, the wikipedia project filters the calculations of this depth metric and reports depth values only for the wikipedia editions with more than 100,000 articles. if a wikipedia edition has a depth @xmath2,000, then the depth metric value is arbitrarily replaced by "--". this has the side effect of placing the english wikipedia edition at the top of wikipedia's ranking by depth metric. to better understand how the depth metric behaves, we manually calculate and report unfiltered depth metric values of all wikipedia editions. returning to in the following subsections, we discuss the formulas of the depth metric, rewrite its mathematical representations, and explain some of its limitations. ## depth + metric the depth metric is a useful indicator of wikipedia's collaborative quality, which reflects how frequently a wikipedia edition's articles are edited or updated by users (wikimedia foundation, 2023c). however, we believe the depth metric must be enhanced to solve some of the limitations spotlighted in this study. in the following subsections, we revise the original depth definitions after quantifying and removing bot activities, propose the depth + metric as an enhanced depth metric for wikipedia corpora quality, mathematically define its definitions, and highlight its key features. ## limitations the depth + metric resolves the problem of botgenerated wikipedia editions that have many botcreated articles and bot-made edits on their articles. yet, the depth + metric does not fix the problem of automatically translated wikipedia editions in the wikipedia project that their articles have been largely translated by poor direct translation or shallow template-based translation. the quantifications of these automatically translated wikipedia editions in the wikipedia project cannot be carried out as systematically as the bot-generated wikipedia editions, and examining each wikipedia edition separately is the only way to accomplish such quantification. another limitation of the depth + met-ric is depending on the active users metric, which dynamically decreases the depth + metric values if there are no editing activities on the articles in the last thirty days. we preferred to use the total unique users who made at least one edit but do not have that figure, so we are approximating it with the already calculated active users metric by the wikipedia project. ## related work due to the widespread use of wikipedia articles as training corpora for many nlp toolchains, especially for low-resource languages, many researchers have addressed the importance of transparency in the wikipedia project, encouraged the transparency values in the project, and proposed improvements on accountability and social transparency through visualizations. for example, @xcite presented a social dynamic analysis tool called "wikidashboard" to improve the social transparency and accountability of wikipedia articles. this tool aims to enhance the interpretation, communication, and trustworthiness of wikipedia articles by visualizing the social dynamics and editing patterns of every article and editor in the wikipedia project. biuk-aghai et al. (2014) also studied the visualization of large-scale human collaboration on the wikipedia project, analyzed the co-authoring across the entire wikipedia editions in various languages (english, german, chinese, swedish, and danish), and found it to follow a geometric distribution in all the investigated language editions. to better understand the geometric distribution of co-author counts across different topics on the wikipedia project, they aggregated wikipedia content by category and visualized it in a form resembling a geographic map. these geographically looking map visualizations show significant differences in co-author counts across different topics in all the visualized wikipedia language editions. at the intersection of transparency and underrepresentation in the wikipedia project, @xcite discussed the available wikipedia corpora for eight languages: english, chinese, arabic, urdu, farsi, french, spanish, and wolof. they closely examined the typical nlp pipeline and highlighted that significant limitations persist even when a language is technically supported, hindering full participation. they specifically compared the number of language speakers to the number of articles in the respective wikipedia edition, using the "articles/1000 speakers" metric. despite the dedicated efforts of numerous wikipedia contributors who have invested substantially in compiling a vast multilingual dataset, not all language speakers have equal opportunities to contribute to the wikipedia project. ## conclusion and future work we have discussed wikipedia's current depth metric in detail, rewritten its mathematical representations, and underlined the limitations of its representation of the depth of collaboration in wikipedia corpora. we also quantified the bot activities in the wikipedia project and excluded the bot-created articles and the bot-made edits on wikipedia articles. we lastly proposed the depth + metric, defined its formal definitions, and highlighted its features, including a better representation of the depth of collaborativeness, a user-centered depth metric, and bot-free wikipedia editions after the removal of the bot-generated articles and the bot-made edits on those wikipedia editions' articles. we hypothesize that a metric that is a better measure of authentic human collaborativeness will be a better measure of the degree to which corpora authentically represents the language and the culture of native speakers. one key aspect of our future work is to find ways to test this hypothesis. specifically, we aim to examine the performance and societal implications of training llms on unrepresentative and inorganic corpora, particularly on the bot-generated wikipedia articles.