An Open Multilingual System for Scoring Readability of W ikipedia (2024)

Abstract: With over 60M articles, Wikipedia has become the largest platform for open and freely accessible knowledge. While it has more than 15B monthly visits, its content is believed to be inaccessible to many readers due to the lack of readability of its text. However, previous investigations of the readability of Wikipedia have been restricted to English only, and there are currently no systems supporting the automatic readability assessment of the 300+ languages in Wikipedia. To bridge this gap, we develop a multilingual model to score the readability of Wikipedia articles. To train and evaluate this model, we create a novel multilingual dataset spanning 14 languages, by matching articles from Wikipedia to simplified Wikipedia and online children encyclopedias. We show that our model performs well in a zero-shot scenario, yielding a ranking accuracy of more than 80% across 14 languages and improving upon previous benchmarks. These results demonstrate the applicability of the model at scale for languages in which there is no ground-truth data available for model fine-tuning. Furthermore, we provide the first overview on the state of readability in Wikipedia beyond English.

## introduction the concept of readability aims to capture how easy it is to read a given text, usually defined as the sum of all factors that affect a reader's understanding, reading speed, and level of interest @xcite . in practice, the goal is often to model and quantify the readability of a text on a pre-defined scale using linguistic features, known as automatic readability assessment (ara) @xcite . common approaches are based on readability formulas such as the flesch-kincaid score @xcite , with a recent shift towards more complex computational models leveraging progress on language models in nlp @xcite . these readability scores are used to better serve readers' information needs in educational contexts for choosing appropriate reading materials to support, e.g., language learners @xcite or readers with learning disabilities @xcite . assessing the accessibility of content in terms of readability is also of relevance more broadly, as general textual information found on the web or in the news is often linguistically too complex for large fractions of the population @xcite . this use case for ara is particularly relevant for wikipedia, which has become the largest platform for open and freely accessible knowledge, read by millions of people worldwide with more than 60m articles across 300+ language versions @xcite . unfortunately, this knowledge is believed to remain inaccessible to many readers because the text is written at a level above their reading ability -denoted as the readability gap @xcite . in fact, studies on english wikipedia have concluded that "overall readability is poor" @xcite . however, the state of readability in wikipedia beyond english is unknown. despite recent advances in ara, there is no currently available system to systematically score articles across many languages due to several challenges (see also @xcite ). there is a lack of availability of ready-to-use multilingual approaches, as existing web interfaces such as translated or readable, support only a limited number of languages. at the same time, there are no established readability formulas, such as the flesch reading ease formula, for most languages beyond english. furthermore, models (or formulas) for ara are often designed only for individual or pairs of languages, which makes it challenging to adapt existing models because (i) they are difficult to scale to hundreds of languages, and (ii) resulting scores cannot be easily compared across languages @xcite . furthermore, there is a general lack of ground-truth data. while there are many resources for english, the datasets are often small in size and some of the most commonly-used ones are not available under an open license (such as newsela @xcite or weebit @xcite ), severely limiting their use in real-world applications. beyond english, resources are scarce and scattered, such that there are no ready-to-use datasets in most languages. in this paper, we develop a multilingual system to score the readability of articles in wikipedia (see figure 1 ). specifically, we make the following contributions: traditional approaches research on how to measure readability dates back to more than 100 years @xcite . these early attempts focused mainly on developing vocabulary lists of common words and/or readability formulas, such as flesch reading ease @xcite , smog @xcite , or the dale-chall readability formula @xcite , some of which are still commonly used today. in the past decades, there has been a shift towards computational models using approaches from nlp and machine learning; we point the interested reader to general overviews by @xcite ; @xcite ; @xcite . language models for automatic readability assessment (ara) more recently, ara has been dominated by approaches using language models based on deep neural networks @xcite . a wide variety of architectures have been proposed based on, among others, word embeddings @xcite , multiattentive recurrent neural networks (azpiazu and pera, 2019), and increasingly common transformers (mohammadi and khasteh, 2019) such as bert @xcite . these models have been combined with traditional linguistic features @xcite . as an alternative to the common approach of modeling ara as a classification task, the formulation as a ranking problem has been shown to perform better in cross-corpus and cross-lingual scenarios @xcite . also, recent work utilizes prompt-based seq2seq models to solve ara as a text-to-text generative task @xcite . multilingual ara while most research on ara is focused on english, there have been many efforts in the past years for a broad range of languages such as arabic @xcite , cebuano @xcite , dutch @xcite , french @xcite , german @xcite , greek @xcite , spanish @xcite , or turkish (uluslu and schneider, 2023). however, most of these studies focus only on a single language, with few exceptions attempting to model several languages jointly @xcite . some studies have demonstrated zero-shot cross-lingual transfer for individual pairs of languages (i.e., a model is trained only in one language and then evaluated on another) for english to french @xcite , spanish to catalan @xcite , or english to german @xcite . our work extends these contributions beyond individual pairs, taking advantage of the more general findings that multilingual transformer models, such as mbert @xcite , per-form surprisingly well at zero-shot cross-lingual transfer learning for a wide range of tasks outside ara @xcite . readability in wikipedia there have been efforts to capture readability in wikipedia, with most studies focusing on english and using traditional readability formulas. a comparison of simple and english wikipedia has shown that articles from simple wikipedia are easier to read @xcite , overall readability is insufficient for its target audience (even for simple wikipedia) @xcite , and readability in wikipedia (both english and simple) lags behind other encyclopedias such as britannica @xcite . @xcite analyzed the temporal evolution of simple wikipedia, showing a gradual decline in readability. some studies focus specifically on wikipedia's health-related content finding that most articles remain written at a level above the reading ability of average adults @xcite . readability has also been discussed as a feature for article quality @xcite . ## data we generate a new multilingual dataset of document-aligned pairs of encyclopedic articles, where each pair contains the same article in two levels of readability (easy/hard). the pairs are obtained by matching wikipedia articles (hard) with the corresponding version from different simplified or children's encyclopedias (wmf-a) (easy). the latter encyclopedias are purposefully designed with the goal of creating articles using simpler language (e.g. vocabulary, grammar, and sentence structure) (wmf-b). additionally, past research has shown that articles from simple english wikipedia are easier to read than articles from english wikipedia, using traditional readability formulas for english @xcite . the proposed approach yields a dataset covering 14 languages summarized in the plain text from articles by parsing the html version instead of wikitext; iv) the dataset is publicly available under an open license in contrast to some of the most commonly used resources in ara, such as newsela @xcite . ## application 6.1 state of readability in wikipedia we use the trank model to get an overview on the state of readability in wikipedia beyond english. for this analysis, we consider overall 24 different wikipedias covering all languages from our dataset (sec.3), in addition to a set of 10 languages considered in a prior study covering different language families and taking into account the number and distribution of speakers worldwide @xcite . for each language, we select a random subsample of 10k articles and extract the text following the same methodology as in sec. 3.2. in figure 4 , we show the distribution of readability scores across articles in each language. for english wikipedia, we observe a median score around 1, with the majority of articles above a score of 0 (roughly corresponding to a flesch-kincaid grade level of 9 (sec. 5.3). for most languages, the distribution of scores is similar or shifted towards much higher values, such as italian, with a median of 2.02. some languages show slightly lower scores, such as hungarian, with a median of -0.32. only simple wikipedia, as expected, shows substantially lower readability scores with a median of -2.04 and the 75-percentile below 0. overall, this demonstrates that previous findings about the poor overall readability of english wikipedia @xcite can be generalized to most other language editions of wikipedia.