Statistical Measures for Readability Assessment (2023)

Abstract: Neural models and deep learning techniques have predominantly been used in many tasks of natural language processing (NLP), including automatic readability assessment (ARA). They apply deep transfer learning and enjoy high accuracy. However, most of the models still cannot leverage long dependence such as inter-sentential topic-level or document-level information because of their structure and computational cost. Moreover, neural models usually have low interpretability. In this paper, we propose a generalization of passage-level, corpus-level, document-level and topic-level features. In our experiments, we show the effectiveness of “Statistical Lexical Spread (SLS)” features when combined with IDF (inverse document frequency) and TF-IDF (term frequency–inverse document frequency), which adds a topological perspective (inter-document) to readability to complement the typological approaches (intra-document) used in traditional readability formulas. Interestingly, simply adding these features in BERT models outperformed state-of-the-art systems trained on a large number of hand-crafted features derived from heavy linguistic processing. In analysis, we show that SLS is also easy-to-interpret because SLS computes lexical features, which appear explicitly in texts, compared to parameters in neural models.

## introduction a large number of readability formulas (also called shallow readability indicators) have been developed since the 1940's, but most of them use superficial intra-sentential information (e.g., average sentence length and average character length) without using inter-sentential information such as document-level, corpus-level and topic-level statistics. to address this issue, we introduce statistical lexical spread (sls), and combine it with features derived from idf and tf-idf to train neural and non-neural models on automatic readability assessment (ara). this set of data-driven features can be extracted from any corpus, preferably where documents are categorized into topics. in this project we utilize wikipedia where articles, by design, are grouped into categories. we use these features to augment a bert-based classifier on some benchmark data sets to determine if any significant improvement can be gained from these features or they are already learned by bert embeddings. for this purpose we develop a 'single-shot' model where bert is fine-tuned on the text alone or text combined with the numerical values of our features, and a 'hybrid' model, where the bert pipeline is augmented with the predictions from a non-neural classifier. interestingly, the 'hybrid' mode shows remarkable improvement on the results of the 'single-shot' mode, and overall our models outperform (or compete with) stateof-the-art methods that rely on heavy linguistic processing. to test the generalizability and crosslinguality of our methods, we evaluate our models on english, spanish, and catalan. the advantage of our approach is that no sophisticated nlp processing tools or resources are needed, apart from an optional lemmatizer, which makes it suitable for lowresourced languages. in case a lemmatizer does not exist, sls+tf-idf can still be narrowed down to statistics on the surface forms with even better performance on some data sets, while on others yielding only 1.43% absolute below the highest scores. ## related work with the recent advancement of machine learning (ml), researchers started to apply it to ara usually modeling it as a classification task. early studies introducing ml to ara developed hand-crafted features extracted mostly from the linguistic analysis of texts. for example, @xcite introduced four syntactic features (average parse tree height, and average number of noun phrases, verb phrases, and sbars) to train an svm classifier. @xcite enriched that with features indicating lexical cohesion (e.g. number of pronouns in a sentence, and word overlap between sentences), and discourse connectivity (e.g. whether connectives between sentences are implicit or explicit). over the years, the number of linguistic features kept growing reaching 155 @xcite and @xcite . although the focus on generating sophisticated linguistic features might help advance the state of the art for english, it does not generalize well, due to the fact that many languages have limited nlp tools and resources. for example, (imperial, 2021) used 155 linguistic features for english and only 54 for filipino due to this limitation. this is why we introduce frequency-based features with minimal nlp tooling requirements (only a lemmatizer) that scales well across languages. we also show that even with the lack of a lemmatizer, the nonmorphology based features can still deliver a comparable performance. there have been a few attempts to depart from linguistic features for ara, particularly using the help of language models. for example, @xcite developed 12 language models matching the 12 american grade levels. their language models are unigrams and assume that the probability of a token given the grade level is independent of the surrounding tokens. @xcite used brown clustering which aims to maximize the mutual information of word bigrams. language models, however, focus on corpus-level information, and they do not have a mechanism to account for passage-level, document-level or topiclevel information, which tf-idf and readability formulas, for example, prove to be more suited for. ## data collection and sampling as is the case with document indexing in information retrieval and text mining, for the construction of data-driven features, we need to compute weights for each word to quantify the degree of its familiarity. instead of using a set of web pages, we use wikipedia articles for our indexing purposes. there are three primary advantages of wikipedia for our approach to ara: first, it is available in many languages, second, it covers a broad variety data point count total titles 6,334,131 total categories 1,347,602 total word count 2,363,334,969 titles with categories 4,049,500 singleton categories 310,997 @xmath0,161 * @xmath1,812,671 * @xmath2,131 due to the large number of titles in the english wikipedia, and the fact that many articles are seed articles, i.e. without any substantial content, we sub-sample the data following pareto's principle which states that 80% of consequences come from 20% of the causes. for a total number of 6.3m articles we found that 28.62% of them cover 80% of the word count. by contrast, there are 1.3m categories, and we found that 0.09% of them cover 80% of the titles. the reason that category selection seems to go off the bounds for pareto's principle is that categories are very liberally used in wikipedia. for example, 23.08% of the categories are singletons, i.e. representing only one article. statistics for the english wikipedia data dump of september 2 nd , 2021 are shown in ## feature design and selection understanding a document is dependent on the reader's level of familiarity with the underlying knowledge base (or the topic), which accounts for the connections in the mental map (liu and yuizono, 2020) of the reader and controls the flow of information for updating these connections. this underlying knowledge base indicates the presence or absence of the shared world knowledge between the writer and the reader. approximating this underlying knowledge map can be obtained by analyzing the connection between words within a document and across a reasonably large collection of documents. topic modeling for ara has been discussed in a number of papers. for example, qumsiyeh and ng (2011) developed a system called readaid that used latent dirichlet allocation (lda), an unsupervised learning algorithm, to determine the ranked probability of topics covered in a document based on the distribution of words in the document, or more precisely the probability of a word given a topic p (w|t), and the probability of a topic given a document p (t|d). their model was trained on 53 subject areas extracted from the english curriculum and college board foot_0 and a set of 100 documents randomly selected for each subject area from dmoz foot_1 . @xcite expanded this approach by training lda models on four variations of 50, 100, 150, and 200 topics, and analyzed the output for semantic richness, clarity, and noise, with the purpose of understanding how the topics are distributed, not just what they are. in this work we use sls+tf-idf to model topics using top-frequency categories already curated in the wikipedia corpus, with the intuition that specialized words will occur in fewer categories than common ones. the categories selected in our analysis are based on a ratio of titles and word counts as explained in section 3, and the number is 1,131 for english, 1,536 for spanish, and 1,516 for catalan. the features used in this project are divided into four groups: statistical lexical spread (sls), tf-idf features, document-based counts, and traditional readability formulas (rf). details are explained in the following sub-sections. ## correlation with readability formulas we conducted a comparison between our statistical measures and the traditional readability formulas to see to what degree they are aligned on their predictions. the data set used in this experiment is the simple wikipedia, with a total number of instances of 142,759. we used a split of 66% for training and 34% for testing. we applied the decision tree random forest algorithm, and the results are shown in figure 1 . generally, there seems to be a strong correlation between our sls and dale_chall_readability_score, while document-based and tf-idf have the highest correlation with 'difficult_words'. we notice that some non-english specific indicators, such as 'osman', 'gulpease_index' and 'gutierrez_polini' have higher correlation with our criteria than some english-specific ones, such as 'gunning_fog' and 'smog_index'. this is why we decided to use all 15 formulas in subsequent experiments. it's also interesting to consider the correlation coefficient among the different traditional readability formulas. many pairs of formulas have high correlation, whether negative or positive, which 5 https://it.wikipedia.org/wiki/indice_gulpease figure 1 : correlation between new and traditional readability formulas means that they are looking at the same or similar pieces of information, while many other pairs have a correlation between -0.5 and 0.5 which indicates low correlation, meaning they are looking at different pieces of information, or interpreting the same pieces of information differently. ## testing on benchmark test sets we use our features, along with the readability scores from the traditional readability formulas and build ml models and apply them to two benchmark data sets: a monolingual one, onestopenglish, and a multilingual one, vikiwiki.