Evaluation of Unsupervised Automatic Readability Assessors Using Rank Correlations (2021)

Abstract: Automatic readability assessment (ARA) is the task of automatically assessing readability with little or no human supervision. ARA is essential for many second language acquisition applications to reduce the workload of annotators, who are usually language teachers. Previous unsupervised approaches manually searched textual features that correlated well with readability labels, such as perplexity scores of large language models. This paper argues that, to evaluate an assessors’ performance, rank-correlation coefficients should be used instead of Pearson’s correlation coefficient ( \rho ). In the experiments, we show that its performance can be easily underestimated using Pearson’s \rho , which is significantly affected by the linearity of the output readability scores. We also propose a lightweight unsupervised readability assessor that achieved the best performance in both the rank correlations and Pearson’s \rho among all unsupervised assessors compared.

## introduction assessing readability plays an essential role in second language acquisition; it can be used for many educational applications such as intelligent reading support systems and placement tests for language classes. readability assessment is a costly task for educational experts and language teachers. to perform it, they must read a text and assess its readability by guessing how difficult the text is for target learning readers. hence, to reduce the cost of the labor required by educational experts, the task of automatically identifying the readability of texts for language learners, known as automatic readability assessment (ara), has been extensively studied in the field of artificial intelligence (ai). unsupervised automatic readability assessment appeared early but has recently been reexamined as a research focus. early studies such as the @xcite , the flesch reading ease formula @xcite , and the flesch-kincaid readability formula (1975) @xcite were unsupervised, as they did not use costly annotated readability labels. given a text, these formulae calculate its readability score based on simple superficial textual features such as the average length of a word in the given text. however, most of these early formulae are designed to assess readability for children who are native speakers. evaluation datasets with readability labels annotated by language teachers targeting second language learners appeared much later, in the 2010s @xcite @xcite . in these works, automatic readability assessment tasks using these evaluation datasets were formalized as a supervised document classification problem, and substantial research efforts were invested into the construction of classifiers by feature engineering to find complicated textual features that correlate well with readability labels. recently, @xcite revisited the unsupervised approach. they proposed that the perplexity scores of neural language models can also be used to represent the readability of text for second language learners and proposed to use them for unsupervised automatic readability assessment. the upper part of fig. 1 show their approach. given a text, their method uses no valuable readability label for training but uses only the language model trained on other large corpora, their method pre-dicts the text's readability score as an output. while this idea is sound, however, their evaluation is validated using the correlation coefficient, or pearson's ρ, as illustrated on the right-hand side of fig. 1 . pearson's ρ measures the degree of linear correlation between two random variables @xcite . as neither the readability levels of the evaluation corpora nor the readability scores output by unsupervised readability assessors are necessarily linear, the use of pearson's ρ can lead to inaccurate evaluation. this study investigates how the use of pearson's ρ affects the evaluation of unsupervised readability assessors. we analyze how unsupervised assessors' performance can be easily underestimated if the readability scores are not linear. for this purpose, we also build a lightweight unsupervised readability assessor, denoted by the lower part of fig. 1 . we show that, alternatively, rank-correlation coefficients are more robust to the linearity and appropriate for this evaluation. the contributions of this paper are summarized as follows. ## automatic readability assessment this section formalizes the problem of automatic readability assessment. let us suppose that we have n texts to assess: we write the set of texts as {t i |@xmath0, . . . , n }}. let y be the set of readability labels. labels are typically ordered in the order of difficulty. for example, in the on-estopenglish dataset @xcite , we can set @xmath1, 1, 2}, where 0 is elementary, 1 is intermediate, and 2 is advanced. the number of levels depends on the evaluation corpus. using y, we write the label for t i as y @xmath2. ## proposed method: vocabulary testing this section describes our unsupervised readability assessor that employs a novel approach: instead of using valuable readability labels as the source of text difficulty for typical second language learners, our proposed method uses vocabulary tests as the source of word difficulty for typical second language learners and obtains readability scores based on accurately estimated word difficulty. to this end, this section explains how to analyze vocabulary test result data to obtain word difficulty. fig. 2 shows example questions from the vocabulary size test, a widely used vocabulary test in applied linguistics @xcite . each question asks about a word in a multiple-choice question format. the test consists of 100 questions like those shown in fig. 2 . @xcite used this test to have 100 second-language learners take the test and to collect their responses. their data were published and made publicly available. we used their dataset to train our classifiers. ## discussion in this paper, we discussed the necessity to use rank correlation coefficients to evaluate automatic readability assessment. the problem that pearson correlation coefficients reflect not only the correlation between two scores but also the linearity of the scores is not particularly novel and has been pointed out for a long time. this study showed that this problem has a significant impact in the evaluation of the ara task. in fact, in the recent evaluation of the ara task @xcite , the problem of linearity in the pearson coefficients was not addressed and its evaluation simply uses the pearson correlation coefficients. to the best of our knowledge, this is the first study to demonstrate the effect of this problem on the performance values in the ara task and examine the extent to which the linearity of the scores affects the scores in tab. 1. in this paper, we termed the proposed method as "unsupervised," according to @xcite . they termed methods that do not use manually annotated readability labels as "unsupervised" even if the methods use supervised machine learning. in fact, the proposed method is trained using the vocabulary test dataset @xcite . phrases, such as "the most average learner" and "learner ability, all refer to the learners on this vocabulary test dataset. in this study, knowing that the term supervised/unsupervised is misleading, we deliberately described the proposed method as "unsupervised" for easy comparison with previous studies. in nlp, the proposed method is closely related to complex word identification (cwi) tasks @xcite . cwi is a task that aims to discover difficult words in a text. the relationship between cwi and personalized text readability was previously studied in @xcite . the task of obtaining the difficulty of an english word for each individual esl learner, as we did in this study, can be regarded as personalized cwi @xcite 8 . personalized cwi has many downstream applications in nlp such as lexical simplification @xcite ), text recommendation for language learners @xcite @xcite , and translator selection in crowdsourcing @xcite . some studies focus on the relationship between word semantics and word difficulty @xcite @xcite . regarding the interpretability of cwi classifiers, ehara (2020a) studied the relationship cwi classifiers' weights and vocabulary sizes. ## conclusions in this paper, we investigated the correlation coefficients to evaluate the performance of unsupervised automatic readability assessors. the experimental results showed that the readability performances measured by pearson's ρ are strongly affected by the linearity of the output scores, whereas those measured by rank correlations are not affected. this indicates the appropriateness of using rank correlation coefficients to evaluate unsupervised automatic readability assessors. we also proposed a lightweight unsupervised assessor based on word difficulty for typical second language learners calculated from a vocabulary test result dataset. this