A Neural Pairwise Ranking Model for Readability Assessment (2022)

Abstract: Automatic Readability Assessment (ARA), the task of assigning a reading level to a text, is traditionally treated as a classification problem in NLP research. In this paper, we propose the first neural, pairwise ranking approach to ARA and compare it with existing classification, regression, and (non-neural) ranking methods. We establish the performance of our approach by conducting experiments with three English, one French and one Spanish datasets. We demonstrate that our approach performs well in monolingual single/cross corpus testing scenarios and achieves a zero-shot cross-lingual ranking accuracy of over 80% for both French and Spanish when trained on English data. Additionally, we also release a new parallel bilingual readability dataset, that could be useful for future research. To our knowledge, this paper proposes the first neural pairwise ranking model for ARA, and shows the first results of cross-lingual, zero-shot evaluation of ARA with neural models.

## introduction automatic readability assessment is the task of assigning a reading level for a given text. it is useful in many applications from selecting age appropriate texts in classrooms @xcite to assessment of patient education materials @xcite and clinical informed consent forms @xcite . contemporary nlp approaches treat it primarily as a classification problem which makes it non-transferable to situations where the reading level scale is different from the model. applying learning to rank methods has been seen as a potential solution to this problem in the past. ranking texts by readability is also useful in a range of application scenarios, from ranking search results based on readability @xcite to controlling the reading level of machine translation output @xcite . however, exploration of ranking methods has not been a prominent direction for ara research. further, recent developments in neural ranking approaches haven't been explored for this task yet, to our knowledge. ara typically relies on the presence of large amounts of data labeled by reading level. further, although linguistic features are common in ara research, it is challenging to calculate them for several languages, due to lack of available software support. though there is a lot of recent interest in neural network based cross-lingual transfer learning approaches for various nlp tasks, there hasn't been much research in this direction for ara yet. with this context, we address two research questions in this paper: the main contributions of this paper are: the rest of this paper is organized as follows: section 2 gives an overview of related research and section 3 describes the proposed neural pairwise ranking model. the next two sections ( 4 5) describe our experimental setup and discuss the results. section 6 summarizes our findings and discusses the limitations of this approach. ## related work readability assessment has been an active area in educational research for almost a century. early research on this topic focused on the creation of readability "formulae", which relied on easy to calculate measures such as word and sentence length, and presence of words from some standard word list @xcite @xcite . more than 200 such formulae were proposed in the past few decades @xcite . the advent of nlp and machine learning resulted in more data driven research on ara over the past two decades. starting from statistical language models @xcite , a range of lexical and syntactic features @xcite @xcite as well as inter-sentential features @xcite @xcite were developed in the past. features motivated by related disciplines such as psycholinguistics (howcroft and demberg, 2017), second language acquisition @xcite and cognitive science @xcite were also explored for this task. in the past few years, ara research has been primarily focused on textual embeddings and deep learning based architectures. word embeddings in combination with other attributes such as domain knowledge or language modeling @xcite and a range of neural architectures, from multi attentive rnn (azpiazu and pera, 2019) to deep reinforcement learning @xcite were proposed. recent research explored combining transformers with linguistic features @xcite @xcite . although a lot of this research evolved on english, the past decade saw ara research in other languages such as german @xcite , french @xcite , italian @xcite , bangla @xcite etc., which employed language specific feature sets. while most ara research modeled one language at a time, some research created language agnostic feature sets and architectures and experimented with 2 to 7 languages @xcite @xcite @xcite . although only one language is considered per model in all this research, there are two important exceptions. madrazo azpiazu and pera (2020b) explored whether combining texts from related languages during training improves ara performance for low resource languages. @xcite used a model trained on english texts on german, based on a common, broad set of handcrafted linguistic features. however, to our knowledge, zero-shot cross lingual transfer of neural network architecture based approaches, without any handcrafted features, was not explored for this task in the past. ara is traditionally treated as a classification problem in nlp research, although there are some exceptions. @xcite compared linear, ordinal, and logistic regression and concluded that ordinal regression with a combination of lexical and grammatical features worked the best for ara, although classification approaches still dominated subsequent research on the topic. there is some past work that considered ara as a pairwise ranking problem, using svm/svm rank and hand crafted linguistic features @xcite @xcite @xcite . while @xcite and @xcite showed that ranking performs better than traditional features and classification/regression respectively, @xcite did not find ranking to be consistently better across the board. given this background, we take a fresh look at the application of ranking for ara, by proposing a new neural pairwise ranking model. ## neural pairwise ranking model the data for our pairwise ranking model takes the form of (document, reading level) pairs. let @xmath0, y 1 ), ..., (x n , y n )] be n such pairs, where x i is the vector representation for document i and y i is the corresponding reading level. we then construct m pairwise permutations from x to form x . the members of x are constructed as follows: if a pair of documents and reading levels (x i , y i ) and (x j , y j ) are chosen, then both permutations ((x i , x j ), (y i , y j )) and ((x j , x i ), (y j , y i )) are added to x . and back-propagate the errors with stochastic gradient descent. this loss function is known as the pairwise logistic loss @xcite . implementation our neural pairwise ranking model (n p rm ) consists of a bert @xcite model as f and a fully connected layer as ψ. we evaluate the performance of the pairwise ranking approach as follows: for a list of texts to be ranked of size s and each text x a within the list, 1 ≤ @xmath1. this pairwise ranking framework allows for n p rm to model relative reading difficulties between texts. while other neural methods have been proposed with more sophisticated learning objectives for ranking problems in the past @xcite , these methods require fixedsize inputs to rank. the n p rm only needs a minimum of two texts to form a ranking for each, and the aggregation process of scores between pairwise permutations of texts can easily produce rankings for an arbitrary list size larger than two. the aggregation process also produces a bounded (by the list size), but continuous score for each document, which results in a ranking with no ties, as long as the input documents are different. additionally, the choice of f in the n p rm framework can allow for multi-lingual predictability, improved performance, or improved efficiency. due to the flexible modeling structure that the n p rm maintains, we hypothesize that with the aid of a multilingual language model, zero-shot cross lingual ara assessment may also be possible with n p rm . we demonstrate this possibility later in the paper foot_0 . ## experimental setup we describe our experimental setup in terms of the datasets used, modeling and evaluation procedures, in this section. ## results we performed within corpus evaluation for classification, and within/cross corpus as well as crosslingual evaluation for regression and ranking. we did not employ classification approaches in the last two evaluation settings as there is no way of resolving ties with classifier predictions. further, regression and ranking gave better performance than classification in monolingual, within-corpus settings. ## conclusions and discussion in this paper we proposed a neural pairwise ranking model for ara (n p rm ). we performed within corpus, cross-corpus and cross-lingual evaluations to benchmark n p rm . our results in the context of the research questions we started with (section 1) are discussed below: limitations of nprm: n p rm models the relative reading difficulty level between texts. while this approach has performed well for our generalizability experiments, there is a general lack of interpretability with n p rm . for example, the newsela-en dataset contains reading level designations that align with the common core educational standards @xcite , and are interpreted to match the school grades of u.s students from kindergarten to high school. since the aggregation process of n p rm sums the predicted scores between pairwise comparisons of an intended ranking, the aggregated score is bounded above by the input list size, is unlikely to correspond to the original reading level scale. further, n p rm takes a list of texts as input and the model forces the constraint of having at least two texts to be ranked as input. hence, n p rm is suitable only for scenarios where ranking by reading level is useful (e.g., ranking of search results by their reading level). outlook: all the five datasets used in these experiments come primarily from news and wikipedia articles. however, the ara is also studied in other domains (e.g., financial disclosures @xcite ). future work can test the validity of the conclusions of this paper in new domains. further, all the datasets are human created texts. it would be interesting to explore how the model works for applications like text simplification and machine translation, which can support existing research on evaluating machine generated text.