Linguistic Features for Readability Assessment (2020)

Abstract: Readability assessment aims to automatically classify text by the level appropriate for learning readers. Traditional approaches to this task utilize a variety of linguistically motivated features paired with simple machine learning models. More recent methods have improved performance by discarding these features and utilizing deep learning models. However, it is unknown whether augmenting deep learning models with linguistically motivated features would improve performance further. This paper combines these two approaches with the goal of improving overall model performance and addressing this question. Evaluating on two large readability corpora, we find that, given sufficient training data, augmenting deep learning models with linguistically motivated features does not improve state-of-the-art performance. Our results provide preliminary evidence for the hypothesis that the state-of-the-art deep learning models represent linguistic features of the text related to readability. Future research on the nature of representations formed in these models can shed light on the learned features and their relations to linguistically motivated ones hypothesized in traditional approaches.

## introduction readability assessment poses the task of identifying the appropriate reading level for text. such labeling is useful for a variety of groups including learning readers and second language learners. readability assessment systems generally involve analyzing a corpus of documents labeled by editors and authors for reader level. traditionally, these documents are transformed into a number of linguistic features that are fed into simple models like svms and mlps @xcite . more recently, readability assessment models utilize deep neural networks and attention mechanisms @xcite . while such models achieve state-of-the-art performance on readability assessment corpora, they struggle to generalize across corpora and fail to achieve perfect classification. often, model performance is improved by gathering additional data. however, readability annotations are time-consuming and expensive given lengthy documents and the need for qualified annotators. a different approach to improving model performance involves fusing the traditional and modern paradigms of linguistic features and deep learning. by incorporating the inductive bias provided by linguistic features into deep learning models, we may be able to reduce the limitations posed by the small size of readability datasets. in this paper, we evaluate the joint use of linguistic features and deep learning models. we achieve this fusion by simply taking the output of deep learning models as features themselves. then, these outputs are joined with linguistic features to be further fed into some other model like an svm. we select linguistic features based on a broad psycholinguistically-motivated composition @xcite . transformers and hierarchical attention networks were selected as the deep learning models because of their state-ofart performance in readability assessment. models were evaluated on two of the largest available corpora for readability assessment: weebit and newsela. we also evaluate with different sized training sets to investigate the use of linguistic features in data-poor contexts. our results find that, given sufficient training data, the linguistic features do not provide a substantial benefit over deep learning methods. the rest of this paper is organized as follows. related research is described in section 2. section 3 details our preprocessing, features, and model construction. section 4 presents model evaluations on two corpora. section 5 discusses the implications of our results. we provide a publicly available version of the code used for our experiments. 1 ## related work work on readability assessment has involved progress on three core components: corpora, features, and models. while early work utilized small corpora, limited feature sets, and simple models, modern research has experimented with a broad set of features and deep learning techniques. labeled corpora can be difficult to assemble given the time and qualifications needed to assign a text a readability level. the size of readability corpora expanded significantly with the introduction of the weeklyreader corpus by @xcite . composed of articles from an educational magazine, the weeklyreader corpus contains roughly 2,400 articles. the weeklyreader corpus was then built upon by @xcite by adding data from the bbc bitesize website to form the weebit corpus. this weebit corpus is larger, containing roughly 6,000 documents, while also spanning a greater range of readability levels. within these corpora, topic and readability are highly correlated. thus, @xcite constructed the newsela corpus in which each article is represented at multiple reading levels thereby diminishing this correlation. early work on readability assessment, such as that of @xcite , extracted simple textual features like character count. more recently, @xcite analyzed a broader set of features including out-of-vocabulary scores and syntactic features such as average parse tree height. @xcite assembled perhaps the broadest class of features. they incorporated measures shown by @xcite to correlate well with second language acquisition measures, as well as psycholinguistically relevant features from the celex lexical database and mrc psycholinguistic database @xcite . traditional feature formulas, like the flesch formula, relied on linear models. later work progressed to more complex related models like svms @xcite . most recently, state-of-art-performance has been achieved on readability assessment with deep neural network incor-porating attention mechanisms. these approaches ignore linguistic features entirely and instead feed the raw embeddings of input words, relying on the model itself to extract any relevant features. specifically, @xcite found that a pretrained transformer model achieved state-of-the-art performance on the weebit corpus while a hierarchical attention network (han) achieved state-of-the-art performance on the newsela corpus. deep learning approaches generally exclude any specific linguistic features. in general, a "featureless" approach is sensible given the hypothesis that, with enough data, training, and model complexity, a model should learn any linguistic features that researchers might attempt to precompute. however, precomputed linguistic features may be useful in data-poor contexts where data acquisition is expensive and error-prone. for this reason, in this paper we attempt to incorporate linguistic features with deep learning methods in order to improve readability assessment. ## results in this section we report the experimental results of incorporating linguistic features into readability assessment models. the two corpora, weebit and newsela, are analyzed individually and then compared. our results demonstrate that, given sufficient data, linguistic features provide little to no benefit compared to independent deep learning models. while the corpus experiment results demonstrate a portion of the approaches tested, the full results are available in appendix b ## conclusion in this paper we explored the role of linguistic features in deep learning methods for readability assessment, and asked: can incorporating linguistic features improve state-of-the-art models? we constructed linguistic features focused on syntactic properties ignored by existing features. we incorporated these features into a variety of model types, both those commonly used in readability research and more modern deep learning methods. we evaluated these models on two distinct corpora that posed different challenges for readability assess-ment. additional evaluations were performed with various training set sizes to explore the inductive bias provided by linguistic features. while linguistic features occasionally improved model performance, particularly at small training set sizes, these models did not achieve state-of-the-art performance. given that linguistic features did not generally improve deep learning models, these models may be already implicitly capturing the features that are useful for readability assessment. thus, future work should investigate to what degree the models represent linguistic features, perhaps via probing methods. although this work supports disusing linguistic features in readability assessment, this assertion is limited by available corpora. specifically, ambiguity in the corpora construction methodology limits our ability to measure label consistency and validity. therefore, the maximal possible performance may already be achieved by state-of-the-art models. thus, future work should explore constructing and evaluating readability corpora with rigorous consistent methodology; such corpora may be assessed most effectively using linguistic features. for instance, accuracy could be improved by averaging across multiple labelers. overall, linguistic features do not appear to be useful for readability assessment. while often used in traditional readability assessment models, these features generally fail to improve the performance of deep learning methods. thus, this paper provides a starting point to understanding the qualities and abilities of deep learning models in comparison to linguistic features. through this comparison, we can analyze what types of information these models are well-suited to learning.