Promoting Pre-trained LM with Linguistic Features on Automatic Readability Assessment (2022)

Abstract: Automatic readability assessment (ARA) aims at classifying the readability level of a passage automatically. In the past, manually selected linguistic features are used to classify the passages. However, as the use of deep neural network surges, there is less work focusing on these linguistic features. Recently, many works integrate linguistic features with pre-trained language model (PLM) to make up for the information that PLMs are not good at capturing. Despite their initial success, insufficient analysis of the long passage characteristic of ARA has been done before. To further investigate the promotion of linguistic features on PLMs in ARA from the perspective of passage length, with commonly used linguistic features and abundant experiments, we find that: (1) Linguistic features promote PLMs in ARA mainly on long passages. (2) The promotion of the features on PLMs becomes less significant when the dataset size exceeds 750 passages. (3) By analyzing commonly used ARA datasets, we find Newsela is actually not suitable for ARA. Our code is available at https://github.com/recorderhou/linguistic-features-in-ARA .

## introduction readability is proved to be an objective and consistent @xcite criterion to level reading materials for language learners. leveled reading materials are extensively needed, since language learners at different stages of language acquisition need readings at different readability levels to build up their reading skills @xcite @xcite . however, judging and selecting the readability levels of materials need time and professional knowledge, which is quite inefficient compared to the ever-increasing demand. to address the need for automatically assessing the readability level of a given text, automatic readability assessment (ara) is proposed. in the early time, experts design formulas @xcite @xcite based on the statistics from text such as word length and sentence length. later, researchers @xcite @xcite mine useful morphological, lexical, syntactic and discourse features from text and use them with traditional machine learning models. deep learning models such as rnn-based models @xcite automatically learn dense word embeddings related to the readability of the texts. recently, the popular pre-trained language models (plms) like bert @xcite with their representative dense embeddings are also reported effective @xcite on ara . however, researchers also find handicaps of these deep learning models. since organizing large-scaled ara dataset is difficult due to the time and expertise required, datasets used in ara are relatively small. the insufficiency of data makes it difficult to train a reliable deep learning model @xcite . what's more, as the materials are designed to guide learners step by step, while describing the same thing, the word use, the structure of sentences and the manner of writing the full passages are made stratified intentionally, which is hard to detect for plms inclined to semantic information @xcite . for these reasons, some of them incorporate linguistic features with plms @xcite and achieve improvements. despite their initial success, insufficient analysis of the long passage characteristic of ara has been done before. we notice that the length of passages in ara datasets consisting of reading materials can easily go beyond the capacity of plms (usually 510 tokens). specifically, as shown in fig 1 , most ara datasets have more than 50% passages longer than 510 tokens. through preliminary experiments ( also, we construct an up-to-date and high-quality dataset called raz from raz-kid 2 's printed leveled books. though small-scaled, texts from this popular website make our research more practical. ## data analysis to analyze the effect of linguistic features as precisely as we could, we select four different datasets namely weebit @xcite , newsela @xcite , onestopenglish @xcite and raz. the characteristics of the 4 datasets are listed in newsela is a text simplification dataset divided into 5 simplification levels. texts from the hardest level are rephrased 4 times to create other 4 easier levels. following previous works, we consider each simplification level a readability level. weebit is an ara dataset. texts from different readability levels focus on different topic. we sample 625 instances each level to construct a balanced dataset. onestopenglish is a relatively small text simplification dataset containing 560 passages. similar to newsela, it is also constructed by rephrasing. raz is an ara dataset constructed by us. we select 370 passages from the raz-kid 2 , an online education platform providing lots of leveled ebooks. we manually annotate them with 3 different readability levels according to the readability level criterion 1 . compared to the above datasets, raz contains more text genres, topics and up-todate vocabulary. more importantly, the average length of raz is much longer than the other three datasets, indicating that it is suitable for exploring the effect of linguistic features on long passages. ## method task description given a dataset @xmath4, p 2 , . . . , p |d| } with d readability levels @xmath5, c 2 , . . . , c d } . each passage p i in dataset d is mapped to one label in c. it can be regarded as a classification task, a ranking task or an ordinal regression task. we take this task as a classification task for its simplicity. model for each passage @xmath6, x p 2 , .., x p l ] which has l tokens, we concatenate our extracted linguistic features f p (see ## conclusion in this paper, we investigate how linguistic features promote plms on ara from the perspective of passage length. firstly, two self-proposed hypotheses are proved: 1. linguistic features provide linguistic information that plm is not good at capturing; 2. linguistic features provide information about the segment dropped by plm. secondly, we observe that the promotion of the features on plms becomes less significant when the dataset size exceeds ∼750 passages. thirdly, our results suggest that newsela dataset is possibly not suitable for ara.