Going Beyond Passages: Readability Assessment for Book-level Long Texts (2024)

Abstract: “Readability assessment for book-level long text is widely needed in real educational applica-tions. However, most of the current researches focus on passage-level readability assessmentand little work has been done to process ultra-long texts. In order to process the long sequenceof book texts better and to enhance pretrained models with difficulty knowledge, we propose anovel model DSDR, difficulty-aware segment pre-training and difficulty multi-view representa-tion. Specifically, we split all books into multiple fixed-length segments and employ unsuper-vised clustering to obtain difficulty-aware segments, which are used to re-train the pretrainedmodel to learn difficulty knowledge. Accordingly, a long text is represented by averaging mul-tiple vectors of segments with varying difficulty levels. We construct a new dataset of GradedChildren’s Books to evaluate model performance. Our proposed model achieves promising re-sults, outperforming both the traditional SVM classifier and several popular pretrained models.In addition, our work establishes a new prototype for book-level readability assessment, whichprovides an important benchmark for related research in future work.”

## introduction the readability assessment is to automatically predict the difficulty level of an input text, which has a wide range of applications, such as automating reader advisory @xcite , clinical informed consent forms @xcite and patient education based on the internet @xcite . the readability assessment has a long history of research. in early days, researchers focus on scoring the reading difficulty of texts by designing various readability formulas. later, many works treat it as a classification task, and various linguistic features are exploited to perform machine learning classification @xcite @xcite . in recent years, traditional linguistic features are combined with the pre-trained model to predict the difficulty level of a text @xcite . in addition, there are works regarding readability assessment as a regression task @xcite or a ranking task @xcite . today, reading a whole book plays a vital role to improve children's reading comprehension ability, thus followed by an increasing demand for the book readability assessment. however, most of the existing work processes readability assessment with a passage as input, which is usually an extracted part of a book. these passage-level models have limitations when it comes to the assessment of booklevel long texts. going from passage-level to book-level readability assessment, a challenging task is how to deal with the long input sequence. as is illustrated in figure 1 , the length of a book text is typically tens times more than that of a passage text. to handle the long sequence simultaneously is a hard work. taking the pre-trained model bert as an example, when the mini-batch is 1, using a 24g video memory gpu, the maximum length of slices should be 16 * 512 = 8192. in our children's books data, only 23% [74/320] of the samples were below this value in length. moreover, the widely applied strategies to process long sequence in passage-level text applications, such as truncating, sliding window and pooling, could not work well for book-level long texts because of too much loss of information. nowadays, pre-trained models have made remarkable achievements in many nlp tasks. but these models, such as the pioneer bert @xcite and the long-sequence-specialized bigbird (zaheer et al., 2020), are mostly based on masked language modeling and next sentence prediction, and thus can not express difficulty knowledge. moreover, the computational complexity of the multi-head self-attention mechanism in the model structure is proportional to the square of the length of the input sequence. consequently, directly applying pre-trained models to book readability assessment cannot obtain satisfying results. in this paper, firstly, we investigate whether the passage-level readability assessment model can be applied to predict the reading difficulty of a book, and we get a negative answer, indicating that there exists a large gap between passage-level and book-level readability assessment. therefore, we propose a two-stage model with difficulty-aware segment pre-training and multi-view difficulty representation (dsdr) for this challenging task. specifically, we split all book texts into fixed-length segments, and perform unsupervised clustering to assign each segment with a difficulty label according to linguistic features. the labeled segment difficulty data is exploited to finetune the pre-trained model with a difficulty classification task, in order to enhance the pre-trained model with the ability to recognize and represent difficulty knowledge. in the next stage, the difficulty-aware pre-trained model is employed to extract difficulty features of segments, and multiple difficulty vectors are fused to form the overall representation of a whole book. we construct a graded children's books (gcb) for book-level readability assessment, including 320 popular stories and famous novels in chinese. we conduct detailed experiments on this dataset. the experimental results demonstrate that our proposed model achieves an accuracy of 73.125, which significantly outperforms the traditional machine learning method svm, neural network classifiers and pretrained models. we will make our data and code publicly available for future research. to sum up, our contributions are as follows: ## can passage-level readability assessment model apply to long texts? comparing with passage-level readability assessment, book-level readability assessment is less addressed and more challenging. for both levels, our experimental datasets come from the educational c c l 2 0 2 4 field, and these texts are selected by experts according to children's cognition. therefore, children who master the classroom content (passage-level data) are expected to be able to read the extended books at the same level. in order to investigate whether the trained passage-level readability assessment models can be directly applied to books, we conduct experiments by training on passages and then perform transfer inference on books. ## readability assessment of long texts to handle the book-level long text readability assessment, we propose a dsdr model and the overall structure is shown in figure 3 . based on the transformer structure, dsdr adds two modules: difficultyaware segment pre-training and difficulty multi-view representation. ## conclusion in this paper, we propose a novel model dsdr to assess the reading difficulty of book-level long texts. through difficulty-aware segment pre-training, the pre-trained model is enhanced with the ability to represent difficulty knowledge, and the difficulty multi-view representations help to compress a long text into a representation without loss of information. in addition, we construct a new dataset graded children's books in which each book is labelled with a difficulty level. experimental results show that our approach achieves superior performance against traditional methods and popular pre-trained language models. c c l 2 0 2 4