A Unified Neural Network Model for Readability Assessment with Feature Projection and Length-Balanced Loss (2022)

Abstract: Readability assessment is a basic research task in the field of education. Traditional methods mainly employ machine learning classifiers with hundreds of linguistic features. Although the deep learning model has become the prominent approach for almost all NLP tasks, it is less explored for readability assessment. In this paper, we propose a BERT-based model with feature projection and length-balanced loss (BERT-FP-LBL) to determine the difficulty level of a given text. First, we introduce topic features guided by difficulty knowledge to complement the traditional linguistic features. From the linguistic features, we extract really useful orthogonal features to supplement BERT representations by means of projection filtering. Furthermore, we design a length-balanced loss to handle the greatly varying length distribution of the readability data. We conduct experiments on three English benchmark datasets and one Chinese dataset, and the experimental results show that our proposed model achieves significant improvements over baseline models. Interestingly, our proposed model achieves comparable results with human experts in consistency test.

## introduction readability assessment is to automatically determine the difficulty level of a given text, aiming to recommend suitable reading materials to readers. there are wide applications of readability assessment, such as automating readers' advisory (pera and ng, 2014), clinical informed consent forms @xcite and internet-based patient education materials @xcite . comparing with other natural language processing (nlp) tasks, readability assessment is less explored. in the early days, researchers exploit linguistic features to develop various readability formulas, such as flesch @xcite , @xcite ) and * corresponding author. @xcite . later, the mainstream research @xcite @xcite is to employ machine learning models to classify a text, by designing a large number of linguistic features. there are also works that treat it as a regression task @xcite or a ranking task @xcite . recently, unlike other nlp tasks, the introduction of deep neural networks for readability assessment does not achieve overwhelming advantages over traditional machine learning methods. employing neural network models for readability assessment, there are several challenges: in order to solve the above problems, we propose a bert-based model with feature projection and length-balanced loss (bert-fp-lbl). with the pre-trained bert as the backbone, we employ feature projection to integrate linguistic features into the neural model, and design a new length-balanced loss function to guide the training. concretely: we conduct experiments on three english benchmark datasets, including weebit @xcite , onestopenglish @xcite and cambridge @xcite , and one chinese dataset collected from school textbooks. experimental results show that our proposed model outperforms the baseline model by a wide margin, and achieves new state-of-the-art results on weebit and cambridge. we also conduct test to measure the correlation coefficient between the bert-fp-lbl model's inference results and three human experts, and the results demonstrate that our model achieves comparable results with human experts. ## related work traditional methods. early research efforts focused on various linguistic features as defined by linguists. researchers use these features to create various formulas for readability, including flesch @xcite , @xcite and smog @xcite . although the readability formula has the advantages of simplicity and objectivity, there are also some problems, such as the introduction of fewer variables during the development, and insufficient consideration of the variables at the discourse level. machine learning classification methods. @xcite develop a method of reading level assessment that uses support vector machines (svms) to combine features from statistical language models (lms), parse trees, and other traditional features used in reading level assessment. subsequently, (petersen and ostendorf, 2009) present expanded results for the svm detectors. @xcite designe 100 factors to systematically evaluate the impact of four levels of linguistic features (shallow, pos, syntactic, discourse) on predicting text difficulty for l1 chinese learners and further selected 22 significant features with regression. @xcite design experiments to analyze the influence of 88 linguistic features on sentence complexity and results suggest that the linguistic features can significantly improve the predictive performance with the highest of 70.78% distance-1 adjacent accuracy. @xcite evaluate the joint application of handcrafted linguistic features and deep neural network models. the handcrafted linguistic features are fused with the features of neural networks and fed into a machine learning model for classification. neural network models. @xcite provide the knowledge-enriched word embedding (kewe) for readability assessment, which encodes the knowledge on reading difficulty into the representation of words. (azpiazu and pera, 2019) present a multi-attentive recurrent neural network architecture for automatic multilingual readability assessment. this architecture considers raw words as its main input, but internally captures text structure and informs its word attention process using other syntax and morphology-related datapoints, known to be of great importance to readability. @xcite propose a new and comprehensive framework which uses a hierarchical selfattention model to analyze document readability. @xcite form a correlation graph among features, which represent pairwise correlations between features as triplets with linguistic features as nodes and their correlations as edges. ## methodology the overall structure of our model is illustrated in figure 1 . we integrate difficulty knowledge to extract topic features using the anchored correlation explanation (corex) @xcite , and fuse linguistic features with neural network representations through projection filtering. further, we propose a new length-balanced loss function to deal with the unbalanced length distribution of the readability assessment data.