Enhancing Automatic Readability Assessment with Pre-training and Soft Labels for Ordinal Regression (2022)

Abstract: The readability assessment task aims to assign a difficulty grade to a text. While neural models have recently demonstrated impressive performance, most do not exploit the ordinal nature of the difficulty grades, and make little effort for model initialization to facilitate fine-tuning. We address these limitations with soft labels for ordinal regression, and with model pre-training through prediction of pairwise relative text difficulty. We incorporate these two components into a model based on hierarchical attention networks, and evaluate its performance on both English and Chinese datasets. Experimental results show that our proposed model outperforms competitive neural models and statistical classifiers on most datasets.

## introduction readability assessment quantifies the difficulty of a text, that is, the degree to which it can be easily read and understood @xcite . since an automatic readability assessment (ara) system can assign a text to a difficulty grade, it is useful for identifying texts or books that are suitable for individuals according to their language proficiency, intellectual and psychological development. ara research harks back to the last century @xcite and has attracted rising attention in recent years, with impressive performance achieved by many neural approaches @xcite @xcite @xcite @xcite @xcite . there are however a number of limitations in the design and training of current ara models. first, even though difficulty grades are clearly ordinal in nature, most systems approach the task as multiclass classification with independent labels. during training, texts in adjacent grades (e.g., grades 2 and 3) are not treated as more similar than those in distant grades (e.g., grades 2 and 6). second, although a good initialization can optimize performance in many natural language processing tasks @xcite , state-of-the-art ara systems generally rely on random initialization @xcite . this paper aims to further improve ara performance by investigating the following research questions: ordinal information can the use of soft labels for ordinal regression @xcite improve performance? model initialization can the model be better initialized through pre-training on pairwise relative prediction of text difficulty? in contrast to most previous work, we conduct both within-corpus and cross-corpus experiments to answer these questions. within-corpus evaluation may not accurately reflect ara performance when the model is deployed on texts from other collections. further, some features (e.g., text length, topics) could be domain-dependent and may not provide the same performance boost in other domains. the rest of the paper is organized as follows. following a review of previous work (section 2), we propose our model (section 3). we then describe our datasets (section 4) and the experimental setup (section 5). finally, we discuss experimental results (section 6). ## related work early studies in ara mostly focused on readability formulas, typically developed through empirical pedagogy and psychology @xcite . although these formulas have the advantage of being easily interpretable, they rely on surface features and cannot measure the structure or semantic complexity of a text. traditional machine learning methods have been applied to train statistical classifiers for ara. these classifiers employ a large number of features related to vocabulary, semantics and syntax @xcite @xcite @xcite @xcite . although they often outperform readability formulas, feature engineering and selection can be timeconsuming and labor-intensive. deep learning methods, which have shown impressive performance in nlp, have recently been applied in ara. pre-trained word embeddings @xcite @xcite and pre-trained language models such as bert @xcite have been exploited by many neural ara models @xcite . vec2read captures important words and sentences through a multi-level attention mechanism, and uses a bidirectional long short term memory (bi-lstm) to create representations of whole sentences and individual words. it has performed well on multilingual readability assessment by applying transfer learning (azpiazu and pera, 2020). most closely related to our model, hierarchical attention networks (han) @xcite consist of both word and sentence encoders to mimic the hierarchical structure of documents. the word encoder uses bidirectional gated recurrent units (bi-gru) @xcite to embed words while summarizing the information from the context. a word-level attention mechanism then aggregates the most informative words to form a sentence vector. at the sentence level, another encoder likewise uses bi-gru to embed sentences, and an attention mechanism aggregates the most formative sentences into a text vector. originally developed for document classification, it has also shown competitive results in ara @xcite . our proposed model follows the architecture of han, but uses a pre-trained bert and a bi-lstm instead of bi-grus in the word encoder and sentence encoder, repectively. further, it adopts soft labels for ordinal regression and a novel pre-training task for initialization. combining neural models with hand-crafted linguistic features can further improve performance @xcite . since our focus is on neural models that do not require feature engineering, we do not pursue this direction of research. ## proposed model we propose an ara model based on hierarchical attention networks (han) @xcite with two novel components: the use of soft labels to exploit the ordinal nature of the readability assessment task, and a novel pre-training task for model initialization. we will henceforth refer to this model as dtra (deep text readability assessment). the proposed model consists of three components (figure 1 ). the feature representation component (section 3.1), similar to han, constructs the representation for an input text. a fully connected layer following this component is utilized as a classifier, where the cross entropy loss is adopted as the loss function. the soft-label component (section 3.2) exploits the ordinal nature of our task by converting discrete grades into soft labels with a distance metric between grades. the pre-training component (section 3.3) aims to produce a good initialization for fine-tuning. ## data our evaluation makes use of five datasets in english and chinese. we used the 8:1:1 ratio for training, development and test data on all datasets. ## experimental set-up this section presents the baselines to which we will compare our proposed model, the evaluation metrics, and implementation details. ## experimental results we report experimental results in english datasets (section 6.1) and chinese datasets (section 6.2). we then present an ablation study (section 6.3) and a comparison between the soft labels and regression (section 6.4). finally, we discuss results of a crosscorpus evaluation with few-shot learning (section 6.5). ## conclusion this paper has proposed a deep learning model for text readability assessment that achieved competitive performance on the benchmark datasets newsela, onestopenglish and weebit. our model, which is based on hierarchical attention network (han) @xcite , incorporates two novel elements: soft labels for ordinal regression @xcite , and a pre-training task on pairwise relative text difficulty that aims to improve the model initialization for fine-tuning. we conducted experiments on both english and chinese datasets to compare this model to a number of competitive neural models, including bert, vec2read and han. the proposed model outperforms all baselines on most datasets in terms of both accuracy and f1. a lite version of the proposed model, with reduced storage memory requirement, also offered competitive performance. an ablation study demonstrated that the pre-training and the soft labels brought benefits in most datasets. the proposed model also outperformed most baselines in the cross-corpus setting, demonstrating its ability to learn features of text difficulty that are transferable to other kinds of texts.