P rompt ARA : Improving Deep Representation in Hybrid Automatic Readability Assessment with Prompt and Orthogonal Projection (2023)

Abstract: Readability assessment aims to automatically classify texts based on readers’ reading levels. The hybrid automatic readability assessment (ARA) models using both deep and linguistic features have attracted rising attention in recent years due to their impressive performance. However, deep features are not fully explored due to the scarcity of training data, and the fusion of deep and linguistic features is not very effective in existing hybrid ARA models. In this paper, we propose a novel hybrid ARA model called PromptARA through employing prompts to improve deep feature representations and an orthogonal projection layer to fuse both deep and linguistic features. A series of experiments are conducted over four English and two Chinese corpora to show the effectiveness of the proposed model. Experimental results demonstrate that the proposed model is superior to state-of-the-art models.

## introduction text readability assessment aims to quantify the difficulty of a text, that is, the degree to which it can be easily read and understood @xcite . due to the superiority of an automatic readability assessment (ara) system on assigning a text to a difficulty grade, ara is useful for identifying texts or books that are suitable for individuals according to their language proficiency, intellectual and psychological development. the studies on ara can be traced back to the last century @xcite and have attracted rising attention in recent years, with impressive performance yielded by many neural approaches @xcite @xcite @xcite @xcite @xcite . in the early stage, studies in ara mainly focused on readability formulas, which are typically developed through empirical pedagogy and psychology @xcite . although these formulas are easily interpretable, they rely on surface features and cannot measure the structure or semantic complexity of a text, resulting in unsatisfactory performance. then, the traditional machine learning methods have been applied to train statistical classifiers for ara. these classifiers exploit a large number of features at various levels of a text, including but not limited to vocabulary, semantics and syntax @xcite @xcite @xcite @xcite . although they often achieve better performance than readability formulas, the feature engineering and selection for these machine learning methods are generally time-consuming and labor-intensive. in recent years, deep learning methods have shown impressive performance in natural language processing and their application in ara has been intensively studied. pre-trained word embedding models @xcite @xcite and masked language models such as bert @xcite have been exploited by many neural ara models @xcite @xcite . however, the performance of these deep learning methods is limited by the scarcity of training data in many ara tasks. to further improve the performance of neural approaches based on deep learning, the hybrid ara models utilizing both linguistic and deep features have been recently studied in the literature @xcite @xcite . the literature @xcite investigated the joint effect of handcrafted linguistic features and deep features extracted by a deep neural network, where deep features and hand-crafted linguistic features were simultaneously fed into a machine learning model without fusion to yield readability levels. the recent literature @xcite introduced certain difficulty-aware topic features through utilizing the word difficulty knowledge to guide the training of topic model, and fused the deep features and linguistic features by a projection scheme. although the hybrid ara models have achieved the state-of-the-art performance, there are two limitations for these models. the first one is that deep features are not fully explored in hybrid ara models due to the scarcity of training data. in most of existing neural ara models such as @xcite , deep features are generally extracted by the pre-trained large language models, which commonly require a large amount of training data. the second one is that the fusion of deep features and linguistic features in existing hybrid ara models is not paid particular attention. in this paper, we employ prompts to improve deep feature representations inspired by the great success of prompt learning @xcite @xcite , and an orthogonal projection layer to effectively fuse the improved deep features and linguistic features and particularly remove the redundant information among these features. the major contributions of this paper can be summarized as follows. promptara through exploiting prompts to improve the extraction of deep features, and an orthogonal projection layer to effectively fuse linguistic features and deep features at various levels of a text, including word, sentence and document levels. the proposed model can focus more on some in-domain information of the text with the help of prompts and reduce the redundant information among different levels of feature representations. ## related work text exhibits an inherent structure, and its difficulty is manifested through various linguistic levels, encompassing words, sentences, and entire documents. how to incorporate more effective information of text into ara is one of the current research directions of ara. as the current mainstream models, deep ara models is mainly based on the hierarchical attention networks (han) and pre-trained models @xcite @xcite . the basic idea of such kind of models is to use multi-level deep representations of texts to yield better representations for prediction. in @xcite , a hierarchical attention model called vec2read was suggested for the multilingual ara task, where the word-level information such as the grammatical and morphological information was introduced. in the recent literature @xcite , a novel han-type model was suggested mainly based on the idea of ordinal regression. noticing that deep features extracted by the kind of han models generally have poor interpretability and their capacity is limited by the scarcity of training data, the hybrid ara models by using both linguistic features and deep features have attracted rising attention in recent years @xcite @xcite . in @xcite , the authors directly used deep and linguistic features to jointly train a statistical classifier. in @xcite , the authors firstly yielded an improved linguistic representation by learning the relevance of the used linguistic features and then fused the improved linguistic representation with the deep representation by a simple concatenation to yield a new representation for prediction. the recent literature @xcite deeply investigated the effectiveness of the linguistic features when incorporated into the deep models for ara. in @xcite , the authors firstly introduced a deep topic feature through introducing the difficulty knowledge, and then suggested a projection scheme to fuse the deep and linguistic features by removing redundant information. although existing hybrid ara models achieve the sate-of-the-art performance, the deep features used in these models are extracted by some pretrained language models, which generally require a large amount of training data when adapted to downstream tasks. thus, the performance of these hybrid ara models is limited by the scarcity of training data. the recently suggested prompt learning @xcite provides a potential way to address this issue. in the recent literature @xcite , the authors introduced prompt learning to the problem of judging the difficulty for two given texts, which can be roughly regarded as a binary classification problem, while the ara task considered in this paper can be generally regarded as a multi-classification task. inspired by the great success of prompt learning, this paper employs prompts to improve the deep representations represented in multiple levels, and an orthogonal projection scheme to effectively fuse the deep and linguistic features for prediction. ## proposed model in this section, we describe the proposed model in detail. the overview of the proposed model is depicted in figure 1 . as shown in figure 1 , four types of features including the document-, sentence-and word-level deep features and the traditional linguistic features are utilized for prediction. the document-and word-level deep features are extracted by texts with prompts, while the sentencelevel deep features and traditional linguistic features are extracted from the original texts. the prompt guided document-level features are utilized to improve the other three types of features via an orthogonal projection layer. with these refined four types of features, the prediction of the readability level for a given text is yielded by a linear classifier. ## experimental settings in this section, we describe the experimental settings in detail. we firstly describe the six corpora used in the experiments including four english benchmark corpora and two chinese corpora, and then present the baseline models, and finally describe some implementation details. ## experimental results in this section, we describe and analyze the experimental results in detail. ## conclusion how to fully explore the deep representations and exploit the linguistic representations of texts is important for ara. this paper proposed a novel hybrid ara model through employing prompts to 11 https://huggingface.co/hfl/cino-base-v2 12 https://huggingface.co/bert-base-chinese 13 https://huggingface.co/bert-base-multilingual-cased 14 https://huggingface.co/schen/longformer-chinese-base4096 improve deep representations and fusing the multilevel deep and linguistic representations with an orthogonal projection scheme. the proposed model uses informative prompts to provide additional information to enhance the contextual semantic understanding of the model, and combines textual information about the different structures of the text to yield a richer textual representation. experimental results show that the proposed model achieves excellent performance on both chinese and english datasets compared with existing approaches. on the english datasets in particular, it shows better generalization performance and robustness. in future work, we will further explore how to achieve automatic generation of textual prompts. capturing richer and more accurate information representation at different text levels will also be investigated.