Exploring hybrid approaches to readability: experiments on the complementarity between linguistic features and transformers (2024)

Abstract: Linguistic features have a strong contribution in the context of the automatic assessment of text readability (ARA). They have been one of the anchors between the computational and theoretical models. With the development in the ARA field, the research moved to Deep Learning (DL). In an attempt to reconcile the mixed results reported in this context, we present a systematic comparison of 6 hybrid approaches along with standard Machine Learning and DL approaches, on 4 corpora (different languages and target audiences). The various experiments clearly highlighted two rather simple hybridization methods (soft label and simple concatenation). They also appear to be the most robust on smaller datasets and across various tasks and languages. This study stands out as the first to systematically compare different architectures and approaches to feature hybridization in DL, as well as comparing performance in terms of two languages and two target audiences of the text, which leads to a clearer pattern of results.

## introduction a significant proportion of the population suffers from their poor reading skills in their everyday life @xcite , for example to access medical information @xcite or to process administrative tasks @xcite . this issue may be tackled with automatic readability assessment (ara); for example by automating recommendations of texts suited to specific reading levels @xcite . ara has leveraged automatic annotation of textual features, and machine learning (ml) algorithms. in this context, ara has largely been modeled using feature engineering @xcite @xcite . current works rely on distributed representations of texts (i.e. embeddings) @xcite and deep learning (dl) @xcite @xcite , yielding improvement over linguistic feature-based systems (e.g., @xcite ; @xcite for english and @xcite for french). consequently, dl has become the new standard in ara. however, linguistic feature engineering has not been completely discontinued @xcite . we emphasize two main reasons for that. first, obtaining audience-specific data to produce large corpora, required for dl, is difficult, and vanilla transformers tend to achieve low performance on small readability datasets @xcite . second, feature-based approaches bring knowledge from cognitive psychology and the modelling of difficulty @xcite , offering insights on how textual characteristics affect readers @xcite . in this work, we focus on hybrid models as a way to combine the accuracy of dl with the grounded interpretability of features, with minimal pre-training costs. 1 we aim to identify an effective architecture for combining linguistic features and transformers for ara, keeping in mind that there may be an overlap of the information encoded in both representations @xcite @xcite . although this work focuses on ara, the methodology presented here can be applied to other tasks, particularly those tasks that rely on a restricted data set. the main contributions of this paper are: (1) a systematic analysis of how hybrid architectures compare with traditional ones 2 , (2) recommendations for the best hybrid architecture for ara, and (3) a study of how those models are impacted by corpora properties (e.g. language, or l1 vs. l2). the paper is structured as follows: we discuss existing work in more details (section 2), we detail our approach (section 3) and present the results we obtained (section 4). we then present an in-depth error analysis (section 5) before concluding (section 6). ## related work the inclusion of linguistic features in dl models has been done in various areas of nlp. in some cases, the purpose is to provide additional information that a dl model does not have access to (e.g. information about products @xcite ). additionally, linguistic information can be included to facilitate the learning task, by providing complementary information or information poorly presented in the model. the inclusion of features in dl requires changes in the architecture, which can be done by adding additional layers or modifying the existing ones foot_2 . in this section, we examine how this modification in architecture is carried out in nlp and particularly in ara. ## methodology in order to find the best approach for combining features and embeddings for ara, we carried out a systematic comparison of architectures by comparing hybrid and non-hybrid (baselines) architectures. to this end, we selected 4 readability corpora with various characteristics (section 3.1), on which we computed linguistic features (section 3.2) before comparing the performance of the 8 architectures described in section 3.3. to this aim, we split each corpus into train, validation and test sets (60/20/20) using stratified cross validation with groups defined based on target difficulty level and text genre (when available). for comparing performance, we applied the friedman and mann-whitney u tests. ## error analysis readability assessment can be strongly influenced by the genre of the documents @xcite . to investigate this effect in the context of our experiments, we computed the best models' performance scores on each genre of the fle corpus (i.e., informative, narrative, dialogue, mail/e-mail and miscellany) and the clear corpus (i.e., informational and literature). results are presented in we also investigated the effect of the task on model performance to assess whether readability predictions could be influenced by the audience (i.e., l1 vs. l2). to ensure a fair comparison between our corpora of different sizes, we used the fle small and clear small corpora in this study. models' performance scores are statistically higher for l2 than for l1 reading ( given the large number of features available after the automatic annotation, we investigated the occurrence of features associated with the prediction error of the models. in this study, the feature selection method described in section 3.2 was used to select the top 100 features associated with error (i.e, statistical residuals). first of all, it is interesting to note that some features used by the models are still correlated with error, hinting that architectures might not have exploited all the information available in the features. the flm corpus is the most impacted since the intersection between errorrelated (100 features) and available in the training (200 features) includes 9 features for sc and 20 for so. moreover, we can note that, while lexical features account for roughly half this intersection for both models, discourse features accounts for 30% in sc, but for only 17% in so. for each architecture, we then looked at the intersections of these feature lists (error-related and feature set) for the two languages (english and french) and the two tasks (l1 and l2). for the sc architecture, the size of the feature intersections for french (10) and english (11) is larger than for l1 (4) and l2 (3). if we compare the two architectures, we observe that the intersections tend to be smaller for so than for the sc, suggesting that this model might be able to make better use of the features, which could then be an explanation for his marginal superiority. we also noted that the large proportion of lexical features for french (80% vs. 10% for english) is specific to the sc architecture. however, in both models, the intersection for french only includes lexical and syntactic features, and does not include any features related to relationships beyond the sentence level, contrary to english. ## conclusion in this paper, seeking to combine the accuracy of dl with the theory-grounded interpretability of features, we carried out a systematic investigation of how to combine transformers and linguistic features. to this end, we compared 8 different architectures (6 hybrid and 2 baselines) on 4 corpora (in different languages and readability tasks). we observed that a soft label architecture obtained the best overall performance, followed by simple con-