LC -Score: Reference-less estimation of Text Comprehension Difficulty (2023)

Abstract: Being able to read and understand written text is critical in a digital era. However, studies shows that a large fraction of the population experiences comprehension issues. In this context, further initiatives in accessibility are required to improve the audience text comprehension. However, writers are hardly assisted nor encouraged to produce easy-to-understand content. Moreover, Automatic Text Simplification (ATS) model development suffers from the lack of metric to accurately estimate comprehension difficulty. We present LC-SCORE, a simple approach for training text comprehension metric for any text without reference i.e. predicting how easy to understand a given text is on a [0, 100] scale. Our objective with this scale is to quantitatively capture the extend to which a text suits to the Langage Clair (LC, Clear Language) guidelines, a French initiative closely related to English Plain Language. We explore two approaches: (i) using linguistically motivated indicators used to train statistical models, and (ii) neural learning directly from text leveraging pre-trained language models. We introduce a simple proxy task for comprehension difficulty training as a classification task. To evaluate our models, we run two distinct human annotation experiments, and find that both approaches (indicator based and neural) outperforms commonly used readability and comprehension metrics such as FKGL.

## introduction the ability to understand text is essential for a wide range of daily tasks. it enables individuals to stay informed, understand administrative forms, and have a full, unimpeded access to social and medical care. studies shows that a large fraction of the population experiences comprehension issues in their daily life. almost half of the oecd population shows reading and written information comprehension difficulties @xcite . such difficulties have a major impact in people's life. in france for example, the national statistic institute @xcite reports that one person out of four has already abandoned an administrative procedure deemed too complicated to follow-along. in order to improve written text accessibility, initiatives such as plain language foot_0 or language clair (lc, translates to clear language) defines writing guidelines to produce clearer texts. moreover, comprehension makes its way into international standards and norms (iso 24495; wcag 2018) but still lacks of concrete solution and measurable objectives. with the rise of deep-learning approaches in natural language processing, as well as its recent successes in a wide variety of tasks (transcription, translation, summarization, question answering), automatic text simplification is an interesting candidate for accessibility improvements at scale. however, system performances are difficult to measure due to the limitations of current automatic metrics @xcite . we hypothesize that the development of better text comprehension metrics could provide automatic text simplification researchers with a way of validating their models while also to giving measurable objectives for the content editors to write clearer texts. in this context, we focus our work in developing models for reference-less text comprehension evaluation as a scoring function for french texts i.e. s : @xmath0, 100] reflecting how clearly written a text is. in this paper, we present the following contributions: ## related work defining what makes a text difficult to understand is a complex task by itself. multiple approaches are explored, like studying the age at which children acquires complex syntactic constructions in french @xcite ; or relying on standardized foreign language levels such as the common european framework of reference (cefr), ranging from a1 to c2. @xcite uses this scale to study french as a foreign language difficulty. in order to improve texts clarity, some organizations produced redaction guidelines i.e. suggestions of good practices to write clear texts, such as plain language (plain) and, in french, @xcite . @xcite also published guidelines for adapting french texts to increase readability and comprehension. more closely related to our work, @xcite introduced a readability formula for french as a foreign language. automatic text simplification aims at generating simpler versions of a source texts. in literature, such models are usually evaluated using automatic metrics. therefore, standard language level and redaction guidelines are hardly suitable to evaluate simplification models since it would require an expert judgement. automatic evaluation instead mostly rely on readability metrics such as fkgl @xcite and gunning fog index @xcite . such metrics were designed with english in mind but can be used on french in practice. on the other hand, samsa @xcite , a semantic metric, is currently not implemented for french, as discussed in section 3.1. other approach include learning regression and classification models @xcite or pretrained language models @xcite . however, @xcite found that automatic metrics remains unsuitable to evaluate progress in automatic text simplification. ## comprehension difficulty annotation we ran two human annotation experiments in two different contexts: the first one using mechanical turk, a crowd-sourcing platform to receive annotations of french speakers from general audience (4.1); the second based on the feedback of langage clair experts in our team (4.2). ## discussions results shows a large improvement of human judgement correlation in favor to our approaches over existing readability metrics. moreover, indicator based method outperform neural models fine-tuned from pre-trained model. neural models' results are promising and could be extended with longer training time and adapting their training objective to produce equally distributed scores. in addition to outperforming neural models, indicator based model are far cheaper to train and predict with since they does not require gpu. being indicator-based makes it easier to interpret and more predictable than neural models, and thus might deliver a better user experience. we observed neural models we trained tend to produce very polarized output probabilities i.e. either very close to 0 or to 1. that's not a problem to quantitatively evaluate the resulting score, but it should probably be adapted to output equally distributed scores in order to be more intuitive. ## conclusion developing methods to accurately measure written text comprehension difficulty is a key challenge that would help better assessing the quality of automatic text simplification models, and provide with a tool for editors to produce texts that are simpler to understand. we explore multiple approaches for training a reference-less metric based on a simple classification task. our systems rely either on linguistic indicators or directly from text. to evaluate our models, we two human annotation experiments. the first involves crowd-sourced workers, asked to compare text based on their comprehension difficulties using best worst scaling with @xmath1. in the second experiment, texts are simplified then rated on a [0, 100] scale by experts from our team. both neural and indicator based methods shows promising results and largely outperform other broadly used readability metrics, on both crowdsourced and expert human annotations. even simple linear models largely outperform readability metrics which adds an evidence against using it to estimate text comprehension complexity. as further researches, we suggest exploring multi-lingual neural training. this would have the obvious benefit of overcoming the language restriction of our work while also mutualizing learning from each language and unifying comprehension difficulties estimation accross languages.