ARTS : Assessing Readability & Text Simplicity (2024)

Abstract: Automatic text simplification aims to reduce a text’s complexity. Its evaluation should quantify how easy it is to understand a text. Datasets with simplicity labels on text level are a prerequisite for developing such evaluation approaches. However, current publicly available datasets do not align with this, as they mainly treat text simplification as a relational concept (“How much simpler has this text gotten compared to the original version?”) or assign discrete readability levels.This work alleviates the problem of Assessing Readability & Text Simplicity. We present ARTS, a method for language-independent construction of datasets for simplicity assessment. We propose using pairwise comparisons of texts in conjunction with an Elo algorithm to produce a simplicity ranking and simplicity scores. Additionally, we provide a high-quality human-labeled and three GPT-labeled simplicity datasets. Our results show a high correlation between human and LLM-based labels, allowing for an effective and cost-efficient way to construct large synthetic datasets.

## introduction text simplification enables participation, promotes equal opportunities, and removes barriers in the digital society. large language models offer effective means of simplifying texts @xcite . however, simplification evaluation lacks in comparison @xcite @xcite . ideally, text simplification evaluation should follow the principles of readability assessment @xcite and quantify how easy it is for a certain group of readers to understand a given text. this quantification should be independent of an original version of a text. we define this task of quantifying a text's simplicity in this manner as simplicity assessment. current text simplification evaluation measures mostly do not align with this, as they mainly treat text simplification as a relational concept. humanlabeled datasets for constructing text simplification evaluation measures typically contain labels based on the extent to which a simplified text is easier to understand than an original text @xcite @xcite @xcite . therefore, they do not quantify simplicity of a text but rather the degree of simplification between an original and modified version of a text. despite their applicability to single texts, automated readability assessment (ara) approaches are not well suited for text simplification evaluation. ara mostly assigns distinct readability levels @xcite and poorly correlates with simplicity scores assigned by human annotators (e.g., fkgl @xcite ). a dataset with manually determined simplicity scores for single texts would be most desirable to enable the development of a text simplicity assessment approach. however, this is a laborious and demanding task. quantification of the simplicity of a text is complex and subjective for annotators; for instance, a person with medical expertise may perceive the simplicity of a medical text differently than a layperson @xcite . manually assigning a score for a text's simplicity is cognitively demanding because it requires determining what constitutes high and low simplicity. even for advanced large language models, the task of simplicity assessment is non-trivial. this work strives to overcome this issue by introducing a method for facilitating simplicity assessment dataset creation. our contribution, arts, lies in i) the application of pairwise comparison and an elo algorithm for dataset creation, including a rating interface, ii) a dataset containing manually rated simplicity scores for texts with the use case of text simplicity assessment, and iii) an evaluation of the use of large language models for the rating task by comparing the rankings with human annotations. finally, we show that arts datasets can be used to train a model for simplicity assessment. arts is language-independent and can accommodate user characteristics such as domain knowledge, english skill level, and subjectivity. we show that using arts, automatically generated simplicity scores correlate highly with human-annotated data, making arts ideal for generating synthetic training data. the code for our project can be found on github foot_0 , the datasets can be found on zenodo @xcite . for modified texts, the licenses of the original texts apply; everything else is published under an mit license. ## methodology figure 1 gives an overview of our approach. we first select and prepare texts and match them into pairs. these pairs are rated and an elo algorithm is applied to the rated pairs. based on that, the arts simplicity ranking is composed. @xcite describe that the elo rating system was increasingly used to compare large language models (llms) through "a vs. b" paired comparisons. we build on this idea and evaluate 2 ## conclusion in this publication, we present arts, a method for constructing datasets with continuous simplicity scores that can be used to develop text simplicity evaluation approaches. additionally, we provide four different-sized datasets with texts spanning multiple domains and target audiences as well as scores derived from human-and llm-based annotations. using a pairwise comparison approach and an elo ranking system, we facilitate simplicity assessment. our approach is language-independent and is able to accommodate readers' subjective characteristics. we found that using arts, automated llm-based labeling produces similar results to human labels, thus allowing for an effective and cost-efficient way of constructing large synthetic datasets. future work will focus on the construction of larger simplicity datasets. we further plan to use the datasets in developing simplicity quantifying models, that are based on arts. as another line of work we want to extend the application of the methodology to the quantification of other subjective phenomena, e.g., offensiveness, bias, or appeal.