BATS : B enchm A rking Text Simplicity 🦇 (2024)

Abstract: Evaluation of text simplification currently focuses on the difference of a source text to its simplified variant. Datasets for this evaluation base on a specific topic and group of readers for which is simplified. The broad applicability of text simplification and specifics that come with intended target audiences (e.g., children compared to adult non-experts) are disregarded. An explainable assessment of the overall simplicity of text is missing. This work is BenchmArking Text Simplicity (BATS): we provide an explainable method to assess practical and concrete rules from literature describing features of simplicity and complexity of text. Our experiments on 15 datasets for text simplification highlight differences in features that are important in different domains of text and for different intended target audiences.

## introduction text simplification aims to reduce a text's difficulty and adapt it to a reader's abilities @xcite . difficulties in fully grasping information excludes individuals from actively participating in society, such as politics, education, or choosing between healthcare options @xcite . simplifying text can help overcome this obstacle. different groups of people (target audiences, short tas) can profit from text simplification, such as children @xcite or nonexperts @xcite . these different groups have distinct needs for text in order to be considered simple @xcite . the domain (short d) of the text that is simplified influences the factors that constitute good simplification @xcite . for example, when simplifying news articles, reducing the level of detail might be helpful as long as the general message stays the same. in contrast, in other domains (e.g., medical texts), preserving all information is imperative @xcite . historically, text simplification was performed by schooled experts. however, advances in computer technology, such as the introduction of (transformer-based) large language models, have enabled the development of highly effective automated text simplification @xcite that can compete with manual efforts. automation and reduced cost, expertise-and time requirements allow for a much broader application of text simplification @xcite . evaluating the quality of simplified text is a task coinciding with the introduction of automation of text simplification. aside from manual evaluation, text can be automatically evaluated by either benchmarking against reference simplifications or using metrics or models. readability scores such as the flesch reading ease @xcite are often described as unsuitable for quantifying the level of simplicity of a text as the measure does not take into account sufficient aspects of what constitutes simplicity. while manual evaluation is labor intensive, metrics often lack precision and poorly approximate human judgments @xcite . recent automatic evaluation approaches using elaborate language models better correlate with human judgments on texts' simplification level but require a high-quality dataset of labeled simplifications for best training results @xcite . current datasets for evaluating metrics do not indicate if a text is simple. instead, they quantify how much more accessible a text has become by it being simplified compared to its complex form @xcite @xcite @xcite . therefore, metrics correlating to the human labels from these datasets do not indicate the overall simplicity of a text but rather its degree of simplification. all these datasets belong to the same domain and thus potentially ignore specificities found only in certain areas. this work focuses on benchmarking text simplicity (bats). we present a reference-free and explainable method to identify characteristics indicating simplicity and complexity of text. our approach does not directly quantify simplicity but rather provides insights into what properties might make a text simple or complex. these characteristics relate to features of simple text which can be found in the literature. bats is target audienceand domain-independent. using bats, we evaluate three research questions: rq 1 which characteristics from literature reflect the simplicity or complexity of texts? rq 2 which dataset-specific, target audiencespecific, and domain-specific characteristics can be found regarding simplicity? rq 3 can bats be used to quantify simplicity effectively? our code and data are publicly available at github 1 under mit license. ## bats figure 1 gives a simplified overview of our approach. we implement rules from literature via snorkel @xcite to determine if these rules appear in a dataset consisting of texts. the rules express heuristics that linguistic experts consider to be salient characteristics of the measure of text simplicity. after applying these rules, binary vectors are obtained. these vectors combined with simplicity scores of texts can then be used to prune the rules to only keep those highly indicative of simplicity and complexity (bats model). we can construct bats vectors for any text with these pruned rules. these vectors indicate which characteristics of simplicity or complexity are satisfied. ## evaluation we evaluate the overall research questions rq 1-3 , described in the introduction. ## conclusion we presented bats, a method to evaluate straightforward and concrete rules that can be used in quantifying the simplicity or complexity of text. our approach is inherently explainable. the potential of bats vectors becomes apparent in their comparison to the state of the art in text representation (openai textual embeddings). through our evaluations we showed the possibility for nuanced evaluation of text simplification for different target audiences or domains while shedding light on their different needs in properties of simplified text. future work should focus on the incorporation of more features into bats and in-depth evaluations of simplification approaches and more datasets. as a direct consequence of our findings, simplification of text should be evaluated in a target audienceand domain-specific manner. here, datasets which contain simplifications that differ in more than one word only from the source texts would be desirable.