Readability Reconsidered A Cross-Dataset Analysis of Reference-Free Metrics (2025)

Abstract: Automatic readability assessment plays a key role in ensuring effective communication between humans and language models. Despite significant progress the field is hindered by inconsistent definitions of readability and measurements that rely on surface-level text properties. In this work we investigate the factors shaping human perceptions of readability through the analysis of 1.2k judgments finding that beyond surface-level cues information content and topic strongly shape text comprehensibility. Furthermore we evaluate 15 popular readability metrics across 5 datasets contrasting them with 5 more nuanced model-based metrics. Our results show that four model-based metrics consistently place among the top 4 in rank correlations with human judgments while the best performing traditional metric achieves an average rank of 7.8. These findings highlight a mismatch between current readability metrics and human perceptions pointing to model-based approaches as a more promising direction.

## introduction readability assessment can be used to determine the level of comprehension of a piece of text @xcite . in domains such as science communication @xcite , health @xcite , law @xcite , and education @xcite , readability assessment plays a key role in making information accessible to individuals regardless of their background or cognitive needs @xcite . it is important for highlyspecialized fields characterized by dense jargon and complex language @xcite , as well as for applications engaging with users of varied familiarity with the domain @xcite . one challenge in advancing automatic readability assessment is that readability is an overloaded term, measured in different ways by prior work. some studies treat readability as text difficulty, using surface-level properties such as word length, word frequency, and various word type counts @xcite @xcite . others broaden the definition of readability to consider syntactic and discourse-level organization, including cohesion and coherence properties @xcite @xcite @xcite . a third line of work views readability as a combination of text characteristics and information content @xcite . taken together, the diversity of interpretations highlight the difficulty of pinning down readability, and have led to the continued use of proxy metrics that may not fit the task, domain, or are misaligned with human comprehension judgments @xcite @xcite . readability metrics. while human judgments remain the gold standard for readability evaluation, their collection is often time-consuming and expensive @xcite . automated metrics have emerged as a cheaper and quicker alternative. examples include metrics relying on basic linguistic features, including sentences, words, and syllables counts, average reading time @xcite , language model perplexity @xcite , and fraction of functional @xcite or uncommon words @xcite . surfaceform features have been further combined to form readability tests, such as the automatic reading index @xcite , dale-chall readability score @xcite , flesch-kincaid reading ease @xcite , and linsear write formula @xcite . despite critiques of brittleness @xcite and limited domain suitability @xcite , these formulas continue to be used. recently, both fine-tuning @xcite and llm-as-a-judge approaches @xcite have been proposed to capture more abstract and nuanced aspects of readability. however, since these methods rely on implicitly learned representations, they are regarded as less interpretable than those grounded in surface-level textual features. ## how do humans perceive readability? given the divergent definitions of readability and continued reliance on surface-form metrics, we take a human-centric perspective, asking: what guides human perceptions of readability? to address this question, we analyze a subset of the eli-why (gpt-4) @xcite dataset, designed to study whether llms can generate explanations tailored to various readability levels. the dataset comprises gpt-4-generated explanations for 299 "why" questions, each annotated by humans into three readability levels-elementary , high school , and graduate -along with accompanying rationales justifying their judgments. each question-explanation pair was independently rated by three annotators, and final labels were determined via majority vote. for additional de-tails, see the original paper. ## re-evaluating readability metrics motivated by the gap between surface-form textual cues and human perceptions of readability, we investigate how well existing readability metrics correlate with human judgments across five diverse datasets (see statistics in eli-why (gpt-4) @xcite includes 897 gpt-4-generated explanations tailored to three readability levels-elementary , high school , and graduate -each annotated with human-assigned labels and rationales. likewise, eli-why (human) is a smaller dataset with 123 answers that were manually curated. scienceqa @xcite ) is a multi-modal science reasoning dataset consisting of 21k multiple choice questions sourced from k-12 curriculum, covering various subjects (e.g., natural science, language science, and social science). each example is associated with a reference solution (or explanation) and reference knowledge (or lecture), both of which are written at the readability level of the intended student audience. we randomly sample 200 text-only examples per grade for our analysis. ## conclusion this work tackles the inconsistency of readability definitions (and metrics) in the literature by showing that human perceptions of readability go beyond lexical and syntactic features, also considering topic and information content. furthermore, we benchmark 20+ reference-less metrics-including llm-as-a-judge and fine-tuned models-across five datasets. our results show that model-based metrics correlate more strongly with human judgments than popular readability metrics, suggesting they capture more nuanced features. together, these findings call for clearer definitions of readability and more rigorous validation of metrics, paving the way for assessments that better reflect how humans understand text.