Toward Human-Centered Readability Evaluation (2025)

Abstract: Text simplification is essential for making public health information accessible to diverse populations, including those with limited health literacy. However, commonly used evaluation metrics in Natural Language Processing (NLP)—such as BLEU, FKGL, and SARI—mainly capture surface-level features and fail to account for human-centered qualities like clarity, trustworthiness, tone, cultural relevance, and actionability. This limitation is particularly critical in high-stakes health contexts, where communication must be not only simple but also usable, respectful, and trustworthy. To address this gap, we propose the Human-Centered Readability Score (HCRS), a five-dimensional evaluation framework grounded in Human-Computer Interaction (HCI) and health communication research. HCRS integrates automatic measures with structured human feedback to capture the relational and contextual aspects of readability. We outline the framework, discuss its integration into participatory evaluation workflows, and present a protocol for empirical validation. This work aims to advance the evaluation of health text simplification beyond surface metrics, enabling NLP systems that align more closely with diverse users’ needs, expectations, and lived experiences.

## introduction text simplification -the process of replacing complex terms with simpler alternatives, removing extraneous details, or breaking down lengthy sentences while preserving essential meaning @xcite -is especially critical in health communication. yet contemporary evaluation metrics often focus on the outcomes of individual simplification operations and fail to fully capture human judgments of overall simplicity. this gap is particularly concerning in the medical domain, where effective information delivery is essential for public well-being, especially for individuals with limited health literacy @xcite . health-related materials such as medication instructions, risk explanations, and care recommendations are often overly complex and cognitively demanding. recent advances in natural language processing (nlp) have enabled automated systems to generate easier-to-read versions of such texts, aiming to improve accessibility and comprehension @xcite @xcite . these systems are typically evaluated with automatic metrics such as sari @xcite , fkgl @xcite , or bleu @xcite , which focus on surface-level features (e.g., lexical simplicity, sentence length, or n-gram overlap). while useful for benchmarking, such metrics fail to capture whether simplified health information is genuinely clear, actionable, and trustworthy to real users-qualities that are critical in high-stakes domains like public health. this challenge mirrors broader efforts in ai to align system performance with human values, where evaluation must go beyond automatic scores to incorporate human judgment. reinforcement learning from human feedback (rlhf) @xcite enables models to learn directly from user preferences and expert evaluations, serving as a gold standard for aligning system behavior with human values. however, collecting large-scale, high-quality human feedback can be costly and impractical. to address this limitation, recent approaches such as reinforcement learning with ai feedback (rlaif) @xcite offer scalable alternatives to costly human evaluations, but risk misalignment with human values and needs in sensitive contexts. to address the gap between metric-driven evaluation and real-world user experience, we investigate: q1 which existing readability metrics (lexi-cal, syntactic, semantic, or neural) align most strongly with human-centered dimensions-clarity, trustworthiness, tone appropriateness, cultural relevance, and actionability-in simplified health texts? q2 to what extent can a composite human-centered readability score (hcrs), integrating automatic features with human feedback, achieve stronger alignment with user evaluations than the best standalone metric? q3 how can human-computer interaction (hci) techniques, such as interactive feedback collection and participatory design, be embedded into the evaluation pipeline to make metric optimization more responsive for real-world needs? readability in health communication is inherently multidimensional: a linguistically simple message can still be unclear, culturally inappropriate, or untrustworthy. addressing these dimensions requires going beyond surface-level metrics. our contributions are threefold: by shifting the evaluation lens from system-centric to user-centric, we aim to advance health-focused nlp systems that are not only technically effective, but also socially and culturally responsive to the needs of diverse real-world users-an imperative in public health communication. ## limitations of current metrics despite widespread adoption, automatic evaluation metrics for text simplification-such as sari, fkgl, and bleu-face critical shortcomings in public health communication. intrinsic metric limitations. these metrics are designed to capture primarily surface-level transformations-such as lexical substitution, sentence compression, and n-gram overlap with reference texts-and are effective for system-level benchmarking. however, they provide no direct evidence regarding whether a simplified text is, in fact, clear, trustworthy, or pragmatically usable for its intended audience. ## human-centered readability score (hcrs) we argue that readability in health communication should be redefined not as a property of the text alone, but as a dynamic, relational, and context-sensitive experience shaped by human perception. to address the limitations of current surface-level metrics, we introduce a five-dimensional framework-clarity, trustworthiness, tone appropriateness, cultural relevance, and actionability-grounded in research from health communication, hci, and participatory design. each dimension captures a distinct yet interdependent aspect of how users interpret and engage with simplified content. figure 1 ## empirical evaluation & discussion this work defines a comprehensive experimental protocol to validate the human-centered readabil-ity score (hcrs) across diverse user populations and health literacy levels. the protocol integrates automatic and human-centered measures to move evaluation beyond current surface-oriented benchmarks. interactive & participatory evaluation. most nlp evaluation workflows are system-facing. to operationalize hcrs dimensions such as tone, trust, and actionability, user-facing interfaces should collect real-time feedback. short prompts-e.g., "was this sentence clear to you?", "did you feel respected by the way it was phrased?"-can be embedded in studies or deployment. inclusive design, informed by participatory methods, engages diverse stakeholders, co-creates evaluation criteria, and adapts interfaces to cultural and literacy contexts. accessibility features (visual icons, read-aloud options) can broaden participation, and lightweight web tools or crowdsourcing can scale data collection. in practice, participatory methods can be embedded in evaluation pipelines through lightweight annotation interfaces where end-users quickly rate sentences on the five hcrs dimensions. feedback can be collected in short micro-surveys (5-10 minutes), aggregated, and then reviewed with stakeholders (e.g., patients, clinicians, or domain experts) during participatory workshops. this creates an iterative loop: (i) immediate micro-ratings during evaluation, (ii) stakeholder workshops to review and refine criteria, and (iii) iterative updates where refinements directly inform hcrs weighting. for instance, in a vaccine information use-case, patients could rate clarity and trustworthiness via inline sliders, while health professionals review aggregated outputs to recalibrate actionability guidelines. this illustrates how participatory methods can be concretely integrated rather than remaining abstract, serving as a proof-of-concept protocol that demonstrates how hcrs could be operationalized through lightweight user studies even before large-scale deployment. structured, multidimensional feedback. binary ratings rarely explain why a sentence fails. feedback channels mapped to hcrs dimensions-e.g., "too technical", "missing information", "poorly structured"-enable more interpretable model training and closer alignment with user perceptions. model integration. human feedback should inform training, not just evaluation. strategies include multi-objective learning, auxiliary classifiers for hcrs dimensions, and prompt engineering for targeted qualities. care is needed with rlaif, as over-reliance on synthetic evaluators risks misalignment with human values in sensitive domains. future directions. future work will extend validation of hcrs beyond health communication, test participatory feedback pipelines at scale, and explore adaptive dimension weighting. ## conclusion this work introduced the human-centered readability score (hcrs), a five-dimensional framework for evaluating simplified health texts beyond surface features. by addressing clarity, trustworthiness, tone, cultural relevance, and actionability, hcrs fills critical gaps in current metrics and aligns evaluation with real-world user needs. we examined how existing metrics relate to these dimensions (q1), proposed a composite score integrating automatic and human feedback (q2), and outlined mechanisms to embed participatory, interactive evaluation into nlp workflows (q3). our empirical protocol sets the stage for validating hcrs across diverse populations and domains, enabling models that are both technically robust and socially responsive. while designed for public health communication, the approach extends to any domain where clarity, trust, and usability are paramount. importantly, as discussed in section 5, hcrs can also serve as a complementary evaluation layer and a calibration signal for rlaif-based training pipelines in sensitive domains-helping to align scalable rl methods with nuanced, humancentered objectives.