Evaluating Health Question Answering Under Readability-Controlled Style Perturbations (2025)

Abstract: Patients often ask semantically similar medical questions in linguistically diverse ways that vary in readability tone and background knowledge. A robust question answering QA system should both provide semantically consistent answers across stylistic differences and adapt its response style to match the users input however existing QA evaluations rarely test this capability creating critical gaps in QA evaluation that undermine accessibility and health literacy. We introduce SPQA an evaluation framework and benchmark that applies controlled stylistic perturbations to consumer health questions while preserving semantic intent then measures how model answers change across correctness completeness coherence fluency and linguistic adaptability using a human-validated LLM-based judge. The style axes include reading level formality and patient background knowledge all perturbations are grounded in human annotations to ensure fidelity and alignment with human judgments. Our contributions include a readability-aware evaluation methodology a style-diverse benchmark with human-grounded perturbations and an automated evaluation pipeline validated against expert judgments. Evaluation results across multiple health QA models indicate that stylistic perturbations lead to measurable performance degradation even when semantic intent is preserved during perturbation. The largest performance drops occur in answer correctness and completeness while models also show limited ability to adapt their style to match the input. These findings underscore the risk of inequitable information delivery and highlight the need for accessibility-aware QA evaluation.

## introduction large language models (llms) have rapidly become central to consumer-facing questionanswering (qa) systems, offering users quick and style transfer (elementary school) "my mom was in a car accident two decades ago which resulted in her left foot being severed and re-attached, her left leg is shorter and pelvis tilted resulting in back pain that has increased over the years. ... her mental health is so bad because she is living in constant pain, she is basically being imprisoned and tortured in her own body with no escape. where do we go from here? we are based in the u. interactive access to information across a wide range of domains @xcite @xcite . with this growing adoption, these systems are increasingly relied upon in critical areas such as healthcare, where users expect accurate and actionable guidance. however, as the user base becomes more diverse, linguistic variability in queries arising from differences in literacy, tone, and background knowledge presents a substantial challenge @xcite . such diversity often affects the accessibility and reliability of responses, creating disparities in the quality of information retrieved. despite its practical significance, this issue remains largely understudied, and existing evaluations rarely assess whether qa models can adapt to stylistic and readability differences, even when the underlying information need is unchanged. while prior research has documented that demographic and stylistic factors influence model outputs @xcite , most evaluations have focused on narrow dimensions such as adversarial perturbations or typographical noise @xcite @xcite . these approaches do not address real-world variability in question phrasing that affects user comprehension and system accessibility. current qa assessments typically emphasize correctness and completeness but overlook whether responses maintain quality and align with the style of diverse user queries. we term this as linguistic adaptability, and it is essential for ensuring accessible information delivery and mitigating inequities in high-stakes domains like health communication. to address this gap, we introduce style perturbed question answering (spqa), an evaluation framework and benchmark (figure 1 ). spqa systematically perturbs user questions along predefined stylistic axes -a) reading level, b) formality, and c) domain knowledge, while preserving semantic intent. for each perturbed question, responses are evaluated against four comprehensive criteria: correctness, completeness, coherence and fluency, and linguistic adaptability. all perturbations are grounded in human annotations to ensure realism, and evaluations are conducted using a human-validated llm-as-judge for scalability and reliability. spqa serves as a general framework for evaluating qa robustness under stylistic variation and provides a structured protocol for stress-testing qa systems under stylistic diversity, operationalizing accessibility as an evaluation dimension. in this work, we apply spqa to consumer health question answering, where the selected stylistic axes capture key factors influencing comprehension and accessibility. our key contributions are summarized below: 1. readability-aware evaluation framework: we introduce spqa 1 , a framework for evaluating qa performance under readability-and stylecontrolled perturbations, addressing an underexplored dimension of accessibility in qa. ## conclusion and future work this study introduces spqa, a framework and benchmark for evaluating linguistic robustness in question-answering systems under controlled stylistic perturbations. spqa systematically assesses how stylistic variations in questions impact qa model performance across multiple evaluation dimensions. unlike prior evaluations focused primarily on accuracy under standard inputs, spqa captures a critical but overlooked dimension: the ability of models to deliver consistent, accessible answers across diverse linguistic contexts. while broadly applicable, we applied spqa to consumer health qa, revealing vulnerabilities in current llms when processing stylistic variations reflecting real-world linguistic diversity. our experiments across ten state-of-the-art llms demonstrate that stylistic changes, even when semantic meaning is preserved, result in measurable performance degradation in answer generation. the largest declines occur in correctness and completeness, and models frequently fail to align response style with question style. these findings reveal systematic risks to health information accessibility, affecting both users with limited literacy and expert users with specialized language needs, thereby reinforcing the urgency of equity-aware qa evaluations. spqa provides a pathway toward addressing this gap by offering a systematic, human-grounded, and readability-aware evaluation protocol validated through llm-as-judge. future research should extend spqa to additional domains, including multimodal inputs, spoken interactions, and lowresource languages. performance improvements may be achieved through adaptive prompting, stylediverse data augmentation, and patient-centered metrics. this work underscores the need for robust evaluation frameworks to ensure equitable access to reliable information for all. future extensions should also incorporate readability-based user modeling to guide real-world deployment strategies.