M ed R ead M e: A Systematic Study for Fine-grained Sentence Readability in Medical Domain (2024)

Abstract: Medical texts are notoriously challenging to read. Properly measuring their readability is the first step towards making them more accessible. Here, we present the first systematic study on fine-grained readability measurements in the medical domain, at both sentence-level and span-level. We first introduce a new dataset MedReadMe, which consists of manually annotated readability ratings and fine-grained complex span annotation for 4,520 sentences, featuring two novel “Google-Easy” and “Google-Hard” categories. It supports our quantitative analysis, which covers 650 linguistic features and additional complex span features, to answer “why medical sentences are so hard.” Enabled by our high-quality annotation, we benchmark several state-of-the-art sentence-level readability metrics, including unsupervised, supervised, and prompting-based methods using recently developed large language models (LLMs). Informed by our fine-grained complex span annotation, we find that adding a single feature, capturing the number of jargon spans, into existing readability formulas can significantly improve their correlation with human judgments, and also make them more stable. We will publicly release data and code.

## introduction if you can't measure it, you can't improve it. an is an unnatural opening between the and . oro-antral communication oral cavity maxillary sinus readability: 5-readability: 5 readability: 4+ readability: 4-, 3+, 3+ the floor of the main sinus near the nose lies directly above the roots of the teeth at the back of the mouth. sometimes following infection or dental treatment , this structure becomes damaged and openings or channels between the mouth and the sinus are formed. these are known as ( ). kmt5c-mediated h4k20 methylation thermo--genic program adipocytes they indicate the activation of might be a potential strategy for . figure 1 : an illustration of our dataset, with sentence readability ratings and fine-grained complex span annotation on 4,520 sentences, including "google-hard" and "google-easy", abbreviations, and general complex terms, etc. we also analyze how medical jargon are being handled during simplification. e.g., a google-hard "oro-antral communication" is copied and elaborated. some jargon are ignored for clarity. to address this gap in research, we present a systematic study for medical text readability in this paper, which includes a manually annotated readability dataset ( §2), a data-driven analysis to answer "why medical sentences are so hard", covering 650 linguistic features and additional medical jargon features ( §3), a comprehensive benchmark of stateof-the-art readability metrics ( §4.1), a simple yet effective method to improve lm-based readability metrics by training on our dataset ( §4.2), and an automatic model that can identify complex words and jargon with fine-grained categories ( §5). our medreadme dataset consists of 4,520 sentences with both sentence-level readability ratings and fine-grained complex span-level annotations (figure 1 ). it covers complex-simple parallel article pairs from 15 diverse data resources that range from encyclopedias to plain-language summaries to biomedical research publications (figure 2 ). the readability ratings are annotated using a rank-and-rate interface @xcite based on the cefr scale @xcite , which is shown to be more reliable than other methods @xcite . we also ask lay annotators to highlight any words/phrases that they find hard to understand and categorize the reason using a 7-class taxonomy. considering that "the majority of people seek health information online began at a search engine", foot_0 we introduce two categories of "google-easy" and "google-hard" to reflect whether jargon is understandable after a quick google search, providing a fresh perspective beyond binary or 5-point likert scales. our new dataset addresses three limitations in prior work: (1) existing work with sentence-level ratings mainly covers data from general domains, such as wikipedia @xcite , news @xcite , and textbooks for esl learners @xcite , which are very different from specialized fields, such as medicine @xcite . (2) prior work separates the research on sentence readability and complex jargon terms, hence missing the possible correlations between them @xcite . (3) previous research on sentence readability uses document-level ratings as an approximation, which is shown to be inaccurate @xcite . our analysis reveals that compared to various linguistic features, complex spans, especially medical jargon from certain domains, more significantly elevate the difficulty of sentences ( §3.1). we also scrutinize the quality of 15 widely used medical text simplification resources ( §3.4), and find that there are non-negligible variances in readability among them, as shown by the differences in the height of the box plots in figure 2 . while evaluating various sentence readability metrics, we find that unsupervised methods based on lexical features perform poorly in the medical domain. prompting large language models such as gpt-4 @xcite with 5-shot achieves strong performance, yet is outperformed by fine-tuned models in a much smaller size. inspired by our analysis, we add a single feature that captures the "number of jargon" in a sentence into existing readability formulas, and find it can significantly improve their performance and also make them more stable. ## constructing medreadme corpus this section presents the detailed procedure for constructing the medical readability measurement (medreadme) corpus, which consists of 4,520 sentences in 180 complex-simple article pairs randomly sampled from 15 data sources ( §2.1). category definition example tok. len. % medical jargon 2.2±1.5 68.6% google-easy medical terms that can be easily understood after a quick search. schistosoma mansoni is a parasitic infection common in the tropics and sub-tropics. 2.0±1.2 56.9% google-hard medical terms that require extensive research before a layperson can possibly understand them. . . . retains limited dna-processing activity, albeit via a distributive binding mechanism. 3.2±2.5 7.5% name entity brand or organization name, excluding general medical terms such as drugs and equipments. while vaccination with biontech and moderna mostly causes only mild and typical . . . 2.7±2.2 4.1% ## key findings enabled by our medreadme corpus, we first analyze the sentence readability measurements for medical texts ( §3.1 and §3.4), then dive into medical jargon of different complexities ( §3.2 and §3.3). ## medical readability prediction in this section, we present a comprehensive evaluation of state-of-the-art readability metrics for medical texts ( §4.1), and design a simple yet effective method to further improve them ( §4.2). ## fine-grained complex span identification based on our analysis in §4.2, identifying complex spans in a sentence can help the judgment of its readability. it can also improve the performance of downstream text simplification system @xcite . we formulate this task as a ner-style sequential labeling problem @xcite , and utilize our annotated dataset to train and evaluate several models. ## conclusion in this work, we present a systematic study for sentence readability in the medical domain, featuring a new annotated dataset and a data-driven study to answer "why medical sentences are so hard.". in the analysis, we quantitatively measure the impact of several key factors that contribute to the complexity of medical texts, such as the use of jargon, text length, and complex syntactic structures. future work could extend to the medical notes from clinical settings to better understand real-time communication challenges in healthcare. additionally, leveraging our dataset that categorizes complex spans by difficulty and type, further research could develop personalized simplification tools to adapt content to the target audience, thereby improving patients' understanding of medical information.