Two-Pronged Human Evaluation of C hat GPT Self-Correction in Radiology Report Simplification (2024)

Abstract: Radiology reports are highly technical documents aimed primarily at doctor-doctor communication. There has been an increasing interest in sharing those reports with patients, necessitating providing them patient-friendly simplifications of the original reports. This study explores the suitability of large language models in automatically generating those simplifications. We examine the usefulness of chain-of-thought and self-correction prompting mechanisms in this domain. We also propose a new evaluation protocol that employs radiologists and laypeople, where radiologists verify the factual correctness of simplifications, and laypeople assess simplicity and comprehension. Our experimental results demonstrate the effectiveness of self-correction prompting in producing high-quality simplifications. Our findings illuminate the preferences of radiologists and laypeople regarding text simplification, informing future research on this topic.

## introduction an increasing number of healthcare providers are interested in sharing health records with patients. that is a positive development because research has shown that sharing medical records with patients might improve patient-doctor communication (ross and @xcite , increase patient involvement in care @xcite , and improve outcomes @xcite . however, the health literacy @xcite of most patients is often not sufficient to enable an understanding of their health records @xcite . the health literacy gap is especially severe for some types of medical reports, such as radiology reports, whose primary purpose is doctordoctor communication. as a result, radiology reports use particularly complex medical jargon and highly specialized descriptions @xcite and present a particular challenge for patients @xcite . for instance, a recent study @xcite found that the mean readability grade level of mri reports was above the 12th-grade reading level. without adequate counseling with an experienced clinician, the severity of the radiology findings may be misinterpreted by the patients. it could lead to unnecessary stress, improper follow-up, and even to increased patient mortality @xcite . there is an increasing interest in patient-friendly reporting. one way to accomplish this is to ask radiologists to supplement their expert-language reports with patient-friendly summaries. one downside of this approach is a negative impact on radiologists' cognitive load and productivity. another downside is the curse of knowledge @xcite , making it challenging for radiologists to simplify their reports. an enticing alternative that has garnered much recent interest @xcite is to generate patient-friendly simplifications with large language models (llms) and ask radiologists to check the generated simplifications before releasing them. there are several open challenges to the generation of patient-friendly radiology reports. the first is that it needs to be clarified what constitutes a good simplification. the existing research has varying views of the trade-offs between factuality, completeness, simplicity, and brevity @xcite . a proper combination of these measures may depend on individual user preferences. as a result, it would be very challenging to create a widely acceptable parallel corpus for radiology report simplification. moreover, very different simplifications could be evaluated as equally successful (longer and more detailed versus shorter with only critical information). thus, even if the parallel corpus was created and used to train and test an llm, automatic evaluation using measures that rely on sequence similarity @xcite @xcite might be misleading. instead, until there is more clarity about what constitutes a reasonable radiology simplification, we think humans should perform the evaluation. there is no broadly accepted protocol for human evaluation of simplified expert text @xcite @xcite , including what questions to ask and who should answer them. in this paper, we propose a novel evaluation protocol following two ideas. first, we observe that laypeople should not be asked factuality and completeness questions due to the lack of expert knowledge and that radiologists should not be asked about simplicity due to the curse of knowledge bias. thus, our protocol employs laypeople and radiologists with slightly different questions. second, we observe that a good simplification is the one that increases understanding compared to the original text, but also that there can be a dangerous mismatch between perceived and actual understanding. thus, laypeople are asked both about their perception and their actual increase in understanding when an expert text is supplemented by its simplification. another contribution of this paper is in evaluating the capabilities of the state-of-the-art llms without constructing a large parallel text corpus. arguably, the best publicly available llm at the moment is chatgpt (openai, 2023), and recent papers @xcite indicate that both its 3.5 and 4 versions can provide high-quality radiology report simplifications only through prompting. in this paper, we provide an in-depth evaluation of chain-of-thought (cot) prompting @xcite and self-correction @xcite . in the cot approach, llms are prompted to justify an answer before providing the answer. in the self-correction approach, llms are prompted to critique their original response and asked to consider the critique to give an improved response. both methods have been shown to work well in several applications @xcite . to our knowledge, they have yet to be evaluated on radiology report simplification. we designed experiments to answer the following research questions: (q 1 ) is the proposed human evaluation protocol insightful? (q 2 ) are cot and self-correction helpful in the simplification of radiology reports? (q 3 ) what is the relationship between perceived and actual understanding of radiology reports? (q 4 ) what kinds of simplifications are preferred by experts and laypeople? the answers should be informative for future research towards high-quality simplifications of expert texts. ## evaluation protocol as described in the previous section, prior text simplification research used human evaluation, but did not clarify the roles of experts and laypeople in evaluation. in the following, we will propose an evaluation protocol that defines those roles. ## conclusion this paper introduces a two-pronged approach for human evaluation of radiology report simplifications. it proposes a specialized variant of the selfcorrection mechanism that allows chatgpt to generate high-quality simplifications. the analysis of results derived from human evaluation show that our proposed evaluation protocol successfully reveals diverse facets of simplification quality.