Style-Aware Radiology Report Generation with R ad G raph and Few-Shot Prompting (2023)

Abstract: Automatically generated reports from medical images promise to improve the workflow of radiologists. Existing methods consider an image-to-report modeling task by directly generating a fully-fledged report from an image. However, this conflates the content of the report (e.g., findings and their attributes) with its style (e.g., format and choice of words), which can lead to clinically inaccurate reports. To address this, we propose a two-step approach for radiology report generation. First, we extract the content from an image; then, we verbalize the extracted content into a report that matches the style of a specific radiologist. For this, we leverage RadGraph—a graph representation of reports—together with large language models (LLMs). In our quantitative evaluations, we find that our approach leads to beneficial performance. Our human evaluation with clinical raters highlights that the AI-generated reports are indistinguishably tailored to the style of individual radiologist despite leveraging only a few examples as context.

## introduction generating radiology reports from medical images is a crucial task in the field of medical imaging. for human interpreters to write such reports is not only time-consuming and labor-intensive but also requires a high level of expertise @xcite . furthermore, these reports are often subject to inter-observer variability, potentially compromising the consistency and accuracy of the reports' findings. as a result, there is a growing interest in methods for automated radiology report generation, which can alleviate these issues and improve overall efficiency of the diagnostic process. recent advances in large language models (llms) have shown great potential for generating high-quality text, with the ability to customize outputs based on user-specified instructions @xcite . these models have been utilized to rephrase existing text, paving the way for new approaches to automated report generation in radiology. however, despite the promise of llms, their application to this task is not without challenges. one principal concern is their tendency to 'hallucinate', i.e., to generate false information (even if plausible sounding), which can be particularly problematic in high-stakes settings such as when generating reports from medical images. previous attempts at automated report generation have largely focused on approaches that aim to produce fully-fledged reports directly from medical images (i.e., a image-to-report modelling task) @xcite @xcite . however, this conflates the content of the report (i.e., the radiology entities and attributes that are described) with its style (i.e., the complement of radiology entities and attributes, or everything needed in terms of language, grammar, and structure to formulate a fully-fledged report, on top of the content representation), which limits the flexibility and applicability of such models. this problem is also reflected in the employed evaluation metrics of existing approaches. they frequently optimize for traditional natural language processing (nlp) metrics, such as bleu, rouge, or meteor, which measure the similarity of the generated report to a reference report based on lexical overlap. these metrics, while generally useful, may not correlate with the clinical correctness or usefulness of the report @xcite . previous work has demonstrated that optimizing for clinical metrics that measure the content's relevance, such as the radgraph scores @xcite , is crucial in generating reports that accurately represent the image findings. the radgraph score gauges similarity in extracted radiology entities and relations with a ground truth annotation. this emphasizes the report's clinical content as opposed to style or lexical overlap, and makes it an amenable measure of the report's usefulness and completeness in clinical practice. here, we propose to generate radiology reports as a two-step procedure for disentangling report content and style. in the first step, a dedicated model is trained to extract pure content from the image. specifically, it generates a structured representation (called radgraph) of the entities and attributes that are present in the image. in the second step, a frozen llm generates a stylized report from this structured representation. given a few report examples as context, the llm can on-the-fly adapt the report style closely to the style of a target radiologist or hospital template for whom the report should be drafted. this model stylization could offer several advantages for radiology workflows: flexibility, in generating reports targeted to their readership, such as ones with less jargon that are more accessible to patients; consistency, in ensuring clear communication between a radiologist and the referring physician, who may be accustomed to a particular style of reporting; and emphasis on preferred information, in highlighting findings most relevant to a specialist's scope of practice (e.g., follow-up on a patch of pneumonia, or the correct location of pacing lead in the heart). in the first step of our approach, we alter the supervision signal of an image-text model to a serialization of the clinical entities (as captured in radgraph) and their attributes, rather than the full report text. this step ensures the content extraction model focuses only on generating the report's clinical content, measured by radgraph score, rather than optimizing for traditional nlp metrics such as bleu that may not correlate with the report's clinical relevance. by generating the clinical content first, we prioritize the report's clinical usefulness over its stylistic quality, which can be improved and even personalized in the second step. for the second step, we leverage gpt-3.5 (a generative llm developed by openai) to transform the predicted serialization of clinical entities and attributes from the image into a styled radiology report @xcite . these models have shown great promise in generating high-quality text @xcite , including summaries and paraphrases of existing text @xcite , which we can use to inject a hospital-specific style, as well as to enhance readability and understandability. by separating the content generation and style injection steps, we can ensure the model optimizes for the relevant criteria for each step, resulting in a high-quality report. our approach addresses the limitations of end-toend image-to-report generation, as the llm does not need to suggest facts about the image. instead, it can focus on rephrasing the structured set of entities and attributes that were first derived from the image and infuse the desired style into the report. at prediction time, the llm in-context learns a serialization-to-report mapping from a few examples of a target report writing style. our method not only offers a novel solution to the challenges posed by previous approaches but also enhances the customization and adaptability of radiology reports generated by llms. this paper makes the following contributions: ## discussion we presented a novel approach for radiology report generation that disentangles the report's content from its style. our experiments showed that our method offers several advantages over the prevailing paradigm of direct image-to-report modeling. first, training a content extraction model to predict a serialized radgraph represention from the input image helps focusing the model on the clinically relevant content which is reflected in improved performance metrics ( ## limitations a limitation of our approach is that it relies on the accuracy and effectiveness of the initially extracted radgraph representation, as this serves as a key supervision signal in our model. notably, radgraphs are extracted at inference time using an automated model rather than manual expert labelling. the model achieves high performance in entity and relation extraction @xcite but is susceptible to error, particularly with report inputs that contain rarer medical entities or ambiguous observations. furthermore, due to our employed llm being served by a third party (openai), reproducing our results comes at the financial costs of using azure openai's service. another consequence of relying on this service is that we cannot guarantee the deterministically exact reproduction of our findings, as the served llm models may change and potentially degrade over time-for instance, if models are replaced by distilled versions thereof. ## ethics considerations a principal ethical consideration is the deidentified, credentialed medical data we worked with. in particular, responsible usage policy of the mimic-cxr dataset @xcite prohibits sharing access to third parties. this disqualifies the use of chatgpt or large language model apis for prompting models to generate radiology reports from our content representations. however, cloud-based services are allowed, including the azure openai platform, which we use for deploying and prompting the gpt-3.5-turbo model. a stipulation is the monetary service costs, which are counted at a rate per one thousand tokens. these can pose financial barriers to equitable access to high-end language models (which are typically more expensive), as well as usage at scale. furthermore, as discussed in the introduction section, the use of large language models is accompanied by their risks of "hallucination" and generating false or misleading content. these risks can be amplified in the critical setting of medical report generation. to mitigate them, we prompt the llm not to synthesize medical content, but rather to rephrase existing content into readable, stylized prose. we assist the model through providing content serialization to report pairs as in-context examples. this is our style injection step, which we intentfully separate from the content generation step to reduce the opportunity for llm hallucination when generating the full report prediction.