ORGAN : Observation-Guided Radiology Report Generation via Tree Reasoning (2023)

Abstract: This paper explores the task of radiology report generation, which aims at generating free-text descriptions for a set of radiographs. One significant challenge of this task is how to correctly maintain the consistency between the images and the lengthy report. Previous research explored solving this issue through planning-based methods, which generate reports only based on high-level plans. However, these plans usually only contain the major observations from the radiographs (e.g., lung opacity), lacking much necessary information, such as the observation characteristics and preliminary clinical diagnoses. To address this problem, the system should also take the image information into account together with the textual plan and perform stronger reasoning during the generation process. In this paper, we propose an Observation-guided radiology Report Generation framework (ORGan). It first produces an observation plan and then feeds both the plan and radiographs for report generation, where an observation graph and a tree reasoning mechanism are adopted to precisely enrich the plan information by capturing the multi-formats of each observation. Experimental results demonstrate that our framework outperforms previous state-of-the-art methods regarding text quality and clinical efficacy.

## introduction radiology reports, which contain the textual description for a set of radiographs, are critical in the process of medical diagnosis and treatment. nevertheless, the interpretation of radiographs is very time-consuming, even for experienced radiologists (5-10 minutes per image). due to its large potential to alleviate the strain on the healthcare workforce, automated radiology report generation @xcite has attracted increasing research attention. one significant challenge of this task is how to correctly maintain the consistency between the image and the lengthy textual report. many previous works proposed to solve this through planningbased generation by first concluding the major observations identified in the radiographs before the word-level realization @xcite @xcite . despite their progress, these methods still struggle to maintain the cross-modal consistency between radiographs and reports. a significant problem within these methods is that, in the stage of word-level generation, the semantic information of observations and radiographs is not fully utilized. they either generate the report only based on the high-level textual plan (i.e., major observations) or ignore the status of an observation (i.e., positive, negative, and uncertain), which is far from adequate. the observations contained in the highlevel plan are extremely concise (e.g., lung opacity), while the final report needs to include more detailed information, such as the characteristics of the observation (e.g., a subtle but new lung opacity) and requires preliminary diagnosis inference based on the observation (e.g., lung infection must be suspected). in order to identify those detailed descriptions and clinical inferences about the observations, we need to further consider the image information together with the textual plan, and stronger reasoning must be adopted during the generation process. in this paper, we propose organ, an observation-guided radiology report generation framework. our framework mainly involves two stages, i.e., the observation planning and the report generation stages, as depicted in figure 1 . in the first stage, our framework produces the observation plan based on the given images, which includes the major findings from the radiographs and their statuses (i.e., positive, negative, and uncertain). in the second stage, we feed both images and the observation plan into a transformer model to generate the report. here, a tree reasoning mechanism is devised to enrich the concise observation plan precisely. specifically, we construct a three-level observation graph, with the high-level observations as the first level, the observation-aware n-grams as the second level, and the specific tokens as the third level. these observation-aware n-grams capture different common descriptions of the observations and serve as the component of observation mentions. then, we use the tree reasoning mechanism to capture observation-aware information by dynamically aggregating nodes in the graph. in conclusion, our main contributions can be summarized as follows: ## conclusion in this paper, we propose organ, an observationguided radiology report generation framework, which first produces an observation plan and then generates the corresponding report based on the radiograph and the plan. to achieve better observation realization, we construct a three-level observation graph containing observations, observationaware n-grams, and tokens, and we propose a tree reasoning mechanism to capture observationrelated information by dynamically aggregating nodes in the graph. experimental results demonstrate the effectiveness of our proposed framework in terms of maintaining the clinical consistency between radiographs and generated reports.