Factual Accuracy is not Enough: Planning Consistent Description Order for Radiology Report Generation (2022)

Abstract: Radiology report generation systems have the potential to reduce the workload of radiologists by automatically describing the findings in medical images.To broaden the application of the report generation system, the system should generate reports that are not only factually accurate but also chronologically consistent, describing images that are presented in time order, that is, the correct order.We employ a planning-based radiology report generation system that generates the overall structure of reports as “plans’” prior to generating reports that are accurate and consistent in order.Additionally, we propose a novel reinforcement learning and inference method, Coordinated Planning (CoPlan), that includes a content planner and a text generator to train and infer in a coordinated manner to alleviate the cascading of errors that are often inherent in planning-based models.We conducted experiments with single-phase diagnostic reports in which the factual accuracy is critical and multi-phase diagnostic reports in which the description order is critical.Our proposed CoPlan improves the content order score by 5.1 pt in time series critical scenarios and the clinical factual accuracy F-score by 9.1 pt in time series irrelevant scenarios, compared those of the baseline models without CoPlan.

## introduction radiologists regularly write qualitative radiology reports to accurately describe the recognized findings in medical images. recently, we can observe two different approaches to radiology imaging: a single-phase and a multiphase imaging method. a single-phase diagnostic approach, as applied in a plain x-ray machine, scans only once, while many modern procedures, including liver contrast ct, use a multiphase diagnostic method, scanning sequentially in a period of several minutes. the timedependent scans are labeled as phases. reports of single-phase diagnoses are time series irrelevant, so the factual accuracy is the most critical quality !"#@xmath0()@xmath1#.+%&/#, %.,@xmath2()@xmath30#&)()'#.'@xmath4+#2'@xmath530%1('-4@xmath6()@xmath70+#5&%+@xmath80.#5*3'"3&%64 !"#@xmath9% &'#(()*)%+ !"#@xmath10:; 7<5./@xmath11:; 7:+#5&%+@xmath12?; 7:.#5*3'"3&%69@xmath13%#@xmath14%."#(,'&/ !"#"@xmath15&/%#%*)@xmath1627,809 @4@xmath17:; 7<5./@xmath18:; a4@xmath19?; 7:.#5*3'"3&%69$?; !!! figure 2: overview of our planning-based report generation system. to broaden the application of report generation systems to multiphase diagnosis, the system should also become capable of generating reports that are written in consistent description order. radiolo-gists intend to write high-quality radiology reports with consistent description order so that doctors can review and understand the radiology reports within a short time (european society of radiology (esr), 2011). the existing studies @xcite can support single-phase diagnostics only, particularly in chest x-ray images. as mentioned in the limitation section of @xcite , contemporary studies face obstacles to generating reports with time series. this study aims to design an automated radiology report generation system that generates factually accurate reports for time series irrelevant diagnoses and consistent description order reports for time series critical diagnostics. as shown in figure 2 , we employ a planning-based report generation system that consists of three modules: image classifier, content planner, and text generator. the content planner generates the "plan," which represents the content and description order of the reports, and then the text generator predicts accurate and consistent description order reports. planning-based models generate more faithful sentences without hallucination than end-to-end models @xcite . however, planning-based approaches have a critical disadvantage; they cascade errors in modules within the system. to solve the error cascading problem, plangen @xcite employs a reinforcement learning that encourages the generated output to adhere to the given content plan, and dyploc @xcite uses multiple plan candidates in the content realization process to reflect the dynamic nature of plans. we propose coordinated planning-based text generation (coplan), a novel unified framework that trains and conducts inferences on the content planner and text generator in coordination to generate more accurate reports in a consistent order. coplan checks cascaded errors in the final output of the system with a report evaluator to generate more appropriate plans for creating accurate reports. we employ two types of report evaluators for coplan, a fact-based evaluator and a description-order-based evaluator, to generate factually accurate reports for time series irrelevant scenarios and consistent description order reports for time series critical scenarios. the contributions of this study are as follows: we evaluate our proposed coplan in both timeseries irrelevant and critical scenarios. the datasets of multiple languages and modalities are used: the jliverct dataset in which the reports are written with time series description and the mimic-cxr dataset @xcite containing time irrelevant reports (i.e. only factual accuracy is critical). the results of the automatic and human evaluations show that our proposed method improves the accuracy of the dataset without times series and improves the consistency of the description order of the dataset with time series, compared to those of the models without coplan. we employed a unified framework that enables both training time and inference time to cope with error cascading. in addition, our coplan addresses factual accuracy and the description order-based output candidate quality estimator for both time series irrelevant scenarios and time series critical scenarios to select the best plans. ## experimental results we conducted two types of experiments: the d2t experiment for evaluating the d2t module and the e2e experiment for evaluating the entire system. ## conclusion we proposed a planning-based neural radiology report generation method for generating reports with the consistent description order on top of the factual accuracy of the content.the results of the evaluations in both time series critical and time series irrelevant datasets revealed that our proposed coplan improved both the factual accuracy and consistency of the description order of the generated reports. however, as shown in sec. 6.2, all radiology report systems have a limitation to generate descriptions regarding normalities without doctors' intentions. in the future, we will combine our system with a human-in-the-loop approach that can reflect doctors' intentions to co-create high-quality reports in a short time.