IBMR esearch at MEDIQA 2021: Toward Improving Factual Correctness of Radiology Report Abstractive Summarization (2021)

Abstract: Although recent advances in abstractive summarization systems have achieved high scores on standard natural language metrics like ROUGE, their lack of factual consistency remains an open challenge for their use in sensitive real-world settings such as clinical practice. In this work, we propose a novel approach to improve factual correctness of a summarization system by re-ranking the candidate summaries based on a factual vector of the summary. We applied this process during our participation in MEDIQA 2021 Task 3: Radiology Report Summarization, where the task is to generate an impression summary of a radiology report, given findings and background as inputs. In our system, we first used a transformer-based encoder-decoder model to generate top N candidate impression summaries for a report, then trained another transformer-based model to predict a 14-observations-vector of the impression based on the findings and background of the report, and finally, utilized this vector to re-rank the candidate summaries. We also employed a source-specific ensembling technique to accommodate for distinct writing styles from different radiology report sources. Our approach yielded 2nd place in the challenge.

## introduction the radiology report is a crucial instrument in patient care and an essential part of every radiological procedure, serving as the official interpretation of a radiological study and the primary means of communication between the radiologist and referring physician. according to the american college of radiology, a radiology report should contain certain components, such as relevant clinical information, imaging findings, limitations of the study, and an impression or conclusion (american college of radiology, 2020). of these, the impression is the most important component of the radiology report, containing conclusions based on the pertinent findings and suggestions for additional diagnostic studies if warranted @xcite . previous studies have shown that oftentimes it is the only part of the report that is read; one previous study found that 43% of referring physicians only read the impression if the report was longer than one page @xcite , while another study found that 23.1% of clinicians agreed with the statement "i usually only read the conclusion of a radiology report" @xcite . in an effort to support radiologists in writing impressions in radiology reports, @xcite introduced the task of automatic generation of radiology impression statements by summarizing textual findings written by radiologists. mediqa 2021 @xcite , as part of naacl-bionlp 2021 workshop, aims to further research efforts in summarization in the medical domain. task 3 of the challenge, radiology report summarization (rrs), focuses specifically on radiology impression generation. the basic task setup is as follows: given the findings and background sections of a radiology report, predict the impression or summary. in this paper, we detail our participation in mediqa 2021 rrs challenge. we developed an approach that utilizes a structured label vector of the impression as our proxy for facts for the impression (predicted using findings and background of the report), to re-rank the generated abstractive summaries from a trained encoder-decoder model. we further employed a source-specific ensembling technique utilizing models fine-tuned to each radiology report source to accommodate for distinct language patterns in each source. our system performed well in the challenge, placing us 2nd on the leaderboard. summarization and factual correctness in radiology reports. @xcite first studied the problem of automatic generation of radiology impressions by summarizing textual radiology findings, and showed that an augmented pointergenerator model achieves high overlap with human references. they also found that about 30% of the radiology summaries generated from neural models contain factual errors. research scholars also integrated radlex ontology into seq2seq models @xcite to enhance the clinical validity of automated impression prediction systems within the radiology workflows. in their next work, @xcite improved upon the problem of factual correctness in radiology reports by optimizing fact scores defined in radiology reports with reinforcement learning methods. they also introduced a new metric factual f 1 comparing the predicted summaries using a descriptor vector of the gold summary. in our work, we extend the ideas put forward by @xcite by utilizing a descriptor vector (generated using off-the-shelf systems like chexpert @xcite or chexbert @xcite ) to re-rank the automatically generated summaries. ## task description and dataset the mediqa-2021 rrs task is defined as follows: given a passage of findings represented as a sequence of tokens @xmath0, x 2 , . . . , x n }, with n being the length of the findings, and a passage of background represented as a sequence of tokens @xmath1, y 2 , . . . , y m } with m being the length of the background, find a sequence of tokens @xmath2, z 2 , . . . , z l } that best summarizes the salient and clinically significant findings in x, with l being an arbitrary length of the impression or summary foot_0 . datasets for training and validation of summarization models provided by the mediqa organizers consisted of radiology reports with findings, background, and impression sections. the training set consists of 91,544 radiology reports from the mimic-cxr database @xcite , while the validation set consists of an additional 4,000 radiology reports -2,000 from mimic-cxr and 2,000 from the indiana network for patient care (indiana) @xcite . as part of the shared task rules, the rest of the publicly available mimic-cxr and indiana radiology reports were not allowed for use in training or validation. however, the organizers allowed the use of validation data for training. at the conclusion of the shared task, to evaluate participant systems, a test set of 600 radiology reports containing only findings and background sections were released with their sources unknown at the time of the challenge. dataset statistics are presented in the ## system description our system is a three-step process in which we (1) utilize pre-trained transformer-based language models in an encoder-decoder setting to get our base summarization models, (2) improve the factual correctness of our base models' predictions by incorporating a re-ranking methodology, and (3) utilize a source-specific ensembling technique which identifies the source of a radiology report, and chooses the prediction of the best performing source-specific model accordingly. we detail the above three steps in the following sections. ## discussion in this section, we present two major findings of our approach. first, we find that radiology reports from different sources have distinct language, and fine-tuning a model trained on source a with a small amount of data from source b provides significant gains in performance on source b, allowing the model to be transferable. as it can be seen in second, fact-aware re-ranking methodology improves performance of the models on natural language metrics (rouge) as well as factual correctness of our predictions, but metrics beyond lexical overlap are needed. as shown in limitations and future work. we acknowledge several limitations to our work. first, we recognize our dependence on an external structured label generator. as we use chexbert labels as our proxy for ground truth for training our 14observations-vector predictor, as well as in our similarity function, any errors in chexbert have a direct impact on our system's performance. second, though far methodology has shown significant gains in performance in factual f 1 and rouge scores, the system is limited by the generated candidate summaries. we aim to build on this approach by incorporating this methodology during training as a modified version of beam search. third, all of our presented results are evaluated using a relatively small set of internal test data, due to the limitations on data during the challenge. though ## conclusion we have presented our system developed during our participation in mediqa 2021 rrs challenge. we found that radiology reports from different sources have distinct language and fine-tuning a trained model with a small amount of data from another source leads to gains in performance and allows the models to be transferable. further, techniques like fact-aware re-ranking, which utilizes a factual vector of the summary to re-rank candidate summaries, not only improves factual correctness of the summary but also improves the performance of the model on the traditional natural language metrics like rouge. we have also identified limitations of our work, and discussed promising areas of future research.