D elta N et: Conditional Medical Report Generation for COVID -19 Diagnosis (2022)

Abstract: Fast screening and diagnosis are critical in COVID-19 patient treatment. In addition to the gold standard RT-PCR, radiological imaging like X-ray and CT also works as an important means in patient screening and follow-up. However, due to the excessive number of patients, writing reports becomes a heavy burden for radiologists. To reduce the workload of radiologists, we propose DeltaNet to generate medical reports automatically. Different from typical image captioning approaches that generate reports with an encoder and a decoder, DeltaNet applies a conditional generation process. In particular, given a medical image, DeltaNet employs three steps to generate a report: 1) first retrieving related medical reports, i.e., the historical reports from the same or similar patients; 2) then comparing retrieved images and current image to find the differences; 3) finally generating a new report to accommodate identified differences based on the conditional report. We evaluate DeltaNet on a COVID-19 dataset, where DeltaNet outperforms state-of-the-art approaches. Besides COVID-19, the proposed DeltaNet can be applied to other diseases as well. We validate its generalization capabilities on the public IU-Xray and MIMIC-CXR datasets for chest-related diseases.

## introduction since december 2019, the world has been suffering from a serious health crisis: the outbreak of covid-19 @xcite . fast screening and diagnosis is critical in covid-19 patient treatment. in clinical practice, the reverse transcription polymerase chain reaction (rt-pcr) is recognized as the golden standard @xcite . however, due to high false-negative rate and shortage of equipment @xcite , medical imaging like x-ray and computed tomography (ct) @xcite also works as * corresponding author an alternative means in covid-19 diagnosis and treatment which generates more timely results than rt-pcr and helps in evaluating the severity degree of covid-19. given medical images of covid-19 patients, radiologists need to write relatively long reports to address the impressions and findings. considering the large volume of covid-19 patients and potentially infected population, writing medical reports becomes a heavy burden for radiologists. furthermore, due to the varied expertise of radiologists, some abnormalities in medical images may be ignored and thus not included in the final reports. to alleviate the heavy workload and aid less experienced radiologists, automatically generating medical reports becomes a critical task. due to its importance in clinical practice, automatically generating medical reports has attracted extensive research interests in recent years @xcite . existing works mainly follow the image captioning approaches and employ an encoderdecoder process. in the encoding stage, the visual features are extracted from medical images via a cnn; in the decoding stage, the reports are generated sequentially via an rnn. such a two-step framework has been proven effective in generating general image captions, such as ms coco. however, when applied to medical images, it may have following two problems: 1) visual bias: for most cases, the abnormal regions only occupy a small fraction of the entire medical images, thus visual features of normal regions dominate the extracted visual embedding, and abnormal regions are difficult to identify; 2) textual bias: in current medical reports, the majority of transcription focuses on describing the normal regions which distract the model from abnormal regions in training. to address two problems mentioned above, we propose deltanet, which is customized for medical images. different from encoder-decoder frameworks that generate reports from a single medical image, deltanet introduces a retrieve-update process which consists of the following three steps: 1) retrieval: deltanet firstly retrieves conditional medical images and reports from medical records. for patients who've already been examined before, we directly obtain his previous medical images and reports. for patients examined for the first time, we retrieve the medical images and reports with similar visual features from other patients; 2) comparison: deltanet compares the embedding difference between two medical images to capture the visual difference; 3) based on the identified visual difference and the retrieved reports, deltanet conditionally generates the final report for current medical image. to prove the effectiveness of the proposed deltanet, we collect a dataset of covid-19 patients. for each patient, we manage to collect all the historical medical images and the corresponding reports that cover the complete treatment process. the experimental results show that the proposed deltanet outperforms the baselines, including both the general image captioning approaches and the existing medical report generation works. besides generating reports, deltanet can also highlight the differences between previous and current medical images and the corresponding updated reports, which provides a clear explanation for the generated report. in addition to covid-19, the proposed deltanet can also be applied to medical report generation of other diseases. we evaluate the generalization ability of deltanet on iu-xray and mimic-cxr datasets. deltanet consistently outperforms baselines. ## model in this section, we introduce the proposed deltanet. firstly, we formulate the conditional medical report generation problem; secondly, we describe the basic encoder-decoder based generation model; then, we propose deltanet, a conditional medical report generation model; finally, we further extend deltanet to exploit multiple conditional reports. ## experiments in this section, we evaluate the proposed deltanet from three perspectives: (1) whether incorporating conditional reports can bring in performance gain in medical report generation; (2) whether increasing the number of conditional reports can further improve the performance; (3) which will perform better, the conditional reports that are acquired from the historical reports of patients themselves or retrieved from other patients according to visual similarity. ## conclusion and future works in this paper, we targeted to automate the medical report generation. different from typical encoderdecoder framework, we proposed a conditional generation model deltanet. for patients with historical reports, we combined the input image with the historical report to generate reports. for first time patients without historical reports, we retrieved visual similar reports from other patients as conditional reports. we proved the advantage of the proposed deltanet over state-of-the-art approaches on the iu-xray, mimic-cxr and covid-19 datasets. in this paper, for new patients without historical reports, we retrieve the conditional reports only via visual similarity. however, more features like age, gender and diseases could be useful in selecting conditional reports. in addition, we may attempt to combine historical reports from the same patient and simliar reports from other patients together to futher improve the performance.