Replace and Report: NLP Assisted Radiology Report Generation (2023)

Abstract: Clinical practice frequently uses medical imaging for diagnosis and treatment. A significant challenge for automatic radiology report generation is that the radiology reports are long narratives consisting of multiple sentences for both abnormal and normal findings. Therefore, applying conventional image captioning approaches to generate the whole report proves to be insufficient, as these are designed to briefly describe images with short sentences. We propose a template-based approach to generate radiology reports from radiographs. Our approach involves the following: i) using a multilabel image classifier, produce the tags for the input radiograph; ii) using a transformer-based model, generate pathological descriptions (a description of abnormal findings seen on radiographs) from the tags generated in step (i); iii) using a BERT-based multi-label text classifier, find the spans in the normal report template to replace with the generated pathological descriptions; and iv) using a rule-based system, replace the identified span with the generated pathological description. We performed experiments with the two most popular radiology report datasets, IU Chest X-ray and MIMIC-CXR and demonstrated that the BLEU-1, ROUGE-L, METEOR, and CIDEr scores are better than the State-of-the-Art models by 25%, 36%, 44% and 48% respectively, on the IU X-RAY dataset. To the best of our knowledge, this is the first attempt to generate chest X-ray radiology reports by first creating small sentences for abnormal findings and then replacing them in the normal report template.

## introduction radiology report generation, which aims to automatically generate a free-text description of a clinical radiograph (like a chest x-ray), has become an important and interesting area of research in both clinical medicine and artificial intelligence. natural language processing (nlp) can speed up report generation and improve healthcare quality and standardization. thus, recently, several methods have been proposed in this area @xcite @xcite @xcite . radiology reports are lengthy narratives, which makes report generation difficult. therefore, applying conventional image captioning approaches to generate the whole report @xcite proves to be insufficient, as such approaches are designed to briefly describe images with short sentences. further, even if benchmark datasets are balanced between normal and abnormal studies, multiple organs' findings are included in a single report, and if at least one organ is abnormal, the report is classified as abnormal, but it still contains more normal sentences than abnormal. as a result, existing text generation models may be overly focused on widely reported normal findings. hence, we propose an approach to generating radiology reports by generating pathological descriptions, i.e., descriptions of abnormal findings, and then replacing corresponding normal descriptions from the normal report template with a generated pathological description. experimental results on the iu chest x-ray (referred to as iu x-ray) and the mimic-cxr benchmark datasets confirm the validity and effectiveness of our approach and demonstrate that our approach achieves better results than the state-of-the-art methods. two experts in the field were involved in this work. one is a radiologist with 30 years of experience, and the other is a doctor with an mbbs, md, and ph.d. and 2 years of experience in medicine. our contributions are: the topic of automatic report generation was researched by @xcite ; @xcite ; @xcite . several attempts have been made in the medical field to create medical reports from the corresponding images. most researchers use multilabel image captioning to produce x-ray reports, and they subsequently use those captions as textual features. the iu x-ray dataset was created by @xcite to generate radiology reports automatically. the iu x-ray dataset's chest x-ray images were used to generate the first structured report using tags predicted by a cnn-rnn model @xcite . a system for generating natural reports for the chest-xray14 dataset, employing private reports, was presented by @xcite . this framework used a non-hierarchical cnn-lstm architecture and focused on semantic and visual aspects. visual attention given to recurrent decoders and convolutionrecurrent architectures (cnn-rnn) was first introduced by @xcite on image captioning. radiology report generation has recently shifted to transformer-based models @xcite . knowledge-driven encode, retrieve, and paraphrase (kerp) @xcite is used for accurate and reliable medical report generation. @xcite suggests pretraining the encoder with several chest x-ray images to properly recognise 14 typical radiographic observations. according to @xcite proposals, radiology reports can be generated using a memorydriven transformer, while @xcite suggests a template-based x-ray report generation approach. @xcite clinically defines the templates for each abnormality to indicate its presence or absence. if the generated tags indicate any disease, then the system will choose the corresponding abnormal sentence template. this method cannot generate patient-specific data like anatomical location, size, etc. @xcite proposed a template-based multi-attention report generation model (tmrgm) for normal and abnormal studies. this work uses template-based techniques exclusively for normal reports. the differences between our work and previous work are as follows: i) instead of generating the whole report at once, we generate smaller sentences for only abnormal findings and then replace them in the normal report template. ii) unlike other stateof-the-art models, our methodology does not put excessive emphasis on normal sentences. iii) if the report's findings are normal, we use a standard template for normal reports. ## methodology the approach that we follow to generate the radiology reports from the radiographs is as follows: ## summary, conclusion and future work we present a template-based approach for generating x-ray reports from radiographs. our model generates small sentences exclusively for abnormalities, which are then substituted in the normal report template to produce a high-quality radiology report. we create a replacement dataset that contains pathological descriptions and their corresponding normal sentences from the normal report template. our experimental results demonstrate that, compared to the state-of-the-art models, the bleu-1, rouge-l, meteor, and cider scores of the full reports generated by our approach are raised by 25%, 36%, 44% and 48%, respectively. also, clinical evaluation metrics show that our method generates more clinically accurate reports than the state-of-the-art methods. unlike other state-of-the-art models, our methodology does not put excessive emphasis on normal sentences. in the future, we plan to apply the proposed method to generate radiology reports for ct, mri, etc. for our experiments, we have used all samples from the iu x-ray dataset. but we have used only 44578 reports out of 227827 reports for the mimic-cxr dataset. our immediate plan is to perform experiments on the whole mimic-cxr dataset. take away from our work is that creating smaller sentences of pathological descriptions and replacing them in the normal template produces better quality reports than generating the whole report at once.