KGVL - BART : Knowledge Graph Augmented Visual Language BART for Radiology Report Generation (2023)

Abstract: Timely generation of radiology reports and diagnoses is a challenge worldwide due to the enormous number of cases and shortage of radiology specialists. In this paper, we propose a Knowledge Graph Augmented Vision Language BART (KGVL-BART) model that takes as input two chest X-ray images- one frontal and the other lateral- along with tags which are diagnostic keywords, and outputs a report with the patient-specific findings. Our system development effort is divided into 3 stages: i) construction of the Chest X-ray KG (referred to as chestX-KG), ii) image feature extraction, and iii) training a KGVL-BART model using the visual, text, and KG data. The dataset we use is the well-known Indiana University Chest X-ray reports with the train, validation, and test split of 3025 instances, 300 instances, and 500 instances respectively. We construct a Chest X-Ray knowledge graph from these reports by extracting entity1-relation-entity2 triples; the triples get extracted by a rule-based tool of our own. Constructed KG is verified by two experienced radiologists (with experience of 30 years and 8 years, respectively). We demonstrate that our model- KGVL-BART- outperforms State-of-the-Art transformer-based models on standard NLG scoring metrics. We also include a qualitative evaluation of our system by experienced radiologist (with experience of 30 years) on the test data, which showed that 73% of the reports generated were fully correct, only 5.5% are completely wrong and 21.5% have important missing details though overall correct. To the best of our knowledge, ours is the first system to make use of multi-modality and domain knowledge to generate X-ray reports automatically.

## introduction medical imaging techniques are widely used in hospitals across the world. the detailed informa-tion extracted from medical images is crucial for proper diagnosis and treatment. an experienced and skilled radiologist is required to prepare an accurate full text diagnostic report. however, due to a lack of experts, many reports contain indecisive findings, forcing patients to undergo further tests involving pathology or other advanced imaging methods. in addition, the time consuming process of full text radiology report generation is one of the biggest challenges. all over the world, the ratio of radiologists to patients is very low. the ratios in the us, china, and india are 1:10,000, 1:14,772, and 1:100,000 respectively @xcite . given the huge number of cases and the shortage of radiology experts, timely report generation and diagnosis is a huge challenge worldwide. in the case of a chest ailment, x-rays produce images of chest organs like, lungs, spinal bones, heart, airways, and blood vessels. these images help doctors ascertain an exact findings such as pneumonia, collapsed lung, emphysema, cancer, broken ribs, etc. manual examination of x-ray images for a large number of patients can be time consuming, leading to delays. human errors may further add to the challenges. this prompted the researchers to use deep learning models capable of automated report generation to address the above mentioned challenges. with deep learning based automatic report generation, the reports can be generated with minimal delay and free from any human errors. large-scale pre-trained language models have recently expanded into multimodality learning, improving representations by combining visual and semantic features @xcite @xcite . however, progress in adapting language models toward conditional natural language generation (nlg) is limited to image and text modalities. the natural language processing (nlp) community is moving towards transformer-based models. the major-ity of nlp research today produces better results by tweaking an already-trained transformer model across a large corpus. this prompts us to research pre-trained transformers like bart that have generative capabilities while conditioning them on visual, textual, and kg data. this study introduces the kgvl-bart conditioned transformer-based model, which, produces a comprehensive report given an x-ray image, tags associated with x-ray image.)), and chest x-ray kg. we train our model using the publicly accessible indiana university chest x-ray dataset (referred to as iu-xray) @xcite . the encoder accepts input from tags, images, and kg in three different modalities. we create embeddings for each of these modalities before sending input to the multimodal encoder. we compare our quantitative findings to earlier transformer-based models. furthermore, we provide qualitative analysis on test set by a radiologist. problem statement: design a system that generates a structured patient-specific report from radiology images, image tags and domain knowledge. the input to the system is ## related work researchers @xcite @xcite studied the issue of automatic report generation. their research looked into the visual attention given to recurrent decoders and convolution-recurrent architectures (cnn-rnn) that were first introduced by @xcite on image captioning. transformers, attention-only based models that have replaced recurrent models in the nlp community @xcite . several attempts have been made in the medical field to create medical reports from the corresponding images. most authors use multilabel image captioning to produce x-ray reports, and they subsequently use those captions as textual features. the iu-xray dataset's chest x-ray images were used to generate the first structured report using tags predicted by a cnn-rnn model @xcite . in @xcite , a system for generating natural reports for the chest-xray14 dataset, employing private reports, was presented. this framework used a non-hierarchical cnn-lstm architecture and focused on semantic and visual aspects. the iu-xray dataset was created by @xcite to generate radiology reports automatically. there is a lot of research performed in nlg on multimodal constraints. nlg models based on transformers @xcite propose a model called mantis, multimodal adaptation for text synthesis, as a general method for multimodal conditionality. in this method, separator tokens are used to separate each modality type, and modality-specific encoders are used to encode each modality type. @xcite contend that pretrained language models and textual concepts by themselves are insufficient to give enough data for generative commonsense reasoning. they supply the conceptnet kg as input to the transformer model for generative commonsense reasoning in addition to text input. @xcite proposes a knowledge-enabled bidirectional encoder representation from transformers (k-bert). since the parameters of all pre-trained bert models are the same, k-bert can load any of them. additionally, k-bert can easily integrate domain knowledge into the models by giving them access to a kg without prior training. @xcite propose km-bart to conduct the task of visual commonsense generation (vcg) by integrating visual features in pretrained bart model. ## methodology as shown in the figure 1 , the model architecture consists of six major components, namely, kg grounding, kg embedding, image embedding, text embedding, encoder, and decoder. the kg grounding module extracts the grounded kg from the chestx-kg. grounded kg includes all of the nodes in the input tag set and their significant neighbors. the kg embedding module converts the grounded kg into vector form using the kge technique. the image feature extractor module generates the feature vectors for input chest x-ray images. to compute the text features, we use the bart text encoder method. the encoder and decoder are multilayer transformers. this section explains all components in detail. ## experiments the datasets, evaluation metrics, and baselines used for the training and evaluation of the kgvl-bart model are covered in detail in this section. ## summary, conclusion and future work we introduce kgvl-bart a knowledge-enhanced vision language model to generate x-ray reports from chest x-ray images and tags. we develop a knowledge graph called chestx-kg, which is verified by two experienced radiologists. overall the reports generated are accurate. our approach to constructing grounded kgs is relatively noise-free since it considers only entities in the hierarchical path from tag set entities to the root node of the knowledge graph and only adds neighbors with "default" properties. experimental results show the efficacy of our method and its superiority over sota.