Knowledge Enhanced Deep Learning Model for Radiology Text Generation (2022)

Abstract: Manual radiology report generation is a time-consuming task. First, radiologists prepare brief notes while carefully examining the imaging report. Then, radiologists or their secretaries create a full-text report that describes the findings by referring to the notes. Automatic radiology report generation is the primary objective of this research. The central part of automatic radiology report generation is generating the finding section (main body of the report) from the radiologists’ notes. In this research, we suggest a knowledge graph (KG) enhanced radiology text generator that can provide additional domain-specific information. Our approach uses a KG-BART model to generate a description of clinical findings (referred to as pathological description) from radiologists’ brief notes. We have constructed a parallel dataset of radiologists’ notes and corresponding pathological descriptions to train the KG-BART model. Our findings demonstrate that, compared to the BART-large and T5-large models, the BLEU-2 score of the pathological descriptions generated by our approach is raised by 4% and 9%, and the ROUGE-L score by 2% and 2%, respectively. Our analysis shows that the KG-BART model for radiology text generation outperforms the T5-large model. Furthermore, we apply our proposed radiology text generator for whole radiology report generation.

## introduction due to the meager ratio of radiologists to patients, radiologists are in high demand. the ratios in the us, china and india are 1:10,000, 1:14,772, and 1:100,000 respectively @xcite . it leads to a large influx of patients, which keeps radiologists extremely busy and under stress. to boost the effectiveness and productivity of radiologists, several hospitals and diagnostic facilities have established radiology information systems (ris) and picture archiving and communications systems (pacs) @xcite . despite this, the current workflow causes a delay in the turnaround time for reports, report inaccuracies, and burnout. our conversations with radiologists have revealed that many radiologists wish to eliminate the tiresome reportgenerating process and concentrate on the diagnosis. the main task in automatic radiology report generation is generating pathological descriptions from radiologists' notes. in natural language processing (nlp), we can look at this as a text generation task. various neural encoder-decoder models have been proposed to accomplish the text generation goal by learning to map input text to output text. however, the input text alone often provides limited knowledge to generate the desired output. hence, researchers have considered incorporating external knowledge from domain-specific kgs along with internal knowledge embedded in the input text @xcite . kg-bart @xcite model incorporates the domain-specific kg in the deep learning model. in our work, we incorporate ultrasound radiology kgs @xcite in the kg-bart model to generate radiology text from radiologists' input notes. we construct a radiology domain dataset to train the kg-bart model. we obtain grounded kg for input sentences. kg grounding finds the most relevant entities and relations from radiology kg to guide the kg-bart model to better understand the relationships among concepts. it considers the inter-concept relation and significant neighbor entities to generate a more natural and plausible output. two high-profile radiologists are associated with this research who help us to get domain insights and to create a dataset. our contributions in this paper are as follows: ## background concept and terminology traditionally, radiologists either dictate on a voice recorder or write the diagnosis notes (referred to as radiologist's notes) on paper. their secretaries are then given access to the notes. next, the secretary access a normal report template, which is a scan-specific normal template (referred to as normal report template) that corresponds to all normal findings, and creates a preliminary report by altering it in accordance with the measurements and findings that the radiologist reported in a more detailed form (pathological description). the radiologist receives the preliminary report once again. the radiologist then reads the report, makes any necessary corrections, and then signs off. finally, the report is provided to a doctor or patient for potential follow-up care. ## related work with prior knowledge of chest findings, zhang et al. (2020) created a graph model that could be used in deep learning models. disease findings are represented in this network as nodes, and related findings are connected closely between them so that they might impact one another during the propagation and aggregation of the graph. to learn specific attributes for each node in the graph, they incorporate this graph into the deep neural network. features extracted from kg are used for multilabel classification. to improve pre-training language understanding, @xcite integrates kg instructive items that are contextually aligned. ke-pler uses a pre-trained language understanding model to encode textual descriptions of entities be-fore integrating the goals of knowledge embedding and language modeling @xcite . by including triples from the kg as supplemental words, k-bert infuses domain information into the models @xcite . in light of these studies, we contend that additional knowledge data can significantly improve the performance of pre-training models used for text generation tasks. ## dataset construction we fetch impressions and corresponding pathological descriptions from the radiology text report corpus to construct a parallel dataset. the radiology text report corpus contains anonymized radiology ultrasound reports that are provided by a company collaborating with us, with due consent of the physicians. we have approximately 10 lac radiology reports, out of which around 1 lac reports are of ultrasound. the radiology report contains the title, history, findings, and impression sections. each section's content is well-structured despite not being uniform. as a result, we use heuristics like regular expressions and word overlap to identify different sections. each section's content is tokenized and lower-cased. impressions are very close to radiologists' notes, but it does not contain patient-specific information like measurement of findings, anatomical location, etc. to convert impressions into radiologists' notes, we manually edit impressions by adding patient-specific information like measurements and anatomical locations by referring to pathological descriptions. examples of impressions, their corresponding pathological descriptions and radiologists' notes prepared using impressions are given in ## method figure 1 shows the architecture of the textgeneration model with input/output flow. the main components of our model are kg grounding, kg embeddings, text embedding, encoder, and decoder. this section explains all these components in detail. ## experimental setup the model input consists of the concept set and kg encoding for the input-concept and conceptexpansion graphs. the output is the target statement, i.e., pathological description. we use above ## baseline and evaluation we compare the performance of kg-bart model with t5-base/large @xcite and bartbase/large @xcite state-of-the-art pretrained conditional text generation models. following other conventional generation tasks, we use several widely-used automatic metrics to automatically assess the performance, such as bleu @xcite and rouge @xcite , which mainly focus on measuring n-gram similarities. ta-ble 4 shows the bleu score and rouge score of generated pathological descriptions by kg-bart and t5-base/large and bart-base/large models vs. gold standard pathological descriptions. ## qualitative analysis kg-bart model performs better on unseen data. sentence formation of the kg-bart model is better than t5 and bart models when input is in abstract form and random in sequence. as shown in ## radiology report generation using kg-bart radiology text generator radiology report generation includes following main tasks: for the first task we use our proposed radiology text generator to generate pathological description from radiologists' notes. this section gives the details of second task; how to replace generated pathological description in normal report template to generate whole report. our domain experts provide us normal report templates. for example, male abdomen pelvis ultrasound normal report, female abdomen ultrasound normal report etc. according to patient's gender and scan procedure we provide an appropriate normal report template to the system. system finds the appropriate normal sentences to replace with generated pathological description and replace it. as we discussed with hospitals, radiologists, physicians, etc., they are happy to provide impression by themselves to generate whole report. we add impressions provided by radiologists in impression section and generates the whole report. ## conclusion we have constructed a parallel dataset of radiologists' notes and corresponding pathological descriptions. kg-bart for radiology text generation produces high-quality sentences by capturing relationships between the concepts in the input. it also considers default properties from the kg if they are missing in the input concept set to generate more logical and natural sentences. our approach to construct grounded kgs does not add noise since it only considers entities in the hierarchical path from concept to root node and only adds neighbors with default properties. experimental results show figure 3 : ui interface of our radiology report generation system. the upper text area shows the inputs (radiologist's notes) given to the system. drop-box gives the option to select the template for male or female. after clicking on generate report button, the system generates a pathological description using our kg-bart radiology text generator and then generates the whole report. the lower first column shows the normal report template. we display the generated pathological description in the lower mid column. the third column shows the generated report. once the report is generated, we can download it by clicking on the