“Knowledge is Power”: Constructing Knowledge Graph of Abdominal Organs and Using Them for Automatic Radiology Report Generation (2023)

Abstract: In conventional radiology practice, the radiologist dictates the diagnosis to the transcriptionist, who then prepares a preliminary formatted report referring to the notes, after which the radiologist reviews the report, corrects the errors, and signs off. This workflow is prone to delay and error. In this paper, we report our work on automatic radiology report generation from radiologists’ dictation, which is in collaboration with a startup about to become Unicorn. A major contribution of our work is the set of knowledge graphs (KGs) of ten abdominal organs- Liver, Kidney, Gallbladder, Uterus, Urinary bladder, Ovary, Pancreas, Prostate, Biliary Tree, and Bowel. Our method for constructing these KGs relies on extracting entity1-relation-entity2 triplets from a large collection (about 10,000) of free-text radiology reports. The quality and coverage of the KGs are verified by two experienced radiologists (practicing for the last 30 years and 8 years, respectively). The dictation of the radiologist is automatically converted to what is called a pathological description which is the clinical description of the findings of the radiologist during ultrasonography (USG). Our knowledge-enhanced deep learning model improves the reported BLEU-3, ROUGE-L, METEOR, and CIDEr scores of the pathological description generation by 2%, 4%, 2% and 2% respectively. To the best of our knowledge, this is the first attempt at representing the abdominal organs in the form of knowledge graphs and utilising these graphs for the automatic generation of USG reports. A Minimum Viable Product (MVP) has been made available to the beta users, i.e., radiologists of reputed hospitals, for testing and evaluation. Our solution guarantees report generation within 30 seconds of running a scan.

## introduction radiology is an integral part of medical care. radiological imaging-based evidence (x-ray, mri, ct, usg, etc.) is crucial for determining the nature of the treatment. the usual radiology process is as follows: a patient gets scanned. then the radiologist prepares the diagnosis notes (referred to as radiologist's dictation) and handing them over to a transcriptionist. the transcriptionist opens a scan-specific standardised template (referred to as normal report template) and edits it refering to the notes in a more descriptive form (referred to as pathological description). radiologists are in huge demand since the ratio of radiologists to patients is very low. these ratios in india, the us, and china are 1:100,000, 1:10000, and 1:14772, respectively @xcite . these low ratios results in a very high patient inflow per radiologist volume, making radiologists incredibly busy and stressed out. the currently adopted transcriptionist-based workflow causes (i) significant delays in report turnaround time, (ii) errors in the reports, and (iii) burnout. to automate the report generation process, domain knowledge is necessary. domain knowledge can be acquired from already existing radiology free-text reports. we need a structured format for knowledge to be able to use it on a computer. our research aims to use natural language processing (nlp) (a) to construct abdominal-organ kg and (b) use these kgs for automatically generating radiology reports. our work is in collaboration with a industry partner. on this project, two experienced radiologists are contributing their domain expertise to our work. our contributions are: ontologies are semantic data models that define the types of things in a specific domain and the properties used to describe those types. ontologies does not include the details about specific individuals in domain. three main components of ontology are classes, relationships, and attributes. domain ontology and individual information together form a knowledge base (kb). we have defined eight logical relations as follows: ## related work research is done for automatic radiology report generation based on scanned images. yuan et al. @xcite propose an automated structured-radiology report generation system using extracted features from images. loveymi et al. @xcite proposed a system that generates descriptions for natural images by image captioning. there is a wealth of research done on building medical kg from electronic medical records (emr). @xcite builds a graph from medical text, clinical notes etc. graph nodes represents diseases, drugs, procedures, and devices. rotmensch et al. @xcite uses the emr to construct the graph of diseases and symptoms. researchers worked on creating medical kg from emr, but no one has built a kg for the radiology domain except zhang et al. @xcite . graph embedding module is proposed by @xcite that helps to generate radiology reports from image reports. each node in their kg represents disease. taira et al. @xcite developed an nlp pipeline to structure critical medical information. extracted information includes the existence, location, properties, and diagnostic interpretation of findings from radiology free-text documents. information is not integrated since they store the structured information for each report separately. also, this system does not accept reports with different reporting styles. however, this is not always the case. every radiologist has his or her own dictation style and reporting style. ie systems that are based on ie patterns are surveyed by muslea et al. @xcite . ghoulam et al. @xcite extract signs of lung cancer, their anatomical location, and the relation between the signs and the locations expressed in the radiology reports. @xcite used a morphosyntactic patterns in their rule-based method to find medical entities like symptoms, disease, exams, medicaments, and treatment. xu et al. @xcite explains that a pattern is a sub dependency tree that indicates a relation instance. pons et al. @xcite give an overview of nlp techniques that can be used in radiology. ## methodology as shown in figure 1 , we first construct the kgs for each abdominal organ from the ultrasound report corpus. then we use these constructed kgs to generate ultrasound radiology reports from the radiologist's dictation. kgs are constructed from anonymized radiology reports provided by our company collaborator. the anonymized report collection consists of approximately 10,000 reports of ultrasound scans. triplets extractor word or phrase processor dependency parser semantic analyser radiology reports corpus triplets to kg radiology ontology radiologist's dictation report generation generate pathological description replace in normal report template normal report template patient specific report kg ## summary, conclusion and future work we have given a systematic method to construct organ-wise kgs from free-text radiology reports. the kgs are stored in standard rdf format, enabling their application to various medical applications. one such example is the generation of radiology reports, which we have described here. our kg-enhanced deep learning model improves the reported bleu-3, rouge-l, meteor, and cider scores of the pathological description generation by 2%, 4%, 2% and 2% respectively. our approach is generalized for other organs and scanned procedures, as evidenced in the eacl paper foot_5 . to the best of our knowledge, this is the first attempt at automatic ultrasound report generation. an mvp (minimum viable product) has been made available to the beta users (practicing radiologists) for testing and evaluation. we are continuously collecting feedback on our system from radiologists and continually refining the tool. we have observed that it can generate a report within 30 seconds of running a scan. yixiao zhang, xiaosong wang, ziyue xu, qihang yu, alan yuille, and daguang xu. 2020. when radiology report generation meets knowledge graph. in proceedings of the aaai conference on artificial intelligence, volume 34, pages 12910-12917.