R ad L ing: Towards Efficient Radiology Report Understanding (2023)

Abstract: Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus. There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks. We present RadLing, a continuously pretrained language model using ELECTRA-small architecture, trained using over 500K radiology reports that can compete with state-of-the-art results for fine tuning tasks in radiology domain. Our main contribution in this paper is knowledge-aware masking which is an taxonomic knowledge-assisted pre-training task that dynamically masks tokens to inject knowledge during pretraining. In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.

## introduction radiology reports are radiologist interpretations of medical images such as x-rays, ct, ultrasound and mri scans. healthcare professionals rely on these reports to monitor and diagnose patients. a radiology report typically includes several sections @xcite , among which the most important ones are the following: the post-bert era @xcite of contextualized pretrained language models (plms) has drastically reduced the need for expensive and hard-to-find human annotated data for biomedical nlp. biomedical plms are usually trained on biomedical publications @xcite @xcite @xcite @xcite and have facilitated drug discovery and healthcare informatics. despite sharing the same concepts, challenges remain in adapting these models to radiology reports. this is because the contents, context and structure of biomedical publications are significantly different from those of the radiology reports. radiology reports are terse, and concept-dense. it is shown that models pretrained on radiology reports improve performance in downstream clinical nlp tasks as opposed to models pretrained on biomedical publications @xcite @xcite . while this is encouraging, we identified some research challenges as well as encountered issues in adapting these plms to industrial setting: research challenges. random masking in masked language modeling (mlm) is known to have context understanding issues @xcite . this is exemplified in tasks like relationship extraction @xcite in radiology reports. figure 1 shows an example sentence from chexpert @xcite . the entities 'pneumonia' and 'lungs' are related concepts, but current state-of-the-art plm-based methods fail to identify their relation in the sentence. industry adaptation issues. industry adaptation issues emanate from data and model sizes. training datasets available for industry are generally small. use of public datasets like mimic-iv (johnson et al., 2020) is not feasible for industry research due to license restrictions. large plms like pubmed-bert (109 million parameters) and best performing radbert variant(125 million) have latency issues in industrial deployment, with low throughput when deployed in low memory settings. reduction in parameters by quantisation or distillation reduces efficiency which is less than ideal. we believe that context understanding problem can be avoided with the help of domain knowledge. in radiology domain, an excellent source of domain knowledge is the taxonomical knowledge base radlex @xcite , which curates radiology lexicons. for the sentence in figure 1 , current plms misclassify the relationship. however, a look at radlex for the entity 'pneumonia' reveals its anatomical_site property to be 'lungs'. the radlex property anatomical_site and the radgraph relation 'located_at' are analogous. infusing this knowledge into plms adds more context and domain knowledge and thus increases prediction capabilities. in this paper, we introduce radling, the first radiology language model based on electra-small architecture with 13.7 million parameters trained with the help of radlex. our major contributions in this paper are: 2 related work radiology report understanding and radiology summarization are the most-addressed nlp tasks that are related to the field of radiology. radiology summarization involves generating a summary of a radiology report, which can help clinicians quickly understand the key findings without having to read the entire report (karn et al., 2022). radiology report understanding involves extracting information from a radiology report, such as the type of imaging study, problem list generation, etc. this information can then be used to assist with diagnosis and treatment planning. radiology report understanding using mlms. yan et al. (2022) has used mlm with several pretrained plms like bert, roberta(liu et al., 2019) and clinical bert (huang et al., 2019) to train on 4.4 million radiology reports. these models show high performance in ner, re, qa, abnormal classification and summarisation. in contrast, we aim to train with knowledge graph pretraining objectives that might help learn with much smaller dataset with 545k reports, and achieve comparable results for ner, re and abnormals classification. other biomedical pretraining objectives. en-titybert (lin et al., 2021) used entity masking pretraining objective to infuse domain knowledge. this pretraining objective can mask context in radiology reports, and hence we improved on this by using radlex. section segmentation is another radiology-specific pretraining objective proposed in biradsbert (kuling et al., 2021) which we have adapted to our work radling-ss. we plan to use fewer sections in keeping with our training dataset structure and test the architecture for our training data. biomedical knowledge graph infusion. kebi-olm (yuan et al., 2021) applies text-only encoding layer to aggregate entity representation. they use mlm, entity detection and entity linking as pretraining objectives and surpassed state-of-theart in biomedical ner and re. biokglm(fei et al., 2021) has compiled large biomedical knowledge graph as well as introduced a separate posttraining procedure between pretraining and finetuning where they experimented several strategies using knowledge embedding algorithms to inject domain knowledge. in contrast, our work radling attempts to use existing radiology domain-specific knowledge graph radlex and pretraining objectives that are better suited to the structure of training data. ## methods our methods consists of preprocessing, tokenization and continuous pretraining. in addition, we have outlined the downstream tasks we have used to evaluate radling models. an overview of our approach is presented in figure 2 . ## conclusion in this work we have explored a cost-effective method to train a high performing radiology plm, radling with a small dataset. radling models took 2 days to train on tesla v100 sxm2 machines with 8 gpus and 16 gb memory per gpu, which using larger models like electra-large required 5 days. we developed a knowledge-aware masking strategy to use radlex to infuse context into radiology plms to train radling-kg. this led us to the following observations. first, electra architecture, without any special pretraining objectives, is able to produce good results with most of the downstream tasks. in a task like relationship extraction, it even outperforms radgraph radiologist benchmarks. second, radling-kg is the best performing radling variant, and outperforms the downstream task benchmarks in all the tasks except qa. third, domain specific vocabulary is helpful in better performance of models. in addition, in tasks that use cross attention like vision-language tasks or explainable ai, having unfragmented radiology tokens is helpful. for example, bert fragments biomedical terms like 'thalamus' into 'tha', '##lam', '##us', thereby losing the domainspecific meaning, whereas in our work, due to domain specific tokenization, the word 'thalamus' is retained. fourth, infusion of radlex information counteracts electra limitations in qa dataset. finally, we have tested the model on proprietary ner datasets and radling-kg has yielded 0.92-0.93 macro f1 on less represented anatomies like neck, while for the highly represented anatomies, f1 is as high as 0.98. this actually shows the potential of using radlex in radiology pretraining. thus, in a real world setting with high imbalances in datasets, radling-kg is more robust. in future we would like to explore more ways to infuse knowledge by (a) using text description of context like @xcite , (b) retrieving context from biomedical knowledge graphs like snomed foot_4 and umls, and (c) more robust knowledge embedding methods. we would like to experiment with larger datasets and models, and work with more downstream radiology applications.