Generating Mammography Reports from Multi-view Mammograms with BERT (2021)

Abstract: Writing mammography reports can be error-prone and time-consuming for radiologists. In this paper we propose a method to generate mammography reports given four images, corresponding to the four views used in screening mammography. To the best of our knowledge our work represents the first attempt to generate the mammography report using deep-learning. We propose an encoder-decoder model that includes an EfficientNet-based encoder and a Transformer-based decoder. We demonstrate that the Transformer-based attention mechanism can combine visual and semantic information to localize salient regions on the input mammograms and generate a visually interpretable report. The conducted experiments, including an evaluation by a certified radiologist, show the effectiveness of the proposed method.

## introduction breast cancer represents a global healthcare @xcite . increasing numbers of new cases and deaths are observed in both developed and less developed countries, only partially attributable to the increasing population age. serial screening with mammography is the most effective method to detect early stage disease and decrease mortality. the goal of screening is to detect breast cancers when still curable to decrease breast cancer-specific mortality @xcite . the european society of breast imaging (eusobi) together with 30 national breast radiology bodies recommend that only qualified radiologists should be involved in screening programs. @xcite . conclusion: x-ray -signs of moderately expressed fibro-cysticchanges with a predominance of the glandularfibrous component. birads category 2 (benign changes)... as the amount of organized breast screening programs grows across the world, the burden on radiologists increases with it. in national screening programs such as in holland or sweden, radiologists may need to read 100 radiology images per hour @xcite . with a growing number of screening programs, we need more trained radiologists and new technologies that can make their workflow more effective. since one of the most time consuming procedures in radiology is writing medical-imaging reports, we explore the potential for deep-learning to automatically generate diagnostic reports of screening mammograms. the rapid evolution of deep learning and artificial intelligence technologies enables them to be used as a strong tool for providing clinical decision-making support to the medical community. while many problems in the area of medical imaging and text analysis have been addressed effectively, there is no known approach to generating clinical reports for mammography studies. there are various reasons for this, such as the requirements regarding the accuracy, completeness and diagnostic relevance of the clinical information contained in the report. in this article, we present a framework (figure 1 ) that takes mammograms as an input, automatically generates mammography reports, and visualizes the attention of the model to provide the interpretability of the process. we use an encoder-decoder architecture, where the encoder extracts visual features and the decoder generates reports. we adopt a convolutional neural network, specifically efficientnet (m tan, 2019), to extract visual features of the four images, corresponding to the four views used in screening mammography. for language modeling, we utilize bert @xcite , inserting an additional attention sub-layer to perform multi-head attention over the regional feature embeddings produced by the encoder. we modify the transformerbased @xcite attention mechanism such that it attends to the visual information on four mammography views and previously generated words. we use the attention scores to build visually interpretable image-text attention mappings. in addition to that, we conduct a series of indepth quantitative and qualitative experiments with the help of an experienced radiologist to demonstrate the clinical validity of our approach. we compare the predictions of our models with the ground truth to understand where the models make mistakes and demonstrate that our best model successfully describes different parts of the breast, and detects pathological regions and abnormalities. we evaluate the image-text attention mappings to demonstrate the interpretability of our model. as far as we are aware, our work represents the first attempt to generate the mammography report using deep-learning. to summarize, we make the following contributions in this paper: ## related work the task of image captioning is creating a model that given a previously unseen query image generates a caption that is both grammatically and semantically correct. the main approaches to image captioning are retrieval-based, template-based and novel caption generation. in retrieval-based methods @xcite , @xcite candidate captions for query images are selected from a pool of existing captions based on some measure of similarity. the downside of this approach is the inability to generate novel image-specific captions. in template-based methods @xcite , @xcite , @xcite image captions are generated by filling the blanks in fixed templates. these methods can generate grammatically and semantically correct novel captions not present in the training set but cannot generate variable-length captions. novel caption generation methods @xcite , @xcite , @xcite use a representation of the query image as an input for a language model responsible for generating the captions. this approach follows the encoder-decoder architecture first applied to machine translation tasks @xcite . to generate an image caption, a representation of the image must first be constructed either via generating handcrafted features or extracting such features automatically, for example using deep neural networks. examples of hand-crafted features are local binary patterns @xcite , scaleinvariant keypoints @xcite , or histograms of oriented gradients @xcite . automatic feature extraction from images is commonly used by applying convolutional neural networks (cnn) @xcite to the query image. these features may be further enhanced, for example by using a spatial transformer @xcite . a sub-field of image captioning is diagnostic captioning (dc). diagnostic captioning is automatic generation of diagnostic text based on a set of medical images of a patient. dc systems can increase the speed of producing a report for experienced physicians and decrease the number of diagnostic errors for inexperienced doctors (for a recent survey on dc methods see @xcite ). the majority of the work in dc is done using encoder-decoder architecture. in addition to evaluation of grammatical and semantical correctness of captions, which is commonly assessed by calculating lexical overlap between generated captions and ground truth @xcite , dc quality can be assessed by clinical correctness by conducting clinical experiments with physicians evaluating the generated reports @xcite , @xcite . language models commonly used in dc usually apply recurrent neural networks (rnn) such as lstm @xcite , see @xcite , with works using transformer-based models beginning to appear @xcite . a common approach in dc is the use of 'visual attention' that allows the decoder to focus on particular areas of input images when generating the captions @xcite , @xcite . such mechanisms also can be used to highlight the regions of interest on the input images adding to the interpretability of the models @xcite . ## data the dataset is based on data from a breast screening program in one of the russian regions. the dataset includes about 25k screening mammography studies with clinical reports. all exams include four standard mammography views: r-cc (right craniocaudal), l-cc (left craniocaudal), r-mlo (right mediolateral oblique), l-mlo (left mediolateral oblique), with image height and width of 4644 by 3510 pixels respectively. each study contains a brief text conclusion, clinical report and bi-rads class. mammography reports are written in russian, examples in this article are translated into english. on average, the mammography report contains 55 words. all personally identifiable information has been deleted by the clinics. we split the dataset into the training, validation and test subsets in the proportion of 91%, 4% and 5% respectively (having 22463, 934 and 1229 cases in each subset). the splits are the same for encoder pretraining and for the text generation model. ## method we start with describing the formal definition of the task. given four mammogram images s we try to generate a sequence of words y that represents the mammography report: the model we introduce is fundamentally an encoder-decoder. the encoder receives the set of projections as input and extracts the set of visual features using a convolutional neural network. next, the transformer-based decoder generates the complete mammography report given the set visual features of the images. ## experiments a series of retrospective data experiments were carried out to evaluate the performance of the developed models. first, we measure the performance of our models with the commonly used natural language generation metrics (nlg), including cider @xcite , meteor @xcite , rouge-l @xcite , and bleu @xcite . we compare four model variants with a random baseline, where the predicted report is a real report for a different patient. then, we evaluate the text reports generated by our model with the help of an experienced radiologist, both quantitatively and qualitatively. we provide a comprehensive description of the experimental procedure together with the obtained results in this and the following section. ## conclusion in this paper we present a first-of-its-kind framework for generating mammography reports given four mammography views using deep-learning. our model utilizes pretrained models including ef-ficientnet for visual extraction and bert for report generation. we demostrate that the transformerbased attention mechanism that simultaneously attends to four mammography views and text from the report significantly improves the performance. our method provides a novel perspective for breast screening: generating mammography reports and providing image-text attention mappings, which makes the automatic breast screening process semantically and visually interpretable. the validity of our approach is confirmed by the corresponding doctor evaluation. in the conducted qualitative analysis we demonstrate that our best model successfully detects pathological regions, and describes abnormalities and parts of the breast.