Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X -rays (2020)

Abstract: Automatic medical image report generation has drawn growing attention due to its potential to alleviate radiologists’ workload. Existing work on report generation often trains encoder-decoder networks to generate complete reports. However, such models are affected by data bias (e.g. label imbalance) and face common issues inherent in text generation models (e.g. repetition). In this work, we focus on reporting abnormal findings on radiology images; instead of training on complete radiology reports, we propose a method to identify abnormal findings from the reports in addition to grouping them with unsupervised clustering and minimal rules. We formulate the task as cross-modal retrieval and propose Conditional Visual-Semantic Embeddings to align images and fine-grained abnormal findings in a joint embedding space. We demonstrate that our method is able to retrieve abnormal findings and outperforms existing generation models on both clinical correctness and text generation metrics.

## introduction understanding abnormal findings on radiographs (e.g. chest x-rays) is a crucial task for radiologists. there has been growing interest in automatic radiology report generation to alleviate the workload of radiologists and improve patient care. following the success of neural network models in imageto-text generation tasks (e.g. image captioning), researchers have trained cnn-rnn encoder-decoder networks to generate reports given radiology images @xcite . although such models are able to generate fluent reports, the generation quality is often limited by biases introduced from training data or the training process. figure 1 shows an example of chest * now at google x-rays (cxrs) and the associated reports from a public dataset @xcite , along with the outputs generated by different models. 1 one issue is that models trained on complete reports tend to generate normal findings as they dominate the dataset @xcite ; another issue is that such generation models struggle to generate long and diverse reports as in other natural language generation (nlg) tasks @xcite . in this work, we focus on reporting abnormal findings on radiology images which are of higher importance to radiologists. to address issues of data bias, we propose a method to identify abnormal findings from existing reports and further use k-means plus minimal mutual exclusivity rules to group these abnormal findings, which reduces the substantial burden of curating templates of abnormal findings. given the fact that radiology reports are highly similar and have a limited vocabulary @xcite , we propose a cross-modal retrieval method to capture relevant abnormal findings from radiology images. our contributions are summarized as: impression: retrocardiac opacity concerning for left lower lobe pneumonia. follow-up radiographs after treatment are recommended to ensure resolution of this finding. findings: heart size is normal. the mediastinal and hilar contours are within normal limits. pulmonary vasculature is not engorged. patchy retrocardiac opacities concerning for left lower lobe pneumonia. right lung is clear. no pleural effusion or pneumothorax is present. clips are seen in the right upper quadrant of the abdomen likely denoting prior cholecystectomy. hier-cnn-rnn (complete): no acute cardiopulmonary process. no evidence of pneumonia. there is no focal consolidation, pleural effusion, or pneumothorax. there is no focal consolidation, pleural effusion, or pneumothorax. the cardiomediastinal silhouette is within normal limits. no acute osseous abnormalities. hier-cnn-rnn (abnormal): lung volumes are low.. low lung volumes. cxr-cvse (abnormal): increased density projecting over the spine which could be due to additional atelectasis; however, pneumonia is also possible.. possible retrocardiac opacity could be prominent vessels but consolidation is not excluded and could represent pneumonia in the appropriate clinical setting. figure 1: example of cxr images (frontal and lateral views) and the associated report. bolded are abnormal findings in the ground-truth and predictions. the cnn-rnn model trained on the complete reports tends to generate normal findings. both cnn-rnn models generate repetitive sentences. 2 related work 2.1 hierarchical encoder-decoder models jing et al. (2017) proposed a co-attention based hierarchical cnn-rnn model that jointly trains two tasks: report generation and medical text indexer (mti) prediction. the model first predicts mti tags and the semantic embeddings of the predictions are fed into the cascaded decoder for generation. similarly, yuan et al. (2019) extracted medical concepts from the cxr reports using semrep 2 as alternatives to mti tags. to address data bias, harzig et al. (2019) proposed a cnn-rnn model with dual word-level decoders: one for abnormal findings and the other for normal findings. it jointly predicts whether the next sentence is a normal or abnormal finding, and uses the corresponding decoder to generate the next sentence. however, it still formulates the task as text generation and has the limitations of such models. ## approach given radiology images i f and i l from the frontal and lateral view, hierarchical cnn-rnn based methods predict complete medical reports @xmath0, s 2 , . . . , s n }, consisting of n sentences. each sentence s i is generated hierarchically: instead of training such generation models, we approach the task as a cross-modal retrieval method. in particular, we propose a model that (1) measures the similarity between images and abnormal find-ings, and (2) identifies fine-grained relevant image regions for each abnormal finding. ## experiments we compare cvse with the state-of-the-art report generation models and simple baseline models to answer two research questions-rq1: does our retrieval-based method outperform generation models? rq2: do the visual-semantic embeddings capture abnormal findings grounded on images? ## conclusions in this paper, we study how to build assistive medical imaging systems that report abnormal findings on the medical images in the form of detailed descriptions. we formulate the problem as a crossmodal retrieval task and apply a metric learningbased method to align visual and semantic features (i.e., image regions and textual descriptions of abnormal findings) without explicit labels. our experiments show that the retrieval-based method outperforms generation-based models by mitigating their weaknesses in generating repetitive sentences and bias toward normal findings. in the future, we will extend our method to other medical image datasets and explore transfer learning.