Towards Visual Dialog for Radiology (2020)

Abstract: Current research in machine learning for radiology is focused mostly on images. There exists limited work in investigating intelligent interactive systems for radiology. To address this limitation, we introduce a realistic and information-rich task of Visual Dialog in radiology, specific to chest X-ray images. Using MIMIC-CXR, an openly available database of chest X-ray images, we construct both a synthetic and a real-world dataset and provide baseline scores achieved by state-of-the-art models. We show that incorporating medical history of the patient leads to better performance in answering questions as opposed to conventional visual question answering model which looks only at the image. While our experiments show promising results, they indicate that the task is extremely challenging with significant scope for improvement. We make both the datasets (synthetic and gold standard) and the associated code publicly available to the research community.

## introduction answering questions about an image is a complex multi-modal task demonstrating an important capability of artificial intelligence. a well-defined task evaluating such capabilities is visual question answering (vqa) @xcite where a system answers free-form questions reasoning about an image. vqa demands careful understanding of elements in an image along with intricacies of the language used in framing a question about it. visual dialog (visdial) @xcite is an extension to the vqa problem, where a system is required to engage in a dialog about the image. this adds significant complexity to vqa where a system should now be able to associate the question in the image, and reason * equal contribution, work done at ibm research over additional information gathered from previous question answers in the dialog. although limited work exploring vqa in radiology exists, visdial in radiology remains an unexplored problem. with the healthcare setting increasingly requiring efficiency, evaluation of physicians is now based on both the quality and the timeliness of patient care. clinicians often depend on official reports of imaging exam findings from radiologists to determine the appropriate next step. however, radiologists generally have a long queue of imaging studies to interpret and report, causing subsequent delay in patient care @xcite . furthermore, it is common practice for clinicians to call radiologists asking follow-up questions on the official reporting, leading to further inefficiencies and disruptions in the workflow @xcite . visual dialog is a useful imaging adjunct that can help expedite patient care. it can potentially answer a physician's questions regarding official interpretations without interrupting the radiologist's workflow, allowing the radiologist to concentrate their efforts on interpreting more studies in a timely manner. additionally, visual dialog could provide clinicians with a preliminary radiology exam interpretation prior to receiving the formal dictation from the radiologist. clinicians could use the information to start planning patient care and decrease the time from the completion of the radiology exam to subsequent medical management @xcite . in this paper, we address these gaps and make the following contributions: 1) we introduce construction of radvisdial -the first publicly available dataset for visual dialog in radiology, derived from the mimic-cxr @xcite dataset, 2) we compare several state-of-the-art models for vqa and visdial applied to these images, and 3) we conduct a comprehensive set of experiments highlighting different challenges of the problem and propose solutions to overcome them. ## related work most of the large publicly available datasets @xcite for radiology consist of images associated with a limited amount of structured information. for example, @xcite make images available along with the output of a text extraction module that produces labels for 13 abnormalities in a chest x-ray. of note recently, the task of generating reports from radiology images has become popular in the research community @xcite . two recent shared tasks at image-clef explored the vqa problem with radiology images @xcite . @xcite al. (2018) 2018 ) use clinical images, the sizes of these datasets are limited. they are a mix of several modalities including 2d modalities such as x-rays, and 3d modalities such as ultrasound, mri, and ct scans. they also cover several anatomic locations from the brain to the limbs. this makes a multimodal task with such images overly challenging, with shared task participants developing separate models @xcite @xcite to first address these subtasks (such as modality detection) before actually solving the problem of vqa. we address these limitations and build up on mimic-cxr @xcite the largest publicly available dataset of chest x-rays and corresponding reports. we focus on the problem of visual dialog for a single modality and anatomy in the form of 2d chest x-rays. we restrict the number of questions and generate answers for them automatically which allows us to report results on a large set of images. ## models for our experiments, we selected a set of models designed for image-based question answering tasks. namely, we experimented with three architectures: stacked attention network (san) @xcite , late fusion network (lf) @xcite , and recursive visual attention network (rva) @xcite . following the original visdial study @xcite , we use an encoder-decoder structure with a discriminative decoder for each of the models. below we give an overview of all the three algorithms. ## experiments this section presents our down-sampling strategy, gives details about conducted ablation studies, and describes experiments with various representations of images and texts. ## results we report macro-averaged f1-scores achieved on the same unbalanced validation set for each of the experiments. when experimenting with different configurations of the same model, we also break down the aggregate score to the f1 scores for individual answer options. ## comparison with the gold-standard data to complement our experiments with the silver data and investigate the applicability of the trained models to real-world scenarios, we also collected a set of gold standard data which consisted of two expert radiologists having a dialog about a particular chest x-ray. these x-ray images were randomly sampled pa views from the test our data. in this section, we present the data collection workflow, outline the associated challenges, compare the resulting data-set with the silver-standard, and report the performance of trained models. ## discussion our main finding is that the introduced task of visual dialog in radiology presents a lot of challenges from the machine learning perspective, including a skewed distribution of classes and a required ability to reason over both visual and textual input data. the best of our baseline models achieved 0.34 macro-averaged f1-score, indicating on a significant scope for potential improvements. our comparison of gold and silver standard data shows some trends are in line with medical doctors' strategies in medical history taking, starting with broader, general questions and then narrowing the scope of their questions to more specific findings @xcite . despite the difficulty and the practical usefulness of the task, it is important to list the limitations of our study. the questions were limited to presence of 13 abnormalities extracted by chexpert and the answers were limited to 4 options. the studies used in this work (from mimic-cxr) originate from a single tertiary hospital in the united states. moreover, they correspond to a specific group of patients, namely those admitted to the emergency department (ed) from 2012 to 2014. therefore, the data and hence the model reflect multiple realworld biases. it should also be noted that chest x-rays are mostly used for screening than diagnostic purposes. a radiology image is only one of the many data points (e.g. labs, demographics, medications) used while making a diagnosis. therefore, although predicting presence of abnormalities (e.g. pneumonia) based on brief knowledge of the patient's medical history and the chest x-ray might be a good exercise and a promising first step in evaluating machine learning models, it is clinically limited. there are plenty of directions for future work that we intend to pursue. to make the synthetic data more realistic and expressive, both questions and answers should be diversified with the help of clinicians' expertise and external knowledge bases such as umls @xcite . we plan to enrich the data with more question types, addressing, for example, the location or the size of a given lung abnormality. we plan to collect more real life dialog between radiologists and augment the two datasets to get a richer set of more expressive dia-log. we anticipate that bridging the gap between the silver-and the gold-standard data in terms of natural language formulations would significantly reduce the difference in model performance for the two setups. another direction is to develop a strategy to manage the uncertain labels such as 'maybe' and 'not in report' to make the dataset more balanced.