Multitask and Multimodal Neural Network Model for Interpretable Analysis of Xray Images The quality and interpretability of the stateoftheart methods for automatic analysis of chest Xray images is still not sufficient. We address this problem by presenting a model that combines the analysis of frontal chest Xray scans with structured patient information contained within radiology records. The proposed model generates a short textual summary with essential information on the found pathologies along with their location and severity; and the 2D heatmaps localizing each pathology on the original Xray images. We test the proposed model on the MIMICCXR dataset. It achieves the stateoftheart performance for image labelling and captioning (78.5% of correctly generated sentences) and defeats other similar solutions that dismiss the additional patient data (by 5.2% of correctly generated sentences). We also propose an automatic approach to label mining that leverages multimodal data: the Xray images, related textual reports, patientsu0027 age and sex.

Computer Assisted Reading of Chest Radiographs Chest radiographs or Xray images are a common diagnostic tool to identify different thoracic diseases and other abnormal cardiopulmonary conditions. The advancements of artificial intelligence paves the way to machine learning based computerassisted systems that can support the radiologists in disease diagnosis and report generation from chest radiographs. In this work we report an implementation of a deeplearning based framework to interpret the disease signature from chest Xrays. The model was trained on a large dataset consisting of both frontal and lateral Xray images of the chest with multiple thoracic disease labels. We report a mean area under ROC curve (AUC) of 0.86, with the AUC of individual diseases in the range of 0.76 to 0.93. We also generated diseaselevel colormaps to visually present the Xray image region most indicative of the disease.