  Convolutional neural networks are showing promise in the automatic diagnosis
of thoracic pathologies on chest x-rays. Their black-box nature has sparked
many recent works to explain the prediction via input feature attribution
methods (aka saliency methods). However, input feature attribution methods
merely identify the importance of input regions for the prediction and lack
semantic interpretation of model behavior. In this work, we first identify the
semantics associated with internal units (feature maps) of the network. We
proceed to investigate the following questions; Does a regression model that is
only trained with COVID-19 severity scores implicitly learn visual patterns
associated with thoracic pathologies? Does a network that is trained on weakly
labeled data (e.g. healthy, unhealthy) implicitly learn pathologies? Moreover,
we investigate the effect of pretraining and data imbalance on the
interpretability of learned features. In addition to the analysis, we propose
semantic attribution to semantically explain each prediction. We present our
findings using publicly available chest pathologies (CheXpert, NIH ChestX-ray8)
and COVID-19 datasets (BrixIA, and COVID-19 chest X-ray segmentation dataset).
The Code is publicly available.
