  We formulate a causal extension to the recently introduced paradigm of
instance-wise feature selection to explain black-box visual classifiers. Our
method selects a subset of input features that has the greatest causal effect
on the models output. We quantify the causal influence of a subset of features
by the Relative Entropy Distance measure. Under certain assumptions this is
equivalent to the conditional mutual information between the selected subset
and the output variable. The resulting causal selections are sparser and cover
salient objects in the scene. We show the efficacy of our approach on multiple
vision datasets by measuring the post-hoc accuracy and Average Causal Effect of
selected features on the models output.
