  Human interpretability of deep neural networks' decisions is crucial,
especially in domains where these directly affect human lives. Counterfactual
explanations of already trained neural networks can be generated by perturbing
input features and attributing importance according to the change in the
classifier's outcome after perturbation. Perturbation can be done by replacing
features using heuristic or generative in-filling methods. The choice of
in-filling function significantly impacts the number of artifacts, i.e.,
false-positive attributions. Heuristic methods result in false-positive
artifacts because the image after the perturbation is far from the original
data distribution. Generative in-filling methods reduce artifacts by producing
in-filling values that respect the original data distribution. However, current
generative in-filling methods may also increase false-negatives due to the high
correlation of in-filling values with the original data. In this paper, we
propose to alleviate this by generating in-fillings with the
statistically-grounded Knockoffs framework, which was developed by Barber and
Cand\`es in 2015 as a tool for variable selection with controllable false
discovery rate. Knockoffs are statistically null-variables as decorrelated as
possible from the original data, which can be swapped with the originals
without changing the underlying data distribution. A comparison of different
in-filling methods indicates that in-filling with knockoffs can reveal
explanations in a more causal sense while still maintaining the compactness of
the explanations.
