  There has been a growing interest in model-agnostic methods that can make
deep learning models more transparent and explainable to a user. Some
researchers recently argued that for a machine to achieve a certain degree of
human-level explainability, this machine needs to provide human causally
understandable explanations, also known as causability. A specific class of
algorithms that have the potential to provide causability are counterfactuals.
This paper presents an in-depth systematic review of the diverse existing body
of literature on counterfactuals and causability for explainable artificial
intelligence. We performed an LDA topic modelling analysis under a PRISMA
framework to find the most relevant literature articles. This analysis resulted
in a novel taxonomy that considers the grounding theories of the surveyed
algorithms, together with their underlying properties and applications in
real-world data. This research suggests that current model-agnostic
counterfactual algorithms for explainable AI are not grounded on a causal
theoretical formalism and, consequently, cannot promote causability to a human
decision-maker. Our findings suggest that the explanations derived from major
algorithms in the literature provide spurious correlations rather than
cause/effects relationships, leading to sub-optimal, erroneous or even biased
explanations. This paper also advances the literature with new directions and
challenges on promoting causability in model-agnostic approaches for
explainable artificial intelligence.
