  In recent years, there has been an explosion of AI research on counterfactual
explanations as a solution to the problem of eXplainable AI (XAI). These
explanations seem to offer technical, psychological and legal benefits over
other explanation techniques. We survey 100 distinct counterfactual explanation
methods reported in the literature. This survey addresses the extent to which
these methods have been adequately evaluated, both psychologically and
computationally, and quantifies the shortfalls occurring. For instance, only
21% of these methods have been user tested. Five key deficits in the evaluation
of these methods are detailed and a roadmap, with standardised benchmark
evaluations, is proposed to resolve the issues arising; issues, that currently
effectively block scientific progress in this field.
