  Explainable AI (XAI) methods focus on explaining what a neural network has
learned - in other words, identifying the features that are the most
influential to the prediction. In this paper, we call them "distinguishing
features". However, whether a human can make sense of the generated explanation
also depends on the perceptibility of these features to humans. To make sure an
explanation is human-understandable, we argue that the capabilities of humans,
constrained by the Human Visual System (HVS) and psychophysics, need to be
taken into account. We propose the {\em human perceptibility principle for
XAI}, stating that, to generate human-understandable explanations, neural
networks should be steered towards focusing on human-understandable cues during
training. We conduct a case study regarding the classification of real vs. fake
face images, where many of the distinguishing features picked up by standard
neural networks turn out not to be perceptible to humans. By applying the
proposed principle, a neural network with human-understandable explanations is
trained which, in a user study, is shown to better align with human intuition.
This is likely to make the AI more trustworthy and opens the door to humans
learning from machines. In the case study, we specifically investigate and
analyze the behaviour of the human-imperceptible high spatial frequency
features in neural networks and XAI methods.
