  Deep neural networks have been well-known for their superb performance in
handling various machine learning and artificial intelligence tasks. However,
due to their over-parameterized black-box nature, it is often difficult to
understand the prediction results of deep models. In recent years, many
interpretation tools have been proposed to explain or reveal the ways that deep
models make decisions. In this paper, we review this line of research and try
to make a comprehensive survey. Specifically, we introduce and clarify two
basic concepts-interpretations and interpretability-that people usually get
confused. First of all, to address the research efforts in interpretations, we
elaborate the design of several recent interpretation algorithms, from
different perspectives, through proposing a new taxonomy. Then, to understand
the results of interpretation, we also survey the performance metrics for
evaluating interpretation algorithms. Further, we summarize the existing work
in evaluating models' interpretability using "trustworthy" interpretation
algorithms. Finally, we review and discuss the connections between deep models'
interpretations and other factors, such as adversarial robustness and data
augmentations, and we introduce several open-source libraries for
interpretation algorithms and evaluation approaches.
