  Saliency methods are widely used to interpret neural network predictions, but
different variants of saliency methods often disagree even on the
interpretations of the same prediction made by the same model. In these cases,
how do we identify when are these interpretations trustworthy enough to be used
in analyses? To address this question, we conduct a comprehensive and
quantitative evaluation of saliency methods on a fundamental category of NLP
models: neural language models. We evaluate the quality of prediction
interpretations from two perspectives that each represents a desirable property
of these interpretations: plausibility and faithfulness. Our evaluation is
conducted on four different datasets constructed from the existing human
annotation of syntactic and semantic agreements, on both sentence-level and
document-level. Through our evaluation, we identified various ways saliency
methods could yield interpretations of low quality. We recommend that future
work deploying such methods to neural language models should carefully validate
their interpretations before drawing insights.
