  BERT, as one of the pretrianed language models, attracts the most attention
in recent years for creating new benchmarks across GLUE tasks via fine-tuning.
One pressing issue is to open up the blackbox and explain the decision makings
of BERT. A number of attribution techniques have been proposed to explain BERT
models, but are often limited to sequence to sequence tasks. In this paper, we
adapt existing attribution methods on explaining decision makings of BERT in
sequence classification tasks. We conduct extensive analyses of four existing
attribution methods by applying them to four different datasets in sentiment
analysis. We compare the reliability and robustness of each method via various
ablation studies. Furthermore, we test whether attribution methods explain
generalized semantics across semantically similar tasks. Our work provides
solid guidance for using attribution methods to explain decision makings of
BERT for downstream classification tasks.
