  Several studies have been carried out on revealing linguistic features
captured by BERT. This is usually achieved by training a diagnostic classifier
on the representations obtained from different layers of BERT. The subsequent
classification accuracy is then interpreted as the ability of the model in
encoding the corresponding linguistic property. Despite providing insights,
these studies have left out the potential role of token representations. In
this paper, we provide an analysis on the representation space of BERT in
search for distinct and meaningful subspaces that can explain probing results.
Based on a set of probing tasks and with the help of attribution methods we
show that BERT tends to encode meaningful knowledge in specific token
representations (which are often ignored in standard classification setups),
allowing the model to detect syntactic and semantic abnormalities, and to
distinctively separate grammatical number and tense subspaces.
