E - V ar M : Enhanced Variational Word Masks to Improve the Interpretability of Text Classification Models (2022)

Abstract: Enhancing the interpretability of text classification models can help increase the reliability of these models in real-world applications. Currently, most researchers focus on extracting task-specific words from inputs to improve the interpretability of the model. The competitive approaches exploit the Variational Information Bottleneck (VIB) to improve the performance of word masking at the word embedding layer to obtain task-specific words. However, these approaches ignore the multi-level semantics of the text, which can impair the interpretability of the model, and do not consider the risk of representation overlap caused by the VIB, which can impair the classification performance. In this paper, we propose an enhanced variational word masks approach, named E-VarM, to solve these two issues effectively. The E-VarM combines multi-level semantics from all hidden layers of the model to mask out task-irrelevant words and uses contrastive learning to readjust the distances between representations. Empirical studies on ten benchmark text classification datasets demonstrate that our approach outperforms the SOTA methods in simultaneously improving the interpretability and accuracy of the model.

## introduction with the widespread adoption of neural networks in text classification tasks @xcite @xcite , the classification models are expected to provide not only highly accurate classification results but also reasonable prediction rationales @xcite . these * corresponding author prediction rationales, which are short yet informative parts of the input for classification predictions @xcite , manifest the interpretability of the model. the better the interpretability of the model, the more reasonable rationales the model provides @xcite . therefore, improving the interpretability of the model helps to boost the reliability of these classifiers in real-world applications @xcite . to improve the interpretability of the classifiers, many methods have been proposed @xcite . some studies rely on additional inputs at the training time, such as pre-defined information and human annotations @xcite , all of which involve high human costs. other works resort to assigning a binary bernoulli variable to each input word with promising results, among which the competitive approaches are the sparse-vib @xcite and vmask (chen and ji, 2020) models . they all employ the variational information bottleneck (vib) @xcite to train stochastic masks to automatically learn task-specific words, namely prediction rationales, for prediction and interpretability simultaneously. however, these methods only utilize the information from the word embedding layer , which holds little task-specific information @xcite , ignoring the multi-level information of the text, such as syntactic and semantic information. that is, this initial layer results in poor interpretability performance and reaches low accuracy on related semantic tasks . additionally, these approaches do not consider the risk of representation overlap arising from vib @xcite , resulting in the model failing to learn discriminative class representations, thus further impairing the classification performance. to address the above issues, we propose an enhanced variational word masks approach, (named e-varm), which combines multi-level information to learn task-specific words in an unsupervised manner and uses contrastive learning to alleviate the representation overlap problem caused by vib. as a result, both the interpretability and accuracy of the model are improved. specifically, as the data flow from the input to the output layer, each hidden layer of the text encoder can capture different information @xcite . for example, the bert @xcite can encode the captured information to a rich hierarchy of linguistic information @xcite with surface features at the bottom, syntactic features in the middle, and semantic features at the top @xcite @xcite . inspired by the feature extraction mechanism of the text encoder, we fuse multilevel information from all hidden layers of the model to generate a task-specific binary bernoulli distribution to extract reasonable interpretations for various text classification tasks, such as sentiment analysis, syntactic judgments, and semantic inference. furthermore, the vib-based methods rely on the lagrangian factor to perform the compressionprediction trade-off @xcite . a larger lagrangian factor means masking out more task-irrelevant information @xcite @xcite for a special class and obtaining better interpretability. however, along with this larger value is the potential risk of representation overlap or even collapse @xcite @xcite since it may hinder the model from learning the discriminative representation and lead to loss of class information. therefore, we leverage supervised contrastive learning @xcite to adjust the classification representation by pulling in samples from the same class and pushing away samples from different classes, thus mitigating the representations overlap problem. additionally, we resort to taskspecific words to construct diverse positive samples to enhance the efficiency of contrastive learning. in a nutshell, we make the following major contributions: (1) we introduce an enhanced variational word masks method to improve classification performance and interpretability simultane-ously. (2) our method produces token-level interpretations that consider the multi-level information from all model layers and are adaptable to various classification tasks. (3) to the best of our knowledge, our approach is the first attempt to use contrastive learning to alleviate the representation overlap caused by vib. (4) the experimental results on ten benchmark datasets validate the effectiveness of our method. ## conclusion in this paper, we propose e-varm to simultaneously boost the model's interpretability and accuracy. e-varm combines multi-level information for task-specific words selection, which can adjust the decision basis of the model, and improve the model's interpretability. further, e-varm adopts contrastive learning for representation optimization to mitigate the risk of representations overlap, enhancing the model's classification performance. experimental results on ten benchmark datasets demonstrate the effectiveness of e-varm.