Efficient Nearest Neighbor Emotion Classification with BERT -whitening (2022)

Abstract: Retrieval-based methods have been proven effective in many NLP tasks. Previous methods use representations from the pre-trained model for similarity search directly. However, the sentence representations from the pre-trained model like BERT perform poorly in retrieving semantically similar sentences, resulting in poor performance of the retrieval-based methods. In this paper, we propose kNN-EC, a simple and efficient non-parametric emotion classification (EC) method using nearest neighbor retrieval. We use BERT-whitening to get better sentence semantics, ensuring that nearest neighbor retrieval works. Meanwhile, BERT-whitening can also reduce memory storage of datastore and accelerate retrieval speed, solving the efficiency problem of the previous methods. kNN-EC average improves the pre-trained model by 1.17 F1-macro on two emotion classification datasets.

## introduction retrieval-based methods enhance model performance by extracting textual information related to the input from large training data. the model interpolates the original model distribution and retrieved results' distribution as the final distribution for output. retrieval-based methods can often improve the performance as the model is exposed to related knowledge not present in the input @xcite . retrieval-based methods have been successfully applied to many tasks such as language modeling @xcite , machine translation @xcite @xcite , question answering @xcite , and computer vision @xcite . retrieval-based methods are expressive, adaptable, and interpretable. here, we propose knn-ec, a simple and efficient non-parametric emotion * corresponding author classification (ec) method using nearest neighbor retrieval. knn-ec can be added to any pre-trained emotion classification model without further training. pre-trained models such as bert @xcite , roberta @xcite , xlnet @xcite have been widely proven effective for emotion classification, and they obtain state-of-the-art results on many emotion classification tasks. however, the sentence representations from the pre-trained model like bert perform poorly in retrieving semantically similar sentences @xcite . previous works found that pre-trained models always induced a non-smooth anisotropic semantic space of sentences, which harmed its performance of semantic similarity. to address this issue, @xcite designed a novel way to mitigate the degeneration problem by regularizing the word embedding matrix. @xcite proposed to transform the bert sentence embedding distribution into a smooth and isotropic gaussian distribution through normalizing flow, an invertible function parameterized by the neural network. because the above methods require additional parameters added to the neural network, this is not suitable to be added to knn-ec to enhance the nearest neighbor retrieval. here, we use bert-whitening @xcite , which utilizes a simple linear transformation to enhance the isotropy of sentence representations and achieve competitive results. the details of bert-whitening are described in section 2.2. we conduct experiments on two emotion classification datasets. knn-ec can achieve 1.21-1.65/0.28-1.44 f1-macro scores improvements over pre-trained models on goemotions/isear. meanwhile, by adding the bert-whitening, our approach can further boost the model performance, optimize memory storage and accelerate retrieval speed. the main contributions of this paper are summawe use whitening transformation to enhance the isotropy of sentence representations and dimension reduction to optimize memory storage and accelerate retrieval speed. in inference, we use the whitening transformation on the test sentence's representation to retrieve the k nearest neighbors from the datastore. we interpolate the model and knn distributions with a hyper-parameter λ as the final distribution. ## background in this section, we will briefly introduce the background of knn-mt @xcite and bert-whitening @xcite . ## conclusion in this paper, we propose a simple and effective method using nearest neighbor retrieval that can be applied to many emotion classification models without further training. meanwhile, we reduce memory storage of datastore and accelerate retrieval speed, significantly decreasing the inference overhead and erasing the limitation of non-parametric knn-ec in practical applications. experimental results show that our proposed model achieves significant improvement.