Text Classification with Negative Supervision (2020)

Abstract: Advanced pre-trained models for text representation have achieved state-of-the-art performance on various text classification tasks. However, the discrepancy between the semantic similarity of texts and labelling standards affects classifiers, i.e. leading to lower performance in cases where classifiers should assign different labels to semantically similar texts. To address this problem, we propose a simple multitask learning model that uses negative supervision. Specifically, our model encourages texts with different labels to have distinct representations. Comprehensive experiments show that our model outperforms the state-of-the-art pre-trained model on both single- and multi-label classifications, sentence and document classifications, and classifications in three different languages.

## introduction text classification generally consists of two processes: an encoder that converts texts to numerical representations and a classifier that estimates hidden relations between the representations and class labels. the text representations are generated using n -gram statistics @xcite , word embeddings @xcite , convolutional neural networks @xcite @xcite , and recurrent neural networks @xcite . recently, powerful pre-trained models for text representations, e.g. bidirectional encoder representations from transformers (bert) @xcite , have shown stateof-the-art performance on text classification tasks using only the simple classifier of a fully connected layer. however, a problem occurs when a classification task is adversarial to text encoders. encoders aim to represent the meanings of texts; hence, seman-sentence label bert a cold is a legit disease. to address this problem, we propose utilizing negative examples, i.e. texts with different labels, to enable negative supervision of the encoder for generating distinct representations for each class. in this study, we design a simple multitask learning model that trains two models simultaneously with a shared text encoder. the first model learns an ordinary classification task (herein referred to as the main task). meanwhile, the second model encourages representations with different class labels to be distinct (herein referred to as the auxiliary encoder i caught a cold. ## multitask learning framework figure 1 shows an overview of our multitask learning framework that consists of main and auxiliary tasks. herein, we refer to the model for the main task as a classifier and the model for the auxiliary task as a discriminator. the overall loss function l sums the loss of the main task l m and that of the auxiliary task l a : the classifier and discriminator share and jointly optimize the text encoder, which encodes an input text into a d-dimensional vector @xmath0. in this paper, we use the terms of text and representation interchangeably when the intention is obvious from the context. ## experiments we conducted a comprehensive evaluation to investigate the performance of our model in terms of (a) single-and multi-label classifications, (b) sentence-and document-level classification, and (c) different languages. we collected the standard evaluation datasets from heterogeneous sources, as summarised in ## related work multitask learning has been employed to improve the performance of text classification @xcite . previous studies aimed to improve multiple tasks; hence, they required multiple sets of annotated datasets. in contrast, our method does not require any extra labelled datasets and is easily applicable to various classification tasks. the methods proposed in @xcite and @xcite improved the bert classification performance by further training the pre-trained model using natural language inference and paraphrase recognition. similar to multitask learning, both methods require an additional largescale labelled dataset. furthermore, these previous studies revealed that the similarity of tasks in training affects the models' final performance @xcite . our method achieved consistent improvements across tasks, indicating its wider applicability. ## conclusion in this paper, we proposed a simple multitask learning model that uses negative supervision to generate distinct representations for texts with different labels. comprehensive evaluation empirically confirmed that our model consistently outperformed bert and han models on single-and multi-label classifications, sentence and document classifications, and classifications in different languages. our multitask learning model provides a general framework that is easily applicable to existing text classification models. in future work, we will examine semantic relations between class labels in the auxiliary task. moreover, we will adapt our model to text generation tasks. we expect that our model will encourage a generation model to generate texts with different labels, such as styles, have distinct representations, which will result in class specific expressions.