Task-Aware Representation of Sentences for Generic Text Classification (2020)

Abstract: State-of-the-art approaches for text classification leverage a transformer architecture with a linear layer on top that outputs a class distribution for a given prediction problem. While effective, this approach suffers from conceptual limitations that affect its utility in few-shot or zero-shot transfer learning scenarios. First, the number of classes to predict needs to be pre-defined. In a transfer learning setting, in which new classes are added to an already trained classifier, all information contained in a linear layer is therefore discarded, and a new layer is trained from scratch. Second, this approach only learns the semantics of classes implicitly from training examples, as opposed to leveraging the explicit semantic information provided by the natural language names of the classes. For instance, a classifier trained to predict the topics of news articles might have classes like “business” or “sports” that themselves carry semantic information. Extending a classifier to predict a new class named “politics” with only a handful of training examples would benefit from both leveraging the semantic information in the name of a new class and using the information contained in the already trained linear layer. This paper presents a novel formulation of text classification that addresses these limitations. It imbues the notion of the task at hand into the transformer model itself by factorizing arbitrary classification problems into a generic binary classification problem. We present experiments in few-shot and zero-shot transfer learning that show that our approach significantly outperforms previous approaches on small training data and can even learn to predict new classes with no training examples at all. The implementation of our model is publicly available at: https://github.com/flairNLP/flair .

## introduction text classification is the task of predicting one or multiple class labels for a given text. it is used in a large number of applications such as spam filtering @xcite , sentiment analysis @xcite , intent detection @xcite or news topic classification @xcite . the current state-of-the-art approach to text classification leverages a bert-style transformer architecture @xcite @xcite with a linear classifier layer on top. the transformer is pre-trained on language modelling task, whereas the classifier is randomly initialized. the entire model is then fine-tuned using training examples for all classes, so that the classifier outputs a distribution over all class labels in the prediction problem. this approach is shown to work well, especially if for each class a reasonable amount of training examples is available. few-shot transfer learning. real world text classification scenarios are often characterized by a lack of annotated corpora and rapidly changing information needs @xcite , motivating research into methods that allow us to train text classifiers for new classes with only a handful of training examples @xcite . in such cases, a standard approach is to transfer knowledge from an existing model for classification task x to initialize the weights for a model for the new classification task y. here, there are two options: if task y differs from x significantly, then we figure 1 : traditional text classifier model training on different tasks x, y and z. while the bert-encoder can be transferred between tasks, the final linear layer and activation are task-specific and therefore must be learned separately from scratch for each task. this limits the transfer learning capability. however, this approach to transfer learning disregards or dismisses two sources of information that may be especially useful in few-shot or zero-shot scenarios: information in the pre-trained decoder first, the traditional transfer learning approach will discard the final linear layer that acts as decoder, since the prediction targets might differ i.e., @xmath0. this effectively results in a loss of all information contained in the decoder and requires us to train a new decoder from scratch given very limited training data in a few-shot scenario. worse, this approach cannot be used in a zero-shot scenario at all since here there is no training data to train the decoder. information provided by class labels second, the traditional approach only learns the semantics of classes implicitly from their training examples. this disregards the explicit semantic information provided by the natural language class labels. for instance, a classifier trained to predict the topics of news articles might have class labels like "business" or "sports" that themselves carry semantic information @xcite . if such a classifier were to be extended to predict a new class named "politics" with only a handful of training examples, it may be sensible to leverage the semantics provided by this class label as well. this would extend its theoretical applicability to zero-shot learning since the name of the new class could suffice as input to learn new classifiers, even without training data. with this paper, we present a straightforward but remarkably effective approach to preserve the two above-mentioned sources of information in transfer learning for text classification. the main idea is to imbue the notion of the task itself into the transformer model, by factorizing arbitrary classification problems into a generic binary classification problem. in other words, we replace the task-specific decoder with a generic binary "true/false" decoder. the input to the transformer then consists not only of the text to be classified, but also of the class label (e.g. a semantically meaningful form of textual labels) prepended to the text. we illustrate this in figure 2 . task-aware representations. our proposed approach therefore reformulates the classification problem as a "query" in which a sentence and a potential class label is given to the transformer which makes a pre-diction whether or not this label holds. the cross-attention mechanism in bert then learns to combine the representation of the text and its label. accordingly, we refer to this approach as tars (task-aware representation of sentences). this addresses the two issues mentioned above as the same decoder can now be used across arbitrary tasks (allowing transfer of the full model) and that the information provided by the class label itself is interpreted by the transformer model. a conceptual advantage of this approach is that it can return predictions even for classes for which no training data exists: it simply needs to prepend the textual label of the new class to text and evaluate the result of the "true/false" decoder. our contributions are therefore as follows: we find that tars significantly outperforms traditional transfer learning in regimes with little to no training data. we also observe surprisingly powerful zero-shot learning abilities, indicating that tars indeed learns to interpret the semantics of the label name and is thus able to correctly predict labels for classes without any training data at all. based on these results, we conclude tars to be a conceptually simple and effective approach for few-shot and zero-shot transfer learning in text classification. ## method we formulate text classification as a universal binary classification problem and use cross-attention to capture the modified objective. we then illustrate how we train, predict and transfer using tars. ## experiments we conduct an experimental evaluation of tars to address the following questions: (1) how well is tars able to transfer to new classification tasks with little training data? (2) how does semantic distance between source and target task affect the transfer learning abilities of tars? (3) and what are the zeroshot capabilities of tars? datasets and labels. to this end, we experiment with 5 widely-used text classification datasets that span different textual domains and classification tasks: two datasets for the task of topic detection, namely agnews @xcite , a corpus of news articles classified into 4 topics, and dbpedia @xcite , a corpus of 14 entity topics. one dataset in two variants for the task of classifying question types @xcite , namely trec-6 with 6 coarse-grained and trec-50 with 50 fine-grained question types. and two corpora for 5-class sentiment analysis, namely amazon-full @xcite for product reviews and yelp-full @xcite for restaurant reviews. an overview of all 6 datasets is given in ## conclusion in this paper we addressed key shortcomings of the existing transfer learning mechanisms for text classification and proposed a novel formulation that transforms it into a generic binary classification problem. the proposed tars architecture captures the similarity between an input text and the task label to perform text classification. we performed an extensive set of transfer learning tasks from multiple datasets from different domains, including topic classification and sentiment analysis. we showed that tars is capable of making zero-shot predictions in multiple text classification tasks, and adapts to a new domain faster than competitive baseline models in few-shot learning settings. the proposed model also generalizes the text classification task to the extent that a single model can perform well on multiple text classification tasks simultaneously. our efforts in the future are aimed towards optimizing the computational complexity of tars. we assume that the specific architecture used here has plausible alternatives: one could use separate transformers for task label and text parts of the input, and cross-attend between the two with a shallow transformer. this way the computational complexity can be significantly reduced by assuming different degrees of task and text independence. in the limit, one could have a task embedding and a text embedding and perform cross-attention only at the embedding level (i.e. dot-product between task and text embeddings). another way of reducing computational complexity is by exploring architectural choices of transformer encoders, e.g. using complementary lighter transformer encoders, or applying model pruning. these options are orthogonal to our contributions and are subject of active research. exploration of these trade-offs, as well as exploring a broader range of tasks, are planned for future work. since, tars can encapsulate multiple tasks in a single model, it would also be interesting to investigate on the effect of training sequence of different tasks. we encourage the nlp community to utilize our model in other domains with our open source implementation.