Probabilistic Ensembles of Zero- and Few-Shot Learning Models for Emotion Classification (2021)

Abstract: Emotion Classification is the task of automatically associating a text with a human emotion. State-of-the-art models are usually learned using annotated corpora or rely on hand-crafted affective lexicons. We present an emotion classification model that does not require a large annotated corpus to be competitive. We experiment with pretrained language models in both a zero-shot and few-shot configuration. We build several of such models and consider them as biased, noisy annotators, whose individual performance is poor. We aggregate the predictions of these models using a Bayesian method originally developed for modelling crowdsourced annotations. Next, we show that the resulting system performs better than the strongest individual model. Finally, we show that when trained on few labelled data, our systems outperform fully-supervised models.

## introduction a large part of natural language processing (nlp) research is focused on building technology to automatically extract information from large collections of texts. however, text contains often not just mere factual information, but also opinions, attitudes and emotions. applications of emotion-aware nlp models range from established tasks such as analysis of product reviews @xcite and development of "emotional" chatbots @xcite to less obvious tasks such as the analysis of developer experience on stack overflow @xcite , author profiling @xcite and the prediction of mental health disorders @xcite . in this work, we focus on the fine-grained emotion classification task @xcite : given a document, an emotion label must be provided. for example, the sentence the angry wolf ate the happy boy could be associated with fear or sadness if the emotion is being modelled from the reader's perspective; alternatively, it could be associated with anger or joy considering the text's perspective. in the emotion classification literature, the targeted emotion perspective is rarely made explicit @xcite , whichin addition to the subjectivity involved in the annotation task -makes it difficult to obtain large amounts of high quality data @xcite . furthermore, with few exceptions @xcite , most of the research has been conducted on english corpora: most of the other languages can be considered low-resourced with respect to affective corpora. in this work, we aim to minimize the amount of annotated data needed to obtain competitive performance in the task of emotion classification. several emotion theories exist, which differ on the emotion inventory and representation type (categorical vs. continuous): in this work we focus on the categorical paradigm, using the mix of different inventories available from the unify emotion dataset @xcite . we describe the use of pretrained language models (plms) for emotion classification in both few-shot and zero-shot scenarios (section 4). few-shot models are supposed to solve a task using only few annotated instances; zeroshot models are supposed to use none. these models have been shown to perform well in different tasks @xcite @xcite . however, in a real unsupervised scenario (i.e., without an evaluation split), their performance is by definition unknown. to mitigate the risks involved in deploying such models, we experiment with a probabilistic ensemble that combines the individual -and potentially biased and noisy -outputs of those models. we use the multi-annotator competence estimation model @xcite , a bayesian method designed to deal with noisy crowdsourced annotations (section 5). experimental results (section 6) show that our ensemble performs better than the strongest individual model. in addition, we show that just fine tuning with few labeled data, our system outperforms fullysupervised models. ## related work attempts to minimize the amount of handlabelled data required to train emotion-aware nlp models have mostly focused on using distant supervision @xcite to collect large amounts of silver labels for training models in a supervised fashion: emoji @xcite , emoji description @xcite , and hashtags @xcite have been shown to be good proxies for emotion classification. the idea of using label templates for unsupervised classification can be traced back at least to hearst patterns @xcite . within the neural paradigm, cloze @xcite label templates are used by @xcite , who obtain strong results on few-shot classification. @xcite and @xcite use templates to generate synthetic data for framing text-classification as entailment. an alternative neural approach to unsupervised classification embeds both the input sequence of text and the set of possible label names in the same semantic space and selects the one which maximizes a defined similarity metric @xcite . pushp and srivastava (2017) concatenates both input and label embeddings and classify their relatedness. a recent survey of template-based or "prompt-based" learning can be found in @xcite . perhaps the closest work to ours is @xcite , who evaluate zeroshot text classification on the unify emotion dataset @xcite ). 1 our main contribution with respect to @xcite is the suggestion of a principled way for a) aggregating multiple predictions without having any access to a model's performance and b) inferring the most probable emotion label given multiple models. an overview of bayesian models of annotation can be found in @xcite . the idea of using a generative model to infer a latent label from multiple signals has recently been presented in a unified framework by @xcite . an alternative to bayesian models for aggregating predictions can be found in @xcite schütze (2019) for all our experiments we use a section of the unify emotion dataset @xcite which aggregates several annotated corpora in a common format. specifically, we use the following datasets: grounded-emotions @xcite , crowdflower @xcite , dailydialog @xcite , tec @xcite , electoral-tweets @xcite , isear @xcite , emotion-stimulus @xcite , tales-emotion @xcite @xcite , and emoint @xcite . we aggregate all the corpora and then sample from each class 60% of the data for the train split, and 20% for the emotion dataset, the actual instances and label set used are different. ## entailment as zero-shot and few-shot learning given two sentences, a premise and a hypothesis, they can be related by an entailment, contradiction or neutral relation. the task of natural language inference (nli) @xcite aims at predicting such relations. recently, the creation of large nli datasets @xcite @xcite has allowed deep learning methods to achieve stateof-the-art performance on the nli task, outperforming logic-based approaches. the high performance of bert-like models @xcite on nli tasks can be exploited to successfully tackle general classification tasks by recasting them as entailment problems: pretrained language models can be finetuned on nli datasets and these finetuned models can be then re-purposed to attack different problems @xcite . for modelling the emotion classification problem, we follow @xcite and given a text to classify (the hypothesis), we build pseudo-sentences to serve as premises, one for each target label. for instance, the input sentence "john said he loved the pizza" can be classified as joy, if an nli models predicts that it entails the artificial sentence "this person expressed a feeling of pleasure". we can substitute pleasure with other emotion-expressing words and map them to specific target labels (e.g., pleasure to joy, sad to sadness, etc.) to build a system for zero-shot, multi-label emotion classification. following @xcite , in this work we explore two options to formulate the hypotheses: based on the label's name and on the label's wordnet @xcite definition. we show the details about our hypotheses for emotion classification in we experiment with six different pretrained nli models that differ in terms of the underlying pretrained language model (bart @xcite , roberta @xcite and xlm-roberta @xcite ) and nli dataset used for training (multigenre nli (mnli) @xcite , adversarial nli (anli) @xcite and xnli @xcite ). in we conduct the few-shot learning experiments by fine-tuning a pretrained entailment model. to build the training data, we fill the templates used in the zero-shot setup with the gold labels and training follows the standard sentence pair classification task used to train the original entailment model. ## a probabilistic ensemble in a true zero-shot classification scenario, no development set is available and therefore a method for estimating the performance of the model on the specific input data is required. in this work, we propose to use several different models and to infer the best possible answer using a probabilistic model. such a model has two advantages over a simple majority voting strategy: first, it has been shown to outperform majority voting @xcite ; second, it provides a confidence value for each instance and estimates the models' accuracy. to aggregate the predictions from the different unsupervised models, we use the multi-annotator competence estimation (mace) model @xcite . this model has been originally developed to analyse crowdsourced annotations for both identifying unreliable annotators and retrieving the true labels. figure 1 shows the plate diagram of the model and we refer to the original publication for further details. algorithm 1 describes the generative process. label label-based hypothesis wordnet-based hypothesis anger (...) feels angry (...) expresses a strong feeling of annoyance, displeasure, or hostility anticipation (...) has a feeling of anticipation (...) is anticipating something, expecting or predicting something about to happen confusion (...) is feeling confused (...) is feeling disoriented and can not think clearly or focus to do something disgust (...) feels disgusted (...) expresses a feeling of revulsion or strong disapproval aroused by something unpleasant or offensive fear (...) is afraid of something (...) expresses an unpleasant emotion caused by the belief that someone or something is dangerous, likely to cause pain, or a threat guilt (...) feels guilty (...) expresses a feeling of having done wrong or failed in an obligation joy (...) feels joyful (...) expresses a feeling of great pleasure and happiness love (...) loves that (...) expresses a great interest and pleasure in something noemo (...) does not feel any emotion (...) is insensitive, showing unfeeling and unresponsive behaviour, with a lack of emotion about the situation sadness (...) feels sad (...) expresses emotions experienced when not in a state of well-being shame (...) feels shameful (...) expresses a painful feeling of humiliation or distress caused by the consciousness of wrong or foolish behavior surprise (...) feels surprised (...) expresses a feeling of mild astonishment or shock caused by something unexpected trust (...) feels trusty about this (...) has a strong belief in the reliability, truth, or ability of someone or something algorithm 1: generative process of the mace model. ## evaluation we evaluate the unsupervised models in a zeroshot configuration against two supervised baselines. we then experiment with adding increasing amounts of supervision to both the baselines and the entailment model in a few-shot setting. we conduct all the evaluation using the standard classification metrics: precision, recall, and macro-averaged f1-score. baselines as upper-baselines for our experiments, we train two supervised models on all the available training data: we train both a neural network based on roberta-base @xcite and a linear svm using character n-grams as representations (henceforth referred to as char-svm). 4 entailment models we assemble 12 different unsupervised zero-shot classification models finetuned on the nli task. the models differ in terms of the pretrained language model, label template and nli finetuning data. : distilled model. rows in a: fully supervised. rows in b: zero-shot. rows in c: aggregations. rows in d: few-shot learning, each row denotes the number of training instances; the results are averaged over three runs and the standard deviation is shown in subscript. statistically significant results according to a χ 2 test, per sub-table, are highlighted in bold. significant results between zero-shot and the aggregations (b and c sub-tables), are highlighted with *. ## conclusions in this work we presented an emotion classification model that does not require large annotated data to be competitive on the unify emotion dataset. we experimented with pretrained language models in both the zero-shot and fewshot settings. we aggregated the predictions of these models using mace, a bayesian method developed for modelling noisy, crowdsourced annotations. experimental results showed that the resulting system performs better than the strongest individual zero-shot model. when evaluated on a diverse dataset, our zero-and few-shot models behave in a comparable way to fully-supervised models, without requiring the same amount of annotated data. noisy text seems to challenge the nli models trained on canonical text, while zero-shot models perform well when the annotation scheme matches the pseudo-sentences used for building the synthetic data: this suggests that different domains might need different templates that take into account elements like vocabulary or stylistic variation. finally, we showed that when the mace and the few-shot systems are trained on few labelled data, they outperform fullysupervised models. in future works we will further explore how to apply zero and few-shot learning for text classification tasks, and how to better aggregate the outputs of different models in a unsupervised manner.