Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora (2022)

Abstract: Within textual emotion classification, the set of relevant labels depends on the domain and application scenario and might not be known at the time of model development. This conflicts with the classical paradigm of supervised learning in which the labels need to be predefined. A solution to obtain a model with a flexible set of labels is to use the paradigm of zero-shot learning as a natural language inference task, which in addition adds the advantage of not needing any labeled training data. This raises the question how to prompt a natural language inference model for zero-shot learning emotion classification. Options for prompt formulations include the emotion name anger alone or the statement “This text expresses anger”. With this paper, we analyze how sensitive a natural language inference-based zero-shot-learning classifier is to such changes to the prompt under consideration of the corpus: How carefully does the prompt need to be selected? We perform experiments on an established set of emotion datasets presenting different language registers according to different sources (tweets, events, blogs) with three natural language inference models and show that indeed the choice of a particular prompt formulation needs to fit to the corpus. We show that this challenge can be tackled with combinations of multiple prompts. Such ensemble is more robust across corpora than individual prompts and shows nearly the same performance as the individual best prompt for a particular corpus.

## introduction to enable communication about emotions, there exists a set of various emotion names, for instance those labeled as basic emotions, by @xcite or @xcite . while such psychological models influence natural language processing and emotion categorization approaches, the choice of emotion concepts is context-dependent. for instance, @xcite and @xcite opted to use guilt and shame as self-directed emotions in addition to ekman's basic emotions, to analyze self-reports of events. for the context of the perception of art it is more appropriate to consider aesthetic emotions @xcite , like beauty, sublime, inspiration, nostalgia, and melancholia. this leads to a potential gap between concepts in emotion-related training data and the application domain, purely because the label set is not compatible. one solution is to resort to so-called dimensional models, in which emotion names are located in vector spaces of affect @xcite or cognitive appraisal (e.g., regarding responsibility, certainty, pleasantness, control, attention with respect to a stimulus event, @xcite . in these vector spaces, classes can be assigned to predicted points with a nearest-neighbor approach, even if these classes have not been seen during training. this approach, however, has the disadvantage of the so-called hubness problem @xcite , namely that the distance between predictions and concepts that have been seen during training tends to be smaller than to novel concepts. we acknowledge ongoing research to tackle this problem @xcite figure 1 : an example of the application of nli to zsl emotion classification. given the premise "i won a trip to greece in a competition", three hypotheses represent the emotions (joy, anger, sadness). the representation of joy is entailed and therefore predicted. @xcite . we take a different, more direct, route to obtaining classifiers for discrete emotion categories which are not known at system development time, namely zero-shot learning (zsl). one instantiation of such zsl systems is via natural language inference models (zsl-nli), in which the inference task needs to perform reasoning @xcite . consequently, the idea of implementing zsl-nli models is not by exemplification and optimizing a classifier, but developing appropriate natural language class name representations which we refer to as prompts. we see an example for the application of an nli model to zsl emotion classification in figure 1 -the nli model needs to decide if the hypothesis (a prompt which represents the class label) entails the premise (which corresponds to the instance to be classified). this paradigm raises the question (which we answer in this paper) of how to formulate the emotion prompt and how much the design choice of the prompt needs to fit the dataset. manually developing intuitive templates based on human data introspection may be the most natural method to produce prompts. in this paper, we provide manually created templates to probe emotion classification in an nli-zsl setup and we analyze whether prompts are language-register dependent according to various corpora (tweets, event descriptions, blog posts). to accomplish this aim, we perform experiments on an established set of emotion datasets with three nli models and we show that (1) prompts are indeed corpus-specific and that the differences follow the same pattern across different pretrained nli models, (2) that an ensemble of multiple prompts behaves more robustly across corpora, and (3) the representation of the emotion concept as part of the textual prompt is an important element, benefiting from representations with synonyms and related concepts, instead of just the emotion name. our code is publicly available at https: //github.com/fmplaza/zsl_nli_emotion_prompts . ## methods in this section, we explain how we apply nli for zsl emotion classification and propose a collection of prompts to contextualize and represent the emotion concept in different corpora. in addition, we propose a prompt ensemble which is more robust across corpora. ## experiments we aim at answering the following research questions: (rq1) do nli models behave the same across prompts? (rq2) should we use synonyms for the emotion representation? (rq3) is an ensemble of multiple prompts more robust across corpora? (rq4) are synonyms sufficient? would it be even more useful to use more diverse representations of emotions? ## conclusion and future work we presented an analysis of various prompts for nli-based zsl emotion classification. the prompts that we chose were motivated by the various particularities of the corpora: single emotions for tec (tweets), "the person feels/the text expresses" for blogs (blogs), and isear (events). in addition, we represented the emotions with emotion names, synonyms, definitions, or with the help of lexicons. our experiments across these data sets showed that, to obtain a superior performance, the prompt needs to fit well to the corpus -we did not find one single prompt that works well across different corpora. to avoid the requirement for manually selecting a prompt, we therefore devised an ensemble model that combines multiple sets of prompts. this model is more robust and is nearly on par with the best individual prompt. in addition, we found that representing the emotion concept more diversely with synonyms or lexicons is beneficial, but again corpus-specific. our work raises a set of future research questions. we have seen that the oracle ensemble showed a good performance, illustrating that the various prompts provide complementary information. this motivates future research regarding other combination schemes, including learning a combination based on end-to-end fine-tuned nli models. we have further seen that including more concepts with the help of a dictionary helps in one corpus, but not across corpora; however, synonyms constantly help. this raises the question about the right trade-off between many, but potentially inappropriate, noisy concepts and hand-selected, highquality concepts. a desideratum is an automatic subselection procedure, which removes concepts that might decrease performance and only keeps concepts that are "compatible" to the current language register and annotation method. ideally, this procedure would not make use of annotated data, because that would limit the advantages of zsl. the main limitation of our current work is that we manually designed the prompts under consideration, based on the corpora we used for evaluation. this is a bottleneck in model development, which should either be supported by a more guided approach which supports humans in developing prompts, or by an automatic model that is able to automatically generate prompts based on the language register and concept representation in the dataset.