Towards Distribution-shift Robust Text Classification of Emotional Content (2023)

Abstract: Supervised models based on Transformers have been shown to achieve impressive performances in many natural language processing tasks. However, besides requiring a large amount of costly manually annotated data, supervised models tend to adapt to the characteristics of the training dataset, which are usually created ad-hoc and whose data distribution often differs from the one in real applications, showing significant performance degradation in real-world scenarios. We perform an extensive assessment of the out-of-distribution performances of supervised models for classification in the emotion and hate-speech detection tasks and show that NLI-based zero-shot models often outperform them, making task-specific annotation useless when the characteristics of final-user data are not known in advance. To benefit from both supervised and zero-shot approaches, we propose to fine-tune an NLI-based model on the task-specific dataset. The resulting model often outperforms all available supervised models both in distribution and out of distribution, with only a few thousand training samples.

## introduction supervised text classification based on transformers has recently achieved considerable performances, benefiting many applications in social @xcite technological @xcite and biomedical @xcite domains, just to mention a few. however, these systems rely on large amounts of manually annotated data that are often expensive to obtain. furthermore, to guarantee reasonable performances, supervised systems need to be trained on data that have the same distribution as the one in the deployed scenario @xcite . this requires a careful choice of data to annotate that is sometimes impossible to achieve because of the difficulty to infer in advance the characteristics of runtime data, and considering the potential evolution of data features during the system's lifetime @xcite . recent work has shown that transformers are more robust than other machine learning models to change in domain and distribution @xcite . however, the decrease in performance due to the distribution shift is still a major issue of supervised models @xcite . figure 1 shows the degradation in performances of models when applied to a different distribution. we consider three emotion classification tasks (with different taxonomies) and a hate speech detection task, and report in-distribution (id) performances, when the model is validated on the same dataset (light blue bars), in comparison with outof-distribution (ood) performances, i.e. validated on different datasets (dark red bars). the drop in performance is significant, often overcoming 30% and sometimes reaching almost 50%. this makes models trained on certain data barely generalizable to other data, limiting drastically their scope. whether a certain textual label is associated with a sentence @xcite . although supervised task-specific trained models typically outperform zero-shot approaches in the training dataset, it is reasonable to question how they compare when the supervised approach is trained on a different dataset. in this work, we make a comprehensive assessment of the ood performances of emotion detection and hate speech detection models in comparison with a nli-based zero-shot model. surprisingly, our results show that the zero-shot approach almost always outperforms the supervised models, suggesting that labeling a large amount of data is not beneficial when the data distribution is not a-priori known. to take advantage of both approaches we propose to adapt and fine-tune a nli model with task-specific data. we show that a small amount of training data is sufficient to achieve performances that are often superior to the top-performing supervised models available, either id or ood. our contribution can be summarized as follows: (1) we perform a comprehensive assessment of the ood performance of supervised models for classification (multi-class, multi-label, binary) of emotive content in comparison to an nli-based approach that does not require specific training, and we show that the latter often achieves higher performance; (2) we propose fine-tuning an nli model on task-specific data and show experimentally that this solution achieves competitive performances both id and ood with only a few thousand samples; (3) we extensively discuss our results and give useful indications for achieving significant id and ood performances with a small annotation cost. ## related works developing models that are robust to domain and distribution shift is one of the most intriguing yet challenging tasks in various machine learning applications @xcite including computer vision @xcite @xcite and nlp @xcite @xcite . we refer to @xcite for an extensive survey on domain generalization. while some works offer a more theoretical perspective on the topic @xcite , general work in the nlp field has been focused mainly on developing benchmarks for evaluating the outof-distribution robustness of models. hendrycks et al. @xcite ) studies ood generalization for seven nlp datasets in the tasks of sentiment classification, semantic similarity, reading comprehension, and textual implication and show that pre-trained transformers adapt better to ood data. yang et al. @xcite propose a unified benchmark called glue-x to evaluate ood robustness in nlp systems. they collect 13 datasets covering tasks such as sentiment analysis, natural language inference, sentence pair similarity, textual similarity, and linguistic acceptability. for each task, they select a dataset for training and other datasets for ood evaluation. the study shows that better ood accuracy is needed for nlp tasks, due to the noticeable loss of performance with respect to the id settings. both works do not compare the performances to zero-shot approaches and do not propose specific methods for increasing ood robustness. furthermore, they do not consider the tasks of emotion detection and hate-speech detection. approaches to deal with the distribution shift problem include ood detection @xcite and mixture of experts (moe) models @xcite . ood detection aims at recognizing ood text to give awareness of the potential degradation in performances, while moe models tend to combine domain-specific models to improve performances in multi-domain contexts. both these approaches are out of the scope of our work since they do not specifically focus on assessing and improving the performances of models over unseen domains and data distribution changes. specific studies on text classification related to ours usually focus on domain generalization by training on text in one domain and testing on a different domain within the same dataset. although related, these approaches do not consider domainindependent differences that occur across datasets concerning e.g. text features (e.g. length), linguistics features (e.g. use of slang) and annotation processes. pada @xcite generates domain-related features and adds them to the text to enable the model to adapt to different domains. other studies refer to specific tasks such as moral value classification @xcite and sentiment analysis @xcite @xcite @xcite . despite not considering the model generalization across datasets, and being often application-specific, these methods do not make any assessment with zero-shot learning nor consider building upon them to improve ood performance. to the best of our knowledge, the only studies on distribution shift that consider the emotion and hate-speech detection tasks are the work of toraman et al. @xcite , which evaluates how bert generalizes across abusive language detection datasets, and zeng et al. @xcite that propose a cnn-based broad learning model for cross-domain emotion classification. the first study on abusive language detection does not compare with zero-shot models nor proposes a method for ood generalization. the zeng et al. work considers a multi-domain dataset obtained by collecting data from chinese e-commerce platforms and performs the assessment across domains. again they do not perform a comparison with zero-shot models nor evaluation across datasets. another line of research related to our work concerns zero-shot and prompt-based models. @xcite propose and evaluate three lstm architectures for zero-shot classification that combine text embedding with label embedding to determine whether the label is related to the input text. yin et al. @xcite provide datasets, a standard for evaluation and state-of-theart baselines for zero-shot classification. barker et al. @xcite propose performing supervised classification on known labels, then applying nli for cases that do not qualify for previously known labels. zhang et al. @xcite propose a meta-learning framework to learn to calibrate the class and sample representations from both seen and virtual unseen classes. other studies focus on the impact of different prompts on performances @xcite . in particular, we highlight the work of plaza-del-arco et al. @xcite which compares different ways to build the hypothesis prompt for nli-based emotion detection. although we adopt an nlibased zero-shot model in our work, taking inspiration from the work of yin et al. @xcite and plaza-del-arco et al. @xcite , no other work that we are aware of makes an extensive comparison of ood performances of supervised models with zero-shot models and fine-tuning of the latter, finding the sweet spot between no-specific training and a fully supervised approach. to the best of our knowledge, there are no extensive state-of-the-art studies focusing on the ood robustness (across datasets) of supervised models in emotion classification tasks. in general, we are not aware of any study that compares ood performances of supervised models for text classification with zero-shot models and assesses the best way to fine-tune a zero-shot model. ## materials and methods we describe in detail the benchmark data and models of our work. in section 3.1 we discuss datasets. section 3.2 focuses on the analysis of the stateof-the-art supervised models and the nli-based systems we employ. ## results and evaluation we assess the ood performances of supervised models in comparison with nli-based classification. we group our evidence by looking at three main classification problems: multi-class (where an item can be associated with only one label), multilabel (where an item can be associated with more than one label) and binary. we set out to investigate performances on emotion-domain-specific detection tasks as a unifying framework across all experiments. we present the details of our experimental settings and the results in sections 4.1, 4.2 and 4.3. all output data are available on github foot_6 . we also evaluate the performances of different prompts in section 4.4. in the last parts of our experimental analysis we evaluate the nli-with-fine-tuning method discussed in section 3.2.3 at varying the number of training samples and in comparison with fully supervised systems (sect. 4.5). unless differently specified, all performances reported refer to the f1-score. for multi-class and multi-label classification, we consider the weighted f1. all experiments have been run on a server with 2 cpu intel xeon gold 6238r 2.20ghz with 640gb ram and two gpu a100 40gb. as a rough estimation, our experiments took in total about 30 gpu days. ## conclusion we made an extensive experimental analysis of the ood performances of supervised transformerbased classifiers trained on task-specific data with emotive content, in comparison with zero-shot approaches based on nli that do not require specific training. our results show that, although nospecific-training approaches are not able to perform as well as supervised models in the same dataset, they often achieve the best performance w.r.t. supervised models evaluated on a different dataset. we found that a mixed approach consisting in fine-tuning nli-based classifiers with limited data reaches a good trade-off between id and ood performances.