Sentiment and Emotion Classification in Low-resource Settings (2023)

Abstract: The popularity of sentiment and emotion analysis has lead to an explosion of datasets, approaches, and papers. However, these are often tested in optimal settings, where plentiful training and development data are available, and compared mainly with recent state-of-the-art models that have been similarly evaluated. In this paper, we instead present a systematic comparison of sentiment and emotion classification methods, ranging from rule- and dictionary-based methods to recently proposed few-shot and prompting methods with large language models. We test these methods in-domain, out-of-domain, and in cross-lingual settings and find that in low-resource settings, rule- and dictionary-based methods perform as well or better than few-shot and prompting methods, especially for emotion classification. Zero-shot cross-lingual approaches, however, still outperform in-language dictionary induction.

## introduction affective computing, including sentiment and emotion classification, has been research focuses inside of the natural language processing (nlp) community for many years @xcite . this has lead to an incredible number of research directions and papers published on these topics, ranging from rule-based and dictionary-based approaches @xcite ; @xcite ; @xcite ; @xcite , to supervised training of deep learning models @xcite @xcite and finally to few-shot and prompting of large language models @xcite @xcite . this also means that a systematic comparison of the benefits and weaknesses of models has not been performed, as each often individual papers compare only against more recent state-of-the-art models, and do not take into account previous approaches. like many other research areas in nlp, sentiment and emotion classification are dependent on domain and language-specific training data for optimal performance and this high-quality task-specific data is always in short supply as we apply our models to a constantly evolving set of scenarios. the objective of this paper is therefore to identify trends in sentiment and emotion classification, especially regarding low-resource settings. as such, we attempt to address the following research questions: to address these questions, we perform experiments foot_0 on 10 sentiment classification datasets and two emotion classification datasets in 8 languages with a number of low-resource approaches. specifically, we compare dictionary-based methods, rulebased methods, few-shot methods and prompting methods on the english datasets. we simultaneously test the out-of-domain performance for each of the methods that demand training data. finally, we also perform cross-lingual experiments. we find that rule-and dictionary-based methods often perform on par with few-shot approaches in low-resource settings, especially on emotion classification and are more robust to domain changes, while prompting similarly provides promising results. zero-shot cross-lingual approaches, however, still outperform in-language dictionary induction for languages other than english, suggesting that more work could be done in this area. ## related work current state-of-the-art models for sentiment and emotion classification are dominated by language models that have been pretrained on large corpora and then fine-tuned for each specific task @xcite . although elmo @xcite , bert @xcite , and its variants were the first to provide evidence for the usefulness of language modeling as a transfer learning objective, there has since been an explosion and it is somewhat difficult to navigate which current models give the best performance on many datasets. besides the fully supervised setup, many larger language models also show signs of being able to learn a task with less data, allowing for non-trivial zero-or few-shot performance. the most common way to achieve this zero or few-shot ability is by prompting a model using a natural language inference model, trained to determine whether a premise is true/false, given a hypothesis. this model can then be applied to new tasks by reformulating the input and labels @xcite . for few-shot prompting, we can make use of the generative abilities of language models by providing demonstrations input/label pairs and asking for a final label @xcite . more recently, the results of models trained using instruction tuning suggests that these models generalize well to unseen tasks @xcite . the same kinds of large language models trained on multilingual corpora also allow for zero-shot cross-lingual transfer, where a model is fine-tuned on a task in a high-resource source language and then tested on an under-resourced language @xcite . however, these approaches have rarely been compared to previous dictionary-based methods. ## data in this section we describe the datasets that are used for experimentation. the statistics are shown in sentiment datasets as we want to explore how well methods work across a number of domains and languages, we choose to explore binary sentiment classification. we use the binary version of the following datasets, where any strong positive/negative has been mapped to positive/negative and neutral has been removed. using only binary sentiment classification allows for us to compare across a larger number of datasets and languages. mpqa: @xcite annotate english news wire texts with a complex set of annotation types. we map the polarities to sentences and keep those sentences that contain a majority of one polarity, such that we have only positive and negative sentence-level annotations. semeval: the semeval 2013 shared task 2 @xcite collected tweets and annotated them as positive, negative, or neutral. we keep only the positive and negative tweets. opener: @xcite annotate english and spanish (among others) hotel reviews for structured and aspect-based sentiment. we use the script from @xcite to map these to sentencelevel binary sentiment classification. es is the spanish data from this dataset. ar: nabil et al. (2015) annotate arabic (both modern standard arabic and various dialects) tweets. we remove the neutral and mixed classes. dz: touileb and barnes (2021) annotate northern african arabizi social media posts for sentiment. in this case, we use the transliterated arabic script version of the dataset and remove the neutral class. mt: the data for maltese (dingli and sant, 2016; cortis and davis, 2019) comes from the combination (martínez-garcía et al., 2021) of two smaller datasets. ## experimental setup in this section, we describe the approaches for three experimental setups (monolingual english sentiment classification, monolingual english emotion classification, and cross-lingual sentiment classification) from most resource intensive to least. ## results in this section we detail the results for sentiment classification, out of domain performance, emotion classification, and cross-lingual transfer. ## conclusion and future work in this paper we have performed experiments on 10 sentiment datasets and two emotion classification datasets in 8 languages with a number of low-resource approaches (dictionary-based methods, rule-based methods, few-shot methods and prompting methods). the main experiments were performed on the english language datasets (3 sentiment and 2 emotion), while further experiments were performed in 7 additional languages. these results confirm that under ideal circumstances, fully supervised models perform much better than low-resource approaches. however, in low-resource settings (lack of training data, domain shift), these same models quickly lose performance and rule-based and dictionary-based approaches perform on par or even better if there is a domain shift involved. while prompting achieved impressive performance in our experiments, given that the models were not explicitly trained, this came at a price. namely, such approaches for languages other than english are currently not available or not on par with english versions. this area will surely be explored in the near future, but this current gap is nonetheless a product of the over-reliance on english in nlp. the strong cross-lingual performance of the xlm-roberta models suggests that cross-lingual approaches, especially those designed for adapting to new languages, scripts @xcite , or generally enabling ever more multilingual pretraining @xcite . we find conflicting evidence on the importance of model size for low-resource performance. on the one hand, prompting the larger opt model for sentiment classification gave consistently better results. on the other hand, roberta-large suffered much more in out-of-domain classification and generally performed worse than robertabase on emotion classification in all data regimens. for prompting in emotion classification, flant5large did not lead to consistent gains over the base version and finally, xlm-roberta-large similarly performed worse than the base version on crosslingual sentiment classification. this finding seems to indicate that some of the promised few-shot performance found in large language models is either lacking or requires careful tuning. in the future, it would be interesting to expand this comparison to other dictionary induction methods, such as cross-lingual propagation @xcite , or high-coverage expansion @xcite . given the promising results from the simple prompting approaches we used in our experiments, further research on how to expand these models to new languages and tasks would be of great use. finally, multi-lingual few-shot approaches @xcite could also be compared, as it is often possible to use a few examples in the target language.