Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge (2023)

Abstract: While transformer-based systems have enabled greater accuracies with fewer training examples, data acquisition obstacles still persist for rare-class tasks – when the class label is very infrequent (e.g. < 5% of samples). Active learning has in general been proposed to alleviate such challenges, but choice of selection strategy, the criteria by which rare-class examples are chosen, has not been systematically evaluated. Further, transformers enable iterative transfer-learning approaches. We propose and investigate transfer- and active learning solutions to the rare class problem of dissonance detection through utilizing models trained on closely related tasks and the evaluation of acquisition strategies, including a proposed probability-of-rare-class (PRC) approach. We perform these experiments for a specific rare-class problem: collecting language samples of cognitive dissonance from social media. We find that PRC is a simple and effective strategy to guide annotations and ultimately improve model accuracy while transfer-learning in a specific order can improve the cold-start performance of the learner but does not benefit iterations of active learning.

## introduction cognitive dissonance occurs during everyday thinking when one experiences two or more beliefs that are inconsistent in some way @xcite . often expressed in language, dissonance plays a role in many aspects of life, for example affecting health-related behavior such as smoking @xcite and contributing to the development of (and exit from) extremism @xcite . however, while the phenomenon is common enough to occur on a daily basis, dissonance is still relatively rare among the myriad of other relationships between * co-lead authors humans annotate acquisition strategy: which are "best" to label? add new examples model retrain/update active learning iteration new examples cumulative (cm) iterative (it) new old data old model fine-tune new old data old model train new model new model initial model: transfer learning start increase chance of rare class? rare class annotation ~ "needle in a haystack" easier to annotate difficult to annotate beliefs that occur across random selections of linguistic expressions and thus makes the automatic detection of it a rare-class problem. despite recent advances in modeling sequences of words, rare-class tasks -when the class label is very infrequent (e.g., < 5% of samples) -remain challenging due to the low rate of positive examples. not only are more random examples necessary to reach a substantial amount of the rare class (e.g., 1,000 examples to reach just 50 examples), but also it is easy for human annotators to miss the rare instances where dissonance is present. here, we develop and address the challenges of creating a resource for language-based assessment of dissonance. active learning using large language models presents both new opportunities and challenges. on the one hand, large language models (llms) offer unmatched representations of documentations, able to achieve state-of-the-art language understanding task performance with transfer learning, often only with a few iterations of fine-tuning @xcite . on the other hand, representations are highdimensional, and models trained or fine-tuned with only a small number of examples are prone to over-fitting, especially when there is a large class imbalance as in rare-class problems. while llms have enabled attempts to tackle increasingly complex semantic challenges across a growing list of tasks, getting annotated examples for such problems can become a bottleneck due to its time-and laborintensiveness @xcite . since data-centric improvements for more novel tasks can provide a faster path than model-centric improvements @xcite , active learning can be a way forward to be both data-centric and address bottlenecks in label acquisition -it aims to reduce annotation costs as well as alleviate the training data deficiency that large language models face. however, while active learning has been studied for multiple natural language tasks @xcite , little is known about active learning acquisition strategies for lm-based approaches, especially for rare-class problems. high data imbalance coupled with very less training data poses the challenge of "absolute rarity" @xcite , as in our task of dissonance detection. we address this problem by using a novel combination of evaluating the ordering of transfer learning from similar tasks to cold-start the active learning loop, and by acquiring with a relatively simple acquisition strategy focused on probability-of-rare-class (prc) to increase the rare class samples. our contributions include: (1) finding that bootstrapping al models with transfer learning on closely related tasks significantly improves rare class detection; (2) a novel systematic comparison of five common acquisition strategies for active learning for a rare class problem 1 ; (3) a systematic comparison of two different approaches to handling al iterations for llms -cumulative and iterative fine-tuned model updates -finding the cumulative approach works best; (4) evaluating annotation costs of a rare-class task, finding that minimum annotation cost does not necessarily lead to better models, especially in realistic scenarios such as absolute rarity; and (5) release of a novel dataset 2 for the task of identifying cognitive dissonance in social media documents. 1 code: https://github.com/humanlab/rare-class-al 2 dataset: https://github.com/humanlab/dissonance-twitterdataset ## related work active learning in nlp has been largely studied as a theoretical improvement over traditional ml for scarce data. in this work, we specifically investigate pool-based active learning, or picking out samples to annotate from a larger pool of unlabeled data, and particularly data for a rare-class problem where lms are not well-understood yet. acquisition strategies sampling strategies for active learning can be broadly classified into three: uncertainty sampling @xcite @xcite , representative (or diversity) sampling @xcite @xcite , and the combination of the two @xcite . the uncertainty sampling strategies that employ classification probabilities, bayesian methods such as variational ratios @xcite , and deep-learning specific methods @xcite often use epistemic (or model) uncertainty. we choose maximum entropy to represent the uncertainty sampling, since it is usually on par with more elaborated counterparts @xcite . as a popular diversity sampling baseline to compare against, we pick select coreset @xcite . the state-of-the-art methods combine these two strategies in novel ways, such as using statistical uncertainty in combination with some form of data clustering for diversity sampling @xcite . our work uses contrastive active learning @xcite to represent this strategy. on the other hand, @xcite and @xcite claim there is rather small to no advantage in using active learning strategies, because a number of samples might be collectively outliers, and existing strategies contribute little to discover them and instead harm the performance of subsequent models. researchers recently have also focused on the futility of complex acquisition functions applied to difficult problems and argued that random acquisition performs competitive to more sophisticated strategies, especially when the labeled pool has grown larger @xcite . furthermore, a large-scale annotation of randomly sampled data could be less expensive than ranking data to annotate in each round of active learning, if there is not much advantage (i.e., such as capturing rare classes) in using a specific strategy. cold-start al while the problem of cold-start exists in acquiring samples through active learning, some work has been done to combat this by leveraging the learned weights in pretrained models @xcite . however, there is much to gain from the field of transfer learning especially for rare class problems, as seen in @xcite . we borrow the concept of heterogeneous transfer learning @xcite and transfer the model weights directly obtained from pretraining on closely related (but different) tasks on completely different domains. this helps models to improve the zero-shot ability for rare class detection. such methods have been explored in traditional machine learning @xcite but not in the era of large language models to the best of our knowledge. rare class al there has been a growing number of applications of active learning in data imbalance and rare class problems. such works include @xcite @xcite which proposed frameworks to improve model performance with data imbalance but failed to check the feasibility and costs in a real-world, active annotation setting where not only is rare class very infrequent ( 4%) but very few (< 70) examples of the rare class exist due to small dataset size ("absolute rarity"). they also fail to compare against a simple, rare class probability of the model. while some work in the pre-llm era use probability outputs of a classifier (certainty-based sampling) which is similar to the proposed prc, they claim to work better in conjunction with co-selection using other uncertainty sampling strategies, and that certainty-based sampling alone performs poorly in terms of increasing rare-class samples @xcite . many studies also focus on rare class discovery, or finding outlying samples that do not fall under the existing categories @xcite @xcite . this is different from our task which focuses on the detection of a rare class. social psychology has used this human tendency to resolve dissonance to understand important psychological processes such as determinants of attitudes and beliefs, consequences of decisions, internalization of values, and the effects of disagreement among persons @xcite . dissonance is also related to anxiety disorders @xcite , relevant to understanding extremism and predicting cognitive styles of users. our approach to annotating cognitive dissonance on social media is motivated by the two-stage annotation approach described in @xcite . to the best of our knowledge this is the first social media dataset for cognitive dissonance. ## conclusion in this work, we have systematically studied approaches to key steps of active learning for tackling a rare-class modeling using a modern large language-modeling approach. while transformerbased systems have enabled greater accuracies with fewer training examples, data acquisition obstacles still persist for rare-class tasks -when the class label is very infrequent (e.g., < 5% of samples). we examined pool-based active annotation and learning in a real-world, rare class, natural language setting by exploring five common acquisition strategies with two different model update approaches. we found that a relatively simple acquisition using the probability of rare class for a model could lead to significant improvement in the rare class samples. we also qualitatively analyzed the data samples extracted from each data acquisition strategy by using subjective scoring and timing the annotators, finding prc to be the most difficult to annotate, while also remaining the best method to improve rare class samples and model performance. our final dataset of 9,589 examples (big train + dev + test) is made available along with an implementation of the prc method and our state-of-the-art model for cognitive dissonance detection.