Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals (2022)

Abstract: For text classification tasks, finetuned language models perform remarkably well. Yet, they tend to rely on spurious patterns in training data, thus limiting their performance on out-of-distribution (OOD) test data. Among recent models aiming to avoid this spurious pattern problem, adding extra counterfactual samples to the training data has proven to be very effective. Yet, counterfactual data generation is costly since it relies on human annotation. Thus, we propose a novel solution that only requires annotation of a small fraction (e.g., 1%) of the original training data, and uses automatic generation of extra counterfactuals in an encoding vector space. We demonstrate the effectiveness of our approach in sentiment classification, using IMDb data for training and other sets for OOD tests (i.e., Amazon, SemEval and Yelp). We achieve noticeable accuracy improvements by adding only 1% manual counterfactuals: +3% compared to adding +100% in-distribution training samples, +1.3% compared to alternate counterfactual approaches.

## introduction and related work for a wide range of text classification tasks, finetuning large pretrained language models @xcite @xcite on task-specific data has been proven very effective. yet, analysis has shown that their predictions tend to rely on spurious patterns @xcite @xcite @xcite @xcite , i.e., features that from a human perspective are not indicative for the classifier's label. for instance, @xcite found the rather neutral words "will", "my" and "has" to be important for a positive sentiment classification. such reliance on spurious patterns were suspected to degrade performance on out-of-distribution (ood) test data, distributionally different from training original source documents manually created counterfactuals (paired with an original) + + + + + + + + + + + + ----document representation vector space ## exploiting few counterfactuals we consider binary sentiment classification of input sentences/documents @xmath0, with associated la-bels @xmath1, 1}. we denote the training set of labeled pairs (x, y) as d id , of size n ≜ |d id |. we further assume that for a limited subset of k ≪ n pairs (x, y) we have corresponding manually constructed counterfactuals (x cad , y cad ), i.e., x cad is a minimally edited version of x that has the opposite label y @xmath2. the resulting set of k counterfactuals is denoted as d cad . we will adopt a vector representation of the input ϕ(x), with ϕ ∶ @xmath3. we aim to obtain a classifier f ∶ r @xmath4, without degrading in-distribution performance, performs well on counterfactual samples and is robust under distribution shift. ## experimental setup datasets for the in-distribution data, we use a training set d train id of 1,707 samples, and a test set d test id of 488 samples, with all of these instances randomly sampled from the original imdb sentiment dataset of 25k reviews @xcite . the counterfactual sets d train cad and d test cad are the revised versions of d train id and d test id , as rewritten by mechanical turk workers recruited by @xcite . see appendix b for further details. we will also test on out-of-distribution (ood) data from amazon @xcite , semeval @xcite and yelp @xcite (we note these datasets as sentence encoders to obtain ϕ(x), we use the sentence encoding frameworks sbert and sim-cse @xcite ; @xcite ). the main results are presented with sroberta large and simcse-roberta large and they are kept frozen at all times. appendix b lists additional details; appendix a shows results for other encoders. baselines as a baseline for our fewcounterfactuals-based approaches, we present results (original) from a classifier trained on twice the amount of original (unrevised, in-distribution) samples. further, we also investigate competitive counterfactual-based approaches as proposed by @xcite , who leverage identified causal sentiment words and a sentiment lexicon to generate counterfactuals in the input space (which we subsequently embed with the same sentence encoders ϕ as before). they adopt three settings, with increasing human supervision, to identify causal words: (i) predicted from top: 32 causal words were identified automatically for imdb; (ii) annotated from top: a human manually marked 65 words as causal from a top-231 word list deemed most relevant for sentiment; and (iii) annotated from all: a human labeled 282 causal words from the full 2,388 word vocabulary. training and evaluation for all presented approaches, the classifier f is implemented by logistic regression with l2 regularization, where the regularization parameter λ is established by 4-fold cross-validation. 1 the results presented further in the main paper body report are obtained by training on the complete training set (i.e., all folds). the mean offset + regression model of §2.2, to artificially generate counterfactuals, is implemented by linear regression with ordinary least squares. the weighted and paired classifiers of §2.1 are trained on n samples from d train id together with k counterfactuals sampled from d train cad . to evaluate our classifiers with generated counterfactuals, as described in §2.2, we train on the n original samples, k manual counterfactuals and nk generated counterfactuals. the original baseline uses 2 ⋅ n original samples d k/2 positive counterfactuals, and (ii) n additional original samples (for original baseline). the reported accuracies are averaged across the 50 seeds. ## conclusion we explored improving the robustness of classifiers (i.e., make them perform well also on out-ofdistribution, ood, data) by relying on just a few manually constructed counterfactuals. we propose a simple strategy, learning from few (original, counterfactual) pairs how to transform originals d id into counterfactuals d cad , in a document vector representation space: shift an original document with the mean offset among the given pairs. thus, using just a small number (1% of |d id |) of manual counterfactuals, we outperform sentiment classifiers trained using either (i) 100% extra original samples, or (ii) a state-of-the-art (lexicon-based) counterfactual generation approach. thus, we suggest that additional annotation budget is better spent on counterfactually revising available annotations, rather than collecting similarly distributed new samples.