A Simple Yet Strong Domain-Agnostic De-bias Method for Zero-Shot Sentiment Classification (2023)

Abstract: Zero-shot prompt-based learning has made much progress in sentiment analysis, and considerable effort has been dedicated to designing high-performing prompt templates. However, two problems exist; First, large language models are often biased to their pre-training data, leading to poor performance in prompt templates that models have rarely seen. Second, in order to adapt to different domains, re-designing prompt templates is usually required, which is time-consuming and inefficient. To remedy both shortcomings, we propose a simple yet strong data construction method to de-bias a given prompt template, yielding a large performance improvement in sentiment analysis tasks across different domains, pre-trained language models, and prompt templates. Also, we demonstrate the advantage of using domain-agnostic generic responses over the in-domain ground-truth data.

## introduction over the past few years, zero-shot prompt-based learning has become a de facto standard in many natural language processing (nlp) tasks where training data is unavailable. for sentiment analysis, much effort has also been dedicated to designing effective prompt templates to trigger the capability of large language models (llms) such as roberta @xcite and gpt @xcite to predict sentiment polarities, e.g., positive or negative. a prompt template typically consists of prompt text and a label token set corresponding to the sentiment class. @xcite demonstrates that it was {good,ok,bad}. is a highperforming prompt template for sentiment analysis task. as figure 1 shows, the input to llm is a must-watch movie. it was @xcite ., and the token ok is the most probable word over the label token set. ## result and analysis interestingly, domain-agnostic generic responses, such as "thank you very much," improve the sentiment classification performances, which touches upon our rq2: how does the quality of our domain-agnostic silver data compared to the in-domain ground-truth data from each of seven sentiment analysis datasets? ## related works zero-shot sentiment classification has attracted a lot of attention as it does not require training data, offering advantages in many real-world use cases, particularly in low-resource scenarios. at the core of this task is the construction of an appropriate prompt, which transforms the sentiment classification task into a fill-in-the-blank format. @xcite explored various prompts for sentiment analysis and identified several highly effective prompt templates. similarly, @xcite experimented with prompt templates in the context of aspect-based sentiment analysis. however, a challenge comes from the sensitivity of accuracy to prompt templates, where even a slight modification in prompt templates can result in a large performance change in zero-shot sentiment analysis tasks. @xcite argues that this "instability" arises from the bias of language models towards predicting certain answers that are common in the pre-training data. to address this issue, they estimate the model's bias towards each sentiment polarity class by requesting its prediction when presented with the prompt and a content-free input, such as "n/a." subsequently, they determine calibration parameters to ensure a uniform probability distribution for this input across all sentiment polarity classes. later on, @xcite borrowed the idea from machine translation research and proposed a noisy channel to shift the probability distribution for few-shot text classification. our work aligns with the approach proposed by @xcite but with a notable distinction. rather than considering the meaningless strings, we go a step further by incorporating positive, neutral and negative sentiment-labeled instances to address the bias in llms. ## conclusion and future in this work, we propose a simple yet effective domain-agnostic data construction method to enhance sentiment classification tasks. our method was evaluated using three popular llms, namely bert, roberta, and gpt2-xl, along with two commonly employed prompt templates. the results show significant improvements in task performance. also, we answered the research questions and demonstrated that our constructed domainagnostic data is superior to in-domain data in terms of overall performance. in the future, we plan to experiment with other dialogue corpora to assess the generalization capabilities of the proposed de-bias method.