Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models (2024)

Abstract: Text classification is a crucial task encountered frequently in practical scenarios, yet it is still under-explored in the era of large language models (LLMs). This study shows that LLMs are vulnerable to changes in the number and arrangement of options in text classification. Our extensive empirical analyses reveal that the key bottleneck arises from ambiguous decision boundaries and inherent biases towards specific tokens and positions.To mitigate these issues, we make the first attempt and propose a novel two-stage classification framework for LLMs. Our approach is grounded in the empirical observation that pairwise comparisons can effectively alleviate boundary ambiguity and inherent bias. Specifically, we begin with a self-reduction technique to efficiently narrow down numerous options, which contributes to reduced decision space and a faster comparison process. Subsequently, pairwise contrastive comparisons are employed in a chain-of-thought manner to draw out nuances and distinguish confusable options, thus refining the ambiguous decision boundary.Extensive experiments on four datasets (Banking77, HWU64, LIU54, and Clinic150) verify the effectiveness of our framework. Furthermore, benefitting from our framework, various LLMs can achieve consistent improvements. Our code and data are available in https://github.com/Chuge0335/PC-CoT .

## preliminary study setup we evaluate various llms to understand factors influencing classification accuracy. our experiments include the gpt-3.5 series, llama2-chat, and qwen-chat models. we use the challenging portions of the banking77 and liu54 datasets (details in section 4) to reduce experimental costs. due to space constraints, the main results are summarized in the main text, with detailed dataset samples and additional results provided in appendix a. ## experiments to evaluate the performance of our framework, we choose four widely recognized classification datasets, including: banking77 (casanueva et al., 2020) with 77 labels in the banking services domain; hwu64 (casanueva et al., 2020) of 64 intents from 21 different domains; liu54 @xcite of 54 classes gathered from amazon mechanical turk; clinc150 @xcite covering 150 classes and 10 domains. notice that we sample challenging segments from these datasets to reduce the experiment cost. the comprehensive sampling process and supervised performance are here we use the instruction versions of models for better instruction following ability. we set the temperature to 0 for all models across different experiments. ## conclusions in this paper, we investigate the vulnerability of llms in terms of option quantity and arrangement for text classification tasks. our empirical analysis reveals that the main challenges lie in their ambiguous decision boundaries and inherent biases towards tokens and positions. to mitigate these challenges, we propose a novel two-stage classification framework for llms, involving reduction and comparison stages. this framework strategically employs pairwise comparisons to address ambiguity and bias. additionally, we introduce innovative self-reduction algorithms, including itr and cbwr strategies, and our pc-cot, a contrastive chain-of-thought technique, to enhance the framework's utility. extensive experiments demonstrate our framework can bring improvement to the stability and debiasing for various prompting techniques. our itr and cbwr methods prove highly effective for reduction, while our pc-cot exhibits superior performance in classification accuracy.