Text Classification via Large Language Models (2023)

Abstract: Despite the remarkable success of large-scale Language Models (LLMs) such as GPT-3, their performances still significantly underperform fine-tuned models in the task of text classification.This is due to (1) the lack of reasoning ability in addressing complex linguistic phenomena (e.g., intensification, contrast, irony etc); (2) limited number of tokens allowed in in-context learning. In this paper, we introduce C lue A nd R easoning P rompting (CARP). CARP adopts a progressive reasoning strategy tailored to addressing the complex linguistic phenomena involved in text classification: CARP first prompts LLMs to find superficial clues (e.g., keywords, tones, semantic relations, references, etc), based on which a diagnostic reasoning process is induced for final decisions. To further address the limited-token issue, CARP uses a fine-tuned model on the supervised dataset for k NN demonstration search in the in-context learning, allowing the model to take the advantage of both LLM’s generalization ability and the task-specific evidence provided by the full labeled dataset. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks, 97.39 (+1.24) on SST-2, 96.40 (+0.72) on AGNews, 98.78 (+0.25) on R8 and 96.95 (+0.6) on R52, and a performance comparable to SOTA on MR (92.39 v.s. 93.3). More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups. Specifically, using 16 examples per class, CARP achieves comparable performances to supervised models with 1,024 examples per class.

## introduction large language models (llms) @xcite @xcite @xcite @xcite have shown the ability for in-context learning (icl). given a few demonstration examples, llms are prompted to generate results for a new test example, and have achieved performance comparable to supervised baselines or even state-of-the-art results in a variety of natural language processing (nlp) tasks such as question answering @xcite , natural language inference, @xcite , named entity recognition @xcite , relation extraction @xcite and information extraction @xcite . in spite of the success, llms with icl still significantly underperform fine-tuned models for text classification. this is due to two reasons: (1) text classification requires models with more powerful reasoning abilities to resolve complex linguistic phenomenon including clause composition (e.g., concession, negation, intensification), irony, etc. recent efforts to improve llms' reasoning capabilities @xcite @xcite ) mainly focus on tackling math problems, and thus are not tailored to addressing the reasoning process necessary for the multitude of intricate linguistic phenomena in text classification; (2) the number of demonstration examples allowed in in-context learning is limited, e.g., the longest context allowed for gpt-3 is 4,096 subtokens. therefore, llms are only able to take the advantage of a small proportion of the training set, performing well below supervised baselines; in this paper, we introduce clue and reasoning prompting (carp), an extensible, annotationfree and efficient framework for text classification via large language models. to address the reasoning process necessary for handling the linguistic phenomena in text classification, carp decomposes the reasoning process into three steps, where llms are first prompted to find superficial clues (e.g., keywords, tones, semantic relations, etc) in the given text; next, carp treats the clues and input as premises and induce a diagnostic reasoning process; and finally determine the final label considering the above two steps. we find this progressive reasoning strategy to be effective in enhancing llms' ability in language reasoning involved in text classification. due to the limited number of tokens allowed in context, a more effective demonstration search is needed. carp uses a fine-tuned (ft) model on the supervised dataset for knn demonstration search for icl. since the fine-tuned model is trained based on taskspecific labels, it guarantees that retrieved samples are close to the input sequence with respect to the task. ft-based demonstration search provides a channel to connect llms with the full training set, in spite of the limited number of tokens allowed in demonstrations. this strategy lets the model take the advantage of both the llms' generalization abilities and all task-specific evidence provided by the training dataset. remarkably, carp yields new sota performances on four out of 5 widely-used text-classification benchmarks, 97.39 (+1.24) on sst-2, 96.40 (+0.72) on agnews, 98.78 (+0.25) on r8 and 96.95 (+0.6) on r52, and a performance comparable to sota on mr (92.39 v.s. 93.3). more importantly, we find that carp delivers impressive ability on low-resource and domain adaptation setups with orders of magnitude fewer training examples. specifically, carp achieves comparable performances with 16 examples per class to supervised models trained on the full training set containing more than 1 thousand examples per class. this demonstrates the capabilities of carp in real-world text classification cases where training data is limited. ## clues collecting and reasoning to enhance the models' reasoning ability in addressing linguistic phenomenon tailored to text classification, we propose a progressive reasoning strategy that involves clue collection, reasoning and decision making. this process also mimics how human decisions: where we first collect evidence from the input, separating chaff from wheat; next we piece together local evidence to form a global picture, which leads to final decision making. next we first given an overview of the the clue collecting and reasoning process, and then describe implementation details. ## experiments in order to evaluate the effectiveness of the proposed method, we conduct experiments on two setups: (1) full training setup, where the model has the access to the full training data; and (2) lowresource setup, where the model can only access partial training dataset. the low-resource setup we conduct experiments on five widely-used datasets, including sst-2 @xcite , r8, r52 foot_2 , agnews @xcite and movie review (mr) @xcite . more details of the benchmarks and low-resource datasets can be found in appendix d. for zero-shot and few-shot experiments, we use @xcite ) (text-davinci-003, 175b) as the backbone. due to the input token limitation, we use @xmath0. prompts on the five datasets are shown in appendix h. model hyper-parameters can be found in ## ablation studies 6.1 impact of the number of demonstrations we explore the effect of the number of demonstrations in prompts using sst-2 . results for carp using different sampling strategies are shown in figure 2 . as can be seen, performances improve as the number of demonstrations increases, which is in line with our expectation. ## conclusion in this paper, we introduce clue and reasoning prompting (carp) for text classification task. carp yields new sota performances on 4 out of 5 widely-used text-classification benchmarks. more importantly, we find that carp delivers impressive abilities on low-resource and domainadaption setups. in the future, we would like to explore carp on more natural language understanding tasks.