Definitions Matter: Guiding GPT for Multi-label Classification (2023)

Abstract: Large language models have recently risen in popularity due to their ability to perform many natural language tasks without requiring any fine-tuning. In this work, we focus on two novel ideas: (1) generating definitions from examples and using them for zero-shot classification, and (2) investigating how an LLM makes use of the definitions. We thoroughly analyze the performance of GPT-3 model for fine-grained multi-label conspiracy theory classification of tweets using zero-shot labeling. In doing so, we asses how to improve the labeling by providing minimal but meaningful context in the form of the definitions of the labels. We compare descriptive noun phrases, human-crafted definitions, introduce a new method to help the model generate definitions from examples, and propose a method to evaluate GPT-3’s understanding of the definitions. We demonstrate that improving definitions of class labels has a direct consequence on the downstream classification results.

## introduction recent success of large language models (llms) is due to their superior performance on various tasks @xcite @xcite , such as text generation @xcite , summarization @xcite , question answering @xcite , information retrieval @xcite , machine translation @xcite , and inductive reasoning @xcite @xcite . however, llms had much less success with solving specific tasks @xcite , such as text classification, where they still lag behind fine-tuned transformer models @xcite . class definitions are commonly communicated to llms in the form of zero-shot prompts and few-shot exam- * equal contribution ples, but both approaches are inferior to "defining" classes via a train set of annotated examples. misinformation has been a major research topic in the last few years, due to its significant impact on society. social media have been a large vector of fake-news, especially during the covid-19 pandemic. we revisit the problem of llm zeroshot classification on the use-case of fine-grained multi-label conspiracy theory detection in tweets @xcite , using gpt-3 model @xcite . we show that the challenging task of distinguishing conspiracy theories related to covid-19 provides valuable insights into the capabilities and limitations of gpt-3. in particular, we aim at exploring the role played by the definition of the labels in zero-shot classification and explore the ability of gpt-3 in generating such definitions from a small set of labelled examples. additionally, we propose a method to test the ability of gpt-3 to understand the definitions and apply them to classification foot_0 . ## related work zero-shot learning @xcite is a machine learning problem where a model predicts classes without access to training data. it is based on the knowledge transfer from seen to unseen classes @xcite , using auxiliary information about the classes that can be provided to the model, such as text descriptions @xcite , semantic attributes @xcite or concepts defined in ontologies @xcite . within the domain of natural language processing (nlp), especially text classification, the zero-shot learning approach has been used in a variety of tasks, such as argument @xcite , keywords @xcite , intent @xcite and utterance @xcite classification. unlike the traditional approach for zero-shot text classification, llms are capable of solving zero-shot classification with no auxiliary information @xcite @xcite . however, there is evidence that llms can outperform crowd workers in zero-shot text annotation tasks @xcite . a different approach is to fine-tune the llm with examples made of manually crafted lists of dictionary/encyclopedia entries related to the target label @xcite or to fine-tune smaller pre-trained language models with examples retrieved @xcite or generated by llms @xcite @xcite . approaches that use class definitions for constructing zero-shot classifiers and prompts have been proposed, both for cnns and transformers @xcite , and for llms @xcite . however, they do not experiment with different variants of class definitions, but use one definition type for classification and report the results. additionally, to the best of our knowledge, no previous research has used llms to generate a class-specific definition directly from examples and used that information for zero-shot text classification as we investigate in this work. semantic abilities of llms are commonly evaluated on nl understanding and reasoning tasks @xcite , but work on targeted evaluation of fine-grained semantic properties is scarce. @xcite propose to evaluate the llm's comprehension of query-related concepts by using a knowledge graph. to the best of our knowledge, there is no previous work focused on the ability llms to understand and apply definitions. ## methodology in this section, we first describe the dataset used for conspiracy theory classification. then, we present the approaches to definition-based zero-shot classification using gpt-3, and describe the method for evaluation of gpt-3's understanding of definitions. in this work, we use the term 'definition' to denote any additional explanation of the textual labels used to describe the class. ## discussion & future work while classification of conspiracy theories using eg definitions is done in a zero-shot fashion, the generation of the definitions still relies on annotated examples. this is different than standard in-context few-shot classification as these examples do not need to be part of the classification prompt. indeed, it can be seen as a way to compress the information from few-shot examples into a shorter descriptive context that can be appended in the zero-shot prompt. further experiments could explore this approach and compare it to standard in-context few-shot classification. the correlation tests of definition understanding in section 4.2 support the claim that gpt-3 can indeed interpret and apply the definitions correctly. this is complemented by the results in section 4.1 which show that better definitions lead to better results. however, further testing should be done on more llms and with other corpora. such experiments are an interesting direction for future work with the potential to shed light on the semantic capabilities of llms. the results of the definition-based zero-shot classifiers imply several practical recommendations and potential applications, all of which represent topics for future work. they include the use of the (high recall) classifiers to create more balanced samples for labeling, application of the classifiers to detect annotation errors, recommendations for mitigation of the low precision, and use of eg definitions for few-shot learning. more details on these topics can be found in appendix c. ## conclusion in this work, we analyze the impact of label definitions on the performance of gpt-3 zero-shot classification, on a challenging task of fine-grained conspiracy theory detection. we show that the use of better definitions leads to a significant gain in most metrics (mcc, precision, f1). we also demonstrate an approach of generating definitions from examples. human-written definitions still provide better results, while example-generated definitions show promising performance. additionally, we successfully tested gpt-3's ability to understand and apply these definitions for classification.