DRAFT : Dense Retrieval Augmented Few-shot Topic classifier Framework (2023)

Abstract: With the growing volume of diverse information, the demand for classifying arbitrary topics has become increasingly critical. To address this challenge, we introduce DRAFT, a simple framework designed to train a classifier for few-shot topic classification. DRAFT uses a few examples of a specific topic as queries to construct Customized dataset with a dense retriever model. Multi-query retrieval (MQR) algorithm, which effectively handles multiple queries related to a specific topic, is applied to construct the Customized dataset. Subsequently, we fine-tune a classifier using the Customized dataset to identify the topic. To demonstrate the efficacy of our proposed approach, we conduct evaluations on both widely used classification benchmark datasets and manually constructed datasets with 291 diverse topics, which simulate diverse contents encountered in real-world applications. DRAFT shows competitive or superior performance compared to baselines that use in-context learning, such as GPT-3 175B and InstructGPT 175B, on few-shot topic classification tasks despite having 177 times fewer parameters, demonstrating its effectiveness.

## introduction with the prevalence of the internet and social media, there is a significant demand for classifying or detecting texts related to specific topics within the vast amount of information pouring in from the internet. for instance, on social media platforms where an overwhelming volume of content is generated, there may exist a need to monitor and filter content associated with particular issues (e.g., drugs). additionally, with the remarkable progress in large language models (llms) @xcite @xcite @xcite in recent times, there is also an increasing * work done while at neosapience. demand to detect specific topics within the text generated by llms. the use of llms often contains ethical concerns with the issues of hallucination, as they possess the capability to generate morally inappropriate or unintended content @xcite . however, as the amount of content shared on social media and generated by llms increase exponentially, verifying each piece of content becomes challenging for individuals. in natural language processing (nlp), research related to the challenges mentioned earlier can be considered a topic classification task since the demand for automatically classifying texts on a specific topic exists. recent pre-trained language models @xcite @xcite have gained considerable recognition for their ability to achieve high performance in topic classification tasks. to train a topic classification model, it is common practice to rely on supervised learning using a training dataset with labeled data. most existing methods on a topic classification have primarily relied on benchmark datasets @xcite . to the best of our knowledge, they have predominantly focused on improving model performance on benchmark datasets that involve a limited number of topics rather than addressing long-tailed arbitrary topics in real-world scenarios. regarding real-world applications, the availability of labeled datasets that cover diverse topics is often limited due to the substantial cost of building such datasets. this constraint poses a challenge for directly deploying topic classification models in practical scenarios, thereby calling for a solution that enables their flexible application. few-shot classification, which performs classification using few text examples, can be applied even without a training dataset for a predefined topic, enabling its extension to tasks classifying a diverse range of arbitrary topics. to the best of our knowledge, existing few-shot classification methods operate exclusively on tasks with two or more defined classes and cannot conduct in oneclass classification tasks @xcite @xcite . however, by leveraging llms with in-context learning @xcite @xcite , we can conduct one-class few-shot topic classification tasks across various topics, showcasing superior performance. this approach garnered significant attention due to its applicability in scenarios with limited labeled data @xcite @xcite . however, successful implementation of in-context learning relies on llms with billions of parameters @xcite . such models suffer from high computational costs, extensive resource requirements, and slow inference speed to apply in real-world applications. to address the few-shot topic classification more efficiently in real-world applications, we propose a simple framework called dense retrieval augmented few-shot topic classifier framework (draft), which can classify arbitrary topics given limited labeled data. draft uses a dense retriever model @xcite to construct customized dataset, using examples of a target topic provided as queries. subsequently, a pre-trained language model is finetuned with the customized dataset for the topic classification task. we evaluate the performance of draft by conducting experiments on general benchmarks and manually constructed datasets with 291 topics. the former datasets are widely regarded as benchmark datasets in recent classification research. in contrast, the latter datasets are manually constructed to replicate real-world scenarios, allowing for a thorough assessment of the diverse topic classification capability. they are used for one-class classification tasks where only a single topic is defined. our experiment results demonstrate that draft consistently achieves competitive or superior performance compared to llms, such as gpt-3 175b and instruct-gpt 175b, which have 177 times more parameters on topic classification tasks. these findings provide strong empirical support for the efficacy of draft in tackling few-shot topic classification tasks. the contributions are summarized as follows: in-context learning with llms recent llms @xcite @xcite @xcite ., query example 1 query example 2 query example 3 query example n ... positive samples negative samples query encoder language model mlp layer data collection customized dataset n number of queries passage encoder semantic search classification results training a classifier building customized dataset (a) (b) queries (positive samples) passages retrieved as positive samples through semantic search remaining passages in data collection (negative samples) figure 1: (a) overall pipeline of draft. draft receives n queries as input, and a trained classifier is only used in the test phase. (b) illustration of mqr in two-dimensional space. a circle represents the normalized embedding space of texts in data collection. for each query, passages only within an angle size θ, calculated as a threshold from n query vectors, are retrieved as positive samples, while others are classified as negative samples. ## method in this section, we begin by describing the application of draft to a simple binary classification and elaborate on the extension of draft. draft comprises two stages for few-shot topic classification: (1) constructing customized dataset from multiple queries using a dense retriever model and ## experiments to evaluate the performance of draft, we conduct three experiments with limited labeled data in classification tasks. ## discussion potential points in draft in draft, the number of queries and the subspace size are considered the most crucial factors among various hyperparameters. queries provide information related to the target topic, while subspace size determines how many passages related to each query are retrieved from data collection. they influence the performance of draft, as they significantly impact the construction of customized dataset, which directly impacts the process of training a classifier. we investigate the relationship between the number of queries and the subspace size by varying both variables using agnews. the experiments are repeated five times with different configurations, and the average accuracy results are presented in figure 2 . increasing the number of queries exhibits a positive correlation with accuracy when the subspace size is fixed. the highest accuracy of 88% is achieved with a subspace size of 10 and 50 queries. although the increase in the number of queries is limited to 50 due to computational resource constraints, the consistent trend implies that more queries can improve performance. we find that draft has the potential for improvement with an increased number of queries. unlike draft, llms can suffer from degraded performance due to the majority label bias @xcite when the number of examples from the same class increases in few-shot samples. in section 4.2, instructgpt 175b demonstrates a noticeable decline in performance as the value of k increases in k-shot settings. considering majority label bias and experimental results, draft shows a more robust performance than llms. ## conclusion in this study, we introduce draft, a simple but effective approach that first applies a dense retriever model for few-shot classification across diverse topics. despite possessing 177 times fewer parameters than llms, draft demonstrates superior performance in few-shot topic classification. these results imply the effectiveness of draft in classifying a diverse array of real-world topics. we anticipate that draft holds the potential to be implemented in practical contexts and actively contribute to addressing diverse societal challenges, including those encompassing specific topics.