BYOC : Personalized Few-Shot Classification with Co-Authored Class Descriptions (2023)

Abstract: Text classification is a well-studied and versatile building block for many NLP applications. Yet, existing approaches require either large annotated corpora to train a model with or, when using large language models as a base, require carefully crafting the prompt as well as using a long context that can fit many examples. As a result, it is not possible for end-users to build classifiers for themselves. To address this issue, we propose a novel approach to few-shot text classification using an LLM. Rather than few-shot examples, the LLM is prompted with descriptions of the salient features of each class. These descriptions are coauthored by the user and the LLM interactively: while the user annotates each few-shot example, the LLM asks relevant questions that the user answers. Examples, questions, and answers are summarized to form the classification prompt. Our experiments show that our approach yields high accuracy classifiers, within 79% of the performance of models trained with significantly larger datasets while using only 1% of their training sets. Additionally, in a study with 30 participants, we show that end-users are able to build classifiers to suit their specific needs. The personalized classifiers show an average accuracy of 90%, which is 15% higher than the state-of-the-art approach.

## overview of byoc in this section, we present the process through which a user can build a personalized classifier using byoc. we do so by way of an example: alice wants to build an email classifier to categorize emails as "important" and "unimportant". the overall flow is summarized in fig. 1 . ## prompting for personalized classification in this section, we first present the formal definition of a classifier model that uses class descriptions, and then present a method to interactively compute those class descriptions. ## evaluation in this section, we evaluate the quality of our proposed approach. our main experiment is a personalized classification use-case where we attempt to classify emails according to the user's provided criteria. for comparison, we also evaluate against an existing dataset for the classification of long text. ## conclusion this paper presented a novel approach to fewshot text classification using large language models (llm). by having the user cooperate with an llm we were able to build highly accurate classifiers while annotating a very minimal amount of data. for many practical applications of classification, our method eliminates the need for large annotated corpora. we have achieved high accuracy classifiers that are competitive with models trained with orders of magnitude larger datasets. we also allow end-users to create their own classifiers without expecting them to be proficient in prompt engineering. through an interactive pro-cess of question-answering and minimal annotation we are able to make llm capture the essential features of each class, enabling a precise classification. our experiments highlights the importance of both personalizing and user-interaction with the llm. first, we show that users do not know how to instruct llms. second, we demonstrate the need for personalization. a fine-tuned model trained on all users at once is outperformed by byoc by 13%, despite outperforming all few-shots baselines. this suggests that there is no one-size-fits-all model that can be applied to every-user. our study involving real participants also confirms the usability and effectiveness of our approach. end-users are able to build classifiers for themselves exhibiting an average accuracy of 90%, rating their experience 8.6 out of 10. overall, our methodology enables users to leverage the power of llms for their specific needs, opening up new possibilities for practical applications of text classification in various domains. this work is available as a commercial product in the bardeen web automation platform foot_2 , where byoc enables end-users to classify several forms of data. popular use-cases include sales lead, inbox classification, and support request categorization.