Zero-shot Topical Text Classification with LLM s - an Experimental Study (2023)

Abstract: Topical Text Classification (TTC) is an ancient, yet timely research area in natural language processing, with many practical applications. The recent dramatic advancements in large LMs raise the question of how well these models can perform in this task in a zero-shot scenario. Here, we share a first comprehensive study, comparing the zero-shot performance of a variety of LMs over TTC23, a large benchmark collection of 23 publicly available TTC datasets, covering a wide range of domains and styles. In addition, we leverage this new TTC benchmark to create LMs that are specialized in TTC, by fine-tuning these LMs over a subset of the datasets and evaluating their performance over the remaining, held-out datasets. We show that the TTC-specialized LMs obtain the top performance on our benchmark, by a significant margin. Our code and model are made available for the community. We hope that the results presented in this work will serve as a useful guide for practitioners interested in topical text classification.

## introduction the recent emergence of transformer-based language models (lms) has led to significant breakthroughs in various nlp tasks @xcite . in particular, lms have shown dramatic performance improvements in text classification (e.g., @xcite ), which is one of the most common use cases considered by nlp practitioners. notably, these improvements are also present in the challenging setting of zero-shot classification, where no labeled data are available for the target categories @xcite . manual collection of labeled data is known to be notoriously costly, complicated, and time consuming, representing a major blocker in the adoption of text classification solutions in practice. hence, gaining a better understanding of the current performance of lms in this zero-shot scenario is a timely issue with great practical importance. text classification is a relatively broad umbrella term, covering tasks such as (i) sentiment analysis (sa) in various forms @xcite ; (ii) style detection -e.g., spam filtering @xcite , authorship attribution, etc.; and (iii) topical text classification (ttc), where the goal is to associate a text example with one or more topics out of a pre-defined set of topics, or categories (e.g., @xcite ). 2 the zero-shot scenario is of crucial importance for the latter, ttc task. specifically, for tasks such as sa, the categories are typically the same between different use cases -positive, negative, and neutral. hence, in principle, one can rely on existing sa labeled datasets to fine-tune the lm. in contrast, in ttc the list of categories is likely to change from one downstream task to another, hence relying on previously labeled datasets is not straightforward. thus, the main motivation for the present work relies on three intertwined pillars -(i) the special importance of the zero-shot setup for ttc; (ii) the wide range of practical ttc use cases -e.g., in domains such as customer-care, healthcare, and legal @xcite ; and (iii) the assumed potential of lms to obtain strong zero-shot performance in ttc tasks. correspondingly, here we report the results of a comparative study -to the best of our knowledge, first of its kind -focused on the zero-shot ttc performance of a plethora of lms over a large collection of ttc datasets. specifically, we consider 23 datasets, across a variety of domains, styles and complexity levels, including news, legal contracts, chats and more. moreover, while previous work used lms for zero-shot ttc without specific tuning for this task, here, we suggest that specializing lms in the ttc task can provide significant performance gains. to that end, we show that one can fine-tune a lm over existing ttc labeled datasets to significantly improve its performance over new ttc tasks, with categories never seen before. we show the value of this approach in zero-shot ttc by training the lm on a subset of the datasets, and evaluating its performance on the remaining, held-out datasets. to summarize, the main contribution of this paper is three-fold: (1) we introduce ttc 23 , a heterogeneous collection of 23 ttc datasets, that we propose as a new benchmark for this task; (2) we share the results of a comprehensive comparative study of the zero-shot performance of various lms at different sizes over ttc 23 , addressing both prediction performance and run-time considerations; and finally, (3) we show that ttc 23 can be leveraged to create models with significantly enhanced zero-shot performance in new ttc tasks. ## the ttc 23 benchmark our aim was to gather a diverse set of datasets containing multiple domains and styles that could serve as a training set for fine-tuning models, and also as a ttc evaluation benchmark. we collected datasets for ttc by searching on huggingface, kaggle, as well as in related papers, focusing on datasets where the input is between one and a few sentences, in english, and labeled in a multi-class or multi-label setting. the collection covers, among others, the following domains and styles: legal -both formal (e.g., ledgar) and informal (e.g., legal advice reddit); finance -both from the news (reuters) and tweets (financial tweets); news items -both headlines (news category classification headline) and discussion threads (20 newsgroups); and chatbot queries -in banking (banking77) and multi-domain (massive). overall we collected 23 datasets, presented in ## experimental setup next we describe the setup in which we evaluate the different methods for zero-shot ttc. ## results and analysis we present results averaged on all datasets in focusing on encoder-only models, and the mc** column, fine-tuning on ttc 23 yields better performance compared to the qa-based model, that uses a different training scheme. ## conclusions ttc is an ancient problem in computer science research, dating back to 1961 @xcite . the recent advancements in large lms led us to explore how well they perform on this problem in a zero-shot setting. in this paper we introduce a comprehensive evaluation of existing approaches to zero-shot ttc over a diverse set of datasets, ttc 23 . our results indicate that large lms indeed exhibit a significant improvement over the smaller encoder-only models that were the main focus on previous studies on zero-shot ttc (e.g., @xcite ). furthermore, we show that fine-tuning roberta and deberta models as well as flan-t5-xxl over existing ttc datasets can significantly boost their respective zero-shot performance on new ttc datasets, with category names not included in the training data. in other words, even llms that represent very strong baselines, can be further significantly improved with additional fine-tuning focused on the target task. our run-time analysis has shown that this superior performance comes with a cost of additional run-time, something the practitioner needs to take into account. our fine-tuned flan-t5-xxl model obtained very impressive zero-shot performance -macroaveraged f1 of 67.32 when averaging across 14 multi-class datasets. nonetheless, these results are typically still not satisfactory in practice, and we expect that a traditional process of collecting labeled data for the target task under consideration to further fine-tune the model would be required. that said, a strong zero-shot ttc model can serve as an excellent starting point to perform domain adaptation on a target dataset, e.g., by means of self-training @xcite or active learning @xcite , as we plan to explore in future work.