Zero-Shot Multi-Label Topic Inference with Sentence Encoders and LLM s (2023)

Abstract: In this paper, we conducted a comprehensive study with the latest Sentence Encoders and Large Language Models (LLMs) on the challenging task of “definition-wild zero-shot topic inference”, where users define or provide the topics of interest in real-time. Through extensive experimentation on seven diverse data sets, we observed that LLMs, such as ChatGPT-3.5 and PaLM, demonstrated superior generality compared to other LLMs, e.g., BLOOM and GPT-NeoX. Furthermore, Sentence-BERT, a BERT-based classical sentence encoder, outperformed PaLM and achieved performance comparable to ChatGPT-3.5.

## introduction topic modeling and inference have been widely studied in the nlp literature @xcite @xcite . in this paper, we focus on zero-shot approaches @xcite @xcite for inferring topics from documents where both the document and topics were never seen by a model previously. for developing zero-shot methods, we exclusively focus on leveraging the recent powerful sentence encoders and large language models (llms) due to their recent success in a wide variety of nlp tasks. the problem of zero-shot topic inference can be described using an intuitive example where the end user (possibly a domain expert) is actively involved in the inference process. consider that the domain expert is analyzing a large volume of health articles and wants to automatically infer topics from those articles, including topics like "autoimmune disorders", "heart health", "arthritis", etc. for this reallife use case, the user will provide the collection of documents as well as a set of topics to be used as labels for categorizing the documents. additionally, the user may also provide a list of relevant keywords/clues associated with each topic, which can be used as expert guidance for the inference process. the zero-shot topic inference algorithm then infers topics for each document. naturally, the zero-shot topic inference is a hard task, and only limited previous works studied this problem @xcite @xcite . however, with the recent developments in llms and pre-trained sentence embeddings like @xcite ; @xcite ; @xcite , we have observed significant performance boosts in many downstream zero-shot nlp tasks. inspired by these, we decided to explore zero-shot methods by leveraging various sentence encoders @xcite , language-agnostic sentence representations (laser) @xcite , sentence-bert (sbert) @xcite , and universal sentence encoder (use) @xcite ] and recent llms @xcite , palm @xcite , gpt-neox @xcite , chat-gpt @xcite ] for topic inference. it is important to highlight that for all of our experiments with llms (except chatgpt-3.5), we did not follow a prompting approach. instead, to make an apple-to-apple comparison with classic sentence encoders, we generated sentence embeddings from these llms (except chatgpt-3.5) and used the embeddings directly to infer topics. in summary, we conducted extensive experiments with classic sentence encoders and llms on the zero-shot topic inference task and, consequently, established a comprehensive benchmark for future work in this direction. experiment results with multiple real-world data sets, including online product reviews, news articles, and healthrelated blog articles, show that among all the models chatgpt-3.5 is superior in terms of generality compared to others, while sentence-bert performs exceptionally well and surpasses llms such as palm, bloom, and gpt-neox. ## related work this work is built upon prior research from multiple areas, including topic modeling and categorization @xcite @xcite , text annotation @xcite @xcite , zero-shot learning @xcite , sentence embeddings @xcite , large language models @xcite etc. a brief discussion on each area and how this work is positioned concerning the state-of-the-art is as follows. ## problem statement the traditional topic inference task is defined as: definition 1 given a collection of documents d and a set of pre-defined topics t , infer one or more topics in t for each document @xmath0. thanks to the pre-defined set of topics t , the traditional topic inference task can benefit from fine-tuning based on a carefully designed training set for supervised learning. on the other hand, we follow the idea of definition-wild zero-shot-text classification coined by @xcite , which is as follows: definition 2 definition-wild 0shot-tc aims at learning a classifier f(•): @xmath1, where classifier f(•) never sees y-specific labeled data in its model development. extending on top of definition-wild (0shot-tc), we formalize our task from the user's standpoint in the following fashion: definition 3 given a collection of documents @xmath2, d 2 , ..., d n }, a user x and a set of user-defined topics t @xmath3, t 2 , ..., t m } provided in real-time, annotate each document d @xmath4. in this dynamic setting, different users may provide varying sets of topics for the same dataset based on their specific application needs and goals. customized training datasets in advance are no longer feasible as the target topics are provided in real time. we assume that each topic t is represented by a word or phrase, and users can include additional topic-related keywords k t . essentially, our ad-hoc problem assumes that the user, typically a domain expert in a specific field (e.g., a cardiologist or a business analyst), provides the documents, target topic, and optional keywords in real time. topics in a document may not be explicitly mentioned but rather implied through related keywords. for example, a document discussing "mental health" may not contain the exact phrase but may reference related terms like "depression", "anxiety", and "antidepressant". these implicit topics are equally significant and should be annotated alongside explicit topics. while userprovided keywords help, it is challenging to create a comprehensive list capturing all possible descriptions. also, the presence of a keyword does not guarantee the document's sole focus on that topic. therefore, keywords alone cannot accurately infer topics; they merely serve as clues from the user. ## method for zero-shot topic inference in this section, we discuss the zero-shot topic inference approach we studied in this paper. the end-to-end inference process is shown in fig 1 . ## performance analysis and findings in this section, we present performance details of sentence encoders using various article and topic encoding techniques (refer to chatgpt and sbert, they attained a score very close to the baseline methods (gflm). another intriguing observation is that palm's performance showed a significant improvement for the "explicit-mention" topic embedding compared to the "topic + keywords" topic embedding. 3. based on our observations, the topic embedding techniques of "topic+keywords" and "explicit-mentions" exhibited superior performance compared to other methods. these embeddings, which incorporate user guidance through topic keywords, significantly improved the accuracy of real-time zero-shot topic inference. as a result, we employed these embeddings only when experimenting with large sentence encoders (lse). 4. the "entire article" approach excelled among other techniques for article embedding, making it the preferred choice when utilizing large sentence encoders (lse). the "sentence embedding average" method followed next in performance, while the "individual sentence embedding" approach proved to be less promising. 5. "explicit-mentions" topic embedding with "entire article" as the article embedding attained the best score, followed by "topic + keywords" topic embedding paired with "entire article". 6. f 1 score obtained by infersent, laser, bloom, and gpt-neox indicates that they failed to generalize over unseen datasets and, therefore, may not be a good choice for zeroshot topic inference. 7. despite the observation stated in (6), we would like to point out that the inclusion of user guidance in the inference process boosted the performance of infersent and laser. for example, "topic-name only" embedding achieved around 7% f 1 score (average over all datasets); however, with "explicit-mentions" embedding, f 1 score reached 30% (average over all datasets). 8. for small datasets, "topic-name-only" embedding presented an interesting case. here, use performed better than sbert. this suggests that, for the product review domain, if additional keywords for each topic are unavailable, use may be a better choice than sbert. however, a detailed investigation is warranted to determine the root cause for this result. considering the real-time nature of our task, it is crucial to consider inference time when selecting the appropriate approach. in order to analyze computation time, we logged the duration taken by different encoders for article and topic embeddings. due to space constraints, we have included the timing information for the best-performing article and topic embedding techniques in the main paper, while the timings for other embeddings such as "sentence embedding average", "individual sentence embedding", "topic-name-only", and "topic + keyword + definition" are provided in the appendix a.5. the generation time (in seconds) for each model is reported in tables 11 and 10 . major observations from these tables are as follows. in essence, comprehensive performance and run ## conclusion the task of zero-shot topic inference is both fundamental and challenging. considering the challenge of zero-shot topic inference and the unexplored potential of recent sentence encoders and large language models (llms) in this area, we investigated their ability to generalize for this task alongside traditional sentence encoders. in our real-time zero-shot topic inference task, we found varying performance among popular sentence encoders and llms. among the mid sentence encoders, sentence-bert showed good performance on unseen data, while use achieved decent accuracy. however, infersent and laser didn't perform at par with use and sbert. among the large sentence encoders, chatgpt performed the best, followed by palm. however, gpt-neox and bloom didn't generalize effectively for the task. we also introduced innovative approaches to incorporate user guidance, improving topic inference accuracy. additionally, we con-ducted a thorough analysis of execution time, revealing that both the f 1 score and the efficiency of certain models (specifically bloom, gpt-neox, infersent, and laser) raise concerns when considering their suitability for the task.