E xploring Universal Sentence Encoders for Zero-shot Text Classification (2022)

Abstract: Universal Sentence Encoder (USE) has gained much popularity recently as a general-purpose sentence encoding technique. As the name suggests, USE is designed to be fairly general and has indeed been shown to achieve superior performances for many downstream NLP tasks. In this paper, we present an interesting “negative” result on USE in the context of zero-shot text classification, a challenging task, which has recently gained much attraction. More specifically, we found some interesting cases of zero-shot text classification, where topic based inference outperformed USE-based inference in terms of F1 score. Further investigation revealed that USE struggles to perform well on data-sets with a large number of labels with high semantic overlaps, while topic-based classification works well for the same.

## introduction what makes a sentence encoder universal? the tantalizing idea is to learn a general sentence encoding technique that can achieve "good" performance on a wide variety of downstream tasks. recently, google's universal sentence encoder (use) @xcite has been shown to achieve great success in various downstream tasks and promising results in a way provided some justification to the name "universal sentence encoder" itself. while @xcite is undoubtedly one of the state-of-the-art sentence encoding techniques available today, it's success has primarily been demonstrated within the "pre-train/fine-tune" paradigm, where, it is assumed that the target labels are known beforehand as well as a small amount of training data is readily available, which can facilitate the fine-tuning process. whereas, a more challenging task is zero-shot text classification @xcite , where, neither the target labels are known beforehand nor any training data is available for fine-tuning. how use performs in case of zero-shot text classification is, therefore an interesting research question, which is relatively underexplored at this moment. to address this knowledge gap, we performed a systematic study, where, we applied use to perform the "zero-shot text classification" task, as defined by @xcite . the goal of our study is to investigate how powerful use is for solving an nlp task for which acquiring training data is almost impractical. to perform this study, we conducted extensive experiments with seven real-world datasets. as a baseline, we implemented two topic-based zeroshot classification techniques for comparative analysis. we evaluated the goal-task performance against the "gold" standard labels annotated by humans and computed f 1 metric for each method compared. experimental results demonstrate that topic-based inference clearly outperformed usebased inference in terms of f 1 score for most of the datasets, essentially yielding the so-called "negative" result. further investigation revealed that use struggles to perform well on datasets with a large number of labels with high semantic overlaps, while topic-based methods work well for the same. universal sentence encoder: the utility of use has been tested for many popular nlp tasks including intent classification casanueva et al. (2020), fake-news detection majumder and das (2020), duplicate record identification lattar et al. (2020) and covid-19 trending topics detection from tweets asgari-chenaghlu et al. (2020). perone et al. (2018); enayet and sukthankar (2020) focused on the performances of different sentence embedding techniques for transfer-learning tasks. rivas and zimmermann (2019) reported that stateof-the-art sentence embeddings are unable to capture sufficient information regarding sentence correctness and quality in the english language. " it is light enough to carry around all day without bother." "weight" article text labels keywords universal sentence encoder label embedding text embedding label(s) cosine distance measure article text user figure 1: steps for zero-shot text classification leveraging universal sentence encoder. zero-shot classification: veeranna et al. (2016) adopted pre-trained word embedding for measuring semantic similarity between a label and documents. @xcite @xcite , performed zeroshot learning using semantic embedding. rios and kavuluru (2018 uniqueness of this work: we explore the efficacy of use for "zero-shot text classification" task and compare against topic-based zero-shot methods, which is unique about this work. ## zero-shot text classification zero-shot text classification (0shot-tc) is a challenging problem which aims to associate an appropriate label with a piece of text, regardless of the text domain without any training/fine-tuning. the idea of zero-shot tc was coined by @xcite , and in this paper, we have specifically focused on definition-wild 0shot-tc discussed by @xcite , a visual depiction of which is presented in figure 1 . more specifically, we formalize our task as below: definition 1. 0shot-tc: given a collection of text articles @xmath0, t 2 , ..., t n }, a user x and a set of user-defined labels l @xmath1, l 2 , ..., l m } provided in real-time, classify each text article t @xmath2. notably, it is possible that two different users will focus on different set of labels for the same dataset based on their application needs. furthermore, creating customized training datasets beforehand is no longer possible because the target labels are provided in real-time by users. ## results and findings we first present the results on the seven datasets used in our experiments for the four variants of the use-based zero-shot text classifiers. we have also presented performance of gflm-w, gflm-s, and use for a fixed number of labels over different threshold in figure 2 . at the end, results were surprising as use was outperformed by simple topic-based inference techniques for zeroshot classification tasks, which motivated us to dig deeper into the reasons of use's score. ## discussion and conclusion in this paper, we present a so-called "negative" result on use in the context of "zero-shot text classification". our experimental results reveal that basic topic-based inference models outperformed use-based inference, which is indeed surprising. further investigation revealed that use struggles to achieve good performance on zero-shot classification tasks with a large number of labels with high semantic overlap. on the other hand, simple topic based inference techniques were found to be pretty robust as a zero-shot classifier. one possible explanation for such performance may be attributed to the fact that topic-distribution vectors are constrained (sums to 1), while use vectors are unbounded (real numbers). such constrained representation of topic-vectors may make them superior in terms of their capability to distinguish between two highly overlapping labels compared to same for unbounded use vectors, which were not trained following such constraints. in normal supervised learning settings, use usually learn those distinctions from training labels, however, in case of zero-shot cases, that distinguishing capability is perhaps not developed well. in summary, this paper highlights a limitation of the use encoding technique and forms a cardinal basis for further research on the limitation of use. our findings also suggest that we may be still far away from a sentence encoding technique that is indeed "universal". educational/research purpose. we have not tried to identify any private information from the data in any way which can result in a privacy violation. additionally, the data we used (publicly release) does not contain personal information (e.g., usernames of users). in the whole experiment, we only used open source packages and libraries, along with proper citations as required also in accordance with its acceptable use policy, and no additional permission was required.