Extreme Z ero- S hot Learning for Extreme Text Classification (2022)

Abstract: The eXtreme Multi-label text Classification (XMC) problem concerns finding most relevant labels for an input text instance from a large label set. However, the XMC setup faces two challenges: (1) it is not generalizable to predict unseen labels in dynamic environments, and (2) it requires a large amount of supervised (instance, label) pairs, which can be difficult to obtain for emerging domains. In this paper, we consider a more practical scenario called Extreme Zero-Shot XMC (EZ-XMC), in which no supervision is needed and merely raw text of instances and labels are accessible. Few-Shot XMC (FS-XMC), an extension to EZ-XMC with limited supervision is also investigated. To learn the semantic embeddings of instances and labels with raw text, we propose to pre-train Transformer-based encoders with self-supervised contrastive losses. Specifically, we develop a pre-training method \textbf{MACLR} , which thoroughly leverages the raw text with techniques including \textbf{M} ulti-scale \textbf{A} daptive \textbf{C} lustering, \textbf{L} abel \textbf{R} egularization, and self-training with pseudo positive pairs. Experimental results on four public EZ-XMC datasets demonstrate that MACLR achieves superior performance compared to all other leading baseline methods, in particular with approximately 5-10% improvement in precision and recall on average. Moreover, we show that our pre-trained encoder can be further improved on FS-XMC when there are a limited number of ground-truth positive pairs in training. Our code is available at https://github.com/amzn/pecos/tree/mainline/examples/MACLR .

## introduction the extreme multi-label text classification (xmc) problem aims at tagging a text input with most relevant subset of labels from an extremely large output space. many web-related applications can be formulated as an xmc task with encouraging results, * work done during an internship at amazon inc. the current xmc setup is built on full label coverage and full supervision, where full label coverage means labels to be predicted have appeared in the training set and full supervision indicates it requires a significant number of annotated (instance, label) pairs. in detail, it is assumed that an xmc algorithm has access to raw text of instances and labels, together with their corresponding relations during training, as shown in figure 1 . however, there are several limitations of this xmc setting. first of all, due to the assumption of full label coverage, it is typical in xmc approaches to simply treat labels as ids for classification and thus they are restricted to making predictions within observed labels. this assumption is unrealistic since the label set usually keeps growing over time, e.g., newly added websites or products which are absent during training yet crucial for applications such as recommendation and advertising. besides, collecting labeled pairs is time-consuming, expensive and sometimes infeasible, for example, launching an e-commerce system in the emerging locale, where no user behavioral signals are avaiable. in spite of these constraints, most existing methods @xcite @xcite followed this xmc setup. it can be seen in figure 2 that astec @xcite , one of the state-of-theart extreme classifiers, is incapable of handling the scenario without supervision, which leads to zero performance in both precision@5 and recall@100. moreover, the increasing trend in astec's perfor-figure 1 : four different settings in xmc. four essential components are considered: instances (raw text), labels (raw text), supervision (positive pairs), and label coverage. in detail, we divide label coverage into 3 groups: full, partial, and none. * in fs-xmc emphasizes that only a limited amount of supervision is available. we can see that ez-xmc is the most general and practical setting, where no supervision and label coverage is required. figure 2: performance of three representative xmc methods on lf-amazon-131k at different ratios of label coverage. a subset covering [0, 1, 5, 10, 25, 50, 100](%) of the whole label set is sampled for fine-tuning. in this paper, we consider an essential yet underexplored xmc setting, called extreme zero-shot xmc (ez-xmc). as depicted in figure 1 , we can access raw text of both instances and labels but do not know their corresponding relations in ez-xmc. moreover, we do not make any assumption on the label coverage, so the labels in the testing set may or may not appear in the training stage. an extension to ez-xmc with a limited number of training pairs, few-shot xmc (fs-xmc), is also taken into account in our paper. either ez-xmc or fs-xmc occurs frequently in the real world since informative and abundant (instance, label) pairs are never easy to obtain. also, it is more practical and worthwhile to reduce labor for manual annotation by solving problems under ez-xmc. note that generalized zero-shot xmc (gz-xmc) proposed in a recent work @xcite can be regarded as a special case of ez-xmc. gz-xmc allows that the set of test labels is not completely overlapped with training labels but still requires supervision from positive pairs, as shown in figure 1 . from figure 2 , we can observe that zestxml @xcite designed for gz-xmc also suffers the issue of no supervision. a natural question then arises: how should we deal with ez-xmc problems? despite the name, ez-xmc is barely easy to tackle. fortunately, although dedicated supervision signals are lacking, raw text of instances and labels, e.g., product descriptions and categories, are still accessible in ez-xmc. thus it is of vital importance to effectively leverage self-information of these data to train a model for classification. to overcome challenges encountered in ez-xmc, we turn to solving the problem from a different perspective without learning classifiers explicitly. in particular, xmc can be cast into a problem which learns a sentence encoder e to map instances and labels into dense embeddings, and predictions are made through approximate nearest neighbor search algorithms in the latent space @xcite . motivated by recent progresses in self-supervised learning @xcite @xcite , we propose maclr (multiscale adaptive clustering & label regularization), a two-stage pre-training procedure with those unpaired raw data to obtain a sentence encoder e under ez-xmc. as to fs-xmc, fine-tuning the encoder on a few paired data is sufficient for the performance boost. figure 2 demonstrates that maclr achieves superior performance when no supervision is available and achieves much higher recall than astec and zestxml by a large margin, even under the higher label coverage ratio. our main contributions are summarized below: ## related work extreme multi-label classification various extreme classifiers have been proposed to address the large output space challenge of xmc problems. we can broadly categorize them into two groups: partitioned-based models with linear classifiers @xcite @xcite ) that partition labels with hierarchical trees, leading to sub-linear inference time complexity, and embedding-based methods @xcite @xcite that learn a classifier for each label and leverage approximated nearest neighbor @xcite to index labels in the large output space. there are also deep learning models such as atten-tionxml @xcite , astec @xcite , siamesexml @xcite , and xr-transformer @xcite that further improve the accuracy of those linear counterparts with various advanced encoder architectures. nevertheless, none of those xmc methods can handle the ez-xmc setup: they not only suffer from the lack of supervised signals, but also fail to generalize to unseen cold-start labels in the test set. the only exception is zestxml @xcite , a recently proposed xmc method that was designed to address the generalized zero-shot xmc (gz-xmc) problem where a number of labels for prediction are absent during training. while zestxml partially resolves the generalization challenge of cold-start labels, just like those conventional xmc models, it still depends heavily on a large number of training data with positive (instance, label) pairs. ## problem formulation in this section, we present the problem formulation of ez-xmc. with x and y denoting the set of instances and labels respectively, the general xmc problem can be viewed as learning a scoring function f : x × @xmath0. f (•, •) maps an (instance, label) pair (x, y) to a similarity score, which is used to make a prediction through approximate nearest neighbor search algorithms. in previous set-tings such as xmc and gz-xmc, a considerable amount of relevant (instance, label) pairs {(x i , y i )} are available. on the contrary, in ez-xmc, we have no knowledge about corresponding relations between instances and labels, but only their raw text, as shown in figure 1 . in this case, existing approaches that depend on the relevant pairs fail to learn an effective scoring function, even with a few paired data under fs-xmc. recent progresses in self-supervised learning have shown that a generalized sentence encoder can be learned through elaborately designed pretraining tasks even without any supervision @xcite , and then adapted to different downstream tasks directly or via slight finetuning. on the other hand, the scoring function f can be modeled as f (x, y) = ⟨e(x), e(y)⟩, where e is a sentence encoder producing semantical dense embeddings, and ⟨•, •⟩ is the similarity measurement such as inner product and cosine similarity. without loss of generality, inner product is adopted in the paper as the similarity metric between embeddings of instances and labels. thus, we formulate the problem as training an encoder e with raw text of x and y through a pre-training task for ez-xmc. as to the few-shot scenario fs-xmc, we can fine-tune e for improvement. ## method in this section, we introduce a two-stage pretraining procedure, maclr, to thoroughly leverage unpaired data with raw text for ez-xmc. specifically, we present the general framework in section 4.1, and then dive into details of two stages, pre-training with the improved inverse cloze task and self-training with pseudo positive pairs, in sections 4.2 and 4.3 respectively. a complete algorithm is presented in algorithm 1 in appendix 4.4. ## conclusions this paper is the first to investigate the problem of extreme zero-shot xmc without any supervision. we develop a two-stage pre-training procedure maclr to train a sentence-bert style encoder on pseudo (context, title) pairs constructed from raw text. we demonstrate that techniques including multi-scale adaptive clustering, label regularization and self-training contribute to the performance gain of the pre-trained encoder. in particular, maclr outperforms all unsupervised baselines significantly when there are no (instance, label) pairs provided. it also offers leading accuracy in both precision and recall after fine-tuning on a limited number of paired data. one limitation is relative low accuracy of top candidates and a future direction could be adding a ranker model after the encoder to improve performance on head labels.