Extreme Zero-Shot Learning for Extreme Text Classification
Abstract
The eXtreme Multi-label text Classification (XMC) problem concerns finding most relevant labels for an input text instance from a large label set.
However, the XMC setup faces two challenges: (1) it is not generalizable to predict unseen labels in dynamic environments, and (2) it requires a large amount of supervised (instance, label) pairs, which can be difficult to obtain for emerging domains.
In this paper, we consider a more practical scenario called
Extreme Zero-Shot XMC (EZ-XMC), in which no supervision is needed and merely raw text of instances and labels are accessible.
Few-Shot XMC (FS-XMC), an extension to
EZ-XMC with limited supervision is also investigated.
To learn the semantic embeddings of instances and labels with raw text, we propose to pre-train Transformer-based encoders with self-supervised contrastive losses.
Specifically, we develop a pre-training method
MACLR , which thoroughly leverages the raw text with techniques including M ulti-scale
A daptive C lustering, L abel R egularization, and self-training with pseudo positive pairs.
Experimental results on four public EZ-XMC datasets demonstrate that MACLR achieves superior performance compared to all other leading baseline methods, in particular with approximately 5-10% improvement in precision and recall on average.
Moreover, we show that our pre-trained encoder can be further improved on FS-XMC when there are a limited number of ground-truth positive pairs in training.
Our code is available at https://github.com/amzn/pecos/ tree/mainline/examples/MACLR .
Introduction
The e X treme M ulti-label text C lassification (XMC) problem aims at tagging a text input with most relevant subset of labels from an extremely large output space.
Many web-related applications can be formulated as an XMC task with encouraging results, such as finding the best matching products from a large catalog in e-commerce systems (Medini et al., 2019; Chang et al., 2021), auto-completing queries given its prefix on search engines (Yadav et al., 2021), predicting search keywords for dynamic advertising (Prabhu et al., 2018; Chang et al., 2020b), tagging categories of Wikipedia articles from a large label taxonomy (Dekel and Shamir, 2010;
Chalkidis et al., 2019), to name just a few.
The current XMC setup is built on full label coverage and full supervision, where full label coverage means labels to be predicted have appeared in the training set and full supervision indicates it requires a significant number of annotated (instance, label) pairs.
In detail, it is assumed that an XMC algorithm has access to raw text of instances and labels, together with their corresponding relations during training, as shown in Figure 1.
However, there are several limitations of this
XMC setting.
First of all, due to the assumption of full label coverage, it is typical in XMC approaches to simply treat labels as IDs for classification and thus they are restricted to making predictions within observed labels.
This assumption is unrealistic since the label set usually keeps growing over time, e.g., newly added websites or products which are absent during training yet crucial for applications such as recommendation and advertising.
Besides, collecting labeled pairs is time-consuming, expensive and sometimes infeasible, for example, launching an e-commerce system in the emerging locale, where no user behavioral signals are avaiable.
In spite of these constraints, most existing methods (Dahiya et al., 2021b;
You et al., 2019;
Mittal et al., 2021; Dahiya et al., 2021a) followed this XMC setup.
It can be seen in Figure 2 that
Astec (Dahiya et al., 2021b), one of the state-of-theart extreme classifiers, is incapable of handling the scenario without supervision, which leads to zero performance in both Precision@5 and Recall@100.
Moreover, the increasing trend in Astec’s perfor-
[Figure]
GZ
[Figure]
XMC
EZ-XMC
Instances
Labels
Antinomianism,Libertarian socialism, Autism,
Sexuality of Abraham Lincoln, Grace Bedell, ……
Supervision
Label coverage
Test Training
Training
Test
Training
Test
Full
Partial
None
Instances
Labels √ √ √ √ √ √ × √ √
Full
Full & Partial & None Full & Partial & None
FS-XMC √ √
Full & Partial & None √ * mance along with the label ratio suggests that it depends highly on the supervision level and is hard to generalize to unseen labels.
This motivates us to investigate how to design an effective XMC model with zero supervision
[Figure]
In this paper, we consider an essential yet underexplored XMC setting, called Extreme Zero-shot
XMC (EZ-XMC).
As depicted in Figure 1, we can access raw text of both instances and labels but do not know their corresponding relations in EZXMC.
Moreover, we do not make any assumption on the label coverage, so the labels in the testing set may or may not appear in the training stage.
An extension to EZ-XMC with a limited number of training pairs, Few-shot XMC (FS-XMC), is also taken into account in our paper.
Either EZXMC or FS-XMC occurs frequently in the real world since informative and abundant (instance, label) pairs are never easy to obtain.
Also, it is more practical and worthwhile to reduce labor for manual annotation by solving problems under EZXMC.
Note that generalized zero-shot XMC (GZXMC) proposed in a recent work (Gupta et al., 2021) can be regarded as a special case of EZXMC.
GZ-XMC allows that the set of test labels is not completely overlapped with training labels but still requires supervision from positive pairs, as shown in Figure 1.
From Figure 2, we can observe that ZestXML (Gupta et al., 2021) designed for
GZ-XMC also suffers the issue of no supervision.
A natural question then arises: how should we deal with EZ-XMC problems?
Despite the name,
EZ-XMC is barely easy to tackle.
Fortunately, although dedicated supervision signals are lacking, raw text of instances and labels, e.g., product descriptions and categories, are still accessible in EZXMC.
Thus it is of vital importance to effectively leverage self-information of these data to train a model for classification.
To overcome challenges encountered in EZ-XMC, we turn to solving the problem from a different perspective without learning classifiers explicitly.
In particular, XMC can be cast into a problem which learns a sentence encoder
E to map instances and labels into dense embeddings, and predictions are made through approximate nearest neighbor search algorithms in the latent space (Shrivastava and Li, 2014).
Motivated by recent progresses in self-supervised learning (Gao et al., 2021;
Chen et al., 2020; He et al., 2020b;
Devlin et al., 2019), we propose MACLR ( M ultiscale A daptive C lustering & L abel R egularization), a two-stage pre-training procedure with those unpaired raw data to obtain a sentence encoder E under EZ-XMC.
As to FS-XMC, fine-tuning the encoder on a few paired data is sufficient for the performance boost.
Figure 2 demonstrates that
MACLR achieves superior performance when no supervision is available and achieves much higher recall than Astec and ZestXML by a large margin, even under the higher label coverage ratio.
Our main contributions are summarized below:
• We propose an essential Extreme Zero-Shot XMC (EZ-XMC) setting without any assumptions on supervision and label coverage, which has not been explored in previous work and is more practical in real applications.
• We leverage unlabeled data to pretrain the sentence encoder E with improved Inverse Cloze Task in Stage I of MACLR.
In particular, multi-scale adaptive clustering and label regularization are proposed to utilize raw text thoroughly.
In Stage II, we further self-train the encoder with pseudo positive pairs constructed from E in Stage I as well as TF-IDF model with complementary information.
• Comprehensive experiments are conducted on four public benchmark EZ-XMC datasets.
Results demonstrate that our pre-trained encoder can outperform existing unsupervised baseline methods notably.
As an example, MACLR achieves Recall@100 of 54.99%, nearly the same level as Astec (one of the SOTA XMC methods) (Dahiya et al., 2021b) trained with a supervised subset covering around 70% labels on LF-Amazon-131K. • MACLR can also achieve comparable or even better performance under the few-shot setting than those models heavily dependent on supervised information.
For example, MACLR is better than the SOTA ZestXML (Gupta et al., 2021) in Recall@100 over 20% ( 57 .
55% v.s. 32 .
69% ) when fine-tuned on the subset covering 1% labels of LF-Amazon-131K.
Related Work
Extreme multi-label classification
Various extreme classifiers have been proposed to address the large output space challenge of XMC problems.
We can broadly categorize them into two groups: partitioned-based models with linear classifiers (Prabhu et al., 2018; Prabhu and
Varma, 2014; Yu et al., 2020) that partition labels with hierarchical trees, leading to sub-linear inference time complexity, and embedding-based methods (Bhatia et al., 2015; Jain et al., 2019;
Guo et al., 2019) that learn a classifier for each label and leverage approximated nearest neighbor (Malkov and Yashunin, 2018; Guo et al., 2016) to index labels in the large output space.
There are also deep learning models such as AttentionXML (You et al., 2019), Astec (Dahiya et al., 2021b), SiameseXML (Dahiya et al., 2021a), and
XR-Transformer (Zhang et al., 2021) that further improve the accuracy of those linear counterparts with various advanced encoder architectures.
Nevertheless, none of those XMC methods can handle the EZ-XMC setup: they not only suffer from the lack of supervised signals, but also fail to generalize to unseen cold-start labels in the test set.
The only exception is ZestXML (Gupta et al., 2021), a recently proposed XMC method that was designed to address the generalized zero-shot XMC (GZXMC) problem where a number of labels for prediction are absent during training.
While ZestXML partially resolves the generalization challenge of cold-start labels, just like those conventional XMC models, it still depends heavily on a large number of training data with positive (instance, label) pairs.
Self-supervised learning techniques
The past few years have witnessed great promise in selfsupervised learning (Lan et al., 2020;
Chen et al., 2020; He et al., 2020b; Devlin et al., 2019; Khosla et al., 2020; Gao et al., 2021), where a pre-training task is defined using only data’s self-information.
Learned representations from the pre-training task can be then leveraged in a wide range of downstream tasks in various domains, such as image classification (Chen et al., 2020;
He et al., 2020b) and object detection (Li et al., 2020) in computer vision, and open-domain question answering (Lee et al., 2019; Guu et al., 2020) in natural language processing.
Specifically, we focus on contrastive approaches for Sentence-BERT (Reimers and Gurevych, 2019) models in this paper, where the intuition is to pull semantically close neighbors together and push apart non-neighbors via noise contrastive estimation or N-pair losses.
Various effective pre-training tasks such as Inverse Cloze
Task (ICT) (Lee et al., 2019) and SimCSE (Gao et al., 2021) have been shown to improve the performance of Sentence-BERT models.
Problem Formulation
In this section, we present the problem formulation of EZ-XMC.
With X and Y denoting the set of instances and labels respectively, the general XMC problem can be viewed as learning a scoring function f : X × Y → R .
f ( · , · ) maps an (instance, label) pair ( x, y ) to a similarity score, which is used to make a prediction through approximate nearest neighbor search algorithms.
In previous set- tings such as XMC and GZ-XMC, a considerable amount of relevant (instance, label) pairs { ( x i , y i ) } are available.
On the contrary, in EZ-XMC, we have no knowledge about corresponding relations between instances and labels, but only their raw text, as shown in Figure 1.
In this case, existing approaches that depend on the relevant pairs fail to learn an effective scoring function, even with a few paired data under FS-XMC.
Recent progresses in self-supervised learning have shown that a generalized sentence encoder can be learned through elaborately designed pretraining tasks even without any supervision (Lee et al., 2019;
Chang et al., 2020a), and then adapted to different downstream tasks directly or via slight finetuning.
On the other hand, the scoring function f can be modeled as f ( x, y
[Equation]
=
[Equation]
, E
[Equation]
, where E is a sentence encoder producing semantical dense embeddings, and ⟨· , ·⟩ is the similarity measurement such as inner product and cosine similarity.
Without loss of generality, inner product is adopted in the paper as the similarity metric between embeddings of instances and labels.
Thus, we formulate the problem as training an encoder
E with raw text of X and Y through a pre-training task for EZ-XMC.
As to the few-shot scenario FSXMC, we can fine-tune E for improvement.
Method
In this section, we introduce a two-stage pretraining procedure, MACLR, to thoroughly leverage unpaired data with raw text for EZ-XMC.
Specifically, we present the general framework in
Section 4.1, and then dive into details of two stages, pre-training with the improved Inverse Cloze Task and self-training with pseudo positive pairs, in Sections 4.2 and 4.3 respectively.
A complete algorithm is presented in Algorithm 1 in Appendix 4.4.
4. Framework
The framework of our pre-training procedure is shown in Figure 3.
MACLR consists of two stages:
• Stage I: title-context pairs are constructed for the Inverse Cloze Task, and the encoder E is then trained on these pairs together with two proposed techniques, multi-scale adaptive clustering and label regularization.
• Stage II: More pseudo positive pairs are crafted using different score functions modeled by the encoder from Stage I and TF-IDF respectively.
E is further trained on additional
pairs to improve the encoding performance.
Details of each component in our pre-training framework are discussed in the following sections.
[Figure]
4. Stage I: Pre-training with improved ICT
Inverse Cloze Task (Lee et al., 2019) is a frequently used pre-training task for the sentence encoder.
Specifically, for an instance x = { s 1 , . . .
, s n } consisting of n sentences, ICT randomly samples a sentence to serve as the pseudo positive label ˆ y =
s i where i ∼ [1 , n ] .
Then the rest of x is the pseudo instance ˆ x = { s 1 , . . .
, s i − 1 , s i +1 , . . .
, s n } .
In XMC, due to the property that the label usually summarizes the instance with one short sentence, which works similarly as the title s 1 , we directly utilize (context, title) pairs in the form of ( ˆ x = { s 2 , . . .
, s n } , ˆ y = s 1 ).
This construction works as the analog of the ground truth (instance, label) pairs and capture the semantics of a sentence.
With these pseudo pairs, the contrastive training objective for a mini-batch of N pairs is as follows:
[Equation]
Based on ICT, we also develop two techniques, multi-scale adaptive clustering and label reguarization, to fully leverage the information of unpaired instances and labels.
4.2. Multi-scale Adaptive Clustering
In the original ICT scheme, we can construct only one positive pair for a particular instance.
It is relatively hard in contrastive learning without enough positive examples, especially for extreme multilabel classification where one instance might be associated with more than one label, and a label is also likely to point to several different instances at the same time.
Thus a question arises naturally: is it possible to construct more positive pairs from purely unpaired raw data to intergrate richer information into the pre-training process?
We solve it by the unsupervised K-means clustering.
In detail, we divide pseudo (context, title) pairs from ICT into
K clusters through K-means based on the embeddings of all instances.
Then if C (ˆ x i ) =
C (ˆ x j ) , i.e., ˆ x i and ˆ x j belong to the same cluster, (ˆ x i , ˆ y j ) and (ˆ x j , ˆ y i ) are regarded as positive pairs besides original ICT pairs.
Furthermore, supervised contrastive loss is adopted for training the encoder with a minibatch of N pairs based on the cluster assignment:
[Equation]
Here, P Y ( i ) =
{ p ∈ { 1 , . . .
, N } : C (ˆ x i ) =
C (ˆ x p ) } is the set of indices of all positives for ˆ x i in the batch, and | P Y ( i ) | is its cardinality.
Minimizing Equation (2) pulls close the representations of instances and their positive labels within the same cluster and pushes away the representations of those from different clusters.
Besides, since the ultimate goal is the minimization of Equation (1), we propose a multi-scale approach with adaptive training, which guides the encoder to learn the easier tasks with sufficient positive examples, and then master harder tasks gradually.
This approach allows the encoder to learn from the coarse scale to the fine scale of clustering assignment, and is similar to the idea of curriculum learning (Bengio et al., 2009) to first focus on learning from a subset of simple examples, and expanding to include the remaining harder samples.
Our adaptive training process can be conducted by modifying the cluster size to adjust the task difficulty accordingly.
To be specific, we initialize the cluster assignment with the number of clusters K 0 , and double the cluster size every T steps.
The cluster assignment is also updated every T update steps along with the training of E .
Such a process lasts for half of the total training steps T total to take advantage of positive examples from constructed clusters.
The obtained intermediate encoder from this adaptive procedure is expected to satisfactorily capture the semantics of a sentence and is ready to deal with the optimization of Equation (1).
Then for the rest half of training steps, we turn to the hardest setting treating each instance as one independent cluster, which exactly falls into the contrastive training objective in Equation (1).
Our multi-scale adaptive clustering is illustrated in Figure 4.
[Table]
+NorVi5gqizswwEcgQdnUIErqEINCAi4h0d4cpTz4Dw7L9PSjDPrKcAfOW8/RLmUcA==</latexit
[Figure]
<latexit sha1_base64="1hpSomqG4oo3nRV2litKBoRdPwk=">AAAB8HicbZDLSgMxFIbP1Futt2qXboJFcFVmqnjZFdy4rGAv0g4lk6ZtaJIZkowwDH0KNy4UcSc+hyuX7nwM38D0stDWHwIf/38OOecEEWfauO6Xk1laXlldy67nNja3tnfyu3t1HcaK0BoJeaiaAdaUM0lrhhlOm5GiWAScNoLh5Thv3FGlWShvTBJRX+C+ZD1GsLHWbXuATZqMOiedfNEtuROhRfBmUKwUvt/LH43Xaif/2e6GJBZUGsKx1i3PjYyfYmUY4XSUa8eaRpgMcZ+2LEosqPbTycAjdGidLuqFyj5p0MT93ZFioXUiAlspsBno+Wxs/pe1YtM791Mmo9hQSaYf9WKOTIjG26MuU5QYnljARDE7KyIDrDAx9kY5ewRvfuVFqJdL3mnp+NorVi5gqizswwEcgQdnUIErqEINCAi4h0d4cpTz4Dw7L9PSjDPrKcAfOW8/RLmUcA==</latexit
[Figure]
Positive
Negative
is doubled to 2 and now ˆ y 1 and ˆ y 3 are positive to ˆ x 1 .
(c) Finally, K is equal to 4 where each instance itself is a cluster, and hence ˆ xonly has one positive label ˆ y 1 .
The process is similar for the rest of the instances.
4.2. Label Regularization
In addition to leveraging information from the instance side, we also have access to the raw texts of the whole label set and can utilize them to boost the encoder’s performance from the label side (Mueller et al., 2022; Müller et al., 2022).
Intuitively, for a randomly sampled label, with a high probability it is an negative example to the instance of interest.
We can take advantage of this intuition to make the embedding of the instance far from its irrelevant labels.
Instead of increasing the distance directly, it is more stable and effective to adopt contrastive losses.
To avoid overfitting, we choose a new positive example for each instance instead of its corresponding pseudo label from ICT which has been used in L cluster .
More concretely, ˆ x + i is selected exactly the same as ˆ x i , since the dropout layer is placed in the standard training of Transformerbased models and can be viewed as a minimal form of data augmentation (Gao et al., 2021).
By feeding the same sentence to the encoder E , two embeddings with different dropout masks are obtained, i.e., ˆ h i
=
E (ˆ x i , z i ) and
ˆ h
+ i
=
E (ˆ x + i , z
+ i ) where z represents a random mask for dropout.
ˆ h i ̸
=
ˆ h
+ i due to the dropout noise, but they hold similar semantics from the same sentence and thus can be used as a positive pair for contrastive learning.
The procedure of label regularization is depicted in Figure 5.
At each step, we sample M real labels from the label set Y , and the reguarization term is computed as follows:
[Equation]
Through minimizing L label , the encoder learns to pull the instance away from its irrelevant labels and incorporate the dropout augmentation at the same time.
Together with L cluster , we have the final objective function for pre-training in the Stage I as
[Equation]
=
L
[Equation]
+
[Equation]
label .
[Equation]
Positive
Negative
[Table]
[Figure]
Stage II: Self-
training with multi-viewed pseudo pairs
After the pre-training procedure in Section 4.2, we can obtain an intermediate encoder E I .
But are there any ways to further improve the encoder?
Inspired by self-training in semi-supervised learning (Yalniz et al., 2019; Xie et al., 2020;
He et al., 2020a; Zoph et al., 2020), E I can be leveraged to make predictions on those unpaired training instances themselves, to generate pseudo positive pairs.
These pseudo pairs are much better than random guessing and can serve as a distinct view from
ICT pairs.
On the other hand, similar pseudo pairs can be constructed by other unsupervised methods such as TF-IDF, which provide different and complementary information about the instance.
With multi-viewed pseudo positive pairs, we can conduct further training on the encoder in State II from a new perspective and self-improve E I .
The detailed process works as follows:
1) Compute the similarity score using E I for each training instance x i , and select labels with topk maximum scores as its pseudo labels; 2) Generate labels similarly with TF-IDF, except that E ( x ) and E ( y ) are replaced with their TFIDF vectors; 3) Mix pseudo positive pairs from 1) and 2) together, and train E I on them with Equation (2).
4. MACLR Algorithm
The whole pre-training procedure of MACLR is shown in Algorithm 1.
Note that for FS-XMC, we simply fine-tune the encoder E from MACLR on available positive pairs for several steps by minimizing the original contrastive loss in Equation (1).
the sentence encoder training parameters and texts in X 2: Feed the context for each pair into the encoder
E and cluster them into K = K 0 clusters via k-means 3: for t = 1 , . .
, T total do 4:
Sample a mini-batch of pseudo pairs of size N and a mini-batch of real labels of size
M 5:
Compute the loss:
L =
L cluster + L label 6:
Train the encoder by minimizing L 7: if t mod T K = 0 and t < T total / 2 then 8:
K = K ∗9: end if 10: if t mod T update = 0 and t < T total / 2 then 11:
Feed raw texts of X again into E , and update current cluster assignment via k-means with the cluster number K 12: end if 13: if t ≥ T total / 2 then 14:
Treat each instance as an independent cluster 15: end if 16: end for ▷ Stage II: Self-training with multi-viewed pseudo pairs 17: Construct pseudo pairs ( X pseu , Y pseu ) by selecting top- k candidate labels with the similarity metric on the encoder E and TF-IDF respectively 18: Train the encoder E for T total steps by minimizing Equation (2)
Experimental Results 5. Experimental Settings Datasets
We evaluate our proposed MACLR on 4 public XMC benchmark datasets (Bhatia et al., 2016; Gupta et al., 2021) where raw text of instances and labels are available.
These datasets are derived from real-world applications, ranging from item-to-item recommendation (LF-Amazon131K, LF-Amazon-1M), to Wikipedia articles category/title tagging (LF-WikiSeeAlso-320K, LF-
Wikipedia-500K).
Detailed dataset statistics are presented in Table 5 in Appendix A.
Evaluation Protocol
We consider two evaluation setups: Extreme Zero-shot Learning of XMC (EZ-XMC) and Few-shot Learning of XMC (FSXMC).
EZ-XMC is a fully unsupervised learning setup where no positive (instance, label) pairs are available.
The only available information is the raw text of training instances and the whole label set.
FS-XMC is a semi-supervised learning setup where only very few positive (instance, label) pairs in the training set are available.
Regardless of the learning procedure, all models are evaluated on the same test set for fair comparison.
We evaluate the models’ performance with precision@k (P@k, k ∈ { 1 , 3 , 5 } ) and recall@k (R@k, k ∈ { 1 , 3 , 5 , 10 , 100 } ), which are two commonly-used evaluation metrics in the XMC literature (Reddi et al., 2019; Chang et al., 2021).
Baseline Methods
For EZ-XMC, we compare our method with the following unsupervised learning algorithms: TF-IDF, XR-Linear, GloVe, SentBERT, MPNet, SimCSE and ICT.
Note that SentBERT and MPNet are pre-trained on external multitask learning datasets with extra supervision.
In contrast, SimCSE and ICT are fully unsupervised pre-rained Siamese-Transformers on the specific
XMC dataset only.
Detailed description of each method can be found in Appendix A.
For FS-XMC, as few-shot (instance, label) pairs are available, we additionally compare fine-tuned MACLR with competitive XMC approaches, including Astec (Dahiya et al., 2021b), SiameseXML (Dahiya et al., 2021a), and
ZestXML (Gupta et al., 2021).
ZestXML is the leading XMC method that improves performance on few-shot labels.
We also take into account SentBERT (Reimers and Gurevych, 2019) with further fine-tuning to demonstrate the effectiveness of our pre-training procedure.
5. Zero-Shot Learning
In this section, we focus on extreme zero-shot learning (EZ-XMC), where no real positive (instance, label) pairs are accessible.
Table 1 presents detailed performance of precision and recall on all four datasets.
Our proposed MACLR consistently outperforms all comparing baselines by a large margin on all four datasets.
Compared to the leading sparse method TF-IDF, MACLR has an average of 5.3% and 9.1% absolute improvement in Precision@
[Table]
and Recall@100, respectively.
Compared to the leading neural model MPNet, MACLR has an average of 3.5% and 10.9% absolute improvement in
Precision@1 and Recall@100, respectively.
Speaking of sparse lexical matching approaches
[Table]
TF-IDF remains a tough-to-beat unsupervised baseline.
Specifically, TF-IDF performs better than many BERT variants (e
[Table]
g., SentBERT, SimCSE,
ICT)
[Table]
which is aligned with the finding in recent zero-shot dense retrieval literature (Thakur et al., 2021; Izacard et al., 2022).
It suggests the importance of designing proper self-supervised learning tasks for Transformer models in unsupervised EZXMC setup.
Note that XR-Linear is based on TFIDF vectors whereas the noise from pseudo pairs makes it even inferior to the original TF-IDF.
As for pre-trained SentBERT models, on the other hand, only MPNet shows comparable performance with TF-IDF.
MPNet remains competitive because it was trained on a large supervised corpus (out-of-domain) to learn semantics between paraphrasing sentences.
Thus, MPNet should be viewed as a multi-task learning baseline with ex-
[Table]
tra supervision.
However, MACLR is significantly better than MPNet with an average improvement of 3.5% in P@1 and over 10% in R@100.
Furthermore, MACLR also outperforms its counterparts which are trained with effective pre-training tasks such as SimCSE and ICT on the target dataset, showing the effectiveness of pre-training strategies like multi-scale adaptive clustering in
MACLR.
Overall, results in Table 1 demonstrates that MACLR is capable to learn informative embeddings and to make useful predictions even with no supervision.
We will investigate each component in MACLR in Section 5.4 thoroughly.
5. Few-Shot Learning
We further conduct few-shot learning (FS-XMC) experiments in which different learning algorithms can access a limited number of positive (instance, label) pairs.
To simulate the scenario of few-shot learning, we first manually sample a small ratio of labels, then collect all their positive instances from the training set as the final subset of positive (instance, label) pairs for model training.
Results of FS-XMC methods fine-tuned with 1% and 5% labels are shown in Tables 2 and 3 respectively.
Our proposed MACLR outperforms all other
[Table]
baselines significantly, including variants of
Siamese-Transformer models (e.g., SentBERT, MPNet) and major competitive XMC methods (e.g.,
XR-Linear, Astec and SiameseXML), on all four datasets.
Note that SiameseXML is the state-of-the-art
XMC method under the full supervision setup of
XMC.
Here, we again witness that existing XMC methods heavily rely on the supervision level as well as the full-coverage of label space for test set.
MACLR, in contrast, still performs robustly under FS-XMC, which enjoy larger applicability to emerging domains with many cold-start labels.
Crucially, even ZestXML tailored to address the challenging scenario of unseen labels cannot match the performance of MACLR.
In particular, when focusing on the few-shot scenario with only 1% sampled labels, MACLR achieves 18.74% in
P@1, improving the performance of Astec with 0.94% and ZestXML with 10.10% significantly.
Besides, MACLR outperforms all Sentence-BERT counterparts, validating the effectiveness of our pre-training procedure.
As to fine-tuning on the subset with 5% labels, performance of all methods are improved as expected with more supervision.
The relative rank among these methods remains the same, with MACLR still performing the best in terms of precision and recall on all four datasets.
5. Ablation Study
In this part, we conduct an ablation study to investigate each component in our pre-training procedure, including multi-scale adaptive clustering, label regularization, and self-training with pseudo positive pairs constructed from the encoder or TF-IDF.
We add a component once a time on LF-Amazon-131K to observe its independent influence on the model performance.
Table 4 presents detailed performance on seven different configurations
[Table] [Table]
For two techniques multi-scale adaptive clustering and label regularization during the Stage I, they can improve the performance of the encoder separately, as shown in the performance gain of the index 2 and 3 over the index 1.
When combined, they can further improve the accuracy of the model, from 8.90% to 10.65% in P @5 and from 47.40% to 51.45% in R @100 .
As to the second stage, we explore the impact of self-training with pseudo positive pairs either from the encoder itself or TF-IDF.
We can see from Table 4 that pairs from both E and TF-IDF contribute to the precision and recall gain over the index 5.
It further validates that the encoder and TF-IDF provides complementary perspective when constructing pseudo positive pairs.
Conclusions
This paper is the first to investigate the problem of Extreme zero-shot XMC without any supervision.
We develop a two-stage pre-training procedure MACLR to train a Sentence-BERT style encoder on pseudo (context, title) pairs constructed from raw text.
We demonstrate that techniques including multi-scale adaptive clustering, label regularization and self-training contribute to the performance gain of the pre-trained encoder.
In particular,
MACLR outperforms all unsupervised baselines significantly when there are no (instance, label) pairs provided.
It also offers leading accuracy in both precision and recall after fine-tuning on a limited number of paired data.
One limitation is relative low accuracy of top candidates and a future direction could be adding a ranker model after the encoder to improve performance on head labels.
Acknowledgement
Cho-Jui Hsieh is partially supported by NSF under
IIS-2008173 and IIS-2048280.
Broader Impact
In the paper all datasets are publicly available without any private or confidential information and to the best of our knowledge, there are no ethical issues with this paper.
For broader impacts of our work, we consider a practical scenario, Extreme
Zero-Shot XMC where no supervision is required.
Solving problems under EZ-XMC can reduce labor of manual annotation significantly.
For example, the unsupervised model can help narrow the tagging space remarkably by selecting a small number of candidate labels for products in the e-commerce domain for efficient annotation.
MACLR is such an effective method to benefit real-world applications like recommendation and advertisement.
A Implementation Details Datasets and Source Codes
Statistics of four datasets are presented in Table 5.
All XMC datasets used in the paper are available in the Extreme Classification Repository (Bhatia et al., 2016) 1 , except for LF-Amazon-1M, which is available from the
ZestXML paper (Gupta et al., 2021) 2 .
Besides, sources code can be accessed in this link.
of Bag-ofWords
(BoW) features.
[Table]
Compared Baselines of EZ-XMC
Here we provide detailed description of each baseline method under the EZ-XMC setting.
• TF-IDF (Rajaraman and Ullman, 2011), which represents instances and labels by sparse TF-IDF features and retrieves top labels for each instance based on the similarity of TF-IDF features; • XR-Linear (Yu et al., 2020), a hierarchical linear model trained with pseudo positive pairs constructed from TF-IDF; • GloVe (Pennington et al., 2014), which adopts dense average word embeddings with the dimension of 300 trained on co-occurrence statistics to measure similarity between instances and labels; • Sentence-BERT (SentBERT) (Devlin et al., 2019; Reimers and Gurevych, 2019), a sentence encoder modeled as a Siamese-Transformer to derive semantically meaningful embeddings for instances and labels; • Paraphrase MPNet (MPNet) (Song et al., 2020), another Sentence-BERT model originally designed for searching sentence paraphrases; • SimCSE (Gao et al., 2021), a SiameseTransformer pre-trained with the contrastive objective using dropout noise as augmentation; • ICT (Lee
et al., 2019), another SiameseTransformer pre-trained with the contrastive objective using (context, title) pairs.
Evaluation Metrics
As mentioned before, we adopt precision and recall as our evaluation metrics.
In detail, P @ k and R @
k are defined as follows:
[Equation]
y ∈ { 0 , 1 } L and y ′
∈ R L are the ground truth vector and the prediction vector respectively.
rank k returns the indices of the top- k highest elements.
Hyper-parameters
We use a Siamese Transformer model to embed both instances and labels.
The encoder consists of a 12 layers BERT-base model, topped with a linear head projecting hidden state of the [CLS] token into a 512 -dimensional embedding.
The sequence length of the instance and the label is set to be 288 and 64 respectively.
We pre-train the model on eight V100 GPUs for 100,000 steps with an Adam optimizer and batch size of 32 per GPU in both Stage I and Stage II.
This pre-training process takes about 1 day.
We adopt an initial learning rate 1 × 10 − 5 with the warm-up ratio 0.1, followed by a linear learning rate decay.
For fine-tuning, the learning rate of
Adam is set to 5 × 10 − 6 with 2000 training steps for the 1% label ratio and 10K training steps for
[Table]
the 5% label ratio.
In the Stage I, we use the initial cluster size K 0 = 2048 and set T K = 10000 and
T update = 5000 .
In Stage II, top 3 ranked labels from predictions of the encoder and TF-IDF are selected to constitute the pseudo set for self-training.
For hyper-parameters of all baselines, we follow their default setups.
All experiments are conducted on the AWS p3dn.24xlarge instance, consisting of 96 Intenl Xeon CPUs with 768 GB of RAM and
Nvidia V100 GPUs with 32 GB of memory each.
It takes about half a day to complete the pre-training procedure of MACLR.
We present error bars of four independent runs to validate our MACLR results are statistically significant under EZ-XMC in Table 6.
It can be observed that the variance is small, showing that our method can produce similar results with different random seeds, and MACLR is statically better than other baselines compared with results in Table 1.
Therefore, we run each method for four times and report the best performance in the main paper.
B Hyperparameter Study
We provide details of hyperparameter tuning process.
For hyperparameters in the Stage I including T K , T update , and K 0 , we choose them from the following ranges: T K ∈ [5000 , 10000 , 25000] ,
T update ∈ [1000 , 5000 , 10000] , and K∈ [1024 , 2048 , 4096] .
Instead of grid search for all possible combinations (in total 3 3 = 27 ), we tried to investigate the impact of different hyperparameters first and then determined the value.
Specifically, we choose the median setting of ( T K = 10000 , T update = 5000 , K 0 = 2048) as the baseline, and for each hyperparameter to be evaluated, we fix the remaining two and observe the performance.
Experimental results can be found in Table 7, 8 and 9.
We summarize our findings below:
• With T update = 5000 , K 0 = 2048 , T K = 10000 performs the best.
We assume that when T K is smaller as 5000 , the training steps are not sufficient to obtain satisfactory representations under the current cluster size; and when T K is larger as 25000 , the algorithm spends too much time on one cluster size and fails to explore different scenarios based on curriculum learning.
• With T K = 10000 , K 0 = 2048 , T update = 1000 outperforms the other two values.
It is expected since the cluster assignment would be more accurate and the learned embeddings would be better accordingly.
• With T K = 10000 , T update = 5000 , K 0 = 2048 is the best among three values.
We hypothesize the trajectory of K 0 = 2048 gives the appropriate curriculum learning process.
Based on
these observations, the best setting should be ( T
K =
T update =
, K
= 2048)
However, we can see that T update = 1000 is only slightly better than T update = 5000 and considering the cost of updating cluster assignment, we select T update = 5000 instead.
[Table]
[Table] [Table]
For the hyperparameter k in the Stage II
[Table]
we use the previous setting ( T K = 10000 , T update = 5000 , K 0 = 2048 ) for pretraining in the stage I, and tune k from [1 , 3 , 5] .
Results are shown in
Table 10 in the paper.
It can be observed that k =leads to the best performance.
This is due to the fact that k = 1 did not introduce a sufficient number of meaningful pseudo (instance, label) pairs while k = 5 was likely to add to the noise of the training set with irrelevant (instance, label) pairs.
[Table]
Therefore, the default hyperparameter choice of
MACLR is determined as ( T K = 10000 , T update = 5000 , K 0 = 2048 , k = 3 ), which is tuned on
LF-Amazon-131K.
Moreover, as we can see that
MACLR is not very sensitive to the choice of hyperparameters and performs better than all the baselines, we assume that this configuration would perform similarly on other datasets and use it without further tuning.
C Additional Experiments on FS-XMC
[Table]
In this section, we present additional experimental results for the setting of FS-XMC on
LF-Amazon-131K and LF-WikiSeeAlso-320K. Instead of sampling a few-shot subset by the label coverage ratio, we turn to sampling based on the pair ratio.
Specifically, suppose a training set
D train = { ( x i , y i ) } has |D train | positive pairs.
Each time we randomly sample a small ratio of δ (1% or 5% in our paper) pairs from the total set to constitute the few-shot subset.
Each subset has δ |D train | pairs for fine-tuning.
Detailed results of δ = 1%
[Table]
and δ = 5% are presented in Table 11 and 12 respectively.
MACLR is still the best-performing method and outperforms all other baselines significantly in precision and recall.