Original: CascadeXML: End-to-end Multi-Resolution Learning for  Extreme Multi-Label Text Classification
Revised: CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Extreme Multi-label Text Classification (XMC) involves learning a classifier thatcan assign an input with a subset of most relevant labels from millions of labelchoices.
Revised: Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices.

Original: Recent approaches, such as XR-Transformer and LightXML, leveragea transformer instance to achieve state-of-the-art performance.
Revised: Recent approaches, such as XR-Transformer and LightXML, leverage a transformer instance to achieve state-of-the-art performance.

Original: However, in thisprocess, these approaches need to make various trade-offs between performance andcomputational requirements.
Revised: However, in this process, these approaches need to make various trade-offs between performance and computational requirements.

Original: A major shortcoming, as compared to the Bi-LSTMbased AttentionXML, is that they fail to keep separate feature representations foreach resolution in a label tree.
Revised: A major shortcoming, as compared to the Bi-LSTM based AttentionXML, is that they fail to keep separate feature representations for each resolution in a label tree.

Original: We thus propose CascadeXML, an end-to-end multiresolution learning pipeline, which can harness the multi-layered architecture of atransformer model for attending to different label resolutions with separate featurerepresentations.
Revised: We thus propose CascadeXML, an end-to-end multiresolution learning pipeline, which can harness the multi-layered architecture of a transformer model for attending to different label resolutions with separate feature representations.

Original: CascadeXML significantly outperforms all existing approacheswith non-trivial gains obtained on benchmark datasets consisting of up to threemillion labels.
Revised: CascadeXML significantly outperforms all existing approaches with non-trivial gains obtained on benchmark datasets consisting of up to three million labels.

Original: Code for CascadeXML will be made publicly available.
Revised: Code for CascadeXML will be made publicly available at https: //github.com/xmc-aalto/cascadexml .

Original:  Introduction
Revised:  Introduction

Original: Extreme multi-label text classification (XMC) deals with the problem of predicting the most relevantsubset of labels from an enormously large label space spanning up to millions of labels.
Revised: Extreme multi-label text classification (XMC) deals with the problem of predicting the most relevant subset of labels from an enormously large label space spanning up to millions of labels.

Original: Over the years,extreme classification has found many applications in e-commerce, like product recommendation [7]and dynamic search advertisement [14], document tagging [1] and open-domain question answering[6, 24] and thus, practical solutions to this problem can have significant and far-reaching impact.
Revised: Over the years, extreme classification has found many applications in e-commerce, like product recommendation [7] and dynamic search advertisement [15], document tagging [1] and open-domain question answering [6, 25] and thus, practical solutions to this problem can have significant and far-reaching impact.

Original: The output space in extreme classification is not only large but also, the  labels follow Zipf’s law wheremost of the labels are tail labels [2, 15].
Revised: The output space in extreme classification is not only large but also, the distribution of instances among labels follows Zipf’s law such that a large fraction of them are tail labels [2, 16].

Original: For example, in a dataset of Wikipedia corpus containing500K labels, only 2.1% labels annotate more than 100 data points and 60% of the labels annotate lessthan 10 data points!
Revised: For example, in a dataset of Wikipedia corpus containing 500K labels, only 2.1% labels annotate more thandata points and 60% of the labels annotate less than 10 data points!

Original: This inherently makes learning meaningful representations at an extreme scale achallenging problem.
Revised: This inherently makes learning meaningful representations at an extreme scale a challenging problem.

Original: Further, as the memory and computational requirements grow linearly withthe size of the label space, naive treatment by the application of classical methods and off-the-shelfsolvers fails to deal with such large output spaces [8, 48].
Revised: Further, as the memory and computational requirements grow linearly with the size of the label space, naive treatment via classical methods and off-the-shelf solvers fails to deal with such large output spaces [8, 53].

Original: In extreme scenarios with millions of labels, a surrogate step of shortlisting candidate labels becomescrucial to perform extreme classification and has been adapted by many popular approaches [10, 17,21, 46, 51].
Revised: In extreme scenarios with millions of labels, a surrogate step of shortlisting candidate labels becomes crucial to perform extreme classification and has been adapted by many popular approaches [10, 18, 22, 51, 56].

Original: For each data point, these approaches first solve a simpler task with coarse label-clustersor meta-labels as the label space to create a shortlist of relevant labels for the extreme classificationtask.
Revised: For each data point, these approaches first solve a simpler task with coarse label-clusters or meta-labels as the label space to create a shortlist of relevant labels for the extreme classification task.

Original: This effectively reduces the training time and computational complexity of the extreme task tothat of the shortlisting step which is O ( √ L ) , where L is the number of labels.[Table]
Revised: This effectively reduces the training time and computational complexity of the extreme task to that of the shortlisting step which is O ( √ L ) , where L is the number of labels.

Original: Existing works leverage label representation as a centroid of their annotated instances to create a
Revised: Existing works leverage label representation as a centroid of their annotated instances to create a

Original: Hierarchical Label Tree (HLT) [34, 49].
Revised: Hierarchical Label Tree (HLT) [36, 54].

Revised: While earlier tree-based methods like Parabel [36] and 36th

Original: Works like [8, 10, 17, 21, 46] use label clusters  at a certainlevel of the HLT as meta-labels which are in turn used to shortlist candidate labels for the extremetask.
Revised: [Table] Bonsai [21] used the entire HLT for extreme classification, many recent works [8, 10, 18, 22, 51] use label clusters only at a certain level of the HLT as meta-labels which are in turn used to shortlist candidate labels for the extreme task.

Original: In contrast, XR-Transformer [51] and AttentionXML [49] leverage multiple levels of the HLT,such that each level corresponds to a certain label resolution in the tree structure [51].
Revised: In contrast, XR-Transformer [56] and AttentionXML [54] leverage multiple levels of the HLT, such that each level corresponds to a certain label resolution in the tree structure [56].

Original: 1.1 Motivation: Strengths and Weaknesses of Current Approaches
Revised: 1.1 Motivation: Strengths and Weaknesses of Current Approaches

Original: The earliest work to successfully combine label-tree based shortlisting and attention-based deepencoders is AttentionXML [49], which employs multiple Bi-LSTMs models that are trained sequentially for each tree resolution.
Revised: The earliest work to successfully combine label-tree based shortlisting and attention-based deep encoders is AttentionXML [54], which employs multiple Bi-LSTMs models that are trained sequentially for each tree resolution.

Original: More recent approaches [8, 17, 46, 51] replaced the model architecturewith a more powerful transformer model [41] and fine-tune a pre-trained instance such as Bert [11].
Revised: More recent approaches [8, 18, 51, 56] replaced the model architecture with a more powerful transformer model [46] and fine-tune a pre-trained instance such as Bert [11].

Original: Without careful designs, such models are computationally very expensive and earlier works [8, 46]could not effectively leverage transformers for both computation and performance on XMC tasks.
Revised: Without careful designs, such models are computationally very expensive and earlier works [8, 51] could not effectively leverage transformers for both computation and performance on XMC tasks.

Original: Next, we discuss two contemporary works LightXML [17] and XR-Transformer [51].
Revised: Next, we discuss two contemporary works LightXML [18] and XR-Transformer [56].

Original: LightXML introduces the concept of dynamic negative sampling, which replaces pre-computed labelshortlists with a dynamically calculated shortlist that changes as the model’s weights get updated.
Revised: LightXML introduces the concept of dynamic negative sampling, which replaces pre-computed label shortlists with a dynamically calculated shortlist that changes as the model’s weights get updated.

Original: This enables end-to-end training with a single model by using the final feature representation of thetransformer encoder for both the meta- and the extreme-classification task.
Revised: This enables end-to-end training with a single model by using the final feature representation of the transformer encoder for both the meta- and the extreme-classification task.

Original: Unfortunately, these twotasks appear to interfere with one another [21].
Revised: Unfortunately, these two tasks appear to interfere with one another [22].

Original: We hypothesize that this is because the meta taskneeds the attention maps to focus on different tokens than the extreme task.
Revised: We hypothesize that this is because the meta task needs the attention maps to focus on different tokens than the extreme task.

Original: Furthermore, it only usesa single-level tree, which prevents scaling to the largest datasets.
Revised: Furthermore, it only uses a single-level tree, which prevents scaling to the largest datasets.

Original: XR-Transformer takes inspiration from multi-resolution approaches in computer vision like superresolution [23] and progressive growing of GANs [18, 19], and enables multiple resolutions throughiterative training.
Revised: XR-Transformer takes inspiration from multi-resolution approaches in computer vision like super resolution [24] and progressive growing of GANs [19, 20], and enables multiple resolutions through iterative training.

Original: However, unlike progressively grown GANs, which predict only at the highestresolution, XR-Transformer needs predictions across all resolutions for its progressive shortlistingpipeline, but uses representations trained at a single resolution.
Revised: However, unlike progressively grown GANs, which predict only at the highest resolution, XR-Transformer needs predictions across all resolutions for its progressive shortlisting pipeline, but uses representations trained at a single resolution.

Original: In practice, this leads to XRTransformer having a complex multi-stage pipeline where the transformer model is iteratively trainedup to a certain resolution and then frozen.
Revised: In practice, this leads to XRTransformer having a complex multi-stage pipeline where the transformer model is iteratively trained up to a certain resolution and then frozen.

Original: This is followed by a re-clustering and re-training ofmultiple classifiers, working at different resolutions, with the same fixed transformer features [51,
Revised: This is followed by a re-clustering and re-training of multiple classifiers, working at different resolutions, with the same fixed transformer features [56,

Original: Alg. 2].
Revised: Alg. 2].

Original: For datasets with over 500K labels, the transformer is only trained on up to 2 15 clusters [51,
Revised: For datasets with over 500K labels, the transformer is only trained on up to 2 15 clusters [56,

Original: Table 6], and the resulting features are used for the extreme task.
Revised: Table 6], and the resulting features are used for the extreme task.

Original: Unlike AttentionXML, using multiple instances of transformer models becomes undesirable dueto their computational overhead.
Revised: Unlike AttentionXML, using multiple instances of transformer models becomes undesirable due to their computational overhead.

Original: This pushes LightXML and XR-Transformer to make differenttrade-offs when leveraging a single transformer model for XMC tasks compared to AttentionXML,see Table 1.
Revised: This enforces LightXML and XR-Transformer to make different trade-offs when leveraging a single transformer model for XMC tasks compared to AttentionXML, see Table 1.

Original: In this paper, we present a method, CascadeXML , that combines the strengths of theseapproaches, thus creating an end to-end trainable multi-resolution learning pipeline which trainsa single transformer model across multiple resolutions in a way that allows the creation of label  αααα
Revised: In this paper, we present a method, CascadeXML , that combines the strengths of these approaches, thus creating an end to-end trainable multi-resolution learning pipeline which trains a single transformer model across multiple resolutions in a way that allows the creation of label resolution specific attention maps and feature embeddings. αααα

Original: R (1)
Revised: R (1)

Original: R (2)
Revised: R (2)

Original: R (3)
Revised: R (3)

Original: Input Embeddings
Revised: Input Embeddings

Original: BERT
Revised: BERT

Original: Layers 1Layer
Revised: Layers 1Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: W W
Revised: W W

Original: W
Revised: W

Original: Wϑ (1) =
Revised: Wϑ (1) =

Original: Φ (6) CLS ( x ) ϑ (2) ϑ (3) ϑ (4)
Revised: Φ (6) CLS ( x ) ϑ (2) ϑ (3) ϑ (4)

Original: HLT
Revised: HLT

Original: C (1) ...
Revised: C (1) ...

Original: C (1) K
Revised: C (1) K

Original: C (2)
Revised: C (2)

Original: C (2) ... C (2) ...
Revised: C (2) ... C (2) ...

Original: C (2) K
Revised: C (2) K

Original: C (3) C (3)
Revised: C (3) C (3)

Original: C (3) ...
Revised: C (3) ...

Original: C (3) ... C (3) ...
Revised: C (3) ... C (3) ...

Original: C (3) K    ...
Revised: C (3) K    ...

Original: L − 3 L − 2 L −
Revised: L − 3 L − 2 L −

Original: L forward pass backward pass
Revised: L forward pass backward pass

Original: S (1) =
Revised: S (1) =

Original: { C (1)1 ,..., C (1) K 1 }
Revised: { C (1)1 ,..., C (1) K 1 }

Original: S (1) top S (2)
Revised: S (1) top S (2)

Original: S (2) top S (3)
Revised: S (2) top S (3)

Original: S (3) top S (4) scores
Revised: S (3) top S (4) scores

Original: Meta-labels:
Revised: Meta-labels:

Original: C (3)1 = { 1 , 2 }
Revised: C (3)1 = { 1 , 2 }

Original: C (2)1 = {{ 1 , 2 } , { 3 , 4 }}
Revised: C (2)1 = {{ 1 , 2 } , { 3 , 4 }}

Original: 1.2 CascadeXML in a Nutshell
Revised: 1.2 CascadeXML in a Nutshell

Original: The key insights that enable calculating resolution-specific representations with a single forward passthrough the transformer are a) that the shortlisting tasks get easier for lower label resolution andb) that the intermediate transformer representations are already quite powerful discriminators [43].
Revised: The key insights that enable calculating resolution-specific representations with a single forward pass through the transformer are a) that the shortlisting tasks get easier for lower label resolution and b) that the intermediate transformer representations are already quite powerful discriminators [48].

Original: Consequently, CascadeXML extracts features for classification at the coarser meta-label resolutionfrom earlier transformer layers and uses the last layer only for classification at the extreme resolution.
Revised: Consequently, CascadeXML extracts features for classification at the coarser meta-label resolution from earlier transformer layers and uses the last layer only for classification at the extreme resolution.

Original: The advantage of this is twofold: First, we postulate that the gain in classification accuracy due toenabling different attention mapsat the different tree resolutions outweighs the loss in accuracy dueto the slightly less powerful representations at the earlier transformer layers.
Revised: The advantage of this is twofold: First, we postulate that the gain in classification accuracy due to enabling tree resolution-specific attention maps outweighs the loss in accuracy due to the slightly less powerful representations at the earlier transformer layers.

Original: Second, this ensuresthat the representation capacity of the later layers is exclusively available to the more difficult tasks.
Revised: Second, this ensures that the representation capacity of the later layers is exclusively available to the more difficult tasks.

Original: The main idea of CascadeXML is illustrated in Figure 1.
Revised: The main idea of CascadeXML is illustrated in Figure 1.

Original: An input text is propagated through thetransformer model until the first shortlisting level is reached.
Revised: An input text is propagated through the transformer model until the first shortlisting level is reached.

Original: There, the embedding ϑ (1) corresponding to the [CLS] token is extracted, and used to determine scores for each level-1 meta-label.
Revised: There, the embedding ϑ (1) corresponding to the [CLS] token is extracted, and used to determine scores for each level-1 meta-label.

Original: Thehighest-scoring meta-labels are selected to create a shortlist of candidate labels S (1) top .
Revised: The highest-scoring meta-labels are selected to create a shortlist of candidate labels S (1) top .

Original: The input ispropagated further to perform another shortlisting step at the next resolution, at which a new [CLS]embedding ϑ (2) is extracted.
Revised: The input is propagated further to perform another shortlisting step at the next resolution, at which a new [CLS] embedding ϑ (2) is extracted.

Original: Crucially, ϑ (2) is the result of a different self-attention operations, andits attention map is a differently weighted combination of the  embeddings of other tokens than thatof ϑ (1) .
Revised: Crucially, ϑ (2) is the result of a separate self-attention operation, and its attention map is a differently weighted combination of the token embeddings as compared to ϑ (1) .

Original: The new embedding ϑ (2) is used to refine the shortlist by selecting the top-scoring level-2meta-labels that are children of the shortlisted level-1 meta-labels.
Revised: The new embedding ϑ (2) is used to refine the shortlist by selecting the top-scoring level-2 meta-labels that are children of the shortlisted level-1 meta-labels.

Original: This process is repeated until atthe final layer of the transformer, where classification is performed at the full label resolution.
Revised: This process is repeated until at the final layer of the transformer, where classification is performed at the full label resolution.

Original: Contributions In this paper, our contributions are the following:
Revised: Contributions In this paper, our contributions are the following:

Original: • We introduce a novel paradigm which can harness the multi-layered architecture of transformers for learning representations at multiple resolutions in an HLT.
Revised: • We introduce a novel paradigm which can harness the multi-layered architecture of transformers for learning representations at multiple resolutions in an HLT.

Original: This is in contrast tothe existing approaches which treat transformer as merely a black box encoder,• Implemented via CascadeXML, it is an end-to-end trainable multi-resolution learningpipeline which is quick to train and simple to implement.
Revised: This is in contrast to the existing approaches which treat transformer as merely a black box encoder, • Implemented via CascadeXML, it is an end-to-end trainable multi-resolution learning pipeline which is quick to train and simple to implement.

Original: It eliminates the need of bells and whistles like multi-stage procedures, reclustering, and bootstrap training to achievestate-of-the-art performance, and
Revised: It eliminates the need of bells and whistles like multi-stage procedures, reclustering and bootstrap training to achieve state-of-the-art performance, and

Original: • CascadeXML not only scales to dataset with millions of labels but also improves thecurrent state-of-the-art in terms of prediction performance by upto ∼ 6% on benchmarkdatasets.
Revised: • CascadeXML not only scales to dataset with millions of labels but also improves the current state-of-the-art in terms of prediction performance by upto ∼ 6% on benchmark datasets.

Original: In addition to this, it is computationally most efficient to train and clocks multi-foldimprovement in inference time as compared to existing transformer-based approaches.
Revised: In addition to this, it is computationally most efficient to train and clocks multi-fold improvement in inference time as compared to existing transformer-based approaches.

Original:  Notation & Preliminaries
Revised:  Notation & Preliminaries

Original: Problem setup:
Revised: Problem setup:

Original: In extreme classification, an instance (document or query) x ∈ Xfrom some inputspace X is mapped to a subset of labels y ⊂
Revised: In extreme classification, the input text instance (document or query) x ∈ X is mapped to a subset of labels y ⊂

Original: [ L ] out of the L available labels identified through theintegers [ L ] := { 1 , . . .
Revised: [ L ] out of the L available labels identified through the integers [ L ] := { 1 , . . .

Original: , L } , such that L ∼ 10 6 .
Revised: , L } , such that L ∼ 10 6 .

Original: Usually, we identify the labels through a binary vectory ∈ { 0 , 1 } L , where y j = 1 ⇔ j ∈ y .
Revised: Usually, we identify the labels through a binary vector y ∈ { 0 , 1 } L , where y j = 1 ⇔ j ∈ y .

Original: The instances and labels are jointly distributed accordingto an unknown distribution ( x , y ) ∼ P of which we have a i.i.d.
Revised: The instances and labels are jointly distributed according to an unknown distribution ( x , y ) ∼ P of which we have a i.i.d.

Original: sample D = (cid:8)(cid:0) x i , y i (cid:1) : i ∈ [ n ] (cid:9)available as a training set.
Revised: sample D = (cid:8)(cid:0) x i , y i (cid:1) : i ∈ [ n ] (cid:9) available as a training set.

Original: Even though the number of labels can be of the order of millions, the labelvectors are sparse, i.e., E [ ∥ y ∥ 1 ] ≪ L . For text data, an instance is a sequence of tokens x = ( t k ) lk =1 ,from a vocabulary t k ∈ V .
Revised: Even though the number of labels is large, the label vectors are sparse.

Original: As part of the model pipeline, a text vectorizer Φ : X (cid:55)→ R d projects the documents to a d -dimensionalembedding space.
Revised: As part of the model pipeline, a text vectorizer Φ : X (cid:55)→ R d projects the documents to a d -dimensional embedding space.

Original: These vectorizers can be simple bag-of-words (BOW) or TF-IDF transformationsΦ tf-idf , or a learnable model Φ dnn that encodes the documents to the embedding space using weightslearnt over the dataset.
Revised: These vectorizers can be simple bag-of-words (BOW) or TF-IDF transformations Φ tf-idf , or a learnable model Φ dnn that encodes the documents to the embedding space using weights learnt over the dataset.

Original: The final classification of an instance x is achieved by computing per labelscores ⟨ w l , Φ( x ) ⟩ using label-wise weight vectors w l ∈ R
Revised: The final classification of an instance x is achieved by computing per label scores ⟨ w l , Φ( x ) ⟩ using label-wise weight vectors w l ∈ R

Original: d
Revised: d

Original: which are either (i) jointly learnt with Φin a deep architecture [8, 10, 17, 31, 46, 49], or (ii) solely learnt in shallow classifiers [2, 50].
Revised: which are either (i) jointly learnt with Φ in a deep architecture [8, 10, 18, 32, 51, 54], or (ii) solely learnt in shallow classifiers [2, 55].

Original: Transformer Encoder: More recent approaches in extreme classification have achieved stateof-the-art performance using pre-trained transformers as text vectorizers [17, 46, 51].
Revised: Transformer Encoder: More recent approaches in extreme classification have achieved stateof-the-art performance using pre-trained transformers as text vectorizers [18, 51, 56].

Original: Populartransformer encoders for NLP [11, 27, 45] consist of an embedding layer, followed by a series ofstacked self-attention layers and task-specific heads that can be fine-tuned along with the encoder.
Revised: Popular transformer encoders for NLP [11, 27, 50] consist of an embedding layer, followed by a series of stacked self-attention layers and task-specific heads that can be fine-tuned along with the encoder.

Original: For a sequence of m input words x ∈ V m , each layer a in these models produces a sequence ofembeddings (cid:16) Φ ( a ) 0 ( x ) , . .{+
+}
Revised: For a sequence of m input words, each layer a in these models produces a sequence of embeddings (cid:16) Φ ( a ) {+
+}

Original:  Φ ( a ) m ( x ) (cid:17) , where Φ ( a ) i ( x ) ∈ R
Revised: ( x ) , . . . , Φ ( a ) m ( x ) (cid:17) , where Φ ( a ) i ( x ) ∈ R

Original: d is a representation for the i ’th token inthe input sequence at the a ’th layer, and Φ ( a ) CLS ( x ) :=
Revised: d is a representation for the i ’th token in the input sequence at the a ’th layer, and Φ ( a ) CLS ( x ) :=

Original: Φ ( a ) 

Revised: Φ ( a )

Original: ( x ) is an embedding corresponding to aspecial [CLS] token that has been pre-trained to capture a representation of the entire input [11, 27].
Revised: ( x ) is an embedding corresponding to a special [CLS] token that has been pre-trained to capture a representation of the entire input [11, 27].

Original: Label Shortlisting: Computing scores ⟨ w l , Φ( x ) ⟩ for each label l ∈
Revised: Label Shortlisting: Computing scores ⟨ w l , Φ( x ) ⟩ for each label l ∈

Original: [ L ] in an extremely largeoutput space is computationally very demanding, especially when the feature space R d ∋ Φ( x )is high dimensional to allow for expressive features.
Revised: [ L ] in an extremely large output space is computationally very demanding, especially when the feature space R d ∋ Φ( x ) is high dimensional to allow for expressive features.

Original: To alleviate this problem, the labels L aregrouped together under K meta-labelsR = { C 1 , . .
Revised: To alleviate this problem, the labels L are grouped together under K meta-labels { C 1 , . .

Original: , C K }  , with K ≈ O ( √ L ) .
Revised: , C K } =: R , with K ≈ O ( √ L ) .

Original: Each meta-label isrepresented as the set of its child labels, C k = { l ∈ [ L ] : l is child of k } and each target set y ⊂
Revised: Each meta-label is represented as the set of its child labels, C k = { l ∈ [ L ] : l is child of k } and each target set y ⊂

Original: [ L ]corresponds to a meta-target ˜
Revised: [ L ] corresponds to a meta-target ˜

Original: y
Revised: y

Original: = R ( y ) =
Revised: = R ( y ) :

Original:  { C k | ∃ i ∈
Revised: = { C k | ∃ i ∈

Original: y : i ∈ C k } .
Revised: y : i ∈ C k } .

Original: A meta-label C l is relevant to adata point if it contains at least one relevant label.
Revised: A meta-label C l is relevant to a data point if it contains at least one relevant label.

Revised: Ideally, the labels grouped under one meta-label should be semantically similar to ensure the common representations learnt for meta- and extreme- tasks are relevant to both tasks [10].

Revised: This has the added advantage that the negative labels selected by the shortlisting procedure will be the most confusing ones, leading to hard negative sampling and improved learning of the extreme classifier [39].

Original: The label vectors areso sparse  thateven the much coarser meta-labels still contain mostly zeros,| ˜ y | ≪ K .
Revised: The label vectors are sparse to the extent that the much coarser meta-labels still contain mostly zeros, | ˜ y | ≪ K .

Original: This means that it is enough to explicitly calculate the scores only for the descendants ofrelevant meta-labels.
Revised: This means that it is enough to explicitly calculate the scores only for the descendants of relevant meta-labels.

Original: In practice, an algorithm will first predict scores for each meta-labels and selectthe top-scoring ones as a shortlist for further classification.
Revised: In practice, an algorithm will first predict scores for each meta-label and select the top-scoring ones as a shortlist for further classification.

Original: The meta- and extreme-classificationtasks can either be learnt sequentially [8, 10] or jointly through dynamic label-shortlisting [17, 21].
Revised: The meta- and extreme-classification tasks can either be learnt sequentially [8, 10] or jointly through dynamic label-shortlisting [18, 22].

Original: Ideally, the labels grouped under one meta-label should be semantically similar to ensure the commonrepresentations learnt for meta- and extreme- tasks are relevant to both tasks [10].

Original: This has the addedadvantage that the negative labels selected by the shortlisting procedure will be the most confusingones, leading to hard negative sampling and improved learning of the extreme classifier [37].

Original: For extremely large label spaces, a single cluster size K is not optimal [51].
Revised: For extremely large label spaces, a single label resolution for shortlisting is not optimal [56].

Original: It is either  large enoughthat the meta-task itself becomes very expensive, or its individual clusters are so large that the shortlist contains too many candidate labels [51].
Revised: It is either too large that the meta-task itself becomes very expensive, or its individual clusters are so large that the shortlist contains too many candidate labels [56].

Original: Thus, methods like AttentionXML and XR-Transformer usea multi-level shortlisting procedure based on a Hierarchical Label Tree (HLT) .
Revised: Thus, methods like AttentionXML and XR-Transformer use a multi-level shortlisting procedure based on a Hierarchical Label Tree (HLT) .

Original: Label Trees: An HLT H = (cid:8) R (1) , . . .
Revised: Label Trees: An HLT H = (cid:8) R (1) , . . .

Original: , R ( T ) (cid:9) is a hierarchical clustering of labels into T levels ofsuccessively refined clusters R ( t ) =
Revised: , R ( T ) (cid:9) is a hierarchical clustering of labels into T levels of successively refined clusters R ( t ) =

Original: (cid:110) C ( t ) 1 , . .
Revised: (cid:110) C ( t ) 1 , . .

Original: , C ( t ) K t (cid:111) .
Revised: , C ( t ) K t (cid:111) .

Original: The clusters C ( t ) k of each level form a disjointpartitioning of the clusters of the next, C ( t ) k
Revised: The clusters C ( t ) k of each level form a disjoint partitioning of the clusters of the next, C ( t ) k

Original: ⊂ R ( t +1) , with the convention that the T + 1 th cluster isthe full label space R ( T +1) =
Revised: ⊂ R ( t +1) , with the convention that the T + 1 th level is the full label space R ( T +1) =

Original: [ L ] .
Revised: [ L ] .

Original: As such, each cluster C ( T ) k (or meta-label) at level T is representedby a set of labels, each cluster at level T − 1 is a set of level- T meta-labels, and thus a set of setsof labels, and so on.
Revised: As such, each cluster C ( T ) k (or meta-label) at level T is represented by a set of labels, each cluster at level T − 1 is a set of level- T meta-labels, and thus a set of sets of labels, and so on.

Original: In practice, label representations z l as an aggregate of their positive instancefeatures are leveraged to create an HLT.
Revised: In practice, label representations z l as an aggregate of their positive instance features are leveraged to create an HLT.

Original: Typically, z l is constructed using Φ tf-idf asz l := v l ∥ v l ∥ , where v l := (cid:88)
Revised: Typically, z l is constructed using Φ tf-idf as z l = v l ∥ v l ∥ , where v l = (cid:88)

Original: i : y ( i ) l = Φ tf-idf ( x ( i ) ) .
Revised: i : y ( i ) l = Φ tf-idf ( x ( i ) ) .

Original: The HLT is often constructed by recursively partitioning the label-space using balanced k-meansclustering in a top-down fashion [8, 17, 49, 51].
Revised: The HLT is often constructed by recursively partitioning the label-space using balanced k-means clustering in a top-down fashion [8, 18, 54, 56].

Original: For example, an HLT constructed using balanced2-means clustering has ⌊ log 2 L ⌋ levels, out of which we select T levels corresponding to differentmeta-label resolutions.
Revised: For example, an HLT constructed using balanced 2-means clustering has ⌊ log 2 L ⌋ levels, out of which we select T levels corresponding to different meta-label resolutions.

Original:  Method: CascadeXML
Revised:  Method: CascadeXML

Original: At its core, the CascadeXML model consists of three components: A pre-trained language model Φwhich allows to extract a document representation Φ ( a ) CLS ( x ) ∈ R d at different layers a ∈ [ A ] , an HLT
Revised: At its core, the CascadeXML model consists of three components: A pre-trained language model Φ which allows to extract a document representation Φ ( a ) CLS ( x ) ∈ R d at different layers a ∈ [ A ] , an HLT

Original: H = (cid:8) R (1) , . .
Revised: H = (cid:8) R (1) , . .

Original: , R ( T ) (cid:9) that divides the label space into K 1 < . . .
Revised: , R ( T ) (cid:9) that divides the label space into K 1 < . . .

Original: <
Revised: <

Original: K T <
Revised: K T <

Original: L = K T +1 successivelyrefined clusters, and a set of linear classifiers (cid:110) W ( t ) =
Revised: L = K T +1 successively refined clusters, and a set of linear classifiers (cid:110) W ( t ) =

Original: [ w ( t ) 1 , . .
Revised: [ w ( t ) 1 , . .

Original: . , w ( t ) K t ] ∈ R
Revised: . , w ( t ) K t ] ∈ R

Original: d
Revised: d

Original: × K t : t ∈ [ T + 1] (cid:111) .
Revised: × K t : t ∈ [ T + 1] (cid:111) .

Original: We select one layer for each meta-classifier to extract resolution-specific feature embeddingsϑ ( t ) :
Revised: For each meta-classifier, we select a layer in order to extract resolution-specific feature embeddings ϑ ( t ) :

Original: =
Revised: =

Original: Φ ( a t ) CLS ( x ) with 1 < a 1 < . . .
Revised: Φ ( a t ) CLS ( x ) with 1 < a 1 < . . .

Original: <
Revised: <

Original: a T <
Revised: a T <

Original: A = a T +1 .
Revised: A = a T +1 .

Original: Refining and Coarsening Label Vectors:
Revised: Refining and Coarsening Label Vectors:

Original: In order to transfer the representation of a shortlist ofa label vector between different layers of the HTL, we need to define two operations.
Revised: In order to transfer the representation of a shortlist of a label vector between different levels of the HLT, we need to define two operations.

Original: The first isrefining the resolution of a shortlist: Given a shortlist S = { C ( t ) k 1 , . . .
Revised: The first is refining the resolution of a shortlist: Given a shortlist S = { C ( t ) k 1 , . . .

Original: , C ( t ) k s } ⊂ R ( t ) of clusters of theresolution at level t , we want to calculate the representation of this shortlist in the next layer, denoted R ( t ) ↓ ( S ) .
Revised: , C ( t ) k s } ⊂ R ( t ) of clusters of the resolution at level t , we want to calculate the representation of this shortlist in the next level, denoted R ( t ) ↓ ( S ) .

Original: Since each cluster is defined as the set of its descendants, this is achieved by taking theunion over the clusters in the list.
Revised: Since each cluster is defined as the set of its descendants, this is achieved by taking the union over the clusters in the list.

Original: The second operation is to find the cover of meta-labels that covera given set of labels, R ( t ) ↑ ( S ) .
Revised: The second operation is to find the cover of meta-labels that envelope a given set of labels, R ( t ) ↑ ( S ) .

Original: This is achieved by identifying each of the clusters in the layer abovefor which there exists at least one descendant in the shortlist.
Revised: This is achieved by identifying each of the clusters in the level above for which there exists at least one descendant in the shortlist.

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: We define R ( t ) ( y ) = (cid:16) R ( t +1) ↑ ◦ . . .
Revised: We define R ( t ) ( y ) = (cid:16) [Equation] ( t + [Equation]

Original:  ◦ R ( T ) ↑ (cid:17) ( y ) to be the level- t cover of the label vector y ∈
Revised: ↑ ◦ . . . ◦ R ( T ) ↑ (cid:17) ( y ) to be the level- t cover of the label set y ⊂

Original: [ L ] .
Revised: [ L ] .

Original: Multi-resolution Dynamic Label Shortlisting: Concisely, we adapt dynamic label shortlisting [17]for multi-resolution training approaches.
Revised: Multi-resolution Dynamic Label Shortlisting: We adapt dynamic label shortlisting [18] for multiresolution training approaches.

Original: Formally, given a shortlist  of s clusters { C ( t ) k 1 , . . .
Revised: Formally, given a shortlist S ( t ) of s clusters { C ( t ) k 1 , . . .

Original: , C ( t ) k s } =:
Revised: , C ( t ) k s } =:

Original: S ( t ) ⊂
Revised: S ( t ) ⊂

Original: R ( t ) at level t , and the corresponding classification features ϑ ( t ) , a new shortlist S ( t +1) ⊂
Revised: R ( t ) at level t , and the corresponding classification features ϑ ( t ) , a new shortlist S ( t +1) ⊂

Original: R ( t +1) is generated as follows:
Revised: R ( t +1) is generated as follows:

Original: First, the k highest-scoring meta-labels are selected into S ( t ) top and thenext shortlist S ( t +1) is then given as the refinement:
Revised: The k highest-scoring meta-labels are selected into S ( t ) top and the next shortlist S ( t +1) is then given as the refinement:

Original: [Equation]
Revised: [Equation]

Original: At the first level, however, the shortlist is initialized to consider all meta labels S (1) =
Revised: At the first level, however, the shortlist is initialized to consider all meta labels S (1) =

Original: R (1) .
Revised: R (1) .

Original: It is ofessence that all true positive (meta-)labels appear at every level during training time [8, 17, 49] and isdifficult to ensure intrinsically due to imperfect recall rate of classifier at resolution t .
Revised: It is of essence that all true positive (meta-)labels appear at every level during training time [8, 18, 54]. However, this is difficult to ensure implicitly due to imperfect recall rate of classifiers. Hence, during training all meta-labels corresponding to ground-truth labels are added to the candidate set, leading to

Original: Hence, during
Revised: [Equation] Consequently, a single training step in CascadeXML looks as given in algorithm 1. Training Objective: At each level of the tree, the goal is to correctly identify the most likely meta-labels, which can be achieved through minimization of a one-vs-all loss, such as BCE [Equation]

Original: Input: instance x , labels y , clusters C , features Φ , classifiers W , weights α ϑ ( t ) ← Φ ( t ) CLS ( x ) ∀ t ∈ [ T + 1] /* Forward pass through Transformer */ loss ←
Revised: Input: instance x , labels y , clusters C , features Φ , classifiers W , weights α ϑ ( t ) ← Φ ( t ) CLS ( x ) ∀ t ∈ [ T + 1] /* Forward pass through Transformer */ loss ←

Original: S (1) ← C (1) /* Initialize shortlist */ for t in 1 , 2 , ..., T do y ( t ) ← R ( t ) ( y ) /
Revised: S (1) ← C (1) /* Initialize shortlist */ for t in 1 , 2 , ..., T do y ( t ) ← R ( t ) ( y ) /

Original: * Ground-truth meta-labels */ p i ← ⟨ w ( t ) k
Revised: * Ground-truth meta-labels */ p i ← ⟨ w ( t ) k

Original: i , ϑ ( t ) ⟩ ∀ i ∈ S ( t ) /
Revised: i , ϑ ( t ) ⟩ ∀ i ∈ S ( t ) /

Original: * Sparse prediction */ loss ← loss + α t |S ( t ) | (cid:80) l ∈S ( t ) L ( t )BC (cid:16) p l , y ( t ) l (cid:17)
Revised: * Sparse prediction */ loss ← loss + α t |S ( t ) | (cid:80) l ∈S ( t ) L ( t ) (cid:16) p l , y ( t ) l (cid:17)

Original: S ( t ) top ← (cid:110) C ( t ) k j : j ∈ Top k (cid:0)(cid:8) p i | i ∈ S ( t ) (cid:9)(cid:1)(cid:111) .
Revised: S ( t ) top ← (cid:110) C ( t ) k j : j ∈ Top k (cid:0)(cid:8) p i | i ∈ S ( t ) (cid:9)(cid:1)(cid:111) .

Original: /* Top-K selection */
Revised: /* Top-K selection */

Original: S ( t + [Equation] [Equation] [Equation]
Revised: S ( t +1) ← R ( t ) ↓ ( S ( t ) top ) ∪ R ( t +1)

Original: t )

Original: [Equation]

Original: R

Original: [Equation]

Original: t +1)

Original: ( y ) /
Revised: ( y ) /

Original: * Add ground-truth positives */ p i ← ⟨ w ( T +1)
Revised: * Add ground-truth positives */ p i ← ⟨ w ( T +1)

Original: k
Revised: k

Original: i , ϑ ( T +1) ⟩ ∀ i ∈ S
Revised: i , ϑ ( T +1) ⟩ ∀ i ∈ S

Original: ( T +1) /
Revised: ( T +1) /

Original: * Final prediction */ loss ← loss + α T +1 |S 
Revised: * Final prediction */ loss ← loss + α T +1 |S ( T +1)

Original: ( T +1) | (cid:80) l 
Revised: | (cid:80) l ∈S L ( T +1)

Original: [Equation] L ( T +1) BC ( p l , y l ) adjust Φ and W to reduce losstraining all meta-labels corresponding to ground-truth labels are added to the candidate set, leading to [Equation]
Revised: ( p l , y l ) adjust Φ and W to reduce loss

Original: Consequently, a single training step in CascadeXML looks as given in algorithm 1. Training Objective: At each level of the tree, the goal is to correctly identify the most likelymeta-labels, which can be achieved through minimization of a one-vs-all loss
Revised: The overall objective is a weighted sum of the individual layer losses

Original: [Equation]
Revised: [Equation]

Original: The overall objective is a weighted sum of the individual layer losses

Original: [Equation]
Revised: [Equation]

Original: where level T + 1 corresponds to the extreme classification task.
Revised: where level T + 1 corresponds to the extreme classification task.

Original: To balance out different loss scalesas a result of normalization by varied shortlist sizes, we re-scale losses from multiple resolutionsusing a simplere-scaling factor α ( t ) =
Revised: To balance out different loss scales as a result of normalization by varied shortlist sizes, we re-scale losses from multiple resolutions using a simple factor α ( t ) =

Original: |S ( t ) | / min t ∈
Revised: |S ( t ) | / min t ∈

Original: [ T +1] ( |S ( t ) | ) .
Revised: [ T +1] ( |S ( t ) | ) .

Original: Inference: The algorithm for inference remains the same as training, except we do not teacher-forcetrue (meta-)labels R ( t ) ( y ) during inference time (Eqn. 3).
Revised: Inference: The algorithm for inference remains the same as training, except we do not teacher-force true (meta-)labels R ( t ) ( y ) during inference time (Eqn. 3).

Original: The inference consists of one forward passthrough the transformer backbone, followed by successive shortlisting at all T resolutions, and finalextreme classification.
Revised: The inference consists of one forward pass through the transformer backbone, followed by successive shortlisting at all T resolutions, and final extreme classification.

Original: For a b -way branching tree with log b L levels, each level t has b t meta-labels,of which |S ( t ) | ∝ √ b t are selected for the shortlist.
Revised: For a b -way branching tree with log b L levels, each level t has b t meta-labels, of which |S ( t ) | ∝ √ b t are selected for the shortlist.

Original: The time complexity is
Revised: Therefore, the time complexity is

Original: [Equation]
Revised: [Equation]

Original: where T dnn denotes the time taken to compute transformer embeddings for an input.
Revised: where T dnn denotes the time taken to compute transformer embeddings for an input.

Original:  Experiments
Revised:  Main results

Original: Benchmark datasets & Evaluation Metrics
Revised: Benchmark datasets & Evaluation Metrics

Original: We benchmark the performance of CascadeXML onstandard datasets and evaluation metrics.
Revised: We benchmark the performance of CascadeXML on standard datasets and evaluation metrics, as shown in Table 2 and Table 3.

Original: In all, we use 5 public XMC datasets [4, 29], the statisticsof which are specified in appendix A.1.
Revised: In all, we use 5 public XMC datasets [4, 30], the statistics of which are specified in appendix A.1.

Original: However, due to space considerations and as our main focusis on extremely large output spaces, we present results for the largest of the publicly available datasets

Original: – Wiki-500K, Amazon-670K, and Amazon-3M, in the main paper.

Original: For all our experiments we use thesame raw text input, train-test split and sparse feature representations as used in [49, 17, 51]. We use
Revised: For all our experiments we use the same raw text input, train-test split and sparse feature representations as used in [54, 18, 56].

Original:  Precision@k (P@k) and Propensity-scored P@k (PSP@k) as the evaluation metrics. PSP@k focusesmore on the model’s performance on tail labels and the results are given in appendix A.3.3.
Revised: We use Precision@k (P@k) and Propensity-scored P@k (PSP@k) [16], which focuses more on the model’s performance on tail labels, as the evaluation metrics.

Original: Baseline Methods: We compare CascadeXML with strong baselines: AnnexML [40], DiSMEC[2], Parabel [34], eXtremeText [42], Bonsai [20], XMLCNN [25], XR-Linear [50], AttentionXML, 
Revised: Baseline Methods We compare CascadeXML with strong baselines: DiSMEC [2], Parabel [36], eXtremeText [47], Bonsai [21], XR-Linear [55], AttentionXML, X-Transformer [8], LightXML and

Original: X-Transformer [8], LightXML and XR-Transformer.Results on Wiki-500K, Amazon-670K and Amazon-3M are shown in Table 2.
Revised: XR-Transformer.

Original: The baseline results for other methods were obtained from [51, Table: 2].
Revised: The baseline results for these methods were obtained from [56, Table: 2].

Original: It may be noted that, in order to incorporate the global statistical information of the dataset, XR-Transformer concatenates sparse tf-idf features to those learnt via deep networks (denoted Φ dnn ) 
Revised: It may be noted that, in order to incorporate the global statistical information of the dataset, XR-Transformer concatenates sparse tf-idf features to those learnt via deep networks (denoted Φ dnn ) as:

Original: [Equation]
Revised: [Equation]

Original: For fair evaluation, we present separate comparison with methods which use (i) features from a DNNencoder only, and (ii) those obtained by combining DNN and tf-idf features. [Table]
Revised: For fair evaluation, we present separate comparison with methods which use (i) features from a DNN encoder only, and (ii) those obtained by combining DNN and tf-idf features. Model Ensemble Settings For CascadeXML, we follow the ensemble settings of LightXML and

Original: P@
Revised: XR-Transformer. Specifically, we ensemble a model each using BERT [11], RoBerta. [27] and XLNet [50] for Wiki10-31K and AmazonCat-13K.

Original: P
Revised: For the larger datasets, we make an ensemble of three BERT models with different seeds for random initialization. On the other hand, AttentionXML uses ensemble of 3 models and X-Transformer uses 9 model ensemble with BERT, RoBerta, XLNet large models with three difference indexers.

Original: [Table]
Revised: Metrics

Original: P@ P@ 
Revised: P@ P@ P@

Original: P@ P@ 
Revised: P@ P@ P@

Original: P@ P@ 
Revised: P@ P@ P@

Original: [Table]
Revised: Datasets

Original: [Table]
Revised: Wiki-500K Amazon-670K Amazon-3M

Original: [Table]
Revised: DiSMEC 70.

Original: Parabel ∗ 68.
Revised: Parabel ∗ 68.

Original: eXtremeText 65.
Revised: eXtremeText 65.

Original: Bonsai ∗ 69.
Revised: Bonsai ∗ 69.

Original: XR-Linear 65.
Revised: XR-Linear 65.

Original: DNN Features
Revised: DNN Features

Original: XML-CNN - - - 33.

Original: AttentionXML ∗ 76.
Revised: AttentionXML ∗ 76.

Original: LightXML ∗ 77.
Revised: LightXML ∗ 77.

Original: CascadeXML ∗ 78.
Revised: CascadeXML ∗ 78.

Original: [DNN(cid:76) tf-idf] Features X [Table] XR-
Revised: [DNN

Original: [Table]
Revised: [Table]

Original: ∗ 79. CascadeXML ∗ 81. respective feature settings are in bold . * implies ensemble result of 3 models, ** implies ensemble result of  9 models  and ‘-’ implies that the model does not scale for that dataset due to GPU memory constraints.
Revised: ‘ * ’ and ‘ ** ’ imply ensemble results of 3 models and 9 models respectively, and ‘-’ implies that the model does not scale for that dataset due to GPU memory constraints.

Original: Details about model ensembles can be found in the appendix.

Original: Empirical Performance:
Revised: Empirical Performance:

Original: As demonstrated in Table 2, CascadeXML, which leverages the multilayered transformer architecture to learn label resolution specific feature representations, is empiricallysuperior to all existing XMC approaches whether using: (i) tf-idf features, or (ii) DNN-baseddensefeatures (iii) or concatenated tf-idf and dense features.
Revised: As demonstrated in Table 2, CascadeXML, which leverages the multilayered transformer architecture to learn label resolution specific feature representations, is empirically superior to all existing XMC approaches whether using: (i) tf-idf features, or (ii) DNN-based dense features, or (iii) both.

Original: This is despite CascadeXML being a simpleend-to-end trainable algorithm without using any bells or whistles like multi-stage procedures[49, 51], reclustering [51] or bootstrapped training [49, 51].
Revised: This is despite CascadeXML being a simple end-to-end trainable algorithm without using any bells or whistles like multi-stage procedures [54, 56], reclustering [56] or bootstrapped training [54, 56].

Original: For example, on Amazon-3M dataset, CascadeXML significantly outperforms all DNN-based approaches whileonly taking  24 hours totrain on a single  GPU.
Revised: For example, on Amazon-3M dataset, CascadeXML significantly outperforms all DNN-based approaches while taking only 24 hours to train on a single Nvidia A GPU.

Original: In contrast, X-Transformer needs to ensemble 9 models and 23 days to trainon 8 GPUs to reach the empirical performance (c.f. page 2, [51]), and LightXML remains unscalable. 
Revised: In contrast, X-Transformer needs to ensemble 9 models and 23 days to train on 8 GPUs to reach the empirical performance (c.f. page 2, [56]), and LightXML remains unscalable. [Table] We report results on Wiki10-31K and AmazonCat-13K in Table 3.

Revised: As a standard practice in the domain [6, 18, 56], we use 256 tokens as an input sequence length to the model after truncation.

Revised: We find this sequence length to be sufficient for competent empirical performance and hence, we do not witness any empirical benefits of leverage sparse tf-idf features for these datasets.

Revised: Notably,

Original:  On the concatenated feature space, CascadeXML also significantly outperforms XR-Transformeron 8 out of 9 dataset-metric combination  with notable ∼ 6% improvement on Wiki-500K dataset for P@5 metric.
Revised: CascadeXML achieves best results on AmazonCat-13K dataset and performs at par with XR-Transformer without leveraging sparse tf-idf features. Thus, CascadeXML’s unique multi-resolution approach is a favorable choice not only for datasets with extremely large output spaces, but also for datasets with tens of thousand labels. On the concatenated feature space, CascadeXML also significantly outperforms XR-Transformer on 8 out of 9 dataset-metric combination (ref: Table 2) with notable ∼ 6% improvement on Wiki-500K dataset for P@5 metric.

Original: As XR-Tranformer learns transformer features for label resolutions only upto 32K( 2 15) clusters [51, Table 6], incorporating tf-idf features via a computation-intensive XR-Linear[50] for final extreme classification task becomes an integral part of their pipeline.
Revised: As XR-Tranformer learns transformer features for label resolutions only up to 2 15 clusters [56, Table 6], incorporating tf-idf features via a computation-intensive XR-Linear [55] for final extreme classification task becomes an integral part of their pipeline.

Original: On the otherhand, CascadeXML does not have any such inherent limitation, and even without tf-idf features, itoutperforms XR-Transformer on 6 out of 9 dataset-metric combinations.
Revised: On the other hand, CascadeXML does not have any such inherent limitation, and even without tf-idf features, it outperforms XR-Transformer on 7 out of 9 dataset-metric combinations (ref: Table 2).  Computational cost & Ablation results

Original: Inference and Training Time: The comparison on inference times is shown in Table 3.
Revised: Single Model Comparison & Training Time Empirical results and training time of a single

Original: CascadeXMLclocks the fastest inference speed as compared to all previous DNN-based XMC models.
Revised: CascadeXML model has been compared to single instance performance of state-of-the-art DNN based

Original:  Notably, itis ∼ 1 . 5 × and ∼ 2 × faster as compared to LightXML and XR-Transformer respectively.
Revised: XMC approaches in Table 4. Notably, CascadeXML, without leveraging any sparse tf-idf features, performs at par with XR-Transformer which benefits from using the extra statistical information about the entire (not truncated) data point.

Original: In terms oftraining time, CascadeXML is upto an order of magnitude faster than LightXML on a single GPU. 
Revised: When comparing training time, CascadeXML is up to an order of magnitude faster than LightXML on a single GPU. In multi-GPU setting, CascadeXML trains in 7.

Original: When using multiple GPUs, its wall clock time on  4  GPUs is lower than both XR-Transformer and
Revised: hours on Wiki-500K dataset using only 4 NVidia V100 GPUs as compared to XR-Transformer and

Original: AttentionXML  on 8 GPUs. 
Revised: AttentionXML which require 12.5 hours on 8 GPUs. CascadeXML not only reduces the training time across datasets but also requires half the number of GPUs to do so. This proves that CascadeXML is significantly more compute efficient as compared to previous DNN based XMC approaches. [Table]

Original: Exact training times can be found in Section A.3.2 in the appendix.
Revised: Inference Time CascadeXML clocks the fastest inference speed as compared to all previous DNN-based XMC models, as shown in Table 5. Notably, CascadeXML is ∼ 1 . 5 × and ∼ 2 × faster at inference time compared to LightXML and XR-Transformer

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: Tail Labels Performance of CascadeXML on tail labels has been compared to baseline XMC methods in Table 6. We note that for smaller datasets - Wiki10-31K and AmazonCat13K - PfastreXML [16] significantly outperforms all other methods on PSP metrics. This is to be expected, as this method specifically optimizes for tail labels.

Revised: However, for larger datasetsAmazon-670K and Wiki-500K - CascadeXML outperforms the strong baselines like XR-Transformer and AttentionXML by 5-8% on PSP@3 and PSP@5 metrics while also significantly outperforming

Revised: PfastreXML over these datasets.

Original: Impact of label clusters size:
Revised: These results conclude that our unique end-to-end multi-resolution training approach is not only empirically superior to previous approaches in P@K metrics but also outperforms strong XMC baselines in performance over tail labels [Table] [Table] [Table]

Revised: 670K

Revised: [Table]

Revised: w/o Rescaling

Revised: [Table]

Revised: Default

Revised: [Table]

Revised: Wiki-500K 2 15 Clusters 76.

Original:  Earlier works have argued that using finegrained clusters leads to better model performance [17, 21, 31].
Revised: w/o Rescaling 76. Default 51. [Table] Impact of label clusters size Earlier works have argued that using finegrained clusters leads to better model performance [18, 22, 32].

Original: With increasednumber of label clusters at the penultimate level of HLT, the multi-resolutiontasks tend to get more in-sync witheach other [21] and hence enables transfer learning across different stages oflearning pipelines [10, 31].
Revised: With increased number of label clusters at the penultimate level of HLT, the multi-resolution tasks tend to get more in-sync with each other [22] and hence enables transfer learning across different stages of learning pipelines [10, 32].

Original: Our mutliresolution training enables CascadeXMLto use 2 16 label clusters for penultimate label resolution, double that of XRTransformer.
Revised: Our mutliresolution training enables CascadeXML to use 2 16 label clusters for penultimate label resolution, double that of XRTransformer.

Original: As Table 4 shows, increasing the resolution at the last shortlistingstep still results in non-trivial improvements, highlighting the need for architectures that can efficiently handle high-resolution meta-labels.
Revised: As Table 7 shows, increasing the resolution at the last shortlisting step results in non-trivial improvements, highlighting the need for architectures that can efficiently handle highresolution meta-labels.

Original: Impact of re-scaling per-resolution loss: For extreme datasets, we witness more efficient trainingby having a larger shortlist size for finer label resolutions (see appendix A.2.2).
Revised: Impact of re-scaling per-resolution loss For extreme datasets, we witness more efficient training by having a larger shortlist size for finer label resolutions (see appendix A.2.2).

Original: This results in variedloss scales across label resolutions leading to one resolution dominating the training.
Revised: This results in varied loss scales across label resolutions leading to one resolution dominating the training.

Original: Comparingthe “Default” and “w/o Rescaling” rows for respective datasets in Table 4, indicates that rescaling asintroduced in section 3 helps CascadeXML train more effectively.
Revised: Comparing the “Default” and “w/o Rescaling” rows for respective datasets in Table 7, indicates that rescaling as introduced in section 3 helps CascadeXML train more effectively.

Original:  Discussion
Revised:  Discussion

Original: As shown above, CascadeXML substantially outperforms current state-of-the-art approaches bothin terms of computational efficiency, and prediction performance.
Revised: As shown above, CascadeXML substantially outperforms current state-of-the-art approaches both in terms of computational efficiency, and prediction performance.

Original: It achieves this while beingsimpler than XR-Transformer and more scalable than LightXML.
Revised: It achieves this while being simpler than XR-Transformer and more scalable than LightXML.

Original: Revisiting Table 1, the extractionof intermediate [CLS] representations enables us to perform end-to-end multi-resolution trainingwith resolution-specific attention maps from a single transformer model.
Revised: Revisiting Table 1, the extraction of intermediate [CLS] representations enables us to perform end-to-end multi-resolution training with resolution-specific attention maps from a single transformer model.

Original: From Table 2, we show improvements on both LightXML and XR-Transformer, which are transformer-based models sharingfeature representations across multiple label resolutions.
Revised: From Table 2, we show improvements over both LightXML and XR-Transformer, which are transformer-based models sharing feature representations across multiple label resolutions.

Original: We attribute this increased performanceto CascadeXML’s end-to-end multi-resolution training pipeline and property of adapted featureembeddings.
Revised: We attribute this increased performance to CascadeXML’s end-to-end multi-resolution training pipeline and property of adapted feature embed- dings.

Original: The presented architecture, therefore, combines the strength of different previousapproaches, without inheriting their limitations.
Revised: The presented architecture, therefore, combines the strength of different previous approaches, without inheriting their limitations.

Original: We postulated that the (meta-)classification tasks at different resolutions need to attend to differenttokens in the input sequence, a property lacking in recent approaches [8, 17, 51].
Revised: We postulated that the (meta-)classification tasks at different resolutions need to attend to different tokens in the input sequence, a property lacking in recent approaches [8, 18, 56].

Original: This intuition iscorroborated by the fact that the attention maps learned by CascadeXML differ significantly betweendifferent resolutions.
Revised: This intuition is corroborated by the fact that the attention maps learned by CascadeXML differ significantly between different resolutions.

Original: We provide graphs and supporting data in the appendix.
Revised: We provide graphs and supporting data in Appendix B.

Original: Obtaining label resolution specific feature embeddings in a pre-trained transformer model requires tomake the compromise of extracting these at earlier transformer layers, where the token representationsare not yet as refined as in the final layer [43].
Revised: Obtaining label resolution specific feature embeddings in a pre-trained transformer model requires to make the compromise of extracting these at earlier transformer layers, where the token representations are not yet as refined as in the final layer [48].

Original: In contrast to the extreme classification task, whichrequires placing true labels exactly at the top-5 positions, the shortlisting task only requires all truemeta-labels be recalled within the shortlist, which is much longer than five elements.
Revised: In contrast to the extreme classification task, which requires placing true labels exactly at the top-5 positions, the shortlisting task only requires all true meta-labels be recalled within the shortlist, which is much longer than five elements.

Original: We observethat the meta-classifiers are able to achieve high-recall (as shown in appendix) at the shortlist lengtheven with the less refined features.
Revised: We observe that the meta-classifiers are able to achieve high-recall (as shown Appendix B) at the shortlist length even with the less refined features.

Original: CascadeXML is, therefore, successful in efficiently leveragingintermediate transformer representations for the task of label shortlisting while keeping featurerepresentations separate for each label resolution.
Revised: CascadeXML is, therefore, successful in efficiently leveraging intermediate transformer representations for the task of label shortlisting while keeping feature representations separate for each label resolution.

Original: The structure of the model means that earlier layers in the transformer receive gradients from multipleresolutions, whereas the final layers’ back propagation is limited to gradients from the extremeresolution (cf.
Revised: The structure of the model means that earlier layers in the transformer receive gradients from multiple resolutions, whereas the final layers’ back propagation is limited to gradients from the extreme resolution (cf.

Original: Figure 1).
Revised: Figure 1).

Original: This setup ensures that a portion of the representational capabilities isexclusively available for the extreme task and hence, the final transformer layers are increasinglymore suitable for classification at the extreme resolution.
Revised: This setup ensures that a portion of the representational capabilities is exclusively available for the extreme task and hence, the final transformer layers are increasingly more suitable for classification at the extreme resolution.

Original: At the same time, since the gradients fromthe finer tasks are also relevant for the coarser meta-label shortlisting task, so additional downstreamtask does not hurt the shortlisting performance at the intermediate transformer layers.
Revised: At the same time, since the gradients from the finer tasks are also relevant for the coarser meta-label shortlisting task, so additional downstream task does not hurt the shortlisting performance at the intermediate transformer layers.

Original:  Other Related Works
Revised:  Other Related Works

Original: The focus of a majority of initial works in XMC has been on using fixed features for learning theclassifier with one of the following class of methods : (i) one-vs-rest [48, 2, 47, 3], (ii) label tree based[33, 20, 42, 16], (iii) decision tree methods [35, 39, 28], and (iv) label embedding based [5, 40, 12]classifiers.
Revised: The focus of a majority of initial works in XMC has been on using fixed features for learning the classifier with one of the following class of methods : (i) one-vs-rest [53, 2, 52, 3, 41], (ii) label tree based [35, 21, 47, 17], (iii) decision tree methods [37, 44, 29], and (iv) label embedding based [5, 45, 13] classifiers.

Original: Beyond the above algorithmic categorization of approaches, computational considerationson scaling the training process via negative sampling [14], and sketching [30] have also been studied.
Revised: Beyond the above algorithmic categorization of approaches, computational considerations on scaling the training process via negative sampling [15], and sketching [31] have also been studied.

Original: Furthermore, the statistical consequences of negative sampling [37] and missing labels [15, 36] haveled to novel insights in designing unbiased loss functions.
Revised: Furthermore, the statistical consequences of negative sampling [39] and missing labels [16, 38, 43, 42] have led to novel insights in designing unbiased loss functions.

Original: With advances in deep learning, joint learning of features and classifier has been the focus of mostrecent approaches.
Revised: With advances in deep learning, joint learning of features and classifier has been the focus of most recent approaches.

Original: Building on the first steps for developing convolutional neural networks for textclassification [22], deep extreme classification was introduced in XML-CNN [26].
Revised: Building on the first steps for developing convolutional neural networks for text classification [23], deep extreme classification was introduced in XML-CNN [26].

Original: This work precededthe recent developments on employing BiLSTM and transformer encoders discussed in detail the
Revised: This work preceded the recent developments on employing BiLSTM and transformer encoders discussed in detail the

Original: Introduction section.
Revised: Introduction section.

Original: There has been a recent surge in works to tackle XMC for instances with shorttext inputs and those in which labels are endowed with textual descriptions [10, 9,31, 32, 38].
Revised: There has been a recent surge in works to tackle XMC for instances with short text inputs and those in which labels are endowed with textual descriptions [10, 9, 32, 33, 40].

Original: Lately,the XMC setting has also been extended to predict unseen labels in zero-shot learning [13, 52, 44].
Revised: Lately, the XMC setting has also been extended to predict unseen labels in zero-shot learning [14, 57, 49].

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduced a novel paradigm for fine-tuning transformer models in XMC settings.
Revised: In this paper, we introduced a novel paradigm for fine-tuning transformer models in XMC settings.

Original: In contrast to the existing methods, which consider transformer as a blackbox encoder, we leverageits multi-layered architecture to learn data representation corresponding to different resolutions in the
Revised: In contrast to the existing methods, which consider transformer as a blackbox encoder, we leverage its multi-layered architecture to learn data representation corresponding to different resolutions in the

Original: HLT as well as fine-grained labels at the level of the extreme classifier.
Revised: HLT as well as fine-grained labels at the level of the extreme classifier.

Original: The proposed instantiation ofour framework in the form of CascadeXML not only yields state-of-the-art prediction performance,but is also end-to-end trainable (without intermediate reclustering steps), simpler to implement,and fast on training and inference.
Revised: The proposed instantiation of our framework in the form of CascadeXML not only yields state-of-the-art prediction performance, but is also end-to-end trainable (without intermediate reclustering steps), simpler to implement, and fast on training and inference.

Original: Beyond further research in extreme classification towards fullyharnessing the representation capabilities of transformer encoders, we believe our approach caninspire future multi-resolution architectures in other domains as well which leverage label hierarchy.
Revised: Beyond further research in extreme classification towards fully harnessing the representation capabilities of transformer encoders, we believe our approach can inspire future multi-resolution architectures in other domains as well which leverage label hierarchy.

Original:  Checklist
Revised:  Acknowledgments

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: The authors would like to thank Devaansh Gupta and Mohadmmadreza Qaraei for useful discussions.

Original: contributions and scope? [Yes] See Section ?? .
Revised: They also acknowledge the support of CSC – IT Center for Science, Finland, as well as the Aalto

Original: (b) Did you describe the limitations of your work? [Yes] In the appendix(c) Did you discuss any potential negative societal impacts of your work?
Revised: Science-IT project, for providing the required computational resources.

Original: [Yes] In theappendix(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: This research is supported in part by Academy of Finland grants : Decision No. 348215 and 347707. A Appendix A.1 Dataset Details & Evaluation Metrics

Original: [Yes]2.
Revised: As stated earlier, the main application of Extreme Multi-label Text Classification is in e-commerce - product recommendation and dynamic search advertisement - and in document tagging, where the objective of an algorithm is to correctly recommend/advertise among the top-k slots. Thus, for evaluation of the methods, we use precision at k (denoted by P @ k ), and its propensity scored variant (denoted by PSP @ k ) These are standard and widely used metrics by the XMC community [4].

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A](b) Did you include complete proofs of all theoretical results?
Revised: For each test sample with observed ground truth label vector y ∈ { 0 , 1 } L and predicted vector ˆ y ∈ R L , P @ k is given by :

Original: [N/A]
Revised: [Equation]

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: where top@ k (ˆ y ) returns the k largest indices of ˆ y . Since P@ k treats all the labels equally, it doesn’t reveal the performance of the model on tail labels.

Original: [Yes] Code is in  the 
Revised: However, because of the long-tailed distribution in XMC datasets, one of the main challenges is to predict tail labels correctly, which may be more valuable and informative compared to head classes. A set of metrics that have been established in XMC to evaluate tail performance are propensity-scored version of precision. These PSP@ k were introduced in [ ? ], and use a weighting factor based on a propensity score p ℓ to give more weight to tail labels: [Equation]

Original: supplementary material
Revised: We use the empirical values for p ℓ as proposed in [ ? ]. [Table] A.2 Model Details [Table]

Revised: Transformer layers put in brackets next to the label resolution imply that a concatenation of the [CLS] token of the respective layers has been used for label shortlisting at that resolution.

Revised: A.2.1 Model Hyperparameters

Revised: CascadeXML optimizes the training objective using Binary Cross Entropy loss as the loss function and AdamW [28] as the optimizer.

Original: , the data is publically available.
Revised: We use different learning rates for the transformer encoder and the (meta-)label weight vectors as we need to train the weight vectors from random initialization in contrast to fine-tuning the transformer encoder. Specifically, the transformer encoder is fine-tuned at a learning rate of 10 − 4 , while the weight vectors are trained at a learning rate of 10 − 3 . The learning rate schedule consists of a constant learning rate for most of the iterations, with a cosine warm-up at the beginning and cosine annealing towards the end of the schedule. In multi-GPU setting, we use a batch size of 64 per GPU (256 total) across 4 GPUs.

Original: (b) Did you specify all the  training details (e.g., data splits, hyperparameters,
Revised: In single GPU setting, we still use a batch size of 256 by accumulating gradients for 4 iterations. The hyperparameter settings in detail have been mentioned in Table 9. A.2.2 Ensemble Training Time As shown in Table 10, CascadeXML achieves the lowest training time across all datasets except

Revised: Amazon-3M using only 4 GPUs as compared to XR-Transformer which leverages 8 GPUs.

Revised: Note that on Amazon-3M, XR-Transformer achieves a slightly lower training time.

Original: how they
Revised: However, XR-Transformer using 2x the number of GPUs and does not train the DNN model on full 3 million label resolution. XR-Transformer trains the DNN model using a classification training object comprising of only 2label clusters [56, Table 7] and then leverages XR-Linear [55], a linear solver, to scale up to 3M labels. On the other hand, CascadeXML trains an ensemble of 3 models to full resolution of 3 million labels in 30 hours using only 4 GPUs. [Table]

Revised: A.3 Leveraging Sparse Features

Revised: As we are using BERT for the transformer backbone of our method, we have to truncate the input sequences to a limited number of tokens (see Table 9).

Revised: This truncation results in loss of information.

Revised: Thus, following XR-Transformer’s lead we combine the features trained by CasadeXML with statistical information in the form of sparse tf-idf representation of the full input in an additional OVA classifier.

Revised: The concatenated features are constructed as [18]:

Revised: [Equation]

Revised: We use a version of DiSMEC [2] instead of using XR-Linear [55] - as done in XR-Transformer - as our external OVA classifier for Φ cat .

Revised: Even though XR-Linear achieves slightly better performance than

Revised: DiSMEC across datasets (Table: 1), we find DiSMEC to be more resource efficient than XR-Linear.

Revised: To quantify, DiSMEC runs successfully on 116GB RAM for all datasets, while XR-Linear requires close to 470GB RAM for Amazon-3M. Next, we discuss the application of DiSMEC over Φ cat .

Revised: DiSMEC DiSMEC is a linear multilabel classifier that minimizes an L 2 -regularized squared hingeloss, followed by a pruning step to only keep the most important weights.

Revised: Thus, the loss for a given weight matrix W =

Revised: [ w 1 , . .

Revised: . , w L ] is given by

Revised: [Equation]

Revised: Crucially, from the point of view of the OVA classifiers, the input features Φ cat ( x i ) are constant.

Revised: This means that the task decomposes into independent optimization problems for each label, minimizing

Revised: [Equation]

Revised: This allows for trivial parallelization of the task across CPU cores, and also means that the weights w j can be pruned as soon as the sub-problem is solved.

Revised: Consequently, there is no need to ever store the entire weight matrix, improving memory efficiency.

Revised: The objective function (7) has a continuous derivative, and its Hessian is well defined everywhere except exactly at the decision boundary.

Revised: Consequently, it can be minimized using a second-order

Revised: Newton method.

Revised: A discussion of this in the context of linear classification can be found in [12].

Revised: B Visualizations and Analysis

Revised: In this section we provide visualizations and additional data that corroborate our interpretation that different attention- and feature maps are needed for classification at different granularities of the label tree.

Revised: In Figure 2, the attention of the [CLS] token to itself in the previous layer is visualized.

Revised: If this self-attention were large, then Φ ( t ) CLS would be mostly a function of Φ ( t − 1) CLS .

Revised: In such a case, Φ ( t − 1) CLS would contain less information than Φ ( t ) CLS (data-processing inequality), but the meta-labels

Revised: R ( t − 1) ( y ) at level t − 1 contain strictly less information than R ( t ) ( y ) .

Original: were chosen)?
Revised: Thus, is the [CLS] token embedding had strong feed-forward characteristics, Φ ( t − 1) CLS would have to contain all the information also about the extreme-level labels, and thus have limited representation capacity for the level-t task. Layer S e l fA tt e n ti on Pretrained Bert

Revised: CascadeXML

Revised: Layer

Original: [Yes]
Revised: [Figure]

Revised: n ti on

Revised: Pretrained Bert

Revised: CascadeXML the later layers much more strongly in

Revised: CascadeXML than in pretrained BERT.

Revised: Luckily, Figure 2 is a sanity-check that shows that this is not the case.

Revised: Starting from layer 6 , where the first meta-task is placed, the [CLS] token is almost entirely re-assembled at each layer from the embeddings of the other tokens – much more strongly than in a pretrained BERT.

Revised: This allows each layer to extract the information best suited for classification at the given hierarchy level.

Revised: We can also detect some qualitative differences in the attention maps at different resolutions: The entropy, i.e. how much the attention is concentrated or spread across different tokens, changes significantly between levels.

Revised: This is not an artifact of the pretrained BERT model, but appears to be learned during fine-tuning.

Revised: In Figure 4, we analyse the flow of information and visualize how much processing is happening in a given layer with respect to the [CLS] token.

Revised: We rely on projection weighted canonical correlation analysis [34] for this task.

Revised: This allows to compare the representations at different layers in a way that is invariant to any linear transformations.

Original: In the appendix A.2.
Revised: We primarily make two observations from Figure 4. First, we observe that [CLS] token embeddings of layers 6, 8, 10 and 12 are more closely related in CascadeXML than in LightXML. This is expected as LigthXML only trains the (meta-)label weight vectors using the [CLS] token embeddings of the bottom layers. Because the multi-resolution training objectives differ only in granularity, many

Original: (c) Did you report error bars (e
Revised: Layer E n t r opy

Original: g., with respect to the random seed after running experi-
Revised: Pretrained

Original: ments multiple times)?
Revised: CascadeXML [Figure]

Original: [No] It is not a standard practice in this domain. (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Layers

Original: [Yes]
Revised: [Figure]

Original: In the appendix A. If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: BERT

Original: [Yes]
Revised: [Figure]

Original: In Section 4: Benchmark datasets (b) Did you mention the license of the assets?
Revised: features required to distinguish coarse meta-labels are useful in determining finer meta-labels as well.

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: Hence, the similarity between consecutive representations is expected to be strong.

Original: using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: On the other hand, when only looking at CascadeXML’s points (in blue) in Figure 4, we observe that the tasks in the first meta-classifier and the extreme classifier are substantially different.

Original: [N/A] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: This implies both training objectives require different representations that cannot be provided by a LightXML-/XRTransformer-style model which use same attention maps (and hence, same [CLS] token embeddings) for all resolutions. Dataset

Original: applicable?
Revised: Level

Original: [N/A]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: The ability to use earlier layers’ [CLS] token representation for the meta-task crucially depends on the fact that these representations are still sufficient for achieving high recall in the shortlisting task.

Original: [N/A]
Revised: As shown in Table 11, the shortlisting achieves very good recall rates. In particular, the very first shortlist, with the “weakest” features, achieves almost perfect recall.