Original: CascadeXML: End-to-end Multi-Resolution Learning for Extreme Multi-Label Text Classification
Revised: CascadeXML: End-to-end Multi-Resolution Learning for Extreme Multi-Label Text Classification

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Extreme Multi-label Text Classification (XMC) involves learning a classifier thatcan assign an input with a subset of most relevant labels from millions of labelchoices.
Revised: Extreme Multi-label Text Classification (XMC) involves learning a classifier thatcan assign an input with a subset of most relevant labels from millions of labelchoices.

Original: Recent approaches, such as XR-Transformer and LightXML, leveragea transformer instance to achieve state-of-the-art performance.
Revised: Recent approaches, such as XR-Transformer and LightXML, leveragea transformer instance to achieve state-of-the-art performance.

Original: However, in thisprocess, these approaches need to make various trade-offs between performance andcomputational requirements.
Revised: However, in thisprocess, these approaches need to make various trade-offs between performance andcomputational requirements.

Original: A major shortcoming, as compared to the Bi-LSTMbased AttentionXML, is that they fail to keep separate feature representations foreach resolution in a label tree.
Revised: A major shortcoming, as compared to the Bi-LSTMbased AttentionXML, is that they fail to keep separate feature representations foreach resolution in a label tree.

Original: We thus propose CascadeXML, an end-to-end multiresolution learning pipeline, which can harness the multi-layered architecture of atransformer model for attending to different label resolutions with separate featurerepresentations.
Revised: We thus propose CascadeXML, an end-to-end multiresolution learning pipeline, which can harness the multi-layered architecture of atransformer model for attending to different label resolutions with separate featurerepresentations.

Original: CascadeXML significantly outperforms all existing approacheswith non-trivial gains obtained on benchmark datasets consisting of up to threemillion labels.
Revised: CascadeXML significantly outperforms all existing approacheswith non-trivial gains obtained on benchmark datasets consisting of up to threemillion labels.

Original: Code for CascadeXML will be made publicly available.
Revised: Code for CascadeXML will be made publicly available.

Original:  Introduction
Revised:  Introduction

Original: Extreme multi-label text classification (XMC) deals with the problem of predicting the most relevantsubset of labels from an enormously large label space spanning up to millions of labels.
Revised: Extreme multi-label text classification (XMC) deals with the problem of predicting the most relevantsubset of labels from an enormously large label space spanning up to millions of labels.

Original: Overthe last decade, extreme classification has found many applications in e-commerce, like productrecommendation [6] and dynamic search advertisement [15] in document tagging [1] and opendomain question answering [7, 25] and thus, practical solutions to this problem can have significantand far-reaching impact.
Revised: Over the years,extreme classification has found many applications in e-commerce, like product recommendation [7]and dynamic search advertisement [14], document tagging [1] and open-domain question answering[6, 24] and thus, practical solutions to this problem can have significant and far-reaching impact.

Original: The output space in extreme classification is not only large but also, the labels follow Zipf’s law wheremost of the labels are tail labels [2, 16].
Revised: The output space in extreme classification is not only large but also, the labels follow Zipf’s law wheremost of the labels are tail labels [2, 15].

Original: For example, in a dataset of Wikipedia corpus containing500K labels, only 2.1% labels annotate more than 100 data points and 60% of the labels annotate lessthan 10 data points!
Revised: For example, in a dataset of Wikipedia corpus containing500K labels, only 2.1% labels annotate more than 100 data points and 60% of the labels annotate lessthan 10 data points!

Original: This inherently makes learning meaningful representations at an extreme scale achallenging problem.
Revised: This inherently makes learning meaningful representations at an extreme scale achallenging problem.

Original: Further, as the memory and computational requirements grow linearly withthe size of the label space, naive treatment by the application of classical methods and off-the-shelfsolvers fails to deal with such large output spaces [8, 51].
Revised: Further, as the memory and computational requirements grow linearly withthe size of the label space, naive treatment by the application of classical methods and off-the-shelfsolvers fails to deal with such large output spaces [8, 48].

Original: In extreme scenarios with millions of labels, a surrogate step of shortlisting candidate labels becomescrucial to perform extreme classification and has been adapted by many popular approaches [10, 18,22, 49, 54].
Revised: In extreme scenarios with millions of labels, a surrogate step of shortlisting candidate labels becomescrucial to perform extreme classification and has been adapted by many popular approaches [10, 17,21, 46, 51].

Original: For each data point, these approaches first solve a simpler task with coarse label-clustersor meta-labels as the label space to create a shortlist of relevant labels for the extreme classificationtask.
Revised: For each data point, these approaches first solve a simpler task with coarse label-clustersor meta-labels as the label space to create a shortlist of relevant labels for the extreme classificationtask.

Original: This effectively reduces the training time and computational complexity of the extreme task tothat of the shortlisting step which is O ( √ L ) , where L is the number of labels.
Revised: This effectively reduces the training time and computational complexity of the extreme task tothat of the shortlisting step which is O ( √ L ) , where L is the number of labels.

Original: [Table]
Revised: [Table]

Original: Existing works leverage label representation as a centroid of their annotated instances to create a
Revised: Existing works leverage label representation as a centroid of their annotated instances to create a

Original: Hierarchical Label Tree (HLT) [36, 52].
Revised: Hierarchical Label Tree (HLT) [34, 49].

Original: Works like [8, 10, 18, 22, 49] use label clusters at a certainlevel of the HLT as meta-labels which are in turn used to shortlist candidate labels for the extremetask.
Revised: Works like [8, 10, 17, 21, 46] use label clusters at a certainlevel of the HLT as meta-labels which are in turn used to shortlist candidate labels for the extremetask.

Original: In contrast, XR-Transformer [54] and AttentionXML [52] leverage multiple levels of the HLT,such that each level corresponds to a certain label resolution in the tree structure [54].
Revised: In contrast, XR-Transformer [51] and AttentionXML [49] leverage multiple levels of the HLT,such that each level corresponds to a certain label resolution in the tree structure [51].

Original: 1.1 Motivation: Strengths and Weaknesses of Current Approaches
Revised: 1.1 Motivation: Strengths and Weaknesses of Current Approaches

Original: The earliest work to successfully combine label-tree based shortlisting and attention-based deepencoders is AttentionXML [52], which employs multiple Bi-LSTMs models that are trained sequentially for each tree resolution.
Revised: The earliest work to successfully combine label-tree based shortlisting and attention-based deepencoders is AttentionXML [49], which employs multiple Bi-LSTMs models that are trained sequentially for each tree resolution.

Original: More recent approaches [8, 18, 49, 54] replaced the model architecturewith a more powerful transformer model [44] and fine-tune a pre-trained instance such as Bert [11].
Revised: More recent approaches [8, 17, 46, 51] replaced the model architecturewith a more powerful transformer model [41] and fine-tune a pre-trained instance such as Bert [11].

Original: Without careful designs, such models are computationally very expensive and earlier works [8, 49]could not effectively leverage transformers for both computation and performance on XMC tasks.
Revised: Without careful designs, such models are computationally very expensive and earlier works [8, 46]could not effectively leverage transformers for both computation and performance on XMC tasks.

Original: Next, we discuss two contemporary works LightXML [18] and XR-Transformer [54].
Revised: Next, we discuss two contemporary works LightXML [17] and XR-Transformer [51].

Original: LightXML introduces the concept of dynamic negative sampling, which replaces pre-computed labelshortlists with a dynamically calculated shortlist that changes as the model’s weights get updated.
Revised: LightXML introduces the concept of dynamic negative sampling, which replaces pre-computed labelshortlists with a dynamically calculated shortlist that changes as the model’s weights get updated.

Original: This enables end-to-end training with a single model by using the final feature representation of thetransformer encoder for both the meta- and the extreme-classification task.
Revised: This enables end-to-end training with a single model by using the final feature representation of thetransformer encoder for both the meta- and the extreme-classification task.

Original: Unfortunately, these twotasks appear to interfere with one another [22].
Revised: Unfortunately, these twotasks appear to interfere with one another [21].

Original: We hypothesize that this is because the meta taskneeds the attention maps to focus on different tokens than the extreme task.
Revised: We hypothesize that this is because the meta taskneeds the attention maps to focus on different tokens than the extreme task.

Original: Furthermore, it only usesa single-level tree, which prevents scaling to the largest datasets.
Revised: Furthermore, it only usesa single-level tree, which prevents scaling to the largest datasets.

Original: XR-Transformer takes inspiration from multi-resolution approaches in computer vision like superresolution [24] and progressive growing of GANs [19, 20], and enables multiple resolutions throughiterative training.
Revised: XR-Transformer takes inspiration from multi-resolution approaches in computer vision like superresolution [23] and progressive growing of GANs [18, 19], and enables multiple resolutions throughiterative training.

Original: However, unlike progressively grown GANs, which predict only at the highestresolution, XR-Transformer needs predictions across all resolutions for its progressive shortlistingpipeline, but uses representations trained at a single resolution.
Revised: However, unlike progressively grown GANs, which predict only at the highestresolution, XR-Transformer needs predictions across all resolutions for its progressive shortlistingpipeline, but uses representations trained at a single resolution.

Original: In practice, this leads to XRTransformer having a complex multi-stage pipeline where the transformer model is iteratively trainedup to a certain resolution and then frozen.
Revised: In practice, this leads to XRTransformer having a complex multi-stage pipeline where the transformer model is iteratively trainedup to a certain resolution and then frozen.

Original: This is followed by a re-clustering and re-training ofmultiple classifiers, working at different resolutions, with the same fixed transformer features [54,
Revised: This is followed by a re-clustering and re-training ofmultiple classifiers, working at different resolutions, with the same fixed transformer features [51,

Original: Alg. 2].
Revised: Alg. 2].

Original: For datasets with over 500K labels, the transformer is only trained on up to 2 15 clusters [54,
Revised: For datasets with over 500K labels, the transformer is only trained on up to 2 15 clusters [51,

Original: Table 6], and the resulting features are used for the extreme task.
Revised: Table 6], and the resulting features are used for the extreme task.

Original: Unlike AttentionXML, using multiple instances of transformer models becomes undesirable dueto their computational overhead.
Revised: Unlike AttentionXML, using multiple instances of transformer models becomes undesirable dueto their computational overhead.

Original: This pushes LightXML and XR-Transformer to make differenttrade-offs when leveraging a single transformer model for XMC tasks compared to AttentionXML,see Table 1.
Revised: This pushes LightXML and XR-Transformer to make differenttrade-offs when leveraging a single transformer model for XMC tasks compared to AttentionXML,see Table 1.

Original: In this paper, we present a method, CascadeXML , that combines the strengths of theseapproaches, thus creating an end to-end trainable multi-resolution learning pipeline which trainsa single transformer model across multiple resolutions in a way that allows the creation of label αααα
Revised: In this paper, we present a method, CascadeXML , that combines the strengths of theseapproaches, thus creating an end to-end trainable multi-resolution learning pipeline which trainsa single transformer model across multiple resolutions in a way that allows the creation of label αααα

Original: R (1)
Revised: R (1)

Original: R (2)
Revised: R (2)

Original: R (3)
Revised: R (3)

Original: Input Embeddings
Revised: Input Embeddings

Original: BERT
Revised: BERT

Original: Layers 1Layer
Revised: Layers 1Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: W W
Revised: W W

Original: W
Revised: W

Original: Wϑ (1) =
Revised: Wϑ (1) =

Original: Φ (6) CLS ( x ) ϑ (2) ϑ (3) ϑ (4)
Revised: Φ (6) CLS ( x ) ϑ (2) ϑ (3) ϑ (4)

Original: HLT
Revised: HLT

Original: C (1) ...
Revised: C (1) ...

Original: C (1) K
Revised: C (1) K

Original: C (2)
Revised: C (2)

Original: C (2) ... C (2) ...
Revised: C (2) ... C (2) ...

Original: C (2) K
Revised: C (2) K

Original: C (3) C (3)
Revised: C (3) C (3)

Original: C (3) ...
Revised: C (3) ...

Original: C (3) ... C (3) ...
Revised: C (3) ... C (3) ...

Original: C (3) K    ...
Revised: C (3) K    ...

Original: L − 3 L − 2 L −
Revised: L − 3 L − 2 L −

Original: L forward pass backward pass
Revised: L forward pass backward pass

Original: S (1) =
Revised: S (1) =

Original: { C (1)1 ,..., C (1) K 1 }
Revised: { C (1)1 ,..., C (1) K 1 }

Original: S (1) top S (2)
Revised: S (1) top S (2)

Original: S (2) top S (3)
Revised: S (2) top S (3)

Original: S (3) top S (4) scores
Revised: S (3) top S (4) scores

Original: Meta-labels:
Revised: Meta-labels:

Original: C (3)1 = { 1 , 2 }
Revised: C (3)1 = { 1 , 2 }

Original: C (2)1 = {{ 1 , 2 } , { 3 , 4 }}
Revised: C (2)1 = {{ 1 , 2 } , { 3 , 4 }}

Original: 1.2 CascadeXML in a Nutshell
Revised: 1.2 CascadeXML in a Nutshell

Original: The key insights that enable calculating resolution-specific representations with a single forward passthrough the transformer are a) that the shortlisting tasks get easier for lower label resolution andb) that the intermediate transformer representations are already quite powerful discriminators [46].
Revised: The key insights that enable calculating resolution-specific representations with a single forward passthrough the transformer are a) that the shortlisting tasks get easier for lower label resolution andb) that the intermediate transformer representations are already quite powerful discriminators [43].

Original: Consequently, CascadeXML extracts features for classification at the coarser meta-label resolutionfrom earlier transformer layers and uses the last layer only for classification at the extreme resolution.
Revised: Consequently, CascadeXML extracts features for classification at the coarser meta-label resolutionfrom earlier transformer layers and uses the last layer only for classification at the extreme resolution.

Original: The advantage of this is twofold: First, we postulate that the gain in classification accuracy due toenabling different attention maps at the different tree resolutions outweighs the loss in accuracy dueto the slightly less powerful representations at the earlier transformer layers.
Revised: The advantage of this is twofold: First, we postulate that the gain in classification accuracy due toenabling different attention maps at the different tree resolutions outweighs the loss in accuracy dueto the slightly less powerful representations at the earlier transformer layers.

Original: Second, this ensuresthat the representation capacity of the later layers is exclusively available to the more difficult tasks.
Revised: Second, this ensuresthat the representation capacity of the later layers is exclusively available to the more difficult tasks.

Original: The main idea of CascadeXML is illustrated in Figure 1.
Revised: The main idea of CascadeXML is illustrated in Figure 1.

Original: An input text is propagated through thetransformer model until the first shortlisting level is reached.
Revised: An input text is propagated through thetransformer model until the first shortlisting level is reached.

Original: There, the embedding ϑ (1) corresponding to the [CLS] token is extracted, and used to determine scores for each level-1 meta-label.
Revised: There, the embedding ϑ (1) corresponding to the [CLS] token is extracted, and used to determine scores for each level-1 meta-label.

Original: Thehighest-scoring meta-labels are selected to create a shortlist of candidate labels S (1) top .
Revised: Thehighest-scoring meta-labels are selected to create a shortlist of candidate labels S (1) top .

Original: The input ispropagated further to perform another shortlisting step at the next resolution, at which a new [CLS]embedding ϑ (2) is extracted.
Revised: The input ispropagated further to perform another shortlisting step at the next resolution, at which a new [CLS]embedding ϑ (2) is extracted.

Original: Crucially, ϑ (2) is the result of a different self-attention operations, andits attention map is a differently weighted combination of the embeddings of other tokens than thatof ϑ (1) .
Revised: Crucially, ϑ (2) is the result of a different self-attention operations, andits attention map is a differently weighted combination of the embeddings of other tokens than thatof ϑ (1) .

Original: The new embedding ϑ (2) is used to refine the shortlist by selecting the top-scoring level-2meta-labels that are children of the shortlisted level-1 meta-labels.
Revised: The new embedding ϑ (2) is used to refine the shortlist by selecting the top-scoring level-2meta-labels that are children of the shortlisted level-1 meta-labels.

Original: This process is repeated until atthe final layer of the transformer, where classification is performed at the full label resolution.
Revised: This process is repeated until atthe final layer of the transformer, where classification is performed at the full label resolution.

Original: Contributions In this paper, our contributions are the following:
Revised: Contributions In this paper, our contributions are the following:

Original: • We introduce a novel paradigm which can harness the multi-layered architecture of transformers for learning representations at multiple resolutions in an HLT.
Revised: • We introduce a novel paradigm which can harness the multi-layered architecture of transformers for learning representations at multiple resolutions in an HLT.

Original: This is in contrast tothe existing approaches which treat transformer as merely a black box encoder,• Implemented via CascadeXML, it is an end-to-end trainable multi-resolution learningpipeline which is quick to train and simple to implement.
Revised: This is in contrast tothe existing approaches which treat transformer as merely a black box encoder,• Implemented via CascadeXML, it is an end-to-end trainable multi-resolution learningpipeline which is quick to train and simple to implement.

Original: It eliminates the need of bells
Revised: It eliminates the need of bells

Original: and whistles like multi-stage procedures, reclustering, and bootstrap training to achievestate-of-the-art performance, and• CascadeXML not only scales to dataset with millions of labels but also improves thecurrent state-of-the-art in terms of prediction performance by upto ∼ 6% on benchmarkdatasets.
Revised: and whistles like multi-stage procedures, reclustering, and bootstrap training to achievestate-of-the-art performance, and • CascadeXML not only scales to dataset with millions of labels but also improves thecurrent state-of-the-art in terms of prediction performance by upto ∼ 6% on benchmarkdatasets.

Original: In addition to this, it is computationally most efficient to train and clocks multi-foldimprovement in inference time as compared to existing transformer-based approaches.
Revised: In addition to this, it is computationally most efficient to train and clocks multi-foldimprovement in inference time as compared to existing transformer-based approaches.

Original:  Notation & Preliminaries
Revised:  Notation & Preliminaries

Original: Problem setup:
Revised: Problem setup:

Original: In extreme classification, an instance (document or query) x ∈ X from some inputspace X is mapped to a subset of labels y ⊂
Revised: In extreme classification, an instance (document or query) x ∈ X from some inputspace X is mapped to a subset of labels y ⊂

Original: [ L ] out of the L available labels identified through theintegers [ L ] := { 1 , . . .
Revised: [ L ] out of the L available labels identified through theintegers [ L ] := { 1 , . . .

Original: , L } , such that L ∼ 10 6 .
Revised: , L } , such that L ∼ 10 6 .

Original: Usually, we identify the labels through a binary vectory ∈ { 0 , 1 } L , where y j = 1 ⇔ j ∈ y .
Revised: Usually, we identify the labels through a binary vectory ∈ { 0 , 1 } L , where y j = 1 ⇔ j ∈ y .

Original: The instances and labels are jointly distributed accordingto an unknown distribution ( x , y ) ∼ P of which we have a i.i.d.
Revised: The instances and labels are jointly distributed accordingto an unknown distribution ( x , y ) ∼ P of which we have a i.i.d.

Original: sample D = (cid:8)(cid:0) x i , y i (cid:1) : i ∈ [ n ] (cid:9)available as a training set.
Revised: sample D = (cid:8)(cid:0) x i , y i (cid:1) : i ∈ [ n ] (cid:9)available as a training set.

Original: Even though the number of labels can be of the order of millions, the labelvectors are sparse, i.e., E [ ∥ y ∥ 1 ] ≪ L .
Revised: Even though the number of labels can be of the order of millions, the labelvectors are sparse, i.e., E [ ∥ y ∥ 1 ] ≪ L .

Original: For text data, an instance is a sequence of tokens x =
Revised: For text data, an instance is a sequence of tokens x =

Original: ( t k ) lk =1 ,from a vocabularyV ∋ t k  .
Revised: ( t k ) lk =1 ,from a vocabulary t k ∈ V .

Original: As part of the model pipeline, a text vectorizer Φ : X (cid:55)→ R d projects the documents to a d -dimensionalembedding space.
Revised: As part of the model pipeline, a text vectorizer Φ : X (cid:55)→ R d projects the documents to a d -dimensionalembedding space.

Original: These vectorizers can be simple bag-of-words (BOW) or TF-IDF transformationsΦ tf-idf , or a learnable model Φ dnn that encodes the documents to the embedding space using weightslearnt over the dataset.
Revised: These vectorizers can be simple bag-of-words (BOW) or TF-IDF transformationsΦ tf-idf , or a learnable model Φ dnn that encodes the documents to the embedding space using weightslearnt over the dataset.

Original: The final classification of an instance x is achieved by computing per labelscores ⟨ w l , Φ( x ) ⟩ using label-wise weight vectors w l ∈ R
Revised: The final classification of an instance x is achieved by computing per labelscores ⟨ w l , Φ( x ) ⟩ using label-wise weight vectors w l ∈ R

Original: d
Revised: d

Original: which are either (i) jointly learnt with Φin a deep architecture [8, 10, 18, 32, 49, 52], or (ii) solely learnt in shallow classifiers [2, 53].
Revised: which are either (i) jointly learnt with Φin a deep architecture [8, 10, 17, 31, 46, 49], or (ii) solely learnt in shallow classifiers [2, 50].

Original: Transformer Encoder: More recent approaches in extreme classification have achieved stateof-the-art performance using pre-trained transformers as text vectorizers [18, 49, 54].
Revised: Transformer Encoder: More recent approaches in extreme classification have achieved stateof-the-art performance using pre-trained transformers as text vectorizers [17, 46, 51].

Original: Populartransformer encoders for NLP [11, 28, 48] consist of an embedding layer, followed by a series ofstacked self-attention layers and task-specific heads that can be fine-tuned along with the encoder.
Revised: Populartransformer encoders for NLP [11, 27, 45] consist of an embedding layer, followed by a series ofstacked self-attention layers and task-specific heads that can be fine-tuned along with the encoder.

Original: For a sequence of m input words x ∈ V m , each layer a in these models produces a sequence ofembeddings (cid:16) Φ ( a ) 0 ( x ) , . .
Revised: For a sequence of m input words x ∈ V m , each layer a in these models produces a sequence ofembeddings (cid:16) Φ ( a ) 0 ( x ) , . .

Original: Φ ( a ) m ( x ) (cid:17) , where Φ ( a ) i ( x ) ∈ R
Revised: Φ ( a ) m ( x ) (cid:17) , where Φ ( a ) i ( x ) ∈ R

Original: d is a representation for the i ’th token inthe input sequence at the a ’th layer, and Φ ( a ) CLS ( x ) :=
Revised: d is a representation for the i ’th token inthe input sequence at the a ’th layer, and Φ ( a ) CLS ( x ) :=

Original: Φ ( a ) 

Revised: Φ ( a )

Original: ( x ) is an embedding corresponding to aspecial [CLS] token that has been pre-trained to capture a representation of the entire input [11, 28].
Revised: ( x ) is an embedding corresponding to aspecial [CLS] token that has been pre-trained to capture a representation of the entire input [11, 27].

Original: Label Shortlisting: Computing scores ⟨ w l , Φ( x ) ⟩ for each label l ∈
Revised: Label Shortlisting: Computing scores ⟨ w l , Φ( x ) ⟩ for each label l ∈

Original: [ L ] in an extremely largeoutput space is computationally very demanding, especially when the feature space R d ∋ Φ( x )is high dimensional to allow for expressive features.
Revised: [ L ] in an extremely largeoutput space is computationally very demanding, especially when the feature space R d ∋ Φ( x )is high dimensional to allow for expressive features.

Original: To alleviate this problem, the labels L aregrouped together under K meta-labels R =
Revised: To alleviate this problem, the labels L aregrouped together under K meta-labels R =

Original: { C 1 , . .
Revised: { C 1 , . .

Original: , C K } , with K ≈ O ( √ L ) .
Revised: , C K } , with K ≈ O ( √ L ) .

Original: Each meta-label isrepresented as the set of its child labels, C k = { l ∈ [ L ] : l is child of k } and each target set y ⊂
Revised: Each meta-label isrepresented as the set of its child labels, C k = { l ∈ [ L ] : l is child of k } and each target set y ⊂

Original: [ L ]corresponds to a meta-target ˜
Revised: [ L ]corresponds to a meta-target ˜

Original: y
Revised: y

Original: = R ( y ) =
Revised: = R ( y ) =

Original: { C k | ∃ i ∈
Revised: { C k | ∃ i ∈

Original: y : i ∈ C k } .
Revised: y : i ∈ C k } .

Original: A meta-label C l is relevant to adata point if it contains at least one relevant label.
Revised: A meta-label C l is relevant to adata point if it contains at least one relevant label.

Original: The label vectors are so sparse that even the much coarser meta-labels still contain mostly zeros,| ˜
Revised: The label vectors are so sparse that even the much coarser meta-labels still contain mostly zeros,| ˜

Original: y
Revised: y

Original: | ≪ K .
Revised: | ≪ K .

Original: This means that it is enough to explicitly calculate the scores only for the descendants ofrelevant meta-labels.
Revised: This means that it is enough to explicitly calculate the scores only for the descendants ofrelevant meta-labels.

Original: In practice, an algorithm will first predict scores for each meta-labels and selectthe top-scoring ones as a shortlist for further classification.
Revised: In practice, an algorithm will first predict scores for each meta-labels and selectthe top-scoring ones as a shortlist for further classification.

Original: The meta- and extreme-classificationtasks can either be learnt sequentially [8, 10] or jointly through dynamic label-shortlisting [18, 22].
Revised: The meta- and extreme-classificationtasks can either be learnt sequentially [8, 10] or jointly through dynamic label-shortlisting [17, 21].

Original: Ideally, the labels grouped under one meta-label should be semantically similar to ensure the commonrepresentations learnt for meta- and extreme- tasks are relevant to both tasks [10].
Revised: Ideally, the labels grouped under one meta-label should be semantically similar to ensure the commonrepresentations learnt for meta- and extreme- tasks are relevant to both tasks [10].

Original: This has the addedadvantage that the negative labels selected by the shortlisting procedure will be the most confusingones, leading to hard negative sampling and improved learning of the extreme classifier [39].
Revised: This has the addedadvantage that the negative labels selected by the shortlisting procedure will be the most confusingones, leading to hard negative sampling and improved learning of the extreme classifier [37].

Original: For extremely large label spaces, a single cluster size K is not optimal [54].
Revised: For extremely large label spaces, a single cluster size K is not optimal [51].

Original: It is either large enoughthat the meta-task itself becomes very expensive, or its individual clusters are so large that the shortlist contains too many candidate labels [54].
Revised: It is either large enoughthat the meta-task itself becomes very expensive, or its individual clusters are so large that the shortlist contains too many candidate labels [51].

Original: Thus, methods like AttentionXML and XR-Transformer usea multi-level shortlisting procedure based on a Hierarchical Label Tree (HLT) .
Revised: Thus, methods like AttentionXML and XR-Transformer usea multi-level shortlisting procedure based on a Hierarchical Label Tree (HLT) .

Original: Label Trees: An HLT H = (cid:8) R (1) , . . .
Revised: Label Trees: An HLT H = (cid:8) R (1) , . . .

Original: , R ( T ) (cid:9) is a hierarchical clustering of labels into T levels ofsuccessively refined clusters R ( t ) =
Revised: , R ( T ) (cid:9) is a hierarchical clustering of labels into T levels ofsuccessively refined clusters R ( t ) =

Original: (cid:110) C ( t ) 1 , . .
Revised: (cid:110) C ( t ) 1 , . .

Original: , C ( t ) K t (cid:111) .
Revised: , C ( t ) K t (cid:111) .

Original: The clusters C ( t ) k of each level form a disjointpartitioning of the clusters of the next, C ( t ) k
Revised: The clusters C ( t ) k of each level form a disjointpartitioning of the clusters of the next, C ( t ) k

Original: ⊂ R ( t +1) , with the convention that the T + 1 th cluster isthe full label space R ( T +1) =
Revised: ⊂ R ( t +1) , with the convention that the T + 1 th cluster isthe full label space R ( T +1) =

Original: [ L ] .
Revised: [ L ] .

Original: As such, each cluster C ( T ) k (or meta-label) at level T is representedby a set of labels, each cluster at level T − 1 is a set of level- T meta-labels, and thus a set of setsof labels, and so on.
Revised: As such, each cluster C ( T ) k (or meta-label) at level T is representedby a set of labels, each cluster at level T − 1 is a set of level- T meta-labels, and thus a set of setsof labels, and so on.

Original: In practice, label representations z l as an aggregate of their positive instancefeatures are leveraged to create an HLT.
Revised: In practice, label representations z l as an aggregate of their positive instancefeatures are leveraged to create an HLT.

Original: Typically, z l is constructed using Φ tf-idf asz l := v l ∥ v l ∥ , where v l := (cid:88)
Revised: Typically, z l is constructed using Φ tf-idf asz l := v l ∥ v l ∥ , where v l := (cid:88)

Original: i : y ( i ) l = Φ tf-idf ( x ( i ) ) .
Revised: i : y ( i ) l = Φ tf-idf ( x ( i ) ) .

Original: The HLT is often constructed by recursively partitioning the label-space using balanced k-meansclustering in a top-down fashion [8, 18, 52, 54].
Revised: The HLT is often constructed by recursively partitioning the label-space using balanced k-meansclustering in a top-down fashion [8, 17, 49, 51].

Original: For example, an HLT constructed using balanced2-means clustering has ⌊ log 2 L ⌋ levels, out of which we select T levels corresponding to differentmeta-label resolutions.
Revised: For example, an HLT constructed using balanced2-means clustering has ⌊ log 2 L ⌋ levels, out of which we select T levels corresponding to differentmeta-label resolutions.

Original:  Method: CascadeXML
Revised:  Method: CascadeXML

Original: At its core, the CascadeXML model consists of three components: A pre-trained language model Φwhich allows to extract a document representation Φ ( a ) CLS ( x ) ∈ R d at different layers a ∈ [ A ] , an HLT
Revised: At its core, the CascadeXML model consists of three components: A pre-trained language model Φwhich allows to extract a document representation Φ ( a ) CLS ( x ) ∈ R d at different layers a ∈ [ A ] , an HLT

Original: H = (cid:8) R (1) , . .
Revised: H = (cid:8) R (1) , . .

Original: , R ( T ) (cid:9) that divides the label space into K 1 < . . .
Revised: , R ( T ) (cid:9) that divides the label space into K 1 < . . .

Original: <
Revised: <

Original: K T <
Revised: K T <

Original: L = K T +1 successivelyrefined clusters, and a set of linear classifiers (cid:110) W ( t ) =
Revised: L = K T +1 successivelyrefined clusters, and a set of linear classifiers (cid:110) W ( t ) =

Original: [ w ( t ) 1 , . .
Revised: [ w ( t ) 1 , . .

Original: . , w ( t ) K t ] ∈ R
Revised: . , w ( t ) K t ] ∈ R

Original: d
Revised: d

Original: × K t : t ∈ [ T + 1] (cid:111) .
Revised: × K t : t ∈ [ T + 1] (cid:111) .

Original: We select one layer for each meta-classifier to extract resolution-specific feature embeddingsϑ ( t ) :
Revised: We select one layer for each meta-classifier to extract resolution-specific feature embeddingsϑ ( t ) :

Original: =
Revised: =

Original: Φ ( a t ) CLS ( x ) with 1 < a 1 < . . .
Revised: Φ ( a t ) CLS ( x ) with 1 < a 1 < . . .

Original: <
Revised: <

Original: a T <
Revised: a T <

Original: A = a T +1 .
Revised: A = a T +1 .

Original: Refining and Coarsening Label Vectors:
Revised: Refining and Coarsening Label Vectors:

Original: In order to transfer the representation of a shortlist ofa label vector between different layers of the HTL, we need to define two operations.
Revised: In order to transfer the representation of a shortlist ofa label vector between different layers of the HTL, we need to define two operations.

Original: The first isrefining the resolution of a shortlist: Given a shortlist S = { C ( t ) k 1 , . . .
Revised: The first isrefining the resolution of a shortlist: Given a shortlist S = { C ( t ) k 1 , . . .

Original: , C ( t ) k s } ⊂ R ( t ) of clusters of theresolution at level t , we want to calculate the representation of this shortlist in the next layer, denoted
Revised: , C ( t ) k s } ⊂ R ( t ) of clusters of theresolution at level t , we want to calculate the representation of this shortlist in the next layer, denoted

Original: R ( t ) ↓ ( S ) .
Revised: R ( t ) ↓ ( S ) .

Original: Since each cluster is defined as the set of its descendants, this is achieved by taking theunion over the clusters in the list.
Revised: Since each cluster is defined as the set of its descendants, this is achieved by taking theunion over the clusters in the list.

Original: The second operation is to find the cover of meta-labels that covera given set of labels, R ( t ) ↑ ( S ) .
Revised: The second operation is to find the cover of meta-labels that covera given set of labels, R ( t ) ↑ ( S ) .

Original: This is achieved by identifying each of the clusters in the layer abovefor which there exists at least one descendant in the shortlist.
Revised: This is achieved by identifying each of the clusters in the layer abovefor which there exists at least one descendant in the shortlist.

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Revised: We define R ( t ) ( y ) = (cid:16) R ( t +1) ↑ ◦ . . .

Revised: ◦ R ( T ) ↑ (cid:17) ( y ) to be the level- t cover of the label vector y ∈

Original:  Multi-resolution Dynamic Label Shortlisting: Concisely, we adapt dynamic label shortlisting [18]for multi-resolution training approaches.
Revised: [ L ] . Multi-resolution Dynamic Label Shortlisting: Concisely, we adapt dynamic label shortlisting [17]for multi-resolution training approaches.

Original: Formally, given a shortlist of s clusters { C ( t ) k 1 , . . .
Revised: Formally, given a shortlist of s clusters { C ( t ) k 1 , . . .

Original: , C ( t ) k s } =:
Revised: , C ( t ) k s } =:

Original: S ( t ) ⊂
Revised: S ( t ) ⊂

Original: R ( t ) at level t , and the corresponding classification features ϑ ( t ) , a new shortlist S ( t +1) ⊂
Revised: R ( t ) at level t , and the corresponding classification features ϑ ( t ) , a new shortlist S ( t +1) ⊂

Original: R ( t +1) is generated as follows:
Revised: R ( t +1) is generated as follows:

Original: First, the scores for all shortlisted meta-labels are calculated through⟨ w ( t ) k i , ϑ ( t ) ⟩ and the k highest-scoring meta-labels are selected into
Revised: First, the k highest-scoring meta-labels are selected into S ( t ) top and thenext shortlist S ( t +1) is then given as the refinement:

Original: [Equation]
Revised: [Equation]

Original: The next shortlist is then given as the refinement S ( t +1) =

Original: R ( t ) ↓ ( S ( t ) top ) .

Original: At the first level, however,the shortlist is initialized to consider all meta labels S (1) =
Revised: At the first level, however, the shortlist is initialized to consider all meta labels S (1) =

Original: R (1) .
Revised: R (1) .

Original: It is of essence that all truepositive (meta-)labels appear at every level during training time [8, 18, 52] and is difficult to ensureintrinsically due to imperfect recall rate of classifier at resolution t .
Revised: It is ofessence that all true positive (meta-)labels appear at every level during training time [8, 17, 49] and isdifficult to ensure intrinsically due to imperfect recall rate of classifier at resolution t .

Original: Hence, duringtraining allmeta-labels corresponding to ground-truth labels are added to the candidate set, leading to [Equation]
Revised: Hence, during

Original: Input: instance x , labels y , clusters C , features Φ , classifiers W , weights α ϑ ( t ) ← Φ ( t ) CLS ( x ) ∀ t ∈ [ T + 1] /* Forward pass through Transformer */ loss ←
Revised: Input: instance x , labels y , clusters C , features Φ , classifiers W , weights α ϑ ( t ) ← Φ ( t ) CLS ( x ) ∀ t ∈ [ T + 1] /* Forward pass through Transformer */ loss ←

Original: S (1) ← C (1) /* Initialize shortlist */ for t in 1 , 2 , ..., T do y ( t ) ← C ( t ) ( y ) /
Revised: S (1) ← C (1) /* Initialize shortlist */ for t in 1 , 2 , ..., T do y ( t ) ← R ( t ) ( y ) /

Original: * Ground-truth meta-labels */ p i ← ⟨ w ( t ) k
Revised: * Ground-truth meta-labels */ p i ← ⟨ w ( t ) k

Original: i , ϑ ( t ) ⟩ ∀ i ∈ S ( t ) /
Revised: i , ϑ ( t ) ⟩ ∀ i ∈ S ( t ) /

Original: * Sparse prediction */ loss ← loss + α t |S ( t ) | (cid:80) l ∈S ( t ) L ( t ) BC (cid:16) p l , y ( t ) l (cid:17)
Revised: * Sparse prediction */ loss ← loss + α t |S ( t ) | (cid:80) l ∈S ( t ) L ( t ) BC (cid:16) p l , y ( t ) l (cid:17)

Original: S ( t ) top ← (cid:110) C ( t ) k j : j ∈ Top k (cid:0)(cid:8) p i | i ∈ S ( t ) (cid:9)(cid:1)(cid:111) .
Revised: S ( t ) top ← (cid:110) C ( t ) k j : j ∈ Top k (cid:0)(cid:8) p i | i ∈ S ( t ) (cid:9)(cid:1)(cid:111) .

Original: /* Top-K selection */
Revised: /* Top-K selection */

Original: S ( t +1) ← S ( t ) top ∪ R ( t +1)
Revised: S ( t + [Equation] [Equation] [Equation]

Original: ( y ) /
Revised: t ) [Equation] R [Equation]

Original:  * Add ground-truth positives */ p i ← ⟨ w ( T +1)
Revised: t +1) ( y ) / * Add ground-truth positives */ p i ← ⟨ w ( T +1)

Original: k
Revised: k

Original: i , ϑ ( T +1) ⟩ ∀ i ∈ S
Revised: i , ϑ ( T +1) ⟩ ∀ i ∈ S

Original: ( T +1) /
Revised: ( T +1) /

Original: * Final prediction */ loss ← loss + α T +1 |S
Revised: * Final prediction */ loss ← loss + α T +1 |S

Original: ( T +1) 
Revised: ( T +1) | (cid:80) l [Equation]

Original: | (cid:80) l ∈S L ( T +1) BC ( p l , y l ) adjust Φ and W to reduce loss 
Revised: L ( T +1) BC ( p l , y l ) adjust Φ and W to reduce loss training all meta-labels corresponding to ground-truth labels are added to the candidate set, leading to [Equation]

Original: Consequently, a single training step in CascadeXML looks as given in algorithm 1.
Revised: Consequently, a single training step in CascadeXML looks as given in algorithm 1.

Original: Training Objective: At each level of the tree, the goal is to correctly identify the most likelymeta-labels, which can be achieved through minimization of a one-vs-all loss
Revised: Training Objective: At each level of the tree, the goal is to correctly identify the most likelymeta-labels, which can be achieved through minimization of a one-vs-all loss

Original: L ( t ) ( x , y ) =

Original: |S ( t ) | (cid:88) l ∈S ( t )

Original: L ( t ) BC (cid:16) ⟨ w ( t ) k l , ϑ ( t ) ( x

Original: [Equation]
Revised: [Equation]

Original: The overall objective is a weighted sum of the individual layer losses
Revised: The overall objective is a weighted sum of the individual layer losses

Original: [Equation]
Revised: [Equation]

Original: where level T + 1 corresponds to the extreme classification task.
Revised: where level T + 1 corresponds to the extreme classification task.

Original: To balance out different loss scalesas a result of normalization by varied shortlist sizes, we re-scale losses from multiple resolutionsusing a  factor α ( t ) =
Revised: To balance out different loss scalesas a result of normalization by varied shortlist sizes, we re-scale losses from multiple resolutionsusing a simple re-scaling factor α ( t ) =

Original: |S ( t ) | / min t ∈
Revised: |S ( t ) | / min t ∈

Original: [ T +1] ( |S ( t ) | ) .
Revised: [ T +1] ( |S ( t ) | ) .

Original: Inference: The algorithm for inference remains the same as training, except we do not teacher-forcetrue (meta-)labels R ( t ) ( y ) during inference time (Eqn. 3).
Revised: Inference: The algorithm for inference remains the same as training, except we do not teacher-forcetrue (meta-)labels R ( t ) ( y ) during inference time (Eqn. 3).

Original: The inference consists of one forward passthrough the transformer backbone, followed by successive shortlisting at all T resolutions, and finalextreme classification.
Revised: The inference consists of one forward passthrough the transformer backbone, followed by successive shortlisting at all T resolutions, and finalextreme classification.

Original: For a b -way branching tree with log b L levels, each level t has b t meta-labels,of which |S ( t ) | ∝ √ b t are selected for the shortlist.
Revised: For a b -way branching tree with log b L levels, each level t has b t meta-labels,of which |S ( t ) | ∝ √ b t are selected for the shortlist.

Original: The time complexity is
Revised: The time complexity is

Original: [Equation]
Revised: [Equation]

Original: where T dnn denotes the time taken to compute transformer embeddings for an input.
Revised: where T dnn denotes the time taken to compute transformer embeddings for an input.

Original:  Experiments
Revised:  Experiments

Original: Benchmark datasets & Evaluation Metrics
Revised: Benchmark datasets & Evaluation Metrics

Original: We benchmark the performance of CascadeXML onstandard datasets and evaluation metrics.
Revised: We benchmark the performance of CascadeXML onstandard datasets and evaluation metrics.

Original: In all, we use 5 public XMC datasets [4, 30], the statisticsof which are specified in appendix A.1.
Revised: In all, we use 5 public XMC datasets [4, 29], the statisticsof which are specified in appendix A.1.

Original: However, due to space considerations and as our main focusis on extremely large output spaces, we present results for the largest of the publicly available datasets
Revised: However, due to space considerations and as our main focusis on extremely large output spaces, we present results for the largest of the publicly available datasets

Original: – Wiki-500K, Amazon-670K, and Amazon-3M , in the main paper.
Revised: – Wiki-500K, Amazon-670K, and Amazon-3M, in the main paper.

Original: For all our experiments we use thesame raw text input, train-test split and sparse feature representations as used in [52, 18, 54].
Revised: For all our experiments we use thesame raw text input, train-test split and sparse feature representations as used in [49, 17, 51].

Original: We use
Revised: We use

Original: Precision@k (P@k) and Propensity-scored P@k (PSP@k) as the evaluation metrics.
Revised: Precision@k (P@k) and Propensity-scored P@k (PSP@k) as the evaluation metrics.

Original: PSP@k focusesmore on the model’s performance on tail labels and the results are given in appendix A.3.3.
Revised: PSP@k focusesmore on the model’s performance on tail labels and the results are given in appendix A.3.3.

Original: Baseline Methods: We compare CascadeXML with strongXMC baselines: AnnexML [43], DiSMEC[2], Parabel [36], eXtremeText [45], Bonsai [21], XMLCNN [26], XR-Linear [53], AttentionXML,
Revised: Baseline Methods: We compare CascadeXML with strong baselines: AnnexML [40], DiSMEC[2], Parabel [34], eXtremeText [42], Bonsai [20], XMLCNN [25], XR-Linear [50], AttentionXML,

Original: X-Transformer [8], LightXML and XR-Transformer.
Revised: X-Transformer [8], LightXML and XR-Transformer.

Original: Results on Wiki-500K, Amazon-670K and
Revised: Results on Wiki-500K, Amazon-670K and

Original: Amazon-3M are shown in Table 2.
Revised: Amazon-3M are shown in Table 2.

Original: The baseline results for other methods were obtained from [54,
Revised: The baseline results for other methods were obtained from [51,

Original: Table: 2].
Revised: Table: 2].

Original: It may be noted that, in order to incorporate the global statistical information of the dataset,
Revised: It may be noted that, in order to incorporate the global statistical information of the dataset,

Original: XR-Transformer concatenatesthe sparse tf-idf features to those learnt from deep neural networks(DNN, denoted Φ dnn )
Revised: XR-Transformer concatenates sparse tf-idf features to those learnt via deep networks (denoted Φ dnn )

Original: [Equation]
Revised: [Equation]

Original: For fair evaluation, we present separate comparison with methods which use (i) features from a DNNencoder only, and (ii) those obtained by combining DNN and tf-idf features.
Revised: For fair evaluation, we present separate comparison with methods which use (i) features from a DNNencoder only, and (ii) those obtained by combining DNN and tf-idf features.

Original: Metrics
Revised: [Table]

Original: P@ P@
Revised: P@ P

Revised: [Table]

Original: P@ P@
Revised: P@ P@

Original: P@ P@ 
Revised: P@ P@ P@

Original: P@
Revised: P@

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Datasets

Original: Wiki

Original: [Table]
Revised: [Table]

Original: AnnexML 64.

Original: DiSMEC 70.

Original: Parabel ∗ 68.
Revised: Parabel ∗ 68.

Original: eXtremeText 65.
Revised: eXtremeText 65.

Original: Bonsai ∗ 69.
Revised: Bonsai ∗ 69.

Original: XR-Linear 65.
Revised: XR-Linear 65.

Original: DNN Features
Revised: DNN Features

Original: [Table]

Original: -CNN - - - 33.
Revised: XML-CNN - - - 33.

Original: AttentionXML [Table]
Revised: AttentionXML ∗ 76.

Original: LightXML [Table]
Revised: LightXML ∗ 77.

Original: [Table]
Revised: CascadeXML ∗ 78. [DNN (cid:76) tf-idf] Features X

Original: [Table]
Revised: [Table]

Original: Features
Revised: XR-

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: ∗ 79. CascadeXML ∗ 81. respective feature settings are in bold . * implies ensemble result of 3 models, ** implies ensemble result of 9 models and ‘-’ implies that the model does not scale for that dataset due to GPU memory constraints.

Original: [Table]
Revised: Details about model ensembles can be found in the appendix.

Original: Empirical Performance:
Revised: Empirical Performance:

Original: As demonstrated in Table 2, CascadeXML, which leverages the multilayered transformer architecture to learn label resolution specific feature representations, is empiricallysuperior to all existing XMC approaches whether using: (i) tf-idf features, or (ii) DNN-based densefeatures (iii) or concatenated tf-idf and dense features.
Revised: As demonstrated in Table 2, CascadeXML, which leverages the multilayered transformer architecture to learn label resolution specific feature representations, is empiricallysuperior to all existing XMC approaches whether using: (i) tf-idf features, or (ii) DNN-based densefeatures (iii) or concatenated tf-idf and dense features.

Original: This is despite CascadeXML being a simpleend-to-end trainable algorithm without using any bells or whistles like multi-stage procedures[52, 54], reclustering [54] or bootstrapped training [52, 54].
Revised: This is despite CascadeXML being a simpleend-to-end trainable algorithm without using any bells or whistles like multi-stage procedures[49, 51], reclustering [51] or bootstrapped training [49, 51].

Original: For example, on Amazon-3M dataset,
Revised: For example, on Amazon-3M dataset,

Original: CascadeXML significantly outperforms all DNN-based approaches while only taking 24 hours totrain on a single GPU.
Revised: CascadeXML significantly outperforms all DNN-based approaches while only taking 24 hours totrain on a single GPU.

Original: In contrast, X-Transformer needs to ensemble 9 models and 23 days to trainon 8 GPUs to reach the empirical performance (c.f. page 2, [54]), and LightXML remains unscalable.
Revised: In contrast, X-Transformer needs to ensemble 9 models and 23 days to trainon 8 GPUs to reach the empirical performance (c.f. page 2, [51]), and LightXML remains unscalable.

Original: On the concatenated feature space, CascadeXML also significantly outperforms XR-Transformeron 8 out of 9 dataset-metric combination with notable ∼ 6% improvement on Wiki-500K datasetfor P@5 metric.
Revised: On the concatenated feature space, CascadeXML also significantly outperforms XR-Transformeron 8 out of 9 dataset-metric combination with notable ∼ 6% improvement on Wiki-500K dataset for P@5 metric.

Original: As XR-Tranformer learns transformer features for label resolutions only upto 32K( 2 15 ) clusters [54, Table 6], incorporating tf-idf features via a computation-intensive XR-Linear[53] for final extreme classification task becomes an integral part of their pipeline.
Revised: As XR-Tranformer learns transformer features for label resolutions only upto 32K( 2 15 ) clusters [51, Table 6], incorporating tf-idf features via a computation-intensive XR-Linear[50] for final extreme classification task becomes an integral part of their pipeline.

Original: On the otherhand, CascadeXML does not have any such inherent limitation, and even without tf-idf features, itoutperforms XR-Transformer on 6 out of 9 dataset-metric combinations.
Revised: On the otherhand, CascadeXML does not have any such inherent limitation, and even without tf-idf features, itoutperforms XR-Transformer on 6 out of 9 dataset-metric combinations.

Original: Inference and Training Time:
Revised: Inference and Training Time:

Original: The comparison on inference times is shown in Table 3.
Revised: The comparison on inference times is shown in Table 3.

Original: CascadeXMLclocks the fastest inference speed as compared to all previous DNN-based XMC models.
Revised: CascadeXMLclocks the fastest inference speed as compared to all previous DNN-based XMC models.

Original: Notably, itis ∼ 1 .
Revised: Notably, itis ∼ 1 .

Original: 5 × and ∼ 2 × faster as compared to LightXML and XR-Transformer respectively.
Revised: 5 × and ∼ 2 × faster as compared to LightXML and XR-Transformer respectively.

Original: In terms oftraining time, CascadeXML is upto an order of magnitude faster than LightXML on a single GPU.
Revised: In terms oftraining time, CascadeXML is upto an order of magnitude faster than LightXML on a single GPU.

Original: When using multiple GPUs, its wall clock time on 4 GPUs is lower than both XR-Transformer and
Revised: When using multiple GPUs, its wall clock time on 4 GPUs is lower than both XR-Transformer and

Original: Attention on 8 GPUs.
Revised: AttentionXML on 8 GPUs.

Original: Exact training times can be found in Section A.3.2 in the appendix.
Revised: Exact training times can be found in Section A.3.2 in the appendix.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Impact of label clusters size:
Revised: Impact of label clusters size:

Original: Earlier works have argued that using finegrained clusters leads to better model performance [18, 22, 32].
Revised: Earlier works have argued that using finegrained clusters leads to better model performance [17, 21, 31].

Original: With increasednumber of label clusters at the penultimate level of HLT, the multi-resolutiontasks tend to get more in-sync with eachother [22] and approaches leveragingtransfer learning across different stages[10, 32] of their training pipeline getweights which are more suitable for theirextreme classification task.
Revised: With increasednumber of label clusters at the penultimate level of HLT, the multi-resolutiontasks tend to get more in-sync witheach other [21] and hence enables transfer learning across different stages oflearning pipelines [10, 31].

Original: Our mutliresolution training enables CascadeXMLto use 2 16 label clusters for penultimate label resolution, double that of XRTransformer.
Revised: Our mutliresolution training enables CascadeXMLto use 2 16 label clusters for penultimate label resolution, double that of XRTransformer.

Original: As Table 4 shows, increasing the resolution at the last shortlistingstep still results in non-trivial improvements, highlighting the need for architectures that can efficientlyhandle high-resolution meta-labels.
Revised: As Table 4 shows, increasing the resolution at the last shortlistingstep still results in non-trivial improvements, highlighting the need for architectures that can efficiently handle high-resolution meta-labels.

Original: Impact of re-scaling per-resolution loss: For extreme datasets, we witness more efficient trainingby having a larger shortlist size for finer label resolutions (see appendix A.2.2).
Revised: Impact of re-scaling per-resolution loss: For extreme datasets, we witness more efficient trainingby having a larger shortlist size for finer label resolutions (see appendix A.2.2).

Original: This results in variedloss scales across label resolutions leading to one resolution dominating the training.
Revised: This results in variedloss scales across label resolutions leading to one resolution dominating the training.

Original: Comparingthe “Default” and “w/o Rescaling” rows for respective datasets in Table 4, indicates that rescaling asintroduced in section 3 helps CascadeXML train more effectively.
Revised: Comparingthe “Default” and “w/o Rescaling” rows for respective datasets in Table 4, indicates that rescaling asintroduced in section 3 helps CascadeXML train more effectively.

Original:  Discussion
Revised:  Discussion

Original: As shown above, CascadeXML substantially outperforms current state-of-the-art approaches bothin terms of computational efficiency, and prediction performance.
Revised: As shown above, CascadeXML substantially outperforms current state-of-the-art approaches bothin terms of computational efficiency, and prediction performance.

Original: It achieves this while beingsimpler than XR-Transformer and more scalable than LightXML.
Revised: It achieves this while beingsimpler than XR-Transformer and more scalable than LightXML.

Original: Revisiting Table 1, the extractionof intermediate [CLS] representations enables us to perform end-to-end multi-resolution trainingwith resolution-specific attention maps from a single transformer model.
Revised: Revisiting Table 1, the extractionof intermediate [CLS] representations enables us to perform end-to-end multi-resolution trainingwith resolution-specific attention maps from a single transformer model.

Original: From Table 2, we showimprovements on both LightXML and XR-Transformer, which are transformer-based models sharingfeature representations across multiple label resolutions.
Revised: From Table 2, we show improvements on both LightXML and XR-Transformer, which are transformer-based models sharingfeature representations across multiple label resolutions.

Original: We attribute this increased performanceto CascadeXML’s end-to-end multi-resolution training pipeline and property of adapted feature embeddings.
Revised: We attribute this increased performanceto CascadeXML’s end-to-end multi-resolution training pipeline and property of adapted featureembeddings.

Original: The presented architecture, therefore, combines the strength of different previousapproaches, without inheriting their limitations.
Revised: The presented architecture, therefore, combines the strength of different previousapproaches, without inheriting their limitations.

Original: We postulated that the (meta-)classification tasks at different resolutions need to attend to differenttokens in the input sequence, a property lacking in recent approaches [8, 18, 54].
Revised: We postulated that the (meta-)classification tasks at different resolutions need to attend to differenttokens in the input sequence, a property lacking in recent approaches [8, 17, 51].

Original: This intuition iscorroborated by the fact that the attention maps learned by CascadeXML differ significantly betweendifferent resolutions.
Revised: This intuition iscorroborated by the fact that the attention maps learned by CascadeXML differ significantly betweendifferent resolutions.

Original: We provide graphs and supporting data in the appendix.
Revised: We provide graphs and supporting data in the appendix.

Original: Obtaining label resolution specific feature embeddings in a pre-trained transformer model requires tomake the compromise of extracting these at earlier transformer layers, where the token representationsare not yet as refined as in the final layer [46].
Revised: Obtaining label resolution specific feature embeddings in a pre-trained transformer model requires tomake the compromise of extracting these at earlier transformer layers, where the token representationsare not yet as refined as in the final layer [43].

Original: In contrast to the extreme classification task, whichrequires placing true labels exactly at the top-5 positions, the shortlisting task only requires all truemeta-labels be recalled within the shortlist, which is much longer than five elements.
Revised: In contrast to the extreme classification task, whichrequires placing true labels exactly at the top-5 positions, the shortlisting task only requires all truemeta-labels be recalled within the shortlist, which is much longer than five elements.

Original: We observethat the meta-classifiers are able to achieve high-recall (as shown in appendix) at the shortlist lengtheven with the less refined features.
Revised: We observethat the meta-classifiers are able to achieve high-recall (as shown in appendix) at the shortlist lengtheven with the less refined features.

Original: CascadeXML is, therefore, successful in efficiently leveragingintermediate transformer representations for the task of label shortlisting while keeping featurerepresentations separate for each label resolution.
Revised: CascadeXML is, therefore, successful in efficiently leveragingintermediate transformer representations for the task of label shortlisting while keeping featurerepresentations separate for each label resolution.

Original: The structure of the model means that earlier layers in the transformer receive gradients from multipleresolutions, whereas the final layers’ back propagation is limited to gradients from the extremeresolution (cf.
Revised: The structure of the model means that earlier layers in the transformer receive gradients from multipleresolutions, whereas the final layers’ back propagation is limited to gradients from the extremeresolution (cf.

Original: Figure 1).
Revised: Figure 1).

Original: This setup ensures that a portion of the representational capabilities isexclusively available for the extreme task and hence, the final transformer layers are increasinglymore suitable for classification at the extreme resolution.
Revised: This setup ensures that a portion of the representational capabilities isexclusively available for the extreme task and hence, the final transformer layers are increasinglymore suitable for classification at the extreme resolution.

Original: At the same time, since the gradients fromthe finer tasks are also relevant for the coarser meta-label shortlisting task, so additional downstreamtask does not hurt the shortlisting performance at the intermediate transformer layers.
Revised: At the same time, since the gradients fromthe finer tasks are also relevant for the coarser meta-label shortlisting task, so additional downstreamtask does not hurt the shortlisting performance at the intermediate transformer layers.

Original:  Other Related Works
Revised:  Other Related Works

Original: The focus of a majority of initial works in XMC has been on using fixed features for learning theclassifier with one of the following class of methods : (i) one-vs-rest [51, 2, 50, 3], (ii) label tree based[35, 21, 45, 17], (iii) decision tree methods [37, 42, 29], and (iv) label embedding based [5, 43, 13]classifiers.
Revised: The focus of a majority of initial works in XMC has been on using fixed features for learning theclassifier with one of the following class of methods : (i) one-vs-rest [48, 2, 47, 3], (ii) label tree based[33, 20, 42, 16], (iii) decision tree methods [35, 39, 28], and (iv) label embedding based [5, 40, 12]classifiers.

Original: Beyond the above algorithmic categorization of approaches, computational considerationson scaling the training process via negative sampling [15], sketching [31], and smart initialization[12, 41] has also been studied.
Revised: Beyond the above algorithmic categorization of approaches, computational considerationson scaling the training process via negative sampling [14], and sketching [30] have also been studied.

Original: Furthermore, the statistical consequences of negative sampling [39]and missing labels [16, 38] have led to novel insights in designing unbiased loss functions.
Revised: Furthermore, the statistical consequences of negative sampling [37] and missing labels [15, 36] haveled to novel insights in designing unbiased loss functions.

Original: With advances in deep learning, joint learning of features and classifier has been the focus of mostrecent approaches.
Revised: With advances in deep learning, joint learning of features and classifier has been the focus of mostrecent approaches.

Original: Building on the first steps for developing convolutional neural networks for textclassification [23], deep extreme classification was introduced in XML-CNN [27].
Revised: Building on the first steps for developing convolutional neural networks for textclassification [22], deep extreme classification was introduced in XML-CNN [26].

Original: This work precededthe recent developments on employing BiLSTM and transformer encoders discussed in detail the
Revised: This work precededthe recent developments on employing BiLSTM and transformer encoders discussed in detail the

Original: Introduction section.
Revised: Introduction section.

Original: There has been a recent surge in works to tackle XMC for instances with shorttext inputs and those in which labels are endowed with textual descriptions [10, 9,  32, 34, 40].
Revised: There has been a recent surge in works to tackle XMC for instances with shorttext inputs and those in which labels are endowed with textual descriptions [10, 9, 31, 32, 38].

Original: Lately,the XMC setting has also been extended to predict unseen labels in zero-shot learning [14, 55, 47]and multi-modal setting in which the input data consists of textual and image features [33].
Revised: Lately,the XMC setting has also been extended to predict unseen labels in zero-shot learning [13, 52, 44].

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduced a novel paradigm for fine-tuning transformer models in XMC settings.
Revised: In this paper, we introduced a novel paradigm for fine-tuning transformer models in XMC settings.

Original: In contrast to the existing methods, which consider transformer as a blackbox encoder, we leverageits multi-layered architecture to learn data representation corresponding to different resolutions in the
Revised: In contrast to the existing methods, which consider transformer as a blackbox encoder, we leverageits multi-layered architecture to learn data representation corresponding to different resolutions in the

Original: HLT as well as fine-grained labels at the level of the extreme classifier.
Revised: HLT as well as fine-grained labels at the level of the extreme classifier.

Original: The proposed instantiation ofour framework in the form of CascadeXML not only yields state-of-the-art prediction performance,but is also end-to-end trainable (without intermediate reclustering steps), simpler to implement,and fast on training and inference.
Revised: The proposed instantiation ofour framework in the form of CascadeXML not only yields state-of-the-art prediction performance,but is also end-to-end trainable (without intermediate reclustering steps), simpler to implement,and fast on training and inference.

Original: Beyond further research in extreme classification towards fullyharnessing the representation capabilities of transformer encoders, we believe our approach caninspire future multi-resolution architectures in other domains as well which leverage label hierarchy.
Revised: Beyond further research in extreme classification towards fullyharnessing the representation capabilities of transformer encoders, we believe our approach caninspire future multi-resolution architectures in other domains as well which leverage label hierarchy.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] In the appendix (c) Did you discuss any potential negative societal impacts of your work?[Yes]
Revised: [Yes] In the appendix(c) Did you discuss any potential negative societal impacts of your work?

Original:  In the appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] In theappendix(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Code is in the
Revised: [Yes] Code is in the

Original: supplementary material
Revised: supplementary material

Original: , the data is publically available.
Revised: , the data is publically available.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: In the appendix A.2.
Revised: In the appendix A.2.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No] It is not a standard practice in this domain.
Revised: [No] It is not a standard practice in this domain.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: In the appendix A.
Revised: In the appendix A.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: In Section 4:
Revised: In Section 4:

Original: Benchmark datasets
Revised: Benchmark datasets

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Original: [Table]
Revised: [N/A]

Original: [Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table]
Revised: [N/A]

Original: [Table]
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Table]
Revised: [N/A]