Original: CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification
Revised: CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification

Original: Abstract
Revised: Abstract

Original: Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices.
Revised: Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices.

Original: Recent approaches, such as XR-Transformer and LightXML, leverage a transformer instance to achieve state-of-the-art performance.
Revised: Recent approaches, such as XR-Transformer and LightXML, leverage a transformer instance to achieve state-of-the-art performance.

Original: However, in this process, these approaches need to make various trade-offs between performance and computational requirements.
Revised: However, in this process, these approaches need to make various trade-offs between performance and computational requirements.

Original: A major shortcoming, as compared to the Bi-LSTM based AttentionXML, is that they fail to keep separate feature representations for each resolution in a label tree.
Revised: A major shortcoming, as compared to the Bi-LSTM based AttentionXML, is that they fail to keep separate feature representations for each resolution in a label tree.

Original: We thus propose CascadeXML, an end-to-end multiresolution learning pipeline, which can harness the multi-layered architecture of a transformer model for attending to different label resolutions with separate feature representations.
Revised: We thus propose CascadeXML, an end-to-end multiresolution learning pipeline, which can harness the multi-layered architecture of a transformer model for attending to different label resolutions with separate feature representations.

Original: CascadeXML significantly outperforms all existing approaches with non-trivial gains obtained on benchmark datasets consisting of up to three million labels.
Revised: CascadeXML significantly outperforms all existing approaches with non-trivial gains obtained on benchmark datasets consisting of up to three million labels.

Original: Code for CascadeXML will be made publicly available at https: //github.com/xmc-aalto/cascadexml .
Revised: Code for CascadeXML will be made publicly available at https: //github.com/xmc-aalto/cascadexml .

Original:  Introduction
Revised:  Introduction

Original: Extreme multi-label text classification (XMC) deals with the problem of predicting the most relevant subset of labels from an enormously large label space spanning up to millions of labels.
Revised: Extreme multi-label text classification (XMC) deals with the problem of predicting the most relevant subset of labels from an enormously large label space spanning up to millions of labels.

Original: Over the years, extreme classification has found many applications in e-commerce, like product recommendation [7] and dynamic search advertisement [15], document tagging [1] and open-domain question answering [6, 25] and thus, practical solutions to this problem can have significant and far-reaching impact.
Revised: Over the years, extreme classification has found many applications in e-commerce, like product recommendation [7] and dynamic search advertisement [15], document tagging [1] and open-domain question answering [6, 25] and thus, practical solutions to this problem can have significant and far-reaching impact.

Original: The output space in extreme classification is not only large but also, the distribution of instances among labels follows Zipf’s law such that a large fraction of them are tail labels [2, 16].
Revised: The output space in extreme classification is not only large but also, the distribution of instances among labels follows Zipf’s law such that a large fraction of them are tail labels [2, 16].

Original: For example, in a dataset of Wikipedia corpus containing 500K labels, only 2.1% labels annotate more thandata points and 60% of the labels annotate less than 10 data points!
Revised: For example, in a dataset of Wikipedia corpus containing 500K labels, only 2.1% labels annotate more thandata points and 60% of the labels annotate less than 10 data points!

Original: This inherently makes learning meaningful representations at an extreme scale a challenging problem.
Revised: This inherently makes learning meaningful representations at an extreme scale a challenging problem.

Original: Further, as the memory and computational requirements grow linearly with the size of the label space, naive treatment via classical methods and off-the-shelf solvers fails to deal with such large output spaces [8, 53].
Revised: Further, as the memory and computational requirements grow linearly with the size of the label space, naive treatment via classical methods and off-the-shelf solvers fails to deal with such large output spaces [8, 53].

Original: In extreme scenarios with millions of labels, a surrogate step of shortlisting candidate labels becomes crucial to perform extreme classification and has been adapted by many popular approaches [10, 18, 22, 51, 56].
Revised: In extreme scenarios with millions of labels, a surrogate step of shortlisting candidate labels becomes crucial to perform extreme classification and has been adapted by many popular approaches [10, 18, 22, 51, 56].

Original: For each data point, these approaches first solve a simpler task with coarse label-clusters or meta-labels as the label space to create a shortlist of relevant labels for the extreme classification task.
Revised: For each data point, these approaches first solve a simpler task with coarse label-clusters or meta-labels as the label space to create a shortlist of relevant labels for the extreme classification task.

Original: This effectively reduces the training time and computational complexity of the extreme task to that of the shortlisting step which is O ( √ L ) , where L is the number of labels.
Revised: This effectively reduces the training time and computational complexity of the extreme task to that of the shortlisting step which is O ( √ L ) , where L is the number of labels.

Original: Existing works leverage label representation as a centroid of their annotated instances to create a
Revised: Existing works leverage label representation as a centroid of their annotated instances to create a

Original: Hierarchical Label Tree (HLT) [36, 54].
Revised: Hierarchical Label Tree (HLT) [36, 54].

Original: While earlier tree-based methods like Parabel [36] and 36th
Revised: While earlier tree-based methods like Parabel [36] and 36th

Original: [Table]
Revised: [Table]

Original: Bonsai [21] used the entire HLT for extreme classification, many recent works [8, 10, 18, 22, 51] use label clusters only at a certain level of the HLT as meta-labels which are in turn used to shortlist candidate labels for the extreme task.
Revised: Bonsai [21] used the entire HLT for extreme classification, many recent works [8, 10, 18, 22, 51] use label clusters only at a certain level of the HLT as meta-labels which are in turn used to shortlist candidate labels for the extreme task.

Original: In contrast, XR-Transformer [56] and AttentionXML [54] leverage multiple levels of the HLT, such that each level corresponds to a certain label resolution in the tree structure [56].
Revised: In contrast, XR-Transformer [56] and AttentionXML [54] leverage multiple levels of the HLT, such that each level corresponds to a certain label resolution in the tree structure [56].

Original: 1.1 Motivation: Strengths and Weaknesses of Current Approaches
Revised: 1.1 Motivation: Strengths and Weaknesses of Current Approaches

Original: The earliest work to successfully combine label-tree based shortlisting and attention-based deep encoders is AttentionXML [54], which employs multiple Bi-LSTMs models that are trained sequentially for each tree resolution.
Revised: The earliest work to successfully combine label-tree based shortlisting and attention-based deep encoders is AttentionXML [54], which employs multiple Bi-LSTMs models that are trained sequentially for each tree resolution.

Original: More recent approaches [8, 18, 51, 56] replaced the model architecture with a more powerful transformer model [46] and fine-tune a pre-trained instance such as Bert [11].
Revised: More recent approaches [8, 18, 51, 56] replaced the model architecture with a more powerful transformer model [46] and fine-tune a pre-trained instance such as Bert [11].

Original: Without careful designs, such models are computationally very expensive and earlier works [8, 51] could not effectively leverage transformers for both computation and performance on XMC tasks.
Revised: Without careful designs, such models are computationally very expensive and earlier works [8, 51] could not effectively leverage transformers for both computation and performance on XMC tasks.

Original: Next, we discuss two contemporary works LightXML [18] and XR-Transformer [56].
Revised: Next, we discuss two contemporary works LightXML [18] and XR-Transformer [56].

Original: LightXML introduces the concept of dynamic negative sampling, which replaces pre-computed label shortlists with a dynamically calculated shortlist that changes as the model’s weights get updated.
Revised: LightXML introduces the concept of dynamic negative sampling, which replaces pre-computed label shortlists with a dynamically calculated shortlist that changes as the model’s weights get updated.

Original: This enables end-to-end training with a single model by using the final feature representation of the transformer encoder for both the meta- and the extreme-classification task.
Revised: This enables end-to-end training with a single model by using the final feature representation of the transformer encoder for both the meta- and the extreme-classification task.

Original: Unfortunately, these two tasks appear to interfere with one another [22].
Revised: Unfortunately, these two tasks appear to interfere with one another [22].

Original: We hypothesize that this is because the meta task needs the attention maps to focus on different tokens than the extreme task.
Revised: We hypothesize that this is because the meta task needs the attention maps to focus on different tokens than the extreme task.

Original: Furthermore, it only uses a single-level tree, which prevents scaling to the largest datasets.
Revised: Furthermore, it only uses a single-level tree, which prevents scaling to the largest datasets.

Original: XR-Transformer takes inspiration from multi-resolution approaches in computer vision like super resolution [24] and progressive growing of GANs [19, 20], and enables multiple resolutions through iterative training.
Revised: XR-Transformer takes inspiration from multi-resolution approaches in computer vision like super resolution [24] and progressive growing of GANs [19, 20], and enables multiple resolutions through iterative training.

Original: However, unlike progressively grown GANs, which predict only at the highest resolution, XR-Transformer needs predictions across all resolutions for its progressive shortlisting pipeline, but uses representations trained at a single resolution.
Revised: However, unlike progressively grown GANs, which predict only at the highest resolution, XR-Transformer needs predictions across all resolutions for its progressive shortlisting pipeline, but uses representations trained at a single resolution.

Original: In practice, this leads to XRTransformer having a complex multi-stage pipeline where the transformer model is iteratively trained up to a certain resolution and then frozen.
Revised: In practice, this leads to XRTransformer having a complex multi-stage pipeline where the transformer model is iteratively trained up to a certain resolution and then frozen.

Original: This is followed by a re-clustering and re-training of multiple classifiers, working at different resolutions, with the same fixed transformer features [56,
Revised: This is followed by a re-clustering and re-training of multiple classifiers, working at different resolutions, with the same fixed transformer features [56,

Original: Alg. 2].
Revised: Alg. 2].

Original: For datasets with over 500K labels, the transformer is only trained on up to 2 15 clusters [56,
Revised: For datasets with over 500K labels, the transformer is only trained on up to 2 15 clusters [56,

Original: Table 6], and the resulting features are used for the extreme task.
Revised: Table 6], and the resulting features are used for the extreme task.

Original: Unlike AttentionXML, using multiple instances of transformer models becomes undesirable due to their computational overhead.
Revised: Unlike AttentionXML, using multiple instances of transformer models becomes undesirable due to their computational overhead.

Original: This enforces LightXML and XR-Transformer to make different trade-offs when leveraging a single transformer model for XMC tasks compared to AttentionXML, see Table 1.
Revised: This enforces LightXML and XR-Transformer to make different trade-offs when leveraging a single transformer model for XMC tasks compared to AttentionXML, see Table 1.

Original: In this paper, we present a method, CascadeXML , that combines the strengths of these approaches, thus creating an end to-end trainable multi-resolution learning pipeline which trains a single transformer model across multiple resolutions in a way that allows the creation of label resolution specific attention maps and feature embeddings.
Revised: In this paper, we present a method, CascadeXML , that combines the strengths of these approaches, thus creating an end to-end trainable multi-resolution learning pipeline which trains a single transformer model across multiple resolutions in a way that allows the creation of label resolution specific attention maps and feature embeddings.

Original: αααα
Revised: αααα

Original: R (1)
Revised: R (1)

Original: R (2)
Revised: R (2)

Original: R (3)
Revised: R (3)

Original: Input Embeddings
Revised: Input Embeddings

Original: BERT
Revised: BERT

Original: Layers 1Layer
Revised: Layers 1Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: W W
Revised: W W

Original: W
Revised: W

Original: Wϑ (1) =
Revised: Wϑ (1) =

Original: Φ (6) CLS ( x ) ϑ (2) ϑ (3) ϑ (4)
Revised: Φ (6) CLS ( x ) ϑ (2) ϑ (3) ϑ (4)

Original: HLT
Revised: HLT

Original: C (1) ...
Revised: C (1) ...

Original: C (1) K
Revised: C (1) K

Original: C (2)
Revised: C (2)

Original: C (2) ... C (2) ...
Revised: C (2) ... C (2) ...

Original: C (2) K
Revised: C (2) K

Original: C (3) C (3)
Revised: C (3) C (3)

Original: C (3) ...
Revised: C (3) ...

Original: C (3) ... C (3) ...
Revised: C (3) ... C (3) ...

Original: C (3) K    ...
Revised: C (3) K    ...

Original: L − 3 L − 2 L −
Revised: L − 3 L − 2 L −

Original: L forward pass backward pass
Revised: L forward pass backward pass

Original: S (1) =
Revised: S (1) =

Original: { C (1)1 ,..., C (1) K 1 }
Revised: { C (1)1 ,..., C (1) K 1 }

Original: S (1) top S (2)
Revised: S (1) top S (2)

Original: S (2) top S (3)
Revised: S (2) top S (3)

Original: S (3) top S (4) scores
Revised: S (3) top S (4) scores

Original: Meta-labels:
Revised: Meta-labels:

Original: C (3)1 = { 1 , 2 }
Revised: C (3)1 = { 1 , 2 }

Original: C (2)1 = {{ 1 , 2 } , { 3 , 4 }}
Revised: C (2)1 = {{ 1 , 2 } , { 3 , 4 }}

Original: 1.2 CascadeXML in a Nutshell
Revised: 1.2 CascadeXML in a Nutshell

Original: The key insights that enable calculating resolution-specific representations with a single forward pass through the transformer are a) that the shortlisting tasks get easier for lower label resolution and b) that the intermediate transformer representations are already quite powerful discriminators [48].
Revised: The key insights that enable calculating resolution-specific representations with a single forward pass through the transformer are a) that the shortlisting tasks get easier for lower label resolution and b) that the intermediate transformer representations are already quite powerful discriminators [48].

Original: Consequently, CascadeXML extracts features for classification at the coarser meta-label resolution from earlier transformer layers and uses the last layer only for classification at the extreme resolution.
Revised: Consequently, CascadeXML extracts features for classification at the coarser meta-label resolution from earlier transformer layers and uses the last layer only for classification at the extreme resolution.

Original: The advantage of this is twofold: First, we postulate that the gain in classification accuracy due to enabling tree resolution-specific attention maps outweighs the loss in accuracy due to the slightly less powerful representations at the earlier transformer layers.
Revised: The advantage of this is twofold: First, we postulate that the gain in classification accuracy due to enabling tree resolution-specific attention maps outweighs the loss in accuracy due to the slightly less powerful representations at the earlier transformer layers.

Original: Second, this ensures that the representation capacity of the later layers is exclusively available to the more difficult tasks.
Revised: Second, this ensures that the representation capacity of the later layers is exclusively available to the more difficult tasks.

Original: The main idea of CascadeXML is illustrated in Figure 1.
Revised: The main idea of CascadeXML is illustrated in Figure 1.

Original: An input text is propagated through the transformer model until the first shortlisting level is reached.
Revised: An input text is propagated through the transformer model until the first shortlisting level is reached.

Original: There, the embedding ϑ (1) corresponding to the [CLS] token is extracted, and used to determine scores for each level-1 meta-label.
Revised: There, the embedding ϑ (1) corresponding to the [CLS] token is extracted, and used to determine scores for each level-1 meta-label.

Original: The highest-scoring meta-labels are selected to create a shortlist of candidate labels S (1) top .
Revised: The highest-scoring meta-labels are selected to create a shortlist of candidate labels S (1) top .

Original: The input is propagated further to perform another shortlisting step at the next resolution, at which a new [CLS] embedding ϑ (2) is extracted.
Revised: The input is propagated further to perform another shortlisting step at the next resolution, at which a new [CLS] embedding ϑ (2) is extracted.

Original: Crucially, ϑ (2) is the result of a separate self-attention operation, and its attention map is a differently weighted combination of the token embeddings as compared to ϑ (1) .
Revised: Crucially, ϑ (2) is the result of a separate self-attention operation, and its attention map is a differently weighted combination of the token embeddings as compared to ϑ (1) .

Original: The new embedding ϑ (2) is used to refine the shortlist by selecting the top-scoring level-2 meta-labels that are children of the shortlisted level-1 meta-labels.
Revised: The new embedding ϑ (2) is used to refine the shortlist by selecting the top-scoring level-2 meta-labels that are children of the shortlisted level-1 meta-labels.

Original: This process is repeated until at the final layer of the transformer, where classification is performed at the full label resolution.
Revised: This process is repeated until at the final layer of the transformer, where classification is performed at the full label resolution.

Original: Contributions In this paper, our contributions are the following:
Revised: Contributions In this paper, our contributions are the following:

Original: • We introduce a novel paradigm which can harness the multi-layered architecture of transformers for learning representations at multiple resolutions in an HLT.
Revised: • We introduce a novel paradigm which can harness the multi-layered architecture of transformers for learning representations at multiple resolutions in an HLT.

Original: This is in contrast to the existing approaches which treat transformer as merely a black box encoder, • Implemented via CascadeXML, it is an end-to-end trainable multi-resolution learning pipeline which is quick to train and simple to implement.
Revised: This is in contrast to the existing approaches which treat transformer as merely a black box encoder, • Implemented via CascadeXML, it is an end-to-end trainable multi-resolution learning pipeline which is quick to train and simple to implement.

Original: It eliminates the need of bells and whistles like multi-stage procedures, reclustering
Revised: It eliminates the need of bells and whistles like multi-stage procedures, reclustering

Original: and bootstrap training to achieve state-of-the-art performance, and
Revised: and bootstrap training to achieve state-of-the-art performance, and

Original: • CascadeXML not only scales to dataset with millions of labels but also improves the current state-of-the-art in terms of prediction performance by upto ∼ 6% on benchmark datasets.
Revised: • CascadeXML not only scales to dataset with millions of labels but also improves the current state-of-the-art in terms of prediction performance by upto ∼ 6% on benchmark datasets.

Original: In addition to this, it is computationally most efficient to train and clocks multi-fold improvement in inference time as compared to existing transformer-based approaches.
Revised: In addition to this, it is computationally most efficient to train and clocks multi-fold improvement in inference time as compared to existing transformer-based approaches.

Original:  Notation & Preliminaries
Revised:  Notation & Preliminaries

Original: Problem setup:
Revised: Problem setup:

Original: In extreme classification, the input text instance (document or query) x ∈ X is mapped to a subset of labels y ⊂
Revised: In extreme classification, the input text instance (document or query) x ∈ X is mapped to a subset of labels y ⊂

Original: [ L ] out of the L available labels identified through the integers [ L ] := { 1 , . . .
Revised: [ L ] out of the L available labels identified through the integers [ L ] := { 1 , . . .

Original: , L } , such that L ∼ 10 6 .
Revised: , L } , such that L ∼ 10 6 .

Original: Usually, we identify the labels through a binary vector y ∈ { 0 , 1 } L , where y j = 1 ⇔ j ∈ y .
Revised: Usually, we identify the labels through a binary vector y ∈ { 0 , 1 } L , where y j = 1 ⇔ j ∈ y .

Original: The instances and labels are jointly distributed according to an unknown distribution ( x , y ) ∼ P of which we have a i.i.d.
Revised: The instances and labels are jointly distributed according to an unknown distribution ( x , y ) ∼ P of which we have a i.i.d.

Original: sample D = (cid:8)(cid:0) x i , y i (cid:1) : i ∈ [ n ] (cid:9) available as a training set.
Revised: sample D = (cid:8)(cid:0) x i , y i (cid:1) : i ∈ [ n ] (cid:9) available as a training set.

Original: Even though the number of labels is large, the label vectors are sparse.
Revised: Even though the number of labels is large, the label vectors are sparse.

Original: As part of the model pipeline, a text vectorizer Φ : X (cid:55)→ R d projects the documents to a d -dimensional embedding space.
Revised: As part of the model pipeline, a text vectorizer Φ : X (cid:55)→ R d projects the documents to a d -dimensional embedding space.

Original: These vectorizers can be simple bag-of-words (BOW) or TF-IDF transformations Φ tf-idf , or a learnable model Φ dnn that encodes the documents to the embedding space using weights learnt over the dataset.
Revised: These vectorizers can be simple bag-of-words (BOW) or TF-IDF transformations Φ tf-idf , or a learnable model Φ dnn that encodes the documents to the embedding space using weights learnt over the dataset.

Original: The final classification of an instance x is achieved by computing per label scores ⟨ w l , Φ( x ) ⟩ using label-wise weight vectors w l ∈ R
Revised: The final classification of an instance x is achieved by computing per label scores ⟨ w l , Φ( x ) ⟩ using label-wise weight vectors w l ∈ R

Original: d
Revised: d

Original: which are either (i) jointly learnt with Φ in a deep architecture [8, 10, 18, 32, 51, 54], or (ii) solely learnt in shallow classifiers [2, 55].
Revised: which are either (i) jointly learnt with Φ in a deep architecture [8, 10, 18, 32, 51, 54], or (ii) solely learnt in shallow classifiers [2, 55].

Original: Transformer Encoder: More recent approaches in extreme classification have achieved stateof-the-art performance using pre-trained transformers as text vectorizers [18, 51, 56].
Revised: Transformer Encoder: More recent approaches in extreme classification have achieved stateof-the-art performance using pre-trained transformers as text vectorizers [18, 51, 56].

Original: Popular transformer encoders for NLP [11, 27, 50] consist of an embedding layer, followed by a series of stacked self-attention layers and task-specific heads that can be fine-tuned along with the encoder.
Revised: Popular transformer encoders for NLP [11, 27, 50] consist of an embedding layer, followed by a series of stacked self-attention layers and task-specific heads that can be fine-tuned along with the encoder.

Original: For a sequence of m input words, each layer a in these models produces a sequence of embeddings (cid:16) Φ ( a ) 

Revised: For a sequence of m input words, each layer a in these models produces a sequence of embeddings (cid:16) Φ ( a )

Original: ( x ) , . . .
Revised: ( x ) , . . .

Original: , Φ ( a ) m ( x ) (cid:17) , where Φ ( a ) i ( x ) ∈ R
Revised: , Φ ( a ) m ( x ) (cid:17) , where Φ ( a ) i ( x ) ∈ R

Original: d is a representation for the i ’th token in the input sequence at the a ’th layer, and Φ ( a ) CLS ( x ) :=
Revised: d is a representation for the i ’th token in the input sequence at the a ’th layer, and Φ ( a ) CLS ( x ) :=

Original: Φ ( a ) 

Revised: Φ ( a )

Original: ( x ) is an embedding corresponding to a special [CLS] token that has been pre-trained to capture a representation of the entire input [11, 27].
Revised: ( x ) is an embedding corresponding to a special [CLS] token that has been pre-trained to capture a representation of the entire input [11, 27].

Original: Label Shortlisting: Computing scores ⟨ w l , Φ( x ) ⟩ for each label l ∈
Revised: Label Shortlisting: Computing scores ⟨ w l , Φ( x ) ⟩ for each label l ∈

Original: [ L ] in an extremely large output space is computationally very demanding, especially when the feature space R d ∋ Φ( x ) is high dimensional to allow for expressive features.
Revised: [ L ] in an extremely large output space is computationally very demanding, especially when the feature space R d ∋ Φ( x ) is high dimensional to allow for expressive features.

Original: To alleviate this problem, the labels L are grouped together under K meta-labels { C 1 , . .
Revised: To alleviate this problem, the labels L are grouped together under K meta-labels { C 1 , . .

Original: , C K } =: R , with K ≈ O ( √ L ) .
Revised: , C K } =: R , with K ≈ O ( √ L ) .

Original: Each meta-label is represented as the set of its child labels, C k = { l ∈ [ L ] : l is child of k } and each target set y ⊂
Revised: Each meta-label is represented as the set of its child labels, C k = { l ∈ [ L ] : l is child of k } and each target set y ⊂

Original: [ L ] corresponds to a meta-target ˜
Revised: [ L ] corresponds to a meta-target ˜

Original: y
Revised: y

Original: =
Revised: =

Original: R ( y ) :
Revised: R ( y ) :

Original: = { C k | ∃ i ∈
Revised: = { C k | ∃ i ∈

Original: y : i ∈ C k } .
Revised: y : i ∈ C k } .

Original: A meta-label C l is relevant to a data point if it contains at least one relevant label.
Revised: A meta-label C l is relevant to a data point if it contains at least one relevant label.

Original: Ideally, the labels grouped under one meta-label should be semantically similar to ensure the common representations learnt for meta- and extreme- tasks are relevant to both tasks [10].
Revised: Ideally, the labels grouped under one meta-label should be semantically similar to ensure the common representations learnt for meta- and extreme- tasks are relevant to both tasks [10].

Original: This has the added advantage that the negative labels selected by the shortlisting procedure will be the most confusing ones, leading to hard negative sampling and improved learning of the extreme classifier [39].
Revised: This has the added advantage that the negative labels selected by the shortlisting procedure will be the most confusing ones, leading to hard negative sampling and improved learning of the extreme classifier [39].

Original: The label vectors are sparse to the extent that the much coarser meta-labels still contain mostly zeros, | ˜ y | ≪ K .
Revised: The label vectors are sparse to the extent that the much coarser meta-labels still contain mostly zeros, | ˜ y | ≪ K .

Original: This means that it is enough to explicitly calculate the scores only for the descendants of relevant meta-labels.
Revised: This means that it is enough to explicitly calculate the scores only for the descendants of relevant meta-labels.

Original: In practice, an algorithm will first predict scores for each meta-label and select the top-scoring ones as a shortlist for further classification.
Revised: In practice, an algorithm will first predict scores for each meta-label and select the top-scoring ones as a shortlist for further classification.

Original: The meta- and extreme-classification tasks can either be learnt sequentially [8, 10] or jointly through dynamic label-shortlisting [18, 22].
Revised: The meta- and extreme-classification tasks can either be learnt sequentially [8, 10] or jointly through dynamic label-shortlisting [18, 22].

Original: For extremely large label spaces, a single label resolution for shortlisting is not optimal [56].
Revised: For extremely large label spaces, a single label resolution for shortlisting is not optimal [56].

Original: It is either too large that the meta-task itself becomes very expensive, or its individual clusters are so large that the shortlist contains too many candidate labels [56].
Revised: It is either too large that the meta-task itself becomes very expensive, or its individual clusters are so large that the shortlist contains too many candidate labels [56].

Original: Thus, methods like AttentionXML and
Revised: Thus, methods like AttentionXML and

Original: XR-Transformer use a multi-level shortlisting procedure based on a Hierarchical Label Tree (HLT) .
Revised: XR-Transformer use a multi-level shortlisting procedure based on a Hierarchical Label Tree (HLT) .

Original: Label Trees: An HLT H = (cid:8) R (1) , . . .
Revised: Label Trees: An HLT H = (cid:8) R (1) , . . .

Original: , R ( T ) (cid:9) is a hierarchical clustering of labels into T levels of successively refined clusters R ( t ) =
Revised: , R ( T ) (cid:9) is a hierarchical clustering of labels into T levels of successively refined clusters R ( t ) =

Original: (cid:110) C ( t ) 1 , . .
Revised: (cid:110) C ( t ) 1 , . .

Original: , C ( t ) K t (cid:111) .
Revised: , C ( t ) K t (cid:111) .

Original: The clusters C ( t ) k of each level form a disjoint partitioning of the clusters of the next, C ( t ) k
Revised: The clusters C ( t ) k of each level form a disjoint partitioning of the clusters of the next, C ( t ) k

Original: ⊂ R ( t +1) , with the convention that the T + 1 th level is the full label space R ( T +1) =
Revised: ⊂ R ( t +1) , with the convention that the T + 1 th level is the full label space R ( T +1) =

Original: [ L ] .
Revised: [ L ] .

Original: As such, each cluster C ( T ) k (or meta-label) at level T is represented by a set of labels, each cluster at level T − 1 is a set of level- T meta-labels, and thus a set of sets of labels, and so on.
Revised: As such, each cluster C ( T ) k (or meta-label) at level T is represented by a set of labels, each cluster at level T − 1 is a set of level- T meta-labels, and thus a set of sets of labels, and so on.

Original: In practice, label representations z l as an aggregate of their positive instance features are leveraged to create an HLT.
Revised: In practice, label representations z l as an aggregate of their positive instance features are leveraged to create an HLT.

Original: Typically, z l is constructed using Φ tf-idf as z l = v l ∥ v l ∥ , where v l = (cid:88)
Revised: Typically, z l is constructed using Φ tf-idf as z l = v l ∥ v l ∥ , where v l = (cid:88)

Original: i : y ( i ) l = Φ tf-idf ( x ( i ) ) .
Revised: i : y ( i ) l = Φ tf-idf ( x ( i ) ) .

Original: The HLT is often constructed by recursively partitioning the label-space using balanced k-means clustering in a top-down fashion [8, 18, 54, 56].
Revised: The HLT is often constructed by recursively partitioning the label-space using balanced k-means clustering in a top-down fashion [8, 18, 54, 56].

Original: For example, an HLT constructed using balanced 2-means clustering has ⌊ log 2 L ⌋ levels, out of which we select T levels corresponding to different meta-label resolutions.
Revised: For example, an HLT constructed using balanced 2-means clustering has ⌊ log 2 L ⌋ levels, out of which we select T levels corresponding to different meta-label resolutions.

Original:  Method: CascadeXML
Revised:  Method: CascadeXML

Original: At its core, the CascadeXML model consists of three components: A pre-trained language model Φ which allows to extract a document representation Φ ( a ) CLS ( x ) ∈ R d at different layers a ∈ [ A ] , an HLT
Revised: At its core, the CascadeXML model consists of three components: A pre-trained language model Φ which allows to extract a document representation Φ ( a ) CLS ( x ) ∈ R d at different layers a ∈ [ A ] , an HLT

Original: H = (cid:8) R (1) , . .
Revised: H = (cid:8) R (1) , . .

Original: , R ( T ) (cid:9) that divides the label space into K 1 < . . .
Revised: , R ( T ) (cid:9) that divides the label space into K 1 < . . .

Original: <
Revised: <

Original: K T <
Revised: K T <

Original: L = K T +1 successively refined clusters, and a set of linear classifiers (cid:110) W ( t ) =
Revised: L = K T +1 successively refined clusters, and a set of linear classifiers (cid:110) W ( t ) =

Original: [ w ( t ) 1 , . .
Revised: [ w ( t ) 1 , . .

Original: . , w ( t ) K t ] ∈ R
Revised: . , w ( t ) K t ] ∈ R

Original: d
Revised: d

Original: × K t : t ∈ [ T + 1] (cid:111) .
Revised: × K t : t ∈ [ T + 1] (cid:111) .

Original: For each meta-classifier, we select a layer in order to extract resolution-specific feature embeddings ϑ ( t ) :
Revised: For each meta-classifier, we select a layer in order to extract resolution-specific feature embeddings ϑ ( t ) :

Original: =
Revised: =

Original: Φ ( a t ) CLS ( x ) with 1 < a 1 < . . .
Revised: Φ ( a t ) CLS ( x ) with 1 < a 1 < . . .

Original: <
Revised: <

Original: a T <
Revised: a T <

Original: A = a T +1 .
Revised: A = a T +1 .

Original: Refining and Coarsening Label Vectors:
Revised: Refining and Coarsening Label Vectors:

Original: In order to transfer the representation of a shortlist of a label vector between different levels of the HLT, we need to define two operations.
Revised: In order to transfer the representation of a shortlist of a label vector between different levels of the HLT, we need to define two operations.

Original: The first is refining the resolution of a shortlist:
Revised: The first is refining the resolution of a shortlist:

Original: Given a shortlist S = { C ( t ) k 1 , . . .
Revised: Given a shortlist S = { C ( t ) k 1 , . . .

Original: , C ( t ) k s } ⊂ R ( t ) of clusters of the resolution at level t , we want to calculate the representation of this shortlist in the next level, denoted R ( t ) ↓ ( S ) .
Revised: , C ( t ) k s } ⊂ R ( t ) of clusters of the resolution at level t , we want to calculate the representation of this shortlist in the next level, denoted R ( t ) ↓ ( S ) .

Original: Since each cluster is defined as the set of its descendants, this is achieved by taking the union over the clusters in the list.
Revised: Since each cluster is defined as the set of its descendants, this is achieved by taking the union over the clusters in the list.

Original: The second operation is to find the cover of meta-labels that envelope a given set of labels, R ( t ) ↑ ( S ) .
Revised: The second operation is to find the cover of meta-labels that envelope a given set of labels, R ( t ) ↑ ( S ) .

Original: This is achieved by identifying each of the clusters in the level above for which there exists at least one descendant in the shortlist.
Revised: This is achieved by identifying each of the clusters in the level above for which there exists at least one descendant in the shortlist.

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: We define R ( t ) ( y ) = (cid:16)
Revised: We define R ( t ) ( y ) = (cid:16)

Original: [Equation]
Revised: [Equation]

Original: ( t +
Revised: ( t +

Original: [Equation]
Revised: [Equation]

Original: ↑ ◦ . . .
Revised: ↑ ◦ . . .

Original: ◦ R ( T ) ↑ (cid:17) ( y ) to be the level- t cover of the label set y ⊂
Revised: ◦ R ( T ) ↑ (cid:17) ( y ) to be the level- t cover of the label set y ⊂

Original: [ L ] .
Revised: [ L ] .

Original: Multi-resolution Dynamic Label Shortlisting: We adapt dynamic label shortlisting [18] for multiresolution training approaches.
Revised: Multi-resolution Dynamic Label Shortlisting: We adapt dynamic label shortlisting [18] for multiresolution training approaches.

Original: Formally, given a shortlist S ( t ) of s clusters { C ( t ) k 1 , . . .
Revised: Formally, given a shortlist S ( t ) of s clusters { C ( t ) k 1 , . . .

Original: , C ( t ) k s } =:
Revised: , C ( t ) k s } =:

Original: S ( t ) ⊂
Revised: S ( t ) ⊂

Original: R ( t ) at level t , and the corresponding classification features ϑ ( t ) , a new shortlist S ( t +1) ⊂
Revised: R ( t ) at level t , and the corresponding classification features ϑ ( t ) , a new shortlist S ( t +1) ⊂

Original: R ( t +1) is generated as follows:
Revised: R ( t +1) is generated as follows:

Original: The k highest-scoring meta-labels are selected into S ( t ) top and the next shortlist S ( t +1) is then given as the refinement:
Revised: The k highest-scoring meta-labels are selected into S ( t ) top and the next shortlist S ( t +1) is then given as the refinement:

Original: [Equation]
Revised: [Equation]

Original: At the first level, however, the shortlist is initialized to consider all meta labels S (1) =
Revised: At the first level, however, the shortlist is initialized to consider all meta labels S (1) =

Original: R (1) .
Revised: R (1) .

Original: It is of essence that all true positive (meta-)labels appear at every level during training time [8, 18, 54].
Revised: It is of essence that all true positive (meta-)labels appear at every level during training time [8, 18, 54].

Original: However, this is difficult to ensure implicitly due to imperfect recall rate of classifiers.
Revised: However, this is difficult to ensure implicitly due to imperfect recall rate of classifiers.

Original: Hence, during training all meta-labels corresponding to ground-truth labels are added to the candidate set, leading to
Revised: Hence, during training all meta-labels corresponding to ground-truth labels are added to the candidate set, leading to

Original: [Equation]
Revised: [Equation]

Original: Consequently, a single training step in CascadeXML looks as given in algorithm 1.
Revised: Consequently, a single training step in CascadeXML looks as given in algorithm 1.

Original: Training Objective: At each level of the tree, the goal is to correctly identify the most likely meta-labels, which can be achieved through minimization of a one-vs-all loss, such as BCE
Revised: Training Objective: At each level of the tree, the goal is to correctly identify the most likely meta-labels, which can be achieved through minimization of a one-vs-all loss, such as BCE

Original: [Equation]
Revised: [Equation]

Original: Input: instance x , labels y , clusters C , features Φ , classifiers W , weights α ϑ ( t ) ← Φ ( t ) CLS ( x ) ∀ t ∈ [ T + 1] /* Forward pass through Transformer */ loss ←
Revised: Input: instance x , labels y , clusters C , features Φ , classifiers W , weights α ϑ ( t ) ← Φ ( t ) CLS ( x ) ∀ t ∈ [ T + 1] /* Forward pass through Transformer */ loss ←

Original: S (1) ← C (1) /* Initialize shortlist */ for t in 1 , 2 , ..., T do y ( t ) ← R ( t ) ( y ) /
Revised: S (1) ← C (1) /* Initialize shortlist */ for t in 1 , 2 , ..., T do y ( t ) ← R ( t ) ( y ) /

Original: * Ground-truth meta-labels */ p i ← ⟨ w ( t ) k
Revised: * Ground-truth meta-labels */ p i ← ⟨ w ( t ) k

Original: i , ϑ ( t ) ⟩ ∀ i ∈ S ( t ) /
Revised: i , ϑ ( t ) ⟩ ∀ i ∈ S ( t ) /

Original: * Sparse prediction */ loss ← loss + α t |S ( t ) | (cid:80) l ∈S ( t ) L ( t ) (cid:16) p l , y ( t ) l (cid:17)
Revised: * Sparse prediction */ loss ← loss + α t |S ( t ) | (cid:80) l ∈S ( t ) L ( t ) (cid:16) p l , y ( t ) l (cid:17)

Original: S ( t ) top ← (cid:110) C ( t ) k j : j ∈ Top k (cid:0)(cid:8) p i | i ∈ S ( t ) (cid:9)(cid:1)(cid:111) .
Revised: S ( t ) top ← (cid:110) C ( t ) k j : j ∈ Top k (cid:0)(cid:8) p i | i ∈ S ( t ) (cid:9)(cid:1)(cid:111) .

Original: /* Top-K selection */
Revised: /* Top-K selection */

Original: S ( t +1) ← R ( t ) ↓ ( S ( t ) top ) ∪ R ( t +1)
Revised: S ( t +1) ← R ( t ) ↓ ( S ( t ) top ) ∪ R ( t +1)

Original: ( y ) /
Revised: ( y ) /

Original: * Add ground-truth positives */ p i ← ⟨ w ( T +1)
Revised: * Add ground-truth positives */ p i ← ⟨ w ( T +1)

Original: k
Revised: k

Original: i , ϑ ( T +1) ⟩ ∀ i ∈ S
Revised: i , ϑ ( T +1) ⟩ ∀ i ∈ S

Original: ( T +1) /
Revised: ( T +1) /

Original: * Final prediction */ loss ← loss + α T +1 |S
Revised: * Final prediction */ loss ← loss + α T +1 |S

Original: ( T +1)
Revised: ( T +1)

Original: | (cid:80) l ∈S L ( T +1)
Revised: | (cid:80) l ∈S L ( T +1)

Original: ( p l , y l ) adjust Φ and W to reduce loss
Revised: ( p l , y l ) adjust Φ and W to reduce loss

Original: The overall objective is a weighted sum of the individual layer losses
Revised: The overall objective is a weighted sum of the individual layer losses

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where level T + 1 corresponds to the extreme classification task.
Revised: where level T + 1 corresponds to the extreme classification task.

Original: To balance out different loss scales as a result of normalization by varied shortlist sizes, we re-scale losses from multiple resolutions using a simple factor α ( t ) =
Revised: To balance out different loss scales as a result of normalization by varied shortlist sizes, we re-scale losses from multiple resolutions using a simple factor α ( t ) =

Original: |S ( t ) | / min t ∈
Revised: |S ( t ) | / min t ∈

Original: [ T +1] ( |S ( t ) | ) .
Revised: [ T +1] ( |S ( t ) | ) .

Original: Inference: The algorithm for inference remains the same as training, except we do not teacher-force true (meta-)labels R ( t ) ( y ) during inference time (Eqn. 3).
Revised: Inference: The algorithm for inference remains the same as training, except we do not teacher-force true (meta-)labels R ( t ) ( y ) during inference time (Eqn. 3).

Original: The inference consists of one forward pass through the transformer backbone, followed by successive shortlisting at all T resolutions, and final extreme classification.
Revised: The inference consists of one forward pass through the transformer backbone, followed by successive shortlisting at all T resolutions, and final extreme classification.

Original: For a b -way branching tree with log b L levels, each level t has b t meta-labels, of which |S ( t ) | ∝ √ b t are selected for the shortlist.
Revised: For a b -way branching tree with log b L levels, each level t has b t meta-labels, of which |S ( t ) | ∝ √ b t are selected for the shortlist.

Original: Therefore, the time complexity is
Revised: Therefore, the time complexity is

Original: [Equation]
Revised: [Equation]

Original: where T dnn denotes the time taken to compute transformer embeddings for an input.
Revised: where T dnn denotes the time taken to compute transformer embeddings for an input.

Original:  Main results
Revised:  Main results

Original: Benchmark datasets & Evaluation Metrics
Revised: Benchmark datasets & Evaluation Metrics

Original: We benchmark the performance of CascadeXML on standard datasets and evaluation metrics, as shown in Table 2 and Table 3.
Revised: We benchmark the performance of CascadeXML on standard datasets and evaluation metrics, as shown in Table 2 and Table 3.

Original: In all, we use 5 public
Revised: In all, we use 5 public

Original: XMC datasets [4, 30], the statistics of which are specified in appendix A.1.
Revised: XMC datasets [4, 30], the statistics of which are specified in appendix A.1.

Original: For all our experiments we use the same raw text input, train-test split and sparse feature representations as used in [54, 18, 56].
Revised: For all our experiments we use the same raw text input, train-test split and sparse feature representations as used in [54, 18, 56].

Original: We use Precision@k (P@k) and Propensity-scored P@k (PSP@k) [16], which focuses more on the model’s performance on tail labels, as the evaluation metrics.
Revised: We use Precision@k (P@k) and Propensity-scored P@k (PSP@k) [16], which focuses more on the model’s performance on tail labels, as the evaluation metrics.

Original: Baseline Methods We compare CascadeXML with strong baselines: DiSMEC [2], Parabel [36], eXtremeText [47], Bonsai [21], XR-Linear [55], AttentionXML, X-Transformer [8], LightXML and
Revised: Baseline Methods We compare CascadeXML with strong baselines: DiSMEC [2], Parabel [36], eXtremeText [47], Bonsai [21], XR-Linear [55], AttentionXML, X-Transformer [8], LightXML and

Original: XR-Transformer.
Revised: XR-Transformer.

Original: The baseline results for these methods were obtained from [56, Table: 2].
Revised: The baseline results for these methods were obtained from [56, Table: 2].

Original: It may be noted that, in order to incorporate the global statistical information of the dataset, XR-Transformer concatenates sparse tf-idf features to those learnt via deep networks (denoted Φ dnn ) as:
Revised: It may be noted that, in order to incorporate the global statistical information of the dataset, XR-Transformer concatenates sparse tf-idf features to those learnt via deep networks (denoted Φ dnn ) as:

Original: [Equation]
Revised: [Equation]

Original: For fair evaluation, we present separate comparison with methods which use (i) features from a DNN encoder only, and (ii) those obtained by combining DNN and tf-idf features.
Revised: For fair evaluation, we present separate comparison with methods which use (i) features from a DNN encoder only, and (ii) those obtained by combining DNN and tf-idf features.

Original: Model Ensemble Settings For CascadeXML, we follow the ensemble settings of LightXML and
Revised: Model Ensemble Settings For CascadeXML, we follow the ensemble settings of LightXML and

Original: XR-Transformer.
Revised: XR-Transformer.

Original: Specifically, we ensemble a model each using BERT [11], RoBerta.
Revised: Specifically, we ensemble a model each using BERT [11], RoBerta.

Original: [27] and
Revised: [27] and

Original: XLNet [50] for Wiki10-31K and AmazonCat-13K.
Revised: XLNet [50] for Wiki10-31K and AmazonCat-13K.

Original: For the larger datasets, we make an ensemble of three BERT models with different seeds for random initialization.
Revised: For the larger datasets, we make an ensemble of three BERT models with different seeds for random initialization.

Original: On the other hand, AttentionXML uses ensemble of 3 models and X-Transformer uses 9 model ensemble with BERT, RoBerta, XLNet large models with three difference indexers.
Revised: On the other hand, AttentionXML uses ensemble of 3 models and X-Transformer uses 9 model ensemble with BERT, RoBerta, XLNet large models with three difference indexers.

Original: Metrics
Revised: Metrics

Original: P@
Revised: P@

Original: P@ P@
Revised: P@ P@

Original: P@ P@
Revised: P@ P@

Original: P@ P@
Revised: P@ P@

Original: P@ P@
Revised: P@ P@

Original: Datasets
Revised: Datasets

Original: Wiki-500K
Revised: Wiki-500K

Original: Amazon-670K
Revised: Amazon-670K

Original: Amazon-3M
Revised: Amazon-3M

Original: DiSMEC 70.
Revised: DiSMEC 70.

Original: Parabel ∗ 68.
Revised: Parabel ∗ 68.

Original: eXtremeText 65.
Revised: eXtremeText 65.

Original: Bonsai ∗ 69.
Revised: Bonsai ∗ 69.

Original: XR-Linear 65.
Revised: XR-Linear 65.

Original: DNN Features
Revised: DNN Features

Original: AttentionXML ∗ 76.
Revised: AttentionXML ∗ 76.

Original: LightXML ∗ 77.
Revised: LightXML ∗ 77.

Original: CascadeXML ∗ 78.
Revised: CascadeXML ∗ 78.

Original: [DNN
Revised: [DNN

Original: [Table]
Revised: [Table]

Original: ‘ * ’ and ‘ ** ’ imply ensemble results of 3 models and 9 models respectively, and ‘-’ implies that the model does not scale for that dataset due to GPU memory constraints.
Revised: ‘ * ’ and ‘ ** ’ imply ensemble results of 3 models and 9 models respectively, and ‘-’ implies that the model does not scale for that dataset due to GPU memory constraints.

Original: Empirical Performance:
Revised: Empirical Performance:

Original: As demonstrated in Table 2, CascadeXML, which leverages the multilayered transformer architecture to learn label resolution specific feature representations, is empirically superior to all existing XMC approaches whether using: (i) tf-idf features, or (ii) DNN-based dense features, or (iii) both.
Revised: As demonstrated in Table 2, CascadeXML, which leverages the multilayered transformer architecture to learn label resolution specific feature representations, is empirically superior to all existing XMC approaches whether using: (i) tf-idf features, or (ii) DNN-based dense features, or (iii) both.

Original: This is despite CascadeXML being a simple end-to-end trainable algorithm without using any bells or whistles like multi-stage procedures [54, 56], reclustering [56] or bootstrapped training [54, 56].
Revised: This is despite CascadeXML being a simple end-to-end trainable algorithm without using any bells or whistles like multi-stage procedures [54, 56], reclustering [56] or bootstrapped training [54, 56].

Original: For example, on Amazon-3M dataset, CascadeXML significantly outperforms all DNN-based approaches while taking only 24 hours to train on a single Nvidia A
Revised: For example, on Amazon-3M dataset, CascadeXML significantly outperforms all DNN-based approaches while taking only 24 hours to train on a single Nvidia A

Original: GPU.
Revised: GPU.

Original: In contrast, X-Transformer needs to ensemble 9 models and 23 days to train on 8 GPUs to reach the empirical performance (c.f. page 2, [56]), and LightXML remains unscalable.
Revised: In contrast, X-Transformer needs to ensemble 9 models and 23 days to train on 8 GPUs to reach the empirical performance (c.f. page 2, [56]), and LightXML remains unscalable.

Original: [Table]
Revised: [Table]

Original: We report results on Wiki10-31K and AmazonCat-13K in Table 3.
Revised: We report results on Wiki10-31K and AmazonCat-13K in Table 3.

Original: As a standard practice in the domain [6, 18, 56], we use 256 tokens as an input sequence length to the model after truncation.
Revised: As a standard practice in the domain [6, 18, 56], we use 256 tokens as an input sequence length to the model after truncation.

Original: We find this sequence length to be sufficient for competent empirical performance and hence, we do not witness any empirical benefits of leverage sparse tf-idf features for these datasets.
Revised: We find this sequence length to be sufficient for competent empirical performance and hence, we do not witness any empirical benefits of leverage sparse tf-idf features for these datasets.

Original: Notably,
Revised: Notably,

Original: CascadeXML achieves best results on AmazonCat-13K dataset and performs at par with
Revised: CascadeXML achieves best results on AmazonCat-13K dataset and performs at par with

Original: XR-Transformer without leveraging sparse tf-idf features.
Revised: XR-Transformer without leveraging sparse tf-idf features.

Original: Thus, CascadeXML’s unique multi-resolution approach is a favorable choice not only for datasets with extremely large output spaces, but also for datasets with tens of thousand labels.
Revised: Thus, CascadeXML’s unique multi-resolution approach is a favorable choice not only for datasets with extremely large output spaces, but also for datasets with tens of thousand labels.

Original: On the concatenated feature space, CascadeXML also significantly outperforms XR-Transformer on 8 out of 9 dataset-metric combination (ref: Table 2) with notable ∼ 6% improvement on Wiki-500K dataset for P@5 metric.
Revised: On the concatenated feature space, CascadeXML also significantly outperforms XR-Transformer on 8 out of 9 dataset-metric combination (ref: Table 2) with notable ∼ 6% improvement on Wiki-500K dataset for P@5 metric.

Original: As XR-Tranformer learns transformer features for label resolutions only up to 2 15 clusters [56, Table 6], incorporating tf-idf features via a computation-intensive XR-Linear [55] for final extreme classification task becomes an integral part of their pipeline.
Revised: As XR-Tranformer learns transformer features for label resolutions only up to 2 15 clusters [56, Table 6], incorporating tf-idf features via a computation-intensive XR-Linear [55] for final extreme classification task becomes an integral part of their pipeline.

Original: On the other hand, CascadeXML does not have any such inherent limitation, and even without tf-idf features, it outperforms XR-Transformer on 7 out of 9 dataset-metric combinations (ref: Table 2).
Revised: On the other hand, CascadeXML does not have any such inherent limitation, and even without tf-idf features, it outperforms XR-Transformer on 7 out of 9 dataset-metric combinations (ref: Table 2).

Original:  Computational cost & Ablation results
Revised:  Computational cost & Ablation results

Original: Single Model Comparison & Training Time Empirical results and training time of a single
Revised: Single Model Comparison & Training Time Empirical results and training time of a single

Original: CascadeXML model has been compared to single instance performance of state-of-the-art DNN based
Revised: CascadeXML model has been compared to single instance performance of state-of-the-art DNN based

Original: XMC approaches in Table 4.
Revised: XMC approaches in Table 4.

Original: Notably, CascadeXML, without leveraging any sparse tf-idf features, performs at par with XR-Transformer which benefits from using the extra statistical information about the entire (not truncated) data point.
Revised: Notably, CascadeXML, without leveraging any sparse tf-idf features, performs at par with XR-Transformer which benefits from using the extra statistical information about the entire (not truncated) data point.

Original: When comparing training time, CascadeXML is up to an order of magnitude faster than LightXML on a single GPU.
Revised: When comparing training time, CascadeXML is up to an order of magnitude faster than LightXML on a single GPU.

Original: In multi-GPU setting, CascadeXML trains in 7.
Revised: In multi-GPU setting, CascadeXML trains in 7.

Original: hours on Wiki-500K dataset using only 4 NVidia V100 GPUs as compared to XR-Transformer and
Revised: hours on Wiki-500K dataset using only 4 NVidia V100 GPUs as compared to XR-Transformer and

Original: AttentionXML which require 12.5 hours on 8 GPUs.
Revised: AttentionXML which require 12.5 hours on 8 GPUs.

Original: CascadeXML not only reduces the training time across datasets but also requires half the number of GPUs to do so.
Revised: CascadeXML not only reduces the training time across datasets but also requires half the number of GPUs to do so.

Original: This proves that CascadeXML is significantly more compute efficient as compared to previous DNN based XMC approaches.
Revised: This proves that CascadeXML is significantly more compute efficient as compared to previous DNN based XMC approaches.

Original: [Table]
Revised: [Table]

Original: Inference Time CascadeXML clocks the fastest inference speed as compared to all previous
Revised: Inference Time CascadeXML clocks the fastest inference speed as compared to all previous

Original: DNN-based XMC models, as shown in Table 5.
Revised: DNN-based XMC models, as shown in Table 5.

Original: Notably, CascadeXML is ∼ 1 .
Revised: Notably, CascadeXML is ∼ 1 .

Original: 5 × and ∼ 2 × faster at inference time compared to LightXML and XR-Transformer
Revised: 5 × and ∼ 2 × faster at inference time compared to LightXML and XR-Transformer

Original: [Table]
Revised: [Table]

Original: Performance on
Revised: Performance on

Original: Tail Labels Performance of CascadeXML on tail labels has been compared to baseline XMC methods in Table 6.
Revised: Tail Labels Performance of CascadeXML on tail labels has been compared to baseline XMC methods in Table 6.

Original: We note that for smaller datasets - Wiki10-31K and AmazonCat13K - PfastreXML [16] significantly outperforms all other methods on PSP metrics.
Revised: We note that for smaller datasets - Wiki10-31K and AmazonCat13K - PfastreXML [16] significantly outperforms all other methods on PSP metrics.

Original: This is to be expected, as this method specifically optimizes for tail labels.
Revised: This is to be expected, as this method specifically optimizes for tail labels.

Original: However, for larger datasetsAmazon-670K and Wiki-500K - CascadeXML outperforms the strong baselines like XR-Transformer and AttentionXML by 5-8% on PSP@3 and PSP@5 metrics while also significantly outperforming
Revised: However, for larger datasetsAmazon-670K and Wiki-500K - CascadeXML outperforms the strong baselines like XR-Transformer and AttentionXML by 5-8% on PSP@3 and PSP@5 metrics while also significantly outperforming

Original: PfastreXML over these datasets.
Revised: PfastreXML over these datasets.

Original: These results conclude that our unique end-to-end multi-resolution training approach is not only empirically superior to previous approaches in P@K metrics but also outperforms strong XMC baselines in performance over tail labels
Revised: These results conclude that our unique end-to-end multi-resolution training approach is not only empirically superior to previous approaches in P@K metrics but also outperforms strong XMC baselines in performance over tail labels

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 670K
Revised: 670K

Original: [Table]
Revised: [Table]

Original: w/o Rescaling
Revised: w/o Rescaling

Original: [Table]
Revised: [Table]

Original: Default
Revised: Default

Original: [Table]
Revised: [Table]

Original: Wiki-500K 2 15 Clusters 76.
Revised: Wiki-500K 2 15 Clusters 76.

Original: w/o Rescaling 76.
Revised: w/o Rescaling 76.

Original: Default 51.
Revised: Default 51.

Original: [Table]
Revised: [Table]

Original: Impact of label clusters size Earlier works have argued that using finegrained clusters leads to better model performance [18, 22, 32].
Revised: Impact of label clusters size Earlier works have argued that using finegrained clusters leads to better model performance [18, 22, 32].

Original: With increased number of label clusters at the penultimate level of HLT, the multi-resolution tasks tend to get more in-sync with each other [22] and hence enables transfer learning across different stages of learning pipelines [10, 32].
Revised: With increased number of label clusters at the penultimate level of HLT, the multi-resolution tasks tend to get more in-sync with each other [22] and hence enables transfer learning across different stages of learning pipelines [10, 32].

Original: Our mutliresolution training enables CascadeXML to use 2 16 label clusters for penultimate label resolution, double that of XRTransformer.
Revised: Our mutliresolution training enables CascadeXML to use 2 16 label clusters for penultimate label resolution, double that of XRTransformer.

Original: As Table 7 shows, increasing the resolution at the last shortlisting step results in non-trivial improvements, highlighting the need for architectures that can efficiently handle highresolution meta-labels.
Revised: As Table 7 shows, increasing the resolution at the last shortlisting step results in non-trivial improvements, highlighting the need for architectures that can efficiently handle highresolution meta-labels.

Original: Impact of re-scaling per-resolution loss For extreme datasets, we witness more efficient training by having a larger shortlist size for finer label resolutions (see appendix A.2.2).
Revised: Impact of re-scaling per-resolution loss For extreme datasets, we witness more efficient training by having a larger shortlist size for finer label resolutions (see appendix A.2.2).

Original: This results in varied loss scales across label resolutions leading to one resolution dominating the training.
Revised: This results in varied loss scales across label resolutions leading to one resolution dominating the training.

Original: Comparing the “Default” and “w/o Rescaling” rows for respective datasets in Table 7, indicates that rescaling as introduced in section 3 helps CascadeXML train more effectively.
Revised: Comparing the “Default” and “w/o Rescaling” rows for respective datasets in Table 7, indicates that rescaling as introduced in section 3 helps CascadeXML train more effectively.

Original:  Discussion
Revised:  Discussion

Original: As shown above, CascadeXML substantially outperforms current state-of-the-art approaches both in terms of computational efficiency, and prediction performance.
Revised: As shown above, CascadeXML substantially outperforms current state-of-the-art approaches both in terms of computational efficiency, and prediction performance.

Original: It achieves this while being simpler than XR-Transformer and more scalable than LightXML.
Revised: It achieves this while being simpler than XR-Transformer and more scalable than LightXML.

Original: Revisiting Table 1, the extraction of intermediate [CLS] representations enables us to perform end-to-end multi-resolution training with resolution-specific attention maps from a single transformer model.
Revised: Revisiting Table 1, the extraction of intermediate [CLS] representations enables us to perform end-to-end multi-resolution training with resolution-specific attention maps from a single transformer model.

Original: From Table 2, we show improvements over both LightXML and XR-Transformer, which are transformer-based models sharing feature representations across multiple label resolutions.
Revised: From Table 2, we show improvements over both LightXML and XR-Transformer, which are transformer-based models sharing feature representations across multiple label resolutions.

Original: We attribute this increased performance to
Revised: We attribute this increased performance to

Original: CascadeXML’s end-to-end multi-resolution training pipeline and property of adapted feature embed- dings.
Revised: CascadeXML’s end-to-end multi-resolution training pipeline and property of adapted feature embed- dings.

Original: The presented architecture, therefore, combines the strength of different previous approaches, without inheriting their limitations.
Revised: The presented architecture, therefore, combines the strength of different previous approaches, without inheriting their limitations.

Original: We postulated that the (meta-)classification tasks at different resolutions need to attend to different tokens in the input sequence, a property lacking in recent approaches [8, 18, 56].
Revised: We postulated that the (meta-)classification tasks at different resolutions need to attend to different tokens in the input sequence, a property lacking in recent approaches [8, 18, 56].

Original: This intuition is corroborated by the fact that the attention maps learned by CascadeXML differ significantly between different resolutions.
Revised: This intuition is corroborated by the fact that the attention maps learned by CascadeXML differ significantly between different resolutions.

Original: We provide graphs and supporting data in Appendix B.
Revised: We provide graphs and supporting data in Appendix B.

Original: Obtaining label resolution specific feature embeddings in a pre-trained transformer model requires to make the compromise of extracting these at earlier transformer layers, where the token representations are not yet as refined as in the final layer [48].
Revised: Obtaining label resolution specific feature embeddings in a pre-trained transformer model requires to make the compromise of extracting these at earlier transformer layers, where the token representations are not yet as refined as in the final layer [48].

Original: In contrast to the extreme classification task, which requires placing true labels exactly at the top-5 positions, the shortlisting task only requires all true meta-labels be recalled within the shortlist, which is much longer than five elements.
Revised: In contrast to the extreme classification task, which requires placing true labels exactly at the top-5 positions, the shortlisting task only requires all true meta-labels be recalled within the shortlist, which is much longer than five elements.

Original: We observe that the meta-classifiers are able to achieve high-recall (as shown Appendix B) at the shortlist length even with the less refined features.
Revised: We observe that the meta-classifiers are able to achieve high-recall (as shown Appendix B) at the shortlist length even with the less refined features.

Original: CascadeXML is, therefore, successful in efficiently leveraging intermediate transformer representations for the task of label shortlisting while keeping feature representations separate for each label resolution.
Revised: CascadeXML is, therefore, successful in efficiently leveraging intermediate transformer representations for the task of label shortlisting while keeping feature representations separate for each label resolution.

Original: The structure of the model means that earlier layers in the transformer receive gradients from multiple resolutions, whereas the final layers’ back propagation is limited to gradients from the extreme resolution (cf.
Revised: The structure of the model means that earlier layers in the transformer receive gradients from multiple resolutions, whereas the final layers’ back propagation is limited to gradients from the extreme resolution (cf.

Original: Figure 1).
Revised: Figure 1).

Original: This setup ensures that a portion of the representational capabilities is exclusively available for the extreme task and hence, the final transformer layers are increasingly more suitable for classification at the extreme resolution.
Revised: This setup ensures that a portion of the representational capabilities is exclusively available for the extreme task and hence, the final transformer layers are increasingly more suitable for classification at the extreme resolution.

Original: At the same time, since the gradients from the finer tasks are also relevant for the coarser meta-label shortlisting task, so additional downstream task does not hurt the shortlisting performance at the intermediate transformer layers.
Revised: At the same time, since the gradients from the finer tasks are also relevant for the coarser meta-label shortlisting task, so additional downstream task does not hurt the shortlisting performance at the intermediate transformer layers.

Original:  Other Related Works
Revised:  Other Related Works

Original: The focus of a majority of initial works in XMC has been on using fixed features for learning the classifier with one of the following class of methods : (i) one-vs-rest [53, 2, 52, 3, 41], (ii) label tree based [35, 21, 47, 17], (iii) decision tree methods [37, 44, 29], and (iv) label embedding based [5, 45, 13] classifiers.
Revised: The focus of a majority of initial works in XMC has been on using fixed features for learning the classifier with one of the following class of methods : (i) one-vs-rest [53, 2, 52, 3, 41], (ii) label tree based [35, 21, 47, 17], (iii) decision tree methods [37, 44, 29], and (iv) label embedding based [5, 45, 13] classifiers.

Original: Beyond the above algorithmic categorization of approaches, computational considerations on scaling the training process via negative sampling [15], and sketching [31] have also been studied.
Revised: Beyond the above algorithmic categorization of approaches, computational considerations on scaling the training process via negative sampling [15], and sketching [31] have also been studied.

Original: Furthermore, the statistical consequences of negative sampling [39] and missing labels [16, 38, 43, 42] have led to novel insights in designing unbiased loss functions.
Revised: Furthermore, the statistical consequences of negative sampling [39] and missing labels [16, 38, 43, 42] have led to novel insights in designing unbiased loss functions.

Original: With advances in deep learning, joint learning of features and classifier has been the focus of most recent approaches.
Revised: With advances in deep learning, joint learning of features and classifier has been the focus of most recent approaches.

Original: Building on the first steps for developing convolutional neural networks for text classification [23], deep extreme classification was introduced in XML-CNN [26].
Revised: Building on the first steps for developing convolutional neural networks for text classification [23], deep extreme classification was introduced in XML-CNN [26].

Original: This work preceded the recent developments on employing BiLSTM and transformer encoders discussed in detail the
Revised: This work preceded the recent developments on employing BiLSTM and transformer encoders discussed in detail the

Original: Introduction section.
Revised: Introduction section.

Original: There has been a recent surge in works to tackle XMC for instances with short text inputs and those in which labels are endowed with textual descriptions [10, 9, 32, 33, 40].
Revised: There has been a recent surge in works to tackle XMC for instances with short text inputs and those in which labels are endowed with textual descriptions [10, 9, 32, 33, 40].

Original: Lately, the XMC setting has also been extended to predict unseen labels in zero-shot learning [14, 57, 49].
Revised: Lately, the XMC setting has also been extended to predict unseen labels in zero-shot learning [14, 57, 49].

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduced a novel paradigm for fine-tuning transformer models in XMC settings.
Revised: In this paper, we introduced a novel paradigm for fine-tuning transformer models in XMC settings.

Original: In contrast to the existing methods, which consider transformer as a blackbox encoder, we leverage its multi-layered architecture to learn data representation corresponding to different resolutions in the
Revised: In contrast to the existing methods, which consider transformer as a blackbox encoder, we leverage its multi-layered architecture to learn data representation corresponding to different resolutions in the

Original: HLT as well as fine-grained labels at the level of the extreme classifier.
Revised: HLT as well as fine-grained labels at the level of the extreme classifier.

Original: The proposed instantiation of our framework in the form of CascadeXML not only yields state-of-the-art prediction performance, but is also end-to-end trainable (without intermediate reclustering steps), simpler to implement, and fast on training and inference.
Revised: The proposed instantiation of our framework in the form of CascadeXML not only yields state-of-the-art prediction performance, but is also end-to-end trainable (without intermediate reclustering steps), simpler to implement, and fast on training and inference.

Original: Beyond further research in extreme classification towards fully harnessing the representation capabilities of transformer encoders, we believe our approach can inspire future multi-resolution architectures in other domains as well which leverage label hierarchy.
Revised: Beyond further research in extreme classification towards fully harnessing the representation capabilities of transformer encoders, we believe our approach can inspire future multi-resolution architectures in other domains as well which leverage label hierarchy.

Original:  Acknowledgments
Revised:  Acknowledgments

Original: The authors would like to thank Devaansh Gupta and Mohadmmadreza Qaraei for useful discussions.
Revised: The authors would like to thank Devaansh Gupta and Mohadmmadreza Qaraei for useful discussions.

Original: They also acknowledge the support of CSC – IT Center for Science, Finland, as well as the Aalto
Revised: They also acknowledge the support of CSC – IT Center for Science, Finland, as well as the Aalto

Original: Science-IT project, for providing the required computational resources.
Revised: Science-IT project, for providing the required computational resources.

Original: This research is supported in part by Academy of Finland grants : Decision No. 348215 and 347707.
Revised: This research is supported in part by Academy of Finland grants : Decision No. 348215 and 347707.

Original: A Appendix A.1 Dataset Details & Evaluation Metrics
Revised: A Appendix A.1 Dataset Details & Evaluation Metrics

Original: As stated earlier, the main application of Extreme Multi-label Text Classification is in e-commerce - product recommendation and dynamic search advertisement - and in document tagging, where the objective of an algorithm is to correctly recommend/advertise among the top-k slots.
Revised: As stated earlier, the main application of Extreme Multi-label Text Classification is in e-commerce - product recommendation and dynamic search advertisement - and in document tagging, where the objective of an algorithm is to correctly recommend/advertise among the top-k slots.

Original: Thus, for evaluation of the methods, we use precision at k (denoted by P @ k ), and its propensity scored variant (denoted by PSP @ k )
Revised: Thus, for evaluation of the methods, we use precision at k (denoted by P @ k ), and its propensity scored variant (denoted by PSP @ k )

Original: These are standard and widely used metrics by the XMC community [4].
Revised: These are standard and widely used metrics by the XMC community [4].

Original: For each test sample with observed ground truth label vector y ∈ { 0 , 1 } L and predicted vector ˆ y ∈ R L , P @ k is given by :
Revised: For each test sample with observed ground truth label vector y ∈ { 0 , 1 } L and predicted vector ˆ y ∈ R L , P @ k is given by :

Original: [Equation]
Revised: [Equation]

Original: where top@ k (ˆ y ) returns the k largest indices of ˆ y .
Revised: where top@ k (ˆ y ) returns the k largest indices of ˆ y .

Original: Since P@ k treats all the labels equally, it doesn’t reveal the performance of the model on tail labels.
Revised: Since P@ k treats all the labels equally, it doesn’t reveal the performance of the model on tail labels.

Original: However, because of the long-tailed distribution in XMC datasets, one of the main challenges is to predict tail labels correctly, which may be more valuable and informative compared to head classes.
Revised: However, because of the long-tailed distribution in XMC datasets, one of the main challenges is to predict tail labels correctly, which may be more valuable and informative compared to head classes.

Original: A set of metrics that have been established in XMC to evaluate tail performance are propensity-scored version of precision.
Revised: A set of metrics that have been established in XMC to evaluate tail performance are propensity-scored version of precision.

Original: These PSP@ k were introduced in [ ? ], and use a weighting factor based on a propensity score p ℓ to give more weight to tail labels:
Revised: These PSP@ k were introduced in [16], and use a weighting factor based on a propensity score p ℓ to give more weight to tail labels:

Original: [Equation]
Revised: [Equation]

Original: We use the empirical values for p ℓ as proposed in [ ? ].
Revised: We use the empirical values for p ℓ as proposed in [16].

Original: [Table]
Revised: [Table]

Original: A.2 Model Details
Revised: A.2 Model Details

Original: [Table]
Revised: [Table]

Original: Transformer layers put in brackets next to the label resolution imply that a concatenation of the [CLS] token of the respective layers has been used for label shortlisting at that resolution.
Revised: Transformer layers put in brackets next to the label resolution imply that a concatenation of the [CLS] token of the respective layers has been used for label shortlisting at that resolution.

Original: A.2.1 Model Hyperparameters
Revised: A.2.1 Model Hyperparameters

Original: CascadeXML optimizes the training objective using Binary Cross Entropy loss as the loss function and AdamW [28] as the optimizer.
Revised: CascadeXML optimizes the training objective using Binary Cross Entropy loss as the loss function and AdamW [28] as the optimizer.

Original: We use different learning rates for the transformer encoder and the (meta-)label weight vectors as we need to train the weight vectors from random initialization in contrast to fine-tuning the transformer encoder.
Revised: We use different learning rates for the transformer encoder and the (meta-)label weight vectors as we need to train the weight vectors from random initialization in contrast to fine-tuning the transformer encoder.

Original: Specifically, the transformer encoder is fine-tuned at a learning rate of 10 − 4 , while the weight vectors are trained at a learning rate of 10 − 3 .
Revised: Specifically, the transformer encoder is fine-tuned at a learning rate of 10 − 4 , while the weight vectors are trained at a learning rate of 10 − 3 .

Original: The learning rate schedule consists of a constant learning rate for most of the iterations, with a cosine warm-up at the beginning and cosine annealing towards the end of the schedule.
Revised: The learning rate schedule consists of a constant learning rate for most of the iterations, with a cosine warm-up at the beginning and cosine annealing towards the end of the schedule.

Original: In multi-GPU setting, we use a batch size of 64 per GPU (256 total) across 4 GPUs.
Revised: In multi-GPU setting, we use a batch size of 64 per GPU (256 total) across 4 GPUs.

Original: In single GPU setting, we still use a batch size of 256 by accumulating gradients for 4 iterations.
Revised: In single GPU setting, we still use a batch size of 256 by accumulating gradients for 4 iterations.

Original: The hyperparameter settings in detail have been mentioned in Table 9.
Revised: The hyperparameter settings in detail have been mentioned in Table 9.

Original: A.2.2 Ensemble Training Time
Revised: A.2.2 Ensemble Training Time

Original: As shown in Table 10, CascadeXML achieves the lowest training time across all datasets except
Revised: As shown in Table 10, CascadeXML achieves the lowest training time across all datasets except

Original: Amazon-3M using only 4 GPUs as compared to XR-Transformer which leverages 8 GPUs.
Revised: Amazon-3M using only 4 GPUs as compared to XR-Transformer which leverages 8 GPUs.

Original: Note that on Amazon-3M, XR-Transformer achieves a slightly lower training time.
Revised: Note that on Amazon-3M, XR-Transformer achieves a slightly lower training time.

Original: However, XR-Transformer using 2x the number of GPUs and does not train the DNN model on full 3 million label resolution.
Revised: However, XR-Transformer uses 2 × the number of GPUs and does not train the DNN model on full 3 million label resolution.

Original: XR-Transformer trains the DNN model using a classification training object comprising of only 2label clusters [56, Table 7] and then leverages XR-Linear [55], a linear solver, to scale up to 3M labels.
Revised: XR-Transformer trains the DNN model using a classification training object comprising of only 2label clusters [56, Table 7] and then leverages XR-Linear [55], a linear solver, to scale up to 3M labels.

Original: On the other hand, CascadeXML trains an ensemble of 3 models to full resolution of 3 million labels in 30 hours using only 4 GPUs.
Revised: On the other hand, CascadeXML trains an ensemble of 3 models to full resolution of 3 million labels in 30 hours using only 4 GPUs.

Original: [Table]
Revised: [Table]

Original: A.3 Leveraging Sparse Features
Revised: A.3 Leveraging Sparse Features

Original: As we are using BERT for the transformer backbone of our method, we have to truncate the input sequences to a limited number of tokens (see Table 9).
Revised: As we are using BERT for the transformer backbone of our method, we have to truncate the input sequences to a limited number of tokens (see Table 9).

Original: This truncation results in loss of information.
Revised: This truncation results in loss of information.

Original: Thus, following XR-Transformer’s lead we combine the features trained by CasadeXML with statistical information in the form of sparse tf-idf representation of the full input in an additional OVA classifier.
Revised: Thus, following XR-Transformer’s lead we combine the features trained by CasadeXML with statistical information in the form of sparse tf-idf representation of the full input in an additional OVA classifier.

Original: The concatenated features are constructed as [18]:
Revised: The concatenated features are constructed as [18]:

Original: [Equation]
Revised: [Equation]

Original: We use a version of DiSMEC [2] instead of using XR-Linear [55] - as done in XR-Transformer - as our external OVA classifier for Φ cat .
Revised: We use a version of DiSMEC [2] instead of using XR-Linear [55] - as done in XR-Transformer - as our external OVA classifier for Φ cat .

Original: Even though XR-Linear achieves slightly better performance than
Revised: Even though XR-Linear achieves slightly better performance than

Original: DiSMEC across datasets (Table: 1), we find DiSMEC to be more resource efficient than XR-Linear.
Revised: DiSMEC across datasets (Table: 1), we find DiSMEC to be more resource efficient than XR-Linear.

Original: To quantify, DiSMEC runs successfully on 116GB RAM for all datasets, while XR-Linear requires close to 470GB RAM for Amazon-3M. Next, we discuss the application of DiSMEC over Φ cat .
Revised: To quantify, DiSMEC runs successfully on 116GB RAM for all datasets, while XR-Linear requires close to 470GB RAM for Amazon-3M. Next, we discuss the application of DiSMEC over Φ cat .

Original: DiSMEC DiSMEC is a linear multilabel classifier that minimizes an L 2 -regularized squared hingeloss, followed by a pruning step to only keep the most important weights.
Revised: DiSMEC DiSMEC is a linear multilabel classifier that minimizes an L 2 -regularized squared hingeloss, followed by a pruning step to only keep the most important weights.

Original: Thus, the loss for a given weight matrix W =
Revised: Thus, the loss for a given weight matrix W =

Original: [ w 1 , . .
Revised: [ w 1 , . .

Original: . , w L ] is given by
Revised: . , w L ] is given by

Original: [Equation]
Revised: [Equation]

Original: Crucially, from the point of view of the OVA classifiers, the input features Φ cat ( x i ) are constant.
Revised: Crucially, from the point of view of the OVA classifiers, the input features Φ cat ( x i ) are constant.

Original: This means that the task decomposes into independent optimization problems for each label, minimizing
Revised: This means that the task decomposes into independent optimization problems for each label, minimizing

Original: [Equation]
Revised: [Equation]

Original: This allows for trivial parallelization of the task across CPU cores, and also means that the weights w j can be pruned as soon as the sub-problem is solved.
Revised: This allows for trivial parallelization of the task across CPU cores, and also means that the weights w j can be pruned as soon as the sub-problem is solved.

Original: Consequently, there is no need to ever store the entire weight matrix, improving memory efficiency.
Revised: Consequently, there is no need to ever store the entire weight matrix, improving memory efficiency.

Original: The objective function (7) has a continuous derivative, and its Hessian is well defined everywhere except exactly at the decision boundary.
Revised: The objective function (7) has a continuous derivative, and its Hessian is well defined everywhere except exactly at the decision boundary.

Original: Consequently, it can be minimized using a second-order
Revised: Consequently, it can be minimized using a second-order

Original: Newton method.
Revised: Newton method.

Original: A discussion of this in the context of linear classification can be found in [12].
Revised: A discussion of this in the context of linear classification can be found in [12].

Original: B Visualizations and Analysis
Revised: B Visualizations and Analysis

Original: In this section we provide visualizations and additional data that corroborate our interpretation that different attention- and feature maps are needed for classification at different granularities of the label tree.
Revised: In this section we provide visualizations and additional data that corroborate our interpretation that different attention- and feature maps are needed for classification at different granularities of the label tree.

Original: In Figure 2, the attention of the [CLS] token to itself in the previous layer is visualized.
Revised: In Figure 2, the attention of the [CLS] token to itself in the previous layer is visualized.

Original: If this self-attention were large, then Φ ( t ) CLS would be mostly a function of Φ ( t − 1) CLS .
Revised: If this self-attention were large, then Φ ( t ) CLS would be mostly a function of Φ ( t − 1) CLS .

Original: In such a case, Φ ( t − 1) CLS would contain less information than Φ ( t ) CLS (data-processing inequality), but the meta-labels
Revised: In such a case, Φ ( t − 1) CLS would contain less information than Φ ( t ) CLS (data-processing inequality), but the meta-labels

Original: R ( t − 1) ( y ) at level t − 1 contain strictly less information than R ( t ) ( y ) .
Revised: R ( t − 1) ( y ) at level t − 1 contain strictly less information than R ( t ) ( y ) .

Original: Thus, is the [CLS] token embedding had strong feed-forward characteristics, Φ ( t − 1) CLS would have to contain all the information also about the extreme-level labels, and thus have limited representation capacity for the level-t task.
Revised: Thus, is the [CLS] token embedding had strong feed-forward characteristics, Φ ( t − 1) CLS would have to contain all the information also about the extreme-level labels, and thus have limited representation capacity for the level-t task.

Original: Layer
Revised: Layer

Original: S e l fA tt e n ti on
Revised: S e l fA tt e n ti on

Original: Pretrained Bert
Revised: Pretrained Bert

Original: CascadeXML
Revised: CascadeXML

Original: Layer
Revised: Layer

Original: [Figure]
Revised: [Figure]

Original: n ti on
Revised: n ti on

Original: Pretrained Bert
Revised: Pretrained Bert

Original: CascadeXML the later layers much more strongly in
Revised: CascadeXML the later layers much more strongly in

Original: CascadeXML than in pretrained BERT.
Revised: CascadeXML than in pretrained BERT.

Original: Luckily, Figure 2 is a sanity-check that shows that this is not the case.
Revised: Luckily, Figure 2 is a sanity-check that shows that this is not the case.

Original: Starting from layer 6 , where the first meta-task is placed, the [CLS] token is almost entirely re-assembled at each layer from the embeddings of the other tokens – much more strongly than in a pretrained BERT.
Revised: Starting from layer 6 , where the first meta-task is placed, the [CLS] token is almost entirely re-assembled at each layer from the embeddings of the other tokens – much more strongly than in a pretrained BERT.

Original: This allows each layer to extract the information best suited for classification at the given hierarchy level.
Revised: This allows each layer to extract the information best suited for classification at the given hierarchy level.

Original: We can also detect some qualitative differences in the attention maps at different resolutions: The entropy, i.e. how much the attention is concentrated or spread across different tokens, changes significantly between levels.
Revised: We can also detect some qualitative differences in the attention maps at different resolutions: The entropy, i.e. how much the attention is concentrated or spread across different tokens, changes significantly between levels.

Original: This is not an artifact of the pretrained BERT model, but appears to be learned during fine-tuning.
Revised: This is not an artifact of the pretrained BERT model, but appears to be learned during fine-tuning.

Original: In Figure 4, we analyse the flow of information and visualize how much processing is happening in a given layer with respect to the [CLS] token.
Revised: In Figure 4, we analyse the flow of information and visualize how much processing is happening in a given layer with respect to the [CLS] token.

Original: We rely on projection weighted canonical correlation analysis [34] for this task.
Revised: We rely on projection weighted canonical correlation analysis [34] for this task.

Original: This allows to compare the representations at different layers in a way that is invariant to any linear transformations.
Revised: This allows to compare the representations at different layers in a way that is invariant to any linear transformations.

Original: We primarily make two observations from Figure 4.
Revised: We primarily make two observations from Figure 4.

Original: First, we observe that [CLS] token embeddings of layers 6, 8, 10 and 12 are more closely related in CascadeXML than in LightXML.
Revised: First, we observe that [CLS] token embeddings of layers 6, 8, 10 and 12 are more closely related in CascadeXML than in LightXML.

Original: This is expected as LigthXML only trains the (meta-)label weight vectors using the [CLS] token embeddings of the bottom layers.
Revised: This is expected as LigthXML only trains the (meta-)label weight vectors using the [CLS] token embeddings of the bottom layers.

Original: Because the multi-resolution training objectives differ only in granularity, many
Revised: Because the multi-resolution training objectives differ only in granularity, many

Original: Layer
Revised: Layer

Original: E n t r opy
Revised: E n t r opy

Original: Pretrained
Revised: Pretrained

Original: CascadeXML
Revised: CascadeXML

Original: [Figure]
Revised: [Figure]

Original: Layers
Revised: Layers

Original: [Figure]
Revised: [Figure]

Original: BERT
Revised: BERT

Original: [Figure]
Revised: [Figure]

Original: features required to distinguish coarse meta-labels are useful in determining finer meta-labels as well.
Revised: features required to distinguish coarse meta-labels are useful in determining finer meta-labels as well.

Original: Hence, the similarity between consecutive representations is expected to be strong.
Revised: Hence, the similarity between consecutive representations is expected to be strong.

Original: On the other hand, when only looking at CascadeXML’s points (in blue) in Figure 4, we observe that the tasks in the first meta-classifier and the extreme classifier are substantially different.
Revised: On the other hand, when only looking at CascadeXML’s points (in blue) in Figure 4, we observe that the tasks in the first meta-classifier and the extreme classifier are substantially different.

Original: This implies both training objectives require different representations that cannot be provided by a LightXML-/XRTransformer-style model which use same attention maps (and hence, same [CLS] token embeddings) for all resolutions.
Revised: This implies both training objectives require different representations that cannot be provided by a LightXML-/XRTransformer-style model which use same attention maps (and hence, same [CLS] token embeddings) for all resolutions.

Original: Dataset
Revised: Dataset

Original: Level
Revised: Level

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The ability to use earlier layers’
Revised: The ability to use earlier layers’

Original: [CLS] token representation for the meta-task crucially depends on the fact that these representations are still sufficient for achieving high recall in the shortlisting task.
Revised: [CLS] token representation for the meta-task crucially depends on the fact that these representations are still sufficient for achieving high recall in the shortlisting task.

Original: As shown in Table 11, the shortlisting achieves very good recall rates.
Revised: As shown in Table 11, the shortlisting achieves very good recall rates.

Original: In particular, the very first shortlist, with the “weakest” features, achieves almost perfect recall.
Revised: In particular, the very first shortlist, with the “weakest” features, achieves almost perfect recall.