Original: D EEP XML: S CALABLE & A CCURATE D EEP E XTREME C LASSIFICATION FOR M ATCHING U SER Q UERIES TO A DVERTISER B ID P HRASES
Revised: D EEP XML: S CALABLE & A CCURATE D EEP E XTREME C LASSIFICATION FOR M ATCHING U SER Q UERIES TO A DVERTISER B ID P HRASES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The objective in deep extreme multi-label learning is to jointly learn feature representations and classiﬁers to automatically tag data points with the most relevant subset of labels from an extremely large label set.
Revised: The objective in deep extreme multi-label learning is to jointly learn feature representations and classiﬁers to automatically tag data points with the most relevant subset of labels from an extremely large label set.

Original: Unfortunately, state-of-theart deep extreme classiﬁers are either not scalable or inaccurate for short text documents.
Revised: Unfortunately, state-of-theart deep extreme classiﬁers are either not scalable or inaccurate for short text documents.

Original: This paper develops the DeepXML algorithm which addresses both limitations by introducing a novel architecture that splits training of head and tail labels.
Revised: This paper develops the DeepXML algorithm which addresses both limitations by introducing a novel architecture that splits training of head and tail labels.

Original: DeepXML increases accuracy by (a) learning word embeddings on head labels and transferring them through a novel residual connection to data impoverished tail labels; (b) increasing the amount of negative training data available by extending state-of-the-art negative sub-sampling techniques; and (c) re-ranking the set of predicted labels to eliminate the hardest negatives for the original classiﬁer.
Revised: DeepXML increases accuracy by (a) learning word embeddings on head labels and transferring them through a novel residual connection to data impoverished tail labels; (b) increasing the amount of negative training data available by extending state-of-the-art negative sub-sampling techniques; and (c) re-ranking the set of predicted labels to eliminate the hardest negatives for the original classiﬁer.

Original: All of these contributions are implemented efﬁciently by extending the highly scalable Slice algorithm for pretrained embeddings to learn the proposed
Revised: All of these contributions are implemented efﬁciently by extending the highly scalable Slice algorithm for pretrained embeddings to learn the proposed

Original: DeepXML architecture.
Revised: DeepXML architecture.

Original: As a result, DeepXML could efﬁciently scale to problems involving millions of labels that were beyond the pale of state-of-the-art deep extreme classiﬁers as it could be more than 10x faster at training than XML-CNN and AttentionXML.
Revised: As a result, DeepXML could efﬁciently scale to problems involving millions of labels that were beyond the pale of state-of-the-art deep extreme classiﬁers as it could be more than 10x faster at training than XML-CNN and AttentionXML.

Original: At the same time, DeepXML was also empirically determined to be up to 19% more accurate than leading techniques for matching search engine queries to advertiser bid phrases.
Revised: At the same time, DeepXML was also empirically determined to be up to 19% more accurate than leading techniques for matching search engine queries to advertiser bid phrases.

Original: Source code for DeepXML can be downloaded from (Anonymous, 2019).
Revised: Source code for DeepXML can be downloaded from (Anonymous, 2019).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Objective : This paper develops the DeepXML algorithm for deep extreme multi-label learning applied to short text documents such as web search engine queries.
Revised: Objective : This paper develops the DeepXML algorithm for deep extreme multi-label learning applied to short text documents such as web search engine queries.

Original: DeepXML is demonstrated to be signiﬁcantly more accurate and an order of magnitude faster to train than state-of-the-art deep extreme classiﬁers XML-CNN
Revised: DeepXML is demonstrated to be signiﬁcantly more accurate and an order of magnitude faster to train than state-of-the-art deep extreme classiﬁers XML-CNN

Original: (Liu et al., 2017) and AttentionXML (You et al., 2018).
Revised: (Liu et al., 2017) and AttentionXML (You et al., 2018).

Original: As a result,
Revised: As a result,

Original: DeepXML could efﬁciently train on problems involving millions of labels on a single GPU that were beyond the scaling capabilities of leading deep extreme classiﬁers.
Revised: DeepXML could efﬁciently train on problems involving millions of labels on a single GPU that were beyond the scaling capabilities of leading deep extreme classiﬁers.

Original: This allowed DeepXML to be applied to the problem of matching millions of advertiser bid phrases to a user’s query on a popular web search engine where it was found to increase prediction accuracy by more than 19 percentage points as compared to the leading techniques currently in production.
Revised: This allowed DeepXML to be applied to the problem of matching millions of advertiser bid phrases to a user’s query on a popular web search engine where it was found to increase prediction accuracy by more than 19 percentage points as compared to the leading techniques currently in production.

Original: Deep extreme multi-label learning : The objective in deep extreme multi-label learning is to learn feature representations and classiﬁers to automatically tag data points with the most relevant subset of labels from an extremely large label set.
Revised: Deep extreme multi-label learning : The objective in deep extreme multi-label learning is to learn feature representations and classiﬁers to automatically tag data points with the most relevant subset of labels from an extremely large label set.

Original: Note that multi-label learning is a generalization of multi-class classiﬁcation which aims to predict a single mutually exclusive label.
Revised: Note that multi-label learning is a generalization of multi-class classiﬁcation which aims to predict a single mutually exclusive label.

Original: Notation : Throughout the paper: N refers to number of training points, d refers to representation dimension, and L refers to number of labels.
Revised: Notation : Throughout the paper: N refers to number of training points, d refers to representation dimension, and L refers to number of labels.

Original: Additionaly, Y refers to the label matrix where y ij =if j th label is relevant to i th instance, and 0 otherwise.
Revised: Additionaly, Y refers to the label matrix where y ij =if j th label is relevant to i th instance, and 0 otherwise.

Original: Please note that differences in accuracies are reported in absolute percentage points unless stated otherwise.
Revised: Please note that differences in accuracies are reported in absolute percentage points unless stated otherwise.

Original: Matching queries to bid phrases : Web search engines allow ads to be served for not just queries bidded on directly by advertisers, referred to as bid phrases, but also for related queries with matching intent.
Revised: Matching queries to bid phrases : Web search engines allow ads to be served for not just queries bidded on directly by advertisers, referred to as bid phrases, but also for related queries with matching intent.

Original: Thus matching a query that was just entered by the user to the relevant subset of millions of advertiser bid phrases in milliseconds is an important research application which forms the focus of this paper.
Revised: Thus matching a query that was just entered by the user to the relevant subset of millions of advertiser bid phrases in milliseconds is an important research application which forms the focus of this paper.

Original: DeepXML reformulates this problem as an extreme multi-label learning task by treating each of the top 3 Million monetizable advertiser bid phrases as a separate label and learning a deep classiﬁer to predict the relevant subset of bid phrases given an input query.
Revised: DeepXML reformulates this problem as an extreme multi-label learning task by treating each of the top 3 Million monetizable advertiser bid phrases as a separate label and learning a deep classiﬁer to predict the relevant subset of bid phrases given an input query.

Original: For example, given the user query "what is diabetes type 2" as input, DeepXML predicts that ads corresponding to the bid phrases "what is type 2 diabetes mellitus", "diabetes type 2 deﬁnition", "do i have type 2 diabetes", etc . could be relevant to the user.
Revised: For example, given the user query "what is diabetes type 2" as input, DeepXML predicts that ads corresponding to the bid phrases "what is type 2 diabetes mellitus", "diabetes type 2 deﬁnition", "do i have type 2 diabetes", etc . could be relevant to the user.

Original: Note that other high-impact applications have also been reformulated as the extreme classiﬁcation of short text documents such as queries, webpage titles, etc .
Revised: Note that other high-impact applications have also been reformulated as the extreme classiﬁcation of short text documents such as queries, webpage titles, etc .

Original: For instance, (Jain et al., 2019) applied extreme multi-label learning to recommend the subset of relevant Bing queries that could be asked by a user instead of the original query.
Revised: For instance, (Jain et al., 2019) applied extreme multi-label learning to recommend the subset of relevant Bing queries that could be asked by a user instead of the original query.

Original: Similarly, extreme multi-label learning could be used to predict which subset of search engine queries might lead to a click on a webpage from its title alone for scenarios where the webpage content might not be available due to privacy concerns, latency issues in fetching the webpage, etc .
Revised: Similarly, extreme multi-label learning could be used to predict which subset of search engine queries might lead to a click on a webpage from its title alone for scenarios where the webpage content might not be available due to privacy concerns, latency issues in fetching the webpage, etc .

Original: State-of-the-art extreme classiﬁers : Unfortunately, state-of-the-art extreme classiﬁers are either not scalable or inaccurate for queries and other short text documents.
Revised: State-of-the-art extreme classiﬁers : Unfortunately, state-of-the-art extreme classiﬁers are either not scalable or inaccurate for queries and other short text documents.

Original: In particular, leading extreme classiﬁers based on bag-of-words (BoW) features (Prabhu et al., 2018b) and pretrained embeddings (Jain et al., 2019) are highly scalable but inaccurate for documents having only 3 or 4 words.
Revised: In particular, leading extreme classiﬁers based on bag-of-words (BoW) features (Prabhu et al., 2018b) and pretrained embeddings (Jain et al., 2019) are highly scalable but inaccurate for documents having only 3 or 4 words.

Original: While feature engineering (Arora, 2017; Joulin et al., 2017; Wieting & Kiela, 2019), including taking sub-word tokens, bigram tokens, etc can ameliorate the problem somewhat, their accuracy still lags that of deep learning methods which learn features speciﬁc to the task at hand.
Revised: While feature engineering (Arora, 2017; Joulin et al., 2017; Wieting & Kiela, 2019), including taking sub-word tokens, bigram tokens, etc can ameliorate the problem somewhat, their accuracy still lags that of deep learning methods which learn features speciﬁc to the task at hand.

Original: However, such methods, as exempliﬁed by the state-of-the-art XML-CNN (Liu et al., 2017) and AttentionXML (You et al., 2018), can have prohibitive training costs and have not been shown to scale beyond a million labels on a single GPU.
Revised: However, such methods, as exempliﬁed by the state-of-the-art XML-CNN (Liu et al., 2017) and AttentionXML (You et al., 2018), can have prohibitive training costs and have not been shown to scale beyond a million labels on a single GPU.

Original: At the same time, there is a lot of scope for improving accuracy as XML-CNN and
Revised: At the same time, there is a lot of scope for improving accuracy as XML-CNN and

Original: AttentionXML’s architectures have not been specialized for short text documents.
Revised: AttentionXML’s architectures have not been specialized for short text documents.

Original: Tail labels : It is worth noting that all the computational and statistical complexity in extreme classiﬁcation arises due to the presence of millions of tail labels each having just a few, often a single, training point.
Revised: Tail labels : It is worth noting that all the computational and statistical complexity in extreme classiﬁcation arises due to the presence of millions of tail labels each having just a few, often a single, training point.

Original: Such labels can be very hard to learn due to data paucity.
Revised: Such labels can be very hard to learn due to data paucity.

Original: However, in most applications, predicting such rare tail labels accurately is much more rewarding than predicting common and obvious head labels.
Revised: However, in most applications, predicting such rare tail labels accurately is much more rewarding than predicting common and obvious head labels.

Original: This motivates DeepXML to have specialized architectures for head and tail labels which lead to accuracy gains not only in standard metrics which assign equal weights to all labels but also in propensity scored metrics (Jain et al., 2016) designed speciﬁcally for long-tail extreme classiﬁcation.
Revised: This motivates DeepXML to have specialized architectures for head and tail labels which lead to accuracy gains not only in standard metrics which assign equal weights to all labels but also in propensity scored metrics (Jain et al., 2016) designed speciﬁcally for long-tail extreme classiﬁcation.

Original: DeepXML : DeepXML improved both accuracy and scalability over existing deep extreme classiﬁers by partitioning all L labels into a small set of head labels, with cardinality less than 0 .
Revised: DeepXML : DeepXML improved both accuracy and scalability over existing deep extreme classiﬁers by partitioning all L labels into a small set of head labels, with cardinality less than 0 .

Original: 1 L , containing the most frequently occuring labels and a large set of tail labels containing everything else.
Revised: 1 L , containing the most frequently occuring labels and a large set of tail labels containing everything else.

Original: DeepXML ﬁrst represented a document by the tf-idf weighted linear combination of its word-vector embeddings as this architecture was empirically found to be more suitable for short text documents than the CNN and attention based architectures of XML-CNN and AttentionXML respectively.
Revised: DeepXML ﬁrst represented a document by the tf-idf weighted linear combination of its word-vector embeddings as this architecture was empirically found to be more suitable for short text documents than the CNN and attention based architectures of XML-CNN and AttentionXML respectively.

Original: The word-vector embeddings of the training documents were learnt on the head labels where there was enough data available to learn a good quality representation of the vocabulary.
Revised: The word-vector embeddings of the training documents were learnt on the head labels where there was enough data available to learn a good quality representation of the vocabulary.

Original: Accuracy was then further boosted by the introduction of a novel residual connection to ﬁne-tune the document representation for head labels.
Revised: Accuracy was then further boosted by the introduction of a novel residual connection to ﬁne-tune the document representation for head labels.

Original: This head architecture could be efﬁciently learnt on a single GPU with a fully connected ﬁnal output layer due to the small number of labels involved.
Revised: This head architecture could be efﬁciently learnt on a single GPU with a fully connected ﬁnal output layer due to the small number of labels involved.

Original: The word-vector embeddings were then transferred to the tail network where there wasn’t enough data available to train them from scratch.
Revised: The word-vector embeddings were then transferred to the tail network where there wasn’t enough data available to train them from scratch.

Original: Accuracy gains could potentially be obtained by ﬁne tuning the embeddings but this led to a dramatic increase in the training and prediction costs.
Revised: Accuracy gains could potentially be obtained by ﬁne tuning the embeddings but this led to a dramatic increase in the training and prediction costs.

Original: As an efﬁcient alternative, DeepXML achieved state-of-the-art accuracies by ﬁne tuning only the residual connection based document representation for tail labels.
Revised: As an efﬁcient alternative, DeepXML achieved state-of-the-art accuracies by ﬁne tuning only the residual connection based document representation for tail labels.

Original: A number of modiﬁcations were made to the highly scalable Slice classiﬁer (Jain et al., 2019) for pre-trained embeddings to allow it to also train the tail residual connection without sacriﬁcing scalability.
Revised: A number of modiﬁcations were made to the highly scalable Slice classiﬁer (Jain et al., 2019) for pre-trained embeddings to allow it to also train the tail residual connection without sacriﬁcing scalability.

Original: Finally, instead of learning an expensive ensemble of base classiﬁers to increase accuracy (Prabhu et al., 2018b;
Revised: Finally, instead of learning an expensive ensemble of base classiﬁers to increase accuracy (Prabhu et al., 2018b;

Original: You et al., 2018), DeepXML improved performance by re-ranking the set of predicted labels to eliminate the hardest negatives for the base classiﬁer with only a 10% increase in training time.
Revised: You et al., 2018), DeepXML improved performance by re-ranking the set of predicted labels to eliminate the hardest negatives for the base classiﬁer with only a 10% increase in training time.

Original: Results : Experiments on medium scale datasets of short text documents with less than a million labels revealed that DeepXML’s accuracy gains over XML-CNN and AttentionXML could be up to 3.
Revised: Results : Experiments on medium scale datasets of short text documents with less than a million labels revealed that DeepXML’s accuracy gains over XML-CNN and AttentionXML could be up to 3.

Original: and 4.32 percentage points respectively in terms of precision@ k and up to 5.32 and 4.2 percentage points respectively in terms of propensity-scored precision@ k .
Revised: and 4.32 percentage points respectively in terms of precision@ k and up to 5.32 and 4.2 percentage points respectively in terms of propensity-scored precision@ k .

Original: At the same time, DeepXML could be up to 15 × and 41 × faster to train than XML-CNN and AttentionXML respectively on these datasets using a single GPU.
Revised: At the same time, DeepXML could be up to 15 × and 41 × faster to train than XML-CNN and AttentionXML respectively on these datasets using a single GPU.

Original: Furthermore, XML-CNN and AttentionXML were unable to scale to a proprietary dataset for matching queries to bid phrases containing 3 million labels and 21 million training points on which DeepXML trained in 14 hours on a single GPU.
Revised: Furthermore, XML-CNN and AttentionXML were unable to scale to a proprietary dataset for matching queries to bid phrases containing 3 million labels and 21 million training points on which DeepXML trained in 14 hours on a single GPU.

Original: On this dataset, DeepXML was found to be at least 19 percentage points more accurate than Slice, Parabel (Prabhu et al., 2018b), and other leading query bid phrase-matching techniques currently running in production.
Revised: On this dataset, DeepXML was found to be at least 19 percentage points more accurate than Slice, Parabel (Prabhu et al., 2018b), and other leading query bid phrase-matching techniques currently running in production.

Original: Contributions : This paper makes the following contributions: (a) It proposes the DeepXML architecture for short text documents that is more accurate than state-of-the-art extreme classiﬁers; (b) it proposes an efﬁcient training algorithm that allows DeepXML to be an order of magnitude more scalable than leading deep extreme classiﬁers; and (c) it demonstrates that DeepXML could be signiﬁcantly better at matching user queries to advertiser bid phrases as compared to leading techniques in production on a popular web search engine.
Revised: Contributions : This paper makes the following contributions: (a) It proposes the DeepXML architecture for short text documents that is more accurate than state-of-the-art extreme classiﬁers; (b) it proposes an efﬁcient training algorithm that allows DeepXML to be an order of magnitude more scalable than leading deep extreme classiﬁers; and (c) it demonstrates that DeepXML could be signiﬁcantly better at matching user queries to advertiser bid phrases as compared to leading techniques in production on a popular web search engine.

Original: Source code for DeepXML and the short text document datasets used in this paper can be downloaded from (Anonymous, 2019).
Revised: Source code for DeepXML and the short text document datasets used in this paper can be downloaded from (Anonymous, 2019).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Much work has been done in extreme multi-label classiﬁcation which can be broadly categorized in two categories: a) learning the classiﬁer with pre-computed features (Agrawal et al., 2013; Cissé et al., 2013; Prabhu & Varma, 2014;
Revised: Much work has been done in extreme multi-label classiﬁcation which can be broadly categorized in two categories: a) learning the classiﬁer with pre-computed features (Agrawal et al., 2013; Cissé et al., 2013; Prabhu & Varma, 2014;

Original: Yu et al., 2014; Weston et al., 2013; Mineiro & Nikos, 2014; Jain et al., 2016; 2017; 2019;
Revised: Yu et al., 2014; Weston et al., 2013; Mineiro & Nikos, 2014; Jain et al., 2016; 2017; 2019;

Original: Yen et al., 2016; 2018;
Revised: Yen et al., 2016; 2018;

Original: Xu et al., 2016; Babbar & Schölkopf, 2017; 2019;
Revised: Xu et al., 2016; Babbar & Schölkopf, 2017; 2019;

Original: Niculescu-Mizil & Abbasnejad, 2017; Papanikolaou & Tsoumakas, 2017; Prabhu et al., 2018a;b;
Revised: Niculescu-Mizil & Abbasnejad, 2017; Papanikolaou & Tsoumakas, 2017; Prabhu et al., 2018a;b;

Original: Barezi et al., 2019; Jasinska et al., 2016), b) jointly learning feature representation along with the classiﬁer (Chen & Lin, 2012; Balasubramanian & Lebanon, 2012; Bi & Kwok, 2013; Liu et al., 2017; Jernite et al., 2017;
Revised: Barezi et al., 2019; Jasinska et al., 2016), b) jointly learning feature representation along with the classiﬁer (Chen & Lin, 2012; Balasubramanian & Lebanon, 2012; Bi & Kwok, 2013; Liu et al., 2017; Jernite et al., 2017;

Original: Wydmuch et al., 2018; Bhatia et al., 2015; Tagami, 2017; You et al., 2018;
Revised: Wydmuch et al., 2018; Bhatia et al., 2015; Tagami, 2017; You et al., 2018;

Original: Krichene et al., 2019; Barezi et al., 2019).
Revised: Krichene et al., 2019; Barezi et al., 2019).

Original: More extensive survey of extreme classiﬁcation and deep learning approaches can be found in section A.5 in the supplementary material.
Revised: More extensive survey of extreme classiﬁcation and deep learning approaches can be found in section A.5 in the supplementary material.

Original: Traditionally, extreme classiﬁcation approaches used sparse BoW features due to fast & efﬁcient feature computation as well as state-of-the-art performance for a large number of labels.
Revised: Traditionally, extreme classiﬁcation approaches used sparse BoW features due to fast & efﬁcient feature computation as well as state-of-the-art performance for a large number of labels.

Original: However, for short text documents such as queries, which form the focus of the paper, deep learning representations are more effective than BoW (Jain et al., 2019).
Revised: However, for short text documents such as queries, which form the focus of the paper, deep learning representations are more effective than BoW (Jain et al., 2019).

Original: Unfortunately, existing deep learning approaches for extreme classiﬁcation yielding state-of-the-art accuracy are not scalable, while scalable approaches have not been shown to yield state-of-the-art accuracy.
Revised: Unfortunately, existing deep learning approaches for extreme classiﬁcation yielding state-of-the-art accuracy are not scalable, while scalable approaches have not been shown to yield state-of-the-art accuracy.

Original: In particular, XML-CNN, acheives state-ofthe-art accuracy on short text documents but have not been shown to scale beyond a million labels, whereas AttentionXML was found to be slightly more scalable but less accurate as shown in table 1.
Revised: In particular, XML-CNN, acheives state-ofthe-art accuracy on short text documents but have not been shown to scale beyond a million labels, whereas AttentionXML was found to be slightly more scalable but less accurate as shown in table 1.

Original: Scalability: Deep learning techniques have been very successful and comprehensively beaten BoW features in small output space (Kim, 2014; Yang et al., 2016).
Revised: Scalability: Deep learning techniques have been very successful and comprehensively beaten BoW features in small output space (Kim, 2014; Yang et al., 2016).

Original: Unfortunately, the scalablity of such techniques degrades in XML setting as the ﬁnal fully connected layer leads to cost linear in number of labels (Liu et al., 2017).
Revised: Unfortunately, the scalablity of such techniques degrades in extreme multi-label (XML) setting as the ﬁnal fully connected layer leads to cost linear in number of labels (Liu et al., 2017).

Original: Traditional approach to solve this problem is negative sampling (Mikolov et al., 2013).
Revised: Traditional approach to solve this problem is negative sampling (Mikolov et al., 2013).

Original: Unfortunately, at the extreme scale, negative sampling has to be applied more aggressively as demonstrated in Fig 5 in the appendix.
Revised: Unfortunately, at the extreme scale, negative sampling has to be applied more aggressively as demonstrated in Fig 5 in the appendix.

Original: To eliminate this problem, many approaches have been proposed such as tree based (Prabhu et al., 2018b; You et al., 2018; Mikolov et al., 2013; Jernite et al., 2017) and hashing based (Shrivastava & Li, 2014; Vijayanarasimhan et al., 2014), approximate nearest neighbor sub-sampling techniques (Jain et al., 2019; Reddi et al., 2018).
Revised: To eliminate this problem, many approaches have been proposed such as tree based (Prabhu et al., 2018b; You et al., 2018; Mikolov et al., 2013; Jernite et al., 2017) and hashing based (Shrivastava & Li, 2014; Vijayanarasimhan et al., 2014), approximate nearest neighbor sub-sampling techniques (Jain et al., 2019; Reddi et al., 2018).

Original: Adding to that, prediction time complexity remains O ( dL ) , which is not suitable for real-time predictions.
Revised: Adding to that, prediction time complexity remains O ( dL ) , which is not suitable for real-time predictions.

Original: Approaches such as Maximum Inner Product Search (Yen et al., 2018) and Local Sensitive Hashing (Niculescu-Mizil & Abbasnejad, 2017) can speed-up one-vs.-all inference but it leads to further loss in accuracy when applied post-training.
Revised: Approaches such as Maximum Inner Product Search (Yen et al., 2018) and Local Sensitive Hashing (Niculescu-Mizil & Abbasnejad, 2017) can speed-up one-vs.-all inference but it leads to further loss in accuracy when applied post-training.

Original: Accuracy: Tail labels are harder to predict as compared to head labels due to training data scarcity but might also be more informative and rewarding in XML setting.
Revised: Accuracy: Tail labels are harder to predict as compared to head labels due to training data scarcity but might also be more informative and rewarding in XML setting.

Original: For instance, predicting tag "Artiﬁcial intelligence researchers" could be more informative than tag "1954 deaths" on Wikipedia page of "Alan Turing".
Revised: For instance, predicting tag "Artiﬁcial intelligence researchers" could be more informative than tag "1954 deaths" on Wikipedia page of "Alan Turing".

Original: Hence, propensity based precision and nDCG are the focus of the paper.
Revised: Hence, propensity based precision and nDCG are the focus of the paper.

Original: (Wei & Li, 2018) demonstrates that trimming tail labels leads to marginal decay in performance on vanilla metrics.
Revised: (Wei & Li, 2018) demonstrates that trimming tail labels leads to marginal decay in performance on vanilla metrics.

Original: However, a) trimming tail labels lead to loss in propensity based metrics, b) this approach has been demonstrated to scale to ≈ 30 K labels only.
Revised: However, a) trimming tail labels lead to loss in propensity based metrics, b) this approach has been demonstrated to scale to ≈ 30 K labels only.

Original: Researchers have also tried to improve performance on tail labels by directly optimizing propensity scored metrics (Jain et al., 2016), posing learning in the presence of adversarial perturbations (Babbar & Schölkopf, 2019) and treating tail labels as outliers (Xu et al., 2016).
Revised: Researchers have also tried to improve performance on tail labels by directly optimizing propensity scored metrics (Jain et al., 2016), posing learning in the presence of adversarial perturbations (Babbar & Schölkopf, 2019) and treating tail labels as outliers (Xu et al., 2016).

Original: Although, these approaches boost performance on tail labels, however they are not well suited for short text documents due to, a) support only for ﬁxed
Revised: Although, these approaches boost performance on tail labels, however they are not well suited for short text documents due to, a) support only for ﬁxed

Original: Matching user queries to bid phrases: Approaches in this domain can be categorized as embeddings based (Jain et al., 2019), sequence-to-sequence models (Gao et al., 2012; Jones et al., 2006;
Revised: Matching user queries to bid phrases: Approaches in this domain can be categorized as embeddings based (Jain et al., 2019), sequence-to-sequence models (Gao et al., 2012; Jones et al., 2006;

Original: Riezler & Liu, 2010; He et al., 2016; Lee et al., 2018;
Revised: Riezler & Liu, 2010; He et al., 2016; Lee et al., 2018;

Original: Lian et al., 2019) and query graph based models (Ioannis et al., 2008).
Revised: Lian et al., 2019) and query graph based models (Ioannis et al., 2008).

Original: Unfortunately, the trigger coverage, suggestion density and quality of recommendations could be poor for many of these techniques.
Revised: Unfortunately, the trigger coverage, suggestion density and quality of recommendations could be poor for many of these techniques.

Original: For instance, query graph based methods can only recommend suggestions for previously seen triggers thereby limiting their trigger coverage.
Revised: For instance, query graph based methods can only recommend suggestions for previously seen triggers thereby limiting their trigger coverage.

Original: Additionally, Sequence-to-sequence models suffers from expensive training and prediction cost.
Revised: Additionally, Sequence-to-sequence models suffers from expensive training and prediction cost.

Original: Although, efﬁcient structures such as trie (Lian et al., 2019) have been deployed to reduce output complexity, however, at the cost of limited bid phrase coverage.
Revised: Although, efﬁcient structures such as trie (Lian et al., 2019) have been deployed to reduce output complexity, however, at the cost of limited bid phrase coverage.

Original: [Figure]
Revised: [Figure]

Original:  D EEP XML
Revised:  D EEP XML

Original: State-of-the-art deep extreme classiﬁers are neither scalable nor accurate for short text documents because: (a) representations learnt through CNNs (XML-CNN), LSTM+Attention (AttentionXML) or context (Bert (Devlin et al., 2018), Elmo (Peters et al., 2018), etc .) might not be accurate given limited tail data with only a few words per document and might also be expensive to compute; and (b) they face scalability issues as the ﬁnal fully connected output layer has millions of outputs making both the forward pass as well as gradient backpropagation prohibitive for even a single training point.
Revised: State-of-the-art deep extreme classiﬁers are neither scalable nor accurate for short text documents because: (a) representations learnt through CNNs (XML-CNN), LSTM+Attention (AttentionXML) or context (Bert (Devlin et al., 2018), Elmo (Peters et al., 2018), etc .) might not be accurate given limited tail data with only a few words per document and might also be expensive to compute; and (b) they face scalability issues as the ﬁnal fully connected output layer has millions of outputs making both the forward pass as well as gradient backpropagation prohibitive for even a single training point.

Original: DeepXML addresses these limitations by using a feature representation inspired by FastText (Joulin et al., 2017) and an output layer inspired by Slice (Jain et al., 2019) as these have been demonstrated to be both accurate and scalable for short text documents in their individual capacities.
Revised: DeepXML addresses these limitations by using a feature representation inspired by FastText (Joulin et al., 2017) and an output layer inspired by Slice (Jain et al., 2019) as these have been demonstrated to be both accurate and scalable for short text documents in their individual capacities.

Original: Unfortunately, combining FastText and Slice in a straight forward fashion also turns out to be neither accurate nor scalable.
Revised: Unfortunately, combining FastText and Slice in a straight forward fashion also turns out to be neither accurate nor scalable.

Original: The rest of this section details the design choices and modiﬁcations that needed to be made in order to get the combination to work.
Revised: The rest of this section details the design choices and modiﬁcations that needed to be made in order to get the combination to work.

Original: DeepXML, FastText & Slice : FastText represents a document as an efﬁciently computable linear combination of its (sub) word-vector embeddings making it highly scalable and well suited for extreme classiﬁcation scenarios.
Revised: DeepXML, FastText & Slice : FastText represents a document as an efﬁciently computable linear combination of its (sub) word-vector embeddings making it highly scalable and well suited for extreme classiﬁcation scenarios.

Original: Unfortunately, the FastText architecture is linear and low-capacity thereby leading to a loss in accuracy when there isn’t a vast amount of unsupervised data available for training.
Revised: Unfortunately, the FastText architecture is linear and low-capacity thereby leading to a loss in accuracy when there isn’t a vast amount of unsupervised data available for training.

Original: Furthermore, previous attempts (Wydmuch et al., 2018) at learning FastText representations in a supervised manner by replacing the fully connected output layer by a ﬁxed tree PLT extreme classiﬁer (Jasinska et al., 2016) have led to even worse accuracies than XML-CNN and AttentionXML.
Revised: Furthermore, previous attempts (Wydmuch et al., 2018) at learning FastText representations in a supervised manner by replacing the fully connected output layer by a ﬁxed Probabilistic Label Trees (PLT) extreme classiﬁer (Jasinska et al., 2016) have led to even worse accuracies than XMLCNN and AttentionXML.

Original: Replacing the ﬁxed tree PLT by learnt Slice improves accuracy somewhat but does not lead to state-ofthe-art results and also greatly increases training time (please see ablation experiments in Section 5).
Revised: Replacing the ﬁxed tree PLT by learnt Slice improves accuracy somewhat but does not lead to state-of-the-art results and also greatly increases training time (please see ablation experiments in Section 5).

Original: DeepXML addresses these limitations by adding a non linearity and residual block to make up for the lack of FastText’s expressive power and training the word-vector embeddings using a fully connected output layer on a small number of head labels rather than through Slice.
Revised: DeepXML addresses these limitations by adding a non linearity and residual block to make up for the lack of FastText’s expressive power and training the word-vector embeddings using a fully connected output layer on a small number of head labels rather than through Slice.

Original: Please refer to Figure 1 for full architecture of DeepXML.
Revised: Please refer to Figure 1 for full architecture of DeepXML.

Original: Once the word-vector embeddings have been trained on the head, they are frozen and transferred to the tail where only the residual block is ﬁne tuned.
Revised: Once the word-vector embeddings have been trained on the head, they are frozen and transferred to the tail where only the residual block is ﬁne tuned.

Original: This increases both accuracy as there isn’t enough training data available to learn good quality word embeddings from scratch on the tail as well as scalability as ﬁne-tuning the head embeddings on the tail would prove too expensive on large problems.
Revised: This increases both accuracy as there isn’t enough training data available to learn good quality word embeddings from scratch on the tail as well as scalability as ﬁne-tuning the head embeddings on the tail would prove too expensive on large problems.

Original: 3. A RCHITECTURE AND TRAINING
Revised: 3. A RCHITECTURE AND TRAINING

Original: DeepXML-h : Model parameters i.e. words embeddings, residual block and fully connected classiﬁer are learnt with Binary Cross Entropy loss and Adam optimizer.
Revised: DeepXML-h : Model parameters i.e. words embeddings, residual block and fully connected classiﬁer are learnt with Binary Cross Entropy loss and Adam optimizer.

Original: DeepXML-h can be efﬁciently trained with a fully connected ﬁnal layer on a single GPU, as L h contains only a small subset of labels.
Revised: DeepXML-h can be efﬁciently trained with a fully connected ﬁnal layer on a single GPU, as L h (Label set containing head labels)contains only a small subset of labels.

Original: In practice, the size of L h does not grow beyond 0.2M even for datasets with millions of labels.
Revised: In practice, the size of L h does not grow beyond 0.2M even for datasets with millions of labels.

Original: Additionally, an ANNS structure was trained over label centroids L ch of head labels.
Revised: Additionally, an approximate nearest neighbour search (ANNS) structure was trained over label centroids L ch of head labels.

Original: Note that label centroids ( L ch ) are computed as:L ch
Revised: Note that label centroids ( L ch ) are computed as:

Original:  =
Revised: L ch =

Original: Y T X (cid:48) .
Revised: Y T X (cid:48) , please note that X (cid:48) = f ( x ) + r h ( x ) from the Fig.

Original: This brings down inference cost of classiﬁer from O ( d | L h | ) to O ( d | s | + d log | L h | ) , given | s | (cid:28) | L h | .
Revised: This brings down inference cost of classiﬁer from O ( d | L h | ) to O ( d | s | + d log | L h | ) , given | s | (cid:28) | L h | .

Original: Here, | s | is the size of label shortlist queried from ANNS structure during prediction which is kept as 300 in practice.
Revised: Here, | s | is the size of label shortlist queried from ANNS structure during prediction which is kept as 300 in practice.

Original: Unfortunately, ANNS trained over label centroids may lead to poor recall values when a single centroid is unable to capture diversity in training instances for labels with highly different contexts (say L (cid:48) h ).
Revised: Unfortunately, ANNS trained over label centroids may lead to poor recall values when a single centroid is unable to capture diversity in training instances for labels with highly different contexts (say L (cid:48) h ).

Original: For instance, 280K articles, ranging from scientists to athletes, are tagged with the ‘living people’ tag in WikiTitle-500K (Bhatia et al., 2016) dataset.
Revised: For instance, 280K articles, ranging from scientists to athletes, are tagged with the ‘living people’ tag in WikiTitle-500K (Bhatia et al., 2016) dataset.

Original: Slice increases the shortlist to improve recall; however, at the cost of 6 × increase in prediction time.
Revised: Slice increases the shortlist to improve recall; however, at the cost of 6 × increase in prediction time.

Original: DeepXML-h tackles this issue by allowing multiple representations for labels in L (cid:48)
Revised: DeepXML-h tackles this issue by allowing multiple representations for labels in L (cid:48)

Original: h .
Revised: h .

Original: Documents are clustered using the KMeans algorithm into c clusters for each label in this set.
Revised: Documents are clustered using the KMeans algorithm into c clusters for each label in this set.

Original: Therefore, c representatives for each label in L (cid:48) h are computed.
Revised: Therefore, c representatives for each label in L (cid:48) h are computed.

Original: This leads to 5% increase in recall@300 and 6% precision@300 with a shortlist of size 300.
Revised: This leads to 5% increase in recall@300 and 6% precision@300 with a shortlist of size 300.

Original: Clustering will lead to | L h − L (cid:48)
Revised: Clustering will lead to | L h − L (cid:48)

Original: h | + c | L (cid:48)
Revised: h | + c | L (cid:48)

Original: h | label representations and hence could potentially lead to increased time complexity.
Revised: h | label representations and hence could potentially lead to increased time complexity.

Original: However, clustering just the top 3 labels into 300 clusters seems to work well in our experiments without signiﬁcant change in training time, and it doesn’t impact prediction time at all.
Revised: However, clustering just the top 3 labels into 300 clusters seems to work well in our experiments without signiﬁcant change in training time, and it doesn’t impact prediction time at all.

Original: DeepXML-t : DeepXML still relies on Slice to ﬁne-tune the residual block in the tail network and learn the weights in the fully connected output layer for millions of tail labels.
Revised: DeepXML-t : DeepXML still relies on Slice to ﬁne-tune the residual block in the tail network and learn the weights in the fully connected output layer for millions of tail labels.

Original: Slice cuts down the time for both the forward pass and the gradient backpropagation from linear to logarithmic in the number of labels.
Revised: Slice cuts down the time for both the forward pass and the gradient backpropagation from linear to logarithmic in the number of labels.

Original: Slice achieves this by ﬁrst representing each label by the unit normalized mean of the feature vectors of the training points tagged with the label.
Revised: Slice achieves this by ﬁrst representing each label by the unit normalized mean of the feature vectors of the training points tagged with the label.

Original: It then learns an approximate nearest neighbour search (ANNS) data structure (Malkov & Yashunin, 2016) over the label representations to determine the most likely labels for a given data point (please see the supplementary material for a more detailed description).
Revised: It then learns an ANNS data structure (Malkov & Yashunin, 2016) over the label representations to determine the most likely labels for a given data point (please see the supplementary material for a more detailed description).

Original: This technique was shown to efﬁciently scale to millions of labels and training points when the feature representation was ﬁxed (Jain et al., 2019).
Revised: This technique was shown to efﬁciently scale to millions of labels and training points when the feature representation was ﬁxed (Jain et al., 2019).

Original: Unfortunately, when the feature representation is being learnt, the ANNS data structure needs to be constantly updated as the label representations change with each training batch.
Revised: Unfortunately, when the feature representation is being learnt, the ANNS data structure needs to be constantly updated as the label representations change with each training batch.

Original: This can lead to a marked slowdown as maintaining the ANNS data structure can incur signiﬁcant computational cost.
Revised: This can lead to a marked slowdown as maintaining the ANNS data structure can incur signiﬁcant computational cost.

Original: DeepXML speeds up training by redeﬁning the label representation to be the unit normalized mean of the document representation before the residual block computed using the learnt word embeddings alone.
Revised: DeepXML speeds up training by redeﬁning the label representation to be the unit normalized mean of the document representation before the residual block computed using the learnt word embeddings alone.

Original: This allows the ANNS data structure to be learnt just once after the word embeddings have been learnt on the head labels and before training starts on the tail.
Revised: This allows the ANNS data structure to be learnt just once after the word embeddings have been learnt on the head labels and before training starts on the tail.

Original: Unfortunately, this also leads to a loss in training accuracy as the label representation is now an approximation to the true representation that should have been deﬁned as the unit normalized mean of the document representations computed after the residual block.
Revised: Unfortunately, this also leads to a loss in training accuracy as the label representation is now an approximation to the true representation that should have been deﬁned as the unit normalized mean of the document representations computed after the residual block.

Original: This loss in accuracy can be compensated by requiring Slice to generate 3x more nearest labels to a given data point.
Revised: This loss in accuracy can be compensated by requiring Slice to generate 3x more nearest labels to a given data point.

Original: This allowed the hard negative labels to now be present in the longer list but this signiﬁcantly increased training time.
Revised: This allowed the hard negative labels to now be present in the longer list but this signiﬁcantly increased training time.

Original: It was empirically determined that a more efﬁcient strategy was to extend the shortlist by adding randomly sampled negative labels as this led to no loss in accuracy with only a minimal increase in training time.
Revised: It was empirically determined that a more efﬁcient strategy was to extend the shortlist by adding randomly sampled negative labels as this led to no loss in accuracy with only a minimal increase in training time.

Original: 3. P REDICTION
Revised: 3. P REDICTION

Original: Initially, classiﬁer scores and ANNS scores from both DeepXML-h and DeepXML-t are merged in a single vector ˆ y .
Revised: Initially, classiﬁer scores and ANNS scores from both DeepXML-h and DeepXML-t are merged in a single vector ˆ y .

Original: As demonstrated in Fig.
Revised: As demonstrated in Fig.

Original: 1, ˆ y clf
Revised: 1, ˆ y clf

Original: −
Revised: −

Original: h is the classiﬁer score (logit) from DeepXML-h and ˆ y anns − h is the cosine similarity from ANNS-h.
Revised: h is the classiﬁer score (logit) from DeepXML-h and ˆ y anns − h is the cosine similarity from ANNS-h.

Original: Similarly, ˆ y clf − t is the classiﬁer score from
Revised: Similarly, ˆ y clf − t is the classiﬁer score from

Original: DeepXML-t and ˆ y ann − t is the cosine similarity from ANNS-t.
Revised: DeepXML-t and ˆ y ann − t is the cosine similarity from ANNS-t.

Original: The ﬁnal DeepXML score vector ( ˆ y ) is computed as follows :
Revised: The ﬁnal DeepXML score vector ( ˆ y ) is computed as follows :

Original: [Equation]
Revised: [Equation]

Original: Note that ˆ y clf − t , ˆ y clf − h , ˆ y anns − h , and ˆ y anns − t are sparse vectors, σ is a sparse-sigmoid function computed only at non-zero entries and β ∈ [0 , 1] .
Revised: Note that ˆ y clf − t , ˆ y clf − h , ˆ y anns − h , and ˆ y anns − t are sparse vectors, σ is a sparse-sigmoid function computed only at non-zero entries and β ∈ [0 , 1] .

Original: The average cost of prediction can be broken down into the following four components: a) computing dense feature representation: O ( dγ ) , b) generating a shortlist:
Revised: The average cost of prediction can be broken down into the following four components: a) computing dense feature representation: O ( dγ ) , b) generating a shortlist:

Original: O ( d log | L t | ) , c) computing classiﬁer scores: O ( d | s | ) .
Revised: O ( d log | L t | ) , c) computing classiﬁer scores: O ( d | s | ) .

Original: Here, γ is the average number of features per document and | s | is the shortlist size in ANNS.
Revised: Here, γ is the average number of features per document and | s | is the shortlist size in ANNS.

Original: 3. D EEP XML-RE
Revised: 3. D EEP XML-RE

Original: Extreme classiﬁers such as Parabel (Prabhu et al., 2018b), AttentionXML (You et al., 2018) learn multiple models to get better prediction accuracy.
Revised: Extreme classiﬁers such as Parabel (Prabhu et al., 2018b), AttentionXML (You et al., 2018) learn multiple models to get better prediction accuracy.

Original: However, this leads to increased training and prediction time ( 200% for both Parabel and AttentionXML) linearly with the number of models.
Revised: However, this leads to increased training and prediction time ( 200% for both Parabel and AttentionXML) linearly with the number of models.

Original: Whereas,
Revised: Whereas,

Original: DeepXML-RE learns a re-ranker with training cost logarithmic in the number of labels by training over a shortlist of negative labels.
Revised: DeepXML-RE learns a re-ranker with training cost logarithmic in the number of labels by training over a shortlist of negative labels.

Original: Speciﬁcally, false positive labels predicted by DeepXML ( a.k.a hardest negatives) are selected as negatives for DeepXML-RE.
Revised: Speciﬁcally, false positive labels predicted by DeepXML ( a.k.a hardest negatives) are selected as negatives for DeepXML-RE.

Original: This leads to only 10 − 20% increase in training time.
Revised: This leads to only 10 − 20% increase in training time.

Original: The architecture of DeepXML-RE is same as DeepXML-t, i.e., a word embedding layer, residual block, and classiﬁer.
Revised: The architecture of DeepXML-RE is same as DeepXML-t, i.e., a word embedding layer, residual block, and classiﬁer.

Original: Model parameters are learnt with binary cross entropy loss and SparseAdam optimizer.
Revised: Model parameters are learnt with binary cross entropy loss and SparseAdam optimizer.

Original: During prediction, DeepXML-RE evaluates on labels shortlisted by
Revised: During prediction, DeepXML-RE evaluates on labels shortlisted by

Original: DeepXML only, thereby incurring a prediction cost of O ( d | s | ) .
Revised: DeepXML only, thereby incurring a prediction cost of O ( d | s | ) .

Original:  E XPERIMENTS AND R ESULTS 4. D ATA SETS ANDEVALUATION METRICS
Revised:  E XPERIMENTS AND R ESULTS 4. D ATA SETS AND

Original: [Figure]
Revised: [Figure]

Original: Experiments were carried out on the Query to Bid phrases (Q2B-3M) dataset, with 3 million labels, by mining the logs of a popular search engine.
Revised: Experiments were carried out on the Query to Bid phrases (Q2B-3M) dataset, with 3 million labels, by mining the logs of a popular search engine.

Original: Each user query was treated as an instance and relevant advertiser bid phrases became its labels.
Revised: Each user query was treated as an instance and relevant advertiser bid phrases became its labels.

Original: Unfortunately, only
Revised: Unfortunately, only

Original: Slice (Jain et al., 2019) and Parabel (Prabhu et al., 2018b) could scale to this dataset.
Revised: Slice (Jain et al., 2019) and Parabel (Prabhu et al., 2018b) could scale to this dataset.

Original: DeepXML is also compared to state-of-the-art methods for query keyword prediction such as Simrank++(Ioannis et al., 2008) and a sequence-to-sequence model based on BERT (Devlin et al., 2018; Lian et al., 2019).
Revised: DeepXML is also compared to state-of-the-art methods for query keyword prediction such as Simrank++(Ioannis et al., 2008) and a sequence-to-sequence model based on BERT (Devlin et al., 2018; Lian et al., 2019).

Original: Experiments were also carried out on four moderate size datasets (Anonymous (2019)).
Revised: Experiments were also carried out on four moderate size datasets (Anonymous (2019)).

Original: The applications considered were tagging Wikipedia pages (WikiTitles-500K), suggesting relevant articles (WikiSeeAlsoTitles-250K), and item-to-item recommendation of Amazon products (AmazonTitles-670K and AmazonTitles-3M).
Revised: The applications considered were tagging Wikipedia pages (WikiTitles-500K), suggesting relevant articles (WikiSeeAlsoTitles-250K), and item-to-item recommendation of Amazon products (AmazonTitles-670K and AmazonTitles-3M).

Original: Please refer to supplementary Table 4 for dataset statistics.
Revised: Please refer to supplementary Table 4 for dataset statistics.

Original: For these moderate size datasets DeepXML was compared to leading deep learning and BoW feature-based methods including XML-CNN, Attention-XML,
Revised: For these moderate size datasets DeepXML was compared to leading deep learning and BoW feature-based methods including XML-CNN, Attention-XML,

Original: Slice, AnnexML
Revised: Slice, AnnexML

Original: (Tagami, 2017), PfastreXML (Jain et al., 2016), Parabel (Prabhu et al., 2018b),
Revised: (Tagami, 2017), PfastreXML (Jain et al., 2016), Parabel (Prabhu et al., 2018b),

Original: XT (Wydmuch et al., 2018), and DiSMEC (Babbar & Schölkopf, 2017).
Revised: XT (Wydmuch et al., 2018), and DiSMEC (Babbar & Schölkopf, 2017).

Original: The implementation of all algorithms was provided by the respective authors.
Revised: The implementation of all algorithms was provided by the respective authors.

Original: 4. H YPERPARAMETERS
Revised: 4. H YPERPARAMETERS

Original: DeepXML has 7 hyperparameters: (a) learning rate and epochs for DeepXML-h; (b) 
Revised: DeepXML has 7 hyperparameters: (a) learning rate and epochs for DeepXML-h; (b) learning rate and epochs for DeepXML-t; (c) embedding dimensions ( d ); (d) shortlist size ( | s | ) and (e) threshold to split label set into head and tail labels. Results are reported for d = 300 and | s | = 300 . The label threshold is chosen via cross-validation (please refer to section A.2 in the supplementary material).

Original: Binary cross-entropy loss and the SparseAdam optimizer were used to update the  parameters.
Revised: Binary cross-entropy loss and the SparseAdam optimizer were used to update the model parameters.

Original: Please refer to Table 5 in the supplementary section for full parameter settings.
Revised: Please refer to Table 5 in the supplementary section for full parameter settings.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: - - - - - - - - - - - XT - - - - - - - - - - -
Revised: - - - - - - - - - - - XT - - - - - - - - - - -

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: AttentionXML- - - - - - - - - - - - AttentionXML - - - - - - - - - - -
Revised: AttentionXML- - - - - - - - - - - - AttentionXML - - - - - - - - - - -

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. R ESULTS ON MODERATE SIZE DATA SETS
Revised: 4. R ESULTS ON MODERATE SIZE DATA SETS

Original: Table 1 compares the performance of DeepXML and DeepXML-RE with state-of-the-art deep learning approaches namely XML-CNN and AttentionXML.
Revised: Table 1 compares the performance of DeepXML and DeepXML-RE with state-of-the-art deep learning approaches namely XML-CNN and AttentionXML.

Original: DeepXML is upto 5 percentage points more accurate than XML-CNN and AttentionXML on propensity-scored metrics.
Revised: DeepXML is upto 5 percentage points more accurate than XML-CNN and AttentionXML on propensity-scored metrics.

Original: Additionally, Figand supplementary Fig 4 demonstrate that tail labels contribute more to DeepXML’s predictions w.r.t other methods.
Revised: Additionally, Figand supplementary Fig 4 demonstrate that tail labels contribute more to DeepXML’s predictions w.r.t other methods.

Original: Furthermore, DeepXML can be 2–4 percentage points more accurate than XML-CNN and AttentionXML on vanilla precision and nDCG.
Revised: Furthermore, DeepXML can be 2–4 percentage points more accurate than XML-CNN and AttentionXML on vanilla precision and nDCG.

Original: Table 1 also demonstrates that DeepXML can be 33–42 × faster to train as compared to XML-CNN and AttentionXML respectively on a single
Revised: Table 1 also demonstrates that DeepXML can be 33–42 × faster to train as compared to XML-CNN and AttentionXML respectively on a single

Original: GPU.
Revised: GPU.

Original: DeepXML-RE can be scaled to AmazonTitles-3M on one GPU and is able to train with only a 10% increase in training time and a 20% increase in prediction time from DeepXML as compared to a 200% increase in training and prediction time of AttentionXML from AttentionXML-1.
Revised: DeepXML-RE can be scaled to AmazonTitles-3M on one GPU and is able to train with only a 10% increase in training time and a 20% increase in prediction time from DeepXML as compared to a 200% increase in training and prediction time of AttentionXML from AttentionXML-1.

Original: This demonstrates that DeepXML and DeepXML-RE can outperform current state-of-the-art accuracy as well as scale to millions of labels .
Revised: This demonstrates that DeepXML and DeepXML-RE can outperform current state-of-the-art accuracy as well as scale to millions of labels .

Original: Table 1 shows that DeepXML and DeepXML-RE can outperform state-of-the-art BoW-based approaches as well.
Revised: Table 1 shows that DeepXML and DeepXML-RE can outperform state-of-the-art BoW-based approaches as well.

Original: Note that on AmazonTitles-3M, PfastreXML is 5 .percentage points better than DeepXML as well as DeepXML-RE for propensity-scored metrics but incurs a loss of 10 percentage points for vanilla precision, which is clearly unacceptable for real world tasks where both metrics are important.
Revised: Note that on AmazonTitles-3M, PfastreXML is 5 .percentage points better than DeepXML as well as DeepXML-RE for propensity-scored metrics but incurs a loss of 10 percentage points for vanilla precision, which is clearly unacceptable for real world tasks where both metrics are important.

Original: Please refer to Section A.3 in the appendix for more discussion.
Revised: Please refer to Section A.3 in the appendix for more discussion.

Original: [Table]
Revised: [Table]

Original: 4. R ESULTS ON Q2B
Revised: 4. R ESULTS ON Q2B

Original: Table 2 indicates that DeepXML-RE is upto 20–45 percentage points more accurate than the methods currently in production, Simrank++ and Seq2Seq respectively.
Revised: Table 2 indicates that DeepXML-RE is upto 20–45 percentage points more accurate than the methods currently in production, Simrank++ and Seq2Seq respectively.

Original: Further, DeepXML-RE is also compared to other scalable extreme classiﬁers such as Parabel and Slice and is seen to be 19 andpercentage points more accurate respectively.
Revised: Further, DeepXML-RE is also compared to other scalable extreme classiﬁers such as Parabel and Slice and is seen to be 19 andpercentage points more accurate respectively.

Original: Additionally, supplementary Table 6 includes example documents which demonstrate that DeepXML can make accurate and diverse predictions.
Revised: Additionally, supplementary Table 6 includes example documents which demonstrate that DeepXML can make accurate and diverse predictions.

Original:  A BLATION , EXTENSIONS
Revised:  A BLATION , EXTENSIONS

Original: [Table]
Revised: [Table]

Original: Table 3 demonstrates that DeepXML is more accurate than other variations such as a version with a vanilla Slice classiﬁer.
Revised: This section discuss impact of feature representations (DeepXML-SW, DeepXML-f) and classiﬁers (DeepXML-fr, DeepXML-ANNS, DeepXML-P, DeepXML-NS).

Original: Note that label centroids for this variant were updated after an interval of 5 epochs.
Revised: The detailed architectures are included in section A.6 in the supplementary section.

Original: Results for DeepXMLfr, a network with a fully connected layer and a residual connection, are also reported in Table 1.
Revised: Sub-word features : DeepXML can also exploit sub-word features (Joulin et al., 2017) for an additional 1% gain in precision as demonstrated in Table 3.

Original: Additionally, DeepXML-P, i
Revised: Label split and classiﬁer : DeepXML-fr refers to a joint

Original: [Table] e. , DeepXML-fr with a tree-based classiﬁer as proposed in AttentionXML, was found to be around 1 percentage point more  accurate than AttentionXML, demonstrating the effectiveness of DeepXML’s feature representation.
Revised: DeepXML architecture, i.e. without splitting labels, trained with a fully connected layer. As demonstrated in Table 1, the proposed algorithm is more scalable and accurate on tail labels w.r.t DeepXML-fr.

Original: DeepXML can also exploit sub-word features (Joulin et al., 2017) for an additional 1 percentage point gain in precision as demonstrated in Table 3.
Revised: DeepXML could be up to 1–3% more accurate then DeepXML-NS & DeepXML-ANNS where the classiﬁer is trained using negative sampling.

Original:  DeepXML was also compared against pre-trained representations  such as FastText, BERT (Devlin et al., 2018) andSIF (Arora, 2017) along with the Slice classiﬁer. As demonstrated in Tables 1 & 3, DeepXML is also upto 10 percentage points more accurate than pre-trained representations.
Revised: Additionally, DeepXML was found to be 1.3% more accurate than DeepXML-P which uses a tree based classiﬁer. Pre-trained features : DeepXML could be upto 10% more accurate than pre-trained representations with Slice classiﬁer (refer to Tables 1 & 3) such as FastText, BERT (Devlin et al., 2018) and

Original: Table 3 and Fig. 5 (in the supplementary material) demonstrate that the performance of DeepXML with negative sampling is worse than DeepXML-fr even with a large shortlist. In conclusion, DeepXML can be more accurate and scalable than alternative conﬁgurations.
Revised: SIF (Arora, 2017).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper developed DeepXML, an algorithm to jointly learn representations for extreme multilabel learning on text data.
Revised: This paper developed DeepXML, an algorithm to jointly learn representations for extreme multilabel learning on text data.

Original: The proposed algorithm addresses the key issues of scalability and low accuracy (especially on tail labels and very short documents) with existing approaches such as Slice,
Revised: The proposed algorithm addresses the key issues of scalability and low accuracy (especially on tail labels and very short documents) with existing approaches such as Slice,

Original: AttentionXML, and XML-CNN, and hence improves on them substantively.
Revised: AttentionXML, and XML-CNN, and hence improves on them substantively.

Original: Experiments revealed that DeepXML-RE can lead to a 1.0–4.3 percentage point gain in performance while being 33–42 × faster at training than AttentionXML.
Revised: Experiments revealed that DeepXML-RE can lead to a 1.0–4.3 percentage point gain in performance while being 33–42 × faster at training than AttentionXML.

Original: Furthermore, DeepXML was upto 15 percentage points more accurate than leading techniques for matching search engine queries to advertiser bid phrases.
Revised: Furthermore, DeepXML was upto 15 percentage points more accurate than leading techniques for matching search engine queries to advertiser bid phrases.

Original: We note that DeepXML’s gains are predominantly seen to be on predicting tail labels (for which very few direct word associations are available at train time) and on short documents (for which very few words are available at test time).
Revised: We note that DeepXML’s gains are predominantly seen to be on predicting tail labels (for which very few direct word associations are available at train time) and on short documents (for which very few words are available at test time).

Original: This indicates that the method is doing especially well, compared to earlier approaches, at learning word representations which allow for richer and denser associations between words – which allow for the words to be well-clustered in a meaningful semantic space, and hence useful and generalisable information about document labels extracted even when the number of direct word co-occurrences observed is very limited.
Revised: This indicates that the method is doing especially well, compared to earlier approaches, at learning word representations which allow for richer and denser associations between words – which allow for the words to be well-clustered in a meaningful semantic space, and hence useful and generalisable information about document labels extracted even when the number of direct word co-occurrences observed is very limited.

Original: In the future we would like to better understand the nature of these representations, and explore their utility for other linguistic tasks.
Revised: In the future we would like to better understand the nature of these representations, and explore their utility for other linguistic tasks.

Original: A S UPPLEMENTARY A. H YPERPARAMETERS OF D EEP XML FOR REPRODUCIBILITY
Revised: A S UPPLEMENTARY A. H YPERPARAMETERS OF D EEP XML FOR REPRODUCIBILITY

Original: Table 5 lists the parameter settings for different data sets.
Revised: Table 5 lists the parameter settings for different data sets.

Original: Experiments were performed with a random-seed of 22 on a P40 GPU card with CUDA 10, CuDNN 7.4, and Pytorch 1.2 (Paszke et al., 2017)
Revised: Experiments were performed with a random-seed of 22 on a P40 GPU card with CUDA 10, CuDNN 7.4, and Pytorch 1.2 (Paszke et al., 2017).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: A. S PLITTING L INTO L h AND L t
Revised: A. S PLITTING L INTO L h AND L t

Original: Label set L is divided into two disjoint sets, i
Revised: Label set L is divided into two disjoint sets, i

Original: [Figure]
Revised: [Figure]

Original: e. L h and L t based on the frequency of the labels.
Revised: e. L h and L t based on the frequency of the labels.

Original: Labels with a frequency more than splitting threshold γ are kept in set L h and others in L t .
Revised: Labels with a frequency more than splitting threshold γ are kept in set L h and others in L t .

Original: The splitting threshold γ is chosen while ensuring that most of the features (or words) are covered in documents that one at least one instances of label in the set L h and | L h | < 0 .
Revised: The splitting threshold γ is chosen while ensuring that most of the features (or words) are covered in documents that one at least one instances of label in the set L h and | L h | < 0 .

Original: 2 M .
Revised: 2 M .

Original: Two components for DeepXML, DeepXML-h and DeepXML-t, are trained on L h and L t .
Revised: Two components for DeepXML, DeepXML-h and DeepXML-t, are trained on L h and L t .

Original: Please note that other strategies like clustering of labels, connected components of labels in a graph were also tried, but the above-mentioned strategy provides good results without any additional overhead.
Revised: Please note that other strategies like clustering of labels, connected components of labels in a graph were also tried, but the above-mentioned strategy provides good results without any additional overhead.

Original: More sophisticated algorithms for splitting such as label clustering, may yield better results, however at the cost of increased training time.
Revised: More sophisticated algorithms for splitting such as label clustering, may yield better results, however at the cost of increased training time.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Ground
Revised: Ground

Original: [Table]
Revised: [Table]

Original: what is diabetes type 2;what is type 2 diabetic diabetes type 1 and 2;type diabetes 1 and 2;type 1 and 2 diabetes, what causes type 2 diabetes
Revised: what is diabetes type 2;what is type 2 diabetic diabetes type 1 and 2;type diabetes 1 and 2;type 1 and 2 diabetes, what causes type 2 diabetes

Original: [Table]
Revised: [Table]

Original: diabetes type 2 deﬁnition
Revised: diabetes type 2 deﬁnition

Original: [Table]
Revised: [Table]

Original: 2 deﬁnition diabetes type
Revised: 2 deﬁnition diabetes type

Original: [Table]
Revised: [Table]

Original: deﬁnition diabetes type
Revised: deﬁnition diabetes type

Original: [Table]
Revised: [Table]

Original: deﬁnition
Revised: deﬁnition

Original: [Table]
Revised: [Table]

Original: diabetes
Revised: diabetes

Original: [Table]
Revised: [Table]

Original: what
Revised: what

Original: [Table]
Revised: [Table]

Original: diabetes
Revised: diabetes

Original: [Table]
Revised: [Table]

Original: diabetes
Revised: diabetes

Original: [Table]
Revised: [Table]

Original: what is type 2 diabetes, type 2 diabetes , what is type 2 diabetes symptoms, diabetes 2;2 diabetes;diabetes 2, whats type 2 diabetes, what is email marketing, what is type 1 and type 2 diabetes, what are the symptoms of diabetes type 2, what is anemia;anemia is what;anemia what is;what anemia is,what is psd2, what identity theft is, what is google adwords, what is radiation therapy, what is emilys list, what is pork belly, what is qualcomm, what is a smartboard, what is an unlocked phone, mpls, what is the invoice price
Revised: what is type 2 diabetes, type 2 diabetes , what is type 2 diabetes symptoms, diabetes 2;2 diabetes;diabetes 2, whats type 2 diabetes, what is email marketing, what is type 1 and type 2 diabetes, what are the symptoms of diabetes type 2, what is anemia;anemia is what;anemia what is;what anemia is,what is psd2, what identity theft is, what is google adwords, what is radiation therapy, what is emilys list, what is pork belly, what is qualcomm, what is a smartboard, what is an unlocked phone, mpls, what is the invoice price

Original: [Table]
Revised: [Table]

Original: what is type 2 diabetes, what is a type 2 diabetes , type 2 diabetes , what are type 2 diabetes , what is the type 2 diabetes, do i have type 2 diabetes , "whats type 2 diabetes", what is type 2 diabetes mellitus , diabetes type 2 deﬁnition
Revised: what is type 2 diabetes, what is a type 2 diabetes , type 2 diabetes , what are type 2 diabetes , what is the type 2 diabetes, do i have type 2 diabetes , "whats type 2 diabetes", what is type 2 diabetes mellitus , diabetes type 2 deﬁnition

Original: [Table]
Revised: [Table]

Original: 2 deﬁnition diabetes type;deﬁnition diabetes type 2;deﬁnition typediabetes
Revised: 2 deﬁnition diabetes type;deﬁnition diabetes type 2;deﬁnition typediabetes

Original: [Table]
Revised: [Table]

Original: diabetes deﬁnition type
Revised: diabetes deﬁnition type

Original: [Table]
Revised: [Table]

Original: type 2 diabetes deﬁnition , what is typeand type 2 diabetes
Revised: type 2 diabetes deﬁnition , what is typeand type 2 diabetes

Original: [Table]
Revised: [Table]

Original: what is type 2 diabetes, type 2 diabetes , what is a type 2 diabetes , do i have type 2 diabetes , what is the type 2 diabetes, what are type 2 diabetes , "what’s type 2 diabetes", what is type 2 diabetes mellitus , diabetes type 2 deﬁnition;2 deﬁnition diabetestype;deﬁnition diabetes type 2;deﬁnition type 2 diabetes;diabetes deﬁnition type 2;type 2 diabetes deﬁnition , what is typeand type 2 diabetes, what are the causes of diabetes type 2, types of diabetes 2, what is diabetes 2, what causes type 2 diabetes
Revised: what is type 2 diabetes, type 2 diabetes , what is a type 2 diabetes , do i have type 2 diabetes , what is the type 2 diabetes, what are type 2 diabetes , "what’s type 2 diabetes", what is type 2 diabetes mellitus , diabetes type 2 deﬁnition;2 deﬁnition diabetestype;deﬁnition diabetes type 2;deﬁnition type 2 diabetes;diabetes deﬁnition type 2;type 2 diabetes deﬁnition , what is typeand type 2 diabetes, what are the causes of diabetes type 2, types of diabetes 2, what is diabetes 2, what causes type 2 diabetes

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. C OMPARISON WITH B O W A PPROACHES
Revised: A. C OMPARISON WITH B O W A PPROACHES

Original: DeepXML, DeepXML-re gives 3 − 4% better accuracy on propsentity scored metrics and can be upto 2% better on vanila metrics with the exception of PfastreXML.
Revised: DeepXML, DeepXML-RE yields 3 − 4% better accuracy on propensity scored metrics and can be upto 2% more accurate on vanilla metrics.

Original: Please note thatDeepXML is able to outperform PfastreXML on vanilla scored metrics for every dataset ranging from 4% on
Revised: Note that PfastreXML outperform DeepXML and

Original: WikiSeeAlsoTitles-350K to 13% on WikiTitles-500K. Also, DeepXML-re is able to compete with the state-of-the-art Parabel with a marginal loss of 0 . 8% in precision @1 while being 5 . 5% better on propensity scored metrics. This demonstrates that DeepXML-re is able to predict correctly from more diverse set of tail labels while maintaining state-of-the-art accuracy.
Revised: DeepXML-RE on AmazonTitles-3M in propensity scored metrics, however suffers a substantial loss of 10% on vanilla precision and nDCG which is unacceptable for real world applications.

Original: A. E VALUATION METRICS
Revised: A. E VALUATION METRICS

Original: Performance has been evaluated using propensity scored precision@ k and nDCG@ k , which are unbiased and more suitable metric in the extreme multi-labels setting (Jain et al., 2016; Babbar &
Revised: Performance has been evaluated using propensity scored precision@ k and nDCG@ k , which are unbiased and more suitable metric in the extreme multi-labels setting (Jain et al., 2016; Babbar &

Original: Schölkopf, 2019; Prabhu et al., 2018a;b).
Revised: Schölkopf, 2019; Prabhu et al., 2018a;b).

Original: The propensity model and values available on The Extreme
Revised: The propensity model and values available on The Extreme

Original: Classiﬁcation Repository (Bhatia et al., 2016) were used.
Revised: Classiﬁcation Repository (Bhatia et al., 2016) were used.

Original: Performance has also been evaluated using vanilla precision@ k and nDCG@ k (with k = 1, 3 and 5) for extreme classiﬁcation.
Revised: Performance has also been evaluated using vanilla precision@ k and nDCG@ k (with k = 1, 3 and 5) for extreme classiﬁcation.

Revised: For a predicted score vector ˆ y ∈ R L and ground truth vector y ∈ { 0 , 1 } L :

Revised: [Equation]

Revised: Here, p l is propensity score of the label l proposed in (Jain et al., 2016).

Original: A. L ITERATURE SURVEY
Revised: A. L ITERATURE SURVEY

Original: Representations: Deep architectures such as CNN (Liu et al., 2017), MLP (Zhang et al., 2018), and
Revised: Representations: Deep architectures such as CNN (Liu et al., 2017), MLP (Zhang et al., 2018), and

Original: LSTM with Attention (You et al., 2018) have been applied to learn semantically and syntactically rich features.
Revised: LSTM with Attention (You et al., 2018) have been applied to learn semantically and syntactically rich features.

Original: Barring AttentionXML (You et al., 2018), these methods suffer from low accuracy, which is more prominent on tail labels indicating inept document representation for tail labels.
Revised: Barring AttentionXML (You et al., 2018), these methods suffer from low accuracy, which is more prominent on tail labels indicating inept document representation for tail labels.

Original: However, performance of AttentionXML also degrades for short text documents as discussed in section 4.3.
Revised: However, performance of AttentionXML also degrades for short text documents as discussed in section 4.3.

Original: Parabel :
Revised: Parabel :

Original: Parabel learns a hierarchy over labels to select hardest negatives for each labels and hence bring down the training cost to O
Revised: Parabel learns a hierarchy over labels to select hardest negatives for each labels and hence bring down the training cost to O

Original: ( Nd log L ) .
Revised: ( Nd log L ) .

Original: However, Parabel is designed speciﬁcally for sparse
Revised: However, Parabel is designed speciﬁcally for sparse

Original: BoW features and its performance degrades on low dimensional features.
Revised: BoW features and its performance degrades on low dimensional features.

Original: Slice : Negative sampling is a popular approach to reduce training complexity in extreme classiﬁcation setting.
Revised: Slice : Negative sampling is a popular approach to reduce training complexity in extreme classiﬁcation setting.

Original: Several strategies have been proposed in literature to select negatives labels for each instance Mikolov et al.
Revised: Several strategies have been proposed in literature to select negatives labels for each instance Mikolov et al.

Original: ; Reddi et al.
Revised: ; Reddi et al.

Original: (2018) or negative examples for each label (Yen et al., 2017; Jain et al., 2019;
Revised: (2018) or negative examples for each label (Yen et al., 2017; Jain et al., 2019;

Original: Prabhu et al., 2018b).
Revised: Prabhu et al., 2018b).

Original: Slice approach has been shown to scale to 100 million labels and more accurate than alternate approaches (Yen et al., 2017; Prabhu et al., 2018b) for low dimensional dense features.
Revised: Slice approach has been shown to scale to 100 million labels and more accurate than alternate approaches (Yen et al., 2017; Prabhu et al., 2018b) for low dimensional dense features.

Original: Slice uses a ANNS to select O ( NL log L ) hardest negative examples and train with selected examples only, thereby reducing the training cost to O ( Nd log L ) . 
Revised: Slice uses a ANNS to select O ( NL log L ) hardest negative examples and train with selected examples only, thereby reducing the training cost to O ( Nd log L ) . Slice also brings down the prediction cost to O ( d log L ) by evaluating only O (log L ) most probable labels selected from ANNS. A. A BLATION STUDY Experiments were carried out with several variations of DeepXML where labels were not splitted in head and tail labels.

Revised: Here are the different conﬁgurations:

Original: Slice also brings down the prediction cost to O ( d log L ) by evaluating only O (log L ) most probable labels selected from ANNS.
Revised: DeepXML-f : This variation refers to a word embedding layer, ReLU non-linearity, Dropout and a fully connected output layer DeepXML-fr : This variation refers to a word embedding layer, ReLU non-linearity, Dropout, a residual block and a fully connected output layer.

Revised: DeepXML-NS :

Revised: This variation refers to a word embedding layer, ReLU non-linearity, Dropout, a residual block and a classiﬁer trained via negative sampling (Mikolov et al., 2013).

Revised: DeepXML-SW :

Revised: DeepXML

Revised: can exploit sub-words features as proposed in (Joulin et al., 2017).

Revised: This variations refers to DeepXML with sub-word information.

Revised: Here, character tri-grams were added to the vocabulary in addition to unigrams.

Revised: DeepXML-ANNS :

Revised: This

Revised: variation

Revised: refers to a word embedding layer, ReLU non-linearity, Dropout, a residual block and a classiﬁer trained via Slice.

Revised: This version trains only with hardest negatives labels selected via ANNS (Jain et al., 2019).

Revised: However, the hardest negatives keeps changing as the document representations are being updated.

Revised: Hence, it requires ANNS graph to be trained multiple time.

Revised: Here, ANNS graph was updated after an interval of 5 epochs.

Revised: DeepXML-P : This variation

Revised: refers

Revised: to

Revised: a

Revised: word

Revised: embedding layer, ReLU non-linearity, Dropout, a residual

Revised: block

Revised: and a

Revised: shallow

Revised: tree based

Revised: classiﬁer

Revised: proposed in AttentionXML.

Revised: [Table]

Revised: ``citrus’’ ``fruits’’

Revised: backpropagates does not backpropagate

Revised: Hard negatives

Revised: Approx NN Search

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]