Open Vocabulary Extreme Classification Using Generative Models (2022)

Abstract: The extreme multi-label classification (XMC) task aims at tagging content with a subset of labels from an extremely large label set. The label vocabulary is typically defined in advance by domain experts and assumed to capture all necessary tags. However in real world scenarios this label set, although large, is often incomplete and experts frequently need to refine it. To develop systems that simplify this process, we introduce the task of open vocabulary XMC (OXMC): given a piece of content, predict a set of labels, some of which may be outside of the known tag set. Hence, in addition to not having training data for some labels–as is the case in zero-shot classification–models need to invent some labels on-thefly. We propose GROOV, a fine-tuned seq2seq model for OXMC that generates the set of labels as a flat sequence and is trained using a novel loss independent of predicted label order. We show the efficacy of the approach, experimenting with popular XMC datasets for which GROOV is able to predict meaningful labels outside the given vocabulary while performing on par with state-of-the-art solutions for known labels.

## introduction extreme multi-label classification (xmc) aims at predicting a set of labels for a given input instance from an extremely large labels set @xcite @xcite . examples for applying extreme classification are labeling a new article with wikipedia's categories, classifying a product with catalog labels, classifying a resume into a collection of pertinent job titles. despite the the scale of the label space, it is challenging to a priori capture all the possible ways in which an input instance can be categorized, especially at the industrial scale. real-world e-commerce platforms, for instance, contain billions of products from thousands of different categories that are continuously updated by human curators for all sort of reasons (e.g., see category changes in ebay ( 2021 )). in this work we introduce the open vocabulary xmc task, where we measure the ability of models to go beyond the given vocabulary and automatically propose new labels that might complement the existing ones and fill gaps in the vocabulary. note that this differs from a zero-shot formulation of the xmc problem @xcite where, although no training instance is available for some labels, they are still present in the given vocabulary. to tackle the problem we propose groov, an autoregressive model that maps input sequences to a set of sequences. inputs are documents/text, and outputs are collections of textual labels from an open vocabulary. we investigate multiple sequenceto-set-of-sequences instantiations, in particular an off-the shelf approach based on a encoder-decoder language model (t5, @xcite ) and a variant that uses a modified softmax function (i.e., multi-softmax) that does not penalize the model for assigning high probability to any gold label. this latter version inherently treats the target as a set of sequences (instead of a flat sequence) and outperforms the off-the shelf approach. to evaluate the out-of-vocabulary behaviour, we use popular xmc datasets for which a portion of the test labels do not appear in the train set. differently from previous works, we assume models are unaware of such labels (i.e., they don't appear in the given label vocabulary) and need to find them with open-ended text generation. we show that groov can indeed generate some of these labels while being competitive with state-of-the-art results on in-vocabulary metrics. in summary, the key contributions of this work are as follows: ## related work traditionally extreme multi-label classification is done by the one-vs-all method. one-vs-all methods such as dismec @xcite , proxml @xcite , pdsparse @xcite , and ppdsparse @xcite , which treat each label as a binary classification problem, can achieve acceptable performance. one-vs-all methods suffer from computationally expensive complexity and large model size. also, the classification tasks are independent of each other, and label dependency is not directly modeled. the high computational complexity in one-vs-all methods can be further improved by incorporating different partitioning techniques on the label spaces. for instance, parabel @xcite partitions the labels through a balanced 2-means label tree using label features constructed from the instances. recently, several approaches have been proposed to improve parabel. bonsai @xcite relaxes two main constraints in parabel; allowing multi-way instead of binary partitionings of the label set at each intermediate node and also removing strict balancing constraints on the partitions. slice @xcite considers building an approximate nearest neighbor (ann) graph as an indexing structure over the labels. for a given instance, the relevant labels can be found quickly from the nearest neighbors of the instance via the ann graph. these models rely on sparse features engineered from the text, which is cumbersome and, most importantly, doesn't benefit from the knowledge of pre-trained lms. moreover, the partitioning of the label space is done independently from the classifier's training. in this paper, we leverage pre-trained language models and show that generative models efficiently partition the label space, token by token, and there is no need for crafting a tree of labels separate from the classifier. deep learning models have improved extreme multi-label classification by learning better text representation from raw text. but the main challenge to these methods is how to couple with millions of labels. attentionxml @xcite shows success in extreme multi-label classification, overpassed all traditional machine learning methods, and proved the superiority of the raw text compared to sparse features. attentionxml uses a label tree, and a new classification model is trained for each layer of this tree that makes inference slow in predicting. x-transformer @xcite ) only uses pre-trained lms to match the label clusters for a given raw text and then ranks these labels by linear classifications with the sparse features. x-transformer is the first method of using pre-trained lms in extreme multi-label classification. due to the high computational complexity of transformer models, it only fine-tunes transformer models as the label clusters matcher, which can not fully exploit the power of transformer models. recently, @xcite showed that seq2seq auto-recursive models using pre-trained models could effectively partition and traverse a set of large labels by generating tokens incrementally. in extreme multi-label classification, the output is a set of labels. turning the set to a sequence of labels requires an ordering among labels, which might not be straightforward in many applications. @xcite shows that this ordering can significantly impact the performance. authors in @xcite propose an rl-based approach to relax the need for a fixed ordering among labels. we propose using a multi-softmax to relax the need for a fixed ordering which is much easier to train and implement than rl algorithms. another advantage of our work to other set-output methods is that we model the multi-label classification as a set of sequences of tokens instead of a set of label identifiers. therefore, we leverage more effectively the lm's knowledge in understanding each label. @xcite tackles the problem of zeroshot learning in extreme multi-label classification in which it tags each input with a set of labels consisting of both seen and unseen labels during the training. not only do we build an effective and efficient zero to few-shot learning, but we also want to go beyond that and tackle the problem of open vocabulary classification in which the taxonomy is not known to us entirely. related to the open vocabulary extreme classification is the open set recognition @xcite in the computer vision community. models proposed to solve the open set recognition have a different signature from our work. they define novel classes only in terms of sets of data points and do not generate names for classes that could then be compared against the true labels in a test set. also, they operate only on images, and the methods' generalization to other modalities is not examined. similar in spirit, @xcite generates hashtags for microblogs and measures the ability of their model in generating new hashtags. the authors use a gru-based dual encoder to generate hashtags. while there are similarities, our work is first in studying large generative pretrained lm for open vocabulary extreme tagging by jointly modeling all golden labels using a novel loss (multi-softmax). ## open-vocabulary tagging consider n training data points {(x i , y i )} @xmath0..n where x i is the text corresponding to the i-th instance and y i ⊆ y * is the set of tags x i was annotated with. importantly, we consider the set of all possible tags y * to be unknown both at training and inference time. we do assume, however, that each tag l @xmath1, that is by a sequence of tokens, tok(l k ) = {t k,j } @xmath2..len(l k ) . lastly let y @xmath3. throughout this work we will pay special attention to labels outside of this set, which we refer to as unseen labels. the above presented formulation of the topic tagging task is incompatible with currently prevalent xmc paradigms in several ways: first, most traditional classifiers require not only y * to be known in advance, but assume that for each label l k there are some examples tagged with l k so that a classifier can be learned for that particular label. these methods often don't rely on the label representation tok(l k ) itself. second, more recent zero-shot work @xcite ) makes tagging possible even for previously unencountered labels y k ̸ ∈ y seen . to our best knowledge, all of these methods rely on access to y * in order to build some kind of index using label features. finally, current datasets have their limitations too: @xcite and @xcite highlight that as the set of possible label grows it is unrealistic to expect that human annotators consider every single possible label in y * when annotating a document, thus we can expect all extreme classification datasets to be generally under-annotated. as we will see in section 7 this hinders our ability to measure the precision of any open vocabulary tagging system. in the following section we introduce a novel class of models that is particularly well-suited for exploring the whole label space y * while maintaining good performance on the set of known labels y seen . ## model below we illustrate how to frame oxmc as seq2seq problem, propose a loss captures the setnature of label sets more directly and then show how individual labels in the sets can be scored. ## quantitative results first, this section looks at our model's overall performance on xmc, considering all the tags. then, we look at the out of vocabulary performance by relaxing the definition of label matching to account for semantically similar labels with different surface forms.