Structural Contrastive Representation Learning for Zero-shot Multi-label Text Classification (2022)

Abstract: Zero-shot multi-label text classification (ZMTC) is a fundamental task in natural language processing with applications in the cold start problem of recommendation systems. Ideally, one would learn an expressive representation of both input text and label features so that ZMTC is transformed into a nearest neighbor search problem. However, the existing representation learning approaches for ZMTC struggle with accuracy as well as poor training efficiency. Firstly, the input text is structural, consisting of both short title sentences and long content paragraphs. It is challenging to model the correlation between short label descriptions and long structural input documents. Secondly, the enormous label space in ZMTC forces the existing approaches to perform multi-stage learning with label engineering. As a result, the training overhead is significant. In this paper, we address both problems by introducing an end-to-end structural contrastive representation learning approach. We propose a randomized text segmentation (RTS) technique to generate high-quality contrastive pairs. This RTS technique allows us to model title-content correlation. Additionally, we simplify the multi-stage ZMTC learning strategy by avoiding label engineering. Extensive experiments demonstrate that our approach leads to up to 2.33% improvement in precision@1 and 5.94x speedup in training time on publicly available datasets. Our code is available publicly.

## introduction zero-shot multi-label text classification (zmtc) @xcite @xcite @xcite defines the following problem: given a set of documents with no labels and the full label description for each class, we would like to correctly classify unseen documents to these classes. zmtc approaches can be leveraged to solve the cold start problem in e-commerce systems @xcite . for instance, we can accurately retrieve newly added products with learning-based retrieval systems. with zmtc, we do not have to worry about whether we have enough supervised data for the retrieval system. without retraining the semantic matching model, zmtc is capable of mapping between the customer query and its matched product descriptions even if they are recently added. challenges of zmtc: we observe two major challenges in zmtc. learning the mapping between input text and its associated class descriptions is hard. in practice, both input text and class description can be a structural document with a short title sentence and a long descriptive paragraph @xcite . as a result, representation learning of multi-categorical input text and class description is likely hard. secondly, the label space is enormous. practitioners deploy zmtc to tasks with number of classes in millions or even billions @xcite @xcite . the explicit zero-short learning approaches that require learning softmax classifiers @xcite would become prohibitive due to the expensive overhead in computing billion-scale embeddings. exploiting representation learning in zmtc: one ideal way of tackling zmtc is to transform it into a nearest neighbor search problem. by learning meaningful representations for both input and class text, the zmtc becomes a similarity search over embeddings. however, the existing representation approaches do not tackle the two major zmtc challenges completely. the current zmtc methods @xcite divide the structural document into a set of sentences. next, they perform sentence-level representation learning by modeling the pairwise similarity of input and label documents. this representation learning method neglects the paragraph-level information and the structural relationship between the title and contents. moreover, the current zmtc approaches index the label space into clusters @xcite , trees @xcite or graphs @xcite to reduce the computation. however, this procedure results in multi-stage training, which is generally hard to optimize end-to-end for zmtc training. our proposal: this paper proposes an end-toend structural contrastive representation learning approach for zmtc. we propose a novel randomized text segmentation (rts) method. we start by creating random chunks of the document contents into subsequences. next, we pair the generated chunks with the document title as well as other chunks from the description to form positive pairs. the pairs are then used for contrastive representation learning. our novel approach of combining titles and text introduces a data-dependent way that trains the model to associate segments of the description with the title. as a result, the relationship between the short title sentence and the long content paragraphs is baked into the representation. we can think of it as a novel self-supervised auxiliary task. this method allows us to learn the representation without label engineering. in other words, we transform zmtc into learning the similarity between different types and modalities of text within documents. specifically, our proposal enables us to represent the long paragraph with random subsequence sampling. our extensive experiments indicate that our approach leads to up to 2.33% improvement in precision@1 and 5.94× speedup in training time on state-of-the-art large-scale zmtc benchmarks. ## methodology in this section, we introduce our proposed structural contrastive learning approach for zmtc. we start with our problem settings. next, we introduce our approach of representation learning for structural text with title and content. finally, we highlight the proposed randomized text segmentation with more intuition. ## experiment in this section, we evaluate the performance of our method and compare it with competitive baselines on 4 zmtc datasets. there are 2 product recommendation datasets, 1 article recommendation dataset, and 1 article categorization dataset in our experiment. we choose these datasets for simulating the cold start problem in large-scale recommendation systems, information retrieval tasks in search engines, and natural language processing of unseen documents. in the experimental evaluation, we would like to answer the following questions: ## conclusion in this paper, we proposed randomized text segmentation (rts) and positive pair construction strategies to exploit the structure within document data for end-to-end contrastive learning to advance state-of-the-art results on zmtc tasks. our proposed method achieves up to 2.33% improvement on precision@1 and up to 5.94× speedup in training time over previous state-of-the-art. we show that it is feasible to efficiently train high-quality models for challenging zmtc tasks without having to resort to time-consuming, multi-stage methods with label engineering or methods that utilize inefficient softmax learning. through extensive ablation experiments, we demonstrate the superiority of rts over naive segmentation methods, and show that the types of positive pairs we proposed are indeed effective for learning better representation. we believe our work has a substantial impact as it can be applied to tackle many large-scale realworld problems such as cold-start recommendation problems, information retrieval, and medical document categorization and classification.