Multi-label Learning with Random Circular Vectors (2024)

Abstract: The extreme multi-label classification (XMC) task involves learning a classifier that can predict from a large label set the most relevant subset of labels for a data instance. While deep neural networks (DNNs) have demonstrated remarkable success in XMC problems, the task is still challenging because it must deal with a large number of output labels, which make the DNN training computationally expensive. This paper addresses the issue by exploring the use of random circular vectors, where each vector component is represented as a complex amplitude. In our framework, we can develop an output layer and loss function of DNNs for XMC by representing the final output layer as a fully connected layer that directly predicts a low-dimensional circular vector encoding a set of labels for a data instance. We conducted experiments on synthetic datasets to verify that circular vectors have better label encoding capacity and retrieval ability than normal real-valued vectors. Then, we conducted experiments on actual XMC datasets and found that these appealing properties of circular vectors contribute to significant improvements in task performance compared with a previous model using random real-valued vectors, while reducing the size of the output layers by up to 99%.

## introduction extreme multi-label classification (xmc) problems arise in various domains, such as product recommendation systems @xcite , labeling large encyclopedia @xcite , instance-level image recognition @xcite and natural language generation @xcite . the xmc task involves learning a classifier which can predict from a large label set the most relevant subset of labels for a data instance. recent work has focused on deep neural network (dnn) models @xcite @xcite @xcite that deliver task performances superior to those of early approaches using linear predictors @xcite . while dnn models have brought great performance improvements, the xmc task still remains a challenge mainly due to the extremely large output space. since a large number of output labels make it difficult to train dnn models efficiently, various methods for improving training efficiency have been proposed @xcite @xcite . among the previous studies, @xcite presented a promising method that employs random real-valued vectors for reducing the output layer size of dnn models. in this approach, a high-dimensional output space vector is replaced with a low-dimensional random vector encoding the relevant label information for a data instance. then, dnn models are trained to predict the labelencoded vector directly. after the model generates a vector, it can be checked approximately whether a label is encoded in it or not through a vector comparison using the cosine similarity between the output vector and a vector that the label is assigned to. the basic idea of the label encoding and retrieval framework relies on the theory of holographic reduced representations @xcite , which was developed in the cognitive neuroscience field. however, random real-valued vectors do not have sufficient ability for representing data instances that belong to many class concepts. as our experiments in § 3 show, the label retrieval accuracy decreases markedly as the number of class labels encoded in a vector increases. to alleviate the issue, this paper presents a novel method that uses circular vectors instead of real-valued vectors. each element of a circular vector takes a complex amplitude as its value; i.e., the vector element is represented by an angle ranging from -π to π. since an angle can be represented by a real value, the memory cost for the circular vector representation is the same as that for a normal real-valued vector. in spite of this fact, surprisingly, circular vectors have better label encoding and retrieval capacities than real-valued vectors. one of the challenges in applying circular vectors to dnn models is how to adapt the output layer to a circular vector. in § 4, we describe our neural network architecture that uses circular vectors in the output layer. our experimental results on xmc datasets show that our method based on circular vectors significantly outperforms a previous model using real-valued vectors, while reducing the size of the output layers by up to 99%. ## previous study: learning with holographic reduced representations several vector symbolic architectures have been developed in the field of cognitive neuroscience, including tensor product representations @xcite , binary spatter code @xcite , binary sparse distributed representations @xcite , multiply-add-permute @xcite , and holographic reduced representations (hrr) @xcite . among them, hrr is a successful architecture for distributed representations of compositional structures. to model complex structured prediction tasks in a vector space that involve key-value stores, sequences, trees and graphs, many prior studies have explored how to use hrr in various machine learning frameworks; recurrent neural networks @xcite , tree kernels (zanzotto and dell'arciprete, 2012), knowledge graph representation learning @xcite , long-short term memory networks @xcite , transformer networks @xcite , and among others. in particular, @xcite presented a general framework based on the hrr architecture for efficient multi-label learning of dnn models. to clarify the motivation of our study, we will review the framework in more detail in the following subsections. ## multi-label representations with circular vectors in this section, we show through experiments that random real-valued vectors actually do not have sufficient ability for representing data instances that belong to many classes. the reason is mainly due to the projection operation in equation 5. as described in § 2, the projection operation was proposed as a way to reduce the effect of the variance of the similarity computation, but each element of the superposition between two normalized vectors via the projection is no longer unitary. thus, the effect of the projection decreases when a label vector encodes more class labels. to alleviate the issue, we developed a simple alternative that forces all vector elements to be unitary in the complex domain even after the superposition operation. we describe the details in the following subsection. ## neural network architecture one of the challenges in adapting chrr to xmc tasks is how to adapt the output layer of dnn models to a circular vector because it has a cyclic feature; i.e., θ = 2πn × θ, where @xmath0. to meet it, we developed a neural network for predicting angles that considers the cyclic feature during the training. the key idea was to represent the output in cartesian coordinates, which can uniquely represent a point on a unit circle. then, we converted the output into polar coordinates to obtain angles. ## experiment on xmc datasets to examine the advantages of circular vectors, we conducted experiments on several xmc datasets. mance on xmc datasets was not the goal of this study, which focuses on the efficiency of the learning method with circular vectors. however, to validate the effectiveness of chrr in the xmc task, we compared our method with several strong baselines. these include treebased fastxml @xcite , pfas-trexml @xcite , and deep learning based xml-cnn @xcite , in addition to fc and hrr. ## conclusion the xmc task still faces challenge of dealing with a large number of output labels. in this paper, we attempted to address this issue by using a low dimensional circular vector to output directly. in theoretical experiments in § 3.2 and § 3.3, we showed that many labels can be accurately encoded by using circular vectors (chrr) rather than normal real-valued vectors (hrr). moreover, using actual xmc datasets, we compared chrr with baseline methods in § 5. chrr reduced the output layer size by up to 99% compared to fc, while it outperformed other baselines in most results. comparing hrr and chrr, chrr outperformed on most results. in the future, we will incorporate circular vector systems into other dnn models such as lstm @xcite and transformer @xcite , as well as associative lstm @xcite and hrrformer @xcite .