Paper:
Multi-label learning aims to automatically assign to an instance (e.g., an image or a document) the most relevant subset of labels from a large set of possible labels. The main challenge is to maintain accurate predictions while scaling efficiently on data sets with extremely large label sets and many training data points. We propose a simple but effective neural net approach, the Semantic Embedding Model (SEM), that models the labels for an instance as draws from a multinomial distribution parametrized by nonlinear functions of the instance features. A Gauss-Siedel mini-batch adaptive gradient descent algorithm is used to fit the model. To handle extremely large label sets, we propose and experimentally validate the efficacy of fitting randomly chosen marginal label distributions. Experimental results on eight real-world data sets show that SEM garners significant performance gains over existing methods. In particular, we compare SEM to four recent state-ofthe-art algorithms (NNML, BMLPL, REmbed, and SLEEC) and find that SEM uniformly outperforms these algorithms in several widely used evaluation metrics while requiring significantly less training time.
1 INTRODUCTION
The multi-label learning problem is to learn to predict potentially multiple relevant labels given an instance. Instances that have multiple labels naturally occur in many application domains, including multimedia information retrieval, tag recommendation, semantic scene classification, query categorization, gene function prediction, medical diagnosis, drug discovery, and marketing.
A popular approach to the multi-label learning problem is to embed the labels in a low-dimensional latent space via linear or local non-linear embeddings. The approach of Hsu et al. (2009) projects the label vectors to a random low-dimensional space, fits a regression model in this space, then projects these predictions back to the original label space. Balasubramanian & Lebanon (2012) use a sparsity-regularized least squares reconstruction objective to select a small set of landmark labels that are used to predict the remaining labels. Bi & Kwok (2013) take a similar approach, with a greatly decreased computation cost, by posing the problem of selecting the landmark labels as one of column subset selection and adopting the leverage score sampling approach (Boutsidis et al., 2009). Recently, Yu et al. (2014) and Jing et al. (2015) propose using trace norm regularization to identify a low-dimensional representation of the original large label space. Mineiro & Karampatziakis (2015) use randomized dimensionality reduction to learn a low-dimensional embedding that explicitly captures correlations between the instance features and their labels. These approaches, like other linear embedding methods, assume that the label matrix is low-rank. However, the label matrix in most applications of multi-label learning is a sparse binary matrix, and thus is extremely likely to violate this low-rank assumption (Bhatia et al., 2015).
Rather than working with the original label and feature matrices, some methods work instead with label or feature similarity matrices, and seek to preserve the local structure of the data in the learned low-dimensional latent space. Tai & Lin (2010) use PCA on the label covariance matrix to extract a low-dimensional latent space for labels and Chen & Lin (2012) extend this method to integrate feature information. Lin et al. (2014) apply PCA to a similarity matrix constructed using both label and feature information; this approach is time-consuming as it requires computing a large similarity matrix. Nam et al. (2014) introduce a neural network model to capture non-linear relationships between the input features and the labels. However, this approach is computationally infeasible when the number of possible labels is large. Similarly, Cissé et al. (2016) shows that using a deep learning approach built on top of an informative partitioning of the label space gives good performance; the scalability of this method was not characterized. Prabhu & Varma (2014) propose a method to efficiently train a classification tree by minimizing the Normalized Discounted Cumulative Gain. Rai et al. (2015) assumes that the label vectors are generated by sampling from a weighted combination of label topics, where the mixture coefficients are determined by the instance features.
Bhatia et al. (2015) proposes a multi-phase algorithm (SLEEC) that first clusters the instances into a number of relatively small groups, learns label embeddings for each group via an SVD, and then trains linear regressors from the input features to the latent label factors for each group. SLEEC empirically outperforms previous state-of-the-art multi-label classifiers, but the label embedding in each group is learned from a nearest neighbor graph that is constructed solely from labelling information, ignoring the available feature matrix; the feature matrix has been shown repeatedly to be a source of useful information for label embedding (Chen & Lin, 2012; Lin et al., 2014; Yu et al., 2014; Jing et al., 2015).
The contribution of this paper is a scalable, accurate, and simple neural network approach to multilabel learning. Experiments establish that our method is faster and more accurate than SLEEC, the current state-of-the-art scalable algorithm.
Notation: In the sequel, n is the number of training instances, c is the cardinality of the set of possible labels, d is the dimensionality of the feature vectors, and r is the dimension of the learned latent space. The matrix X ∈ Rn×d contains the instance features, and Y ∈ 0, 1n×c indicates the labels assigned to each instance. We denote the number of observed labels for instance i with `i = ∑c k=1 yik. The notations Ai· and A·j respectively refer to the ith row and jth column of the matrix A. Unless otherwise specified, the notation f(A) denotes the elementwise application of an arbitrary function f to the A, so for example exp(A)ij = exp(aij).
2 THE SEMANTIC EMBEDDING MODEL
Our Semantic Embedding Model (SEM) assumes that the underlying parameters determining the observed labels are low-rank rather than that the observed label matrix is itself low-rank, and it uses a nonlinear model to fit the probability distributions over the labels, conditioned on the instance features.
SEM models the i-th row of Y as the result of `i draws from a multinomial distribution: Yi· ∼ Multinomial(`i;Pi), where P = [
exp(hij)∑c k=1 exp(hik) ] i=1,...,n j=1,...,c . (1)
The parameter matrix H = UVT + 1nbT is the sum of label priors b ∈ Rc and the product of explanatory latent factors associated with the instances (U ∈ Rn×r) and the labels (V ∈ Rc×r). Further, we allow the latent factors associated with each instance to be a nonlinear function of the features associated with that instance, U = f(X,W) for some W to be learned. We note that if f(X,W) = XW, SEM could be viewed as fitting a Bayesian Exponential Family PCA (Mohamed et al., 2009). However, throughout this paper we take f(XW) = σ(XW), where σ(X) = (1 + exp(−X))−1 denotes the elementwise application of the sigmoid function, as we find this gives good results; with this choice, SEM is more naturally viewed as a neural network model.
We fit the SEM parameters by maximizing the likelihood of the observed labels. This is equivalent to minimizing the sum of the KL divergences between the empirical label distributions for each instance and the label distributions predicted by the model (Pawitan, 2001). Accordingly, we define the empirical label distribution matrix G, whose ith row satisfies Gi = Yi/`i, then minimize the
row-wise Kullback-Leibler distance (Yang et al., 2011) between G and P:
JG||P = n∑
i=1 c∑ j=1 Gij log Gij Pij ∼= − n∑ i=1 c∑ j=1 Gij logPij . (2)
Recalling that
Pij = exp(hij)∑c k=1 exp(hik) = exp
( (σ(XW)VT )ij + bj )∑c k=1 exp ((σ(XW)V T )ik + bk) ,
some algebraic manipulations give the final objective
J (W,V,b) = JG||P = − n∑
i=1 c∑ j=1 Gij log exp(σ(XW)i·(V T )·j + bj)∑c k=1 exp(σ(XW)i·(V T )·k + bk)
= − n∑
i=1 c∑ j=1 Gij(σ(XW)i·(V T )·j + bj) + n∑ i=1 log ( c∑ k=1 exp(σ(XW)i·(V T )·k + bk) ) = −Tr(G(σ(XW)VT + 1nbT )T ) + 1Tn log ( exp(σ(XW)VT + 1nb T )1c ) .
(3)
Thus the SEM parameters are learned by solving the optimization problem min
W,V,b J (W,V,b). (4)
Here V ∈ Rc×r are the representations of the labels in a latent semantic space, W ∈ Rd×r controls the nonlinear mapping from the instance features to the same semantic space, and the offsets b ∈ Rc allow for label-specific offsets in the mapping from the semantic space to the log probabilities.
3 MODEL FITTING
The optimization problem (4) is non-convex. To solve it efficiently, we use a Gauss-Siedel approach combined with mini-batching.
Namely, we cyclically update each of W,V, and b using AdaGrad (Duchi et al., 2011) while keeping the other two variable fixed. We compute the gradients using mini-batches. To state the expressions for the gradients with respect to the model parameters, we introduce some helpful notation: A B denotes the entry-wise product of two matrices, M = σ(XW) (1− σ(XW)), and D = Diag ( exp (σ(XW)VT + 1nb T )1c ) . The gradients are readily computed from (2):
G(W) = XT ( M [( D−1 exp ( σ(XW)VT + 1nb T ) −G ) V ])
(5) G(V) = ( exp ( σ(XW)VT + 1nb T )T D−1 −GT ) σ(XW) (6)
G(b) = ( exp ( σ(XW)VT + 1nb T )T D−1 −GT ) 1n. (7)
Using AdaGrad, the update rule for W(τ) is
W(τ) = W(τ−1) − α(τ)W G(W (τ−1)) (8)
where τ is the timestep and αW is a matrix of step sizes computed via( α (τ) W ) iq = ρ√∑τ−1
m=1
( G(W(m)) G(W(m)) ) iq + ε , (9)
where ε and the learning rate ρ determine how much an entry Wij is updated during the first timestep.
V(τ) and b(τ) are computed according to similar updating rules obtained from (8) and (9) by substituting G(W) with G(V) (or G(b)), W with V (or b), and αW with αV (or αb). A listing of the proposed algorithm is given in Algorithm 1. Its computational complexity is O(Tnr(d + c)), where T is the number of epochs. We note that the gradient calculations in lines 7–9 of Algorithm 1 are amenable to parallelization.
Algorithm 1 Mini-Batched Gauss-Siedel Adaptive Gradient Descent for learning SEM parameters Input: Instance feature matrix X ∈ Rn×d, observed label matrix Y ∈ Rn×c, dimensionality of the
latent space r, learning rate ρ and > 0, mini-batch size m < n, and number of epochs T . 1: Initialize W(0), V(0), and b(0) 2: for t = 1, 2, . . . , T do 3: Randomly choose n/m mini-batchs Iz ⊂ {1, ..., n} of size m 4: for b = 1, 2, . . . n/m do 5: Set τ = (t− 1)(n/m) + b 6: Select the data instances in the z-th mini-batch by working with XIz,· in lieu of X 7: Update W(τ) via (8) while fixing V = V(τ−1) and b = b(τ−1) 8: Update V(τ) via the analog of (8) for V while fixing W = W(τ) and b = b(τ−1)
9: Update b(τ) via the analog of (8) for b while fixing W = W(τ) and V = V(τ) 10: end for 11: end for Output: W(τ), V(τ) and b(τ)
3.1 INCREASED EFFICIENCY BY FITTING MARGINALS
Although Algorithm 1 runs in time linear in the dimensions of the model parameters and the input datasets, it can be computationally expensive when there are more than a few thousand labels. To further reduce the running time of our algorithm, we note that in practice, each instance is often associated with `i c labels. To speed up the training, at each timestep τ , rather than attempting to minimize the divergence between the entire empirical and predicted label distributions of each instance, we sample a set of labels L(τ)i for each instance and attempt to minimize the empirical and predicted marginal label distributions over that set of labels L(τ)i . Let PLi denote the set of labels assigned to the ith instance, and ALi denote the set of labels not assigned to that instance. We sample from ALi to form a set NLi, and use L (τ) i = PLi ∪ NL (τ) i . This leads to the modified objective
JMarginal(W,V,b)(τ) = − n∑ i=1 ∑ j∈L(τ)i Gij log exp(σ(XW)i·(V T )·j + bj)∑ k∈L(τ)i exp(σ(XW)i·(VT )·k + bk) . (10)
Note that J (τ) is a random function that changes at each timestep. Minimizing this stochastic objective effectively seeks SEM parameters which fit all the randomly sampled marginals encountered during training. Thus it is important to sample the sets NLi so that the selected marginals capture non-trivial information about the label distributions. One can imagine that uniformly sampling from ALi will not provide very informative marginals. As an improvement on this naïve scheme, we sample labels from ALi with probability proportional to their frequency of occurrence in the training data set. The number of negative labels is set to be β times the number of positive labels i.e., |NLi| = β|PLi| = β`i. Further, when m > 1, to faciliate efficient BLAS operations while mini-batching, we use the same marginals for each instance in the same minibatch, i.e., we fit marginals over L(τ) := ⋃ i∈Iz L (τ) i , where Iz denotes the set of instances in the current minibatch.
In the experiments presented in Section 4, we found that β around 10 suffices when c is relatively small, and β around 100 suffices when c is on the order of tens of thousands.
3.2 LABEL PREDICTION
We present two methods for predicting the labels for a new instance x ∈ Rd given the fitted SEM parameters.
The first uses the generative model behind SEM: form h = σ(xTW)VT+bT and note the probability that the jth label is assigned to that instance is given by
P(yj = 1) = exp(hj)/ c∑
k=1
exp(hk). (11)
Accordingly we assign the most probable labels to x. We call this prediction scheme the direct SEM method; it simply requires choosing the labels corresponding to the largest entries of h.
The second method builds a kernel classifier in the semantic space obtained from the SEM factorization. Following Mineiro & Karampatziakis (2015), a classifier is trained on these semantic representations by solving the optimization problem
min Z∈Rc×s n∑ i=1 `(Yi·,Zψ(xiW)) + λ‖Z‖2F , (12)
where ` is the log-loss penalty and ψ : Rr → Rs is an s-dimensional Random Fourier Feature (RFF) map (Rahimi & Recht, 2007):
ψ(x) = cos (Φx + θ) , (13)
where Φ ∈ Rs×r is a matrix of i.i.d. standard Gaussians and θ ∈ [0, 2π)s is a vector of i.i.d uniform samples from [0, 2π).
At test time, the predicted label probabilities for an instance x are given by Zψ(xW), so we assign the most probable labels according to this model. We refer to this scheme as the kernelized SEM method.
4 EXPERIMENTS
In the sequel we refer to the direct SEM scheme as simply SEM, and the kernelized SEM scheme as SEM-K. We compare SEM and SEM-K with several alternative multi-label learning algorithms: NNML (Nam et al., 2014), REmbed (Mineiro & Karampatziakis, 2015), SLEEC (Bhatia et al., 2015), and BMLPL (Rai et al., 2015). We do not compare to the models proposed in (Tai & Lin, 2010; Chen & Lin, 2012; Bi & Kwok, 2013; Yu et al., 2014; Prabhu & Varma, 2014) because earlier works (Yu et al., 2014; Bhatia et al., 2015) have shown that they are inferior to SLEEC.
4.1 DATASETS
Table 1 summarizes the eight datasets used in our experiments. Here ntrain and ntest are the numbers of training and testing instances, d is the number of features, c is the number of labels/classes, and the avg(`i) column reports the average number of labels per instance. In these datasets, the number of labels varies from 23 to 30938, the average label cardinality varies from 2.508 to 19.020, and the number of instances in different classes varies over a large range. Thus predicting the labels assignments correctly over this collection of datasets is a challenging task.
4.2 METHODOLOGY
The codes of the methods we compare to are provided by the authors, in particular, we note that the computationally intensive portions of REmbed, SLEEC and NNML are implemented in C; by way of comparison, our algorithms are entirely implemented in Matlab. Due to there being several parameters for each method, we hand-tuned the parameters for each dataset as suggested by the authors. All methods were run in MATLAB on a Windows server with 4GB memory and four 2.3GHz CPUs with eight cores.
The prediction performance for each algorithm is evaluated according to widely-used metrics in the field of multi-label classification, viz., label-based Macro-F1 (MaF1) and Micro-F1 (MiF1) and instance-based Precision-at-k (P@k, esp. P@1 and P@3) (Zhang & Zhou, 2014). MaF1 and MiF1 require predefining a threshold to determine the number of labels to be assigned to the testing data. In our experiments, the number of labels assigned to each testing instance was set according to its ground truth.
4.3 PERFORMANCE ON DATASETS WITH SMALL LABEL SETS
First we compare the performance on six multi-label learning problems with c < 1000. To fit both SEM models, we take the number of epochs be 30 and the mini-batch size be 200—i.e., T = 30 and m = 200 in Algorithm 1—and because c is small, we fit the full label distributions. The classification performances of our SEM algorithms and the baseline methods are shown in Table 2. SEM or SEM-K outperform the alternative algorithms in most cases.
Table 3 compares the running times of the algorithms as the size of the dataset is increased, using MediaMill. We see that SEM is the fastest model, followed by REMBED, then closely by SEM-K; the remaining three models are significantly more costly. It is clear that NNML, the previous neural
network approach to multi-label learning costs the most. In the other five algorithms, the latent space dimensionality (r) is set to be 50. SLEEC is expensive because it constructs the nearest neighbor graph among training data and computes the top r eigenvectors of the corresponding similarity matrix, which costs O(n2r + d2r). REmbed is efficient because its main cost is to find the singular vectors of a c× (r + q) matrix (here c is the number of labels and q is a small integer), but its performance is inferior to SEM-K. The BMLPL code provided by the author applies SVD to the training data to initialize than model parameters and then uses conjugate gradient to update the parameters, thus it costs much more than REmbed and our proposed methods.
4.4 PERFORMANCE ON DATASETS WITH LARGE LABEL SETS
We proposed using SEM to fit marginals rather than the entire label distribution when c is large, for computational efficiency. To judge the effectiveness of this proposal, we compare the accuracy and running times of the SEM and SEM-K models with baselines on EurLex-des and Wiki10K, two datasets with c > 1000. As baselines, we use REmbed and SLEEC in accordance with the above discussion which showed that these two methods are efficient and/or have good performance.
The hyperparameters in SLEEC were set according to the original authors’ code: r for EurLex-des and Wiki10K is 100 and 75 respectively, and 3 clusters are used for Eurlex-des and 5 are used for Wiki10K. To fit the SEM models, we used the same value of r as SLEEC on these two datasets and used 10 training epochs. For REmbed, the latent space size r was tuned via cross-validation; r = 300 for Eurlex-des and r = 150 for Wiki10K. The number of Random Fourier Features is 2000 for both REmbed and SEM-K. The latent space size r in SEM is same with SLEEC. The mini-batch sizes and number of epochs are set to be 200 and 10 respectively when fitting the SEM models. The number of threads is set to be 8 for all methods.
Table 4 compares the classification performances of the methods on these two datasets. It is clear that SEM-K with a small set of negative labels obtains better performance than both REmbed and SLEEC. Table 5 shows that, additionally, the SEM-K models are fit much faster than than the other models.
4.5 IMPACT OF THE SIZE OF THE MARGINALS
Figure 1 illustrates the impact of the choice of β on the prediction performance (in terms of P@1) of SEM and SEM-K. The performances of SLEEC and REmbed are included for comparison. The hyperparameters of SLEEC, REmbed and SEM were set as in Section 4.4.
It is evident that the performance of SEM increases significantly in a monotonic fashion with β. However, SEM-K is insensitive to β once it passes a dataset-dependent threshold (e.g., β = 60
for Eurlex-des and β = 100 for Wiki10K). Note that on Wiki10K, even the simpler direct SEM outperforms REmbed when there are sufficient negative labels.
Figure 2 illustrates the effect of β on the running times of SEM and SEM-K. Note that the additional time to fit the classifier in the semantic space required by SEM-K is negligible compared to the time it takes to first fit the direct SEM model.
4.6 ADDITIONAL CONSIDERATIONS
There are other important ways in which the proposed SEM methods can be compared to the baseline multi-label learning methods, including their performance as a function of the latent space dimensionality and as a function of the amount of training. Due to space constraints, a discussion of these two concerns and the convergence behavior of Algorithm 1 is provided in the Supplementary material.
5 CONCLUSION
We proposed a new semantic embedding model (SEM) for handling the multi-label learning task. A framework based on Gauss-Siedel mini-batched adaptive gradient descent was proposed for efficiently solving the non-convex optimization problem required to learn the SEM parameters. For large label sets, we proposed fitting the SEM to marginal distributions rather than the full label distribution. A series of experiments on eight real-world datasets empirically demonstrated that the proposed method is superior to state-of-the-art methods in terms of prediction performance and running time.
A. Effect of Latent Space Dimensionality
It can be seen that the latent space dimensionality r plays an important role to learn latent factors V and a feature mapping matrix W in our proposed methods, as it does in the three baselines BMLPL, REmbed and SLEEC. In order to investigate this dependence, we conducted a series of experiments on the training data sets using 5-fold cross-validation, comparing BMLPL, REmbed, SLEEC and our proposed SEM and SEM-K.
In this experiment, we take Delicious dataset as an example. The training data is separated into five folds where four folds are used as training and one fold as validating, and the averaged results in terms of P@1 and MiF1 are given by Figure 3. It can be seen that their performances usually improve with increasing r until they reach an optimum value. However, once r becomes too large, their performances degrade. This is reasonable: when r is too small, the learned parameters cannot fully characterize the hidden semantic structure in the classification problem, while when r is too large, the benefits of dimensionality reduction are lost, as the model begins to over-fit to the idiosyncrasies of the training data rather than capturing the semantic structure common to both the training and validation data. Usually, these methods could obtain good performance at small r, say 45 for Delicious dataset.
B. Effect of training data size
Meanwhile, we studied the label prediction performance as a function of the amount of labeled training data. In this experiment, we fixed the testing data size, and randomly selected training data from the training set so that the training data size varies from 1% to 70% of the combined training and testing data. In order to avoid the presence of empty categories and instances with no labels, at least one instance is kept for each label and at least one label is kept for each instance during this sampling process. For each fixed size of the training set, the desired amount of data is randomly
sampled ten times, and the resulting average P@1 and MiF1 on the testing data are recorded. During training, the latent dimensionality parameter r is selected via 5-fold cross-validation.
Figure 4 shows these results for the Mediamill dataset which contains the largest number of instances. As expected, the performance of all the methods is positive correlated with the size of the training data set, and we also see that the proposed SEM-K uniformly outperforms the other methods regardless of the training data size. As it is often expensive to obtain large labeled data sets in real applications, this observation suggests that SEM-K is a better choice for these situations.
C. Convergence
In order to demonstrated the convergence of the proposed method, we show the value of objective function (4) (at r = 45) via Figure 5(a) and the prediction result (P@1) via Figure 5(b) along with the number of passes to the dataset (i.e., τ in Algorithm 1). It can be seen that SEM could be convergent and the prediction performance becomes stable in less than 50 epochs, which will leverage SEM dealing with large-scale data.

Review:
This paper proposes SEM, a simple large-size multilabel learning algorithm which models the probability of each label as softmax(sigmoid(W^T X) + b), so a one-layer hidden network. This in and of itself is not novel, nor is the idea of optimizing this by adagrad. Though it's weird that the paper explicitly derives the gradient and suggests doing alternating adagrad steps instead of the more standard adagrad steps; it's unclear whether this matters at all for performance. The main trick responsible for increasing the efficiency of this model is the candidate label sampling, which is done in a relatively standard way by sampling labels proportionally to their frequency in the dataset.
Given that neither the model nor the training strategy is novel, it's surprising that the results are better than the state-of-the-art in quality and efficiency (though non-asymptotic efficiency claims are always questionable since implementation effort trades off fairly well against performance). I feel like this paper doesn't quite meet the bar.