M ODELING L ABEL S PACE I NTERACTIONS IN M ULTILABEL C LASSIFICATION USING B OX E MBEDDINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
Multi -label classification is a challenging structured prediction task in which a set of output class labels are predicted for each input
Real world datasets often have natural or latent taxonomic relationships between labels making it desir - able for models to employ label representations capable of capturing such taxonomies
Most existing multi-label classification methods do not do so , resulting in label predictions that are incoherent w .r t the taxonomic constraints, thus failing to accurately represent the fundamentals of the problem setting.
In this work we introduce the multi label box model MBM) a multi-label classification method that combines the encoding power of neural networks with the inductive bias and probabilistic semantics of box embeddings (Vilnis et al., 2018), which can be understood as trainable Venn-diagrams based on hyper-rectangles
Representing labels by boxes rather than vectors MBM is able to capture taxonomic relations among labels
Furthermore, since box embeddings allow these relations to be learned by stochastic gradient descent from data, and to be read as calibrated con - ditional probabilities , our model provides a high degree of interpretability.
This interpretability also facilitates the injection of partial information about label-label relationships into model training, to further improve its consistency.
We provide theoretical grounding for our method and show experimentally the model’s ability to learn the true latent taxonomic structure from data.
Through extensive empirical evaluations on both small and large-scale multi-label classification datasets we show that MBM can significantly improve taxonomic consistency while preserving or surpassing state-of-the-art predictive performance
I NTRODUCTION
Multi-label classification is a machine learning task in which an input is associated with multiple categories.
Many real-world multi-label classification datasets in modalities such as text categorization (Lewis et al., 2004), image classification (Lin et al., 2015; Krishna et al., 2016), entity typing (Murty et al., 2018; Onoe et al., 2021), functional genomics (Barutcuoglu et al., 2006; Clare, 2003), and so on, have a rich inter-dependent label structure that can be expressed using a taxonomy graph or a hierarchy.
To be useful in practice, a model should produce predictions that are coherent with respect to the label taxonomy.
For example, if a book is classified as drama , it should also be classified as fiction according to the label taxonomy of book genres in the left-hand side of Figure 1.
More formally, we are given a label taxonomy in the form of a directed acyclic graph G =
( L , T ) , where ( a, b ) ∈ T if and only if a is a parent of b in the taxonomy. A model will assign scores s ℓ ( x ) for each label ℓ ∈ L .
We say the scores are coherent with respect to the taxonomy if, for all edges ( a, b ) ∈ T , s a ( x ) ≥ s b ( x ) , and the model is consistent if this is the case for all inputs x .
In the case of the book genre classification example, this implies that s fiction must be greater than or equal to s drama , regardless of the input.
The problem of producing coherent predictions for multi-label classification has garnered a lot of attention in the machine learning literature (Wehrmann et al., 2018c; Giunchiglia & Lukasiewicz, 2020; Murty et al., 2018; Cerri et al., 2014).
Most methods that proposed to improve the coherence in predictions require the complete label taxonomy at inference time, and occasionally at training time as well, making these models hard to scale to large label spaces (Giunchiglia & Lukasiewicz, 2020; Wehrmann et al., 2018c).
This brings forth a question: Can we utilize representation learning to model the label-label relationships implicitly in the embedding space?
Vilnis et al.
(2018) show that probabilistic box embeddings, which represent concepts as high dimensional hyper-rectangles, can embed DAGs efficiently using the explicit information about the edges.
Box embeddings represent edges by box-box containment as shown in Figure 1b.
Representing the input and output labels in the same geometric space of boxes allows the multi-label taxonomy to be learned without an explicit taxonomic training signal.
Moreover, there exists a large space of possible configurations that represent the same taxonomy, and if the label embeddings in the model gets close to any such configuration, then the model will always produce classifications that are coherent w.r.t the taxonomy, regardless of the input.
We show through empirical evidence this is the case, and provide a formal proof for latter.
In this work, we propose the multi-label box model (MBM) that utilizes the geometry and probabilistic semantics of box embeddings to model label-label interactions in multi-label classification.
MBM represents labels as boxes using free parameters and uses a deep neural network to embed the inputs in the same space.
We perform coherence analysis of the model using two measures and show that MBM not only achieves state-of-the-art predictive performance but it also significantly improves the coherence of predicted scores w.r.t latent label taxonomy.
Our analysis further shows that it is possible to retrieve the latent label-label relationships solely by analysing the learnt label representations inside the MBM, endowing the model with high degree of interpretability.
Finally, we also present a way to utilize the interpretability of MBM to inject partial information about label-label relationships into the model thereby improving the coherence even further.
R ELATED W ORK
Multi-label classification tasks that exhibit strong label space structure in the form of explicit label taxonomy are termed hierarchical multi-label classification (HMLC) in machine learning literature.
Most approaches for such tasks make use of the complete hierarchy at training time.
These approaches can be categorized into two buckets (Silla & Freitas, 2010): (1) Local approaches that focus on local information for each label or clusters of labels in the hierarchy and classify them independently (Cerri et al., 2014; Huang et al., 2019), and (2) Global approaches that treat the problem as a structured classification task and take global interactions into account (Belanger & McCallum, 2016).
In the most general setting, however, both local and global interactions between labels exist.
The recent advances in deep learning (Wehrmann et al., 2018a) propose a specialized neural network architecture called Hierarchical Multi-Label Classification Network (HMCN-R and HMCN-F) that takes into account both local and global interactions by creating an ensemble of classifiers that can be trained using end-to-end gradient based training.
However, HMCN does not try to enforce coherence strongly, focusing solely on predictive performance.
In order to improve prediction coherence, recent works employ special loss functions on top of a neural network classifier to enforce coherence w.r.t the label taxonomy (Murty et al., 2018; Giunchiglia & Lukasiewicz, 2020).
While effective, these approaches still use the label taxonomy explicitly, making them difficult to scale to very large label spaces.
Recent advances in representation learning provide various methods to embed large graphs and taxonomies parsimoniously in non-euclidean spaces (Nickel & Kiela, 2017; Vilnis et al., 2018; Vilnis & McCallum, 2015; Gu et al., 2019).
The most prominent of these embedding methods include hyperbolic embeddings (Nickel & Kiela, 2017; Ganea et al., 2018a;b) and box embedding (Vilnis et al., ; Li et al., 2019; Dasgupta et al., 2020a).
The use of representations other than Euclidean vectors for improving the coherence of multi-label classification has been limited to specific domains like text (Chatterjee et al., 2021) or specific tasks like entity typing (Onoe et al., 2021).
Moreover, while both hyperbolic and box embedding can model hierarchical relationships, it has been shown that the box embedding can also model more general graphs like DAGs much more efficiently than hyperbolic embeddings (Patel et al., 2020).
Hence, we propose a model that uses box embedding to capture general label-label relationships without the explicit use of label taxonomy to improve the coherence of model predictions.
O VERVIEW OF B OX E MBEDDINGS
Notations: In the problem of multi-label classification, we are given a set of labels L where
L = |L| , and an instance can be labeled with an element s ∈ { 0 , 1 } L , where projection to the i th coordinate π i ( s ) = 1 means that the i th label is true.
We call the set of all such labelings S , and the associated probability space ( S, P ( S ) , P S ) .
We use I to denote the set of all finite closed intervals [ µ − , µ + ] in Ω ⊂ R plus the empty set, i.e. I := { [ µ − , µ + ] ⊂ Ω | µ + ≥ µ − } ∪ ∅ . We denote the smallest σ -algebra containing I as σ ( I ) and, given a valid finite measure ν , we consider the measure space (Ω , σ ( I ) , ν ) .
As a high dimensional generalization, I d will denote a d -dimensional Cartesian product of I .
Definition 1 (Box Embedding (Vilnis et al., 2018)) .
Let B : I d
→ S be a measurable function such that B − 1 ◦ π − 1 i (1)
= (cid:81) di [ µ − i , µ
+ i ] ∈ I d .
A box embedding is defined as the function Box :
L → I d which maps a label ℓ ∈
L to B − 1 ◦ π − 1 ℓ ( { 1 } ) ∈
I d .
The definition of box embeddings induces a push-forward measure Q on S such that for any R ⊆ S ,
Q ( S ) =
ν ◦ B − 1 ( R ) .
The complete joint probability distribution over the labels can be modeled using Q as defined above; however, computing B − 1 ( R ) requires the use of inclusion-exclusion principle and hence is intractable for a general R .
In order to avoid local identifiability issues in training, Dasgupta et al.
(2020b) interpret µ − i (resp.
µ + i ) as the location parameters of random variables M − i (resp.
M + i ) that are distributed according to GumbelMax (resp.
GumbelMin) distributions, leading to a meta-probabilistic generalization of box embedding which they call Gumbel Box Process .
Since GumbelMax (resp.
GumbelMin) is a max (resp.
min) stable distribution, it enables the computation of the location parameters of the intersection box as given in the following definition.
Definition 2 (Intersection Box (Dasgupta et al., 2020b)) .
Let A = (cid:81) di
=1 [ a − i , a + i ] and B = (cid:81) [-d i =
-]di
=1 [ b − i , b
+ i ] be two gumbel boxes expressed using their location parameters, then the location parameters of the intersection of these two gumbel boxes are given as
[Equation]
where lse( x, y ) =
log(exp( x ) + exp( y )) .
The expected volume of Gumbel boxes involves the Bessel Function of the Second Kind, however, as shown in Dasgupta et al.
(2020b)
, this integral can be reasonably approximated using softplus function leading to the following definition for approximate bessel volume .
Definition 3 (Approximate Bessel Volume (Dasgupta et al., 2020b)) .
For a gumbel box B = (cid:81) di =1 [ b − i , b + i ] we define the approximate Bessel volume λ : I d → R + as
[Equation]
In the next section, we formally demonstrate the suitability of box embeddings for capturing taxonomic label relationships, and for that we first prove a couple of useful facts regarding the Gumbel intersection and Bessel approximate volume.
Proposition 1.
Approximate bessel volume is monotonic with respect to set containment.
That is for two Gumbel boxes A, B
[Equation]
Proof.
Follows from the monotonicity of log(1 + exp( . )) .
Proposition 2.
For any two Gumbel boxes A, B , λ ( A ˜ ∩ B ) ≤ λ ( B ) .
Proof.
The fact that max( x, y ) ≤ lse( x, y ) , and the statement of proposition 1 together imply the desired result.
Since λ is neither normalized nor additive, it cannot be used as a probability measure on (Ω , σ ( I d )) .
However, we can use proposition 1 and 2 to define a conditional probability model as follows.
Corollary 1.
For two gumbel boxes A, B , let P Box ( A | B )
=
λ ( A ˜ ∩ B ) λ
( B ) , then
(i) For any two gumbel boxes A, B , we have 0 ≤ P Box ( A | B ) ≤ 1 .
(ii) P Box ( A | C ) ≤ P Box ( B | C ) for any three gumbel boxes A, B, C , with a − i ≥ b − i , a + i ≤ b + i .
M ULTI - LABEL B OX M ODEL
In order to perform the task of multi-label classification we need to model the conditional probabilities P ( Y | X ) where Y ∈ S and X is the input.
Using definition 1, we define label box embeddings
Box ψ : L → I d as
Box ψ ( ℓ i ) :
= d (cid:89) j
= [ ψ − i,j , ψ − i,j + log(1 + exp ψ
+ i,j )] , where ψ − , ψ + ∈ R L
×
d are trainable parameters.
The input instance X is encoded as a fixed-width element of I d using a parametric instance box embedding Box θ =
I d ◦ F θ : X → I d , where
F θ : X → R d is a neural network with parameters θ
and I d : R d → I d defined as
[Equation]
where δ = 10 − 5 .
The conditional probability for Y ∈ S given input X is computed using conditional probability under the Gumbel box model as
[Equation]
Using the definition of P Box as stated through corollary 1, we get the following expression for the conditional probability of Y i under the model, where the intersection ˜ ∩ is the Gumbel Intersection and measure λ is Approximate Bessel Volume .
[Equation]
4. M ODELING LABEL - SPACE INTERACTIONS
In Section 1, we alluded to the fact that the inductive bias of the MBM allows it to efficiently model partially specified first-order label interactions.
Now we make this remark more concrete.
If the partial specification of label interaction is defined using a taxonomy that can be represented as a directed acyclic graph (DAG), the following proposition shows that MBM has a strong inductive bias towards maintaining coherence in its scores.
Proposition 3.
Let G =
( L , T ) denote a DAG defined over the labels where L is the set of all labels and T = { ( ℓ i , ℓ j )
| ℓ i , ℓ j ∈ L , P D ( y i = [-
-]1 | y j = 1) = 1 } is the set of edges.
Then there exists some ψ such that P ( ψ,θ ) ( y i = 1 | x ) ≥ P ( ψ,θ ) ( y j = 1 | x ) , for all x, θ .
Proof.
For all ( ℓ i , ℓ j ) ∈ T , let ψ be such that Box ψ ( ℓ j ) ⊆
Box ψ ( ℓ i ) .
Note that such ψ exists since for each i ∈ { 1 , . . .
, L } , Box ψ ( ℓ i ) is defined using only ψ i .
It follows from corollary 1 that
P Box (Box ψ | Box
) ≥ P Box Box ψ ℓ j | Box θ X )) for any X, θ .
4. L EARNING
The entire MBM is specified using parameters ( ψ, θ ) where ψ ∈ R 2 d × L are the label embedding parameters and θ are the parameters of the instance encoder neural network F θ .
Given data
D = { ( x (1) , y (1) ) , . . .
, ( x ( N ) , y ( N ) ) } , the model parameters are learnt by minimizing negative loglikelihood loss
[Equation]
using the ADAM optimizer (Kingma & Ba, 2017).
In order to empirically verify the intuition behind proposition 3, we also propose the use of label interaction loss
[Equation]
that utilizes the geometry of box embeddings to inject partial information about label interactions specified using a label taxonomy G =
( L , T ) .
When label interaction loss is applied (MBM-T), the total loss is L T =
L nll + νL G , where ν is a hyperparameter.
For the Box model, label interaction score for a pair of labels is defined as
[Equation]
B ASELINES
Our choice of baselines reflects the focus of this work, i.e., introducing coherence in prediction using suitable representation spaces.
To this end, our baselines consist of a high-performing neural network that only uses Euclidean vector representations (MVM), and another that uses hyperbolic representations (MHM).
In order to test the importance of the probabilistic semantics used to formulate MBM, we also include as baseline, a non-probabilistic box model as defined in Abboud et al.
The base input encoder architecture F θ in all these baselines is same as the one used in MBM.
Multi-label Vector Model (MVM)
An input encoder neural network F θ : X → R d is used to encode the inputs and a label embedding matrix ψ is used to represent the labels.
The conditional probability of labels given the input and label interaction scores are given as
P MVM ( y l = 1 | x ; ψ, θ ) : = σ ( F θ ( X ) T M θ ψ l ) , and s MVM ( l i , l j ) :
= σ ( ψ Tl i M θ ψ l j ) , respectively, where σ is the logistic sigmoid function, and G θ ∈ R d × d is a matrix of trainable parameters.
The parameters ( θ, ψ ) are learnt through L T (Eq. 3,4).
Note that, when label interaction is not being explicitly modeled i.e., ν = 0 , M is fixed as the identity matrix and MVM reduces to a special case of multi-layer perceptron.
Multi-label Hyperbolic Model (MHM) As discussed in Chatterjee et al.
(2021), the isometry between the Lorentz model and Poincare disk model for hyperbolic geometry can be used produce the retraction formula used to project d -dimensional euclidean vectors to d -dimensional Poincar´e ball
B d = (cid:8) x ∈ R d | ∥ x ∥ < 1 (cid:9) .
Here the projection operator Π : R d → B d and the distance d : B d × B d → R + in the hyperbolic space are given as:
[Equation]
Specifically, the input is first encoded using F θ and then projected into B d .
The unnormalized score for each label is computed as the negative of the distance between the hyperbolic projections of encoded input and label representation.
Since the hyperbolic distance function consists of arcosh , the negative distance is interpreted as log-probability score.
Hence, the conditional log-probability of the labels and the label interaction score is given, respectively, by the following expressions, which are then used to learn the parameters ( ψ, θ ) in the same way as MVM and MBM.
log
[Equation]
[Equation] ( F
[Equation] ( x ) [Equation] , Π( ψ l )
[Equation]
and s MHM ( l i , l j
[Equation]
:= − d (Π( ψ l i ) , Π( ψ l j )) ,
Non-probabilistic box model (BoxE) : In order to test the importance of the probabilistic semantics used to formulate the MBM, we also include as baseline, a box model that does not use probabilistic semantics. For this, we replace the P MBM with the non-probabilistic score defined in Abboud et al. and replace the log-likelihood loss ( L nll ) with a margin based loss L margin .
E VALUATION AND R ESULTS
We evaluate the performance of MBM using 12 real-world multi-label classification datasets (Dimitrovski et al., 2011; Clare, 2003) spanning across three domains: text (Enron), images (Imclef07a, Imclef07d, Diatoms), and functional genomics (Expr, Cellcycle, Derisi, Spo) .
These datasets are the ideal test bed as they provide explicit label taxonomies with different connectivity: trees, forests, and more general DAGs.
Moreover, all the labels of all training and test instances respect the label taxonomy.
THe datasets include both categorical and continuous input features.
We convert the categorical features into one-hot feature vectors and standardize all continuous features.
The input encoder F θ uses a common architecture for all models consisting of an MLP with a maximum of 3 layers.
We perform a grid search over activations, hidden dimensions, dropout, learning rate and use the best parameters for each model.
The Mean Average Precision (MAP), that is the mean over the average precision values across instances in the test set, is used to evaluate the predictive performance of the models.
Table 1 reports average metric values across 5 runs with different random seeds (the standard error intervals are small and are omitted for the sake of readability). As seen in Table 1, the predictive performance (MAP) of MBM is better than all other methods on at least 9 out of 12 datasets.
To check the statistical significance of our results, following Demˇsar (2006), we first perform the Friedman test and obtain a p-value of 2 × 10 − 9 .
The critical diagram of the post-hoc Nemenyi test (Figure 2a) performed after the Friedman test allows us to conclude that difference in predictive performance between MBM and all other models except MVM is statistically significant. Through the more powerful pairwise Wilcoxon test, we find that the performance of MBM and MVM is comparable (p-value of 0.1). Furthermore, from these results, one can conclude with confidence that MBM with its probabilistic formulation preforms significantly better than BoxE, which is the non-probabilistic variant of box embeddings. In the following section, we discuss another important aspect of performance, that is, the coherence of predictions.
6. C OHERENCE ANALYSIS
MAP is a good metric for predictive performance, however it does not take into account the inconsistencies in the predicted scores w.r.t.
the label taxonomy.
For instance, recalling the earlier example in Figure 1, a consistent model would always assign higher score to fiction when compared to drama , since a book classified as drama should also be classified as fiction .
Since MAP is incapable of capturing such coherence conditions, we perform further analysis to check the coherence of the predicted scores.
This is done by considering two opposite perspectives towards coherence evaluation.
The first one is a punitive approach, where we count the number of instances that are given inconsistent scores w.r.t the taxonomy.
In the second one, a more constructive approach, we try to check the amenability of the model, by applying post-hoc corrections to the predicted scores to enforce coherence.
The results of these two approaches are presented by using Constraint Violation (CV) and Mean Average Precision post Coherence correction (CMAP), which are described below.
Note that we do not recommend the use of either CV or CMAP as the sole metrics to evaluate the model, we present these as measures to analyse the coherence of the model predictions.
Constraint violation is a punitive measure that quantifies the extent to which the label scores generated by the model violate the partial ordering of the latent label taxonomy regardless of true labels for the instances.
Hence, lower value of CV implies higher taxonomic coherence in the predictions. [Equation]
[Equation]
Mean Average Precision post Coherence correction (CMAP) :
Given a complete or partial taxonomy G = ( L , T ) for the labels, coherence can be imposed post-hoc by applying a modification [Table] [Table]
[Table]
[Table]
[Table]
↑
[Table]
[Table]
[Table]
[Table]
60 CMAP ↑ 41.
33.60 SpoFUN
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
90 CellcycleGO
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
98 DerisiGO
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
52.32 SpoGO
CV ↓ 5.
MAP ↑ 83.
82.29 CMAP ↑ 83.
82.29 Enron
CV ↓ 0.
MAP ↑ 79.
62.1 CMAP ↑ 79.
62.1 Diatoms
CV ↓ 6.
MAP ↑ 72.
74.64 CMAP ↑ 72.
74.64 Imclef07a
CV ↓ 3. MAP ↑ 83. 84.69 CMAP ↑ 84. 84.69 Imclef07d
CV ↓ 8.
MAP 4.
4.4 CMAP 4. [Table] function δ : R L → R L to the label-scores produced by the model such that δ ( s i ) − δ ( s j ) < 0 for all ( l i , l j ) ∈ T .
This can be achieved by modifying the score s i for each label l i to be either the maximum ( δ MG ) of the scores of any of its descendants or the minimum of the scores of its ancestors ( δ mG ) in the taxonomy G .
Concretely, given the scores s ∈ R L produced by a model, for an input, the two modification functions are given as
[Equation]
[Figure] where Anc G ( l ) and Des G ( l ) is the set of ancestors and descendants, respectively, of l in the graph
G .
In practice, one would select whichever post-hoc modification performs better, and we refer to the maximum MAP obtained after applying one of the modification functions δ mG , δ MG to the scores as CMAP.
2 Intuitively, a CMAP value close to MAP value implies that the model is more perceptive of the latent hierarchy in the label space.
As seen from the critical diagram of the
Nemenyi test (Figure 2b), when coherence is considered along with the predictive performance (CMAP), MBM outperforms MVM and MHM (with statistical significance) indicating that MBM is much more perceptive of the latent taxonomy compared to MVM.
In conclusion, while on one extreme there is MVM, which exhibits good predictive performance but fails to maintain coherence w.r.t the taxonomy, on the other extreme we have MHM that exhibits lowest constraint violations but gives inadequate predictive performance.
MBM, however, demonstrates good characteristics on both fronts–predictive performance as well as coherence.
Comparison with the state-of-the-art: C-HMCNN (Giunchiglia & Lukasiewicz, 2020), is the current state-of-the-art method for hierarchical MLC, and includes the modification function δ MG (called MCM in their work) in their model directly, applying it to scores produced by a multi-layer perceptron at training as well as inference time.
This is done by cleverly modifying BCE loss to MCLoss which works well with the modification function during training.
There are two key differences between C-HMCNN and MBM that one needs to keep in mind when comparing performance.
First, C-HMCNN does not attempt to use the label representations themselves to increase coherence.
Second, unlike MBM, C-HMCNN requires the complete label taxonomy at training and inference time. Due to the use of label taxonomy at inference time, C-HMCNN has 0 constraint violations. However, as seen from the Table 1 and the Figure 2, MBM performs significantly better than C-HMCNN w.r.t MAP as well as CMAP (with p-value of 0.002 in Wilcoxon test).
A NALYSIS OF L EARNED L ABEL E MBEDDINGS
In this section, we analyze the geometry of the learned label embeddings, finding that the simple geometry of box embeddings endows the MBM model with high degree of interpretability.
In order to verify that label box embeddings are producing consistent scores by using inclusion in the box space, we inject into the model the taxonomy information through the additional loss term (Eq. 4).
As seen from the two right most columns in Table 1, injecting explicit taxonomic information into the label embeddings further reduces the extent of constrain violation for MBM and MHM models.
However, it does not aid the MVM model and even drops its predictive performance significantly.
This underscores the importance of embedding geometry for inducing taxonomic coherence, and validates our intuition about the arrangement of label embeddings in MBM.
To determine the extent to which the label embeddings capture the latent label taxonomy without it being explicitly provided, we perform ancestor-descendant classification solely using the learned label embeddings.
Each pair of labels ( l i , l j ) get a score β ( i, j ) that is determined using their corresponding label embeddings ψ i , ψ j .
Since MVM and MBM have different geometrical interpretations, we use different scoring for each.
Specifically, β MVM ( i, j ) =
ψ i · ψ j ∥ ψ j ∥ , β MBM ( i, j ) = s MBM ( l i , l j )
These scores are then compared to true ancestor-descendant relations 1.0 FPR 0. T P R Random 0.
BoxEMVMMHMMBM
Dataset
BoxE MVM MHM MBM
ExprFUN 0.
0.89 CellcycleFUN 0.
0.93 DerisiFUN 0.
0.87 SpoFUN 0.
0.87 Enron 0.
0.92 Diatoms 0.
0.96 Imclef07a 0. [Table] in the taxonomy to obtain respective ROC curves as shown in Figure 3. As seen in Table 3, MBM captures the true label taxonomy the best ( AUC ≥ 0 . 87 ) for all datasets.
Furthermore, for hyperberbolic space, it is suggested that the magnitude of embeddings relate to the level of generality in taxonomy (Nickel & Kiela, 2017).
We show that the same observation holds for box embeddings, with the vector embedding magnitude replaced by box embedding volume.
To see this, we compute the Spearman rank correlation between the number of descendants of a node in the true taxonomy and the embedding magnitude, negative embedding magnitude, and embedding volume for MVM, MHM, and MBM, respectively.
The correlation values reported in Table 2 confirm our intuition regarding box embeddings stated above.
It is interesting to note that as shown by good correlation between the vector magnitude and level of the label node as well as low constraint violations, the MHM model does capture the depth of the label nodes in the taxonomy.
However, as shown by the ROC curves in Figure 3, unlike MBM, it fails to capture the exact connectivity of the nodes at different depths in the taxonomy.
C ONCLUSION
In this work, we demonstrate that box embeddings with its probabilistic formulation can effectively capture taxonomic relations present between labels in multi-label classification without requiring explicit access to the taxonomy.
The multi-label box model achieves a fine balance between predictive performance and coherence.
Furthermore, we find that the graph relationships between labels can be easily injected via extra supervision during training, increasing the coherence of the model predictions.
Since the model has the same computational complexity as a simple vector neural network model (MVM), and unlike C-HMCNN it does not require the label taxonomy, we believe that the proposed model can be scaled to work with extremely large label spaces.
Moreover, due to the flexibility of using any neural network encoder this model can be easily extended to different input modalities, such as raw text, images, etc.
We wish to pursue these two directions in our future work.
R EPRODUCIBILITY S TATEMENT
We have strove make the results reported in this work easily reproducible.
In this section, we provide pointers to information necessary to reproduce the results mentioned in this paper.
The description of the datasets with various statistics, links to download them, and instructions to pre-process them are provided in
Appendix B. The BoxE baseline is further described with expressions for distance and the margin based loss function in Appendix A.1.
The hyper-parameter settings for each modeldataset combination (all 96 settings) used to produce the reported results are provided in 3.
We also include a discussion on computational complexity of the models in Appendix A.2.
Finally, an anonymized version of the code used in this paper is publicly available and can be obtained by following the instructions in A.3.
In this section, we describe the implementation details: training procedure and hyper-parameter search.
Frameworks used: We implement all the models described in this work using PyTorch (Paszke et al., 2019).
We also make use of NLP specific abstractions over PyTorch provided by AllenNLP (Gardner et al., 2017).
Data pre-processing: The datasets were pre-processed to remove noisy characters, fix encoding issues, standardize the continuous variables and map the labels to binary one hot encoding label vector.
Training:
The learning algorithm used for training is minibatch gradient descent with a fixed batch size of 64.
The ADAM (Kingma & Ba, 2017) optimizer was used during training, along with
ReduceLROnPlateau learning rate scheduler with early stopping.
Since the naive implementation of the label interaction loss described in Eq.
4 is too expensive to compute at each mini-batch step, we approximate it by randomly sampling (without replacement), at each mini-batch step, a subset of edges ˜ T ∼ T .
The size of the sampled set is a hyper-parameter (last column in Table 3). Hyper-parameter search:
The following hyper-parameters were searched using the performance on the validation set: optimizer learning rate, hidden dimensions of the MLP and activation functions, dropout probabilities, label space dimensions/box space dimensions(half of hidden dimensions), weight and sampling percentage for the labels for computing the label interaction loss( L G ) for the MHM-T, MBM-T and MVM-T models.
All the results are reported using the best hyperparameters found using grid search for each model-dataset combination.
Table 3 summarizes the search ranges used and Table 4 presents the final hyper-parameters obtained.
[Figure] [Table] [Table]
[Table]
- - Enron
MBM-T 0.
1e-7 Enron
MHM
[Table]
- - Enron
MHM-T 0.
[Table]
1e-7 Enron
MVM 0.
- - Enron
[Table]
1eExprFUN
BoxE 0.
- - ExprFUN
C
[Table]
HMCNN
[Table]
- ExprFUN
MBM
[Table]
- - ExprFUN
MBM
[Table]
[Table]
1e-7 ExprFUN
[Table]
[Table]
MHM
[Table]
[Table]
1e-
[Table]
[Table]
[Table]
1eExprGO
BoxE 0.
- - ExprGO
[Table]
- ExprGO
MBM 0.
- - ExprGO
MBM-T 0.
1e-9 ExprGO
MHM 0.
- - ExprGO
MHM-T 0.
1e-7 ExprGO
MVM 0.
- - ExprGO
MVM-T 0.
1eImclef07a
BoxE 0.
- - imclef07a
C-HMCNN 0.
- Imclef07a
MBM 0.
- - Imclef07a
MBM-T 0.
1e-7 Imclef07a
MHM 0.
- - Imclef07a
MHM-T 0.
1e-7 Imclef07a
MVM 0.
- - Imclef07a
MVM-T 0.
1eImclef07d
BoxE 0.
- - Imclef07d
C-HMCNN 0.
- Imclef07d
MBM 0.
- - Imclef07d
MBM-T 0.
1e-9 Imclef07d
MHM 0.
- - Imclef07d
MHM-T 0.
1e-7 Imclef07d
MVM 0.
- - Imclef07d
MVM-T 0.
1e-
[Table]
A. F URTHER D ETAILS FOR THE BASELINES
BoxE : Let p ∈ R d be a point and z, Z ∈ R d with z i <
Z i be the lower left and upper right coordinates of a non-probabilistic label box.
Then a non-probabilistic compatibility score between the point and the box can be defined as shown in Abboud et al. Concretely, Abboud et al. (2020) define per-dimension distance as [Equation]
where c i =
( Z i − z i ) / 2 , w i
=
Z i − z
i are the center and width of the projection of the box (i.e., an interval) in dimension i , and κ = 0 .
5 ∗ w i ∗ ( w i + 1 − 1 w i +1 ) is the width dependent factor.
The score function that measures the compatibility of the point p and box ( z, Z ) is defined as s BoxE ( p, z, Z ) =
−∥ v ∥ 2 , where v i = dist ( p i , z i , Z i ) .
Given the trainable parameters ψ − , ψ + ∈ R L × d for representing the labels, the lower left and the upper right coordinates of a non-probabilistic box for label l i are taken to be z i = ψ − i, ∗ and Z i = ψ − i, ∗ + log(1 + exp ψ
+ i, ∗ ) , respectively.
The input x is encoded as a point p ∈ R d using a multi-layer perceptron F θ , just like MVM.
However, since the score is negative distance, the parameters are learnt using margin based loss given as:
[Equation]
[Equation]
where P i and N i are the positive and negative labels for i -th data instance, respectively.
A. C OMPUTATIONAL COMPLEXITY
The computational complexity of the MBM can be divided into two parts–the computation of the instance box Box θ ( X ) , and the computation of probability score P ( Y | X ) given the encoding of the instance and label box parameters.
Computing Box θ ( X ) amounts to splitting the output of a feed-forward network ϕ
( x ) into min and max parameters ϕ ( x ) − , ϕ ( x ) + , and thus has the same complexity as that of the MVM. Assuming that the operations (cid:80) , (cid:81) , log and exp have unit complexity, computing P ( Y | X ) also has equivalent complexity. The computation of conditional probability for a single label y k using the MVM model involves computing σ ( ϕ ( x ) · y k ) , an O ( dL ) operation. Now, with same assumptions for the MBM, computing λ (Box θ ( x ) ∩ Box ψ ( y k )) involves d invocations of log(exp( · ) + exp( · )) followed by d subtractions and d invocations of log(1 + exp( . )) , resulting in O ( d ) operations.
The computation of λ (Box θ ( x )) omits the intersection calculation, but otherwise is the same, and thus is also O ( d ) .
Hence, calculating λ (Box( x ) ∩ Box( y k )) λ (Box( x )) is O ( d ) , resulting in overall complexity of O ( dL ) , which is equal to that of MVM.
We note that the complexity of our model is not dependent on the depth of the hierarchy, as in C-HMCNN, and thus can reasonably scale to arbitrarily deep hierarchies as might be present in extreme multi-label classification.
These theoretical statements are supported by Table 5, which provides average epoch duration in seconds for MVM, MBM, MBM-T, and C-HMCNN with the same hidden size. Hence, we can conclude that the computational complexity of MVM ≈ MBM < MBM-T.
[Table]
Computational resources used: For datasets with number of labels less than 500, i.e., the 4 FUNCAT datasets, Imclef07a, Imclef07d, Diatoms and Enron, all the models were trained on TitanX
GPU (memory=12GB).
For the 4 GO datasets that have number of labels greater than 4000, all the models are trained on M40 GPU (memory=24GB).
A. C ODE
Completely anonymized and executable code with detailed instructions is provided using
Anonymous
Github at anonymous.4open.science/r/modeling-label-spaceinteractions-in-multi-label-classification-using-box-embeddings .
The instructions cover all the use cases, i.e., obtaining the pre-processed datasets, training a new model from scratch (MBM or any baseline), evaluating a pre-trained model on test set, directly downloading the pre-trained models for datasets, and reproducing the graphs reported in the analysis section.
B D ATASETS
The 12 datasets used in this work wary greatly in terms of domain, number of labels, number of instances, and the connectivity of label taxonomy. The characteristics of each dataset w.r.t these properties is summarized in 6. These datasets do not require a licence and are available for public usage. The links to the sources for all the datasets are provided in Table 7 below.
The continuous features are standardized by removing mean and scaling to unit variance, and the categorical features are encoded as one-hot vectors.
C S IGNIFICANCE T ESTING
In section 6, we provide the results for Friedman test followed by post-hoc Nemenyi test, and also the result of pairwise Wilcoxon test where the former are not sufficient, i.e. for the case of MVM vs MBM w.r.t MAP.
For the sake of completeness, here in table 8, we provide the complete table of pairwise Wilcoxon test from comparing all models with MBM and MBM-T w.r.t all three metrics.
D C OMPARISON TO C-HMCNN AND HMCN
In our experiments with the official code of C-HMCNN(Giunchiglia & Lukasiewicz, 2020) as well as our re-implementation of the same, we found that C-HMCNN consistently gets lower MAP compared to a well tuned MVM (note that the base MVM results are not reported in the C-HMCNN or HMCN paper).
In Giunchiglia & Lukasiewicz (2020), C-HMCNN was shown to outperform
HMCN(Wehrmann et al., 2018c).
Based on these observations, we considered a well-tuned MVM to be the stronger baseline and only included the results of C-HMCNN (our implementation) in table [Table] [Table]
to test the null hypothesis that two related paired samples come from the same distribution.
The test considers the ranking as well as the differences in performance between the models across the 12 datasets.
[Table] Table 9 compares our models to the C-HMCNN and HMCN models.
Here, the columns with * are taken from their respective papers and these models use the union of train and validation set to train their final model.
However, in all of our experiments, we only train on the train set to obtain the final test performance.
As seen from the table, except for Imclef07a, MBM performs better than
HMCN and C-HMCNN (even with lesser training data).
E MAP AND CMAP
In MCL problems, where predictions consistent with a taxonomy are necessary, the model designer might be ready to sacrifice some absolute predictive performance for increased coherence, where the predictive performance can be measure using Mean Average Precision (MAP).
First recall that
Average Precision (AP) for an instance is the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight, and Mean Average
Precision (MAP) is the mean of AP across instances. Given data D , let s ( k ) i [Equation] denote the score generated by the model for label i given the input x ( k ) .
Then MAP is computed as [Table] [Equation] [Table]
As described in Section 6.1, given the scores s ∈ R L produced by a model, for an input, the two post-hoc modification functions that can be applied to make these score coherent are given as
[Equation]
where Anc G ( l ) and Des G ( l ) is the set of ancestors and descendants, respectively, of l in the graph
G . One can compute MAP after applying either one of the two modification functions. If a human were to make consistent score corrections based on the complete hierarchy, they would either pick the score of a label to be the minimum of its ancestors or maximum of its descendants.
Once picked, the same approach has to be applied to all the labels.
We argue that in terms of correlation with human judgment, both these methods are equivalent.
Hence, one can pick the modification function based on empirical performance of the post-hoc correction algorithm. We observe that δ mG , i.e., “min of ancestors” approach, produced higher MAP with both MVM and MBM models compared to δ M G . as can be seen in table 10.
We believe that ”min of ancestors” works better because it promotes sparsity in predictions, which is essential for the task of MLC. As seen, CMAP produces better results for both and
[Table]
F P ERFORMANCE ON LONG TAIL OF LABELS
Since MVM and MBM both are embedding based models, we expect the performance of the MBM model to wary with the label frequency in the same manner as MVM.
This hypothesis is corroborated by the following table which shows the spearman rank correlation between the label frequency and the mean average precision (MAP) for that label.
The high positive correlation for all the models suggests that the performance of both MVM and MBM often degrades as the frequency of the labels drop.
However, as shown by the table 11, both the models perform reasonably well for extremely low frequency labels.
Here, tail MAP is the mean average precision for labels with frequency 0.001% or lower.
[Table]
G E FFECT OF EMBEDDING DIMENSION
To study the effect of embedding dimensions on the performance, we plot the MAP, CMAP and CV vs the hidden dimension size for a subset of the datasets.
As shown in figure 4, at extremely low dimensions, the MBM model consistently outperforms the MVM.
This shows that the favorable inductive bias of the box space induces parsimonious representations.
H E FFECT OF HIERARCHY ON THE MARGINAL SCORE
[Figure] [Figure]