M ODELING L ABEL S PACE I NTERACTIONS IN M ULTILABEL C LASSIFICATION USING B OX E MBEDDINGS
A BSTRACT
Multi-label classification is a challenging structured prediction task in which a set of output class labels are predicted for each input.
Real-world datasets often have taxonomic relationships between labels which can be explicit, implicit, or partially observed.
Most existing multi-label classification methods either ignore the label taxonomy or require the complete specification of the taxonomy at training and inference time to enforce coherence in their predictions.
In this work we introduce the multi-label box model (MBM), a multi-label classification method that combines the encoding power of neural networks with the inductive bias of probabilistic box embeddings (Vilnis et al., 2018), which can be understood as trainable Venn-diagrams based on hyper-rectangles.
By representing labels as boxes, MBM is able to capture taxonomic relations among labels without them being provided explicitly.
Furthermore, since MBM learns the label-label relationships from data and represents them as calibrated conditional probabilities, it provides a high degree of interpretability.
This interpretability also facilitates the injection of partial information about label-label relationships into model training, to further improve its consistency.
We provide theoretical grounding for our method and show experimentally the model’s ability to learn the true latent taxonomic structure from data.
Through extensive empirical evaluations on twelve multi-label classification datasets, we show that MBM can significantly improve taxonomic consistency while maintaining the state-of-the-art predictive performance.
I NTRODUCTION
Multi-label classification is a machine learning task in which an input is associated with multiple categories.
Many real-world multi-label classification datasets in modalities such as text categorization (Lewis et al., 2004), image classification (Lin et al., 2015; Krishna et al., 2016), entity typing (Murty et al., 2018; Onoe et al., 2021), functional genomics (Barutcuoglu et al., 2006; Clare, 2003), and so on, have a rich inter-dependent label structure that can be expressed using a taxonomy graph or a hierarchy.
To be useful in practice, a model should produce predictions that are coherent with respect to the label taxonomy.
For example, if a book is classified as drama , it should also be classified as fiction according to the label taxonomy of book genres in the left-hand side of Figure 1.
More formally, we are given a label taxonomy in the form of a directed acyclic graph G =
( L , T ) , where ( a, b ) ∈ T
if and only if a is a parent of b in the taxonomy.
A model will assign scores s ℓ ( x ) for each label ℓ ∈ L .
We say the scores are coherent with respect to the taxonomy if, for all edges ( a, b ) ∈ T , s a ( x ) ≥ s b ( x ) , and the model is consistent if this is the case for all inputs x .
In the case of the book genre classification example, this implies that s fiction must be greater than or equal to s drama , regardless of the input.
The problem of producing coherent predictions for multi-label classification has garnered a lot of attention in the machine learning literature (Wehrmann et al., 2018a; Giunchiglia & Lukasiewicz, 2020; Murty et al., 2018; Cerri et al., 2014).
Most methods proposed to improve the coherence in predictions require the complete label taxonomy at inference time, and occasionally at training time as well, making these models hard to scale to large label spaces (Giunchiglia & Lukasiewicz, 2020; Wehrmann et al., 2018b).
This brings forth a question: Can we utilize representation learning to model the label-label relationships implicitly in the embedding space?
Vilnis et al.
(2018) introduced probabilistic box embeddings, which represent concepts as high dimensional hyper-rectangles, and demonstrated that they can embed DAGs efficiently using explicit information about the edges.
Box embeddings represent edges by box-box containment as shown in Figure 1b.
Representing the input and output labels in the same geometric space of boxes allows the multi-label taxonomy to be learned without an explicit taxonomic training signal.
Moreover, there exists a large space of possible configurations that represent the same taxonomy, and if the label embeddings in the model gets close to any such configuration, then the model will always produce classifications that are coherent w.r.t the taxonomy, regardless of the input.
We show through empirical evidence this is the case, and provide a formal proof for latter.
In this work, we propose the multi-label box model (MBM) that utilizes the geometry and probabilistic semantics of box embeddings to model label-label interactions in multi-label classification.
MBM represents labels as boxes using free parameters and uses a deep neural network to embed the inputs in the same space.
We perform coherence analysis of the model using two measures and show that MBM not only achieves state-of-the-art predictive performance but it also significantly improves the coherence of predicted scores w.r.t latent label taxonomy.
Our analysis further shows that it is possible to retrieve the latent label-label relationships solely by analysing the learnt label representations inside the MBM, endowing the model with high degree of interpretability.
Finally, we also present a way to utilize the interpretability of MBM to inject partial information about label-label relationships into the model thereby improving the coherence even further.
R ELATED W ORK
Multi-label classification tasks that exhibit strong label space structure in the form of explicit label taxonomy are termed hierarchical multi-label classification (HMLC) in machine learning literature.
Most approaches for such tasks make use of the complete hierarchy at training time.
These approaches can be categorized into two buckets (Silla & Freitas, 2010): (1) Local approaches that focus on local information for each label or clusters of labels in the hierarchy and classify them independently (Cerri et al., 2014; Huang et al., 2019), and (2) Global approaches that treat the problem as a structured classification task and take global interactions into account (Belanger & McCallum, 2016).
In the most general setting, however, both local and global interactions between labels exist.
The recent advances in deep learning (Wehrmann et al., 2018a) propose a specialized neural network architecture called Hierarchical Multi-Label Classification Network (HMCN-R and HMCN-F) that takes into account both local and global interactions by creating an ensemble of classifiers that can be trained using end-to-end gradient based training.
However, HMCN does not try to enforce coherence strongly, focusing solely on predictive performance.
In order to improve prediction coherence, recent works employ special loss functions on top of a neural network classifier to enforce coherence w.r.t the label taxonomy (Murty et al., 2018; Giunchiglia & Lukasiewicz, 2020).
While effective, these approaches still use the label taxonomy explicitly, making them difficult to scale to very large label spaces.
Recent advances in representation learning provide various methods to embed large graphs and taxonomies parsimoniously in non-euclidean spaces.
The most prominent of these embedding methods include hyperbolic embeddings (Nickel & Kiela, 2017; Ganea et al., 2018a;b) and box embedding (Vilnis et al., 2018; Dasgupta et al., 2021a).
The use of representations other than Euclidean vectors for improving the coherence of multi-label classification has been limited to specific domains like text (Chatterjee et al., 2021) or specific tasks like entity typing (Onoe et al., 2021).
Moreover, while both hyperbolic and box embedding can model hierarchical relationships, it has been shown that the box embedding can also model more general graphs like DAGs much more efficiently than hyperbolic embeddings (Patel et al., 2020; Boratko et al., 2021; Dasgupta et al., 2021b).
Hence, we propose a model that uses box embedding to capture general label-label relationships without the explicit use of label taxonomy to improve the coherence of model predictions.
O VERVIEW OF B OX E MBEDDINGS
Notations: In the problem of multi-label classification, we are given a set of labels L where
L = |L| , and an instance can be labeled with an element s ∈ { 0 , 1 } L , where projection to the i th coordinate π i ( s ) = 1 means that the i th label is true.
We call the set of all such labelings S , and the associated probability space ( S, P ( S ) , P S ) .
We use I to denote the set of all finite closed intervals [ µ − , µ + ] in Ω ⊂ R plus the empty set, i.e. I := { [ µ − , µ + ] ⊂ Ω | µ + ≥ µ − } ∪ ∅ .
We denote the smallest σ -algebra containing I as σ ( I ) and, given a valid finite measure ν , we consider the measure space (Ω , σ ( I ) , ν ) .
As a high dimensional generalization, I d will denote a d -dimensional Cartesian product of I .
Definition 1 (Box Embedding (Vilnis et al., 2018)) .
Let B : I d
→ S be a measurable function such that B − 1 ◦ π − 1 i (1)
= (cid:81) di [ µ − i , µ
+ i ] ∈ I d .
A box embedding is defined as the function Box :
L → I d which maps a label ℓ ∈ L to B − 1 ◦ π − 1 ℓ ( { 1 } ) ∈
I d .
The definition of box embeddings induces a push-forward measure Q on S such that for any R ⊆ S ,
Q ( S ) =
ν ◦ B − 1 ( R ) .
The complete joint probability distribution over the labels can be modeled using Q as defined above; however, computing B − 1 ( R ) requires the use of inclusion-exclusion principle and hence is intractable for a general R .
In order to avoid local identifiability issues in training, Dasgupta et al.
(2020) interpret µ − i (resp.
µ + i ) as the location parameters of random variables M − i (resp.
M + i ) that are distributed according to GumbelMax (resp.
GumbelMin) distributions, leading to a meta-probabilistic generalization of box embedding which they call Gumbel Box Process .
Since GumbelMax (resp.
GumbelMin) is a max (resp.
min) stable distribution, it enables the computation of the location parameters of the intersection box as given in the following definition.
Definition 2 (Intersection Box (Dasgupta et al., 2020)) .
Let A = (cid:81) di
=1 [ a − i , a + i ] and B = (cid:81) di
=1 [ b − i , b
+ i ] be two gumbel boxes expressed using their location parameters, then the location parameters of the intersection of these two gumbel boxes are given as
[Equation]
, (1) where lse( x, y ) =
log(exp( x ) + exp( y )) .
The expected volume of Gumbel boxes involves the Bessel Function of the Second Kind, however, as shown in Dasgupta et al.
, this integral can be reasonably approximated using softplus function leading to the following definition for approximate bessel volume .
Definition 3 (Approximate Bessel Volume (Dasgupta et al., 2020)) .
For a gumbel box B = (cid:81) di =1 [ b − i , b + i ] we define the approximate Bessel volume λ : I d → R + as
[Equation]
In the next section, we formally demonstrate the suitability of box embeddings for capturing taxonomic label relationships, and for that we first state a couple of useful facts regarding the Gumbel intersection and Bessel approximate volume.
Proposition 1.
Approximate bessel volume is monotonic with respect to set containment.
That is for two Gumbel boxes A, B ,
[Equation]
Proof.
Follows from the monotonicity of log(1 + exp( . )) .
Proposition 2.
For any two Gumbel boxes A, B , λ ( A ˜ ∩ B ) ≤ λ ( B ) .
Proof.
The fact that max( x, y ) ≤ lse( x, y ) , and the statement of proposition 1 together imply the desired result.
Since λ is neither normalized nor additive, it cannot be used as a probability measure on (Ω , σ ( I d )) .
However, we can use proposition 1 and 2 to define a conditional probability model as follows.
Corollary 1.
For two gumbel boxes A, B , let P Box ( A | B )
=
λ ( A ˜ ∩ B ) λ
( B ) , then
(i) For any two gumbel boxes A, B , we have 0 ≤ P Box ( A | B ) ≤ 1 .
(ii) P Box ( A | C ) ≤ P Box ( B | C ) for any three gumbel boxes A, B, C , with a − i ≥ b − i , a + i ≤ b + i .
M ULTI - LABEL B OX M ODEL
In order to perform the task of multi-label classification we need to model the conditional probabilities P ( Y | X ) where Y ∈ S and X is the input.
Using definition 1, we define label box embeddings
Box ψ : L → I d as
Box ψ ( ℓ i ) :
= d (cid:89) j
= [Equation]
)] , where ψ − , ψ + ∈ R L × d are trainable parameters.
The input instance X is encoded as a fixed-width element of I d using a parametric instance box embedding Box θ =
I d ◦ F θ : X → I d , where
F θ : X → R d is a neural network with parameters θ
and I d : R d → I d defined as
[Equation]
where δ = 10 − 5 .
The conditional probability for Y ∈ S given input X is computed using conditional probability under the Gumbel box model as
[Equation]
Using the definition of P Box as stated through corollary 1, we get the following expression for the conditional probability of Y i under the model, where the intersection ˜ ∩ is the Gumbel Intersection and measure λ is Approximate Bessel Volume .
[Equation]
4. M ODELING LABEL - SPACE INTERACTIONS
In Section 1, we alluded to the fact that the inductive bias of the MBM allows it to efficiently model partially specified first-order label interactions.
Now we make this remark more concrete.
If the partial specification of label interaction is defined using a taxonomy that can be represented as a directed acyclic graph (DAG), the following proposition shows that MBM has a strong inductive bias towards maintaining coherence in its scores.
Proposition 3.
Let G =
( L , T ) denote a DAG defined over the labels where L is the set of all labels and T = { ( ℓ i , ℓ j )
| ℓ i , ℓ j ∈ L , P D ( y i = 1 | y j = 1) = 1 } is the set of edges.
Then there exists some ψ such that P ( ψ,θ ) ( y i = 1 | x ) ≥ P ( ψ,θ ) ( y j = 1 | x ) , for all x, θ .
Proof.
For all ( ℓ i , ℓ j ) ∈ T , let ψ be such that Box ψ ( ℓ j ) ⊆
Box ψ ( ℓ i ) .
Note that such ψ exists since for each i ∈ { 1 , . . .
, L } , Box ψ ( ℓ i ) is defined using only ψ i .
It follows from corollary 1 that
P Box (Box ψ ( ℓ i ) | Box θ ( X )) ≥ P Box (Box ψ ( ℓ j ) | Box θ ( X )) for any X, θ .
4. L EARNING
The entire MBM is specified using parameters ( ψ, θ ) where ψ ∈ R 2 d × L are the label embedding parameters and θ are the parameters of the instance encoder neural network F θ .
Given data
D = { ( x (1) , y (1) ) , . . .
, ( x ( N ) , y ( N ) ) } , the model parameters are learnt by minimizing negative loglikelihood loss
[Equation]
using the ADAM optimizer (Kingma & Ba, 2017).
In order to empirically verify the intuition behind proposition 3, we also propose the use of label interaction loss
[Equation]
that utilizes the geometry of box embeddings to inject partial information about label interactions specified using a label taxonomy G =
( L , T ) .
When label interaction loss is applied (MBM-T), the total loss is L T =
L nll + νL G , where ν is a hyperparameter.
For the Box model, label interaction score for a pair of labels is defined as
[Equation]
B ASELINES
Our choice of baselines reflects the focus of this work, i.e., introducing coherence in prediction using suitable representation spaces.
To this end, our baselines consist of a high-performing neural network that only uses Euclidean vector representations (MVM), and another that uses hyperbolic representations (MHM).
In order to test the importance of the probabilistic semantics used to formulate MBM, we also include as baseline, a non-probabilistic box model as defined in Abboud et al.
The base input encoder architecture F θ in all these baselines is same as the one used in
MBM.
Multi-label Vector Model (MVM)
An input encoder neural network F θ : X → R d is used to encode the inputs and a label embedding matrix ψ is used to represent the labels.
The conditional probability of labels given the input and label interaction scores are given as
P MVM ( y l = 1 | x ; ψ, θ ) :
= σ ( F θ ( X ) T M θ ψ l ) , and s MVM ( l i , l j ) :
= σ ( ψ Tl i M θ ψ l j ) , respectively, where σ is the logistic sigmoid function, and G θ ∈ R d × d is a matrix of trainable parameters.
The parameters ( θ, ψ ) are learnt through L T (Eq. 3,4).
Note that, when label interaction is not being explicitly modeled i.e., ν = 0 , M is fixed as the identity matrix and MVM reduces to a special case of multi-layer perceptron.
Multi-label Hyperbolic Model (MHM)
As discussed in Chatterjee et al.
(2021), the isometry between the Lorentz model and Poincare disk model for hyperbolic geometry can be used produce the retraction formula used to project d -dimensional euclidean vectors to d -dimensional Poincar´e ball
B d = (cid:8) x ∈ R d | ∥ x ∥ < 1 (cid:9) .
Here the projection operator Π : R d → B d and the distance d : B d × B d → R + in the hyperbolic space are given as:
[Equation]
Specifically, the input is first encoded using F θ and then projected into B d .
The unnormalized score for each label is computed as the negative of the distance between the hyperbolic projections of encoded input and label representation.
Since the hyperbolic distance function consists of arcosh , the negative distance is interpreted as log-probability score.
Hence, the conditional log-probability of the labels and the label interaction score is given, respectively, by the following expressions, which are then used to learn the parameters ( ψ, θ ) in the same way as MVM and MBM.
log P
MHM ( y l = 1 | x ) = − d (Π( F θ ( x )) , Π( ψ l )) and s MHM ( l i , l j ) :
= − d (Π( ψ l i ) , Π( ψ l j )) ,
Non-probabilistic box model (BoxE) :
In order to test the importance of the probabilistic semantics used to formulate the MBM, we also include as baseline, a box model that does not use probabilistic semantics.
For this, we replace the P MBM with the non-probabilistic score defined in Abboud et al.
and replace the log-likelihood loss ( L nll ) with a margin based loss L margin .
E VALUATION AND R ESULTS
We evaluate the performance of MBM using 12 real-world multi-label classification datasets (Dimitrovski et al., 2011; Clare, 2003) spanning across three domains: text (Enron), images (Imclef07a,
Imclef07d, Diatoms), and functional genomics (Expr, Cellcycle, Derisi, Spo) .
These datasets are the ideal test bed as they provide explicit label taxonomies with different connectivity: trees, forests, and more general DAGs.
Moreover, all the labels of all training and test instances respect the label taxonomy.
The datasets include both categorical and continuous input features.
We convert the categorical features into one-hot feature vectors and standardize all continuous features.
The input encoder F θ uses a common architecture for all models consisting of an MLP with a maximum of 3 layers.
We perform a grid search over number of MLP layers, activation function, hidden dimensions, dropout, learning rate and use the best parameters for each model.
The Mean Average Precision (MAP), that is the mean over the average precision values across instances in the test set, is used to evaluate the predictive performance of the models.
Table 1 reports average metric values across 10 runs with different random seeds (the standard error intervals are small and are omitted for the sake of readability).
As seen in Table 1, the predictive performance (MAP) of MBM is better than all other embedding based methods on at least 11 out of 12 datasets.
To check the statistical significance of our results, following Dem ˇ sar (2006), we first perform the Friedman test and obtain a p-value of 2 × 10 − 9 .
The critical diagram of the post-hoc Nemenyi test (Figure 2a) performed after the Friedman test allows us to conclude that difference in predictive performance between MBM and all other embedding based models is statistically significant.
Furthermore, from these results, one can conclude with confidence that MBM with its probabilistic formulation preforms significantly better than BoxE, which is the non-probabilistic variant of box embeddings.
In the following section, we discuss another important aspect of performance, that is, the coherence of predictions.
6. C OHERENCE ANALYSIS
MAP is a good metric for predictive performance, however it does not take into account the inconsistencies in the predicted scores w.r.t.
the label taxonomy.
For instance, recalling the earlier example in Figure 1, a consistent model would always assign higher score to fiction when compared to drama , since a book classified as drama should also be classified as fiction .
Since MAP is incapable of capturing such coherence conditions, we perform further analysis to check the coherence of the predicted scores.
This is done by considering two opposite perspectives towards coherence evaluation.
The first one is a punitive approach, where we count the number of instances that are given inconsistent scores w.r.t the taxonomy.
In the second one, a more constructive approach, we try to check the amenability of the model, by applying post-hoc corrections to the predicted scores to enforce coherence.
The results of these two approaches are presented by using Constraint Violation (CV) and Mean Average Precision post Coherence correction (CMAP), which are described below.
Note that we do not recommend the use of either CV or CMAP as the sole metrics to evaluate the model, we present these as measures to analyse the coherence of the model predictions.
Constraint violation is a punitive measure that quantifies the extent to which the label scores generated by the model violate the partial ordering of the latent label taxonomy regardless of true labels for the instances.
Hence, lower value of CV implies higher taxonomic coherence in the predictions.
[Equation]
Mean Average Precision post Coherence correction (CMAP) :
Given a complete or partial taxonomy G =
( L , T ) for the labels, coherence can be imposed post-hoc by applying a modification function δ : R L → R L to the label-scores produced by the model such that δ ( s i ) − δ ( s j ) < 0 for
[Table]
MAP ↑ 27.
29.18 CMAP ↑ 28.
29.18 SpoFUN CV ↓ 2.
MAP ↑ 46. 48.61 CMAP ↑ 47.
48.61 ExprGO
CV ↓
[Table]
MAP ↑ 43.
45.61 CMAP ↑ 43.
45.61 CellcycleGO
CV ↓ 5.
MAP ↑ 40.
42.24 CMAP ↑ 40.
42.24 DerisiGO
CV ↓ 3.
MAP ↑ 40.
42.77 CMAP ↑ 41.
42.77 SpoGO
CV ↓ 3.
MAP ↑ 80.
80.04 CMAP ↑ 80.
80.04 Enron
CV ↓ 0.
MAP ↑ 43. 76.23 CMAP ↑ 45.
76.23 Diatoms
CV ↓ 6.
MAP ↑ 83.
90.26 CMAP ↑ 84.
90.26 Imclef07a
CV ↓ 12.
MAP ↑ 87.
89.22 CMAP ↑ 88.
89.22 Imclef07d
CV ↓ 11.
MAP 4.
2.17 CMAP 4.
[Table]
all ( l i , l j ) ∈ T .
This can be achieved by modifying the score
s i for each label l i to be either the maximum ( δ MG ) of the scores of any of its descendants or the minimum of the scores of its ancestors ( δ m G ) in the taxonomy G .
Concretely, given the scores s ∈ R L produced by a model, for an input, the two modification functions are given as
[Equation]
where Anc G ( l ) and Des G ( l ) is the set of ancestors and descendants, respectively, of l in the graph
G .
In practice, if one is given a partial label taxonomy, one would select whichever post-hoc modifi- cation performs better, and we refer to the maximum MAP obtained after applying one of the modification functions δ mG , δ MG to the scores as CMAP.
3 Intuitively, a CMAP value close to MAP value implies that the model is more perceptive of the latent hierarchy in the label space.
As seen from the critical diagram of the Nemenyi test (Figure 2b), when coherence is considered along with the predictive performance (CMAP), MBM outperforms MVM and MHM (with statistical significance) indicating that MBM is much more perceptive of the latent taxonomy compared to other embedding based models (MVM and MHM).
In conclusion, while on one extreme there is MVM, which exhibits reasonable good predictive performance but fails to maintain coherence w.r.t the taxonomy, on the other extreme we have MHM that exhibits lowest constraint violations but gives inadequate predictive performance.
MBM, however, demonstrates good characteristics on both fronts–predictive performance as well as coherence.
Comparison with the state-of-the-art: C-HMCNN (Giunchiglia & Lukasiewicz, 2020), is the current state-of-the-art method for hierarchical MLC, and includes the modification function δ MG (called MCM in their work) in their model directly, applying it to scores produced by a multi-layer perceptron at training as well as inference time.
At training time, this modification is accompanied by a clever modification of BCE loss to form a novel MCLoss which works well with the modification function during training.
There are two key differences between C-HMCNN and MBM that one needs to keep in mind when comparing their performance.
First, C-HMCNN does not attempt to use the label representations themselves to increase coherence—an idea central to MBM and this work.
Second, unlike MBM, C-HMCNN requires the complete label taxonomy at training and inference time to enforce the coherence.
Owing to the use of complete label taxonomy at inference time, C-HMCNN always has 0 constraint violations.
However, as seen from the Table 1 and the Figure 2, MBM, which does not require the label taxonomy at all, performs comparably to C-HMCNN in terms of MAP (p-value of 0.677 in two sided Wilcoxon test between MBM and C-HMCNN shows that there is no statistical difference between their performances). [Figure]
A NALYSIS OF L EARNED L ABEL E MBEDDINGS
In this section, we analyze the geometry of the learned label embeddings, finding that the simple geometry of box embeddings endows the MBM model with high degree of interpretability.
In order to verify that label box embeddings are producing consistent scores by using inclusion in the box space, we inject into the model the taxonomy information through the additional loss term (Eq.
As seen from the right section in Table 1, injecting explicit taxonomic information into the label embeddings further reduces the constrain violation for MBM on all twelve datasets.
However, the same does not aid the MVM model and even drops its predictive performance significantly.
This underscores the importance of embedding geometry for inducing taxonomic coherence, and validates our intuition about the arrangement of label embeddings in MBM.
To determine the extent to which the label embeddings capture the latent label taxonomy without it being explicitly provided, we perform ancestor-descendant classification solely using the learned label embeddings.
Each pair of labels ( l i , l j ) get a score β ( i, j ) that is determined using their corresponding label embeddings ψ i , ψ j .
Since MVM and MBM have different geometrical interpretations, we use different scoring for each.
Specifically, β MVM ( i, j ) =
ψ i · ψ j / ∥ ψ j ∥ , β MBM ( i, j ) =
s MBM ( l i , l j ) .
These scores are then compared to true ancestor-descendant relations in the taxonomy to obtain respective ROC curves as shown in Figure 3.
As seen in Table 3, MBM captures the true label taxonomy the best ( AUC ≥ 0 . 87 ) for all datasets.
1.0 FPR 0.
T P R
Random 0.
BoxEMVMMHMMBM
Dataset
BoxE MVM MHM MBM
ExprFUN 0.
0.89 CellcycleFUN 0.
0.93 DerisiFUN 0.
0.87 SpoFUN 0.
0.87 Enron 0.
0.92 Diatoms 0.
0.96 Imclef07a 0.
[Table]
Furthermore, for hyperberbolic space, it is suggested that the magnitude of embeddings relate to the level of generality in taxonomy (Nickel & Kiela, 2017).
We show that the same observation holds for box embeddings, with the vector embedding magnitude replaced by box embedding volume.
To see this, we compute the Spearman rank correlation between the number of descendants of a node in the true taxonomy and the embedding magnitude, negative embedding magnitude, and embedding volume for MVM, MHM, and MBM, respectively.
The correlation values reported in Table 2 confirm our intuition regarding box embeddings stated above.
It is interesting to note that as shown by the correlation between the vector magnitude and level of the label node as well as low constraint violations, the MHM model does capture the depth of the label nodes in the taxonomy.
However, as shown by the ROC curves in Figure 3, unlike MBM, it fails to capture the exact connectivity of the nodes at different depths in the taxonomy.
C ONCLUSION
In this work, we demonstrate that box embeddings with its probabilistic formulation can effectively capture taxonomic relations present between labels in multi-label classification without requiring explicit access to the taxonomy.
The proposed model achieves a fine balance between predictive performance and coherence.
Furthermore, we find that the taxonomic relationships between labels can be easily injected via extra supervision during training, increasing the coherence of the predictions further.
Since the model has the same computational complexity as a simple neural network model, and unlike C-HMCNN it does not require the label taxonomy, the proposed model can be scaled to work with extremely large label spaces.
Moreover, due to the flexibility of using any neural network encoder this model can be easily extended to different input modalities, such as raw text, images, etc.
We wish to pursue these two directions in our future work.
R EPRODUCIBILITY S TATEMENT
In this section, we provide pointers to information necessary to reproduce the results mentioned in this paper.
The description of the datasets with various statistics, links to download them, and instructions to pre-process them are provided in
Appendix B. The BoxE baseline is further described with expressions for distance and the margin based loss function in Appendix A.1.
The hyperparameter settings for each model-dataset combination (all 96 settings) used to produce the reported results are provided in 3.
We also include a discussion on computational complexity of the models in Appendix A.2.
Finally, the code for all the models (including the baselines) used in this paper is available at https://github.com/iesl/box-mlc-iclr-2022 . A CKNOWLEDGEMENTS The authors would like to thank the members of the Information and Extraction Synthesis Laboratory (IESL) at UMass Amherst for helpful discussions. This work was partially supported by IBM
Research AI through the AI Horizons Network and the Chan Zuckerberg Initiative under the project
Scientific Knowledge Base Construction.
Additional support was provided by the National Science Foundation (NSF) under Grant Numbers IIS-1763618, IIS-1922090, and IIS-1955567, the Defense Advanced Research Projects Agency (DARPA) via Contract No. FA8750-17-C-0106 under Subaward No. 89341790 from the University of Southern California, and the Office of Naval Research (ONR) via Contract No. N660011924032 under Subaward No. 123875727 from the University of Southern California. The U.S. Government is authorized to reproduce and distribute reprints for
Governmental purposes notwithstanding any copyright notation thereon.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IBM, CZI, NSF, DARPA, ONR, or the U.S. Government.
A I MPLEMENTATION D ETAILS
In this section, we describe the implementation details: training procedure and hyper-parameter search.
Frameworks used: We implement all the models described in this work using PyTorch (Paszke et al., 2019).
We also make use of NLP specific abstractions over PyTorch provided by AllenNLP (Gardner et al., 2017).
We also make use of the abstractions provided in the Box Embedding library
Chheda et al. (2021) in our implementation of MBM. For the extensive hyper-parameter search, we use the academic version of Weights & Biases library (Biewald, 2020). Data pre-processing: The datasets were pre-processed in the exact same manner as in Giunchiglia & Lukasiewicz (2020), i.e., the categorical features were converted to one-hot feature vectors and the continuous input features were standardized.
Also, following the Giunchiglia & Lukasiewicz (2020) ; Wehrmann et al.
(2018a), the labels corresponding to the root nodes were removed from training and evaluation. Further details of the original datasets is provided in B.
Training: We used ADAM (Kingma & Ba, 2017) with a batch size of 4 to learn the model parameters for all the models.
Since the naive implementation of the label interaction loss described in
Eq. 4 is too expensive to compute at each mini-batch step, we approximate it by randomly sampling (without replacement), at each mini-batch step, a subset of edges ˜ T ∼ T .
The size of the sampled set is a hyper-parameter (last column in Table 3).
Hyper-parameter search:
The following hyper-parameters were searched using the performance on the validation set: optimizer learning rate, hidden dimensions of the MLP, number of hidden layers (maximum of 3), activation functions, dropout probabilities, weight decay and sampling percentage for the labels for computing the label interaction loss( L G ) for the MHM-T, MBM-T and MVM-T models.
After an initial investigation, we identified that the softplus activation performed the best for all models and hence, the activation was fixed to softplus for the final grid search.
Also note that in order to be comparable to the previous work that uses these datasets, we had to divide our experiments into two kinds of runs—the search runs and the final runs. First, in the search runs , we use a validation set to obtain the best hyper-parameters for each model-dataset combination. After identifying the best set of hyper-parameters the training and validation sets were combined. We call the models trained on this combined dataset as the final runs .
Since one cannot use early stopping without a validation set, the number of epochs to train the models in the final runs was also obtained as a hyper-parameter and was set to the best epoch for the corresponding search run.
All the results are reported using the metrics obtained in the final runs (10 runs with different random seeds for each dataset-model pair with best hyper-parameters obtained from the corresponding search runs).
Table 3 summarizes the search ranges used and Table 4 presents the final hyper-parameters obtained.
[Table]
It should be noted that the activation function chosen for all model configurations was softplus.
[Table]
[Table]
1e-07 DerisiGO
MHM 0.
- - DerisiGO
MHM-T 0.
1e-07 DerisiGO
MVM 0.
- - DerisiGO
MVM-T 0.
1eDiatoms
BoxE 0.
- - Diatoms
C-HMCNN 0.
- Diatoms
MBM 0.
- - Diatoms
MBM-T 0.
1e-07 Diatoms
MHM 0.
- - Diatoms
MHM-T 0.
1e-07 Diatoms
MVM 0.
- - Diatoms
MVM-T 0.
1eEnron
BoxE 0.
- - Enron
C-HMCNN 0.
- Enron
MBM 0.
- - Enron
MBM-T 0.
1e-07 Enron
MHM 0.
- - Enron
MHM-T 0.
1e-07 Enron
MVM 0.
- - Enron
MVM-T 0.
1eExprFUN
BoxE 0.
- - ExprFUN
C-HMCNN 0.
- ExprFUN
MBM 0.
- - ExprFUN
MBM-T 0.
1e-07 ExprFUN
MHM 0.
- - ExprFUN
MHM-T 0.
1e-07 ExprFUN
MVM 0.
- - ExprFUN
MVM-T 0.
1eExprGO
BoxE 0.
- - ExprGO
C-HMCNN 0.
- ExprGO
MBM 0.
- - ExprGO
MBM-T 0.
1e-07 ExprGO
MHM 0.
- - ExprGO
MHM-T 0.
1e-07 ExprGO
MVM 0.
- - ExprGO
MVM-T 0.
1eImclef07a BoxE 0. - - imclef07a C-HMCNN 0.
[Table]
A. F URTHER D ETAILS FOR THE BASELINES
BoxE : Let p ∈
R
d be a point and z, Z ∈ R d with z
i <
Z i be the lower left and upper right coordinates of a non-probabilistic label box.
Then a non-probabilistic compatibility score between the point and the box can be defined as shown in Abboud et al.
Concretely, Abboud et al.
(2020) define per-dimension distance as dist
[Equation]
Z
[Equation]
otherwise
[Equation]
[Equation]
where c i =
( Z i − z i ) / 2 , w i
=
Z i − z
i are the center and width of the projection of the box (i.e., an interval) in dimension i , and κ = 0 .
5 ∗ w i ∗ ( w i + 1 − [Equation] 1 ) is the width dependent factor.
The score function that measures the compatibility of the point p and box ( z, Z ) is defined as s BoxE ( p, z, Z ) =
−∥ v ∥ 2 , where v i = dist ( p i , z i , Z i ) .
Given the trainable parameters ψ − , ψ + ∈ R L × d for representing the labels, the lower left and the upper right coordinates of a non-probabilistic box for label l i are taken to be z i = ψ − i, ∗ and Z i = ψ − i, ∗ + log(1 + exp ψ
+ i, ∗ ) , respectively.
The input x is encoded as a point p ∈ R d using a multi-layer perceptron F θ , just like MVM.
However, since the score is negative distance, the parameters are learnt using margin based loss given as:
[Equation]
where P i and N i are the positive and negative labels for i -th data instance, respectively.
A. C OMPUTATIONAL COMPLEXITY
The computational complexity of the MBM can be divided into two parts–the computation of the instance box Box θ ( X ) , and the computation of probability score P ( Y | X ) given the encoding of the instance and label box parameters.
Computing Box θ ( X ) amounts to splitting the output of a feed-forward network ϕ
( x ) into min and max parameters ϕ ( x ) − , ϕ ( x ) + , and thus has the same complexity as that of the MVM.
Assuming that the operations (cid:80) , (cid:81) , log and exp have unit complexity, computing P ( Y | X ) also has equivalent complexity.
The computation of conditional probability for a single label y k using the MVM model involves computing σ ( ϕ ( x ) · y k ) , an O ( dL ) operation.
Now, with same assumptions for the MBM, computing λ (Box θ ( x ) ∩ Box ψ ( y k )) involves d invocations of log(exp( · ) + exp( · )) followed by d subtractions and d invocations of log(1 + exp( . )) , resulting in O ( d ) operations.
The computation of λ (Box θ ( x )) omits the intersection calculation, but otherwise is the same, and thus is also O ( d ) .
Hence, calculating λ (Box( x ) ∩ Box( y k )) λ (Box( x )) is O ( d ) , resulting in overall complexity of O ( dL ) , which is equal to that of MVM.
We note that the complexity of our model is not dependent on the depth of the hierarchy, as in C-HMCNN, and thus can reasonably scale to arbitrarily deep hierarchies as might be present in extreme multi-label classification.
These theoretical statements are supported by Table 5, which provides average epoch duration in seconds for MVM, MBM, MBM-T, and C-HMCNN with the same hidden size.
Hence, we can conclude that the computational complexity of MVM ≈ MBM <
MBM-T. with the same hidden size
[Table]
Computational resources used: For datasets with number of labels less than 500, i.e., the 4 FUNCAT datasets, Imclef07a, Imclef07d, Diatoms and Enron, all the models were trained on TitanX
GPU (memory=12GB).
For the 4 GO datasets that have number of labels greater than 4000, all the models are trained on M40 GPU (memory=24GB).
A. C ODE
Executable python code with detailed instructions to reproduce the results reported in 1 is provided using at https://github.com/iesl/box-mlc-iclr-2022 .
We also include instructions for obtaining the pre-processed datasets, training a new model from scratch (MBM or any baseline), evaluating a pre-trained model on test set, directly downloading the pre-trained models for datasets.
B D ATASETS
The 12 datasets used in this work wary greatly in terms of domain, number of labels, number of instances, and the connectivity of label taxonomy.
The characteristics of each dataset w.r.t these properties is summarized in 6.
These datasets do not require a licence and are available for public usage.
The links to the sources for all the datasets are provided in Table 7 below.
The continuous features are standardized by removing mean and scaling to unit variance, and the categorical features are encoded as one-hot vectors. [Table] [Table]
C S IGNIFICANCE T ESTING
In section 6, we provide the results for Friedman test followed by post-hoc Nemenyi test, and also the result of pairwise Wilcoxon test where the former are not sufficient, i [Table] e. for the case of MBM vs
C-HMCNN w.r.t MAP. For the sake of completeness, table 8 provides results of pairwise Wilcoxon test comparing all models with MBM and MBM-T w.r.t all three metrics. [Table]
D C OMPARISON TO C-HMCNN AND HMCN IN TERMS OF AU(PRC)
Due to our focus on coherence and predictive performance at the same time [Table] we use mean average precision as the metric to evaluate predictive performance.
One could, however, also use AU(PRC) as the metric for predictive performance. For the sake of completeness, in Table 9, we present the AU(PRC) values for the final models obtained in our experiments, and compare it with those provided in Giunchiglia & Lukasiewicz (2020) and Wehrmann et al.
(2018a).
We believe that the difference between the AU(PRC) reported for C-HMCNN in Giunchiglia & Lukasiewicz (2020) and our work is due to the difference in the hyper-parameter search strategy. Specifically, we use MAP as the reference metric to identify the best hyper-parameters while Giunchiglia & Lukasiewicz (2020) uses AU(PRC) itself.
[Table]
[Table]
E MAP AND CMAP
In MCL problems, where predictions consistent with a taxonomy are necessary, the model designer might be ready to sacrifice some absolute predictive performance for increased coherence, where the predictive performance can be measure using Mean Average Precision (MAP).
First recall that
Average Precision (AP) for an instance is the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight, and Mean Average
Precision (MAP) is the mean of AP across instances.
Given data D , let s ( k ) i
[Equation]
denote the score generated by the model for label i given the input x ( k ) .
Then MAP is computed as
[Equation]
As described in Section 6.1, given the scores s ∈ R L produced by a model, for an input, the two post-hoc modification functions that can be applied to make these score coherent are given as
[Equation]
where Anc G ( l ) and Des G ( l ) is the set of ancestors and descendants, respectively, of l in the graph
G .
One can compute MAP after applying either one of the two modification functions.
If a human were to make consistent score corrections based on the complete hierarchy, they would either pick the score of a label to be the minimum of its ancestors or maximum of its descendants.
Once picked, the same approach has to be applied to all the labels.
We argue that in terms of correlation with human judgment, both these methods are equivalent.
Hence, one can pick the modification function based on empirical performance of the post-hoc correction algorithm.
We observe that δ mG , i.e., “min of ancestors” approach, produced higher MAP with both MVM and MBM models compared to δ MG .
as can be seen in table 10.
We believe that ”min of ancestors” works better because it promotes sparsity in predictions, which is essential for the task of MLC.
[Table]
F P ERFORMANCE ON LONG TAIL OF LABELS
Since MVM and MBM both are embedding based models, we expect the performance of the MBM model to wary with the label frequency in the same manner as MVM.
This hypothesis is corroborated by the following table which shows the spearman rank correlation between the label frequency and the mean average precision (MAP) for that label.
The high positive correlation for all the models suggests that the performance of both MVM and MBM often degrades as the frequency of the labels drop.
However, as shown by the table 11, both the models perform reasonably well for extremely low frequency labels.
Here, tail MAP is the mean average precision for labels with frequency 0.001% or lower.
[Table]
G E FFECT OF EMBEDDING DIMENSION
To study the effect of embedding dimensions on the performance, we plot the MAP and CV vs the hidden dimension size for a subset of the datasets.
As shown in figure 4, at extremely low dimensions, the MBM model consistently outperforms the the most competitive embedding based model
MVM, demonstrating the usefulness of the favorable inductive bias of the box space in inducing parsimonious representations. [Figure]
H E FFECT OF HIERARCHY ON THE MARGINAL SCORE
[Figure]