Hyperbolic Embedding Inference for Structured Multi-Label Prediction
Abstract
We consider a structured multi-label prediction problem where the labels are organized under implication and mutual exclusion constraints.
A major concern is to produce predictions that are logically consistent with these constraints.
To do so, we formulate this problem as an embedding inference problem where the constraints are imposed onto the embeddings of labels by geometric construction .
Particularly, we consider a hyperbolic Poincaré ball model in which we encode labels as
Poincaré hyperplanes that work as linear decision boundaries.
The hyperplanes are interpreted as convex regions such that the logical relationships (implication and exclusion) are geometrically encoded using insideness and disjointedness of these regions, respectively.
We show theoretical groundings of the method for preserving logical relationships in the embedding space.
Extensive experiments on 12 datasets show 1) significant improvements in mean average precision; 2) lower number of constraint violations; 3) an order of magnitude fewer dimensions than baselines.
Introduction
Structured multi-label prediction is a task aiming to associate every object with multiple labels that are semantically constrained in a structured manner (e.g., by implication and exclusion constraints).
This task is of growing importance in many applications such as image annotation [1, 2], text categorization [3, 4] and functional genomics [5, 6].
One of the central concerns of the task is to produce predictions that are logically consistent with the constraints of the labels.
For example, a protein must be labeled to have the function nucleic acid binding if it is already labeled to have the function RNA binding (i.e., implication) and must not have the function drug binding (i e., mutual exclusion).
Various works have been proposed to improve the prediction consistency [7, 8, 9, 10, 11].
One line of work called label embedding aims to represent labels as low-dimensional vectors [12, 13].
A key disadvantage of the vector-based representations is that they only capture weak forms of correlation or “similarity’ between labels, but do not strongly enforce the logical relationships.
Another line of work [7, 9, 14, 8] imposes these logical constraints directly to the losses of neural networks.
However, they do not explicitly learn the representations of labels and typically require a complete label taxonomy, which is not always available in and scalable to real-world settings [11].
[Figure]
p e r s o n w o m e n g i r l p a r e n t m o t h e r p l a n t t r e e f a t
h e r 0.
(c)
Embedding-based inference [15], which imposes logical constraints directly to the label embeddings, is able to inductively infer the underlying label relationships from incomplete labelings [16].
Once all embeddings are adhering to the constraints, each label can be predicted independently without accessing the label relationships, which significantly reduces the computation cost during inference [15].
The key idea, which is inspired by the Venn diagram [16, 17] or set-theoretic semantics [18], is to represent each label as a convex region [15].
A prominent example is the multi-label box model (MBM)
[11] that models label implications as box containments.
However, MBM learns box-like decision boundaries, which are typically not compatible with standard classifiers (i.e., hyperplane margin-based models such as logistic regression [19]).
Besides, box models suffer from a theoretical limitation, i.e., lower-way intersections enforce higher-way interactions [20].
Finally, current methods ignore the importance of constraining mutual exclusion, which is essential as otherwise, a model could trivially obtain zero implication violation by assigning the same score to all labels.
In this paper, we consider a structured multi-label prediction problem with implication and mutual exclusion constraints that are jointly described by a hierarchy and exclusion (HEX) graph (see
Figure 1(a) for an example).
The key idea of our method is to transform the logical constraints into soft geometric constraints in the embedding space.
In particular, we consider a hyperbolic Poincaré ball model that has demonstrated advantages in representing hierarchical data [21, 22] and assign each label a Poincaré hyperplane that has several favorable theoretical properties in classification.
Each Poincaré hyperplane can be interpreted as a convex region such that the implication and mutual exclusion are modeled by geometric insideness and disjointness between the corresponding regions, respectively.
In this way, a multi-label classifier can be defined by measuring the confidence of an instance having a label as geometric membership .
Unlike other hyperbolic region-based models such as hyperbolic cones [23] and hyperbolic disks [24], Poincaré hyperplane works as a linear decision boundary and can be seamlessly integrated into existing margin-based classifiers such as hyperbolic logistic regression [19].
Figure 1(b) shows an example of the learned label representations that respect all the constraints given in Figure 1(a).
We show theoretical groundings of the proposed method on modeling implication and mutual exclusion .
Extensive experiments on 12 multi-label classification tasks show the model’s capability to improve the mean average precision significantly while keeping the number of constraint violations low and requiring an order of magnitude fewer dimensions.
Preliminaries
Poincaré ball model
The Poincaré ball (cid:0) D n , g D (cid:1) is one of the models of hyperbolic geometry that is very suitable for representing hierarchies due to its exponentially growing volume [25].
The
Poincaré ball is defined as an open n -ball D n = { x ∈ R n : ∥ x ∥ < 1 } equipped with a Riemannian metric g D x
= λ 2 x g E , where λ x = 2 1 −∥ x ∥ 2 , g E =
I n is the Euclidean metric tensor, λ x is the conformal factor , and ∥·∥ 2 denotes the L 2 norm in Euclidean space.
The distance between two points x , y ∈ D n can be defined by d D ( x , y ) =
cosh − 1 (cid:16) 1 +∥ x − y ∥ 2 (1 −∥ x ∥ 2 )(1 −∥ y ∥ 2 ) (cid:17) .
Structured multi-label prediction Let X ⊆ R n denote an n -dimensional instance space and L = { l 1 , l 2 , . . .
} , |L| ≥ 2 denote the finite set of possible labels.
Given a set of N training examples D
= { ( x i , L i ) | 1 ≤ i ≤ N, x
i ∈ X , L i ⊂ L} , multi-label prediction aims to learn a labeling function f : X → 2 L mapping from the instance space to the powerset of the label space, f ( x ) ⊂ L .
Structured multi-label prediction additionally imposes a set of prior-known logical constraints over the labels, namely, the predictions must be logically consistent with these constraints.
Analogous to Mirzazadeh et al.
[15], we consider two forms of logical constraints between labels: implication and mutual exclusion.
Specifically, an implication of the form l a ⇒ l b imposes the constraint that whenever an instance is labeled as l a then it must also be labeled as l b , i.e., l a ⇒ l b is a shorthand notation for ∀ x ∈ X , l a ∈ f ( x ) ⇒ l b ∈ f ( x ) .
Mutual exclusions are constraints of the form ¬ l a ∨ ¬ l b , implying that an instance cannot be simultaneously labeled as l a and l b , i.e., ¬ l a ∨ ¬ l b is a shorthand notation for ∀ x ∈ X , l a / ∈ f ( x ) ∨ l b / ∈ f ( x ) .
We can concisely represent a set of implication and exclusion constraints with a hierarchy and exclusion (HEX) graph [2].
Definition 1 (HEX graph [2] 2 ) .
A HEX graph G =
( V, E h , E e ) is a graph consisting of a set of nodes V
= { v 1 , . .
. , v n } , directed (hierarchy) edges E h ⊆
V × V , and undirected (exclusion) edges
E e ⊆ V × V , such that the subgraph G h
= ( V, E h ) is a DAG and the subgraph G e
=
( V, E e ) has no self loop.
Each node v i ∈ V represents the label l i .
A directed edge ( v i , v
j ) ∈
E h represents the implication l i ⇒ l j , and an undirected edge ( v i , v
j ) ∈
E e represents the exclusion ¬ l i ∨ ¬ l j .
Note that an arbitrary HEX graph may contain redundant edges.
A hierarchy edge ( v i , v j ) is redundant when there is a path in G h from v i to v j which does not contain the edge ( v i , v j ) .
Similarly, an exclusion edge ( v i , v j ) is redundant when there is another exclusion edge connecting their ancestors (or connecting one node’s ancestor to the other node).
We can transform a HEX graph into an equivalent HEX graph by adding or removing redundant edges.
In this paper, we only consider HEX graphs that have a minimal number of edges, we call such HEX graph a minimal sparse HEX graph (see Fig. 1(a) for an example).
Given a minimal sparse HEX graph, we define the HEX-property as
Definition 2 (HEX-property) .
A labeling function f has the HEX property with respect to a HEX graph G if for all x ∈ X , f ( x ) respects all constraints represented by G .
We also call such function f logically consistent w.r.t G .
Given the HEX graph and the HEX-property, structured multi-label prediction is formally defined as a constrained optimization problem.
Definition 3 (Structured multi-label prediction) .
The structured multi-label prediction task with respect to a training set D = { ( x i , L i ) | 1 ≤ i ≤ N, x i ∈ X , L i ⊂ L} , minimal HEX graph G =
( V, E h , E e ) , and multi-label prediction function f , is the task of learning f such that the function f minimizes (cid:80) ( x i ,L i ) ∈D loss( f ( x i ) , L i ) , with loss a predefined function, while attempting to maintain the HEX-property with respect to G .
Note that this definition allows for a soft interpretation of the constraints, meaning that the goal is to adhere to all of them, but we do allow for loosening some if necessary.
For example, a mutual exclusion constraint is allowed to loosen when an instance (e.g., image), though rarely happens, is simultaneously labeled as two mutual exclusive labels (e.g., dog and cat ).
Hyperbolic embedding inference
We consider learning a real-valued ranking function h : X × L (cid:55)→ [0 , 1] , where the output is interpreted as the confidence of an instance x ∈ X having a label l ∈ L .
Afterward, a binary multi-label classifier f : X → 2 L can be simply obtained by thresholding the ranking function with a threshold t , i.e., f ( x ) =
{ l | h ( x, l ) ≥ t, ∀ l ∈ L} .
The objective of h is to assign higher scores to positive instance-label pairs than that of negative instance-label pairs.
3.1 Geometric construction
Given an n -dimensional Poincaré ball D n , we associate each instance x i ∈ X with a point in the
Poincaré ball and associate each label l i ∈ L with a Poincaré hyperplane, such that its corresponding positive and negative instances are correctly separated by the hyperplane.
Poincaré hyperplanes Let B n denote the set of n -balls in R n whose boundaries ∂ B n intersect the
Poincaré ball D n perpendicularly.
Poincaré hyperplanes are defined by ∂ B n ∩ D n (see Fig. 2(a)) plus all linear subspaces going through the origin.
For the former cases, a Poincaré hyperplane can be uniquely defined by its center point that has a minimal distance to the origin.
Definition 4.
Given a (center) point c ∈ D n where c ̸ = 0 , the Poincaré hyperplane is defined as
[Equation]
1) where c is the center point and ⃗ c ∈ T c D n 3 is the normal vector passing through the origin 0 .
Intuitively, this corresponds to the union of all geodesics passing through c while orthogonal to the normal vector ⃗ c ∈ T c D n .
In the case where c is the center of the hyperplane, ⃗ c must simultaneously pass through c and the origin.
Hence, ⃗ c can be simply taken as c without loss of generality.
For the special case where c = 0 , the Poincaré hyperplanes are all linear subspaces (Euclidean planes) passing through the origin.
In this paper, we exclude these special cases by assuming c ̸ = 0 .
Geometric intuition Essentially, the Poincaré hyperplane works as a linear decision boundary that separates the embedding space into two regions, 4 where the smaller region (i.e., convex hull) is interpreted as the space of positive samples while the other one is interpreted as the space of negative samples.
Two reasons motivate us to model labels as Poincaré hyperplanes: 1) Modeling labels as hyperplanes has several desired theoretical advantages in margin-based classifiers.
Our model shares the same philosophy as existing learning frameworks such as hyperbolic logistic regression [19] and hyperbolic SVM [26]; 2) More importantly, unlike Euclidean space that is flat, hyperbolic Poincaré ball is a curved space in which there are infinitely many non-parallel hyperplanes which do not intersect, implying that linear decision boundaries in hyperbolic space can capture more complicated set-theoretic interactions, such as implication and mutual exclusion.
Enclosing balls Given a Poincaré hyperplane H c , we call the corresponding n -ball B n c that encloses H c its enclosing n -ball.
Formally, an enclosing n -ball B n ( o , r ) is defined by B n ( o , r )
= { p : ∥ p − o ∥ ≤ r } , where o ∈ R n and r are the center point and the radius, respectively.
Given H c , we have the following closed-form representation of B n c .
Proposition 1.
Given a Poincaré hyperplane H c where c ̸ = 0 , there exists an n -ball B n c ( o c , r c ) such that H c ⊂ B n c ( o c , r c ) , i.e., H c is a subset of B n c ( o c , r c ) .
B n c is uniquely given by
[Equation]
Proof sketch.
The key idea is to solve a quadratic equation given by the fact that the radius of B n c , the radius of D n , and the distance from the center of D n to the center of B n c must satisfy the Pythagorean theorem [27].
Full proof is in the supplementary material.
3.2 Geometric interpretation
Our main idea is to transform the logical relationships between labels into geometric relationships between their corresponding enclosing n -balls.
In particular, the implication is modeled by the geometric insideness while the mutual exclusion is modeled by the geometric disjointness.
Implication The logical implication between two labels is interpreted as geometric relations between n -balls, i.e., n -ball insideness illustrated in Fig. 2(b).
In particular, an n -ball B w ( o w , r w ) contains B u ( o u , r u )
if and only if ∥ o u − o w ∥ + r u < r w , and thus we can create an insideness loss defined by
[Equation]
r w } .
Clearly, the insideness loss term satisfies the properties of correctness and transitivity
Lemma 1 (Correctness) .
B u is inside of B w if and only if L inside ( B u , B w ) = 0 .
Lemma 2 (Transitivity) .
If L inside ( B u , B w ) =
and L inside ( B w , B v ) = 0 , we have L inside ( B u , B v ) ≤
L inside ( B u , B w ) + L inside ( B w , B v ) ≤ L inside ( B w , B v ) = 0 .
[Figure]
𝔹
[Figure]
𝔹 # 𝔹
[Figure]
Mutual exclusion Similarly, we interpret mutual exclusion as geometric disconnectedness between n -balls illustrated in Fig. 2(c).
B u disconnecting from B w can be measured by subtracting the distance between their center points from the sum of their radii.
Inversely, the corresponding loss is
[Equation]
Again, the disjointedness loss term satisfies the correctness property
Lemma 3 (Correctness) .
B u disconnects from B w if and only if L disjoint ( B u , B w ) = 0 .
3.3 Classification and learning
Given the embeddings of instances and labels, an instance can be classified by measuring the geometric membership , i.e., the confidence of a point
p ∈ D n being inside the enclosing ball B .
Membership and non-membership Formally, given an instance embedding p ∈ D n and a label embedding associated with an enclosing n -ball B c .
The confidence of an instance p being inside the enclosing n -ball B c can be measured by subtracting the distance between the center point of B c and p from the radius of B c .
The corresponding loss is defined as the inverse of the measure, given by
[Equation] [Equation]
Symmetrically, for negative instance-label relations, the loss of non-membership can be defined as
[Equation]
Clearly, we have the following properties that follow directly from the definitions.
Lemma 4.
A point p is a member of B c if and only if L membership ( p , B c ) = 0 .
Lemma 5.
A point p is not a member of B c if and only if L non-membership ( p , B c ) = 0 .
Lemma 1-2, Lemma 3, Lemma 4-5 immediately follow the definitions of geometric insideness, disjointedness, and membership, respectively.
We aim to learn an encoder E θ (i.e., a hyperbolic neural network whose designs depend on the datasets), where θ is the trainable parameter, and a function C which maps labels to the center points of the corresponding Poincaré hyperplanes in the Poincaré ball.
Now, we define h ( x, l ) =
σ ( L non-membership ( E θ ( x ) , C ( l )) − L membership ( E θ ( x ) , C ( l ))) , as our ranking function, where σ is the sigmoid function.
The final classification function is then defined by f ( x ) = { l | h ( x, l ) ≥ 0 .
We call our classifier hyperbolic multi-label embedding inference (HMI).
Given a HEX graph, HMI has the following guarantee.
Proposition 2 (HEX-property) .
The classification function f of HMI has the HEX property with respect to G if for every constraint in G , the corresponding loss term is 0 .
Learning with soft constraints Let D + =
{ ( x i , l n ) | ( x i , L i ) ∈ D , l n ∈ L i } be the set of positive instance-label pairs and D - = { ( x i , l n ) | ( x i , L i ) ∈ D , l n ∈ L , l n / ∈ L i } be the set of negative instance-label pairs.
By combining the loss functions of membership, non-membership, insideness and disjointedness, the learning objective can be formulated as min θ, C (cid:88)
[Equation]
∈D +
L membership
[Equation]
θ (
[Equation]
+ (cid:88) ( x i
[Equation]
l n ) ∈D −
L non-membership (cid:0) E θ ( x i )
[Equation]
B C ( l n ) (cid:1)
+
[Equation]
  (cid:88)
[Equation]
v i
[Equation]
v
[Equation]
h
L inside (cid:0)
[Equation]
C ( l
[Equation]
C ( l
[Equation]
(cid:1) + (cid:88) ( v i ,v
[Equation]
[Equation]
E e
L disjoint (cid:0)
[Equation]
C ( l
[Equation]
C ( l
[Equation]
(cid:1) (7)
The first two terms are losses for positive and negative samples while the last two terms are implication and exclusion constraints, respectively, with λ being the penalty weight of the constraints.
The following corollary shows that our model has a strong inductive bias for preserving consistency.
Corollary 1.
Given a HEX graph G of labels, if the loss terms L inside and L disjoint are 0 , then the learned prediction function is logically consistent.
Classification via hyperbolic logistic regression A key advantage of our method is that the losses of constraints are compatible with other (margin-based) hyperbolic classifiers such as hyperbolic logistic regression (HLR) [19] and hyperbolic support vector machine (HSVM) [26].
In our experiment we explore HLR, which formulates the logits as the distances from an instance to a Poincaré hyperplane of a label.
That is, h ( x, l ) =
d ( E θ ( x ) , H C ( l )) .
d ( p , H c ) has the following closed form:
[Equation]
where ⊕ is the Möbius addition [19].
The classifier is defined by f ( x ) = { l | σ ( h ( x, l )) ≥ 0 .
5 , ∀ l ∈
L} where σ is the sigmoid function.
We dub such classifier combined with HMI as HMI+HLR.
Evaluation 4.1 Experiment setup
Datasets We consider 12 datasets that have been used for evaluating multi-label prediction methods [11, 8, 10].
These consist of 8 functional genomic datasets [28], 3 image annotation datasets [29, 30], and 1 text classification dataset [31].
All input features are pre-processed in the same way as described by Patel et al.
For all datasets, the implication constraints (label taxonomy) are given.
Following Mirzazadeh et al.
[15] we add exclusion constraints between sibling nodes whenever this does not create a contradiction (i.e., they share no common descendant nodes).
We also explore other strategies for deriving exclusions, but no significant difference was observed (see the supplement for an analysis).
Similar to MBM [11] and its baselines, we sample 30% of the implications and exclusions constraints for training the model.
Hyperbolic encoder We adopt a simple hyperbolic linear layer as the instance encoder for all datasets.
A single-layer hyperbolic fully-forward linear layer is defined by f θ = { W , b } ( x ) = tanh ⊗ ( W ⊗ x ⊕ b ) , with ⊗ being Möbius matrix-vector multiplication defined by M ⊗ x = tanh (cid:16)
∥
M x ∥ ∥ x ∥ tanh − 1 ( ∥ x ∥ ) (cid:17)
M x ∥
M x ∥ , where W ∈ R n × d is a trainable matrix and x is a point x ∈
D n , M x ̸ = 0 .
⊕ denotes Möbius addition given by x ⊕ y =
( 1+2 ⟨ x , y ⟩ + ∥ y ∥ 2 ) x
+ ( 1 −∥ x ∥ 2 ) y 1+2 ⟨ x , y ⟩ + ∥ x ∥ 2 ∥
y ∥. tanh ⊗ denotes an Möbius version of pointwise non-linearity given by tanh ⊗ ( x ) = exp 0 (tanh (log 0 ( x ))) , with exp 0 and log 0 being the exponential and logarithmic maps, see [19] for more details.
Baselines We compare our approach with both classical vector-based and state-of-the-art regionbased embedding methods.
In particular, we consider two vector-based models: 1) The multi-label vector model (MVM) [32], which encodes both inputs and labels as Euclidean vectors; 2) the multilabel hyperbolic model (MHM) used by Chen et al.
[13], which represents inputs and labels as hyperbolic points; and two box models: 3) the non-probabilistic box model (BoxE) [33] and 4) the probabilistic multi-label box model (MBM) [11] that encodes both instances and labels as axis-parallel hyper-rectangles.
Besides, we compare with 5) hyperbolic logistic regression (HLR) [19] since it also encodes labels as Poincaré hyperplanes (but does not use geometric constraints).
Furthermore, we
[Table]
CellcycleGO
[Table]
DerisiGO
[Table]
Enron
[Table]
compare with 6) C-HMCNN, a state-of-the-art non-embedding based method that injects hierarchy constraints directly into the loss function without embedding labels.
A notable difference is that
C-HMCNN needs the full hierarchy constraints as its input.
Finally, we also implement HMI+HLR, a combination of our proposed constraints with HLR for an ablation study.
Implementation details We implement HMI, HLR and HMC-HLR using PyTorch [34] and train the models on NVIDIA A100 with 40GB memory.
We train HMI, HLR and HMI+HLR using
Riemannian Adam [35] optimizer implemented by the Geoopt library [36] with a batch size of 4 .
We also explore some larger batch sizes but it does not yield better results, which is also observed in Wehrmann et al.[14].
We set the dropout rate to 0 .
6 suggested by [14] to avoid the case that the model overfits the small training sets.
We employ an early-stopping strategy with patience 20 to save training time.
The results of other baselines are as reported by Patel et al.[11] that we closely follow.
The learning rate is searched from { 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 , 1 e − 2 }.
The penalty weight of the violation is searched from { 1 e − 5 , 5 e − 4 , 1 e − 4 , 5 e − 3 , 1 e − 2 } and we also show its impact in an ablation.
The best dimension per dataset is searched from { 32 , 64 , 128 , 256 } , which is one order of magnitude lower than that used by Patel et al.
All methods have been run 10 times with random seeds and the average results are reported.
We omit the standard deviations since they are in a very small range ( [2 × 10 − 4 , 2 . 3 × 10 − 3 ] ).
Our code is open available at 5 .
[Figure]
Evaluation protocols In line with Patel et al.
[11], we consider Mean Average Precision (mAP) ,which summarizes the information of precisions and recalls with varied thresholds.
We also report two metrics that additionally take the constraints into account: 1) Constrained mAP (CmAP) is a variant of mAP that replaces the score of each label with the maximum scores of its descendant labels in the hierarchy [11].
2) Hierarchy Constraint Violation (HCV) [11] measures the extent to which the label scores violate the implication constraints regardless of true labels for the instances.
HCV is computed as HCV =
[Equation]
, where h i means the prediction score of label l i .
Clearly, a lower value of HCV implies higher consistency in the predictions.
4.2 Main results
As Table 1 shows, our method HMI either achieves the best (7-8/12 datasets) or competitive (4-5/ datasets) performance (mAP and CmAP) over all compared methods.
HMI outperforms all methods w.r.t the average ranking of mAP/CmAP, showcasing the advantages of HMI.
We observed that the
CmAP is close to mAP, indicating that the model is adhering to the label constraints [11].
In terms of predictive consistency (HCV), HMI consistently achieves the best or the second-best results.
Note that C-HMCNN always gets zero HCV because it exploits the complete hierarchy.
HMI achieves competitive HCV, despite only using 30% of the hierarchy.
[Table]
Statistical significance Following Patel et al.
[11] and Giunchiglia and Lukasiewicz [8], we test the statistical significance of the performance across all datasets.
First, we perform the Friedman test [37] and show that there exists a significant difference w.r.t.
all metrics with p-values ≪ 0 .
Next, we conduct the post-hoc Nemenyi test to verify the statistical differences of the average ranking.
The critical diagram w.r.t the average ranking of mAP/CmAP is shown in Fig.
3, in which the methods that have no significant differences (significance level 0 . 05 ) are connected by a horizontal line.
As shown in the diagrams, it is clear to conclude that there is a statistically significant difference w.r.t mAPs/CmAPs of HMI and HMI+HLR against MVM, BoxE,
MHM, and HLR but not the two strong baselines (MBM and C-HMCNN).
We further perform the
Wilcoxon test that considers not only the differences in rankings but also the numerical differences in the performance.
The Wilcoxon test results show that there is a statistically significant difference between the mAPs/CmAPs of HMI and the two strong baselines with p-value ≪ 0 .
In terms of HCV, our statistical significance test in Figure 3 and Table 2 shows that HMI and HMI+HLR significantly outperform MVM, BoxE, MHM, HLR, and MBM but not C-HMCNN since it has zero
HCV.
However, we observed that the predictive performance (mAP, CmAP) is not fully proportional to the HCV, e.g., HMI outperforms C-HMCNN w.r.t.
mAP/CmAP on many of the datasets even though C-HMCNN has zero CV.
Classification via hyperbolic logistic regression To validate whether our proposed geometric constraints are able to improve hyperbolic logistic regression (HLR) [19], we implement HMI+HLR, a combination of our proposed constraints with HLR as described in Section 3.3.
Table 1 show that
HMI+HLR outperforms HLR with statistical confidence, showcasing that HMI is able to improve the predictive performance and consistency of HLR.
However, there is no significant difference (with p -value larger than 0 . 05 in Table 2) between the two variants of our method (HMI and HMI+HLR).
4.3 Ablation studies & parameter sensitivity.
For further ablation
[Table]
we introduce one additional metric.
Exclusion Constraint Violation (ECV) measures, analogous to HCV, the fraction of the exclusion constraints violated by the predictions i.e., ECV =
[Equation]
kj (cid:1) .
We introduce this because HCV can be made zero trivially by associating all labels with the same score.
Hence, in the ablation study, we will show how the exclusion constraints (the results of ECV) complement HCV and influence the overall performance.
[Table]
Impact of penalty weight Ta-
ble 3 shows the results of HMI on "CellcycleFUN" and "CellcycleGO" dataset.
We observed that with different penalty weights, the obtained results are slightly different.
Even without penalty ( λ = 0 ), the model already achieves acceptable results, in particular, it outperforms MVM, MHM
[Table]
and
BoxE, indicating that our hyperbolic model, to some extent, is capable of capturing label hierarchies without any explicit constraints.
However, as Table 3 shows, a proper λ = 0 .
001 , λ = 0 .
005 and λ = 0 . 01 indeed improves the performance and consistency.
Finally, we observed that increasing λ to 0 .
1 , though further improves consistency (HCV and ECV), does not further improve mAP and CmAP.
We conjecture that this is because a large λ would encourage the model to "overfit" the given constraints while "underfitting" the classification loss.
[Table]
Impact of implication & exclu-
sion To study the roles of implication and exclusion.
We implemented three variants of HMI by removing either implication or exclusion, or removing both of them.
Table 4 depicts the results of these variants.
It is clear that both implication and exclusion constraints improve the base model that has no constraints.
When implication and exclusion are jointly constrained, the performance is significantly improved again.
We also observed that implication and exclusion constraints, to some extent, do complement each other, e.g., by only using implication (resp.
exclusion), the model archives lower ECV (resp.
HCV).
Finally, we observed that even without exclusion, our model still slightly outperforms MBM, showcasing the advantages of hyperbolic space for modeling hierarchies.
Impact of sampling ratio To study whether our method is able to preserve logical constraints from incomplete label constraints we compare the performance of HMI with different ratios for sampling the training constraints.
As Figure 4(a) depicts, with zero sampling ratio, our method already achieves acceptable results.
We conjecture that this is because some constraints can be learned from the data.
However, Figure 4(a) clearly shows that including constraints indeed helps to improve the performance.
Making the sampling ratio larger than 30 - 40% does not lead to a significant performance gain.
We conjecture that this is because certain ratio of training constraints is sufficient for inferring the full set of constraints.
[Figure]
Impact of embedding dimensionality We study how the choice of dimensionality affects performance.
As Figure 4(b) depicts, HMI achieves acceptable results even in a very low dimension ( n ≤ 100) .
When increasing the dimension an order of magnitude ( n = 1000 ), the performance grows only slightly.
Note that all reported baselines achieved acceptable results with dimensions in [500 , 1000 , 1750]
(see hyperparameter settings in the Appendix of Patel et al.
We conjecture that the reason we can achieve good performance with fewer dimensions is that the hyperbolic hyperplane is more suitable for representing hierarchical decision boundaries.
Comparison with MBM with only implication or without any constraint To faithfully study the advantages of hyperbolic hyperplane on modeling label relations than that of the box model (MBM), we also implement two versions of HMI by considering only (30%) implication constraints and without any constraint (sampling ratio = 0 ), respectively.
Our Wilcoxon test in Table 5 shows that HMI with only implication and HMI without any constraint still outperform their corresponding counterparts of MBM on CmAP and HCV (with p-value < 0 .
05 ) while achieving comparable results on mAP (i.e., with better average ranks but without statistical significance, we believe this is because mAP is less sensitive to the constraints than CmAP).
[Table]
Conclusion
In this paper, we focus on a structured multi-label prediction task whose output is supposed to respect the implication and exclusion constraints.
We show that such a problem can be formulated in a hyperbolic Poincaré ball space whose linear decision boundaries (Poincaré hyperplanes) can be interpreted as convex regions.
The implication and exclusion constraints are geometrically interpreted as insideness and disjointedness, respectively.
Experiments on 12 datasets show significant improvements in mean average precision and lower constraint violations, even with an order of magnitude fewer dimensions than baselines.
Acknowledgments
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Bo Xiong.
Bo Xiong is funded by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No: 860801.
Mojtaba
Nayyeri is funded by the German Federal Ministry for Economic Affairs and Climate Action under
Grant Agreement Number 01MK20008F (Service-Meister).
We thank Prof. Martin Keller-Ressel and all anonymous reviewers for providing valuable suggestions that improved the paper.
A Proof of theorems
Proposition 1.
Given a Poincaré hyperplane H c where c ̸ = 0 , there exists an n -ball B c ( o c , r c ) such that H c ⊂ B c ( o c , r c ) , i.e., H c is a subset of B c ( o c , r c ) .
B c is uniquely given by
[Equation]
Proof.
Since c is the center point of the Poincaré hyperplane, the vector −→ c must be a normal vector of the tangent space T c B n of B n at c .
Let q be one of the point that the Poincaré hyperplane and the
Poincaré ball intersect at.
Then, the radius of B c ( o c , r c ) , the radius of D n , and the distance from the centers of D n to the center of B c ( o c , r c ) must satisfy the Pythagorean theorem [27], i.e., the three
Euclidean distances d ( 0 , q ) , d ( q, o c ) and d ( o c , 0 ) must satisfy
[Equation]
Since we have d ( c , o c ) =
d ( q , o c ) =
r c , by solving this quadratic equation, we have r c = 1 −∥ c ∥ 2 2 ∥ c ∥ .
Since o c = c (1 + r c d ( 0 , c ) ) , we have o c =
c ( 1+ ∥ c ∥ 2 ) 2 ∥ c ∥ .
Thus, B c =
[Equation]
Proposition 2 (HEX-property) .
The classification function f has the HEX property with respect to
G if and only if for any constraint in G , the corresponding loss term is 0 .
Proof.
Note that the loss term of the constraint being 0 implies that the corresponding constraint is respected.
Our loss terms clearly connect the HEX property.
That is, for any point p ∈ D n and a pair of enclosing n -balls ( B w , B u ) , L membership ( p, B w ) ≥ L membership ( p, B u ) for all ( B w , B u ) where L inside ( B w , B u ) = 0 and ¬L membership ( p, B w ) ∨ ¬L membership ( p, B u ) for all ( B w , B u ) where
L disjoint ( B u , B w ) = 0 .
According to the definition of HEX-property, f has the HEX property with respect to G if and only if the corresponding loss term of the corresponding constraint is 0 .
Corollary 1.
Given a HEX graph G of labels and if the loss of the embeddings is 0 , then the learned prediction function is logically consistent with respect to G .
Proof.
Note that the loss terms L inside , L disjoint , L membership , L non-membership in Eq.7 are all non-negative.
Hence, the loss being 0 implies that all losses are zeros (all constraints are satisfied).
According to the definition of consistency, the prediction function is consistent.
B Supplementary experiments and details
Datasets and pre-processing The functional genomic datasets (Expr, Spo, Derisi, Cellcycle) are available at 7 .
The image datasets (Imclef07a, Imclef07d, Diatoms) and text dataset (Enron) are all available at 8 .
All licenses of the datasets can be found in the corresponding links and references.
The number of labels, types of features, the number of instances vary significantly.
The diversity of these datasets makes them suitable for evaluating the multi-label classification task.
The input features are pre-processed in the same way as described in [11, 8, 10].
In particular, all categorical features were transformed using one-hot encoding.
The missing values were replaced by the mean value (for numeric features) or zero-valued vector (for categorical features).
All continuous features were standardized before feeding into the encoder.
The labels of the root nodes are removed from training and evaluation.
[Table] [Table]
Deriving mutual exclusion In real-world applications, exclusion relations could be annotated by human experts by exploiting domain knowledge.
In this paper, we explore various strategies to generate possible exclusion relations: 1) Deriving exclusion from the label taxonomy .
Following the "exclusive whenever possible" assumption [1], we add mutual exclusion edges between two nodes whenever they do not share any descendant nodes (i.e., it does not create a contradiction).
Deriving exclusion from the label co-occurrence .
We add mutual exclusion edges between two labels whenever there is no instance in the training set simultaneously belonging to them.
Clearly, strategy 1 generates all possible exclusion edges entailed by the label taxonomy, while strategy 2 generates exclusion edges that are reflected by the dataset itself.
Strategy 1 might create false positive exclusions (i.e., exclusions that violate the label co-occurrence in the datasets), while strategy 2 might suffer from the noisy labeled data (e.g., an instance might be incorrectly or incompletely labeled).
However,
Table 7 shows that there is no statistical difference between the generated exclusions from these two methods.
Hence, we may conclude that the "exclusive whenever possible" assumption almost holds.
One common problem of these two methods is that there are many redundant edges generated.
To efficiently exploit the constraints, we only generate exclusions between sibling nodes whenever it does not create contradiction [15].