Hyperbolic Embedding Inference for Structured Multi-Label Prediction
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider a structured multi-label prediction problem where the labels are organized under implication and mutual exclusion constraints.
A major concern is toproduce predictions that are logically consistent with these constraints.
To do so, weformulate this problem as an embedding inference problem where the constraintsare imposed onto the embeddings of labels by geometric construction .
Particularly, we consider a hyperbolic Poincaré ball model in which we encode labels as
Poincaré hyperplanes that work as linear decision boundaries.
The hyperplanes areinterpreted as convex regions such that the logical relationships (implication andexclusion) are geometrically encoded using insideness and disjointedness of theseregions, respectively.
We show theoretical groundings of the method for preservinglogical relationships in the embedding space.
Extensive experiments on 12 datasetsshow 1) significant improvements in mean average precision; 2) lower number ofconstraint violations; 3) an order of magnitude fewer dimensions than baselines.
Introduction
Structured multi-label prediction is a task aiming to associate every object with multiple labels that aresemantically constrained in a structured manner (e g., by implication and exclusion constraints).
Thistask is of growing importance in many applications such as image annotation [1, 2], text categorization[3, 4] and functional genomics [5, 6].
One of the central concerns of the task is to produce predictionsthat are logically consistent with the constraints of the labels.
For example, a protein must be labeledto have the function nucleic acid binding if it is already labeled to have the function RNA binding(i e., implication) and must not have the function drug binding (i e., mutual exclusion).
Various works have been proposed to improve the prediction consistency [7, 8, 9, 10, 11].
One lineof work called label embedding aims to represent labels as low-dimensional vectors [12, 13].
A keydisadvantage of the vector-based representations is that they only capture weak forms of correlationor “similarity’ between labels, but do not strongly enforce the logical relationships.
Another lineof work [7, 9, 14, 8] imposes these logical constraints directly to the losses of neural networks.
However, they do not explicitly learn the representations of labels and typically require a completelabel taxonomy, which is not always available in and scalable to real-world settings [11].
Embedding-based inference [15], which imposes logical constraints directly to the label embeddings,is able to inductively infer the underlying label relationships from incomplete labelings [16].
Once allembeddings are adhering to the constraints, each label can be predicted independently without accessing the label relationships, which significantly reduces the computation cost during inference [15].
The key idea, which is inspired by the Venn diagram [16, 17] or set-theoretic semantics [18], is torepresent each label as a convex region [15].
A prominent example is the multi-label box model(MBM)
[11] that models label implications as box containments.
However, MBM learns box-likedecision boundaries, which are typically not compatible with standard classifiers (i.e., hyperplane
[Figure]
p e r s o n w o m e n g i r l p a r e n t m o t h e r p l a n t t r e e f a t
h e r 0.
(c) margin-based models such as logistic regression [19]).
Besides, box models suffer from a theoreticallimitation, i.e., lower-way intersections enforce higher-way interactions [20].
Finally, current methodsignore the importance of constraining mutual exclusion, which is essential as otherwise, a modelcould trivially obtain zero implication violation by assigning the same score to all labels.
In this paper, we consider a structured multi-label prediction problem with implication and mutualexclusion constraints that are jointly described by a hierarchy and exclusion (HEX) graph (see
Figure 1(a) for an example).
The key idea of our method is to transform the logical constraints intosoft geometric constraints in the embedding space.
In particular, we consider a hyperbolic Poincaréball model that has demonstrated advantages in representing hierarchies and assign each label
a
Poincaré hyperplane that has several favorable theoretical properties in classification.
Each Poincaréhyperplane can be interpreted as a convex region such that the implication and mutual exclusion aremodeled by geometric insideness and disjointness between the corresponding regions, respectively.
In this way, a multi-label classifier can be defined by measuring the confidence of an instance havinga label as geometric membership .
Unlike other hyperbolic region-based models such as hyperboliccones [21] and hyperbolic disks [22], Poincaré hyperplane works as a linear decision boundaryand can be seamlessly integrated into existing margin-based classifiers such as hyperbolic logisticregression [19].
Figure 1(b) shows an example of the learned label representations that respect allthe constraints given in Figure 1(a).
We show theoretical groundings of the proposed method onmodeling implication and mutual exclusion .
Extensive experiments on 12 multi-label classificationtasks show the model’s capability to improve the mean average precision significantly while keepingthe number of constraint violations low and requiring an order of magnitude fewer dimensions.
Preliminaries
Poincaré ball model
The Poincaré ball (cid:0) D n , g D (cid:1) is one of the models of hyperbolic geometrythat is very suitable for representing hierarchies due to its exponentially growing volume [23].
The
Poincaré ball is defined as an open n -ball D n = { x ∈ R n : ∥ x ∥ < 1 } equipped with a Riemannianmetric g D x = λ 2 x g E , where λ x =
[Equation]
−∥ x
[Equation]
2 , g E =
I n is the Euclidean metric tensor, λ x is the conformalfactor , and ∥·∥ 2 denotes the L 2 norm in Euclidean space.
The distance between two points x , y ∈ D ncan be defined by d D ( x , y ) =
cosh − 1 (cid:16) 1 +
[Equation]
Structured multi-label prediction Let X ⊆ R n denote an n -dimensional instance space and L ={ l 1 , l 2 , . . .
} , |L| ≥ 2 denote the finite set of possible labels.
Given a set of N training examples D
={ ( x i , L i ) | 1 ≤ i ≤ N, x
i ∈ X , L i ⊂ L} , multi-label prediction aims to learn a labeling functionf : X → 2 L mapping from the instance space to the powerset of the label space, f ( x ) ⊂ L .
Structured multi-label prediction additionally imposes a set of prior-known logical constraints overthe labels, namely, the predictions must be logically consistent with these constraints.
Analogousto Mirzazadeh et al.
[15], we consider two forms of logical constraints between labels: implicationand mutual exclusion.
Specifically, an implication of the form l a ⇒ l b imposes the constraint that whenever an instance is labeled as l a then it must also be labeled as l b , i.e., l a ⇒ l b is a shorthandnotation for ∀ x ∈ X , l a ∈ f ( x ) ⇒ l b ∈ f ( x ) .
Mutual exclusions are constraints of the form¬ l a ∨ ¬ l b , implying that an instance cannot be simultaneously labeled as l a and l b , i.e., ¬ l a ∨ ¬ l bis a shorthand notation for ∀ x ∈ X , l a / ∈ f ( x ) ∨ l b / ∈ f ( x ) .
We can concisely represent a set ofimplication and exclusion constraints with a hierarchy and exclusion (HEX) graph [2].
Definition 1 (HEX graph [2] 1 ) .
A HEX graph G =
( V, E h , E e ) is a graph consisting of a set ofnodes V
= { v 1 , . .
. , v n } , directed (hierarchy) edges E h ⊆
V × V , and undirected (exclusion) edges
E e ⊆ V × V , such that the subgraph G h
= ( V, E h ) is a DAG and the subgraph G e
=
( V, E e ) hasno self loop.
Each node v i ∈ V represents the label l i .
A directed edge ( v i , v
j ) ∈
E h represents theimplication l i ⇒ l j , and an undirected edge ( v i , v
j ) ∈
E e represents the exclusion ¬ l i ∨ ¬ l j .
Note that an arbitrary HEX graph may contain redundant edges.
A hierarchy edge ( v i , v j ) is redundantwhen there is a path in G h from v i to v j which does not contain the edge ( v i , v j ) .
Similarly, anexclusion edge ( v i , v j ) is redundant when there is another exclusion edge connecting their ancestors(or connecting one node’s ancestor to the other node).
We can transform a HEX graph into anequivalent HEX graph by adding or removing redundant edges.
In this paper, we only consider HEXgraphs that have a minimal number of edges, we call such HEX graph a minimal sparse HEX graph(see Fig.
1(a)
for an example).
Given a minimal sparse HEX graph, we define the HEX-property as
Definition 2 (HEX-property) .
A labeling function f has the HEX property with respect to a HEXgraph G if for all x ∈ X , f ( x ) respects all constraints represented by G .
We also call such function f logically consistent w.r.t G .
Given the HEX graph and the HEX-property,structured multi-label prediction is formally defined as a constrained optimization problem.
Definition 3 (Structured multi-label prediction) .
The structured multi-label prediction task withrespect to a training set D = { ( x i , L i ) | 1 ≤ i ≤ N, x
i ∈ X , L i ⊂ L} , minimal HEX graph G =( V, E h , E e ) , and multi-label prediction function f , is the task of learning f such that the function fminimizes (cid:80) ( x i ,L i ) ∈D loss( f ( x i ) , L i ) , with loss a predefined function, while attempting to maintainthe HEX-property with respect to G .
Note that this definition allows for a soft interpretation of the constraints, meaning that the goal isto adhere to all of them, but we do allow for loosening some if necessary.
For example, a mutualexclusion constraint is allowed to loosen when an instance (e.g., image), though rarely happens, issimultaneously labeled as two mutual exclusive labels (e.g., dog and cat ).
Hyperbolic embedding inference
We consider learning a real-valued ranking function h : X × L (cid:55)→ [0 , 1] , where the output isinterpreted as the confidence of an instance x ∈ X having a label l ∈ L .
Afterward, a binarymulti-label classifier f : X → 2 L can be simply obtained by thresholding the ranking function with athreshold t , i.e., f ( x ) =
{ l | h ( x, l ) ≥ t, ∀ l ∈ L} .
The objective of h is to assign higher scores topositive instance-label pairs than that of negative instance-label pairs.
3.1 Geometric construction
Given an n -dimensional Poincaré ball D n , we associate each instance x i ∈ X with a point in the
Poincaré ball and associate each label l i ∈ L with a Poincaré hyperplane, such that its correspondingpositive and negative instances are correctly separated by the hyperplane.
Poincaré hyperplanes Let B n denote the set of n -balls in R n whose boundaries ∂ B n intersect the Poincaré ball D n perpendicularly.
Poincaré hyperplanes are defined by ∂ B n ∩ D n (see Fig. 2(a))
plusall linear subspaces going through the origin.
For the former cases, a Poincaré hyperplane can beuniquely defined by its center point that has a minimal distance to the origin.
Definition 4.
Given a (center) point c ∈ D n where c ̸ = 0 , the Poincaré hyperplane is defined as
[Equation]
1) where c is the center point and ⃗ c ∈ T c D n 2 is the normal vector passing through the origin 0 . [Figure] 𝔹
[Figure]
Intuitively, this corresponds to the union of all geodesics passing through c while orthogonal to thenormal vector ⃗ c ∈ T c D n .
In the case where c is the center of the hyperplane, ⃗ c must simultaneouslypass through c and the origin.
Hence, ⃗ c can be simply taken as c without loss of generality.
Forthe special case where c = 0 , the Poincaré hyperplanes are all linear subspaces (Euclidean planes)passing through the origin.
In this paper, we exclude these special cases by assuming c ̸ = 0 .
Geometric intuition Essentially, the Poincaré hyperplane works as a linear decision boundarythat separates the embedding space into two regions, 3 where the smaller region (i.e., convex hull) isinterpreted as the space of positive samples while the other one is interpreted as the space of negativesamples.
Two reasons motivate us to model labels as Poincaré hyperplanes: 1) Modeling labels ashyperplanes has several desired theoretical advantages in margin-based classifiers.
Our model sharesthe same philosophy as existing learning frameworks such as hyperbolic logistic regression [19] andhyperbolic SVM [24]; 2) More importantly, unlike Euclidean space that is flat, hyperbolic Poincaréball is a curved space in which there are infinitely many non-parallel hyperplanes which do notintersect, implying that linear decision boundaries in hyperbolic space can capture more complicatedset-theoretic interactions, such as implication and mutual exclusion.
Enclosing balls Given a Poincaré hyperplane H c , we call the corresponding n -ball B n c that encloses H c its enclosing n -ball.
Formally, an enclosing n -ball B n ( o , r ) is defined by B n ( o , r )
={ p : ∥ p − o ∥ ≤ r } , where o ∈ R n and r are the center point and the radius, respectively.
Given H c ,we have the following closed-form representation of B n c .
Proposition 1.
Given a Poincaré hyperplane H c where c ̸ = 0 , there exists an n -ball B n c ( o c , r c )such that H c ⊂ B n c ( o c , r c ) , i.e., H c is a subset of B n c ( o c , r c ) .
B n c is uniquely given by
[Equation]
Proof sketch.
The key idea is to solve a quadratic equation given by the fact that the radius of B n c , theradius of D n , and the distance from the center of D n to the center of B n c must satisfy the Pythagoreantheorem [25].
Full proof is in the supplementary material.
3.2 Geometric interpretation
Our main idea is to transform the logical relationships between labels into geometric relationshipsbetween their corresponding enclosing n -balls.
In particular, the implication is modeled by thegeometric insideness while the mutual exclusion is modeled by the geometric disjointness.
Implication The logical implication between two labels is interpreted as geometric relationsbetween n -balls, i.e., n -ball insideness illustrated in Fig. 2(b).
In particular, an n -ball B w ( o w , r w )contains B u ( o u , r u ) if and only if ∥ o u − o w ∥ + r u < r w , and thus we can create an insideness lossdefined by
[Equation]
Clearly, the insideness loss term satisfies the properties of correctness and transitivity
Lemma 1 (Correctness) .
B u is inside of B w if and only if L inside ( B u , B w ) = 0 .
Lemma 2 (Transitivity) .
If L inside ( B u , B w ) =
and L inside ( B w , B v ) = 0 , we have L inside ( B u , B v ) ≤
L inside ( B u , B w ) + L inside ( B w , B v ) ≤ L inside ( B w , B v ) = 0 .
Mutual exclusion Similarly, we interpret mutual exclusion as geometric disconnectedness betweenn -balls illustrated in Fig. 2(c).
B u disconnecting from B w can be measured by subtracting thedistance between their center points from the sum of their radii.
Inversely, the corresponding loss is
[Equation]
Again, the disjointedness loss term satisfies the correctness property
Lemma 3 (Correctness) .
B u disconnects from B w if and only if L disjoint ( B u , B w ) = 0 .
3.3 Classification and learning
Given the embeddings of instances and labels, an instance can be classified by measuring thegeometric membership , i.e., the confidence of a point
p ∈ D n being inside the enclosing ball B .
Membership and non-membership Formally, given an instance embedding p ∈ D n and a labelembedding associated with an enclosing n -ball B c .
The confidence of an instance p being inside theenclosing n -ball B c can be measured by subtracting the distance between the center point of B c andp from the radius of B c .
The corresponding loss is defined as the inverse of the measure, given by
[Equation]
Symmetrically, for negative instance-label relations, the loss of non-membership can be defined as L
[Equation]
Clearly, we have the following properties that follow directly from the definitions.
Lemma 4.
A point p is a member of B c if and only if L membership ( p , B c ) = 0 .
Lemma 5.
A point p is not a member of B c if and only if L non-membership ( p , B c ) = 0 .
We aim to learn an encoder E θ (i.e., a hyperbolic neural network whose designs depend on thedatasets), where θ is the trainable parameter, and a function C which maps labels to the center pointsof the corresponding Poincaré hyperplanes in the Poincaré ball.
Now, we define h ( x, l ) =
σ ( L non-membership ( E θ ( x ) , C ( l )) − L membership ( E θ ( x ) , C ( l ))) , as our ranking function, where σ is the sigmoid function.
The final classification function is then defined byf ( x ) =
{ l | h ( x, l ) ≥ 0 . 5 } .
We call our classifier hyperbolic multi-label embedding inference (HMI).
Given a HEX graph, HMI has the following guarantee.
Proposition 2 (HEX-property) .
The classification function f of HMI has the HEX property withrespect to G if for any constraint in G , the corresponding loss term is 0 .
Learning with soft constraints Let D + =
{ ( x i , l n ) | ( x i , L i ) ∈ D , l n ∈ L i } be the set of positiveinstance-label pairs and D - = { ( x i , l n ) | ( x i , L i ) ∈ D , l n ∈ L , l n / ∈ L i } be the set of negativeinstance-label pairs.
By combining the loss functions of membership, non-membership, insidenessand disjointedness, the learning objective can be formulated as
[Equation]
The first two terms are losses for positive and negative samples while the last two terms areimplication and exclusion constraints, respectively, with λ being the penalty weight of the constraints.
The following corollary shows that our model has a strong inductive bias for preserving consistency.
Corollary 1.
Given a HEX graph G of labels, if the loss terms L inside and L disjoint are 0 , then thelearned prediction function is logically consistent.
Classification via hyperbolic logistic regression A key advantage of our method is that the lossesof constraints are compatible with other (margin-based) hyperbolic classifiers such as hyperboliclogistic regression (HLR) [19] and hyperbolic support vector machine (HSVM) [24].
In our experiment we explore HLR, which formulates the logits as the distances from an instance to a Poincaréhyperplane of a label.
That is, h ( x, l ) =
d ( E θ ( x ) , H C ( l )) .
d ( p , H c ) has the following closed form:
[Equation]
where ⊕ is the Möbius addition [19].
The classifier is defined by f ( x ) = { l | σ ( h ( x, l )) ≥ 0 .
5 , ∀ l ∈
L} where σ is the sigmoid function.
We dub such classifier combined with HMI as HMI+HLR.
Evaluation
4.1 Experiment setup
Datasets We consider 12 datasets that have been used for evaluating multi-label prediction methods [11, 8, 10].
These consist of 8 functional genomic datasets [26], 3 image annotation datasets[27, 28], and 1 text classification dataset [29].
All input features are pre-processed in the same way asdescribed by Patel et al.
For all datasets, the implication constraints (label taxonomy) are given.
Following Mirzazadeh et al.
[15] we add exclusion constraints between sibling nodes whenever thisdoes not create a contradiction (i.e., they share no common descendant nodes).
We also explore otherstrategies for deriving exclusions, but no significant difference was observed (see the supplementfor an analysis).
Similar to MBM [11] and its baselines, we sample 30% of the implications andexclusions constraints for training the model.
We adopt a one-layer hyperbolic linear layer [19] asthe encoder E θ .
Details of the encoder are described in the supplemental material.
Baselines We compare our approach with both classical vector-based and state-of-the-art regionbased embedding methods.
In particular, we consider two vector-based models: 1) The multi-labelvector model (MVM) [30], which encodes both inputs and labels as Euclidean vectors; 2) the multilabel hyperbolic model (MHM) used by Chen et al.
[13], which represents inputs and labels ashyperbolic points; and two box models: 3) the non-probabilistic box model (BoxE) [31] and 4) theprobabilistic multi-label box model (MBM)
[11] that encodes both instances and labels as axis-parallelhyper-rectangles.
Besides, we compare with 5) hyperbolic logistic regression (HLR) [19] since it alsoencodes labels as Poincaré hyperplanes (but does not use geometric constraints).
Furthermore, wecompare with 6) C-HMCNN, a state-of-the-art non-embedding based method that injects hierarchyconstraints directly into the loss function without embedding labels.
A notable difference is that
C-HMCNN needs the full hierarchy constraints as its input.
Finally, we also implement HMI+HLR,a combination of our proposed constraints with HLR for an ablation study.
Hyperparameter settings We train HMI, HLR and HMI+HLR using Riemannian Adam [32] witha batch size of 4 , a learning rate of 1 e − 3 , a dropout rate of 0 . 6 and an early-stopping with patience20 .
We also explore some larger batch sizes but it does not yield better results, which is also observedin Wehrmann et al.[14].
The results of other baselines are as reported by Patel et al.[11] that weclosely follow.
We empirically set the penalty weight of the violation as λ = 5 e − 3 but we willshow its impact in an ablation.
The best dimension per dataset is searched from { 32 , 64 , 128 , 256 } ,which is one order of magnitude lower than that used by Patel et al.
All methods have been run 10 times with random seeds and the average results are reported.
We omitthe standard deviations since they are in a very small range ( [2 × 10 − 4 , 2 . 3 × 10 − 3 ] ).
Evaluation protocols In line with Patel et al.
[11], we consider Mean Average Precision (mAP) , 4which summarizes the information of precisions and recalls with varied thresholds.
We also reporttwo metrics that additionally take the constraints into account: 1) Constrained mAP (CmAP) is avariant of mAP that replaces the score of each label with the maximum scores of its descendant labelsin the hierarchy [11].
2) Hierarchy Constraint Violation (HCV) [11] measures the extent to whichthe label scores violate the implication constraints regardless of true labels for the instances.
HCV
[Table]
CellcycleGO
[Table] [Table]
is computed as HCV = 1 |D|| E h | (cid:80) |D| k =1 (cid:80) ( l i ,l j
) ∈
E h 1 (cid:0) h
ki − h kj > 0 (cid:1) , where h i means the predictionscore of label l i .
Clearly, a lower value of HCV implies higher consistency in the predictions.
4.2 Main results
As Table 1 shows, our method HMI either achieves the best (7-8/12 datasets) or competitive (4-5/12datasets) performance (mAP and CmAP) over all compared methods.
HMI outperforms all methodsw.r.t the average ranking of mAP/CmAP, showcasing the advantages of HMI.
We observed that the
CmAP is close to mAP, indicating that the model is adhering to the label constraints [11].
In terms ofpredictive consistency (HCV), HMI consistently achieves the best or the second-best results.
Notethat C-HMCNN always gets zero HCV because it exploits the complete hierarchy.
HMI achievescompetitive HCV, despite only using 30% of the hierarchy.
[Table]
Statistical significance Following Patel et al.
[11] and Giunchigliaand Lukasiewicz [8], we test thestatistical significance of the performance across all datasets.
First, weperform the Friedman test [33] and
[Figure]
show that there exists a significant difference w.r.t.
all metrics with p-values ≪ 0 .
Next, weconduct the post-hoc Nemenyi test to verify the statistical differences of the average ranking.
Thecritical diagram w.r.t the average ranking of mAP/CmAP is shown in Fig.
3, in which the methodsthat have no significant differences (significance level 0 . 05 ) are connected by a horizontal line.
Asshown in the diagrams, it is clear to conclude that there is a statistically significant difference w.r.tmAPs/
CmAPs of HMI and HMI+HLR against MVM, BoxE, MHM, and HLR but not the two strongbaselines (MBM and C-HMCNN).
We further perform the Wilcoxon test that considers not only thedifferences in rankings but also the numerical differences in the performance.
The Wilcoxon testresults show that there is a statistically significant difference between the mAPs/CmAPs of HMI andthe two strong baselines with p-value ≪ 0 .
In terms of HCV, our statistical significance test in
Figure 3 and Table 2 shows that HMI and HMI+HLR significantly outperform MVM, BoxE, MHM,
HLR, and MBM but not C-HMCNN since it has zero HCV.
However, we observed that the predictiveperformance (mAP, CmAP) is not fully proportional to the HCV, e.g., HMI outperforms C-HMCNNw.r.t.
mAP/CmAP on many of the datasets even though C-HMCNN has zero CV.
Classification via hyperbolic logistic regression To validate whether our proposed geometricconstraints are able to improve hyperbolic logistic regression (HLR) [19], we implement HMI+HLR,a combination of our proposed constraints with HLR as described in Section 3.3.
Table 1 show that
HMI+HLR outperforms HLR with statistical confidence, showcasing that HMI is able to improve thepredictive performance and consistency of HLR.
However, there is no significant difference (withp -value larger than 0 . 05 in Table 2) between the two variants of our method (HMI and HMI+HLR).
4.3 Ablation studies & parameter sensitivity.
For further ablation, we introduce one additional metric.
Exclusion Constraint Violation (ECV)measures, analogous to HCV, the fraction of the exclusion constraints violated by the predictionsi.e.
, ECV =
[Equation]
We introduce this because HCV can be madezero trivially by associating all labels with the same score.
Hence, in the ablation study, we willshow how the exclusion constraints (the results of ECV) complement HCV and influence the overallperformance.
[Table]
Impact of penalty weight Ta-
ble 3 shows the results of HMIon "CellcycleFUN" and "CellcycleGO" dataset.
We observed thatwith different penalty weights, theobtained results are slightly different.
Even without penalty ( λ =0 ), the model already achieves acceptable results, in particular, itoutperforms MVM, MHM, and
BoxE, indicating that our hyperbolic model, to some extent, is capable of capturing label hierarchies without any explicit constraints.
However, as Table 3 shows, a proper λ = 0 .
001 , λ = 0 . 005and
λ
= 0 .
01 indeed improves the performance and consistency.
Finally, we observed that increasingλ to 0 .
1 , though further improves consistency (HCV and ECV), does not further improve mAP and CmAP.
We conjecture that this is because a large λ would encourage the model to "overfit" the givenconstraints while "underfitting" the loss of classification itself.
[Table]
Impact of implication & exclusion To study the roles of implication and exclusion.
We implemented three variants of HMI byremoving either implication or exclusion, or removing both of them.
Table 4 depicts the results of thesevariants.
It is clear that both implication and exclusion constraintsimprove the base model that hasno constraints.
When implication and exclusion are jointly constrained, the performance is significantly improved again.
We also observed that implication and exclusion constraints, to some extent,do complement each other, e.g., by only using implication (resp.
exclusion), the model archiveslower ECV (resp.
HCV).
Finally, we observed that even without exclusion, our model still slightlyoutperforms MBM, showcasing the advantages of hyperbolic space for modeling hierarchies.
[Table]
[Figure]
Impact of sampling ratio
To study whether our methodis able to preserve logical constraints from incomplete label constraints we comparethe performance of HMI withdifferent ratios for samplingthe training constraints.
As
Figure 4(a) depicts, with zerosampling ratio, our method already achieves acceptable results.
We conjecture that thisis because some constraintscan be learned from the data.
However, Figure 4(a) clearly shows that including constraints indeed helps to improve the performance.
Making the sampling ratio larger than 30 - 40% does notlead to a significant performance gain.
We conjecture that this is because certain ratio of trainingconstraints is sufficient for inferring the full set of constraints.
Impact of embedding dimensionality We study how the choice of dimensionality affects performance.
As Figure 4(b) depicts, HMI achieves acceptable results even in a very low dimension( n ≤ 100) .
When increasing the dimension an order of magnitude ( n = 1000 ), the performancegrows only slightly.
Note that all reported baselines achieved acceptable results with dimensions in[500 , 1000 , 1750]
(see hyperparameter settings in the Appendix of Patel et al.
We conjecturethat the reason we can achieve good performance with fewer dimensions is that the hyperbolichyperplane is more suitable for representing hierarchical decision boundaries.
Conclusion
In this paper, we focus on a structured multi-label prediction task whose output is supposed torespect the implication and exclusion constraints.
We show that such a problem can be formulatedin a hyperbolic Poincaré ball space whose linear decision boundaries (Poincaré hyperplanes)
canbe interpreted as convex regions.
The implication and exclusion constraints are geometricallyinterpreted as insideness and disjointedness, respectively.
Experiments on 12 datasets show significantimprovements in mean average precision and lower constraint violations, even with an order ofmagnitude fewer dimensions than baselines.
One potential limitation of our work is that we currently only consider implication and exclusion.
While very powerful when combined, one could come up with other logical constraints such aslogical equivalence l a ∧ l b ⇔ l c .
We do believe that incorporating such prior knowledge (constraints)into embeddings could further improve the performance and plan them for our future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Some proofs have
been included in the supplementary material only.
Some Lemmas which follow directly
from a definition do not have an explicitly stated proof.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All code, dataand instructions to reproduce the results are attached as supplementary material.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] All training details, hyperparameters are described in the supplementarymaterial.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We run all experiments 10 times and reported the averageresults.
The standard deviations are very small (in range [2 × 10 − 4 , 2 . 3 × 10 − 3 ] ) andhence we omitted the concrete numbers.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] The computing resources are
described in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] We mention the links of the datasets,
where the respective licences can be found.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots
if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Proof of theorems
Proposition 1.
Given a Poincaré hyperplane H c where c ̸ = 0 , there exists an n -ball B c ( o c , r c )such that H c ⊂ B c ( o c , r c ) , i.e., H c is a subset of B c ( o c , r c ) .
B c is uniquely given by
[Equation]
Proof.
Since c is the center point of the Poincaré hyperplane, the vector −→ c must be a normal vectorof the tangent space T c B n of B n at c .
Let q be one of the point that the Poincaré hyperplane and the
Poincaré ball intersect at.
Then, the radius of B c ( o c , r c ) , the radius of D n , and the distance from thecenters of D n to the center of B c ( o c , r c ) must satisfy the Pythagorean theorem [25], i.e., the three
Euclidean distances d ( 0 , q ) , d ( q, o c ) and d ( o c , 0 ) must satisfy
[Equation]
Since we have d ( c , o c ) =
d ( q , o c ) =
r c , by solving this quadratic equation, we haver c = 1 −∥ c ∥ 2 2 ∥ c ∥ .
Since o c = c (1 + r c d ( 0 , c ) ) , we have o c =
c ( 1+ ∥ c ∥ 2 ) 2 ∥ c ∥ .
Thus, B c =
B (cid:18) o
c =
{+c ( 1+ ∥ c ∥ 2 ) 
+}
∥ c ∥ , r c = 1 −∥ c ∥ 2 2 ∥ c ∥ (cid:19) .
Lemma 1-2, Lemma 3, Lemma 4-5 immediately follow the definitions of geometric insideness,disjointedness, and membership, respectively.
Proposition 2 (HEX-property) .
The classification function f has the HEX property with respect to
G if and only if for any constraint in G , the corresponding loss term is 0 .
Proof.
Note that the loss term of the constraint being 0 implies that the corresponding constraintis respected.
Our loss terms clearly connect the HEX property.
That is, for any point p ∈ D nand a pair of enclosing n -balls ( B w , B u ) , L membership ( p, B w ) ≥ L membership ( p, B u ) for all ( B w , B u )where L inside ( B w , B u ) = 0 and ¬L membership ( p, B w ) ∨
¬L membership ( p, B u ) for all ( B w , B u ) where
L disjoint ( B u , B w ) = 0 .
According to the definition of HEX-property, f has the HEX property withrespect to G if and only if the corresponding loss term of the corresponding constraint is 0 .
Corollary 1.
Given a HEX graph G of labels and if the loss of the embeddings is 0 , then the learnedprediction function is logically consistent with respect to G .
Proof.
Note that the loss terms L inside , L disjoint , L membership , L non-membership in Eq.7 are all non-negative.
Hence, the loss being 0 implies that all losses are zeros (all constraints are satisfied).
According tothe definition of consistency, the prediction function is consistent.
B Theoretical inexpressivity of box models
One of the main problem of box models is that a set of lower-way interactions automatically enforcesa higher-way interaction.
To illustrate the problem , we first consider a counter example of HEXgraph shown in Fig. 5(a), where there are three labels A, B and C that have pair-wise
interactionsbut three of them together has no three-way interaction, i.e., A ∧ B ∧ C = ∅ .
With box embeddings,by satisfying pair-wise interactions we will automatically have three-way interaction Fig. 5(b).
Thefollowing proposition generalizes the theoretical inexpressivity into arbitrary dimensions.
Proposition 3 (see Sec 7.3 of [20]) .
Let (Ω , E , P ) be a probability box space, and let A, B, C : Ω → { 0 , 1 } be binary random variables such that P ( A ∧ B ) , P ( A ∧ C ) , P ( B ∧ C ) > 0 and
P ( A ∧ B ∧ C ) = 0 .
Then there exists an ϵ > 0 and some event E ∈ E such that, for any box model, we have
[Equation]
Fig. 5(c) shows that such counter example can be easily encoded in a 2 -dimensional Euclidean ballspace.
For Poincaré hyperplane in the Poincaré ball model, we only need to add one additionaldimension such that all labels are embeded in a half of the Poincaré ball.
[Figure]
Hence, such counter example can be encoded in a (half of) 3 -dimensional Poincaré ball.
Note that inhyperbolic space, the intersection of two hyperplanes has infinite and exponentially growing volume,which is a key difference compared with the normal n -balls (whose intersection has finite volume).
C Supplementary experiments and detailsDatasets and pre-processing
The functional genomic datasets (Expr, Spo, Derisi, Cellcycle) areavailable at 5 .
The image datasets (Imclef07a, Imclef07d, Diatoms) and text dataset (Enron) are allavailable at 6 .
All licenses of the datasets can be found in the corresponding links and references.
The number of labels, types of features, the number of instances vary significantly.
The diversityof these datasets makes them suitable for evaluating the multi-label classification task.
The inputfeatures are pre-processed in the same way as described in [11, 8, 10].
In particular, all categoricalfeatures were transformed using one-hot encoding.
The missing values were replaced by the meanvalue (for numeric features) or zero-valued vector (for categorical features).
All continuous featureswere standardized before feeding into the encoder.
The labels of the root nodes are removed fromtraining and evaluation.
[Table]
Hyperbolic encoder We adopt a simple hyperbolic linear layer as the instance encoder forall datasets.
A single-layer hyperbolic fully-forward linear layer is defined by f θ = { W , b } ( x ) =tanh ⊗ ( W ⊗ x ⊕ b ) , with ⊗ being Möbius matrix-vector multiplication defined by M ⊗ x =
[Table]
[Table]
tanh (cid:16) ∥
M x ∥ ∥ x ∥ tanh − 1 ( ∥ x ∥ ) (cid:17)
M x ∥
M x ∥ , where W ∈ R n × d is a trainable matrix and x is a point x ∈
D n , M x ̸ = 0 .
⊕ denotes Möbius addition given by x ⊕ y =
( 1+2 ⟨ x , y ⟩ + ∥ y ∥ 2 ) x
+ ( 1 −∥ x ∥ 2 ) y 1+2 ⟨ x , y ⟩ + ∥ x ∥ 2 ∥
y ∥. tanh ⊗denotes an Möbius version of pointwise non-linearity given by tanh ⊗ ( x ) = exp 0 (tanh (log 0 ( x ))) ,with exp 0 and log 0 being the exponential and logarithmic maps, see [19] for more details.
Implementation details We implement HMI, HLR and HMC-HLR using PyTorch [34] and trainthe models on NVIDIA A100 with 40GB memory.
The results of other baselines are taken from [11].
We use Riemannian Adam [32] with a batch size of 4 and learning rate of 1 e − 3 for training the modelparameters for all datasets.
For Riemannian Adam optimizer, we use the geoopt implementation [35].
We also explore some larger batch sizes but it does not yield better results, which is also observedin [14].
We set the dropout rate to 0 .
6 suggested by [14] to avoid the case that the model overfitsthe small training sets.
We employ an early-stopping strategy with patience 20 to save training time.
For the constraints weight λ , we empirically set it as λ = 5 e − 3 , and in an ablation study we showhow λ effects the performance.
Note that different from [11] that obtains the results by conductingper-dataset hyperparameter optimization (dimensions, layers, learning rate, etc.), we only searchthe best number of dimensions (from [32 , 64 , 128 , 256] ) for each dataset on the validation set, anddimension is an order of magnitude fewer than that used in [11].
The best searched dimensionsis shown in Table 6.
Hence, our treatment significantly reduces the overall running time and thehyperparameter settings are more robust.
All methods have been run 10 times and the average resultsare reported.
Following previous works, we omit the standard deviations since they are in a verysmall range ( [2 × 10 − 4 , 2 . 3 × 10 − 3 ] ).
Deriving mutual exclusion In real-world applications, exclusion relations could be annotatedby human experts by exploiting domain knowledge.
In this paper, we explore various strategies togenerate possible exclusion relations: 1) Deriving exclusion from the label taxonomy .
Following the"exclusive whenever possible" assumption [1], we add mutual exclusion edges between two nodeswhenever they do not share any descendant nodes (i.e., it does not create a contradiction).
2) Derivingexclusion from the label co-occurrence .
We add mutual exclusion edges between two labels wheneverthere is no instance in the training set simultaneously belonging to them.
Clearly, strategy 1 generatesall possible exclusion edges entailed by the label taxonomy, while strategy 2 generates exclusionedges that are reflected by the dataset itself.
Strategy 1 might create false positive exclusions (i.e.,exclusions that violate the label co-occurrence in the datasets), while strategy 2 might suffer fromthe noisy labeled data (e.g., an instance might be incorrectly or incompletely labeled).
However,
Table 7 shows that there is no statistical difference between the generated exclusions from these twomethods.
Hence, we may conclude that the "exclusive whenever possible" assumption almost holds.
One common problem of these two methods is that there are many redundant edges generated.
Toefficiently exploit the constraints, we only generate exclusions between sibling nodes whenever itdoes not create contradiction [15].
[Table]
Comparison on AU(PRC)
We also compare the performance of all methods on the area under theprecision-recall curve (AU(PRC)), which is another metric used in [14, 8] for evaluating multi-labelclassifiers.
Baselines results are taken from [11].
Table 8 shows that HMI either achieves the best orcompetitive results against all baselines.
Comparison with MBM with only implication or without any constraint To faithfully studythe advantages of hyperbolic hyperplane on modeling label relations than that of the box model(MBM), we also implement two versions of HMI by considering only (30%) implication constraintsand without any constraint (sampling ratio = 0 ), respectively.
Table 9 summarizes the comparison.
Our Wilcoxon test in Table 10 shows that HMI with only implication and HMI without any constraintstill outperform their corresponding counterparts of MBM on CmAP and HCV (with p-value < 0 .
05 )while achieving comparable results on mAP (i.e., with better average ranks but without statisticalsignificance, we believe this is because mAP is less sensitive to the constraints than CmAP).
Correlation between hierarchy level vs embeddings
As suggested by [36], for hyperbolic space,the magnitude of embeddings correlate to the level of hierarchy in the taxonomy.
Analogously, weconjecture that the magnitude of learned label embeddings (e.g., hyperplane) of HMI also correlateto the level of hierarchy in the taxonomy.
To confirm our assumption, we compute the Spearmanrank correlation between the number of descendants of a node in the true taxonomy and the negativeembedding magnitude of the learned label embeddings (i.e., the center points of the hyperplanes)of HMI.
Table 11 compares the correlation scores with the values reported in [11].
The positivecorrelation scores shown in the table confirm our conjecture that the HMI is able to capture thehierarchy of the labels in the taxonomy.
[Table]
[Table]
[Table]