Imposing Hard Logical Constraints on Multi-label Classification Neural Networks
Abstract
Machine Learning, and in particular deep learning, is becom- ing increasingly ubiquitous, and it is likely to be applied in almost every aspect of our lives in the next few years.
However, the careless application of such methods in the real world can have, and has already had, disastrous consequences.
In order to avoid such undesirable and potentially dangerous scenarios, a standard approach is to formally specify the de- sired behavior of the system and then ensure its compliancy to the specified properties.
In this paper, we thus propose to enhance deep learning models by incorporating background knowledge as hard logical constraints.
The constraints rule out the models’ undesired behaviors and can be exploited to gain better performance.
In order to achieve the above, we propose CCN( h ), a novel model for multi-label classification problems with hard constraints expressed as normal logic rules.
Given any multi-label classification neural network h , CCN( h ) is able to exploit the information expressed by the constraints to: (i) produce predictions that are guaranteed to satisfy the constraints, and (ii) improve the performances.
We conduct an extensive experimental analysis showing the superior performance of CCN( h ) when compared to state-of-the- art models in the setting of multi-label classification problems with hard logical constraints.
Introduction
Machine Learning, and in particular Deep Learning (DL), is becoming increasingly ubiquitous, and it is likely to be applied in almost every aspect of our lives in the next few years.
This is happening thanks to the successes of
DL, which allowed to solve problems that were previously thought to be unbreakable (see e g., (Senior et al. 2020;
Stokes et al. 2020)).
Such results, though ground-breaking, overshadow the dangers that come with the careless application of DL models in the real world.
Indeed, while these models report astonishingly high-performance in terms of accuracy (or alternatively chosen metric), they do not give any guarantee that the model will not have any unintended behaviour when used in practice.
In this paper we thus propose to enhance DL models by incorporating background knowledge as hard logic con- straints.
The constraints rule out the models’ undesired behaviors and can be exploited to gain better performance.
In order to achieve the above, we propose coherent-byconstruction network 1 CCN( h ), a novel model for Multilabel Classification (MC) problems with hard constraints over the output space expressed as normal logic rules (Lloyd 1987), that is, expressions of the form:
[Equation]
imposing that whenever the classes A 1 , . .
A k are predicted, while A k +1 , . . .
, A n are not, then also the class A should be predicted.
Such constraints generalize hierarchy constraints (corresponding to the case n = k = 1 ) used in the field of hierarchical MC problems. Indeed, consider, for example, the task proposed by (Dimitrovski et al. 2008), where a radiological image has to be annotated with an IRMA code specifying, among others, the biological system examined. In this setting, we will be able to specify not only that the prediction { stomach } alone is not possible thanks to the hierarchical constraint
[Equation]
but we will be also able to capture constraints like “if a medical image contains the abdomen but neither the middle nor the upper abdomen, then it contains the lower abdomen”, which can we written as:
[Equation] CCN( h ) is the first model able to deal with MC problems with such expressive hard constraints.
CCN( h ) can be built on top of any multi-label classification neural network h (to be trained) and is based upon two basic ingredients:
the Constraint Module (CM): a constraint layer built on top of h , which extends the predictions made by h in order to ensure that the predictions satisfy the constraints, and 2.
the Constraint Loss (CLoss) a loss function, teaching CCN( h ) when to exploit the constraints, that
is, in the presence of (1), when to exploit the prediction on { A 1 , . . .
, A n } to make predictions on A .
We further show that, if we restrict ourselves to constraints with stratified negation (Apt, Blair, and Walker 1988), given a set H of initial predictions made by the underlying model h , CCN( h ) is able to compute the unique minimal set of classes M such that
M extends H , that is,
such that H ⊆ M , and 2.
M is coherent with (satisfies) the constraints, that is, such that, given (1), A ∈ M whenever { A 1 , . . .
A k } ⊆ M and { A k +1 , . . .
, A n } ∩ M = ∅ .
Indeed, for a non-stratified set of constraints expressed as normal rules, there can be no or more than one minimal set of classes having the above two properties, and determining the non-existence or computing one of them can take exponential time.
Coherent-by-Construction Networks
Preliminaries and Notation Let Π be a set of hard constraints expressed as normal logic rules, h be a generic neural network for a MC problem, and H the set of classes predicted by h .
Given a constraint r ∈ Π , we call head ( r ) =
A the head of r , and body ( r ) = body + ( r ) ∪ body − ( r ) the body of r , where body + ( r ) =
{ A 1 , . . .
, A k } and body − ( r ) =
{¬ A k +1 , . . .
¬ A n } .
Definition 2.1 A set of constraints Π is stratified if there is a partition Π 1 , Π 2 , . . .
, Π s of Π , with Π 1 possibly empty, such that, for every i ∈ { 1 , . . .
, s } ,
for every class A ∈ ∪ r ∈ Π i body + ( r ) , all the constraints with head A in Π belong to ∪ ij =1 Π j ; 2.
for every ¬ A ∈ ∪ r ∈ Π i body − ( r ) , all the constraints with head A in Π belong to ∪ i − 1 j =1 Π j .
Π 1 , Π 2 , . .
, Π s is a stratification of Π , and each Π i is a stratum
Given a rule like the one in Equation (1) and a generic model m we need to compute the values associated to ¬ A k +1 , . . .
¬
A n given the output of m .
We call such values m A k +1 , . . .
, m A n , and they must be such that for each class
A i ∈ { A k +1 , . . .
, A n } and threshold θ ,
m A i = 1 when m A i = 0 , and m A i
= 0 when m A i = 1 , 2.
m
A i is strictly decreasing and continuous (small changes to the value of m A i should correspond to small changes in the value of m A i ), and 3.
m
A i
= θ when m A
i
= θ .
For any threshold θ there are infinitely many functions v satisfying such requirements.
A simple solution is to require v to be piecewise linear with two segments joining when v = v = θ , in which case, if θ = 0 .
5 , we obtain v = 1 − v , that is the standard negation in fuzzy logics (Metcalfe 2005).
For simplicity, from here on, we assume to have the standard negation, that is, to fix the threshold θ to 0 .
5 and v = 1 − v , for each v ∈ [0 , 1] .
All the definitions and results generalize to the case in which we have an arbitrary strict negation with θ = θ .
Constraint Module Given h and a set of stratified constraints Π , our goal is to get CCN( h ) to predict a final set of classes M such that:
M extends the set of classes H predicted by h (i.e., H ⊆ M ) and be coherent with Π ; 2.
M is such that any class in M \ H is in the head of a constraint r in Π , with the chain of rules used to satisfy body ( r ) grounded in H ; 3.
M includes only those classes that are either in H or are forced to be in M through the explicit use of chains of rules grounded in H ; 4.
M is unique, that is, there will be no other set of classes M ′ satisfying the above requirements.
The first property is the obvious one: the constraints in Π must be satisfied, and CCN( h ) can only derive more classes in the head of the constraints.
Whereas the second property is formalized by the concept of supportedness as defined in the work by (Apt, Blair, and Walker 1988).
Definition 2.2 A set of classes M is supported relative to
H and Π , if for any class A ∈ M , A ∈ H , or there exists a constraint r ∈ Π such that head ( r )
=
A , body + ( r ) ⊆
M , and for each ¬ B ∈ body − ( r ) , we have B ̸∈ M .
The third property is a minimality condition.
Definition 2.3 M is minimal relative to H and Π , if there exists no set of classes M ′ with H ⊆ M ′ ⊂
M that is coherent with Π .
The four properties together ensure that the final predictions made by CCN( h ) are coherent with Π and that can be uniquely explained on the grounds of the initial predictions made by h and the constraints in Π .
Assuming Π is stratified, in order to compute the output of CCN( h ), we need to first compute the stratification of Π in the following way:
we compute the acyclic component graph (Cormen et al. 2009) of the dependency graph G Π of Π (as defined in (Apt, Blair, and Walker 1988)), that is, the DAG obtained by shrinking each strongly connected component in G Π into a single vertex (notice that since Π is stratified, negative edges are not involved in any cycle in G Π ), 2.
we assign to the classes in each node of the DAG the number 1 plus the maximum number of negative edges connecting a root to the node, and 3.
we define: (a) A i as the set of classes having the number i assigned at the previous step, and (b) Π i as the set of constraints in Π whose head is in A i .
Define Π ∗ i to be the set of constraints 1.
initially equal to Π i , and then 2. obtained by recursively adding the constraints obtained from a constraint r already in Π ∗ i by substituting a class A ∈ body + ( r ) ∩
A i with body ( r ′ ) for any rule r ′ with head ( r ′ )
=
A (hence, r ′ ∈ Π i ), and finally
eliminating the constraints r such that head ( r ) ∈ body + ( r ) , or for which there exists another constraint r ′ ∈ Π i with head ( r ) = head ( r ′ ) and body ( r ′ )
⊂ body ( r ) .
Π ∗ i is guaranteed to be finite, since the set of classes A is finite, and we do not allow for repetitions in the body of constraints.
The constraints being eliminated in the third step are redundant.
Then,foreach class A ∈
A i , we define the output CM A of the constraint module CM via
[Equation]
where 1. r 1 , . .
, r p are all the constraints in Π ∗ i with head A , and 2.
assuming r ∈ Π ∗ i has the form (1),
[Equation]
, with v A 1 = h A 1 if A 1 ∈ A i , and v A 1 = CM A 1 if A 1 ∈ ∪ i − 1 j =
A j .
Analogously for v A 2 , . . .
, v A k .
The above definition is well-founded:
Π ∗ 1 does not contain negated classes, and thus the definition of CM A
when
A ∈
A 1 relies only on the outputs of the bottom module h , and 2.
the definition of CM A when A ∈ A i ( i > 1 ) uses only outputs of h or of already defined outputs of CM.
Given the above the definition, we can now state that
CCN( h ) has the desired properties.
Theorem 2.4 Assume Π is stratified, and let M be the set of classes predicted by CCN ( h ).
Then, M 1. extends H and is coherent with Π , 2. is supported relative to H and Π , 3. is minimal relative to H and Π , and 4.
is the unique set satisfying the previous properties.
Constraint Loss For every data point, the value of the loss function CLoss used to train CCN( h ) is defined as:
CLoss
[Equation]
A ∈A
CLoss A ,
CLoss A being the value of the loss for class A , defined as:
CLoss
[Equation]
, where:
• y A is the ground
truth for class
A
, • CM + A is the value to optimize when y A
= 1 , and • CM − A is the value to optimize
when y A = 0 .
CM + A and CM − A differ from the output value CM A of CCN( h ) for class A , that is, from CCN( h ) A .
Consider a class A in the i th stratum (i.e., A ∈ A i ).
To each constraint r with head A in Π ∗ i we associate two values 1.
h + ,rA to be used with y A = 1 , and
[Figure]
Stackvector
Concatenatevector
[Figure]
Assume y A
= 1 .
Consider a constraint r in Π ∗ i with head
A of the form (1).
Then, we want to teach CCN( h ) to possibly exploit the constraint r for predicting A if y A
= . . .
=
y A k
= 1 and y A k
+1 = . . .
=
y A n = 0 .
We thus define, h + ,rA = min( v A 1 y A 1 , . .
. , v A k y A k , v A k
+1 y
A k
+1 , . .
. , v A n y A n ) , where v A l is h
A l
if A l ∈ A i (and thus 1 ≤ l ≤
k ), 2. CM + A l if A l ∈ ∪ i − 1 j =
A j and 1 ≤ l ≤ k , 3.
CM − A l if A l ∈ ∪ i − 1 j =
A j and k + 1 ≤ l ≤ n .
The value CM + A associated to class A when y A = 1 is
CM + A = max( h A , h + ,r 1 A , . . .
, h + ,r p A ) , where r 1 , . . .
, r p are all the constraints in Π ∗ i with head A .
Assume y
A
= 0 .
Consider a constraint r in Π ∗ i with head
A of the form (1).
Then, for some class A l ∈ body + ( r ) y
A l = 0 , or for some class
A l ∈ body − ( r ) y
A l = 1 ,
and we want to teach CCN( h ) to not fire the constraint r .
We thus define
[Equation]
) , where v A l now is
h
A l if A l ∈ A i (and thus 1 ≤ l ≤ k ), 2.
CM − A l if A l ∈ ∪ i − 1 j =
A j and 1 ≤ l ≤ k , 3. CM + A l if A l ∈ ∪ i − 1 j =
A j and k + 1 ≤ l ≤ n .
The value CM − A associated with the class A when y A = 0 is
CM − A = max( h A , h − ,r 1 A , . .
, h − ,r p A ) , where r 1 , . . .
, r p are all the constraints in Π ∗ i with head A .
GPU Implementation
Constraint Module The basic idea, starting from the vector CM 0 (which contains the l values resulting from the bottom module h ) is to iteratively compute the vector of values
CM i corresponding to the outputs of CM if given the set of constraints ∪ ij =1 Π ∗ j .
The final output of CM will correspond to CM s .
Figure 1 shows a visual representation of the process and of CCN( h ).
For each i ∈ { 1 , . . .
, s } , p i is the number of constraints in Π ∗ i ( p i = | Π ∗ i | ), r ij denotes the j th constraint in Π ∗ i , and
• B + i is the p i × l matrix whose j, k element is 1 if A k ∈ body + ( r ij ) , and 0 otherwise; • B − i is the p i × l matrix whose j, k element is 1 if ¬ A k ∈ body − ( r ij ) , and 0 otherwise; • C i − 1 is the p i × l matrix obtained by stacking p i times CM
i − 1 .
Stacking p times a vector v of size q returns the p × q matrix 1 Tp × v whose j, k element is v [ k ] .
Then, the j th value v i [ j ] of the vector v i associated with the body of the constraint r ij is
[Equation]
where
⊙ represents the Hadamard product, 2.
J p,q is the p × q matrix of ones, and 3.
given an arbitrary p × q matrix Q , min( Q, dim = 1) (resp., max( Q, dim = 1) ) returns a vector of length p whose i th element is equal to min( Q i 1 , . . .
, Q iq ) (resp., max( Q i 1 , . . .
, Q iq ) ).
Then, the output of the i th layer associated with the i th stratum is given by:
[Equation]
= max( IH i ⊙ V
i , dim = 1) , where
• H
i is the l × p i matrix whose j, k element is 1 if A j = head ( r ik ) , and 0 otherwise, • IH i is the l × ( l + p i ) matrix obtained by stacking the l × l identity matrix and H i , • V
i is the l × ( l + p i ) matrix obtained by stacking l times the concatenation of CM i − 1 and v i .
Constraint Loss
The computation of constraint loss on
GPU follows the same reasoning of the constraint module
[Table]
Experimental Analysis
In order to prove the superiority of our model
[Table]
we adopted the same methodology presented in (Feng, An, and He 2019) and consider the three well-established MC models and the state-of-the-art MC model tested in (Feng, An, and He 2019), which can be characterized by the order of classes correlations they exploit.
Thus, CCN( h ) is compared with 1.
BR (Boutell et al.
2004), a first order model which considers each class separately, ignoring class correlations, and
[Table]
ECC (Read et al. 2009), RAKEL (Tsoumakas et al. 2009) and CAMEL (Feng, An, and He
), which exploit correlations among two or more classes.
BR, ECC, and RAKEL are the well-established MC models, and CAMEL is the current state-of-the-art MC model (Feng,
An, and He 2019).
Since these models are not guaranteed to output predictions that are coherent with the constraints, we applied CM as additional post-processing steps.
We tested the models on 16 publicly available datasets.
About the metrics, the analysis of 64 papers on MC problems conducted by (Spola ˆ or et al. 2013) and reported by (Pereira et al. 2018), shows that already in 2013 as many as 19 different metrics have been used to evaluate MC models, and still today different papers use different subsets of such metrics.
However, as suggested by (Pereira et al. 2018), not all subsets can be used, as the experimental results may appear to favor a specific behavior depending on the subset of measures chosen, thus possibly leading to misleading conclusions.
To avoid such undesired results, the authors conducted a correlation analysis of the metrics that led to the individuation of clusters of correlated measures and thus to the proposal of various subsets of metrics.
Following the above criteria, we used the six metrics listed in Table 1.
Due to lack of space, we cannot report the results of each model, however, in Table 1, the average rankings of each model can be found, 2 and we can see that CCN( h ) outranks all the other models on all metrics.
We also verified the statistical significance of the results—as advised by (Demsar 2006)—by performing the Friedman test for each metric.
As reported in the last column of Table 1, the Friedman test returned pvalue < 0 .
01 for each metric.
Conclusions
In this paper, we proposed to enhance DL models by incorporating background knowledge as hard logic constraints.
To this end, we developed CCN( h ), a novel model whose predictions are always coherent with the given constraints and such that it is able to exploit the background knowledge expressed by the constraints to gain better performance, as demonstrated by the fact that it outperforms the state-of-theart MC models on 16 datasets.
The paper is an extended abstract of (Giunchiglia and Lukasiewicz 2021).