The Curious Case of Stacking Boosted Relational Dependency Networks
Abstract
Reducing bias while learning and inference is an important requirement to achieve generalizable and better performing models.
The method of stacking took the ﬁrst step towards creating such models by reducing inference bias but the question of combining stacking with a model that reduces learning bias is still largely unanswered.
In statistical relational learning, ensemble models of relational trees such as boosted relational dependency networks (RDN-Boost) are shown to reduce the learning bias.
We combine RDN-Boost and stacking methods with the aim of reducing both learning and inference bias subsequently resulting in better overall performance.
However, our evaluation on three relational data sets shows no signiﬁcant performance improvement over the baseline models.
Introduction
Statistical relational learning (SRL) [10, 17] combines the two formalisms of probability (capturing uncertainty) and ﬁrst order logic (capturing symmetries).
This has resulted in the ability to represent complex relations between objects or entities.
One of the key success stories inside SRL models have been ensembles of relational regression trees that overcome the assumption of a propositional representation of the data.
This model called boosted relational dependency networks (RDN-Boost) [15], has been shown to reduce the learning bias since it combines multiple prediction models while learning thereby resulting in far stronger discriminative relational models.
A common assumption is that simple reduction in the learning bias is not generally sufﬁcient to learn generalizable models especially in relational domains.
A recent work presented theoretical frameworks for relational models that can reduce the bias in both learning and inference time [6].
Kou et al.
[11] proposed the method of relational stacking for collective inference which is a meta learning algorithm consisting of a base learner, that can be any relational machine learning model.
The base learner is augmented by expanding an instance’s features with predictions on other related instances.
Stacking has shown to improve collective inference by reducing the inference bias [7] and thus we hypothesized that a combination model that integrates the stacking model (reduces inference bias) with RDN-Boost (reduces learning bias) can result in better performing models.
We propose the combination of the stacking method and RDN-Boost using three relational templates which are dependent on the notion of distance between entities in the given relational data set.
This is based on the observation that the entire data set can be viewed as a graph G =
( V, E ) where V is the set of nodes denoting the entities and E is the set of edges denoting the relations between the entities.
This transformation allows for efﬁcient computation of the distances.
We test our model with three relational data sets and show that, against expectations, using any of these relational templates do not improve the performance of the combination model when compared to the underlying RDN-Boost model.
We speculate that the reason could be the direct dependence of the stacking model on the strength of the underlying base model .
We make a few key contributions: (1) We propose a combination model of relational ensemble learner RDN-Boost (lower learning bias) and stacking (lower inference bias).
(2) We proposerelational templates for effective combination of RDN-Boost and stacking models.
(3) Our empirical results demonstrate that such a combination does not yield in a statistically signiﬁcant increase in performance.
Related work
Relational dependency networks An effective and efﬁcient way of learning relational dependency networks (RDNs) [16] by turning learning RDNs into a sequence of relational functionapproximations was introduced in [15].
The key idea was to learn each conditional distribution using a set of relational gradient-boosted trees.
Given that dependency networks approximate the true joint distribution as a product of the conditionals, i.e., P ( x 1 , ..., x n ) =
πP ( y i |
y ß) , one could imagine learning each conditional separately and combine them using Gibbs sampling [9].
This has the added advantage that unlike a directed probabilistic model such as a Bayesian network, one does not have to ensure acyclicity, thus resulting in faster learning and inference.
Schulte et al.
[20] transformed Bayesian networks (BNs) to RDNs.
RDNs have further been used for problems pertaining to hybrid domains [18], collective classiﬁcation [12] and applications such as relation extraction [21].
Stacking method Kou et al.
[11] introduced stacked graphical learning, which takes an efﬁcient way of approaching the problem of collective classiﬁcation by composing additional feature sets from predictions on related instances to enhance the performance of base learner.
The stacked graphical learning combines the underlying approaches by Cohen [3] and Wolpert [23] to build stacks and predict labels for instances respectively.
Fast et al.
[7] demonstrated further that the stacked graphical learning can achieve effectiveness while being efﬁcient.
On synthetic data, RDN gets much lower learning bias while stacked graphical model gets much lower inference bias.
Brophy et al.
[2] combines stacked graphical learning with probabilistic graphical models to build a relational model for social network spam.
Eldardiry et al.
[6] evaluated the performance of relational ensemble models via both theoretical analysis and experimental results.
It is shown that relational ensemble learning approach combined with relational ensemble inference approach can target the error in both learning and inference processes resulting in a better performing model.
As mentioned earlier, Natarajan et al.
[15] proposed boosted relational dependency networks (RDN-Boost) by learning relational dependency networks through boosting approach, which reduces the learning bias of relational dependency network and as analyzed in [6].
Combination of gradient boosting with collective inference has been investigated in [1] focusing on continuous prediction.
A recent work [19] transfers RDN-Boost by mapping predicates in learnt trees from one domain to another similar domain and modifying the mapped trees to achieve better performance.
Stacking Boosted RDNs
Stacking [11] can reduce the inference bias with respect to the underlying base algorithm but combining it with relational machine learning algorithms that reduce learning bias is still largely an unexplored problem.
In this work, we combine the stacking method with RDN-Boost [15] as the base learner which have shown to reduce the learning bias.
[Figure]
Stacked-RDN-B-learn train
J
D train is training data, relational template, number the number of iterations; learn a local model RDN-B [-
-]0 ← RDN-B( D train ); learn stacked models RDN-B 1 · · · RDN-B K ; for i = 1 · · · K do randomly evenly split D train into J folds: D train 1 , · · · , D trainJ ; for j = 1 · · · J do learn RDN-B (cid:48)
j ← RDN-B( D train \ D train j ); predict ˆ y j =
RDN-B (cid:48)
j ( D train j ) ; end
D train (cid:48)
=
D train ; ˆ y
= ˆ y 1 ∪ · · · ∪ ˆ y J ; ( N is the number of instances) ; for n = 1 · · · N do new predicate of instance n = C
( ˆ y , instance n , D train ) ;
D train (cid:48)
=
D train (cid:48) ∪ {new predicate of instance n } ; end learn RDN-B
i ← RDN-B( D train (cid:48) ); end
Stacked-RDN-B = ensemble(RDN-B 0 ,RDN-B 1 , · · · ,RDN-B K ); ;
Stacked-RDN-B-infer (Stacked-RDN-B, D test ); ˆ y 0 =
RDN-B 0 ( D test ); for i = 1 · · · K do for n = 1 · · · N do new predicate of instance n = C
( ˆ y i − 1 , instance n , D test ) ;
D test (cid:48)
=
D test (cid:48) ∪ {new predicate of instance n } ; end predict ˆ y i =
RDN-B i ( D test (cid:48) ) ; end inference =
ˆ y K ; ;
C ( ˆ y , instance, D ) ; ﬁnd all related instances {instance m , m ∈ M } ( M is the set of the indices of all related instances to this instance) of this instance in the data D using relational templates ; aggregate predictions { ˆ y m , m ∈ M } of those related instances to generate new predicate for this instance ; An overview of the stacking approach is shown in ﬁgure 1a.
Stacking consists of two components: the learning component and the inference component. Both of these components need to be consistent to guarantee collective classiﬁcation. During the learning step of stacking, initially a local model f 0 is learned from the given training data. After learning the local model, K stacked models f 1 · · · f K are learned in K iterations.
For each iteration, label predictions of instances are generated by following a procedure similar to cross-validation and a new feature/predicate is created for each instance according to label predictions of its related instances. A stacked model f i is learned from the training data including the new predicates for this iteration i . In the subroutine, the training data is split into J folds, where, one fold j ∈ J constitutes the testing data and the other folds make up the training data.
A model is then trained on the training data and predict on the testing data.
The predictions on the testing data become the label predictions for this fold j .
After J iterations, instances in each fold have their label predictions. Figure 1b shows an overview of RDN-Boost which learns a new ﬁrst-order logic regression tree ∆ j based on all previous relational regression trees F ( ψ 0 , ∆ 1 , · · · , ∆ j − 1 ) . At each step, a new relational regression tree ∆ j is ﬁtted to reduce the predicting error ( I − F ) further from the combination of previous trees F ( ψ 0 , ∆ 1 , · · · , ∆ j − 1 ) and is run for a speciﬁed number of iterations, say t , to learn t trees.
The overall model is an ensemble of all the learned trees.
For each gradient step, regression examples are ﬁrst updated using the functional gradient ∂logP ( y i ; x i ) ψ
( y i =1; x i ) =
I ( y i = 1; x i ) − P
y i = 1; x i ) , where y i is the target predicate, x i is other predicates and ψ ( y i = 1; x i ) is the potential function.
Then a relational regression tree ∆ j is ﬁtted on the regression examples and the ensemble model is updated with the new ﬁtted relational regression tree.
The ﬁnal model is simply a sum of these trees.
Note that the trees follow single path semantics – each example will satisfy only one branch of the trees.
Thus for every example, each tree returns a single regression value and all the values are then summed to get the ﬁnal value.
The stacking framework requires the design of a relational template.
The relational template is important for the performance of the system, since it is used to collect predictions from related examples.
In [11], the collected predictions are aggregated to generate a new feature for assisting the inference.
Since RDN-Boost learns and extracts logic rules from the data, a new predicate is generated for assisting the inference.
We present our method Stacked RDN-Boost in algorithm 1 which consists of 2 procedures describing the learning process Stacked-RDN-B-learn and the inference process Stacked-RDN-B-infer .
The learning procedure, Stacked-RDN-B-learn , consists of ﬁrst learning a local model RDN-B 0 by ﬁtting the base learner RDN-Boost (RDN-B) on the training data D train .
The stacked models,
RDN-B 1 · · · RDN-B K , are then learned in K iterations.
For each iteration i = 1 · · · K , the training data D train is split into J folds where every fold forms the testing data with rest of the folds serving as training data.
We then learn RDN-B (cid:48)
j on D train \ D train j using the base learner RDN-B, and predict on D train j to get ˆ y j for this fold D train j .
After J iterations and thus predictions on the J folds, let us denote the label predictions for all instances as ˆ y .
For each instance n = 1 · · · N , we create a new predicate by aggregating label predictions of related instances retrieved by relational template C on the relations (facts) in D train .
The new predicates are included into D train although only for the present iteration, say i .
At the end of this iteration, we learn the stacked model RDN-B i on D train (cid:48)
(=
D train ∪ {all new predicates}) using the base learner RDN-B.
For the inference process Stacked-RDN-B-infer , we ﬁrst predict on the testing data D test using the local model RDN-B 0 to get label predictions ˆ y 0 .
Then, for each instance n = 1 · · · N , a new predicate is created via aggregating label predictions from ˆ y i − 1 of related instances retrieved by relational template C , as in the training phase, on the relations in D test .
These new predicates are included into D test only for this iteration i as in training.
At the end of the iteration, we predict on the D test (cid:48)
(= D test ∪ {all new predicates}) using the stacked model RDN-B i , and obtain the label predictions ˆ y i .
The inference result ˆ y K of the last iteration is the ﬁnal inference result of our model.
The relational template C is used to ﬁnd all related instances {instance m , m ∈ M } ( M is the set of the indices of all related instances to the current instance) to the current instance via relations in the data D , and then generate a new predicate for the current instance by aggregating over label predictions { ˆ y m , m ∈ M } of those related instances.
The details of the different relational templates considered in this work are given next.
3.1 Relational template
We propose 3 relational templates to combine the methods of stacking and RDN-Boost effectively
. 3.1.1 Aggregate over adjacent entities
For each entity, all other entities which appear in any relation with this entity are retrieved and their label predictions are collected.
If the majority of predictions of related entities is positive, then a new predicate is created for this entity.
Otherwise, no new predicate is created.
This is akin to a majority vote from related entities.
The predictions for entities here are inferred by the model during each cross-validation step similar to the subroutine of stacking method.
3.1.2 Aggregate over entities within ﬁxed depth
The given relational data set can be treated as a graph G =
( V, E ) where relations are treated as edges E and the entities are treated as nodes V .
The depth is the least number of edges between two entities.
For each entity, all other entities within the the depth are retrieved and their label predictions are collected.
If the majority of predictions of related entities is positive, a new predicate is created for this entity.
Otherwise, no new predicate is created for this entity.
3.1.3 Aggregate by Hsu-Lyuu-Flandrin-Li distance
This distance metric [4, 13] is based on the concept of m -diameter, and is given as, [Equation]
[Equation]
The deﬁnition of graph G =
( V, E ) is similar as aggregate over entities within ﬁxed depth .
Relations are treated as edges E and entities in the relations are treated as nodes V .
The m -diameter D Gm ( a, b ) between node a and node b is the minimum integer d such that there are at least m internally disjoint paths of length at most d between two nodes [13].
Since the time complexity of this distance metric is very high and there is no available implementation, to simplify this distance metric, m is set to beand we have a special case.
Then the Hsu-Lyuu-Flandrin-Li distance between two nodes D G 1 ( a, b ) is simpliﬁed to be the length of the shortest path between two nodes.
Thus, for every entity e i , breadth-ﬁrst search is applied here to ﬁnd the length d j of the shortest path between entity e i and entity e j .
The positive and negative label predictions ˆ y are mapped to +1 and -1.
For each entity e i , the inverse 1 /d j of distance between entity e i and entity e j is used as the weight w j of entity e j .
A weighted voting (cid:80) Nj =1 ,j (cid:54) =
i w j · ˆ y j is done over all other entities.
If the weighted sum is larger than or equal to 0, then a new predicate is created for entity e i else no new predicate is created.
Experiments and analysis 4.1 Datasets
We consider 3 relational data sets: Carcino : is a biomedical data set [22] of the structures of various chemical compounds and the task is to predict if these compounds are carcinogenic in nature.
PPMI : is a study [14] designed to identify bio-markers that impact Parkinson’s and the task is to predict if a patient has Parkinson’s [5].
UW-CSE : is a task over a representation of staff, students and faculty of 5 different computer science departments; the target is to predict whether someone is a professor.
4.2 Experiments setup
Each dataset is split into training data and testing data with a ratio of 60% - 40%.
We varied relational templates, number of trees of RDN-Boost, number of iterations and number of folds of stacking method.
The details of the three relational templates are displayed in the section 3.
We ran the experiments on a Intel(R) Xeon(R) CPU E5-1620 v3 @
3.50GHz with 32 GB RAM machine.
We implemented our approach in Python.
For the base learner/baseline RDN-Boost, we used the srlearn python wrapper [8] and implemented the stacking approach. [Table] [Table]
4.3 Results
We ﬁrst present our results with the relational template aggregate over adjacent entities .
The number of trees of RDN-Boost is set at 10 and the values of K and J are varied as on [(1,3), (2,3), (1,5), (2,5)], since as shown in [11], large K does not help the performance.
And the results on the dataset
UW-CSE with this relational template are not shown due to lack of space as well as because there is no signiﬁcant improvement compared to RDN-Boost, which is similar in behavior with the data sets
Carcino and PPMI, the results for which are shown in tables 1 and 2 respectively.
From these results it can be seen that the improvement is not signiﬁcant or consistent as expected.
A better relational template may be needed to exploit the strengths of stacking method.
Thus, we next implemented the relational template aggregate over entities within ﬁxed depth .
The depth is varied, such as 2 or 3, to evaluate the model and the performance is not improved as expected.
The experiment results with this relational template are omitted since there is no signiﬁcant improvement in the results and are similar to previous results.
Since assisting prediction from neighbors in a ﬁxed range does not work well, the next approach may be to try an appropriate distance metric for aggregating predictions from related entities.
We next used the Hsu-Lyuu-Flandrin-Li distance [4, 13] for the relational template.
Besides the updating of relational template, parameters of RDN-Boost and stacking method are explored more in order to capture the performance trends of our approach.
For the base learner RDN-Boost, we changed the number of trees from small to large (2 to 20) in order to see the effect of different learning ability of base learner on the system.
For each such setting of the number of trees, we generated results for settings of the number of iteration K and the number of folds J of stacking method as [(1,3), (2,3), (1,5), (2,5), (3,5)].
The results for the 3 data sets are shown in tables 3, 4 and 5.
The results for the setting of ( K , J ) of stacking method = (3,5) is not shown since there is no signiﬁcant difference with other ( K , J ) settings.
As mentioned in Kou et al.
[11], the stacking method converges very fast and the larger number of iterations does not raise the level of performance.
The results of the number of trees as 5 are not shown since the results are also similar and the case of the number of trees as 2 can cover the cases with small number of trees. [Table] [Table] [Table]
F [Table] AUC-PR [Table]
AUC-ROC
[Table]
[Table]
The improvement of performance is not statistically signiﬁcant contrary to our hypothesis and expectations.
Most importantly, the improvement is not consistent through all settings of the number of iterations K and the number of folds J of the stacking method.
There appears to be no discernable pattern in the results that would provide a deeper understanding of the effects of stacking.
From tables 3 and 4, it can be seen that though the variation of the number of trees of RDN-Boost changes its learning ability, it still performs fairly well on both data sets.
The selected data sets may be not complex enough for the strong base learner RDN-Boost so that there is not much space left for the stacking method to improve .
As analyzed in Fast et al.
[7], on synthetic data, RDN gets lower learning bias and higher inference bias; stacked graphical learning gets higher learning bias and lower inference bias.
Combining them together may not necessarily improve the overall performance since their under-performance on inference and learning respectively also decides the ﬁnal performance.
On real data, it shows that stacked graphical learning gets much lower inference variance than RDN which is not obvious in our experiments either.
Since the system was run only 5 times under each setting to get the mean and standard deviation, more runs may be required to get a precise evaluation of variance .
Conclusion
Initial results clearly show that the approach of combining RDN-Boost with stacking method does not perform signiﬁcantly better than the baseline i.e. RDN-Boost as expected.
The reasons might be from the design of the approach or the complexity of data.
The relational template is key part of the stacking method.
A proper relational template could potentially facilitate the combination of RDN-Boost and stacking method.
On a different note, the data sets in the section 4 may not necessarily be complex.
Since RDN-Boost is a strong base learner, though the paper [6] mentioned that boosted approach introduces inference bias, the inference bias on those data sets might not be signiﬁcant enough in performance.
Investigating this deeper is an interesting direction. Acknowledgments
SY, DSD and SN gratefully acknowledge DARPA Minerva award FA9550-19-1-0391.
Any opinions, ﬁndings, and conclusion or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the view of the DARPA or the US government.
Broader Impact
Machine learning models encounter issues of bias, variance and noise while handling real problems.
Advancing boosted-RDN with stacking method attempts to leverage strengths of different deliberate approaches to target multiple aspects of the error.
An approach may be outstanding at one part of solving problems while the cooperation of different approaches makes the whole system more complete and robust.