Semi-Supervised Learning with Decision Trees: Graph Laplacian Tree Alternating Optimization
azharmagambetovu mer ed.edu m arreira-perpinanu mer ed.edu
Abstract
Semi-supervised learning seeks to learn a machine learning model when only a small amount of the available data is labeled.
The most widespread approach uses a graph prior, which encourages similar instances to have similar predictions.
This has been very successful with models ranging from kernel machines to neural networks, but has remained inapplicable to decision trees, for which the optimization problem is much harder.
We solve this based on a reformulation of the problem which requires iteratively solving two simpler problems: a supervised tree learning problem, which can be solved by the Tree Alternating Optimization algorithm; and a label smoothing problem, which can be solved through a sparse linear system.
The algorithm is scalable and highly effective even with very few labeled instances, and makes it possible to learn accurate, interpretable models based on decision trees in such situations.
Introduction
Semi-supervised learning (SSL) is an important subﬁeld of machine learning which has received a lot of attention in recent years given today’s growing amount of data and widespread deployment of machine learning systems.
One of the major reasons is that SSL is applicable when labels are scarce.
This is in contrast to the traditional fully supervised learning, which requires access to a large amount of high-quality labeled data.
However, obtaining such samples is often costly, time-consuming and sometimes even impractical.
Therefore, SSL methods have received much praise in the machine learning literature [40] and they are widely used in many applications.
A common strategy in SSL is to assume that similar instances have similar predictions, which is commonly incorporated into an objective as a graph prior (e g. graph Laplacian).
In this paper we study the problem of training a decision tree model by leveraging a small percentage of labeled data and a much larger sample of unlabeled data.
Why trees?
First, trees are considered to be interpretable models, since the prediction is obtained by routing an input along a unique rootto-leaf path, which can be reformulated as if-then rules.
Second, they are widely used in a wide spectrum of applications, such as data mining [30], computer vision [13], ﬁnance, etc.
They are typically employed as base learners in an ensemble (e.g. bagging [3] or boosting [15]).
However, they are used as standalone predictors as well.
For instance, recent algorithmic advances in training non-greedy trees have shown that oblique trees can perform competitively well in a number of tasks and strike a good balance between accuracy and interpretability [6, 5, 16–18, 34, 36].
However, like many non-linear methods, decision trees are well known to overﬁt for small-sized (labeled) data, which is the case in SSL.
As an illustration, consider ﬁg. 1, that shows a synthetic binary classiﬁcation problem in 2D.
An oblique tree achieves a certain good performance when it is provided the entire population of labeled data.
But the error signiﬁcantly increases if a tree is trained on six labeled instances only (plot 3).
Whereas the beneﬁt is evident when we provide all ground truth
LapTAO (ours) error: 0.0% error: 4.5% error: 23.46% error: 8.78% -   -   -   -   -   -   -   -   data (six labeled and the rest are unlabeled) and properly optimize a tree within SSL framework.
In our proposed approach, we ﬁrst state the objective, which consists of a supervised loss (for the labeled data only) and a graph Laplacian regularization (also known as manifold regularization [2]).
The resulting optimization problem is long considered to be hard to solve since trees deﬁne a nondifferentiable, non-convex mapping.
By reformulating the problem as a constrained optimization, we derive an efﬁcient and scalable iterative algorithm (section 3) which requires solving two simpler problems at each step: a sparse linear system and a supervised tree learning problem.
For the latter, we use the Tree Alternating Optimization algorithm [6, 5], which is crucial for the success of our approach (section 3.1).
Moreover, for a special case where the tree structure as well as the parameters in each decision node (not leaves) are ﬁxed, we derive the exact solution given by another linear system (section 3.3).
Experimental results (section 4) show the algorithm is able to learn accurate and interpretable decision trees even with very few labeled instances.
Related work
Although the literature on this topic is immense (see [40, 29]), incorporating decision trees into SSL has received almost no attention.
One possible explanation is the difﬁculty of the optimization problem.
Most SSL methods are based on adding a graph prior (or similarity matrix) as regularization to exploit the geometry of the underlying data distribution [2, 39, 41, 42].
Minimizing an objective with such regularization is non-trivial when we have decision trees as a predictive model.
That said, several attempts have been made to apply SSL for trees.
Levati ´ c et al.
[23] modify a splitting criterion for recursive tree induction by taking into account unlabeled data.
That is, a splitting score for each [feature, threshold] pair consists of two parts: the traditional purity score (e.g. Gini index) for labeled data and a “clustering” score for all available data.
Tanha et al.
[27] apply a self-training framework [33] to train classiﬁcation trees with labeled and unlabeled data.
Note that self-training is a generic framework that can be applied with any classiﬁer which predicts class probabilities.
It trains a classiﬁer iteratively where the ﬁrst iteration includes only labeled data.
After that, it uses model predictions and adds the most conﬁdently predicted instances to enrich the amount of supervision.
Another direct approach is to apply a label smoothing technique [41] to propagate label information to all data and then ﬁt a tree.
This method has been recently reused within the graph neural networks framework to train boosted trees [12, 21].
Finally, Kemp et al.
[22] proposed a Bayesian approach where unlabeled data assist in inferring a latent tree structure from a distribution over trees.
The limitations of this approach are that it covers classiﬁcation tasks only, and it uses Markov Chain Monte Carlo to sample from the distribution over trees, which leads to scalability issues.
Unlike all these methods, we attempt to directly minimize a regularized SSL objective (with a graph prior) over the parameters of a decision tree.
The method is applicable to both regression and classiﬁcation problems, and can train trees of arbitrary type (i.e., axis-aligned, oblique, etc.).
In fact, our optimization framework is generally applicable to any type of predictive functions, not just trees.
LapTAO: semi-supervised learning framework for decision trees
We are given the dataset D
= D l ∪ D u , where D l = { x n , y n } ln =1 ⊂ R D
×
R is the labeled portion of the data, with l points, and D u = { x n } Nn = l +1 ⊂ R D is the unlabeled portion, with N − l points.
Then our goal is to minimize the following regularized objective:
[Equation]
Here, w nm are the weights in the afﬁnity (similarity) matrix based on a graph on all the data points D , usually a nearest-neighbor graph; T : R D → R is the tree predictive mapping, with parameters
Θ
= { θ i } nodes ; φ ( · ) is a regularization penalty, such as k·k 1 ; and γ , α are regularization hyperparameters.
Rather than using a greedy recursive partitioning procedure (such as CART [4] or C5.0 [26]), which does not optimize any loss function, we consider T as a parametric model with trainable weights in each node (like ﬁxing a neural net architecture and optimizing over its parameters).
If T was differentiable, one could optimize (1) via gradient-based methods, as can be done for neural nets [31].
Similarly, the solution is relatively straightforward to obtain if problem (1) is convex [2].
However, solving problem (1) is non-trivial with a tree which deﬁnes a non-differentiable and nonconvex mapping.
Instead, we apply the method of auxiliary coordinates [8, 9], a generic method for optimizing nested systems.
We proceed as follows by reformulating problem (1) in an equivalent form (similar to that in [7, 36, 17] for dimensionality reduction and clustering).
Introduce a new auxiliary variable z n ∈ R for each training instance n and consider the constrained problem:
[Equation]
Obviously, by putting the constraints (3) back into eq.
(2), we end up with the same objective as in (1), so these two problems are equivalent.
Let us denote y =
[ y 1 , y 2 , . . .
, y l , 0 , 0 , . . .
T ∈ R N the augmented ground truth vector, i.e., we put zeros in the unlabeled portion of the data.
Similarly, introduce a diagonal matrix J = diag (1 , . .
., 0) ∈ R N × N with the ﬁrst l diagonal entries equal to 1 and the rest to 0.
Also, let the graph Laplacian be L
= D − W with a diagonal matrix
D ∈ R N × N (the degree matrix) having entries d nn =
P
N m =1 w nm , and let W = ( w nm ) ∈
R N × N be the afﬁnity matrix.
Finally, call z =
[ z 1 , . .
, z N ] T and t ( X ; Θ ) =
[ T ( x 1 ; Θ ) , . . .
, T ( x N ; Θ )] T , where X =
( x 1 , . .
, x N ) .
Then we can rewrite eq.
(2)-(3) as follows:
[Equation]
Now we solve this using the augmented Lagrangian method [25].
This deﬁnes a new, unconstrained optimization problem:
[Equation]
where λ ∈ R N are the estimates of the Lagrange multipliers.
Optimizing this for each µ >produces a sequence of ( z µ , t µ ( X ; Θ )) and, as µ → ∞ , we gradually force the minimizer to be in the feasible set of the constrained problem.
Finally, in order to minimize (5) over z and t ( X ; Θ ) for ﬁxed µ , we apply alternating optimization over z and Θ :
• Label-step (optimizing over z given ﬁxed t ( X ; Θ ) ).
The objective in eq.
(5) is a quadratic function and a minimizer is obtained by solving the linear system:
[Equation]
where A =
J + µ
I
+ γ L is a positive deﬁnite
matrix.
Moreover,
A
is a
sparse
matrix if the graph Laplacian L is sparse, which is the case in practice if we construct W by using a nearest neighbors graph.
This allows us to solve a large scale linear system in an efﬁcient way (e.g. by caching a matrix factorization or using the conjugate gradient method).
Intuitively, the label-step can be interpreted as approximating the labels (for D u ) using the graph Laplacian and predictions obtained from the current tree (i.e., label smoothing).
• Tree-step (optimizing over Θ given ﬁxed z ).
Problem (5) reduces to a regression ﬁt of a tree:
[Equation]
Note that
here we use ( z − 12 µ λ ) as “ground-truth” labels (not y n , which is not deﬁned for D u anyway)
We solve
this problem
using the Tree Alternating Optimization (TAO) algorithm
which
we describe in section 3.1 below.
Intuitively,
this step can be understood as ﬁtting a tree with the current estimates of the labels
Finally, the step over Lagrange multipliers is done by the update λ ← λ − µ ( z − t ( X ; Θ )) .
In summary, our algorithm alternates between solving a linear system and training a tree.
After each (label,tree)-step, we increase the penalty parameter µ , we update λ and we keep iterating until approximate convergence or other stopping criterion (e.g. maximum number of iterations reached).
We call our algorithm LapTAO and provide detailed pseudocode in the suppl.
mat.
Initialization for LapTAO
To start our iterative algorithm, we need to obtain initial solutions ( z 0 , t 0 ) for eq.
(5) when µ → 0 + .
This is straightforward to achieve for z 0 , as it involves solving the same linear system as in eq.
(6) but with µ = 0 and λ = 0 :
( J + γ L ) z =
Jy .
This can be considered as a non-parametric smoothing of the labels obtained by propagating (diffusing) the ground-truth labels over all points (labeled and unlabeled) through the graph Laplacian.
Although these smoothed labels are not optimal in problem (1), which requires optimizing them jointly with the tree, they do provide a good initialization, and it is convenient to solve the linear system exactly for µ = 0 .
After that, we ﬁt a tree using z 0 as ground-truth labels (tree-step).
Note that TAO requires an initial tree (see section 3.1), which we obtain by generating a complete tree of depth ∆ (a hyperparameter) with a Gaussian random weight vector at each decision node.
Hyperparameters of LapTAO
The hyperparameters are γ for the graph prior and α (sparsity) and ∆ (depth) for the tree.
They can be selected by cross-validation.
Extension to multioutput regression and classiﬁcation We can extend LapTAO for multiple outputs in a straightforward way: the label-step solves the linear system (6) for each dimension separately and the tree-step applies TAO as usual, as it can handle a vector-valued output.
For classiﬁcation, we use a one-hot encoding of the labels and consider the problem as a regression task, as is commonly done in the decision tree literature [19] (especially for boosted trees).
It is possible to extend our framework for other losses (e.g. hinge, logistic) that may work better for classiﬁcation.
This requires certain changes in the label-step and we will explore it in our future works.
Extension to models other than trees Although our focus in this paper are decision trees, the semi-supervised learning optimization algorithm
we propose is perfectly general.
The model ( T ( x ; Θ ) ) appears in the algorithm in the tree-step, with the form of a regression problem having the smoothed labels as ground-truth.
Obviously, we can use other regression models, such as random forests, gradient boosted trees, neural networks, etc.
The motivation to use our approach was the fact that trees are not differentiable, so one cannot optimize problem (1) by gradient-based methods.
But our approach has another, computational advantage: by separating terms through the auxiliary variables z , the quadratic-cost term is conﬁned to the label-step.
For large datasets, this is a sparse linear system, for which efﬁcient algorithms exist.
Hence, the complexity associated with the model (tree) is linear on the dataset size (in the tree-step).
This is much faster than having to deal directly with the quadratic term and the model, as in eq.
(1), whether for trees or other models.
3.1 Overview of the tree alternating optimization (TAO) algorithm
Potentially, one could apply any tree ﬁtting algorithm to solve the tree-step in LapTAO, such as
CART [4], C5.0 [26], OC1 [24], etc.
(we do show such results in the suppl. mat.).
But there are several important considerations.
Firstly, from an optimization point of view, it is known that alternating optimization is most effective when the step over each block is (ideally) exact.
This is computationally achievable for the label-step, which involves a linear system.
However, training a tree optimally even in the simplest case (axes-aligned with binary inputs and output) is NP-hard [20].
Therefore, we need an approximate but good solution.
Most traditional tree learning algorithms, based on greedy recursive partitioning (such as CART), are highly suboptimal [19] and do not even consider any speciﬁc loss function over trees.
In contrast, the Tree Alternating Optimization (TAO) algorithm [6, 5] ﬁts decision trees by monotonically decreasing a well-deﬁned and very general loss function and regularization over a well-deﬁned parametric space of trees, given an initial tree structure and parameter values.
This makes it also possible to use warm-start in the tree-step, i.e., to continue improving the tree from the previous iteration—which greedy recursive partitioning cannot do, as it constructs a new tree from scratch every time.
Warm-start is essential to speed up the optimization and to achieve stability in the results (CART-type algorithms are notoriously unstable in that small changes in the training set can result in drastically different tree structures and parameters [19]).
A further, important advantage of TAO is that it can learn trees of quite general types, such as oblique trees (see section 3.2), which are far more powerful that the traditional axesaligned trees.
TAO has been shown to ﬁnd much better trees under a variety of losses, regularization and types of tree, as well as forests, and scales well to large datasets, see e.g. [37, 34, 38, 35, 16, 10].
Below we provide a summary of TAO for completeness; more details can be found in [6, 5, 34].
TAO considers a decision tree as a parametric model with a given structure and trainable parameters in each node ( Θ = { θ i } nodes ), and it optimizes an objective consisting of a loss and a regularization term, such as that in eq.
TAO requires the loss to be a sum over individual points, so it cannot handle eq.
(1), which includes pairwise distances.
However, our reformulation of the problem allows us to apply
TAO since the minimization in eq.
(7) is now in the desirable form.
TAO takes as input an initial tree, for example a complete tree of depth ∆ with random weights at each node, or a tree created by another algorithm.
It uses a separability condition to decompose the loss function (7) over subsets of non-descendant nodes.
For instance, all nodes at the same depth are non-descendant w.r.t.
each other and can be trained independently (and in parallel).
Then, TAO uses an alternating optimization scheme such as the following: 1) pick all nodes at a certain depth and ﬁx all remaining nodes (i.e., we consider them as a function with constant parameters); 2) optimize each node at that depth efﬁciently by solving a reduced problem (see below); 3) repeat this for all depths.
One pass through all nodes is a TAO iteration, and we keep iterating until convergence occurs or until we reach a maximum number of iterations (see pseudocode in the suppl.
mat.). Optimizing a single node i (decision node or leaf) over its parameters can be shown to be equivalent to a simpler, reduced problem operating only on those training points which currently reach i (denoted as reduced set R i ).
The solution of the reduced problem depends on the type of node:
• Decision node .
With binary trees, this involves solving a certain binary classiﬁcation problem.
Given the reduced set R i , we pass each input { x n , y n } ∈ R i to both children of i and calculate the error ( e left n , e right n ) induced by each child (note that all descendant nodes and their parameters are ﬁxed).
Next, assign a pseudolabel ¯ y n ∈ {− 1 , 1 } depending on which child brings the lower error (either left or right).
Each pseudolabel comes with the weight | e left n − e right n | since the error made for each instance is different.
Solve a weighted 0/1 loss binary classiﬁcation on { x n , ¯ y n } .
• Leaf .
The actual prediction of a tree occurs in its leaves; the decision nodes are just responsible for routing a point to the corresponding leaf.
Therefore, the optimization problem for a leaf is equivalent to the original problem (7) but on its reduced set.
Its solution depends on what type of leaf model we use.
With a constant label, it is the average of the response values ( y n ) in that leaf.
With linear leaves, the solution is to ﬁt a linear regressor on R i (possibly with regularization).
3.2 Sparse oblique decision trees
The TAO algorithm is applicable to a large spectrum of decision tree types.
Here we pick a sparse oblique tree with constant leaves as our main model.
Each decision node i makes a hyperplanebased split: go to the left child if w Ti x < w i 0 , else go to the right one.
Therefore, the decision node reduced problem (which is NP-hard) is approximated by ﬁtting a logistic regression.
Additionally, we apply an ℓ 1 penalty as regularization term φ ( · ) to encourage sparsity.
We use LIBLINEAR [14] to ﬁt this model.
As for the leaves, we take the average of the response values ( y ) in that leaf.
The motivation behind choosing sparse oblique trees is twofold.
First, traditional axis-aligned trees are very restrictive since each split uses a single feature, which neglects interactions or correlations between features.
Indeed, empirical results show oblique trees achieve far better performance [38].
Besides, thanks to the sparsity, only few features are active at each split, and nodes in the initial tree can become redundant and be pruned if their weight w i becomes zero [6].
This, together with the fact that oblique trees typically have small depth, makes the ﬁnal model more interpretable.
3.3 Special case: exact solution when the tree structure is ﬁxed
Problem (1) is hard to solve over the entire space of decision trees, but we can obtain an exact solution if the structure and decision node parameters are ﬁxed, since then the problem reduces to solving a linear system.
In this case, the only parameters to optimize are in the leaves.
Assuming each leaf outputs a constant value, we can reformulate the tree prediction as a sum of basis functions
T ( [Equation] =
[Equation]
i [Equation] ) , where m is the number of leaves, c i ∈ R is leaf i ’s label and b i ( · ) ∈ { 0 , 1 } is 1 only if x ends up in leaf i .
Now we can rewrite (1) as the following minimization problem over the parameters of all the leaves c = ( c 1 , . . .
, c m ) T :
[Equation]
where B =
( b i ( x n ))
∈ R N × m can be precomputed since we ﬁx the tree structure and parameters in all decision nodes.
Minimizing this over c yields the following linear system:
[Equation]
(10) where A = B T JB + γ B T LB is a matrix of m × m .
This is very fast to solve since oblique trees are quite shallow, so the number of leaves m is not large (at most 1 000 in our experiments).
Once
LapTAO is ﬁnished, we apply the above procedure as a post-processing to the ﬁnal tree.
3.4 Computational complexity of LapTAO
At the top level
LapTAO runs a ﬁxed number of iterations (depending on the µ schedule, typically less than 20).
Each iteration has to solve (approximately) two subproblems:
• Label-step : this is a large, sparse linear system of N × N (where N is the sample size).
We solve it approximately with conjugate gradients (CG), initialized by the previous iterate (warm-start).
Each CG iteration is O ( Nk ) where k is the average number of neighbors in the graph, and we run just a few CG iterations.
The total runtime of the label step is less than 30 seconds in the largest experiment we conducted (1M points).
Convergence can be further improved via preconditioning (e.g. Jacobi).
We can also solve the linear system exactly in O ( N 2 ) by caching its SVD, as noted in the suppl. mat. (section 3), but this is only convenient if N is a few thousands at most.
• Tree-step : ﬁtting an oblique tree with TAO to the N training points.
Each iteration of TAO updates each decision node and leaf node.
For each leaf, we compute the average of the labels of its reduced set (training points reaching it), so this is O ( N ) over all the leaves.
For each decision node, we train a logistic regression on its reduced set.
Assuming logistic regression is linear on the sample size and dimensionality, this is O ( ND ) total for all the decision nodes at the same depth, although with a larger constant factor in the big-O notation than for the leaves.
Hence, processing all the decision nodes in the tree is O (∆ ND ) , or equivalently, running ∆ logistic regressions on the whole training set.
See more details in [6, 5].
A critical computational advantage of TAO is
due to the fact that each node (decision or leaf node) only handles the points in its reduced set.
Therefore, TAO itself can be parallelized depthwise.
In summary, the overall runtime of TAO is O (∆ ND ) per TAO iteration.
We run 10 TAO iterations in our experiments.
Since the tree-step dominates the label-step, in terms of runtime our algorithm is almost like sequentially training decision trees (as in boosting).
Additionally, each tree-step can be parallelized.
Further acceleration can be done using GPUs.
This is possible with GPU-friendly implementations of logistic regression, and also because oblique trees involve scalar products (unlike axis-aligned trees).
Finally, this computational cost should also include computing the nearest-neighbor graph and its afﬁnity matrix W .
This is indeed a large cost, and it affects all semi-supervised learning methods based on the graph Laplacian.
A naive implementation requires O ( DN 2 ) to calculate the distance vector for each point and determine the nearest neighbors.
For large datasets, one usually uses approximate nearest neighbors (e.g. via Locality Sensitive Hashing or other techniques [1]).
In terms of storage, besides the tree T , we need to store the auxiliary variables z (predicted labels) for the entire training set, which are N scalars for 1D regression or binary classiﬁcation.
Experiments
This section shows our experimental ﬁndings.
We demonstrate that the proposed method dominates over other semi-supervised learning frameworks in accuracy and approaches fully supervised baseline with far less amount of labeled data .
This is true with practically no exceptions against baseline tree-based models where accuracy margin is often quite large.
As for the other methods (non tree-based), we either outperform them or achieve similar error, which makes LapTAO a strong competitor.
To show that, we consider several regression and classiﬁcation benchmarks of varying sizes and across different domains.
We were able to run our algorithm on a dataset with up tomillion instances on a regular PC, which shows its scalability.
Moreover, using fashion-mnist as an example, we demonstrate that the ﬁnal model, a shallow oblique tree with sparse parameter vector in each node, provides insights into how it achieves a prediction allowing model interpretability.
4.1 Experimental setup
We compare our proposed approach (LapTAO) with the following baselines: 1) oblique–all ﬁts an oblique tree with full supervision, this shows the theoretical maximum performance we can achieve; 2) oblique–lbl is the oblique trees trained on labeled portion of data D l (this completely discards large portion of unlabeled data); 3) Self-training ( axis–self , oblique–self ) is an iterative procedure that uses the model predictions to enlarge the portion of labeled data (see section 2 for details and references), we closely follow the implementation by Yarowsky [33]; here, “axis” means traditional axis-aligned trees; 4) Laplacian SVM is a seminal work by Belkin et al.
[2] which has similar problem formulation as in eq.
(1) but for SVM.
Additionally, the suppl. mat. include the comparison with semi-supervised classiﬁcation trees by Levati´c et al.
[23] and EBBS by Chen et al.
Regarding hyperparameters, given the ﬁxed cross-validation set (1% of train data), we explored as best as we could all important hyperparameters for all methods (see details in the suppl. mat.).
These include: controlling a tree depth ( ∆ ), conﬁdence threshold for self-training, σ and C values for LapSVM, etc.
It worth to mention that the hyperparameter settings suggested by authors or their default values work best in most cases.
We use TAO to train oblique trees and CART [4] to train axis-aligned trees.
For all methods that use TAO, we set the total number of TAO iterations to 15.
The depth ∆ as well as the regularization parameter α are tuned via cross-validation.
As for the settings that are speciﬁc to LapTAO, we proceed as follows.
To construct the graph Laplacian, we use the Gaussian afﬁnities with k -nearest neighbors and perplexity parameter K tuned for each dataset.
The linear system in the label-step is solved either using direct methods (less than 20k dimensions) or Conjugate Gradient method for large scale problems.
We use γ = 0 .
1 in all experiments.
As for the main loop of the augmented
Lagrangian, we iterate 20 times starting from small value for µ 0 = 0 .
001 multiplied by 1.5 after each iteration.
The remaining details as well as dataset descriptions can be found in the suppl. mat.
4.2 Main results
Fig.
2 summarizes the main results which are the trade-off plots of test error versus the percentage of labeled data on two regression and classiﬁcation tasks.
Intuitively, the error should go down monotonically as we increase the amount of supervision which is clearly the case in all ﬁgures.
According to our ﬁndings, KNN and “axis–self” show the worst results in almost all benchmarks.
The only case when KNN performed reasonably good was on MNIST, which is known to work well with “template classiﬁers” (e.g. RBF network, kernel SVM, KNN, etc.).
Even in that case it has a large error gap with respect to LapTAO.
The poor performance of the “axis–self” can be explained by suboptimality of greedily grown trees [19] and suboptimality of the self-training approach, which is mostly based on heuristics.
Next, oblique trees trained on supervised data only (“oblique–lbl”)
←−−−−−−−−−− Regression −−−−−−−−−−→ ←−−−−−−−−−− Classiﬁcation −−−−−−−−−−→ cpu_act (8k,21,1) year_pred(0.5M,90,1) mnist(70k,784,10) susy(1M,18,2)
[Figure]
% of labeled data
[Figure]
E t e s t (
M S E )
[Figure]
[Figure]
leads to the signiﬁcant drop in accuracy (magenta vs black lines).
This shows that relying only on labeled data is not enough to achieve a decent performance.
Incorporating an oblique tree into selftraining framework brings certain beneﬁts (“oblique–self”), notably for classiﬁcation tasks (green dashed vs solid lines).
Finally, LapTAO consistently improves over all other SSL methods , often by a considerable margin.
For instance, in case of 3% in cpu_act and 1% in MNIST, the difference in the error with the second best SSL approach is several orders of magnitude.
It shows acceptable results even in extreme label scarcity scenarios, e.g. when we provide < 0 .
5% of labeled data on year_pred and susy.
Moreover, LapTAO approaches the fully supervised baseline more quickly: for MNIST, we can achieve the same ∼ 5% test error as “oblique–all” using only 20% of labeled training points.
[Figure]
Comparison with Laplacian SVM LapSVM is a natural baseline to compare with since it uses the same problem formulation as in eq.
(1) but for the support vector machines.
However, we were not able to include it to the previous comparison (in ﬁg. 2) due to implementation issues: 1) it is impractical to apply it for problems beyond 30k instances as it requires computing the inverse of the modiﬁed dense Gram matrix on the entire dataset; 1 2) it can handle a classiﬁcation task only.
Therefore, we pick the subset of Fashion-MNIST (3 classes: “shirt”, “bag” and “ankle boot”) resulting in 18k training points.
To make the comparison as fair as possible, both of the algorithms use the same graph Laplacian matrix and we enforce the same penalty on it ( γ = 0 . 1 ).
For LapSVM, we use the rbf kernel with σ = 5 and the hyperparameter for
LIBSVM [11] is set to C = 100 .
The results are illustrated in ﬁg. 3.
It is worth to mention that similar to the original MNIST, “template classiﬁers”, such as kernel SVM, show quite good performance on this task which makes LapSVM a strong baseline [32].
On top of that, the problem formulation in eq.
(1) is still convex for SVM and can be efﬁciently solved, whereas we are dealing with much harder problem for oblique trees.
Therefore, it is nice to see that LapTAO performs similarly (but slightly worse) up to some point.
However, it is surprising to see that the error gap between our approach and LapSVM narrows as we introduce more label scarcity and eventually we start to outperform when % of labeled data = 3%.
From that point on, the table turns to the side of LapTAO and the difference becomes more and more noticeable (especially for 0 .
One possible explanation for this behavior is the overﬁtting issue on small datasets, which
[Table]
is a known problem for kernel SVM, whereas sparse oblique trees are shown to be relatively robust to that [38].
Training time Table 1 reports the training time for different baseline methods.
Overall, LapTAO algorithm for the largest experiment we performed (on susy) took less than 7 hours and aroundhours for the moderate dataset size (mnist).
This is comparable to the self-training baseline (i.e., oblique–self) but LapTAO produces far better trees in terms of accuracy.
Please note that we ran our code on a regular PC (Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz, 32GB RAM), with little parallel processing and using unoptimized Python implementation.
Therefore, the training runtime for LapTAO can be signiﬁcantly improved.
We did not use any GPUs.
4.3 Model interpretability
Model explainability and interpretability is a topic of renewed interest due to the widespread usage of machine learning and the risks associated with privacy, algorithmic bias, etc.
In order to trust and rely on such automated systems, it is crucial to understand how they achieve a certain prediction.
In contrast to “black box” models, decision trees are long considered as interpretable models due to the hierarchical structure.
This allows to transform the model prediction as “if-then” rules extracted from root-to-leaf path.
Speciﬁcally for oblique trees, each logical clause takes the following form: go to left child if w T x < w 0 , else go to right.
This makes the interpretation a little harder since we need to look at linear combination of features at each split.
However, in our case, we add ℓ 1 penalty which encourages parameter vector at each node to be sparse, i.e., only few features participate in decision making.
In this section, we argue that the oblique trees trained using LapTAO strike a good trade-off between accuracy and interpretability which is controlled via hyperparameter α .
To illustrate this, we use the same subset of Fashion-MNIST as in section 4.2 and train a sparse oblique tree using LapTAO (10% of training data are labeled).
By decreasing the value of α we enforce more sparsity resulting into shallower and more interpretable trees (since complexity decreases).
However, we sacriﬁce the performance since the error goes up.
Fig.
4 shows the results for α = 1 and α = 10 , more results can be found in the suppl. mat.
For simplicity, let us focus on the bottom tree ( α = 10 ).
Clearly, each leaf contains instances of nearly the same class since the average image looks like a representative “template” from the corresponding class.
As for the decision node, consider the root (node #1).
All “Boot” images are sent to the right child of the root.
Also, it is easy to notice that such images do not contain any pixels in the top-left quadrant of the image.
Therefore, the weight vector at the root has negative (blue) values in the corresponding elements and we know that negative values are responsible for sending an instance to the left.
In other words, all images that have something in the top-left quadrant are sent to the left.
Therefore, boots will end up in the right child.
Similarly, node #2 sends most of the images that have
M-shaped stroke in the top-center part (e.g. large bags, shirts with collar) to the left child.
Following the same logic, we can obtain meaningful insights for each decision node.
Also note that all nodes have majority of values equal to zero (thanks to the sparsity) which makes the interpretation easier.
Conclusion
Semi-supervised learning is most commonly formulated with a graph-based regularization term, which encourages the labels of nearby (similar) points to be similar.
This has been very successful with models such as neural nets and kernel machines, but until now it remained an open problem with decision trees, which deﬁne a nondifferentiable function.
We have shown how to reformulate the problem in a way that is amenable to iterative optimization.
By introducing auxiliary variables, we isolate the difﬁcult part (the tree optimization), and all the algorithm needs to do is to ﬁt a regression tree in alternation with solving a sparse linear system that smoothes the predicted labels.
The tree
[Figure]
ﬁtting can be done reliably and efﬁciently using the Tree Alternating Optimizing (TAO) algorithm, which also allows us to use more powerful trees, such as sparse oblique trees.
Our experimental results demonstrate that the algorithm can train accurate and interpretable decision trees even in extreme label scarcity situations.
Our framework can be generalized to other machine learning models, such as ensembles of trees, which will be the future direction of our work
Acknowledgments and Disclosure of Funding
Work partially supported by NSF awards IIS–1423515 and IIS–2007147.