On Margins and Generalisation for Voting Classifiers Anonymous Author(s)
AffiliationAddress email
Abstract
We study the generalisation properties of majority voting on finite ensembles ofclassifiers, proving margin-based generalisation bounds via the PAC-Bayes theory.
These provide state-of-the-art guarantees on a number of classification tasks.
Ourcentral results leverage the Dirichlet posteriors studied recently by Zantedeschiet al.
[1] for training voting classifiers; in contrast to that work our bounds apply tonon-randomised votes via the use of margins.
Our contributions add perspectiveto the debate on the “margins theory” proposed by Schapire et al.
[2] for thegeneralisation of ensemble classifiers.
Introduction
Weighted ensemble methods are among the most widely-used and effective algorithms known inmachine learning.
Variants of boosting [3, 4] are state-of-the-art in a wide variety of tasks [5, 6]and methods such as random forest [7] are among the most commonly-used in machine learningcompetitions [see, e.g. , 8, 9], valued both for their excellent results and interpretability.
Evenwhen these algorithms do not directly produce the best learners for a task, the best performance incompetitions is often obtained by an ensemble of “strong learners”—the output of a collection ofdifferent algorithms trained on the data—contrasted to the weak learners usually considered in theensemble learning literature.
Among the oldest ideas to explain the performance of ensemble classifiers, and machine learningmethods in general, is the concept of margins.
First introduced to analyse the Perceptron algorithm [10], margins relate closely to the idea of confidence in predictions in ensemble learning, witha large margin implying that a considerable weighted fraction of voters chose the same answer.
Thiswas first leveraged to obtain early margin-based generalisation bounds for ensembles by Schapireet al.
[2], in an attempt to understand the excellent generalisation of boosting, a surprising result givenclassical Vapnik-Chervonenkis theory.
This “margins theory” was explored further in a number ofworks [11–13] and is among the leading explanations for the success of such methods and boostingin particular.
The same thread of margin bounds for ensemble methods has also been taken up in parallel in
PAC-Bayes theory by Langford and Seeger [14], Biggs and Guedj [15].
PAC-Bayes provides anatural framework both for deriving margin bounds, and for considering ensemble methods in general,particularly majority votes where the largest-weighted ensemble prediction is taken.
Within theframework, the weightings are typically considered as the parameter of a categorical distributionover individual voters.
PAC-Bayes theorems [see the comprehensive surveys of 16, 17] then directlyprovide generalisation bounds for the performance of this “randomised” proxy for the majority vote,a.k.a.
Gibbs classifier.
These can then be de-randomised by such margin-based techniques, or througha variety of oracle bounds [18–21], motivating new learning algorithms [22–28].
Uniquely among PAC-Bayesian approaches, Zantedeschi et al.
[1] instead consider Dirichlet distributions over the voters.
Any sample from this distribution already implies a vector of voting weights,and it is on the performance and optimisation of these “stochastic majority votes” they primarily focus.
As an aside, they provide an oracle result which allows their bounds to be de-randomised, butthis introduces an irreducible factor such that the bound on the true fixed vote can never be less thandouble that of the stochastic version.
It also neglects to leverage the generally high confidence ofpredictions obtained by their algorithm.
Our contribution.
By combining tools from margin bounds and the use of Dirichlet majority votes,we provide a new margin bound for non-randomised majority votes.
This is in contrast to Zantedeschiet al.
[1] which primarily considers stochastic majority votes.
Our bound empirically comparesvery favourably to existing margin bounds and in contrast to them are applicable to multi-classclassification.
Remarkably, our empirical results are also sharper than existing PAC-Bayesian ones,even when the algorithm optimising those bounds is used.
Our primary tool is a new result relating the margin loss of these stochastic votes to the misclassification loss of the non-randomised ones in a surprisingly sharp way.
This tool can additionallybe utilised alongside a further idea from Zantedeschi et al.
[1] to obtain an alternative form of thebound which is more amenable to optimisation.
Through this work we provide further support tothe margins theory for ensembles, showing that near-sharp bounds based on margins alone can beobtained on a variety of real-world tasks.
Outline.
The rest of this section introduces the problem setup, notation and summarises main results.
Section 2 provides background on PAC-Bayes, Dirichlet majority votes and margin bounds, relatingthem to our new results.
Section 3 states and summarises our new theoretical results, giving the mostrelevant proofs (all remaining proofs are deferred to appendices).
Section 4 empirically evaluatesthese new results before we conclude with an overall discussion in Section 5.
1.1 Notation and setting
Majority voting algorithms combine the predictions of a finite set of “base” classifiers, H , from X to
Y =
[ c ] := { 1 , . .
, c } .
The classifiers h i ∈ H take the form h
i : X → Y for i ∈
[ d ] so that |H| = d .
Majority votes consider as set of weightings θ in ∆ d , the simplex, and return the highest-weightedoverall prediction.
Using the indicator function I A of a set A , this is expressed as
[Equation]
We are primarily interested in learning a weighting θ with small misclassification risk (and guaranteesof this) based on a sample S ∼ D m , where D ∈ M +1 ( X × Y ) is the data-generating distributionand m ∈ N + the sample size.
We let M +1 ( A ) denote the set of probability measures on a set A .
For h ∈ H the misclassification loss is ℓ
( h, x, y ) :
= I h ( x ) ̸
= y , the misclassification out-of-samplerisk is L 0 ( h ) :
=
E ( x,y )
∼D ℓ 0 ( h, x, y ) and a hat denotes the in-sample estimate of this quantity,ˆ L 0 ( h ) :
= E ( x,y ) ∼ Uniform( S ) ℓ 0 ( h, x, y ) .
In a slight abuse of notation we will also often write the riskof the majority vote L 0 ( θ ) =
L 0 ( f θ ) and similarly for its empirical counterpart.
The margin of majority vote f θ on example ( x, y ) is derived from the minimal gap between the totalweight assigned to the true class y and to any other predicted class:
[Equation]
The corresponding margin loss is ℓ γ ( θ , x, y ) :
=
I M ( θ ,x,y ) ≤ γ for margin γ ≥ 0 , with the corresponding in-sample and out-of-sample risks notated as ˆ L γ ( θ ) and L γ ( θ ) respectively.
1.2 Overview of results
Our main result is a margin bound of the following form: with high probability ≥ 1 − δ over thesample, simultaneously for any θ ∈ ∆ d and K > 0 ,
[Equation]
where D Dir ( α , β ) is the KL divergence between Dirichlet random vectors with parameters α andβ , with 1 a vector of ones implying a uniform Dirichlet prior distribution on the simplex.
The term e − Kγ 2 is a de-randomisation penalty.
The parameter K is chosen freely in an arbitrary data-dependentway to balance the requirements of the different terms: it must be large enough to decrease thisexponential term, while too-large a parameter increases the KL divergence from the uniform prior.
This result is surprisingly strong; in particular there is no dependence on the dimensionality ( i.e. ,number of voters d ) in the exponential term, an advantage discussed further in Section 3.2.
In Equation (1), ˆ L γ ( θ ) is the 0-1 valued γ -margin loss which enables comparison with existingmargin bounds for trained weighted ensembles.
We further consider a second scenario, where thegeneralization bound is also used to train the model itself.
We note that the γ -margin loss ˆ L γ ( θ )appearing in Equation (1) has null gradients, so the bound cannot be directly optimised by gradientdescent.
To rectify this we also prove a variation of the bound, replacing the above loss by itsexpectation under a Dirichlet stochastic vote, E ξ ∼ Dir( K θ )
ˆ L γ ( ξ ) , which is bounded in differentiableclosed form to give an alternative, optimisation-friendly bound.
In our evaluations we focus on these two complementary scenarios, obtaining state-of-the-art empirical results.
Across different scenarios and tasks our results outperform both existing margin bounds(including a sharpened version of the result from Biggs and Guedj [15] which may be of independentinterest), and PAC-Bayes bounds, even when it is not used as the objective.
Further, in contrast toexisting margin bounds our results also hold for multi-class majority votes.
Background2.1 PAC-Bayes bounds
PAC-Bayes bounds are among the tightest known generalisation bounds, as for example the onlyframework in which non-vacuous generalisation bounds for neural networks have been obtained [seee.g. 29–36].
However, unlike many other such bounds they usually apply to randomised Gibbs(-like)prediction functions rather than deterministic ones.
These are typically re-drawn for every newtest evaluation.
Thus a high-probability bound is obtained on the expectation of the risk w.r.t.
the
PAC-Bayes posterior Q , with the complexity of Q ∈ M +1 ( H ) appearing in the bound in terms of a
Kullback-Leibler (KL) divergence from a pre-chosen PAC-Bayes prior P ∈ M +
( H )
[which is notrequired to be a true prior in the Bayesian sense – see the discussion in 16].
A particularly sharp [asdiscussed in 37] and widely-used result is given in Theorem 1, valid for any bounded loss function ℓwith values in [0 , 1] .
Theorem 1 (Seeger et al.
[38], Maurer [39]) .
For any D ∈ M +1 ( X × Y ) , m ∈ N + , prior P ∈
M +1 ( H ) and δ ∈ (0 , 1) , with probability ≥ 1 − δ over S ∼ D m , simultaneously for all Q ∈ M +
( H )
[Equation]
where the generalised inverse kl − 1 ( u, c ) :
= sup { v ∈ [0 , 1] : kl( u, v ) ≤ c } and kl( u, v ) :
=u log uv + (1 − u ) log 1 − u 1 − v is a KL divergence between Bernoulli random variables.
The above bound uses the inverse small-kl function which will be seen in our later results anda number of pre-existing ones.
To lend intuition we note that kl − 1 ( u, c ) ∈
O ( u + c ) , giving
Equation (1) from Theorem 2 when using a uniform prior.
The following upper bounds are alsouseful: kl − 1 ( u, c ) ≤ u + (cid:112) c/ 2 giving “slow-rates” and kl − 1 ( u, c ) ≤ u + √ 2 cu + 2 c .
From thiswe can see that when the loss ˆ L → 0 then the overall rate improves to O (1 /m ) , so the small-klformulation interpolates between the traditional fast and slow rate regimes of learning theory.
2.2 Margin bounds
In the learning theory literature there exists a rich tradition of using the concept of a margin, whichquantifies the confidence of predictions, to explain generalisation.
This is particularly evident in thecase of voting algorithms such as boosting, where traditional Vapnik-Chervonenkis based techniquespredict classical overfitting which is not ultimately observed.
The “margins theory” was developed by
Schapire et al.
[2] to explain this discrepancy.
By considering the weightings θ as the parameter of acategorical distribution, they proved a bound of the form (holding with probability greater than 1 − δ over the sample, as for all bounds in this section)
L 0 ( θ ) ≤ L γ ( θ [Equation] + ˜ O (cid:16)
[Equation]
m (cid:17) . Although there wasinitially some debate about the validity of the theory [40], eventually Gao and Zhou [12, Theorem 4]provided the following improved bound which further supported that a large-margin voting classifiercould generalise: simultaneously for any γ > (cid:112) 2 /d and θ ∈ ∆ d ,
[Equation]
More recently, a similar bound (proved through a PAC-Bayesian method based on Seeger et al.
[38])was proved in Biggs and Guedj [15, Theorem 8].
Here we give a bound provided as an intermediatestep in their proof that is strictly (and empirically considerably) sharper than their final result: for anyfixed margin γ > 0 , simultaneously for any θ ∈ ∆ d
[Equation]
Since γ ∈ (0 , 12 ) for non-vacuous results, a union bound argument can be used to extend the above tofixed-precision γ , and this result has the advantage of being valid for small γ as are often observedempirically.
Our contributions.
Firstly we mention the smaller contribution of the improved form of the boundfrom Biggs and Guedj [15] given in Equation (3); a proof is given in Appendix B alongside furtherrefinements and evaluation.
However we show that in many cases even this improved version and
Equation (2) give weak or vacuous results.
As a result of this weakness (and thus perhaps null resultfor the margins theory applied to voting classifiers)
we present a new margin bound in Theorem 2based on Dirichlet distributions as a theoretical intermediate step.
This is also valid in the multi-classcase, unlike the above results which are only for binary classification.
Empirically the bound isobserved to give an enormous improvement in tightness than the existing margin bounds and in somecases is near-sharp.
2.3 Dirichlet stochastic majority votes
In most results from the PAC-Bayes framework, and in the proof of the existing results given in
Section 2.2, the majority vote weightings θ are considered the parameters of a categorical distributionover voters.
Zantedeschi et al.
[1] instead consider PAC-Bayesian bounds (specifically, Theorem 1)applied to a hypothesis class of majority votes of the form f ξ , where ξ ∼ Dir( α ) is drawn from a
Dirichlet distribution with parameter α .
This distribution has mean E ξ = α / (cid:80) di
[-P di =
-]=1 α i with a largersum (cid:80) di
=
α i giving a more concentrated or peaked distribution (see Appendix A for more details).
Since ξ is randomised, the bounds from Zantedeschi et al.
[1] apply to “stochastic majority votes”rather than the more typical deterministic ones we consider here.
However, the use of such Dirichletdistributions over voters in the PAC-Bayes bounds rather than the more usual categorical ones is amajor step forward as it allows the correlation between voters to be more carefully considered.
Wewill utilise and de-randomise these as a stepping stone to ones for deterministic predictors f θ directly.
As is common in the PAC-Bayes literature, Zantedeschi et al.
[1] use their new bound as an optimisation objective to obtain a new algorithm, here using stochastic gradient descent.
The boundwith Dirichlet posterior obtained directly from Theorem 1 includes the expected misclassificationloss with respect to the Dirichlet parameters, E ξ ∼ Dir( α ) ℓ ( f ξ , x, y ) , which has null gradient for anysampled ξ .
They therefore additionally upper bound this term by the differentiable closed form
[Equation]
where I z ( a, b ) is the regularised incomplete beta function, which has a sigmoidal shape.
Theinequality is sharp in the binary classification case, and is used in the training objective and finalevaluation of their method.
As an aside, Zantedeschi et al.
[1] also proved an oracle bound whichallows their result to be de-randomised, but this introduces a irreducible factor of two.
Our contributions.
Firstly, we provide a new margin bound for majority vote algorithms utilising
Dirichlet posteriors as a theoretical stepping stone.
We show that this bound gives sharper bounds on the misclassification loss than the bound from Zantedeschi et al.
[1], doing better than the irreduciblefactor, even when applied to the output of their algorithm.
We show further that the bound is alsotighter when applied to the outputs of other PAC-Bayes algorithms derived from “categorical”-typeposteriors.
Finally, we give an altered form of the bound involving the expectation of the marginloss E ξ ∼ Dir( α ) ℓ γ ( f ξ , x, y ) and a result analogous to Equation (4) for this case.
Through this we areable to obtain a new PAC-Bayes objective which is compared to existing PAC-Bayes optimisationmethods.
Main results
Our main results use the idea of Dirichlet stochastic majority votes from Zantedeschi et al.
[1] as anintermediate step to prove new margin bounds for deterministic majority votes.
In this section, firstwe give our main result in Theorem 2 and discuss further.
In Section 3.1 we give an alternative boundobtained by a very similar method which is more amenable to optimisation, and we provide proofsfor these results in Section 3.2.
The central step in these proofs is in constructing a proxy Dirichlet distribution ξ ∼ Dir( K θ ) over voters, the loss of which is bounded à la PAC-Bayes, and de-randomised using margins to obtain boundsdirectly for f θ .
The primary complexity term appearing in our bounds is therefore D Dir ( K θ , β ) ,the KL divergence between Dirichlet distributions with parameters
K θ and β
respectively.
As with
PAC-Bayes priors, β can be chosen in arbitrary sample-independent fashion, but we typically chooseit as a vector of ones, giving a uniform distribution on the simplex as prior as in Equation (1).
Thebounds also involve a de-randomisation penalty of O ( e − Kγ 2 ) where γ is the margin appearing in theloss; this term upper bounds the difference between our randomised proxy ξ and its mean θ and getssmaller with K as the distribution concentrates tightly around its mean.
This parameter K can beoptimised in any data-dependent way to obtain the tightest final bound.
Theorem 2.
For any D ∈ M +1 ( X × Y ) , m ∈ N + , margin γ > 0 , δ ∈ (0 , 1) , and prior β ∈ R d + ,with probability at least 1 − δ over the sample S ∼ D m simultaneously for every θ ∈ ∆ d and K > 0 ,
[Equation]
Theorem 2 differs from the existing margin bounds of Equations (2) and (3), and Schapire et al.
[2]in a specific and significant way, with θ appearing not only in the loss function ˆ L γ ( θ ) , but also inthe KL complexity term.
Empirically we find our bound to be an improvement but it is possibleto generate scenarios where the pre-existing bounds are non-vacuous while ours is not, since the
KL divergence is unbounded for certain choices of θ , for example when one of the components isexactly zero.
This difference arises because the existing bounds all use the idea of a categoricaldistribution with parameter θ in their proofs (which has KL divergence from a uniform prior upperbounded by log d ), while we use a Dirichlet.
This gains us the surprisingly tight de-randomisationresult (Theorem 4) used in all proofs.
3.1 PAC-Bayes bound as objective
We note here that it is non-trivial to directly obtain a training objective for optimisation from
Theorem 2, due to the non-differentiability of the margin loss ˆ L γ ( θ ) .
Therefore, in order to compareresults with a wide variety of methods that optimise PAC-Bayes bounds [including those used by 1,as baselines], we obtain a relaxed and differentiable formulation in Theorem 3 for direct optimisation.
Theorem 3.
Under the conditions of Theorem 2 the following bound also holds
[Equation]
Using the incomplete Beta function I z ( a, b ) we also have the following result, which is sharp in thebinary classification case,
[Equation]
Theorem 3 has a stronger PAC-Bayesian flavour than Theorem 2, with an expected loss under somedistribution appearing (complicating the final optimisation of K ), while Theorem 2 takes a form muchcloser to that of a classical margin bound.
The second part of the result is analogous to Equation (4)used by Zantedeschi et al.
We combine both parts to calculate the overall bound in closed formand obtain gradients for optimisation.
3.2 Proof of main results
The proof of Theorems 2 and 3 essentially follow from applying a simple PAC-Bayesian bound incombination with the key Theorem 4 below.
In some sense this is our most important and novelresult.
Our whole approach is largely motivated by its surprising tightness; in particular there isno dependence on the dimension, which is avoided by careful use of the aggregation property ofthe Dirichlet distribution.
This surprise arises because to obtain a tightly concentrated Dirichletdistribution on ξ ∼ Dir( α ) , the concentration parameter K = (cid:80) di[-=
-]
=1 α i must grow linearly with thedimension.
In fact, even a uniform distribution (which will be less peaked than our final posterior)has (cid:80) di
=1 α i =
d , so the de-randomisation step is effectively very cheap in higher dimensions.
Theorem 4.
Let θ ∈ ∆ d and K > 0 .
Then for any γ > 0 and ( x, y ) ,
[Equation]
For our proofs we first recall the aggregation property of the Dirichlet distribution: if ( ξ 1 , . . . , ξ d ) ∼
Dir(( α 1 , . . . , α d )) , then ( ξ 1 , . . . , ξ d − 1 + ξ d ) ∼ Dir(( α 1 , . . . , α d − 1 + α d )) .
We further note thefollowing crucial concentration-of-measure result.
The aforementioned lack of dimensionality in
Theorem 4 is possible because Theorem 5 depends only on (cid:80) di
=1 α i , and this value is unchangedby aggregation, which avoids the dimension dependence that would otherwise be introduced by therequirement ∥ u ∥ 2 = 1 below.
Theorem 5 (41) .
Let X ∼ Dir( α ) , t > 0 , and u ∈ R d with ∥ u ∥ 2 = 1 .
Then
[Equation]
Proof of Theorem 2 and Theorem 3.
The proof of our main results is completed by applying the PACBayes bound Theorem 1 with the γ -margin loss to a Dirichlet prior and posterior with parameters
βand K θ respectively.
Substituting the first part of Theorem 4 gives the first part of Theorem 3, andadditionally substituting the second part and re-scaling γ → γ/ 2 gives Theorem 2.
For the second part of Theorem 3, define w =
i : h i ( x ) = y } for fixed ( x, y )
so W :=
(cid:80) i ∈ w ξ i ∼ Beta (cid:0)(cid:80) i ∈ w α
i , (cid:80) i/ ∈ w α i (cid:1) by the aggregation property of the Dirichlet distribution.
Then
E ξ ℓ γ ( ξ , x, y ) ≤ E ξ (cid:26)
[Equation]
γ (cid:27)
[Equation]
 (cid:88)
[Equation]
(cid:88) i/
[Equation]
w α
[Equation]


[Equation]
 (cid:88) j/ [Equation] w α j
[Equation]
(cid:88) i ∈ w α j   using I a
i − max
[Equation] a j ≥ 12 − γ for a ∈ ∆ c (with equality for c = 2classes), and that I z ( a, b ) is the CDF of a Beta distribution with parameters ( a, b ) .
Proof of Theorem 4.
Define γ 2 > γ 1 such that γ := γ 2 − γ 1 , and α = K θ .
From the trivialinequality I x ∈ A − I x ∈ B ≤ I x ∈
A
I x/ ∈ B we derive
[Equation]
≤ P ξ ∼ Dir( α )   (cid:88)
i : h i ( x )=
y ξ i − (cid:88)
i : h i ( x )=
k ξ i − (cid:88) i : h i ( x )=
y
θ i + (cid:88) i : h i ( x )= k θ i > 2 γ   where in the last inequality we set k = argmax k ̸ =
y (cid:80) i : h i ( x )= k θ i , and use that max j (cid:80) i : h i ( x )=
j θ i −max
j (cid:80) i : h i ( x )=
j ξ i ≤ max j (cid:80) i : h i ( x )=
j θ i − (cid:80)
i : h i ( x )=
k ξ i for any k .
We rewrite the above in vectorform (with inner product denoted u · v ) as
[Equation]
where by the aggregation property of the Dirichlet distribution ˜ ξ ∼ Dir( ˜ α ) with
[Equation]


T .
Applying Theorem 5 we obtain ∆ ≤ e − 4( (cid:80) i ˜
α i +1)
γ 2 =
e − 4( (cid:80) di[-=
-]
=1 α i +1)
γ 2 .
This gives the firstinequality by setting γ 1 = 0 , γ 2 = γ .
Setting γ 1 = γ, γ 2 = 2 γ and swapping θ and ξ gives an almostidentical proof (with some signs reversed) of the second inequality.
Empirical evaluation
In this section we empirically validate our results against existing PAC-Bayesian and margin boundson several classification datasets from UCI [42], LIBSVM 1 and Zalando [43].
Since our main resultin Theorem 2 is not associated with any particular algorithm, we use θ outputted from PAC-Bayesderived algorithms to evaluate this result against other margin bounds (Figure 1) and PAC-Bayesbounds (Figure 2).
We then compare optimisation of our secondary result Theorem 3 with optimisingthose PAC-Bayes bounds directly (Figure 3).
All generalisation bounds given are evaluated with aprobability 1 − δ =0 .
Further details not provided here including tabulated results, description ofdatasets, training mechanisms and compute are provided in Appendix C.
Strong and weak voters.
Similarly to Zantedeschi et al.
[1] we consider both using data-independentand data-dependent voters.
This brings our experimental setup in line with a common workflowfor machine learning practicioners: the training set is sub-divided into a set for training severaldifferent strong algorithms, and a second set on which the weightings of these are optimised.
Morespecifically, the weak voter setting, used only for binary classification, uses axis-aligned decisionstumps (denoted stumps ), with thresholds evenly spread over the input space ( 6 per feature and perclass).
The stronger voters (denoted rf ) are learned from half of the training data, while the other halfis used for evaluating and optimising the different generalisation bounds (note this reduces m ).
Thesetake the form of random forests [7] of M =10 trees optimising Gini impurity score on n 2 baggedsamples and √ d drawn features for each tree, with unbounded maximal depth.
Optimising γ and K in bounds.
In reporting margin bounds we optimise over a grid of marginγ values in (0 , 12 ) , and additionally over K for Theorem 2.
Since Theorem 2 and Equation (3) asstated require a fixed margin, we apply a union bound over the values in the grid, replacing δ in thesebounds with δ/N where N is the number of grid points.
Existing PAC-Bayes bounds.
We compare to state-of-the-art PAC-Bayesian bounds (and derivedalgorithms) for weighted majority vote classifiers: the First Order [18], the Second Order [21],
Binomial [20] (with the number of voters set to 100 ) and the two Chebyshev-Cantelli-based [28]empirical bounds from categorical-type Gibbs classifiers with parameter θ , and we refer to these as [Figure] HABERMAN TICTACTOE
ADULT
CODRNA
MUSHROOMS [Figure] f [Figure]
+ o
[Figure]
r
[Figure]
t e s t
[Figure]
e rr G Z B G + o u r
s t e s t - e rr [Figure]
e c i s i o n S t
u m p s
G Z B G + o u r
s
t
e s t - e rr G Z B G + o u r
s t
e s t - e rr G Z B G + o u r
s t
e s t - e rr G Z B G + o u r
s t
e s t - e rr G Z B G + o u r
s t
e s t - e rr G Z B G + o u r
s t
e s t - e rr G Z B G + o u r
s t
e s t - e rr G Z B G + o u r
s t
e s t - e rr
[Figure]
HABERMAN
[Figure]
Bin or f2 ( original bound ), and test errors.
For each column grouping, θ is the output from optimising the corresponding PACBayes bound for rf on the given dataset.
FO , SO , Bin , CCPBB and CCTND respectively (more details are given in Appendix C).
We denote byf2 the factor two bound derived in Zantedeschi et al.
[1, Annex A.4] from Dirichlet majority votes.
All prior distributions for PAC-Bayes bounds, including ours, are set to uniform.
We also refer by thesame names to the outputs of optimising these bounds with stochastic gradient descent; details ontraining and initialisation are given in Appendix C.
Description of figures.
In Figure 1 we compare Theorem 2 with the existing margin bound of
Equation (2) and the improved Biggs and Guedj [15] bound given in Equation (3).
Since Equation (3)is strictly better than the original result and the latter was vacuous in almost all cases considered (see
Appendix B), we do not include it.
All datasets are for binary classification as the existing results onlycover this case, and the θ values considered are the outputs of either the FO- or f2-optimisation usingeither the weak or
the strong voters described above.
Figure 2 extends this evaluation of Theorem 2to improve generalisation results, by applying it to the models optimised with the PAC-Bayes bounds
FO , SO , Bin and f2 as objective.
In this case, we consider both binary and multiclass datasets.
In
Figure 3 we directly compare the outputs of optimising state-of-the-art PAC-Bayesian bounds withour optimisation-ready variant result Theorem 3.
These experiments were carried out on strong voters,as standard in the literature [ e.g. 21, 26, 28].
Discussion and conclusion
We observe overall that in many cases the existing margin and PAC-Bayes bounds are insufficient toexplain the generalisation observed, while our new bound is consistently tight, and sometimes sharp( i.e. it approaches the true test error).
PROTEIN
FASHION-MNIST
HABERMAN
TICTACTOE
ADULT
F O S O B i n
CC P BB CC T N D f 2 o u r s 0.
MNIST
F O S O B i n
CC P BB CC T N D f 2 o u r s
CODRNA
F O S O B i n
CC P BB CC T N D f 2 o u r s
PENDIGITS
F O S O B i n
CC P BB CC T N D f 2 o u r s
SENSORLESS
F O S O B i n
CC P BB CC T N D f 2 o u r s
MUSHROOMS bound test error
Figure 1 demonstrates that existing margin bounds can be insufficient to explain the generalisationobserved, which could be construed as a null result for the “margins theory”.
However, our newbound obtains empirically very sharp results in almost all cases, reaffirming to the theory.
Note thatdue to the non-convexity of our bound, the reported values are local minima and can potentially beimproved by applying a thorougher search for the optimal K , still giving a similarly valid bound.
Forinstance, simply by enlarging the search space for K our bound drops to 0 .
36 ± 0 .
10 on ADULT withdecision stumps as voters, beating existing bounds also in this setting. Unlike the existing results, θalso arises in the complexity (KL divergence) term and so the bound is not equally tight for every θ atfixed margin loss.
Further examination of this property could add additional nuance and perspectiveto the theory.
When comparing to existing PAC-Bayes bounds in Figure 2, remarkably Theorem 2 is always tighterthan just using the bound which is being optimised.
We speculate that this arises partially due to theirreducible factors appearing in those bounds; for example the FO or f2 bounds can never be tighterthan twice the train loss of the associated Gibbs classifier, while ours has no such limitation.
Thisresult is quite valuable as it demonstrates that Theorem 2 can be readily used in an algorithm-freemanner: the choice of learning algorithm is up to the practitioner, but the bound will then oftenprovide an excellent guarantee on the obtained weights θ .
Finally, in Figure 3, our optimisation-friendly variant bound Theorem 3 is seen to be competitive interms of test error while giving an improved-or-equal final bound on all datasets.
When consideringthe less-common setting of binary stumps (see Appendix C)
we found that sometimes this objectiveconverged to a sub-optimal local minimum.
We speculate that this arises due to the highly non-convexnature of the objective combined with a strong K -inflating gradient signal from the O ( e − Kγ 2 ) term.
Thus future work to improve these results even further could start with the use of the quasi-convexsmall-kl relaxation from Thiemann et al.
We note however that this is overall less importantthan our main results, as both our bounds are still extremely tight when used in an algorithm-freeway and applied to the output of another algorithm as discussed above.
Overall, we note that in many cases (a majority in Figure 2) our main bound of Theorem 2 is veryclose to the test set bound and thus cannot actually be improved any further, with the problem ofproviding sharp guarantees based on the training data alone effectively solved in many cases.
Conclusion.
We obtain empirically very strong generalisation bounds for voting classifiers usingmargins.
We believe these are highly relevant to the community, since voting-based classifiers andmargin-maximising algorithms are among the most popular and influential in machine learning.
Dirichlet majority votes have already obtained excellent results in the stochastic setting [1], butour new result in Theorem 4 showing they are well-approximated by their mean should open newdirections in the more conventional deterministic setting.
Our results also have practical relevance: for example, in the strong voter machine learning workflowdescribed above, instead of setting data aside as a test set, this data can be freed up to learn even stronger voters, since a strong out-of-sample ensemble guarantee can still be provided even without atest set.
In future work we hope to expand these results further to other (non-majority) voting schemes likethose with score-output voters [as in e.g. 2], and ensembles of voters with finite VC dimension
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See, e.g. discussion in Section (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
is of a primarily theoretical nature and therefore no negative societal impact is foreseen.
All data is from publicly-available datasets.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Included in the
theorem statements and setting introduction in Section 1.
(b) Did you include complete proofs of all theoretical results?
[Yes] Included primarily in the main paper with some extra result in the appendices.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4 and
Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] In Appendix C
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] In Appendix C
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Appendix C
(b) Did you mention the license of the assets?
[Yes] In Appendix C
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include links in Appendix C
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All data was publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We did not conduct any such research.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
We did not conduct any such research.
Did you the estimated hourly wage paid to participants and the total amountspent on participant compensation N/A
did not conduct any such research.
A Properties of the Dirichlet distribution
The Dirichlet measure has probability density function w.r.t.
Lebesgue measure given by: [Equation] where B ( α ) is the multivariate Beta function, [Equation]
The mean of a Dirichlet is E ξ ∼ Dir( α ) ξ
= α / (cid:80) di
=1 α i .
The KL divergence between two Dirichlet distributions is the following, given in e.g. Zantedeschiet al.
[Equation]
B Additional details on margin bounds
Here we first note the original result from Biggs and Guedj [15] that is adapted in Equation (3); sincethis is obtained by applying an upper bound to the inverse small-kl and an additional step, it is strictlylooser than the result we give in Equation (3).
Biggs and Guedj [15] also uses a dimension doublingtrick to allow negative weights (as they consider only the binary case), which we remove here toreplace the factor log(2 d ) by log d .
Theorem 6.
For any margin γ > 0 , δ ∈ (0 , 1) , sample size m ∈ N , each of the following resultsholds with probability at least 1 − δ over the sample S ∼ D m simultaneously for any θ ∈ ∆ d ,
[Equation]
where C := 2 log(2 /
[Equation]
B.1 Definition of the margin
We here note that the definition of the margin given in Gao and Zhou [12] and Biggs and Guedj [15]is slightly different from our own, leading to a scaling of the margin definition by a factor of one-half.
We show this below.
Both the above papers consider prediction functions like F θ ( x ) = (cid:80) di
{+=
 θ i h+}
i ( x ) with output set
Y = { +1 , − 1 } .
The functions h i ( x ) can be positive or negative.
The margin is defined as asyF theta ( x ) .
We translate this into our equivalent but scaled version as follows:
[Equation]
which is double the margin as we define it.
Thus ℓ γ ( θ , x, y ) =
I M ( θ ,x,y ) ≤ γ
=
I yF θ ≥ 2 γ and thecondition on the margin yF ( x ) ≥ (cid:112) 8 /d
given in Gao and Zhou [12] translates to M ( θ , x, y ) > 565 (cid:112)
2 /d as we give.
B.2 Proof of Theorem 6 and Equation (3)
For completeness we provide here short proofs of Equation (3) and Theorem 6.
The central propositionused in Biggs and Guedj [15] to prove their margin bound and these results for voting algorithms isthe following, proved implicitly there and here adapted to our setting.
Theorem 7 (Biggs and Guedj [15]) .
Let θ ∈ ∆ d and define ρ = Categ( θ ) and i ∼ ρ T as T i.i.d.samples from ρ indexed by j ∈ [ T ] .
Then for any γ > 0 , T ∈ N + and ( x, y ∈ { +1 , − 1 } ) ,
[Equation]
where we have defined the margin loss for a sum of Categoricals as ℓ C γ ( i , x, y )
=
I yT −
[Equation] Tt =1 h it ( x ) ≤ γ . Proof of Equation (3) .
We apply the PAC-Bayes bound Theorem 1 to ℓ Cγ with ρ T as defined in
Theorem 7 and a uniform prior of the same form, π T .
We then substitute the results from Theorem 7to show that
[Equation]
With a uniform prior, KL( ρ T , π T ) =
T D Cat ( θ, d − 1 1 ) ≤ T log d .
Substitution of this upper boundand T = (cid:6) 2 γ − 2 log m (cid:7) gives the result.
Proof of Theorem 6.
Beginning with Equation (3), we relax the ceiling using γ ≤ 12 and m ≥ 2 for anon-vacuous bound to obtain
[Equation]
with C := 2 log(2 /δ )
+ 194 γ − 2 log d log m .
Then using the small-kl upper bound kl − 1 ( u, c ) ≤u + √ 2 cu + 2 c we have
[Equation]
which is the result given.
Proof of Theorem 7.
Using the same method as the beginning of the proof of Equation (6),
[Equation]
In the last line we used Hoeffding’s inequality for a sum of T random variables bounded in [ − 1 , 1] .
The other side follows using an identical method with the margin losses reversed.
B.3 Further improvement to the bound
A question which naturally arises from looking at the proof of Equation (3) and Theorem 6 is whetherwe can do better by choosing T in a more optimal way, rather than just setting it to (cid:6) 2 γ − 2 log m (cid:7) .
We thus prove a bound here which is valid for the optimal choice of T ; in practice this is seen tobe slightly tighter than Equation (3), although the improvement from Theorem 6 to that result is fargreater.
For any θ ∈ ∆ d with probability at least 1 − δ over the sample,
[Equation]
A slightly weaker version of this result, with an extra m − 1 log(2 √ m ) term, can be proved from
[Equation]
which is shown in the proof of Equation (3).
We note that the optimal T depends on the data onlythrough ˆ L γ ( θ ) ∈ { 0 , m − 1 , 2 m − 1 , . . .
The last possibility gives a trivial bound.
A union boundover the m non-vacuous possibilities gives Equation (7) with the extra logarithmic factor.
In order to remove this term, we use a slightly more sophisticated argument applied to a different PACBayes bound (Theorem 8) given below.
This result uses the function (defined for C > 0 , p ∈ [0 , 1] )
[Equation]
which relates to the small KL (Theorem 9).
Theorem 8 (Catoni [45]) .
Given data distribution D on X × Y , prior P ∈ M 1 ( H ) , C > 0 andδ ∈ (0 , 1) , the following hold each with probability ≥ 1 − δ over S ∼ D m , for all Q ∈ M 1 ( H )
[Equation]
Theorem 9 (Germain et al.
[46], Proposition 2.1) .
For any 0 ≤ q ≤ p < 1 ,
[Equation]
Proof of Equation (7) .
We substitute Theorem 7 into Theorem 8 with the categorical loss and auniform prior, π T .
and KL upper bound as in the above proof.
as we obtain for any data-independent
C > 0 , T ∈ N + , γ > 0 that
[Equation]
where k :=
m ˆ
L γ ( θ ) is the number of margin errors.
Since the only quantity on the left hand side in this bound unknown before we see data is the valueof k , there exists a C k dependent on the value of k that optimises the bound, and a T k that dependson this pair.
Since there are only m such values giving non-vacuous bounds ( k = m is triviallyvacuous), we can apply a union bound over all these bounds with δ = δ/m to give the following withprobability ≥ 1 − δ :
[Equation]
Applying the inversion of Theorem 9 gives the second result.
B.4 Comparison of margin bounds
In Figure 4 we compare the various bounds given above in a non-experimental way, fixing the marginloss ˆ L γ to a particular value and seeing how the bounds change if that value of the loss is achievedfor different values of the margin γ ∈ (0 , 0 . 5) .
Since (uniquely among the bounds), the value of θappears in our bound Theorem 2, we show three different sampled possible values, drawn uniformlyfrom the simplex.
[Figure]
[Figure]
B o un d V a l u e
[Figure]
[Figure]
[Figure]
B o un d V a l u e
[Figure]
The results for “categorical”-based bounds demonstrate that the refined bounds Equations (3) and (7)are much tighter that the result as given in Theorem 6 by Biggs and Guedj [15].
Both these refinementsare also tighter than Equation (2) from Gao and Zhou [12].
We used Equation (3) in the main paperbecause it is closer to an exiting result (as it appears in the proof from 15), and is not much worse thanthe refinement Equation (7), particularly when compared to our far stronger new result Theorem 2.
This figure also shows that, at least for some values of θ , this new bound can be far tighter than allthe existing bounds.
One interesting facet of this is that the bound is improved very little for γ abovea certain point, quite a different behaviour to the other bounds.
Empirically this was seen too in ourother experiments, with the optimised γ often being quite small.
Of course, for some values of θthis bound will be weaker, but we observe the same kind of results in our main experimental results,where this is a learned value.
C Additional experimental details and evaluations
Dataset descriptions.
We provide the description of the classification datasets considered in our empirical evaluation.
Haberman (UCI) prediction of survival of n = 306 patients who had undergone surgery fromd = 3 anonymized features.
TicTacToe (UCI) determination of a win for player x at TicTacToe game of any of the n = 958board configurations ( d = 9 categorical states).
Mushrooms (
UCI) prediction of edibility of n = 8 ,
mushroom sample, given their d =
categorical
features describing their aspect.
Adult (LIBSVM a1a) determining whether a person earns more than 50K a year ( n = 32 , 561people and d = 123 binary features).
CodRNA (LIBSVM) detection of non-coding RNAs among n = 59 , 535 instances and fromd = features.
Pendigits (UCI) recognition of hand-written digits ( 10 classes, d = 9 features and n = 12 , 992 ).Protein
(LIBSVM) d = 357 features, n = 24 , 387 instances and 3 classes.
Sensorless (LIBSVM) prediction of motor condition ( n = 58 , 509 instances and 11 classes),with intact and defective components, from d = 48 features extracted from electric current
drive signals.
MNIST (LIBSVM) prediction of hand-written digits ( n = 70 , 000 instances and 10 classes)
from d = 28 × 28 gray-scale images.
Fashion-MNIST (Zalando) prediction of cloth articles (
In all experiments, we convert all categorical features to numerical using an ordinal encoder and westandardize all features using the statistics of the training set.
Baseline descriptions.
We report the generalization bounds of the literature used for trainingweighted majority vote classifiers in our comparison.
We additionally note: Cat( θ ) the categorical distribution over the base classifiers (with θ i the weight associated to voter h i ∈ H ),and D Cat ( θ ) , π ) the KL-divergence between two categorical distribution with parameters θand π ; ℓ TND ( h, h ′ , x, y ) :
= I h ( x ) ̸
=
y
∧
h ′ ( x ) ̸
=
y
the tandem loss proposed in Masegosa et al.
[21]and ˆ L TND ( h, h ′ ) :
=
E ( x,y ) ∼ Uniform( S ) ℓ TND ( h, h ′ , x, y ) its in-sample estimate; ℓ Bin ( θ , N, x, y ) :
=(cid:80)
Nk
= N 2 (cid:0) Nk (cid:1) M
( θ , x, y ) k (1 − M ( θ , x, y ))
( N − k ) the probability that among N voters randomly drawnfrom Cat( θ ) at least N 2 of them are incorrect, as defined in Lacasse et al.
{+• First Order [FO, 18]:For any D ∈ M +
+}
( X × Y ) , m ∈ N + , δ ∈ (0 , 1) , and prior π ∈ ∆ d , with probability atleast 1 − δ over the sample S ∼ D m simultaneously for every θ ∈ ∆ d ,
[Equation]
0 ( θ )
[Equation]
kl −
[Equation]
• Second Order [SO, 21]:
For any D ∈ M +1 ( X × Y ) , m ∈ N + , δ ∈ (0 , 1) , and prior π ∈ ∆ d , with probability at
least 1 − δ over the sample S ∼ D m simultaneously for every θ ∈ ∆ d ,
[Equation]
• Binomial [Bin, 20]:
For any D ∈ M +1 ( X × Y ) , m ∈ N + , N ∈ N + , δ ∈ (0 , 1) , and prior π ∈ ∆ d , withprobability at least 1 − δ over the sample S ∼ D m simultaneously for every θ ∈ ∆ d ,
[Equation]
[Equation]
θ )
[Equation]
[Equation]
• Chebyshev-Cantelli tandem loss bound [CCTND, 28, Theorem 12];
• Chebyshev-Cantelli tandem loss bound with an offset [CCPBB, 28, Theorem 15];
• Dirichlet Factor-Two [f2, 1]:
For any D ∈ M +1 ( X × Y ) , m ∈ N + , δ ∈ (0 , 1) , and prior β ∈ R d + , with probability atleast 1 − δ over the sample S ∼ D m simultaneously for every θ ∈ ∆ d
and K > 0 ,
[Equation]
Optimisation of PAC-Bayesian bounds.
To optimize the baselines CCPBB and CCTND , we rely onthe code released by its authors 2 , with the Gradient Descent option and building random forests asdescribed in our main text.
When optimising the PAC-Bayesian bounds FO , SO , Bin , f2 and ours, weinitialize θ ’s to be uniform, i.e. θ i = 1 /d , and K = 2 .
We then optimise the posterior parameters of check Vale?the method ( α = Kθ for Dirichlet, and θ for Categorical distributions) with the Adam optimiser [47]with running average coefficients (0 . 9 , 0 . 999) , batch size equal to 100 and learning rate set to 0 .
All methods are run for a maximum of 100 epochs with patience of 25 epochs for early stopping anda learning rate scheduler reducing it by a factor of 10 with 2 epochs patience.
At each run of an algorithm, we randomly split a dataset into training and test sets of sizes 80% − 20%respectively, and optimise/evaluate the bounds only with the half of the training set that was not
[Figure]
used for learning the voters (in the case of data-dependent ones).
Note that we do not make use of avalidation set, as we use the risk certificates as estimate of the test error for model selection.
Finally,we report the value of Seeger’s "small-kl" bound of Theorem 1, even when a different type of boundhas been optimised ( e.g. for the CCPBB and CCTND baselines), and we average all results over 5different trials.
Margin bound comparison.
Given a pre-trained model, hence fixed θ and initial K init (which isdifferent from 1 . only for the models trained via Dirichlet bounds), we search for its optimal riskcertificate by evaluating a given bound at 1 , 000 values of γ , spaced evenly on a log scale withbase 10 and in the interval [10 − 4 , 0 . 5) .
For our margin bound, for each of these γ values we alsooptimise K ∈
[ K init , K init 2 16 ] using the golden-section search technique to obtain the tightestupper bound.
Notice that this does not add significant computational overhead to the search.
Also forthese experiments, the bounds are evaluated with the portion of training data that was not used forlearning the voters.
Compute.
All experiments were run on a virtual machine with 16 vCPUs and 128 Gb of RAM.
C.1 Additional results
In Figure 5, Figure 6 and Figure 7 we report the results from Figure 1, Figure 2 and Figure 3 in themain text.
Here we deploy a different scale per dataset so that they can be easily read, also whenthe bounds and test errors are very small.
Additionally, in Figure 8 we provide the test errors andrisk certificates obtained by optimising the generalization bounds with decision stumps as voters.
Although our certificates are always the tightest, we found that in some cases our method convergesto sub-optimal solutions.
We speculate that this arises due to the highly non-convex nature of theobjective combined with a strong K -inflating gradient signal from the O ( e − Kγ 2 ) term.
Thus futurework to improve these results even further could start with the use of the quasi-convex small-klrelaxation from Thiemann et al.
We note however that this is overall less important than ourmain results, as both our bounds are still extremely tight when used in an algorithm-free way andapplied to the output of another algorithm.
[Figure]
[Figure]
[Figure]