On Margins and Generalisation for Voting Classifiers
Abstract
We study the generalisation properties of majority voting on finite ensembles ofclassifiers, proving margin-based generalisation bounds via the PAC-Bayes theory.
These provide state-of-the-art guarantees on a number of classification tasks.
Ourcentral results leverage the Dirichlet posteriors studied recently by Zantedeschiet al.
[1] for training voting classifiers; in contrast to that work our bounds apply tonon-randomised votes via the use of margins.
Our contributions add perspectiveto the debate on the “margins theory” proposed by Schapire et al.
[2] for thegeneralisation of ensemble classifiers.
Introduction
Weighted ensemble methods are among the most widely-used and effective algorithms known inmachine learning.
Variants of boosting [3, 4] are state-of-the-art in a wide variety of tasks [5, 6]and methods such as random forest [7] are among the most commonly-used in machine learningcompetitions [see, e.g. , 8, 9], valued both for their excellent results and interpretability.
Evenwhen these algorithms do not directly produce the best learners for a task, the best performance incompetitions is often obtained by an ensemble of “strong learners”—the output of a collection ofdifferent algorithms trained on the data—contrasted to the weak learners usually considered in theensemble learning literature.
Among the oldest ideas to explain the performance of ensemble classifiers, and machine learningmethods in general, is the concept of margins.
First introduced to analyse the Perceptron algorithm [10], margins relate closely to the idea of confidence in predictions in ensemble learning, witha large margin implying that a considerable weighted fraction of voters chose the same answer.
Thiswas first leveraged to obtain early margin-based generalisation bounds for ensembles by Schapireet al.
[2], in an attempt to understand the excellent generalisation of boosting, a surprising result givenclassical Vapnik-Chervonenkis theory.
This “margins theory” was explored further in a number ofworks [11–13] and is among the leading explanations for the success of such methods and boostingin particular.
The same thread of margin bounds for ensemble methods has also been taken up in parallel in
PAC-Bayes theory by Langford and Seeger [14], Biggs and Guedj [15].
PAC-Bayes provides anatural framework both for deriving margin bounds, and for considering ensemble methods in general,particularly majority votes where the largest-weighted ensemble prediction is taken.
Within theframework, the weightings are typically considered as the parameter of a categorical distributionover individual voters.
PAC-Bayes theorems [see the comprehensive surveys of 16, 17] then directlyprovide generalisation bounds for the performance of this “randomised” proxy for the majority vote,a.k.a.
Gibbs classifier.
These can then be de-randomised by such margin-based techniques, or througha variety of oracle bounds [18–21], motivating new learning algorithms [22–28].
Uniquely among PAC-Bayesian approaches, Zantedeschi et al.
[1] instead consider Dirichlet distributions over the voters.
Any sample from this distribution already implies a vector of voting weights,and it is on the performance and optimisation of these “stochastic majority votes” they primarily focus.
As an aside, they provide an oracle result which allows their bounds to be de-randomised, butthis introduces an irreducible factor such that the bound on the true fixed vote can never be less thandouble that of the stochastic version.
It also neglects to leverage the generally high confidence ofpredictions obtained by their algorithm.
Our contribution.
By combining tools from margin bounds and the use of Dirichlet majority votes,we provide a new margin bound for non-randomised majority votes.
This is in contrast to Zantedeschiet al.
[1] which primarily considers stochastic majority votes.
Our bound empirically comparesvery favourably to existing margin bounds and in contrast to them are applicable to multi-classclassification.
Remarkably, our empirical results are also sharper than existing PAC-Bayesian ones,even when the algorithm optimising those bounds is used.
Our primary tool is a new result relating the margin loss of these stochastic votes to the misclassification loss of the non-randomised ones in a surprisingly sharp way.
This tool can additionallybe utilised alongside a further idea from Zantedeschi et al.
[1] to obtain an alternative form of thebound which is more amenable to optimisation.
Through this work we provide further support tothe margins theory for ensembles, showing that near-sharp bounds based on margins alone can beobtained on a variety of real-world tasks.
Outline.
The rest of this section introduces the problem setup, notation and summarises main results.
Section 2 provides background on PAC-Bayes, Dirichlet majority votes and margin bounds, relatingthem to our new results.
Section 3 states and summarises our new theoretical results, giving the mostrelevant proofs (all remaining proofs are deferred to appendices).
Section 4 empirically evaluatesthese new results before we conclude with an overall discussion in Section 5.
1.1 Notation and setting
Majority voting algorithms combine the predictions of a finite set of “base” classifiers, H , from X to
Y =
[ c ] := { 1 , . .
, c } .
The classifiers h i ∈ H take the form h
i : X → Y for i ∈
[ d ] so that |H| =
d . 62 Majority votes consider as set of weightings θ in ∆ d , the simplex, and return the highest-weightedoverall prediction.
Using the indicator function I A of a set A , this is expressed as
[Equation]
We are primarily interested in learning a weighting θ with small misclassification risk (and guaranteesof this) based on a sample S ∼ D m , where D ∈ M +1 ( X × Y ) is the data-generating distributionand m ∈ N + the sample size.
We let M +1 ( A ) denote the set of probability measures on a set A .
For h ∈ H the misclassification loss is ℓ
( h, x, y ) :
= I h ( x ) ̸
= y , the misclassification out-of-samplerisk is L 0 ( h ) :
=
E ( x,y )
∼D ℓ 0 ( h, x, y ) and a hat denotes the in-sample estimate of this quantity,ˆ L 0 ( h ) :
= E ( x,y ) ∼ Uniform( S ) ℓ 0 ( h, x, y ) .
In a slight abuse of notation we will also often write the riskof the majority vote L 0 ( θ ) =
L 0 ( f θ ) and similarly for its empirical counterpart.
The margin of majority vote f θ on example ( x, y ) is derived from the minimal gap between the totalweight assigned to the true class y and to any other predicted class:
[Equation]
The corresponding margin loss is ℓ γ ( θ , x, y ) :
=
I M ( θ ,x,y ) ≤ γ for margin γ ≥ 0 , with the corresponding in-sample and out-of-sample risks notated as ˆ L γ ( θ ) and L γ ( θ ) respectively.
1.2 Overview of results
Our main result is a margin bound of the following form: with high probability ≥ 1 − δ over thesample, simultaneously for any θ ∈ ∆ d and K > 0 ,
[Equation]
where D Dir ( α , β ) is the KL divergence between Dirichlet random vectors with parameters α andβ , with 1 a vector of ones implying a uniform Dirichlet prior distribution on the simplex.
The term e − Kγ 2 is a de-randomisation penalty.
The parameter K is chosen freely in an arbitrary data-dependentway to balance the requirements of the different terms: it must be large enough to decrease thisexponential term, while too-large a parameter increases the KL divergence from the uniform prior.
This result is surprisingly strong; in particular there is no dependence on the dimensionality ( i.e. ,number of voters d ) in the exponential term, an advantage discussed further in Section 3.2.
In Equation (1), ˆ L γ ( θ ) is the 0-1 valued γ -margin loss which enables comparison with existingmargin bounds for trained weighted ensembles.
We further consider a second scenario, where thegeneralization bound is also used to train the model itself.
We note that the γ -margin loss ˆ L γ ( θ )appearing in Equation (1) has null gradients, so the bound cannot be directly optimised by gradientdescent.
To rectify this we also prove a variation of the bound, replacing the above loss by itsexpectation under a Dirichlet stochastic vote, E ξ ∼ Dir( K θ )
ˆ L γ ( ξ ) , which is bounded in differentiable 91 closed form to give an alternative, optimisation-friendly bound.
In our evaluations we focus on these two complimentary scenarios, obtaining state-of-the-art empiricalresults.
Across different scenarios and tasks our results outperform both existing margin bounds(including a sharpened version of the result from Biggs and Guedj [15] which may be of independentinterest), and PAC-Bayes bounds, even when it is not used as the objective.
Further, in contrast toexisting margin bounds our results also hold for multi-class majority votes.
Background2.1 PAC-Bayes bounds
PAC-Bayes bounds are among the tightest known generalisation bounds, as for example the onlyframework in which non-vacuous generalisation bounds for neural networks have been obtained [seee.g. 29–36].
However, unlike many other such bounds they usually apply to randomised Gibbs(-like)prediction functions rather than deterministic ones.
These are typically re-drawn for every newtest evaluation.
Thus a high-probability bound is obtained on the expectation of the risk w.r.t.
the
PAC-Bayes posterior Q , with the complexity of Q ∈ M +1 ( H ) appearing in the bound in terms of a
Kullback-Leibler (KL) divergence from a pre-chosen PAC-Bayes prior P ∈ M +
( H )
[which is not 106 required to be a true prior in the Bayesian sense – see the discussion in 16].
A particularly sharp [asdiscussed in 37] and widely-used result is given in Theorem 1, valid for any bounded loss function ℓwith values in [0 , 1] .
Theorem 1 (Seeger et al.
[38], Maurer [39]) .
For any D ∈ M +1 ( X × Y ) , m ∈ N + , prior P ∈
M +1 ( H ) and δ ∈ (0 , 1) , with probability ≥ 1 − δ over S ∼ D m , simultaneously for all Q ∈ M +
( H )
[Equation]
where the generalised inverse kl − 1 ( u, c ) :
= sup { v ∈ [0 , 1] : kl( u, v ) ≤ c } and kl( u, v ) :
= 112 u log uv + (1 − u ) log 1 − u 1 − v is a KL divergence between Bernoulli random variables.
The above bound uses the inverse small-kl function which will be seen in our later results anda number of pre-existing ones.
To lend intuition we note that kl − 1 ( u, c ) ∈
O ( u + c ) , giving
Equation (1) from Theorem 2 when using a uniform prior.
The following upper bounds are alsouseful: kl − 1 ( u, c ) ≤ u + p c/ 2 giving “slow-rates” and kl − 1 ( u, c ) ≤ u + √ 2 cu + 2 c .
From thiswe can see that when the loss ˆ L → 0 then the overall rate improves to O (1 /m ) , so the small-klformulation interpolates between the traditional fast and slow rate regimes of learning theory.
2.2 Margin bounds
In the learning theory literature there exists a rich tradition of using the concept of a margin, whichquantifies the confidence of predictions, to explain generalisation.
This is particularly evident in thecase of voting algorithms such as boosting, where traditional Vapnik-Chervonenkis based techniquespredict classical overfitting which is not ultimately observed.
The “margins theory” was developed by
Schapire et al.
[2] to explain this discrepancy.
By considering the weightings θ as the parameter of acategorical distribution, they proved a bound of the form (holding with probability greater than 1 − δ over the sample, as for all bounds in this section)
L 0 ( θ ) ≤ L γ ( θ ) + ˜ O
[Equation]
Although there wasinitially some debate about the validity of the theory [40], eventually Gao and Zhou [12, Theorem 4]provided the following improved bound which further supported that a large-margin voting classifiercould generalise: simultaneously for any γ > [Equation] and θ ∈ ∆ d ,
[Equation]
More recently, a similar bound (proved through a PAC-Bayesian method based on Seeger et al.
[38])was proved in Biggs and Guedj [15, Theorem 8].
Here we give a bound provided as an intermediatestep in their proof that is strictly (and empirically considerably) sharper than their final result: for anyfixed margin γ > 0 , simultaneously for any θ ∈ ∆ d
[Equation]
Since γ ∈ (0 , 12 ) for non-vacuous results, a union bound argument can be used to extend the above tofixed-precision γ , and this result has the advantage of being valid for small γ as are often observedempirically.
Our contributions.
Firstly we mention the smaller contribution of the improved form of the boundfrom Biggs and Guedj [15] given in Equation (3); a proof is given in Appendix B alongside furtherrefinements and evaluation.
However we show that in many cases even this improved version and
Equation (2) give weak or vacuous results.
As a result of this weakness (and thus perhaps null resultfor the margins theory applied to voting classifiers)
we present a new margin bound in Theorem 2based on Dirichlet distributions as a theoretical intermediate step.
This is also valid in the multi-classcase, unlike the above results which are only for binary classification.
Empirically the bound isobserved to give an enormous improvement in tightness than the existing margin bounds and in somecases is near-sharp.
2.3 Dirichlet stochastic majority votes
In most results from the PAC-Bayes framework, and in the proof of the existing results given in
Section 2.2, the majority vote weightings θ are considered the parameters of a categorical distributionover voters.
Zantedeschi et al.
[1] instead consider PAC-Bayesian bounds (specifically, Theorem 1)applied to a hypothesis class of majority votes of the form f ξ , where ξ ∼ Dir( α ) is drawn from a
Dirichlet distribution with parameter α .
This distribution has mean E ξ = α /
{+P di =
+} α i with a largersum
P di =
α i giving a more concentrated or peaked distribution (see Appendix A for more details).
Since ξ is randomised, the bounds from Zantedeschi et al.
[1] apply to “stochastic majority votes”rather than the more typical deterministic ones we consider here.
However, the use of such Dirichletdistributions over voters in the PAC-Bayes bounds rather than the more usual categorical ones is amajor step forward as it allows the correlation between voters to be more carefully considered.
Wewill utilise and de-randomise these as a stepping stone to ones for deterministic predictors f θ directly.
As is common in the PAC-Bayes literature, Zantedeschi et al.
[1] use their new bound as an optimisation objective to obtain a new algorithm, here using stochastic gradient descent.
The boundwith Dirichlet posterior obtained directly from Theorem 1 includes the expected misclassificationloss with respect to the Dirichlet parameters, E ξ ∼ Dir( α ) ℓ ( f ξ , x, y ) , which has null gradient for anysampled ξ .
They therefore additionally upper bound this term by the differentiable closed form
[Equation]
where I z ( a, b ) is the regularised incomplete beta function, which has a sigmoidal shape.
Theinequality is sharp in the binary classification case, and is used in the training objective and finalevaluation of their method.
As an aside, Zantedeschi et al.
[1] also proved an oracle bound whichallows their result to be de-randomised, but this introduces a irreducible factor of two.
Our contributions.
Firstly, we provide a new margin bound for majority vote algorithms utilising
Dirichlet posteriors as a theoretical stepping stone.
We show that this bound gives sharper bounds on the misclassification loss than the bound from Zantedeschi et al.
[1], doing better than the irreduciblefactor, even when applied to the output of their algorithm.
We show further that the bound is alsotighter when applied to the outputs of other PAC-Bayes algorithms derived from “categorical”-typeposteriors.
Finally, we give an altered form of the bound involving the expectation of the marginloss E ξ ∼ Dir( α ) ℓ γ ( f ξ , x, y ) and a result analogous to Equation (4) for this case.
Through this we areable to obtain a new PAC-Bayes objective which is compared to existing PAC-Bayes optimisationmethods.
Main results
Our main results use the idea of Dirichlet stochastic majority votes from Zantedeschi et al.
[1] as anintermediate step to prove new margin bounds for deterministic majority votes.
In this section, firstwe give our main result in Theorem 2 and discuss further.
In Section 3.1 we give an alternative boundobtained by a very similar method which is more amenable to optimisation, and we provide proofsfor these results in Section 3.2.
The central step in these proofs is in constructing a proxy Dirichlet distribution ξ ∼ Dir( K θ ) over voters, the loss of which is bounded à la PAC-Bayes, and de-randomised using margins to obtain boundsdirectly for f θ .
The primary complexity term appearing in our bounds is therefore D Dir ( K θ , β ) ,the KL divergence between Dirichlet distributions with parameters
K θ and β
respectively.
As with
PAC-Bayes priors, β can be chosen in arbitrary sample-independent fashion, but we typically chooseit as a vector of ones, giving a uniform distribution on the simplex as prior as in Equation (1).
Thebounds also involve a de-randomisation penalty of O ( e − Kγ 2 ) where γ is the margin appearing in theloss; this term upper bounds the difference between our randomised proxy ξ and its mean θ and getssmaller with K as the distribution concentrates tightly around its mean.
This parameter K can beoptimised in any data-dependent way to obtain the tightest final bound.
Theorem 2.
For any D ∈ M +1 ( X × Y ) , m ∈ N + , margin γ > 0 , δ ∈ (0 , 1) , and prior β ∈ R d + ,with probability at least 1 − δ over the sample S ∼ D m simultaneously for every θ ∈ ∆ d and K > 0 ,
[Equation]
Theorem 2 differs from the existing margin bounds of Equations (2) and (3), and Schapire et al.
[2]in a specific and significant way, with θ appearing not only in the loss function ˆ L γ ( θ ) , but also inthe KL complexity term.
Empirically we find our bound to be an improvement but it is possibleto generate scenarios where the pre-existing bounds are non-vacuous while ours is not, since the
KL divergence is unbounded for certain choices of θ , for example when one of the components isexactly zero.
This difference arises because the existing bounds all use the idea of a categoricaldistribution with parameter θ in their proofs (which has KL divergence from a uniform prior upperbounded by log d ), while we use a Dirichlet.
This gains us the surprisingly tight de-randomisationresult (Theorem 4) used in all proofs.
3.1 PAC-Bayes bound as objective
We note here that it is non-trivial to directly obtain a training objective for optimisation from
Theorem 2, due to the non-differentiability of the margin loss ˆ L γ ( θ ) .
Therefore, in order to compareresults with a wide variety of methods that optimise PAC-Bayes bounds [including those used by 1,as baselines], we obtain a relaxed and differentiable formulation in Theorem 3 for direct optimisation.
Theorem 3.
Under the conditions of Theorem 2 the following bound also holds
[Equation]
Using the incomplete Beta function I z ( a, b ) we also have the following result, which is sharp in thebinary classification case,
[Equation]
Theorem 3 has a stronger PAC-Bayesian flavour than Theorem 2, with an expected loss under somedistribution appearing (complicating the final optimisation of K ), while Theorem 2 takes a form muchcloser to that of a classical margin bound.
The second part of the result is analogous to Equation (4)used by Zantedeschi et al.
We combine both parts to calculate the overall bound in closed formand obtain gradients for optimisation.
3.2 Proof of main results
The proof of Theorems 2 and 3 essentially follow from applying a simple PAC-Bayesian bound incombination with the key Theorem 4 below.
In some sense this is our most important and novelresult.
Our whole approach is largely motivated by its surprising tightness; in particular there isno dependence on the dimension, which is avoided by careful use of the aggregation property ofthe Dirichlet distribution.
This surprise arises because to obtain a tightly concentrated Dirichletdistribution on ξ ∼ Dir( α ) , the concentration parameter K = P di {+=
+}
α i must grow linearly with the 223 dimension.
In fact, even a uniform distribution (which will be less peaked than our final posterior)has
[Equation]
d , so the de-randomisation step is effectively very cheap in higher dimensions.
Theorem 4.
Let θ ∈ ∆ d and K > 0 .
Then for any γ > 0 and ( x, y ) ,
[Equation]
For our proofs we first recall the aggregation property of the Dirichlet distribution: if ( ξ 1 , . . . , ξ d ) ∼
Dir(( α 1 , . . . , α d )) , then ( ξ 1 , . . . , ξ d − 1 + ξ d ) ∼ Dir(( α 1 , . . . , α d − 1 + α d )) .
We further note thefollowing crucial concentration-of-measure result.
The aforementioned lack of dimensionality in
Theorem 4 is possible because Theorem 5 depends only on
P di =1 α i , and this value is unchanged 230 by aggregation, which avoids the dimension dependence that would otherwise be introduced by therequirement ∥ u ∥ 2 = 1 below.
Theorem 5 (41) .
Let X ∼ Dir( α ) , t > 0 , and u ∈ R d with ∥ u ∥ 2 = 1 .
Then
[Equation]
Proof of Theorem 2 and Theorem 3.
The proof of our main results is completed by applying the PACBayes bound Theorem 1 with the γ -margin loss to a Dirichlet prior and posterior with parameters
βand K θ respectively.
Substituting the first part of Theorem 4 gives the first part of Theorem 3, andadditionally substituting the second part and re-scaling γ → γ/ 2 gives Theorem 2.
For the second part of Theorem 3, define w =
i : h i ( x ) = y } for fixed ( x, y )
so W :=
P i ∈ w ξ i ∼ 238 Beta (cid:0)P i
[Equation]
α i ,
P
[Equation]
/ ∈
[Equation]
α i (cid:1) by the aggregation property of the Dirichlet distribution.
Then
[Equation]
 using I a i − max j ̸
= i a j ≤ 2 γ ≤ I a i −
P j ̸
= i a j ≤ 2 γ
= I
P j ̸
= i a j ≥ 12 − γ for a ∈ ∆ c (with equality for c = 2classes), and that I z ( a, b ) is the CDF of a Beta distribution with parameters ( a, b ) .
Proof of Theorem 4.
Define γ 2 > γ 1 such that γ := γ 2 − γ 1 , and α = K θ .
From the trivialinequality I x ∈ A − I x ∈ B ≤ I x ∈
A
I x/ ∈ B we derive
[Equation]
[Equation]
[Equation]
where in the last inequality we set k = argmax k ̸
= y
P i : h i ( x )=
k θ i , and use that max j P i : h i ( x )=
j θ i − 244 max j
P i : h i ( x )= j
[Equation]
i ≤ max j P i : h i ( x )=
j θ i −
P i : h i ( x )=
k ξ i for any k .
We rewrite the above in vectorform (with inner product denoted u · v ) as
[Equation]
o where by the aggregation property of the Dirichlet distribution ˜ ξ ∼ Dir( ˜ α ) with˜ α :=   X i : h i ( x )= y α i ,
X i : h i ( x )=
k α i ,
X i : [Equation] i [Equation]
x
[Equation]
/ ∈{ k
[Equation]
α
[Equation]


T .
Applying Theorem 5 we obtain ∆ ≤ e − 4(
[Equation]
2 =
e − 4( P di {+=
+}
α i +1)
γ 2 .
This gives the firstinequality by setting γ 1 = 0 , γ 2 = γ .
Setting γ 1 = γ, γ 2 = 2 γ and swapping θ and ξ gives an almostidentical proof (with some signs reversed) of the second inequality.
Empirical evaluation
In this section we empirically validate our results against existing PAC-Bayesian and margin boundson several classification datasets from UCI [42], LIBSVM 1 and Zalando [43].
Since our main resultin Theorem 2 is not associated with any particular algorithm, we use θ outputted from PAC-Bayesderived algorithms to evaluate this result against other margin bounds (Figure 1) and PAC-Bayesbounds (Figure 2).
We then compare optimisation of our secondary result Theorem 3 with optimisingthose PAC-Bayes bounds directly (Figure 3).
All generalisation bounds given are evaluated with aprobability 1 − δ =0 .
Further details not provided here including tabulated results, description ofdatasets, training mechanisms and compute are provided in Appendix C.
Strong and weak voters.
Similarly to Zantedeschi et al.
[1] we consider both using data-independentand data-dependent voters.
This brings our experimental setup in line with a common workflowfor machine learning practicioners: the training set is sub-divided into a set for training severaldifferent strong algorithms, and a second set on which the weightings of these are optimised.
Morespecifically, the weak voter setting, used only for binary classification, uses axis-aligned decisionstumps (denoted stumps ), with thresholds evenly spread over the input space ( 6 per feature and perclass).
The stronger voters (denoted rf ) are learned from half of the training data, while the other halfis used for evaluating and optimising the different generalisation bounds (note this reduces m ).
Thesetake the form of random forests [7] of M =10 trees optimising Gini impurity score on n 2 baggedsamples and √ d drawn features for each tree, with unbounded maximal depth.
Optimising γ and K in bounds.
In reporting margin bounds we optimise over a grid of marginγ values in (0 , 12 ) , and additionally over K for Theorem 2.
Since Theorem 2 and Equation (3) asstated require a fixed margin, we apply a union bound over the values in the grid, replacing δ in thesebounds with δ/N where N is the number of grid points.
Existing PAC-Bayes bounds.
We compare to state-of-the-art PAC-Bayesian bounds (and derivedalgorithms) for weighted majority vote classifiers: the First Order [18], the Second Order [21],
Binomial [20] (with the number of voters set to 100 ) and the two Chebyshev-Cantelli-based [28]empirical bounds from categorical-type Gibbs classifiers with parameter θ , and we refer to these as
[Figure]
[Figure]
FO , SO , Bin , CCPBB and CCTND respectively (more details are given in Appendix C).
We denote byf2 the factor two bound derived in Zantedeschi et al.
[1, Annex A.4] from Dirichlet majority votes.
All prior distributions for PAC-Bayes bounds, including ours, are set to uniform.
We also refer by thesame names to the outputs of optimising these bounds with stochastic gradient descent; details ontraining and initialisation are given in Appendix C.
Description of figures.
In Figure 1 we compare Theorem 2 with the existing margin bound of
Equation (2) and the improved Biggs and Guedj [15] bound given in Equation (3).
Since Equation (3)is strictly better than the original result and the latter was vacuous in almost all cases considered (see
Appendix B), we do not include it.
All datasets are for binary classification as the existing results onlycover this case, and the θ values considered are the outputs of either the FO- or f2-optimisation usingeither the weak or
the strong voters described above.
Figure 2 extends this evaluation of Theorem 2to improve generalisation results, by applying it to the models optimised with the PAC-Bayes bounds
FO , SO , Bin and f2 as objective.
In this case, we consider both binary and multiclass datasets.
In
Figure 3 we directly compare the outputs of optimising state-of-the-art PAC-Bayesian bounds withour optimisation-ready variant result Theorem 3.
These experiments were carried out on strong voters,as standard in the literature [ e.g. 21, 26, 28].
Discussion and conclusion
We observe overall that in many cases the existing margin and PAC-Bayes bounds are insufficient toexplain the generalisation observed, while our new bound is consistently tight, and sometimes sharp( i.e. it approaches the true test error). [Figure]
PROTEIN
FASHION-MNIST
HABERMAN
TICTACTOE
ADULT
F O S O B i n
CC P BB CC T N D f 2 o u r s 0.
MNIST
F O S O B i n
CC P BB CC T N D f 2 o u r s
CODRNA
F O S O B i n
CC P BB CC T N D f 2 o u r s
PENDIGITS
F O S O B i n
CC P BB CC T N D f 2 o u r s
SENSORLESS
F O S O B i n
CC P BB CC T N D f 2 o u r s
MUSHROOMS
[Figure] Figure 1 demonstrates that existing margin bounds can be insufficient to explain the generalisationobserved, which could be construed as a null result for the “margins theory”.
However, our newbound obtains empirically very sharp results in all cases, reaffirming to the theory.
Unlike the existingresults, θ also arises in the complexity (KL divergence) term and so the bound is not equally tight forevery θ at fixed margin loss.
Further examination of this property could add additional nuance andperspective to the theory.
When comparing to existing PAC-Bayes bounds in Figure 2, remarkably Theorem 2 is always tighterthan just using the bound which is being optimised.
We speculate that this arises partially due to theirreducible factors appearing in those bounds; for example the FO or f2 bounds can never be tighterthan twice the train loss of the associated Gibbs classifier, while ours has no such limitation.
Thisresult is quite valuable as it demonstrates that Theorem 2 can be readily used in an algorithm-freemanner: the choice of learning algorithm is up to the practitioner, but the bound will then oftenprovide an excellent guarantee on the obtained weights θ .
Finally, in Figure 3, our optimisation-friendly variant bound Theorem 3 is seen to be competitive interms of test error while giving an improved-or-equal final bound on all datasets.
When consideringthe less-common setting of binary stumps (see Appendix C)
we found that sometimes this objectiveconverged to a sub-optimal local minimum.
We speculate that this arises due to the highly non-convexnature of the objective combined with a strong K -inflating gradient signal from the O ( e − Kγ 2 ) term.
Thus future work to improve these results even further could start with the use of the quasi-convexsmall-kl relaxation from Thiemann et al.
We note however that this is overall less importantthan our main results, as both our bounds are still extremely tight when used in an algorithm-freeway and applied to the output of another algorithm as discussed above.
Overall, we note that in many cases (a majority in Figure 2) our main bound of Theorem 2 is veryclose to the test set bound and thus cannot actually be improved any further, with the problem ofproviding sharp guarantees based on the training data alone effectively solved in many cases.
Conclusion.
We obtain empirically very strong generalisation bounds for voting classifiers usingmargins.
We believe these are highly relevant to the community, since voting-based classifiers andmargin-maximising algorithms are among the most popular and influential in machine learning.
Dirichlet majority votes have already obtained excellent results in the stochastic setting [1], butour new result in Theorem 4 showing they are well-approximated by their mean should open newdirections in the more conventional deterministic setting.
Our results also have practical relevance: for example, in the strong voter machine learning workflowdescribed above, instead of setting data aside as a test set, this data can be freed up to learn evenstronger voters, since a strong out-of-sample ensemble guarantee can still be provided even without atest set.
In future work we hope to expand these results further to other (non-majority) voting schemes likethose with score-output voters [as in e.g. 2], and ensembles of voters with finite VC dimension.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See, e.g. discussion in Section 5(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
is of a primarily theoretical nature and therefore no negative societal impact is foreseen.
All data is from publicly-available datasets.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Included in the
theorem statements and setting introduction in Section 1.
(b) Did you include complete proofs of all theoretical results?
[Yes] Included primarily inthe main paper with some extra result in the appendices.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4 and
Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] In Appendix C
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] In Appendix C
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Appendix C
(b) Did you mention the license of the assets?
[Yes] In Appendix C
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include links in Appendix C
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All data was publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We did not conduct any such research.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
We did not conduct any such research.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[-=
 θ i h-][N/A]
We did not conduct any such research.
[-• First Order [FO, 18]:For any D ∈ M +
-]