Online PAC-Bayes Learning
Abstract
Most PAC-Bayesian bounds hold in the batch learning setting where data is collected at once, prior to inference or prediction.
This somewhat departs from many contemporary learning problems where data streams are collected and the algorithms must dynamically adjust.
We prove new PAC-Bayesian bounds in this online learning framework, leveraging an updated deﬁnition of regret, and we revisit classical PAC-Bayesian results with a batch-to-online conversion, extending their remit to the case of dependent data.
Our results hold for bounded losses, potentially non-convex , paving the way to promising developments in online learning.
Introduction
Batch learning is somewhat the dominant learning paradigm in which we aim to design the best predictor by collecting a training dataset which is then used for inference or prediction.
Classical algorithms such as SVMs [see Cristianini et al., 2000, among many others] or feedforward neural networks [Svozil et al., 1997] are popular examples of efﬁcient batch learning.
While the mathematics of batch learning constitute a vivid and well understood research ﬁeld, in practice this might not be aligned with the way practitionners collect data, which can be sequential when too much information is available at a given time ( e.g. the number of micro-transactions made in ﬁnance on a daily basis).
Indeed batch learning is not designed to properly handle dynamic systems.
Online learning (OL)
[Zinkevich, 2003, Shalev-Shwartz, 2012, Hazan, 2016] ﬁlls this gap by treating data as a continuous stream with a potentially changing learning goal.
OL has been studied with convex optimisation tools and the celebrated notion of regret which measures the discrepancy between the cumulative sum of losses for a speciﬁc algorithm at each datum and the optimal strategy.
It led to many fruitful results comparing the efﬁciency of prediction for optimisation algorithms such that
Online Gradient Descent (OGD), Online Newton Step through static regret [Zinkevich, 2003, Hazan et al., 2007].
OL is ﬂexible enough to incorporate external expert advice onto classical algorithms with the optimistic point of view that such advices are useful for training [Rakhlin and Sridharan, 2013a,b] and then having optimistic regret bounds.
Modern extensions also allow to compare to moving strategies through dynamic regret [see e.g. Yang et al., 2016, Zhang et al., 2018, Zhao et al., 2020].
However, this notion of regret has been challenged recently: for instance, Wintenberger [2021] chose to control an expected cumulative loss through PAC inequalities in order to deal with the case of stochastic loss functions.
Statements holding with arbitrarily large probability are widely used in learning and especially within the PAC-Bayes theory.
Since its emergence in the late 90s, the PAC-Bayes theory (see the seminal works of Shawe-Taylor and Williamson, 1997, McAllester, 1998, 1999 and the recent surveys by
Guedj, 2019, Alquier, 2021) has been a powerful tool to obtain generalisation bounds and to derive efﬁcient learning algorithms.
Classical PAC-Bayes generalisation bounds help to understand how a learning algorithm may perform on future similar batches of data.
More precisely, PAC-Bayes learning exploits the Bayesian paradigm of explaining a learning problem through a meaningful distribution over a space of candidate predictors [see e.g. Maurer, 2004, Catoni, 2007, Tolstikhin and Seldin, 2013, Mhammedi et al., 2019].
An active line of research in PAC-Bayes learning is to overcome classical assumptions such that data-free prior, bounded loss, iid data [see Lever et al., 2010, 2013, Alquier and Guedj, 2018, Holland, 2019, Rivasplata et al., 2020, Haddouche et al., 2021,
Guedj and Pujol, 2021] while remaining in a batch learning sprit.
Finally, a pioneering line of work led by [Seldin et al., 2012a,b] on PAC-Bayes learning for martingales and independently developed by [Gerchinovitz, 2011, Foster et al., 2015, Li et al., 2018] boosted PAC-Bayes learning by providing sparsity regret bound, adaptive regret bounds and online algorithms for clustering.
Our contributions.
Our goal is to provide a general online framework for PAC-Bayesian learning.
Our main contribution (Thm. 2.3 in Sec. 2) is a general bound which is then used to derive several online PAC-Bayesian results (as developed in Secs.
3 and 4).
More speciﬁcally, we derive two types of bounds, online PAC-Bayesian training and test bounds .
Training bounds exhibit online procedures while the test bound provide efﬁciency guarantees.
We propose then several algorithms with their associated training and test bounds as well as a short series of experiments to evaluate the consistency of our online PAC-Bayesian approach.
Our efﬁciency criterion is not the classical regret but an expected cumulative loss close to the one of Wintenberger [2021].
More precisely, Sec.
3 propose a stable yet time-consuming Gibbs-based algorithm, while Sec. 4 proposes time efﬁcient yet volatile algorithms.
We emphasize that our PAC-Bayesian results only require a bounded loss to hold: no assumption is made on the data distribution, priors can be data-dependent and we do not require any convexity assumption on the loss, as commonly assumed in the OL framework.
Outline.
Sec.
2 introduces the theoretical framework as well as our main result.
Sec.
3 presents an online PAC-Bayesian algorithm and draws links between PAC-Bayes and OL results.
Sec.
4 details online PAC-Bayesian disintegrated procedures with reduced computational time and Sec.
5 gathers supporting experiments.
We include reminders on OL and PAC-Bayes in Appendixes A.1 and C.
Appendix B provide disucssion about our main result.
All proofs are deferred to Appendix D.
An online PAC-Bayesian bound
We establish a novel PAC-Bayesian theorem (which in turn will be particularised in Sec. 3) which overcomes the classical limitation of data-independent prior and iid data.
We call our main result an online PAC-Bayesian bound as it allows to consider a sequence of priors which may depend on the past and a sequence of posteriors that can dynamically evolve as well.
Indeed, we follow the online learning paradigm which considers a continous stream of data that the algorithm has to process on the ﬂy, adjusting its outputs at each time step w.r.t.
the arrival of new data and the past.
In the
PAC-Bayesian framework, this paradigm translates as follows: from an initial (still data independent) prior Q 1 = P and a data sample S
=
( z 1 , ..., z m ) , we design a sequence of posterior ( Q i ) 1 ≤ i ≤ m where Q i = f ( Q 1 , ..., Q i − 1 , z i ) .
Framework.
Consider a data space Z (which can be only inputs or pairs of inputs/outputs).
We ﬁx an integer m >
and our data sample S ∈
Z m is drawn from an unknown distribution µ .
We do not make any assumption on µ .
We set a sequence of priors, starting with P 1 = P a data-free distribution and ( P i )
i ≥ 2 such that for each i , P i is F i − 1 measurable where ( F i )
i ≥ 0 is an adapted ﬁltration to S .
For P, Q ∈ M
( H ) , the notation P (cid:28) Q indicates that Q is absolutely continuous wrt P (i.e. Q ( A )
=
if P ( A ) = 0 for measurable A ⊂ H ).
We also denote by Q i our sequence of candidate posteriors.
There is no restriction on what Q i could be.
In what follows we ﬁx a ﬁltration ( F i ) i ≥
and we denote by KL the Kullback-Leibler divergence between two distributions.
We consider a predictor space H and a loss funtion (cid:96) :
H ×
Z → R + bounded by a real constant
K > 0 .
This loss deﬁnes the (potentially moving) learning objective.
We denote by M 1 ( H ) the set of all probability distributions on H .
We now introduce the notion of stochastic kernel [Rivasplata et al., 2020] which formalise properly data-dependent measures within the PAC-Bayes framework.
First, for a ﬁxed predictor space H , we set Σ H to be the considered σ -algebra on H .
Deﬁnition 2.1 (Stochastic kernels) .
A stochastic kernel from S =
Z m to H is deﬁned as a mapping
Q : Z m × Σ H → [0; 1] where
• For any B ∈ Σ H , the function S =
( z 1 , ..., z m ) (cid:55)→ Q ( S, B ) is measurable, • For any S ∈ Z m , the function B (cid:55)→ Q ( S, B ) is a probability measure over H .
We denote by Stoch ( S , H ) the set of all stochastic kernels from S to H and for a ﬁxed S , we set
Q S :=
Q ( S, . )
the data-dependent prior associated to the sample S through Q.
From now, to refer to a distribution Q S depending on a dataset S , we introduce a stochastic kernel
Q ( ., . )
such that Q S = Q ( S, . ) .
Note that this notation is perfectly suited to the case when Q S is obtained from an algorithmic procedure A .
In this case the stochastic kernel Q of interest is the learning algorithm A .
We use this notion to characterise our sequence of priors.
Deﬁnition 2.2.
We say that a sequence of stochastic kernels ( P i ) i =1 ..m is an online predictive sequence if (i) for all i ≥ 1 , S ∈ Z m , P i ( S, . ) is F i − 1 measurable and (ii) for all i ≥ 2 , P i ( S, . )
(cid:29)
P i − 1 ( S, . ) .
Note that (ii) implies that for all i, P i ( S, . )
(cid:29) P 1 ( S, . )
with P 1 ( S, . )
a data-free measure (yet a classical prior in the PAC-Bayesian theory).
We can now state our main result.
Theorem 2.3.
For any distribution µ over Z m , any λ > 0 and any online predictive sequence (used as priors)
( P i ) , for any sequence of stochastic kernels ( Q i ) we have with probability 1 − δ over the sample
S ∼ µ , the following, holding for the data-dependent measures Q i,S :
= Q i ( S, . ) , P i,S :=
P i ( S, . ) : m (cid:88)
i
[Equation]
Remark 2.4.
For the sake of clarity, we assimilate in what follows the stochastic kernels Q i , P i to the data-dependent distributions Q
i ( S, . ) , P i ( S, . ) .
Then, an online predictive sequence is also assimilated to a sequence of data-dependent distributions.
Concretely this leads to the switch of notation Q i,S
→ Q i in Thm.
The reason of this switch is that, even though stochastic kernel is the right theoretical structure to state our main result, we consider in Secs.
3 and 4 practical algorithmic extensions which focus only on data-dependent distributions, hence the need to alleviate our notations.
The proof is deferred to Appendix D.1.
See Appendix B for context and discussions.
A batch to online conversion.
First, we remark that our bound slightly exceeds the OL framework: indeed, it would require our posterior sequence to be an online predictive sequence as well, which is not the case here (for any i , the distribution Q i,S can depend on the whole dataset ).
This is a consequence of our proof method (see Appendix D.1), which is classically denoted as a "batch to online" conversion (in opposition to the "online to batch" procedures as in Dekel and Singer, 2005).
In other words, we exploited PAC-Bayesian tools designed for a ﬁxed batch of data to obtain a dynamic result.
This is why we refer to our bound as online as it allows to consider sequences of priors and posteriors that can dynamically evolve.
Analysis of the different terms in the bound.
Our PAC-Bayesian bound formally differs in many points from the classical ones.
On the left-hand side of the bound, the sum of the averaged expected loss conditioned to the past appears.
Having such a sum of expectations instead of a single one is necessary to assess the quality of all our predictions.
Indeed, because data may be dependent, one can not consider a single expectation as in the iid case.
We also stress that taking an online predictive sequence as priors leads to control losses conditioned to the past, which differs from classical PAC-Bayes results designed to bound the expected loss.
This term, while original in the
PAC-Bayesian framework (to the best of our knowledge) recently appeared (in a modiﬁed form) in
Wintenberger [2021, Prop 3].
See Appendix B.2 for further disucssions.
On the right hand-side of the bound, online counterparts of classical PAC-Bayes terms appear.
At time i , the measure Q i (i.e. Q i,S according to Remark 2.4) has a tradeoff to achieve between an overﬁtted prediction of z i (the case Q i =
δ z i where δ is a Dirac measure) and a too weak impact of the new data with regards to our prior knowledge (the case Q i = P i ).
The quantity λ > 0 can be seen as a regulariser to adjust the relative impact of both terms.
Inﬂuence of λ .
The quantity λ also plays a crucial role on the bound as it is involved in an explicit tradeoff between the KL terms, the conﬁdence term log(1 /δ ) and the residual term mK 2 / 2 .
This idea of seeing λ as a trading parameter is not new [Thiemann et al., 2017, Germain et al., 2016].
However, the results from Thiemann et al.
[2017] stand w.p. 1 − δ for any λ while ours and the ones from Germain et al.
[2016] hold for any λ w.p. 1 − δ which is weaker and implies to discretise R + onto a grid to estimate the optimal λ .
We now move on to the design of online PAC-Bayesian algorithms.
An online PAC-Bayesian procedure
OL algorithms (we refer to Hazan, 2016 an introduction to the ﬁeld) are producing sequences of predictors by progressively updating the considered predictor (see Appendix A.1 for an example).
Recall that, in the OL framework, an algorithm outputs at time i a predictor which is F i − 1 -measurable.
Here, our goal is to design an online procedure derived from Thm.
2.3 which outputs an online predictive sequence (which is assimilated, according to Remark 2.4, to a sequence of distributions).
Online PAC-Bayesian (OPB) training bound.
We state a corollary of our main result which paves the way to an online algorithm.
This constructive procedure motivates the name Online
PAC-Bayesian training bound (OPBT RAIN in short).
Corollary 3.1 (OPBT RAIN ) .
For any distribution µ over Z m , any λ > 0 and any online predictive sequences ˆ Q, P , the following holds with probability 1 − δ over the sample S ∼ µ :
[Equation]
Here, λ is seen as a scale parameter as precised below.
The proof consists in applying Thm.
2.3 with for all i , Q i =
ˆ Q
i +1 and P
i .
Note that in this case, our posterior sequence is an online predictive sequence in order to ﬁt with the OL framework.
Corollary 3.1 suggests to design ˆ Q as follows, assuming we have drawn a dataset S
= { z 1 , ..., z m } , ﬁxed a scale parameter λ > 0 and an online predictive sequence P i :
[Equation]
which leads to the explicit formulation
[Equation]
Thus, the formulation of Eq.
(2), which has been highlighted by Catoni [2003, Sec. 5.1] shows that our online procedure produces Gibbs posteriors.
So, PAC-Bayesian theory provides sound justiﬁcation for the somewhat intuitive online procedure in Eq.
(1): at time i , we adjust our new measure ˆ Q
i +1 by optimising a tradeoff between the impact of the newly arrived data z i and the one of prior knowledge
ˆ Q i .
Notice that ˆ Q is an online predictive sequence: ˆ Q i is F i −
-measurable for all i as it depends only on ˆ Q i − 1 and z i − 1 .
Furthermore, one has ˆ Q i (cid:29) ˆ Q i − 1 for all i as ˆ Q i is deﬁned as an argmin and the KL term is ﬁnite if and only it is absolutely continuous w.r.t.
ˆ Q
i − 1 .
Remark 3.2.
In Corollary 3.1, while the right hand-side is the reason we considered Eq.
(1) , the left hand side still needs to be analysed.
It expresses how the posterior ˆ Q
i +1 (designed from ˆ Q i , z i ) generalises well on average to any new draw of z i .
More precisely, this term measures how much the training of ˆ Q i +1 is overﬁtting on z i .
A low value of it ensures our online predictive sequence, which is obtained from a single dataset, is robust to the randomness of S , hence the interest of optimising the right hand side of the bound.
This is a supplementary reason we refer to Corollary 3.1 as an
OPBT RAIN bound as it provide robustness guarantees for our training.
[Table]
PAC
[Table]
Bayesian (OPB) test bound.
However, Corollary 3.1 does not say if ˆ Q i +1 will produce good predictors to minimise (cid:96) ( .,
z i +1 ) , which is the objective of ˆ Q
i +1 in the OL framework (we only have access to the past to predict the future).
We then need to provide an Online PAC-Bayesian (OPB) test bound (OPBT EST bound) to quantify our prediction’s accuracy.
We now derive an OPBT EST bound from Thm. 2.3.
Corollary 3.3 (OPBT EST ) . .
For any distribution µ over Z m , any λ > 0 , and any online predictive sequence ( ˆ Q i ) , the following holds with probability 1 − δ over the sample S ∼ µ :
[Equation]
Optimising in λ gives λ = (cid:112) 2log(1 /δ ) / mK 2 and ensure that:
[Equation]
The proof consists in applying Thm.
2.3 with for all i , Q i =
ˆ Q i = P
i .
Corollary 3.3 quantiﬁes how efﬁcient will our predictions be.
Indeed, the left hand side of this bound relates for all i , how good ˆ Q i is to predict z i (on average) which is what ˆ Q
i is designed for.
Note that here, the involved λ can differ from the scale parameter of Eq.
(1), it is now a way to compensate for the tradeoff between the two last terms of the bound.
The strength of this bound is that since ˆ Q is an online predictive sequence, the Kullback-Leibler terms vanished, leaving terms depending only on hyperparameters.
Links with previous approaches
We now present a speciﬁc case of Corollary 3.1 where we choose as priors the online predictive sequence ˆ Q ( i.e. in Thm.
2.3, we choose Q i
=
ˆ Q i +1 , P i
=
ˆ Q i ).
The reason we focus on this speciﬁc case is that it enables to build strong links between PAC-Bayes and OL.
We then adapt our OPBT RAIN bound (Corollary 3.1).
The online procedure becomes:
[Equation]
which leads to the explicit formulation
[Equation]
Links with classical PAC-Bayesian bounds.
We denote that the optimal predictor in this case is such that at any time i , d ˆ Q
i +1 ( h ) ∝ exp( − λ(cid:96)
( h, z i ))
d ˆ Q i ( h ) hence
d ˆ Q m +1 ( h ) ∝ exp ( − λ (cid:80) mi =1 (cid:96) ( h, z i ))
d ˆ Q 1 ( h ) .
One recognises, up to a multiplicative constant, the optimised predictor of Catoni [2007,
Th 1.2.6] which solves argmin Q E h ∼ Q [ 1 m (cid:80) mi =1 (cid:96) ( h, z i )]
+ KL
[Equation]
Q (cid:107)
[Equation]
λ , thus one sees that in this case, the output of our online procedure after m steps coincides with Catoni’s output.
This shows consistency of our general procedure which recovers classical result within an online framework: when too many data are available, treating data sequentially until time
m leads to the same Gibbs posterior than if we were treating the whole dataset as a batch.
Analogy with Online Gradient Descent (OGD).
We propose an analogy between the procedure
Eq.
(3) and the celebrated OGD algorithm (see Appendix A.1 for a recap).
First we remark that our minimisation problem is equivalent to argmin Q λ E h
i ∼ Q
[ (cid:96) ( h i , z i )]
+ KL( Q (cid:107) ˆ Q i ) .
Then we assume that for any i, ˆ Q i = N ( ˆ m i , I d ) with ˆ m i ∈
R d
and we set L i ( ˆ m i ) =
E h i ∼ ˆ Q
i [ (cid:96) ( h i , z i )] .
The minimisation problem becomes: argmin ˆ m λ L i ( ˆ m ) + 12 (cid:107) ˆ m − ˆ
m i (cid:107) 2 .
And so using the ﬁrst order
Taylor expansion, we use the approximation L i ( ˆ m )
≈ L i ( ˆ m i ) + (cid:104) ˆ m − ˆ m i , ∇ L i ( ˆ m i ) (cid:105) which ﬁnally transform our argmin into the following optimisation process: ˆ m i +
=
ˆ m i − λ ∇ L i ( ˆ m i ) which is exactly OGD on the loss sequence L i .
We draw an analogy between the scale parameter λ and the step size η in OGD.
the KL term translates the inﬂuence of the previous point and the expected loss gives the gradient.
This analogy has been already exploited in Shalev-Shwartz [2012] where they approximated E h i ∼ q µ [ (cid:96)
( h i , z i )] :
= ¯ L i ( µ ) ≈ µ T ∇ ¯ L i ( µ i ) where µ is their considered online predictive sequence.
Finally, we remark that the optimum rate in Corollary 3.3 is a O ( √ m ) which is comparable to the best rate of Shalev-Shwartz [2012, Eq (2.5)] (see proposition A.2).
Comparison with previous work.
We acknowledge that the procedure of Eq.
(3) already appeared in literature.
Li et al.
[2018, Alg. 1] propose a Gibbs procedure somewhat similar to ours, the main difference being the addition of a surrogate of the true loss at each time step.
Within the OL literature, the idea of updating measures online has been recently studied for instance in Chérief-Abdellatif et al.
More precisely, our procedure is similar to their Streaming Variational Bayes (SVB) algorithm.
A slight difference is that they approximated the expected loss similarly to Shalev-Shwartz [2012].
The guarantees Chérief-Abdellatif et al.
[2019] provided for SVB hold for Gaussian priors and comes at the cost of additional constraints that do not allow to consider any aggregation strategies contrary to what Corollary 3.1 propose.
Their bounds are deterministic and are using tools and assumptions from convex optimisation (such that convex expected losses) while ours are probabilistic and are using measure theory tools which allow to relax these assumptions.
Strength of our result.
We emphasize two points.
First, to the best of our knowledge, Corollary 3. is the ﬁrst bound which theoretically suggests Eq.
(3) as a learning algorithm.
Second, we stress that Eq.
(3) is a particular case of Corollary 3.1 and our result can lead to other fruitful routes.
For instance, we consider the idea of adding noise to our measures at each time step to avoid overﬁtting (this idea has been used e.g. in Neelakantan et al., 2015 in the context of deep neural networks): if our online predicitve sequence ( ˆ Q i ) can be deﬁned through a sequence of parameter vectors ˆ µ , then we can deﬁne P i by adding a small noise on ˆ µ i and thus giving more freedom through stochasticity.
Thus, we see that our procedure led us to the use of the Gibbs posteriors of Catoni.
However, in practice, Gaussian distributions are preferred [ e.g. Dziugaite and Roy, 2017, Rivasplata et al., 2019,
Perez-Ortiz et al., 2021b,a, Pérez-Ortiz et al., 2021]).
That is why we focus next on new online
PAC-Bayesian algorithms involving Gaussian distributions.
Disintegrated online algorithms for Gaussian distributions.
We dig deeper in the ﬁeld of disintegrated PAC-Bayesian bounds, originally explored by Catoni [2007], Blanchard and Fleuret [2007], further studied by Alquier and Biau [2013], Guedj and Alquier [2013] and recently developed by Rivasplata et al.
[2020], Viallard et al.
[2021] (see Appendix C for a short presentation of the bound we adapted and used).
The strength of the disintegrated approach is that we have directly guarantees on the random draw of a single predictor, which avoids to consider expectations over the predictor space.
This fact is particularly signiﬁcant in our work as the procedure precised in Eq.
(2), require the estimation of an exponential moment to be efﬁcient, which may be costful.
We then show that disintegrated PAC-Bayesian bounds can be adapted to the OL framework, and that they have the potential to generate proper online algorithms with weak computational cost and sound efﬁciency guarantees.
Online PAC-Bayesian disintegrated (OPBD) training bounds.
We present a general form for online PAC-Bayes disintegrated (OPBD) training bounds .
The terminology comes from the way we craft those bounds: from PAC-Bayesian disintegrated bounds we use the same tools as in Thm.
2.3 to create the ﬁrst online PAC-Bayesian disintegrated bounds.
OPBD training bounds have the following form.
For any online predictive sequences ˆ Q, P , any λ > 0 w.p. 1 − δ over S ∼ µ and ( h 1 , ..., h m ) ∼ ˆ Q 2 ⊗ ... ⊗ ˆ Q m +1 : m (cid:88) i =
E [ (cid:96) ( h i , z i ) | F i − 1 ] ≤ m (cid:88) i = (cid:96) ( h i , z i )
+ Ψ( h
[Equation]
with Ψ , Φ being real-valued functions.
Ψ controls the global behaviour of Q i +1 w.r.t.
the F i − 1measurable prior P i .
If one has no dependency on h i this behaviour is global, otherwise it is local.
Note that those functions may depend on λ, δ .
However, since they are ﬁxed parameters, we do not make these dependencies explicit.
Similarly to Corollary 3.1, this kind of bound allows to derive a learning algorithm
(cf Algorithm 1) which outputs an online predicitve sequence ˆ Q .
Finally we draw ( h 1 , ..., h m ) ∼ ˆ Q 2 ⊗ ...
⊗ ˆ Q m +1 (and not ˆ Q 1 ⊗ ... ⊗ ˆ Q m ) since an OPBD bound is designed to justify theoretically an OPBD procedure in the same way Corollary 3.1 allowed to justify Eq.
Why focus on Gaussian measures?
The reason is that a Gaussian variable h ∼ N ( w, σ 2 I d ) can be written as h = w + ε with ε ∼ N (0 , σ 2 I d ) , and this expression totally deﬁnes h ( I d being the identity matrix).
A general OPBD algorithm for Gaussian measure with ﬁxed variance We use an idea presented in Viallard et al.
[2021] which restrict the measure set to Gaussian on R d with known and ﬁxed covariance matrix σ 2 I d .
Then we present in Algorithm 1 a general algorithm (derived from an
OPBD training bound) for Gaussian measures with ﬁxed variance which outputs a sequence of gaussian ˆ Q i
=
N ( ˆ w i , σ 2 I d ) from a prior sequence P i
=
N ( w 0 i , σ 2 I d ) where for each
i , w
i is F i − 1measurable.
Because the variance is ﬁxed, the distribution is uniquely deﬁned by its mean, thus we identify ˆ Q i and ˆ w i , P i and w 0 i .
Parameters : Time m, scale parameter λ
Initialisation : Variance σ 2 , Initial mean ˆ w 1 ∈ R d , epoch m 1 for each iteration i in 1 ..m do
Observe z i , w
i and draw ε i ∼ N (0 , σ 2 I d )
Update: ˆ w i +1 :=
argmin w ∈ R d (cid:96) ( w + ε i , z i ) + Ψ( w + ε i , w, w 0 i ) 4 end 5 Return ( ˆ w i ) i =1 ..m +
At each time i , Algorithm 1 requires the draw of ε i ∼ N (0 , σ 2 I d ) .
Doing so, we generated the randomness for our h i (because our bound holds for a single draw of ( h 1 , .., h m ) ∼ ˆ Q 2 ⊗ ... ⊗ ˆ Q m +1 ), we then write h i =
w + ε i
and we optimise w.r.t.
Ψ to ﬁnd ˆ w
i
+1 .
Bounds of interest.
We present two possible choices of pairs (Ψ , Φ) derived from the disintegrated results presented in Appendix C. Doing so, we explicit two ready-to-use declinations of Algorithm 1.
Corollary 4.1.
For any distribution µ over Z m , any online predictive sequences of Gaussian measures with ﬁxed variance
ˆ Q i = N ( ˆ w i , σ 2 I d ) and P i = N ( w 0 i , σ 2 I d ) , any λ > 0 , w.p. 1 − δ over S ∼ µ and ( h i =
ˆ w i +
+ ε i ) i =1 ..
m ∼ ˆ Q 2 ⊗ ... ⊗ ˆ Q m +1 , the bound of Eq.
(4) holds for the two following pairs Ψ , Φ :
[Equation]
Where the notation 1 , 2 denote whether the functions have been derived from adapted theorems of
Rivasplata et al., 2020, Viallard et al., 2021 recalled in Appendix C We then can use algorithm 1 with
Eq.
(5) , Eq.
Proof is deferred to Appendix D.2.
Note that in Corollary 4.1, we identiﬁed ˆ Q
i to ˆ w i and for the last formula, Ψ has no dependency on h i .
Comparison with Eq.
The main difference with Eq.
(1) provided by the disintegrated framework is that the optimisation route does not include an expected term within the optimisation objective.
The main advantage is a weaker computational cost when we restrict to Gaussian distributions.
The main weakness is a lack of stability as our algorithm now depends at time i on (cid:96) ( h + ε i , z i ) so on ε i directly.
We denote that Eq.
(5) is less stable than Eq.
(6) as it involves another dependency on ε i through Ψ .
The reason is that Rivasplata et al.
[2020] proposed a bound involving a disintegrated KL divergence while Viallard et al.
[2021] proposed a result involving a Rényi divergence avoiding a dependency on ε i .
We refer to Appendix C for a detailed statement of those properties.
Comparison with van der Hoeven et al.
Theorem 3 of van der Hoeven et al.
[2018] recovers
OGD from the exponential weights algorithm by taking a sequence of moving distributions being
Gaussians with ﬁxed variance which is exactly what we consider here.
From these, they retrieve the classical OGD algorithm as well as its classical convergence rate.
Let us compare our results with theirs.
First, if we ﬁx a single step η in their bound and assume two traditional assumptions for OGD (a ﬁnite diameter D of the convex set and an uniform bound G on the loss gradients), we recover for the OGD (greedy GD in van der Hoeven et al., 2018) a rate of D 2 2 σ 2 η + ησ 2 TG 2.
This is, up to constants and notation changes, exactly our Ψ i ( i ∈ { 1 , 2 } ).
Also, we notice a difference in the way to use Gaussian distributions: Theorem 3 of van der Hoeven et al.
[2018] is based on their
Lemma 1 which provides guarantees for the expected regret.
This is a clear incentive to consider as predictors the mean of the sucessive Gaussians of interest.
On the contrary, Corollary 4.1 involves a supplementary level of randomness by considering predictors
h i drawn from our Gaussians.
This additional randomness appears in our optimisation process (algorithm 1).
Finally, notice that van der
Hoeven et al.
[2018] based their whole work on the use of a KL divergence while Corollary 4.1 not only exploit a disintegrated KL ( Ψ 1 ) but also a Rényi α -divergence ( Ψ 2 ).
Note that we propose a result only for α = 2 for the sake of space constraints but any other value of α leads to another optimisation objective to explore.
OPBD test bounds.
Similarly to what we did in Sec. 3, we also provide OPBD test bounds to provide efﬁciency guarantees for online predicitve sequences (e.g. the output of algorithm 1).
Our proposed bounds have the following general form.
For any online predictive sequence ˆ Q , any λ > 0 w.p. 1 − δ over S and ( h 1 , ..., h m ) ∼ ˆ Q 1 ⊗
... ⊗ ˆ Q m :
[Equation]
i
[Equation] [Equation]
with Φ being a real-valued function(possibly dependent on λ, δ though it is not explicited here).
Note that our predictors ( h 1 , ..., h m ) are now drawn from ˆ Q 1 ⊗ ... ⊗ ˆ Q m .
Thus, the left-hand side of the bound considers a h i drawn from an F i − 1 -measurable distribution evaluated on (cid:96) ( ., z i ) : this is effectively a measure of the prediction performance.
We now state a corollary which gives disintegrated guarantees for any online predicitve sequence.
Corollary 4.2.
For any distribution µ over Z m , any λ > 0 , and any online predictive sequence ( ˆ Q i ) , the following holds with probability 1 − δ over the sample S ∼ µ and the predictors ( h 1 , ..., h m ) ∼ ˆ Q 1 ⊗ ... ⊗ ˆ Q m , the bound of Eq.
(7) holds with :
[Equation]
Where the notation 1 , 2 denote whether the functions have been derived from adapted theorems of
Rivasplata et al., 2020, Viallard et al., 2021 recalled in Appendix C.
The optimised λ gives in both cases a O ( (cid:112)
m log(1 /δ ) .
Proof is deferred to Appendix D.2.
Experiments
We adapt the experimental framework introduced in Chérief-Abdellatif et al.
[2019, Sec.5] to our algorithms (anonymised code available here).
We conduct experiments on several real-life datasets, in classiﬁcation and linear regression.
Our objective is twofold: check the convergence of our learning methods and compare their efﬁciencies with classical algorithms.
We ﬁrst introduce our experimental setup.
Algorithms.
We consider four online methods of interest: the OPB algorithm of Eq.
(3) which update through time a Gibbs posterior.
We instantiate it with two different priors ˆ Q 1 : a Gaussian distribution and a Laplace one.
We also implement Algorithm 1 with the functions Ψ 1 , Ψ 2 from Corollary 4.1.
To assess efﬁciency, we implement the classical OGD (as described in Alg. 1 of Zinkevich, 2003) and the SVB method of Chérief-Abdellatif et al.
Binary Classiﬁcation.
At each round i the learner receives a data point x i ∈ R d and predicts its label y i ∈ {− 1 , +1 } using (cid:104) x
i , h i (cid:105) , with h i =
E h ∼ ˆ Q
i
[ h ] for OPB methods or h i being drawn under ˆ Q i for OPBD methods.
The adversary reveals the true value y i , then the learner suffers the loss (cid:96) ( h i , z i ) = (cid:0) 1 − y
i
h
Ti x i (cid:1) + with z i
=
( x i , y i ) and a + = a if a > 0 and a + = 0 otherwise.
This loss is unbounded but can be thresholded.
[Figure]
Linear Regression.
At each round i , the learner receives a set of features x i ∈ R d and predicts y i ∈ R using (cid:104) x
i , h i (cid:105) with h i =
E h ∼ ˆ Q
i
[ h ] for SVB and OPB methods or h i being drawn under ˆ Q i for OPBD methods.
Then the adversary reveals the true value y t and the learner suffers the loss (cid:96) ( h i , z i )
=
(cid:0) y
i − h Ti x i (cid:1) 2 with z i =
( x i , y i ) .
This loss is unbounded but can be thresholded.
Datasets.
We consider four real world dataset: two for classiﬁcation (Breast Cancer and Pima
Indians)
[Figure]
and two for regression (Boston Housing and California Housing).
All datasets except the
Pima Indians have been directly extracted from sklearn [Pedregosa et al., 2011].
Breast Cancer dataset [Street et al., 1993] is available here and comes from the UCI ML repository as well as the
Boston Housing dataset [Belsley et al., 2005] which can be obtained here.
California Housing dataset [Pace and Barry, 1997] comes from the StatLib repository and is available here.
Finally, Pima Indians dataset [Smith et al., 1988] has been recovered from this Kaggle repository.
Note that we randomly permuted the observations to avoid to learn irrelevant human ordering of data (such that date or label).
Parameter settings.
We ran our experiments on a 2021 MacBookPro with an M1 chip and
Gb RAM.
For OGD, the initialisation point is 0 R d and the values of the learning rates are set to η = 1 / √ m .
For SVB, mean is initialised to 0 R d and covariance matrix to Diag (1) .
Step at time i is η i = 0 .
1 / √ i .
For both of the OPB algorithms with Gibbs posterior, we chose λ = 1 /m .
As priors, we took respectively a centered Gaussian vector with the covariance matrix Diag ( σ 2 ) ( σ = 1 . 5 ) and an iid vector following the standard Laplace distribution.
For the OPBD algorithm with Ψ 1 , we chose λ = 10 − 4 /m , the initial mean is 0 R d and our ﬁxed covariance matrix is Diag ( σ 2 ) with σ = 3 .
10 − 3 .
For the OPBD algorithm with Ψ 1 , we chose λ = 2 .
10 − 3 /m , the initial mean is 0 R d and our covariance matrix is Diag ( σ 2 ) with σ = 10 − 2 .
The reason of those higher scale parameters and variance is that Ψ from Rivasplata et al.
[2020] is more stochastic (yet unstable) than the one
Viallard et al.
Experimental results.
For each dataset, we plot the evolution of the average cumulative loss (cid:80) ti =1 (cid:96) ( h i , z i ) /t as a function of the step t = 1 , . . .
, m , where m is the dataset size and h
i is the decision made by the learner h i at step i .
The results are gathered in Fig.
Empirical ﬁndings.
OPB with Gaussian prior (’Gibbs Gauss’) outperforms OGD on all datasets except California Housing (on which this method is not implemented ) while OPB with Laplace prior (’Gibbs Laplace’) always fail w.r.t.
OGD.
OPB methods fail to compete with SVB on the Boston
Housing dataset.
OPBD methods compete with SVB on regression problems and clearly outperforms
OGD on classiﬁcation tasks.
OPBD with Ψ 2 (labeled as ’OPBD Via’ in Fig.
) performs better on the California Housing dataset while OPBD with Ψ 1 (labeled as ’OPBD Riva’) is more efﬁcient on the Boston Housing dataset.
Both methods performs roughly equivalently on classiﬁcation tasks.
This brief experimental validation shows the consistency of all our online procedures as we observe a visible decrease of the cumulative losses through time.
It particularly shows that OPBD procedures improve on OGD on these dataset.
We refer to Appendix E for additional table gathering the error bars of our OPBD methods.
Why do we perform better than OGD?
As stated in Sec. 4, OGD can be recovered as a Gaussian approximation of the exponential weights algorithm (EWA).
Thus, a legitimate question is why do we perform better than OGD as our OPBD methods are also based on a Gaussian surrogate of EWA?
van der Hoeven et al.
[2018] only used Gaussians distributions with ﬁxed variance as a technical tool when the considered predictors are the Gaussian means.
In our work, we exploited a richer characteristic of our distributions in the sense our predictors are points sampled from our
Gaussians and not only the means.
This also has consequences in our learning algorithm as at time i of our algorithm 1, our optimisation step involves a noise ε i ∼ N (0 , σ 2 I ) .
Thus, we believe that
OPBD methods should perform at least as well as OGD.
We write ’at least’ as we think that the higher ﬂexibility due to this additional level of randomness might result in slightly better empirical performances, as seen on the few datasets in Fig. 1.
Conclusion
We establish links between Online Learning and PAC-Bayes.
We show that PAC-bayesian bounds are useful to derive new OL algorithms.
We also prove sound theoretical guarantees for such algorithms.
We emphasise that all of our results stand for any general bounded loss, especially no convexity assumption is needed.
Having no convexity assumption on the loss paves the way to exciting future practical studies, starting with Spiking Neural Network which is investigated in an online fashion (see
Lobo et al., 2020 for a recent survey).
A follow-up question on the theoretical part is whether we can relax the bounded loss assumption: we leave this for future work.
Acknowledgements
We warmly thank reviewers and the Area Chair who provided insigthful comments and suggestions which greatly helped us improve our manuscript. B.G. acknowledges partial support by the U S. Army Research Laboratory and the U
S. Army Research Ofﬁce and by the U.K. Ministry of
Defence and the U.K. Engineering and Physical Sciences Research Council (EPSRC) under grant number EP/R013616/1; B.G. also acknowledges partial support from the French National Agency for
Research, grants ANR-18-CE40-0016-01 and ANR-18-CE23-0015-02. A Background A.1 Reminder on Online Gradient Descent
For the sake of completeness we re-introduce the projected Online Gradient Descent (OGD) on a convex set K .
This is a ﬁrst example of online learning philosophy.
It may be the algorithm that applies to the most general setting of online convex optimization.
This algorithm, which is based on standard gradient descent from ofﬂine optimization, was introduced in its online form by Zinkevich [2003].
In each iteration, the algorithm takes a step from the previous point in the direction of the gradient of the previous cost.
This step may result in a point outside of the underlying convex set.
In such cases, the algorithm projects the point back to the convex set, i.e. ﬁnds its closest point in the convex set.
We precise this algorithm works with the assumptions of a convex set K bounded in diameter by D and of bounded gradients (by a certain G ).
We also assume here to have a dataset
S =
( z t ) t =1 ..T and to be coherent with the online learning philosophy, we assume that for each t > 0 , we possess a loss function (cid:96) t depending on the points ( z 1 , ...
z t ) .
We present OGD in algorithm
Parameters : Epoch T, step-size ( η )
Initialisation : Convex set K , Initial point θ 0 ∈ K , T, step sizes ( η t ) t 1 for each iteration t in 1 ..
T do
Compute f (cid:48) ( θ n )
Play (observe) θ t and compute the cost f t ( θ t ) Update and project
[Equation]
4 end 5 Return θ T
One now deﬁnes the notion of regret which is the classical quantity to evaluate the performance of an online algorithm.
Deﬁnition A.1.
One deﬁnes the regret of a decision sequence ( θ t ) at time T w.r.t.
a point θ as:
Regret T ( θ ) :=
T (cid:88) t = (cid:96) t ( θ t )
[Table]
Now we state a regret bound which can be found in [Shalev-Shwartz, 2012, Eq 2.5] although we slightly modiﬁed the result, which uses additional hypotheses from Hazan [2016].
Proposition A.2.
Assume that K has a ﬁxed diameter D and that the gradients of any point is bounded by G .
Then for any θ ∈ K , the regret of projected OGD with ﬁxed step η satisﬁes:
[Equation]
A.2 About PAC-Bayes learning
We provide here more information about PAC-Bayes learning.
We propose a local framework; then expose what APC-Bayes lerning ais to do and ﬁnally state two celebrated PAC-Bayes theorems.
An usual framework We ﬁrst state the following framework:
• H is a space of considered predictors • Z is a data space.
z can be an unlabeled data x or a couple ( x, y ) of a point with its label.
We assume that µ is a distribution over Z which rules the distribution of our data.
• (cid:96) :
H ×
Z → R + is a loss function i.e. the learning objective we want to minimise.
• S =
( z 1 , ...
z m ) an iid dataset following µ .
• The generalisation risk for h ∈ H :
R ( h ) =
E z ∼ µ [ (cid:96) ( h, z )] .
• The empirical risk R m ( h ) =
1 m (cid:80) mi =1 (cid:96) ( h, z i ) .
What is PAC-Bayes learning?
PAC-Bayes learning is about learning a meaningful data-dependent posterior distribution Q from a (classically data-free) prior P without necessarily exploiting the
Bayes formula.
PAC-Bayes learning controls the expected generalisation error :
[Equation]
which is the averaged error that would make a predictor drawn from our posterior distribution on a new point (usually this objective holds under an iid assumption on our dataset).
Two classical theorems.
We state below two celebrated PAC-Bayesian results: the McAllester bound enriched with Maurer’s remark as stated in [Guedj, 2019, Thm.1] and Catoni’s bound (Catoni, 2007, Thm 1.2.6).
Those theorems both holds with the assumptions of iid data and loss bounded by 1 .
Note that in Thm.
B.1, we also proposed another bound which is a corollary of Catoni’s one.
Theorem A.3 (McAllester’ bound) .
For any prior distribution P , we have with probability 1 − δ over the m -sample S , for any posterior distribution Q such that Q (cid:28) P :
[Equation]
where KL is the Kullback-Leibler divergence.
Theorem A.4.
For any prior distribution P , any λ > 0 , we have with probability 1 − δ over the m -sample S , for any posterior distribution Q such that Q (cid:28) P :
[Equation]
where KL is the Kullback-Leibler divergence.
B Discussion about Thm. 2. B.1 Comparison with classical PAC-Bayes
The goal of this section is to show how good Thm.
2.3 compared to a naive approach which consists in applying classical PAC-Bayes results sequentially.
The interest of this section is twofold:
• First, presenting a classical PAC-Bayes result extracted and adapted from Alquier et al.
[2016] which is formally close to what we propose.
• Second, showing that a naive (yet natural) approach to obtain online PAC-Bayes bound leads to a deteriorated bound.
We ﬁrst state our PAC-Bayes bound of interest.
Theorem B.1 (Adapted from Alquier et al.
[2016], Thm 4.1) .
Let S =
( z 1 , ..., z m ) be an iid sample from the same law µ 0 .
For any data-free prior P , for any loss function (cid:96) bounded by K , any λ > 0 , δ ∈ ]0; 1[ , one has with probability 1 − δ for any posterior Q ∈ M 1 ( H ) :
[Equation]
Remark B.2.
Two remarks about this result:
• Thm.
B.1 is a particular case of the original theorem from Alquier et al.
[2016] as we take the case of a bounded loss which implies the subgaussianity of the random variables (cid:96) ( ., z i ) and then allows us to recover the factor λK 2 m
• This theorem is derived from Catoni [2007] and constitutes a good basis to compare ourselves with as it similar formally similar.
Naive approach A naive way to obtain OPB bounds is to apply m times Thm.
B.1 (one per data) on batches of size 1 and then summing up the associated bounds.
Thus one has the beneﬁts of classical
PAC-Bayes bound without having no more the need of data-free priors nor the iid assumption.
The associated result is stated below:
Theorem B.3.
For any distributions µ 1 , ..., µ m over Z (such that z i ∼ µ i ), any λ > 0 and any online predictive sequence (used as priors) ( P i ) , the following holds with probability 1 − δ over the sample
S ∼ µ for any posterior sequence ( Q i ) :
[Equation]
Recall that here again we assimilate the stochastic kernels Q i , P i to the data-dependent distributions
Q i ( S, . ) , P i ( S, . )
Proof.
First of all, for any i , we apply Thm.
B.1 m to the batch { z i } .
This allows us to consider P i as a prior as it does not depend on the current data.
We then have, taking δ (cid:48)
=
δ/m , for any i ∈ { 1 ..m } with probability 1 − δ/m :
[Equation]
Then, taking an union bound on those m events ensure us that with probability 1 − δ , for any i ∈ { 1 ..m } :
[Equation]
Finally summing those m inequalities ensure us the ﬁnal result with probability 1 − δ .
Comparison between Thm. 2.3 and Thm.
B.3 Three points are noticeable between those two theorems:
• First of all, the main issue with Thm.
B.3 is that has a strongly deteriorated rate of O (cid:16) m log( m/δ ) λ (cid:17) instead of the rate in O (cid:16) log(1 /δ )
λ (cid:17) proposed in Thm.
More precisely, the problem is that we do not have a sublinear bound: one cannot ensure any learning through time.
This point justiﬁes the need of the heavy machinery exploited in Thm. 2.
proof
as it allows a tighter convergence rate.
• The second point point lies in the controlled quantity on the left hand-side of the bound.
Thm.
B.3 controls A :=
[Equation]
i
[Equation]
]] instead of B :
[Equation]
A is a less dynamic quantity than B in the sense that it does not imply any evolution through time, it just considers global expectations.
Doing so, A does not take into account that at each time step we have acces to all te past data to predict the future, this may explain the deteriorated convergence rate.
Thus B , which appears to be a suitable quantity to control to perform online PAC-Bayes (see Appendix B.2 for additional explanations) • Finally,
an interesting point is that in Thm.
B.3 the bound, while looser, holds unformly for any posterior sequence contrary to Thm. 2.3 which holds only for a speciﬁc posterior sequence.
This point will have a consequence for optimisation.
We will come back later on this in Appendix B.3.
B.2 A deeper analysis of Thm. 2.
This section includes discussion about our proof technique and why all the assumptions made are necessary.
We also propose a short discussion about the beneﬁts and limitations of an online
PAC-Bayesian framework as well as a deeper reﬂexion about the new term our bound introduce.
Why do we need an online predictive sequence as priors?
This condition is fully exploited when dealing with the exponential moment ξ m in the proof (see Lemma D.2 proof).
Indeed, the fact of having P i being F i − 1 -measurable is essential to apply conditional Fubini (Lemma D.3).
Note that the condition ∀ i, P i − 1 (cid:28) P
i is not necessary as the weaker condition ∀ i, P 1 (cid:28) P i would sufﬁce here.
However, note that when we particularise our theorem, for instance if we choose in Corollary 3.
P i
=
ˆ Q i , one recovers the condition
ˆ Q i (cid:28) ˆ Q i +1 to have ﬁnite KL divergences.
Hence the interest of taking directly an online predictive sequence.
About the boundedness assumption The only moment where we invoke the boundedness assumption is in Lemma D.2’s
proof where we apply the conditionnal Hoeffding lemma.
This lemma actually translates that the sequence of r.v.
( (cid:96) ( ., z i ) i
=1 ..m is conditionally subgaussian wrt the past i.e for any
i , h i ∈ H ; λ ∈ R :
[Equation]
where ˜ (cid:96) i ( h i , z i ) =
E [ (cid:96) ( h i , z i ) | F i − 1 ] − (cid:96) ( h i , z i ) .
This condition is the one truly involved in our heavy machinery.
However, we chose to restrict ourselves to the stronger assumption of bounded loss function for the sake of clarity.
However, an interesting open direction is to ﬁnd whether there exists concrete classes of unbounded losses which may satisfy either conditional subgaussianity or others conditions (such as conditional Bernstein condition for instance).
Reﬂections about the left hand side of Thm.
We study in this paragraph the following term
[Equation]
has naturally arisen in our work as the right term to compare our empirical loss with to perform the conditional Hoeffding lemma.
Taking a broader look, we now interpret this term as the right quantity to control if one wants to perform online PAC-Bayes learning.
Indeed this term is a ’best of both world’ quantity bridging PAC-Bayes and online learning:
• From the PAC-Bayes point of view one keeps the control on average (cf the conditional expectation in B ) on a novel data drawn at each time step.
This point is crucial in the PAC-Bayes literature as our posteriors are designed to generalise well to unseen data.
• From the Online Learning point of view, one keeps the control of a sequence of points generated from an online algorithm.
Because an online learning algorithm generate a prediction for future points while having access to past data, the conditional expectation in B translates this.
Finally this conditional expectation appears to be a good tradeoff between the classical expectation on data appearing in the PAC-Bayes literature (see e.g. Thm.
B.1) and the local control that we have in online learning by only dealing with the performance of a sequence of points generated from a learning algorithm
(see e.g. proposition A.2)
About the interest of an Online PAC-Bayesian framework The main shift our work does with classical online learning literature is that it does not consider the celebrated regret but instead focuses on B which is a cumulative expected loss conditionned to the past.
This shift does not invalidate our work but put some relief to hte guarantees Online PAC-Bayes learning can provide that Online
Learning cannot and reversely.
• Online PAC-Bayes ensure a good potential for generalisation as it deals with the control of conditional expectation.
This can be useful if one wants to deal with a periodic process for instance.
• Online Learning through the regret compares the studied sequence of predictors (typically generated from an online learning algorithm) and tries to compare it to the best ﬁxed strategy (static regret) or the best dynamic one (dynamic regret).
In this way, OL algorithms want to ensure that their predictions are closed from the optimal solution.
This point is not guaranteed by our online PAC-Bayesian study.
However the limitations of online learning can arise if the studied problem has a huge variance (for instance micro-transactions in ﬁnance).
In this case those algorithms can follow an unpredictable optimisation route while PAC-Bayes still ensure a good performance on average (knowing the past) in this case.
• Finally, we want to emphasize that PAC-Bayesian learning circumvent a problem of memoryless learning which appears in classical OL algorithms.
For instance, the OGD algorthm (see Appendix A.1) uses once a data and do not memorise it for further use.
This problem does not happen in Online PAC-Bayes learning.
Indeed, we take the example of the procedure Eq.
(3) which generates Gibbs posterior which keep in mind the inﬂuence of past data.
B.3 Thm. 2.3 and optimisation
In this section we discuss about the way Thm 2.2 can be thought in the framework of an optimisation process as we did in Secs.
3 and 4.
A signiﬁcant change compared to classical PAC-Bayes Thm.
2.3 holds ’for any posterior sequence ( Q i ) the following holds with probability 1 − δ over the sample S ∼ µ ’ while most classical PACBayesian results such that Thm.
B.1 holds ’with probability 1 − δ over the sample S ∼ µ for any posterior Q ’.
This change is signiﬁcant as our theorem does not control simultaneaously all possible sequences of posteriors but only holds for one.
Thus, Thm. 2.3 has to be seen as a local or pointwise theorem and not as a global one.
In classical PAC-Bayes, this local behavior is a brake on the optimisation process.
But as we develop below, it is not the case in our online framework.
Thm.
2.3 is compatible with online optimisation We ﬁrst recall that classically, an online algorithm like OGD (see Appendix A.1) performs one optimisation step per arriving data.
Thus, at time m , such algorithm will perform m optimisation steps and generate m predictors.
Similarly the OPB algorithm of Eq.
(1) generates m distribution in m time steps.
We insist on the fact that, Thm. 2.3 and all its corollaries throughout our paper are valid for a sequence of m posteriors and not only a single one.
A key point is that whatever the number m of data, our theoretical guarantee wil still be valid for m posterior distributions with the approximation term log(1 /δ )
(and not log( m/δ ) as an union bound would provide for a classical PAC-Bayes theorem).
For this reason, given an online PAC-Bayes algorithm, Thm. 2.3 is suited for optimisation.
Indeed, having a bound valid for a sequence of posteriors ensures guarantees for a single run of our OPB algorithm.
This point is crucial to bridge a link with online learning as regret bounds (e.g. proposition A.2) also provide guarantees for a single sequence of predictors.
In online learning however, those guarantees are mainly deterministic (because based on convex optimisation properties) but not totally: the recent work of Wintenberger [2021] proposed PAC regret bounds for its general
Stochastic Online Convex Optimisation framework.
An interesting open challenge is to overcome the pointwise behavior of our theorem, for that, we need to rethought
[Rivasplata et al., 2020, Thm 2.1] as this basis is pointwise itself.
Given we consider a sequence of data-dependent priors one cannot apply the classical change of measure inequality to ensure guarantees holding uniformly on posterior sequences.
A crucial point: having an explicit OPB/OPBD algorithm
In our previous paragraph we said that our bound were suitable for optimisation given an OPB/OPBD algorithm.
We now provide some precision about this point.
All the procedures provided in the paper (i.e. Eq.
(1), algorithm 1) take into account an update phase implying an argmin.
Luckily for our procedures, this argmin is explicit:
• For the OPB algorithm of Eq.
(1), the argmin is solved thanks to the variational formulation of the Gibbs posterior • For OPBD algorithms, given the explicit choices of Ψ given in Corollary 4.1, argmin becomes explicit when one has a derivable loss function.
In both cases, this explicit argmin ensure our procedure of interest generates explictly a single posterior per time step: we have a well-deﬁned sequence of m posteriors at time m .
Doing so the guarantees of Thm.
2.3 holds for this sequence.
C A reminder on PAC-Bayesian disintegrated bounds
We present two PAC-Bayesian disintegrated bounds valid with data-dependent priors (i.e. any stochastic kernels).
• The ﬁrst one is Th.
1) i) from Rivasplata et al.
[2020] which provides a disintegrated version of Thm.
• The second one is Thm 2.
from Viallard et al.
[2021] which involves Rényi divergence instead of the classical KL .
Note that this bound has originally been stated for dataindepedent prior, which is why we revisit the proof to adapt it to the stochastic kernel framework.
Proposition C.1 (Th 1) i) Rivasplata et al.
Let P ∈ M 1 ( S ) , Q 0 ∈ Stoch ( Z m , F ) .
Let f : S × H
→ R be any measurable function.
Then for any Q ∈ Stoch ( Z m , F ) and any δ ∈ (0 , 1) , with probability at least 1 − δ over the random draw of S ∼ P and h ∼ Q S , we have:
[Equation]
where ξ m := (cid:82) S (cid:82)
H e f ( s,h ) Q 0 s ( dh ) P ( ds ) and dQ S dQ 0 S is the Radon Nykodym derivative of Q S w.r.t.
Q 0 S .
Proposition C.2 (Adapted from Th.
2 of Viallard et al.
Let µ ∈ M 1 ( S ) , Q 0 ∈
Stoch ( Z m , F ) .
Let α > 1 and f : S × H → R
+ be any measurable function.
Then for any Q ∈ Stoch ( Z m , F ) such that for any S ∈ Z m , Q S >> Q 0 S , Q 0 S >> Q S and any δ ∈ (0 , 1) , with probability at least 1 − δ over the random draw of S ∼ µ and h ∼ Q S , we have:
[Equation]
where D α ( Q (cid:107) P )
=
α − 1 log (cid:16) E (cid:104) E h ∼ P (cid:16) dQdP ( h ) (cid:17) α (cid:105)(cid:17) is the Rényi diverence of order α .
Note that Viallard et al.
original bound only stand for data-free priors and i.i.d data.
However it appears their proof works with any stochastic kernel as prior and any distribution over the dataset.
We propose below an adaptation of their proof below to ﬁt with those more general assumptions.
C.1 Proof of proposition C.
Proof.
For any sample S and any stochastic kernel Q , note that f ( S, h ) is a non-negative random variable.
Hence, from Markov’s inequality we have
[Equation]
Taking the expectation over S ∼ µ to both sides of the inequality gives
[Equation]
Taking the logarithm to both sides of the equality and multiplying by αα − 1 >
[Equation]
we obtain
[Equation]
We develop the right side of the inequality in the indicator function and make the expectation of the hypothesis over Q 0 S our "prior" stochadtic kernel appears.
Indeed, because for any S ∈ S , Q S >> Q 0 S and Q 0 S >> Q S one can write properly dQ S dQ 0 S and dQ 0 S
[Equation]
S =
[Equation]
S
[Equation]
S
[Equation]
− 1 the Radon-Nykodym derivatives.
Thus we have
[Equation]
Remark that 1 r + 1 s = 1 with r = α and s = αα − 1 .
Hence
[Equation]
we can apply Hölder’s inequality:
[Equation]
Then, by taking the logarithm; adding log (cid:0) 2 δ (cid:1) and multiplying by αα − 1 > 0 to both sides of the inequality
[Equation]
we obtain α
[Equation]
From this inequality
[Equation]
we can deduce that
P S
[Equation]
h ∼
[Equation]
(cid:20) α
[Equation]
Note that E h (cid:48) ∼ Q 0 S f ( S, h (cid:48) )
αα − 1 is a non-negative random variable, hence, we apply Markov’s inequality to have
[Equation]
Since the inequality does not depend on the random variable h ∼ Q S , we have
[Equation]
Taking the logarithm to both sides of the inequality and adding αα − 1 log 2 δ give us
[Equation]
Combining Equation Eq.
(8) and Eq.
(9) with a union bound gives us the desired result.
D Proofs D.1 Proof of Thm. 2.
Background We ﬁrst recall [Rivasplata et al., 2020, Thm 2].
Theorem D.1.
Let µ ∈ M 1 ( S ) , Q 0 ∈ Stoch ( S , F ) .
Let k be a positive integer, any A : S × H → R k a measurable function and F : R k
→ R be a convex function .
Then for any Q ∈ Stoch ( S , F ) and any δ ∈ (0 , 1) , with probability at least 1 − δ over the random draw of S ∼ µ we have
[Equation]
where ξ m := (cid:82) S (cid:82)
H e f ( s,h ) Q 0 s ( dh ) P ( ds ) and Q S
[ A S ] := Q S
[ A ( S, . )]
= (cid:82) H A ( S, h ) Q S ( dh ) .
Proof of Thm.
To fully exploit the generality of Thm.
D.1, we aim to design a m -tuple of probabilities.
Thus, our predictor set of interest is H m :=
H ⊗ m
and then, our predictor h is a tuple ( h 1 , .., h m ) ∈ H .
Throughout our study, our stochastic kernels Q, Q 0 will belong to the speciﬁc class C deﬁned below:
[Equation]
Thus our kernels are such that conditionally to a given sample, our predictors ( h 1 , ..., h m ) are drawn independently.
We now apply Thm.
D.1.
To do so, we consider the following function A : S × H m → R 2 such that
[Equation]
A is indeed measurable in both of its variables.
For a ﬁxed λ > 0 , we set the function F to be
F ( x, y )
=
λ ( x − y ) .
The only thing left to set up is our stochastic kernels.
To do so, let P =
( P 1 , ...
P m ) be an online predictive sequence, we then deﬁne Q 0 ∈ C (deﬁned in Eq.
s.t.
for any sample S , Q 0 S =
P 1 ( S, . )
⊗
... ⊗ P m ( S, . ) .
We also ﬁx Q 1 , ..., Q m to be any (posterior) stochastic kernels and similarly we deﬁne the stochastic kernel Q ∈ C such that for any sample S , Q ( S, . )
=
Q 1 ( S, . )
⊗
... ⊗ Q m ( S, . ) .
From now, we ﬁx a dataset S and, for the sake of clarity, we assimilate in what follows the stochastic kernels Q i , P i to the data-dependent distributions
Q i ( S, . ) , P i ( S, . )
(i.e. we drop the dependency in
S ).
Under those choices, one has:
Q S [ A S ] = (cid:90) h ∈ H m
A ( S, h ) Q S ( dh 1 , ..., dh m ) =
(cid:32)(cid:90)
h ∈
H m m (cid:88) i =
E [ (cid:96) ( h i , z i ) | F i − 1 ] Q S ( dh 1 , ..., dh m ) , (cid:90)
h ∈
H m m (cid:88) i = (cid:96) ( h i , z i ) Q S ( dh 1 , ..., dh m ) (cid:33) .
Furthermore, Q ∈ C , thus Q S ( dh 1 , ..., dh m ) =
Π mi =1 Q
i ( dh i ) so:‘
Q S [ A S ] = (cid:32) m (cid:88) i =
E h i ∼ Q i
[ E [ (cid:96) ( h i , z i ) |
F i − 1 ]] , m (cid:88) i =
E
[Equation]
Finally:
[Equation]
Applying Thm.
D [Equation] 1 and re-organising the terms gives us with probability 1 − δ : m (cid:88)
[Equation]
Thus: m
[Equation]
The last line holding because for a ﬁxed S , Q S
= Q 1 ⊗ ... ⊗ Q m and Q 0 S
= P 1 ⊗ ... ⊗ P m .
The last term to control is
[Equation]
, with ˜ (cid:96) i ( h i , z i ) =
E [ (cid:96) ( h i , z i ) | F i − 1 ] − (cid:96) ( h i , z i ) .
Hence the following lemma.
Lemma D.2.
One has for any m , ξ m ≤ exp (cid:16) λ 2 mK 2(cid:17) with K bounding (cid:96) .
The proof of this lemma is deferred to Appendix D.1.
To conclude the proof, we just bound ξ m by the result of Lemma D.2 within Eq.
D.1.1 Proof of Lemma D.
Proof of Lemma D.2.
We prove our result by recursion: for m = 1 , S =
z 1 and one knows that P 1 is F 0 measurable
yet it does not depend on S .
Thus for any h 1 ∈ H , E [ (cid:96) ( h 1 , z 1 ) | F 0 ]
=
E [ (cid:96) ( h 1 , z 1 )] .
We then has:
[Equation]
Fubini
[Equation]
The last line holding because for any h 1 ∈ H , ˜ (cid:96) 1 ( h 1 , z 1 ) is a centered variable belonging in [ − K, K ] a.s.
and so one can apply Hoeffding’s lemma to conclude.
Assume the result is true at rank m − 1 ≥ 0 .
We then has to prove the result at rank m .
Our strategy consists in conditioning by F m − 1 within the expectation over S :
[Equation]
First, we use that Q 0 ∈ C , thus Q 0 S
= P 1 ⊗ ... ⊗ P m (i.e. our data are drawn independently for a given S ):
[Equation]
We now condition by F m − 1 and use that Π m − 1 i =1 E h
i ∼ P
i [Equation] i
[Equation]
z i ) (cid:17)(cid:105) is a F m − 1 -measurable r.v.
[Equation]
Now our next step is to use a variant of Fubini valid for F m − 1 - measurable measures.
Lemma D.3 (Conditional Fubini) .
Let f : H × Z
→ R + .
For a sigma -algebra F over Z and a measure P over H such that
• P is a F -measurable r.v. •
There exists a constant measure (a.s.)
P 0 such that P >> P 0 .
Then one has almost surely, for any r.v.
z over Z :
[Equation]
The proof of this lemma lies at the end of this section.
We then ﬁx F = F m − 1 and f ( h, z ) =
exp( λ ˜ (cid:96) i
( h, z )) .
Furthermore, because we assumed the sequence ( P i ) to be an online predictive sequence, P m is F m − 1 -measurable and P m >> P 1 with Pa data-free prior.
One then applies Lemma D.3:
[Equation]
Yet, injecting this result onto ξ m provides:
[Equation]
The ﬁnal remark is to notice that for any h m ∈ H , E [˜ (cid:96) m
( h m , z m ) | F m − 1 ] = 0 and ˜ (cid:96) m
( h m , z m ) ∈
[ − K, K ] a.s. then one can apply the conditional Hoeffding’s lemma which ensure us that for any λ > 0 :
[Equation]
One then has ξ m ≤ exp (cid:16) λ
2 K 2 2 (cid:17) ξ m − 1 .
The recursion assumption concludes the proof.
Proof of Lemma D.3.
Let A be a F -measurable event.
One wants to show that
[Equation]
Where the ﬁrst expectation in each term is taken over z .
This will be enough to conclude that
[Equation]
thanks to the deﬁnition of conditional expectation.
We ﬁrst start by using the fact that P is Fmeasurable and that P 0 (cid:28) P with P 0 a constant measure.
This is enough to obtain that the
Radon-Nykodym derivative dPdP 0 is a F -measurable function, thus:
[Equation]
Because f ( h, z ) dPdP 0 ( h ) 1 A ( z ) is a positive function, and that P 0 is ﬁxed, one can apply the classical
Fubini-Tonelli theorem:
[Equation]
One now conditions by F and use the fact that [Equation] , 1 A are F -measurable:
[Equation]
We ﬁnally re-apply Fubini-Tonelli to re-intervert the expectations:
[Equation]
This ﬁnally proves the announced results, yet concludes the proof.
D.2 Proofs of Sec.
We prove here Corollary 4.1 and Corollary 4.2.
D.2.1 Proof of Corollary 4.
We ﬁx ˆ Q, P to be online predictive sequences (with ˆ Q 1 , P 1 being data-free priors).
Recall that we assimilated the stochastic kernels ˆ Q i , P i to the their associated data-dependent sitribution given a sample S
ˆ Q
i ( S, . ) , P i ( S, . ) .
As in Thm.
2.3, our predictor set of interest is H m :=
H ⊗ m
and then, our predictor h is a tuple ( h 1 , .., h m ) ∈ H .
We consider the stochastic kernel Q belonging to the class C deﬁned in Eq.
(10) such that for any S ∈ S , Q ( S, . )
=
ˆ Q 2 ⊗ ... ⊗ ˆ Q m +1 .
Similarly one deﬁnes Q 0 ∈ C such that for any
S ∈ S , Q 0 ( S, . )
=
P 1 ⊗ ... ⊗ P m
Proof for (Ψ 1 , Φ 1 ) :
For λ > 0 , we set our function f to be for any dataset S and predictor tuple
[Equation]
[Equation]
We then apply proposition C.1 with the function f , Q, Q 0 deﬁned above.
One then has by dividing by λ with probability 1 − δ over S ∼ µ and h =
( h 1 , ..., h m ) ∼ ˆ Q 2 ⊗
... ⊗ ˆ Q m +1 : m (cid:88)
[Equation]
And then using the fact that S ∈ S , Q S
=
ˆ Q 2 ⊗ ... ⊗ ˆ Q m +1 , Q 0 S
= P 1 ⊗ ...
⊗ P m gives us: m (cid:88) i = [Equation] [Equation] h
[Equation]
z i ) | F
[Equation]
1 ] ≤
[Equation]
, with ξ m =
[Equation]
i ( h i , z i ) (cid:17)(cid:105)(cid:105) and for any i , ˜ (cid:96) i ( h i , z i ) =
E [ (cid:96) ( h i , z i ) | F i − 1 ] − (cid:96) ( h i , z i )
Notice that, because P is an online predictive sequence, then one can apply directly Lemma D.2 to conclude that ξ m ≤ exp (cid:16) λ 2 K 2 m(cid:17) .
We also use [Viallard et al., 2021, Lemma 11] which derives the calculation of the disintegrated KL divergence between two Gaussians.
One then has for any i , with h i =
ˆ w i
+
+ ε i :
[Equation]
Combining those facts altogether allows us to conclude.
Proof for (Ψ 2 , Φ 2 ) :
For λ > 0 , we set our function f to be for any dataset S and predictor tuple ( h = h 1 , ..., h m ) , [Equation] = [Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:96)
[Equation]
z
[Equation]
(cid:33)(cid:33) .
We take α = 2 and apply this time proposition C.2.
One then has by dividing by 2 λ with probability 1 − δ over S ∼ µ and h =
( h 1 , ..., h m ) ∼ ˆ Q 2 ⊗
... ⊗ ˆ Q m +1 : m (cid:88) i =
E [ (cid:96) ( h i , z i ) | F i − 1 ] ≤ m (cid:88) i = (cid:96)
[Equation]
  . We ﬁrst notice that D 2 (cid:0) Q S (cid:107) Q 0 S (cid:1) =
(cid:80) m i =
1 D 2 ( ˆ Q i +1 (cid:107) P i ) as our predictors are drawn independently once S is given.
We also use that for any i , the Rényi divergence with α = 2 between ˆ Q
i +1 and P
i (two multivariate
Gaussians with same covariance matrix) is (cid:107) ˆ w
[Equation] w[-i +1 − w 
-]
[Equation] (as recalled in Gil et al.
We then remark that: ξ (cid:48) m = E S (cid:48) ∼ µ
E h (cid:48) ∼ Q 0 S (cid:48) exp (cid:32)
2 λ (cid:32) m (cid:88) i =
E [ (cid:96) ( h (cid:48)
i , z (cid:48) i ) | F i − 1 ] − m (cid:88) i
= (cid:96) ( h (cid:48)
i , z (cid:48) i ) (cid:33)(cid:33) .
Thus we recover the exponential moment ξ m from the Rivasplata’s case up to a factor 2 within the exponential.
We then apply Lemma D.2 with λ (cid:48)
= 2 λ to obtain that ξ (cid:48) m ≤ exp (cid:0) 2 λ 2 K 2 m (cid:1) .
Combining all those facts allows us to conclude.
D.2.2 Proof of Corollary 4.
We apply the exact same proof than Corollary 4.1.
The only difference is the way to deﬁne our stochastic kernels.
We now take, for a single online predictive sequence
ˆ Q the following stochastic kernels:
We consider the stochastic kernel Q belonging to the class C deﬁned in Eq.
(10) such that
for any
S ∈ S , Q ( S, . )
=
ˆ Q 1 ⊗ ... ⊗ ˆ Q m
and we take Q 0 = Q .
This fact allows the divergence terms (Rényi or KL depending on which bound we consider) to vanish.
The rest of the proof remains unchanged.
E Additional experiment
In this section we perform error bars for our OPBD methods in order to evaluate their volatility.
We ran n = 50 times our algorithms and then show in the table below for each data set the means and the standard deviation of our averaged cumulative losses at regular time steps.
We denote for i ∈ { 1 , 2 } ’OPBD Ψ i ’ to indicate that this algorithm is our OPBD method used with thev optimisation objective Ψ i .
Analysis Those tables shows the robustness of our OPBD methods to their intrinsic randomness: we always have a decreasing mean through time as well as an overall variance reduction.
Note that for the most complicated problem (California Housing dataset), the variance is the highest.
More precisely, we notice that the standard deviation of OPBD with Ψ 1 is always greater than the one of
OPBD with Ψ 2 which is not a surprise as Ψ 1 involves a disintegrated KL divergence while Ψ 2 is a proper Rényi divergence.
Hence the additional volatility for OPBD with Ψ 1 .
This fact is particurlaly noticeable on the California Housing dataset where both the means and variance of OPBD with Ψ 1 increase drastically between t=16000 and t=20000 while the increase is more attenuated for OPBD with Ψ 2 .
This fact is also visible on Fig.
[Table] [Table]
[Table] [Table]