Generalization Error Bounds on Deep Learning with Markov Datasets
Abstract
In this paper, we derive upper bounds on generalization errors for deep neural networks with Markov datasets.
These bounds are developed based on Koltchinskii and Panchenko’s approach for bounding the generalization error of combined classiﬁers with i.i.d.
datasets.
The development of new symmetrization inequalities in high-dimensional probability for Markov chains is a key element in our extension, where the absolute spectral gap of the inﬁnitesimal generator of the Markov chain plays a key parameter in these inequalities.
We also propose a simple method to convert these bounds and other similar ones in traditional deep learning and machine learning to Bayesian counterparts for both i.i.d.
and Markov datasets.
Extensions to m -order homogeneous Markov chains such as AR and ARMA models and mixtures of several Markov data services are given.
Introduction
In statistical learning theory, understanding generalization for neural networks is among the most challenging tasks.
The standard approach to this problem was developed by Vapnik [1] and it is based on bounding the difference between the prediction error and the training error.
These bounds are expressed in terms of the so called VC-dimension of the class.
However, these bounds are very loose when the VC-dimension of the class can be very large or even inﬁnite.
In 1998, several authors [2, 3] suggested another class of upper bounds on generalization error that are expressed in terms of the empirical distribution of the margin of the predictor (the classiﬁer).
Later, Koltchinskii and Panchenko [4] proposed new probabilistic upper bounds on generalization error of the combination of many complex classiﬁers such as deep neural networks.
These bounds were developed based on the general results of the theory of Gaussian, Rademacher and empirical processes in terms of general functions of the margins, satisfying a Lipschitz condition.
They improved previously known bounds on generalization error of convex combination of classiﬁers.
In the context of supervised classiﬁcation, PAC-Bayesian bounds have proved to be the tightest [5–7].
Several recent works have focused on gradient descent based PAC-Bayesian algorithms, aiming to minimise a generalisation bound for stochastic classiﬁers [8–10].
Most of these studies use a surrogate loss to avoid dealing with the zero-gradient of the misclassiﬁcation loss.
Several authors used other methods to estimate of the misclassiﬁcation error with a non-zero gradient by proposing new training algorithms to evaluate the optimal output distribution in PAC-Bayesian bounds analytically [11–13].
Recently, there have been some interesting works which use information-theoretic approach to ﬁnd PAC-bounds on generalization errors for machine learning [14, 15] and deep learning [16].
All of the above-mentioned bounds are derived based on the assumption that the dataset is generated by an i.i.d.
process with unknown distribution.
However, in many applications in machine learning such as speech, handwriting, gesture recognition, and bio-informatics, the samples of data are usually correlated.
Some of these datasets are time-series ones with stationary distributions such as samples via MCMC, ﬁnite-state random walks, or random walks on graph.
In this work, we develop some upper bounds on generalization errors for deep neural networks with Markov or hidden Markov datasets.
Our bounds are derived based on the same approach as Koltchinskii and Panchenko [4].
To deal with the Markov structure of the datasets, we need to develop some new techniques in this work.
The development of new symmetrization inequalities in high-dimensional probability for Markov chains is a key element in our extension, where the absolute spectral gap of the inﬁnitesimal generator of the Markov chain plays as a key parameter in these inequalities.
Furthermore, we also apply our results to m -order
Markov chains such as AR and ARMA models and mixtures of Markov chains.
Finally, a simple method to convert all our bounds for traditional deep learning to counterparts for
Bayesian deep learning is given.
Our method can be applied to convert other similar bounds for i.i.d. datasets in the research literature as well.
Bayesian deep learning was introduced by [20, 21].
The key distinguishable property of a Bayesian approach is marginalization, rather than using a single setting of weights in (traditional) deep learning [22].
Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspeciﬁed by the data, and can represent many compelling but different solutions.
Analysis of machine learning algorithms for Markov and Hidden Markov datasets already appeared in research literature [17–19].
In practice, some real-world time-series datasets are not stationary Markov chains.
However, we can approximate time-series datasets by stationary Markov chains in many applications.
There are also some other methods of approximating non-stationary Markov chains by stationary ones via MA and ARMA models in the statistical research literature. The i.i.d. dataset is a special case of the Markov dataset with stationary distribution.  Preliminaries 2.1 Mathematical Backgrounds
Let a Markov chain { X n } ∞ n =1 on a state space S with transition kernel Q ( x, dy ) and the initial state
X 1 ∼ ν , where S is a Polish space in R .
In this paper, we consider the Markov chains which are irreducible and positive-recurrent, so the existence of a stationary distribution π is guaranteed.
An irreducible and recurrent Markov chain on an inﬁnite state-space is called Harris chain [23].
A Markov chain is called reversible if the following detailed balance condition is satisﬁed:
[Equation]
Deﬁne
[Equation]
and [Equation] [Equation] Let L 2 ( π ) be the Hilbert space of complex valued measurable functions on S that are square integrable w.r.t.
π .
We endow L 2 ( π ) with inner product (cid:104) f, g (cid:105) :
= (cid:82) fg ∗ dπ , and norm (cid:107) f (cid:107) 2 ,π := (cid:104) f, f (cid:105) 1 / 2 π .
Let
E π be the associated averaging operator deﬁned by ( E π )( x, y )
=
π ( y ) , ∀ x, y ∈ S , and
[Equation]
(4) where (cid:107) B (cid:107) L 2 ( π )
[Equation]
L 2 ( π ) = max v : (cid:107) v (cid:107) 2 , [Equation] =1 (cid:107) Bv (cid:107) 2 , π . Q can be viewed as a linear operator (inﬁnitesimal generator) on L 2 ( π ) , denoted by Q , deﬁned as ( Q f )( x ) :
= E Q ( x, · ) ( f ) , and the reversibility is equivalent to the self-adjointness of Q .
The operator Q acts on measures on the left, creating a measure µ Q , that is, for every measurable subset A of S , µ Q ( A ) :
= (cid:82) x ∈S Q ( x, A ) µ ( dx ) .
For a
Markov chain with stationary distribution π , we deﬁne the spectrum of the chain as
[Equation]
[Equation]
It is known that λ = 1 − γ ∗ [24], where
[Equation]
if eigenvalue 1 has multiplicity 1 , 0 , otherwise is the the absolute spectral gap of the Markov chain.
The absolute spectral gap can be bounded by the mixing time t mix of the Markov chain by the following expression:
[Equation]
where π ∗ =
min x ∈S π x is the minimum stationary probability , which is positive if Q k > 0 (entrywise positive) for some k ≥ 1 .
See [25] for more detailed discussions.
In [25, 26], the authors provided algorithms to estimate t mix and γ ∗ from a single trajectory.
Deﬁne
[Equation]
where (cid:107) · (cid:107) 2 is the standard L 2 norm in the Hilbert space of complex valued measurable functions on
S .
2.2 Problem settings
In this paper, we consider a uniformly bounded class of functions: F := (cid:8) f : S → R (cid:9) such that sup f ∈F (cid:13)(cid:13)
f (cid:13)(cid:13) ∞ ≤ M for some ﬁnite constant M .
Deﬁne the probability measure P ( A ) :
= (cid:82)
A π ( x ) dx, for any measurable set A ∈ S .
In addition, let P n be the empirical measure based on the sample ( X 1 , X 2 , · · · , X n ) , i.e., P {+n := 1 n (cid:80) ni =
+}
δ X i .
We also denote Pf := (cid:82) S fdP and P n f := (cid:82) S fdP n .
Then, we have
Pf = (cid:90)
S f ( x ) π ( x ) dx and P n f = 1 n n (cid:88) i = f
( X i ) .
On the Banach space of uniformly bounded functions F , deﬁne an inﬁnity norm: (cid:107) Y (cid:107) F = sup f ∈F | Y ( f ) | .
Let
[Equation]
where { g i } is a sequence of i.i.d.
standard normal variables, independent of { X i } .
We will call n (cid:55)→ G n ( F ) the Gaussian complexity function of the class F .
Similarly, we deﬁne
[Equation]
[Equation]
and
[Equation]
(11) where { ε i } is a sequence of i.i.d.
Rademacher (taking values +1 and − 1 with probability 1 / 2 each) random variables, independent of { X i } .
We will call n (cid:55)→ R n ( F ) the Rademacher complexity function of the class F .
For times-series datasets in machine learning, we can assume that feature vectors are generated by a Markov chain { X n } ∞ n =1 with stochastic matrix Q , and { Y n } ∞ n =1 is the corresponding sequence of labels.
Furthermore, Q is irreducible and recurrent on some ﬁnite set S .
An i.i.d.
sequence of feature vectors can be considered as a special Markov chain where Q ( x, x (cid:48) ) only depends on x (cid:48) .
In the supervised learning, the sequence of labels { Y n } ∞ n =1 can be considered as being generated by a Hidden Markov Model (HMM), where the emission probability P Y n | X n ( y | x ) =
g ( x, y ) for all n ≥and g : S × Y → R + .
It is easy to see that { ( X n , Y n ) } ∞ n =1 is a Markov chain with the transition probability
[Equation]
Let ˜ Q
( x 1 , y 1 , x 2 , y 2 ) :
=
Q ( x 1 , x 2 ) g ( x 2 , y 2 ) for all x 1 , x 2 ∈ S and y 1 , y 2 ∈ Y , which is the transition probability of the Markov chain { ( X n , Y n ) } ∞ n =1 on ˜ S :
= S × Y .
Then, it is not hard to see that { ( X n , Y n ) } ∞ n =1 is irreducible and recurrent on ˜ S , so it has a stationary distribution, say ˜ π .
The associated following probability measure is deﬁned as
[Equation]
and the empirical distribution P n based on the observations { ( X k , Y k ) } nk =1 is
[Equation]
2.3 Contributions
In this paper, we aim to develop a set of novel upper bounds on the generalization errors for deep neural networks with Markov dataset.
More specially, our target is to ﬁnd a relationship between Pf and P n f which holds for all f ∈ F in terms of Gaussian and Rademacher complexities.
Our main contributions include:
• We develop general bounds on generalization errors for machine learning (and deep learning) on Markov datasets.
• Since the dataset is non-i.i.d., the standard symmetrization inequalities in high-dimensional probability can not be applied.
In this work, we extend some symmetrization inequalities for i.i.d.
random processes to Markov ones.
• We propose a new method to convert all the bounds for machine learning (and deep learning) models to Bayesian settings. • Extensions to m -order homogeneous Markov chains such as AR and ARMA models and mixtures of several Markov services are given.
Main Results 3.1 Probabilistic Bounds for General Function Classes
In this section, we develop probabilistic bounds for general function classes in terms of Gaussian and
Rademacher complexities.
First, we prove the following key lemma, which is an extension of the symmetrization inequality for i.i.d.
sequences (for example, [27]) to a new version for Markov sequences { X n } ∞ n =1 with the stationary distribution π and the initial distribution ν ∈ M 2 :
Lemma 1.
Let F be a class of functions which are uniformly bounded by M .
For all n ∈ Z + , deﬁne
[Equation]
Then, the following holds:
[Equation]
where [Equation] The proof of this lemma can be found in Appendix A of the supplement material.
Compared with the i.i.d.
case, the symmetrization inequality for Markov chain in Lemma 1 are different in two perspectives: (1) The expectation E (cid:2) (cid:107) P 0 n (cid:107) F (cid:3) is now is under the joint distributions of Markov chain and Rademacher random variables and (2)
The term A n appears in both lower and upper bounds to compensate for the difference between the initial distribution ν and the stationary distribution π of the Markov chain 2 .
Later, we will see that A n represents the effects of data structures on the generalization errors in deep learning.
By applying Lemma 1, the following theorem can be proved.
See a detailed proof in Appendix C in the supplement material.
Theorem 2.
Denote by [Equation] Let ϕ be a non-increasing function such that ϕ ( x ) ≥ 1 ( −∞ , 0] ( x ) for all x ∈ R .
For any t > 0 ,
[Equation]
and
[Equation]
Since B n = O (cid:0) 1 / √ n (cid:1) , Theorem 2 shows that with high probability, the generalization error can be bounded by Rademacher or Gaussian complexity functions plus an O (1 / √ n ) term, where n is the length of the training set.
This fact also happens in i.i.d.
case [4].
However, because the dependency among samples in Markov chain, the constant in O (1 / √ n ) term is larger than the i.i.d. case.
It follows, in particular, in the example of the voting methods of combining classiﬁers [2], from
Theorem 2, we achieve the following PAC-bound:
[Equation]
with probability at least 1 − α (PAC-Bayes bound), where V ( H ) is the VC-dimension of the class H and C is some positive constant.
3.2 Bounding the generalization error in deep neural networks
In this section, we consider the same example as [4, Section 6].
However, we assume that feature vectors in the dataset are generated by a Markov chain instead of an i.i.d. process.
Let H be a the class of all uniformly bounded functions f : S → R .
H is called the class of base functions .
Consider a feed-forward neural network with l layers of neurons V = { v i } ∪ (cid:83) lj
{+=
+}
V j where V l = { v o } .
The neurons v i and v o are called the input and the output neurons, respectively.
To deﬁne the network, we will assign the labels to the neurons in the following way.
Each of the base neurons is labelled by a function from the base class H .
Each neuron of the j -th layer V j , where j ≥ 1 , is labelled by a vector w := ( w 1 , w 2 , · · · , w m ) ∈ R m , where m is the number of inputs of the neuron.
Here, w will be called the vector of weights of the neuron.
Given a Borel function σ from R into [ − 1 , 1] (for example, sigmoid function) and a vector w := ( w 1 , w 2 , · · · , w m ) ∈ R m where m = |H| + 1 , let
[Equation]
Let σ j : j ≥ 1 be functions from R into [ − 1 , 1] , satisfying the Lipschitz conditions
[Equation]
The neural network works can be formed as the following.
The input neuron inputs an instance x ∈ S .
A base neuron computes the value of the base function on this instance and outputs the value through its output edges.
A neuron in the j -th layer ( j ≥ 1 ) computes and outputs through its output edges the value N σ j ,w ( u 1 , u 2 , · · · , u m ) (where u 1 , u 2 , · · · , u m are the values of the inputs of the neuron).
The network outputs the value f ( x ) (of a function f it computes) through the output edge.
We denote by N l the set of such networks.
We call N l the class of feed-forward neural networks with base H and l layers of neurons (and with sigmoid σ j ).
Let N ∞ :
= (cid:83) ∞
j =1 N j .
Deﬁne H 0 :
= H , and then recursively
[Equation]
Denote H ∞ :
= (cid:83) ∞
j =1 H j .
Clearly, H ∞ includes all the functions computable by feed-forward neural networks with base H .
Let { b j } be a sequence of positive numbers.
We also deﬁne recursively classes of functions computable by feed-forward neural networks with restrictions on the weights of neurons:
[Equation]
where (cid:107) w (cid:107) 1 denotes the 1 -norm of the vector w .
Clearly,
[Equation]
Denote by ˜ H the class of measurable functions ˜ f : S × Y → R , where Y is the alphabet of labels.
˜ H is introduced for real machine learning applications where we need to work with a new Markov chain generated from both feature vectors and their labels { ( X n , Y n ) } ∞ n =1 instead of the feature-based
Markov chain { X n } ∞ n =1 . See Subsection 2.2 for detailed discussions.
For binary classiﬁcation, ˜
H := { ˜ f : f ∈ H} , where ˜ f ( x, y ) = yf ( x ) . Let ϕ be a function such that ϕ ( x ) ≥
I ( −∞ , 0] ( x ) for all x ∈ R and ϕ satisﬁes the Lipschitz condition with constant L ( ϕ ) .
Then, the following is a direct application of Theorem 2.
Theorem 3.
For any t ≥ 0 and for all l ≥ 1 ,
[Equation]
(28) where B n is deﬁned in (19) .
Remark 4.
P { ˜ f ≤ 0 } represents the probability of mis-classiﬁcation in the deep neural network.
Proof.
Let H (cid:48) l :=
H ( b 1 , b 2 , · · · , b l ) .
As the proof of [4, Theorem 13], it holds that
[Equation]
Hence, (28) is a direct application of Theorem 2 and (29).
Now, given a neural network f ∈ N ∞ , let l ( f
[Equation]
For any number k such that 1 ≤ k ≤ l ( f ) , let V k ( f ) be the set of all neurons of layer k in the neural network which is represented by f .
Denote by
[Equation]
where w ( u ) is the coefﬁcient-vector associated with the neuron u in this layer. Deﬁne
[Equation]
where α > 0 is the number
such that ζ ( α ) < 3 / 2 , ζ being the Riemann zeta-function: ζ ( α ) :
= (cid:80) ∞
k =1 k − α . Then, by using Theorem 3 with b k → ∞ and the same arguments as [4, Proof of Theorem 14], we obtain the following result.
See Section 1.3 in the supplement material for more detailed derivations.
Theorem 5.
For any t ≥ 0 and for all l ≥ 1 ,
P (cid:18) ∃ f ∈ H ∞ : P { ˜ f ≤ 0 } > inf
[Equation]
where B n is deﬁned in (19) .
3.3 Generalization Error Bounds on Bayesian Deep Learning
For Bayesian machine learning and deep learning, F := (cid:8) f : S × W → R (cid:9) , where S is the state space of the Markov chain and W is the domain of (random) coefﬁcients.
We assume that S and W are Polish spaces on R , which include both discrete sets and R .
For example, in binary classiﬁcation, f ( X, W ) = sgn( W T X + b ) where the feature X and the coefﬁcient W are random vectors with speciﬁc prior distributions.
In practice, the distribution of W is known which depends on our design method, and the distribution of X is unknown.
For example, W is assumed to be Gaussian in
Bayesian deep neural networks [22].
Since all the bounds on Subsections 3.1 and 3.2 hold for any function f in F at each ﬁxed vector
W = w , hence, they can be directly applied to Bayesian settings where W is random.
However, these bounds are not expected to be tight enough since we don’t use the prior distribution of W when deriving them.
In the following, we use another approach to derive new (and tighter) bounds for Bayesian deep learning and machine learning from all the bounds in Subsections 3.1 and 3.2.
For illustration purposes, we only derive a new bound.
Other bounds can be derived in a similar fashion.
We assume that W 1 , W 2 , · · · , W n are i.i.d.
random variables as in [22].
Let
[Equation]
and deﬁne a new probability measure ˜ P on S × W such that
[Equation]
[Equation] for all (Borel) set A on S ×W .
Here, ˜ π is the stationary distribution of the irreducible Markov process { ( X n , W n ) } ∞ n =1 with stochastic matrix ˜ Q :=
{ Q ( x, w ) P W ( w ) } x ∈S ,w ∈W .
In addition, deﬁne two new (averaging) linear functionals:
[Equation]
and
[Equation]
In practice, the prior distribution of W is known, so we can estimate ˆ P n ( f ) based on the training set { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , · · · , ( X n , Y n ) } , which is a Markov chain on X × Y (cf.
Section 2.2).
The following result can be proved.
Theorem 6.
Let ϕ be a sequence of function such that ϕ ( x ) ≥
I ( −∞ , 0] ( x ) .
For any t > 0 ,
[Equation]
and
[Equation]
Proof.
Let W 1 , W 2 , · · · , W n be an n samples of W ∼ P W on W (or samples of some set of random coefﬁcients).
For simplicity, we assume that { W n } ∞ n =1 is an i.i.d.
sequence.
Then, it is obvious that { ( X n , W n ) } ∞ n =1 forms a Markov chain with probability transition probability
[Equation]
From Theorem 2, it holds that
[Equation] [Equation]
This means that with probability at least 1 − π 2 3 exp( − 2 t 2 ) , it holds that ˜
[Equation]
From (43), it holds that with probability at least 1 − π 2 3 exp( − 2 t 2 ) ,
[Equation]
From (44), with probability at least 1 − π 2 3 exp( − 2 t 2 ) , it holds that
[Equation]
+
[Equation]
From (46), we obtain (38).
Similarly, we can achieve (39).
Extension to High-Order Markov Chains
In this subsection, we extend our results in previous sections to m -order homogeneous Markov chain. The main idea is to convert m -order homogeneous Markov chains to 1 -order homogeneous Markov chain and use our results in previous sections to bound the generalization error. We start with the following simple example. Example 7.
[ m -order moving average process without noise] Consider the following m -order Markov chain X
[Equation] Let Y k := [ X k + m − 1 , X k + m − 2 , · · · , X k ] T . Then, from (47) , we obtain Y k
{++
+}
=
G Y k , ∀ k ∈ Z
+ where
[Equation]
It is clear that { Y n } ∞ n =1 is an order- 1 Markov chain.
Hence, instead of directly working with the m -order Markov chain { X n } ∞ n =1 , we can ﬁnd an upper bound for the Markov chain { Y n } ∞ n =1 .
To derive generalization error bounds for the Markov chain { Y n } ∞ n =1 , we can use the following arguments.
For all f ∈ F and ( x k , x k +1 , · · · , x k + m − 1 ) , by setting ˜ f
( x k , x k +1 , · · · , x k + m − 1 ) =
f ( x k ) where ˜ f : S m → R , we obtain
[Equation]
Hence, by applying all the results for 1 -order Markov chain { Y n } ∞ n =1 , we obtain corresponding upper bounds for the sequence of m -order
Markov chain { X n } ∞ n =1 .
This approach can be extended to more general Markov chain. See Section 4 in the Supplementary Material for details.
Conclusions
In this paper, we derive upper bounds on generalization errors for machine learning and deep neural networks based on a new assumption that the dataset has Markov or hidden Markov structure.
We also propose a new method to convert all these bounds to Bayesian deep learning and machine learning.
Extension to m -order Markov chains and a mixture of Markov chains are also given.
An interesting future research topic is to develop some new algorithms to evaluate performance of these bounds on real Markov datasets.