R OBUST G ENERALIZATION OF Q UADRATIC N EURAL N ETWORKS VIA F UNCTION I DENTIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
A key challenge facing deep learning is that neural networks are often not robust to shifts in the underlying data distribution.
We study this problem from the perspective of the statistical concept of parameter identiﬁcation.
Generalization bounds from learning theory often assume that the test distribution is close to the training distribution.
In contrast, if we can identify the “true” parameters, then the model generalizes to arbitrary distribution shifts.
However, neural networks are typically over-parameterized, making parameter identiﬁcation impossible.
We show that for quadratic neural networks, we can identify the function represented by the model even though we cannot identify its parameters.
Thus, we can obtain robust generalization bounds even in the over-parameterized setting.
We leverage this result to obtain new bounds for contextual bandits and transfer learning with quadratic neural networks.
Overall, our results suggest that we can improve robustness of neural networks by designing models that can represent the true data generating process.
In practice, the true data generating process is often very complex; thus, we study how our framework might connect to neural module networks, which are designed to break down complex tasks into compositions of simpler ones.
We prove robust generalization bounds when individual neural modules are identiﬁable.
I NTRODUCTION
Recent work has demonstrated that neural networks are not robust to shifts in the underlying data, including both distribution shifts (i.e., where the data comes from a new distribution independent of the neural network parameters)
(Hendrycks & Dietterich, 2019; Taori et al., 2020) as well as adversarial shifts (i.e., where the shift can depend on the neural network parameters) (Szegedy et al., 2013).
Accordingly, there has been a great deal of interest in better understanding why neural networks fail to be robust (Tsipras et al., 2018; Ilyas et al., 2019) and on improving robustness (Goodfellow et al., 2014; Raghunathan et al., 2018;
Cohen et al., 2019).
From the perspective of learning theory, there is little reason to expect neural networks to be robust, since generalization bounds typically assume that the test examples are from the same distribution as the training examples.
PAC-Bayesian generalization bounds allow for a limited amount of robustness, but only if the support of the target distribution q is contained in that of the source distribution p , since it requires that the KL divergence D KL ( q (cid:107) p ) is ﬁnite.
Yet, distribution shifts (Hendrycks &
Dietterich, 2019) often shift probability mass to inputs completely outside the source distribution.
Instead, the reason we might expect neural networks to be robust to these shifts is that humans are robust to them; for instance, small pixel-level shifts considered in adversarial examples are typically unnoticeable to humans, yet these shifts can move the image completely off of the distribution of natural images.
This fact indicates a gap in our theoretical understanding of neural networks.
In particular, the key question is understanding settings under which we may expect neural networks to be robust to distribution shifts that are “large” (e.g., in terms of KL divergence).
We study a strategy for closing this gap based on the statistical concept of identiﬁability (Hsu et al., 2012).
At a high level, this concept assumes that the true model belongs to the model family; then, in the limit of inﬁnite training data, the learning algorithm can exactly recover the parameters of the true model.
For instance, in linear regression, the data is generated according to the model
y = (cid:104)
θ ∗ , x (cid:105) + ξ , where ξ is σ -subgaussian noise.
Then, under mild assumptions on the training data Z =
( X, Y ) , the ordinary least squares (OLS) estimator
ˆ θ ( Z ) recovers the true parameter—i.e., in the limit of inﬁnite data, ˆ θ ( Z ) =
θ ∗ .
With ﬁnite samples, OLS satisﬁes high-probability convergence rates of the form
[Equation]
The connection to robustness is that if (1) holds, then for any input x such that (cid:107) x (cid:107) 2 ≤ x max , we have
[Equation]
∗
[Equation]
Thus, for any distribution q ( x ) with support on B 2 (0 , x max ) = { x ∈ X | (cid:107) x (cid:107) 2 ≤ x max } , ˆ θ ( Z ) obtains bounded error—i.e., we have
E q ( x ) [( (cid:104) ˆ θ ( Z ) , x (cid:105) − (cid:104)
[Equation]
∗
[Equation]
x (cid:105) )
[Equation] [Equation]
(cid:15) 2 x 2 max with high probability.
Thus, a natural question is whether we can obtain similar kinds of parameter identiﬁcation bounds for neural networks.
A key complication is that practical neural networks are often over-parameterized , possibly to facilitate optimization (Du & Lee, 2018; Jacot et al., 2018).
In this setting, identiﬁcation is impossible, since multiple parameters can yield the same model.
Nevertheless, it may be possible to obtain bounds of the form (2)—in particular, even if we do not recover the true parameters θ ∗ , we can still recover the function f θ ∗ ( x ) .
We refer to this notion as function identiﬁcation .
Furthermore, we show that quadratic neural networks satisfy function identiﬁcation bounds under mild conditions.
To demonstrate its utility, we show how function identiﬁcation can be leveraged to obtain regret guarantees for a bandit (Rusmevichientong & Tsitsiklis, 2010) where each arm is a quadratic neural network.
Linear bandits fundamentally involve covariate shift since their “covariates” are arms, which are adaptively chosen through the learning process as a function of past observations; thus, existing approaches have all operated in the setting where there is a unique and identiﬁable global minimizer.
Similarly, we build on recent work proving bounds on transfer learning in the setting of bounded label shift and unbounded covariate shift (Bastani, 2020; Xu et al., 2021); again, we show that we can leverage function identiﬁcation to easily transfer learn quadratic neural networks.
Our results suggest that one strategy for improving robustness of neural networks is to design models that can represent the true data generating process.
However, doing so can be challenging due to the complexity of most real-world data generating processes mapping covariates to labels—e.g., mapping natural language to semantic meaning or images to object detections and labels.
As a consequence, we study how our results can connect to neural module networks (Andreas et al., 2016), which are designed to break down complex tasks into smaller ones that can each be solved by a neural network.
For instance, we may break down the task “count the number of red balls in image x ” into (i) detecting balls, (ii) detecting red objects, and (iii) intersecting the two sets, and (iv) summing the results.
Intuitively, neural modules can generalize more robustly since (i) it is more likely that an individual neural module designed to solve a simple task can be identiﬁed from training data, and (ii) even if module composition is not itself identiﬁable, shifts in the compositional structure tend to be smaller than shifts in the underlying data distribution.
We study a simpliﬁed form of neural module networks, where modules are quadratic neural networks composed in sequence according to a given input; for simplicity, we assume they can be trained in a supervised way, ensuring robust generalization.
Then, we show that under certain conditions, compositions of these models are also robust, including the case where there are shifts in the distribution over compositions.
Related work.
Prior work has connected misspeciﬁcation (i.e., the true model is in the model family) and robustness to covariate shift (Shimodaira, 2000; Wen et al., 2014); however, having a correctly speciﬁed model is insufﬁcient if the true parameters are not identiﬁable—e.g., in linear regression, if the covariance matrix Σ
= E p ( x ) [ xx (cid:62) ] is singular, then θ is not identiﬁable; thus, the estimated model may not be robust.
Quadratic neural networks cannot be identiﬁed even if the model is correctly speciﬁed since the parameters have a continuous symmetry (i.e., orthogonal transformations).
Recent work has studied learning under adversarial examples (Goodfellow et al., 2014; Raghunathan et al., 2018;
Cohen et al., 2019) and corrupted training data (Steinhardt et al., 2017; Diakonikolas et al., 2019).
In contrast, we are interested in robustness to covariate shift; there has been recent work empirically showing that neural networks are sensitive to distribution shift (Hendrycks & Dietterich, 2019; Taori et al., 2020; Ruis et al., 2020;
Ribeiro et al., 2020; Koh et al., 2020).
Distributionally robust optimization enables training of models robust to small shifts (Duchi & Namkoong, 2018), but we are interested in potentially large shifts.
Unsupervised domain adaptation (Ben-David et al., 2007;
Blitzer et al., 2008) learns a model on a covariate shifted target distribution; however, they rely on unlabeled examples from the target domain, whereas we do not.
There has been recent theory on robustness to adversarial perturbations—e.g., showing there may be a tradeoff between robustness and on-distribution generalization (Tsipras et al., 2018), and that non-robust algorithms tend to learn predictive but brittle representations compared to adversarially robust ones (Ilyas et al., 2019).
In contrast, we show that these tradeoffs are mitigated when the true model function can be identiﬁed despite over-parameterization.
Furthermore, adversarial shifts are typically bounded (e.g., small (cid:96) ∞ norm), whereas the shifts we consider may be large.
There has been a great deal of recent work on deep learning theory, including on quadratic neural networks; however, it has largely focused on optimization (Ge et al., 2017b;
Jacot et al., 2018;
Du et al., 2019; Gao et al., 2019; Soltanolkotabi et al., 2018; Li et al., 2018), and on-distribution generalization (Neyshabur et al., 2017; Du & Lee, 2018; Jacot et al., 2018; Arora et al., 2018; Long & Sedghi, 2019).
In contrast, we are interested in out-of-distribution generalization.
We discuss additional related work on matrix factorization and multi-armed bandits in Appendix A, as well as a discussion of the novelty of our results.
P ROBLEM F ORMULATION
We consider a model f θ : X → Y , with covariates X ⊆ R d , labels Y ⊆ R , and parameters θ ∈ Θ ⊆
R m .
A generalization bound from learning theory typically has the form
[Equation]
where
[Equation]
where (cid:15), δ ∈ R > 0 , Z = { ( x 1 , y 1 ) , ..., ( x n , y n ) } ⊆
X × Y with y i =
f θ ∗ ( x i ) + ξ i is a training set of i.i.d.
observations from a distribution p (i.e., p ( Z ) =
p ( x 1 , y 1 ) · ... · p ( x n , y n ) ), ξ i is bounded random noise independent of x i with | ξ i | ≤ ξ max , θ ∗ ∈
Θ are the true parameters, and ˆ θ ( Z ) =
arg min θ ∈ Θ
ˆ L ( θ ; Z ) where ˆ L ( θ ; Z ) = 1 n n (cid:88)
i =
( f θ ( x i ) − y i )is an estimator based on the training data Z .
In particular, they assume that the training inputs x i ∼ p are i.i.d.
samples from the same distribution as the test example x ∼ p .
Deﬁnition 2.1.
The model f θ and distribution p satisfy function identiﬁcation if for any (cid:15), δ ∈ R > 0 , we have P p ( Z ) [ ∀ x ∈ X .
( f ˆ θ ( Z ) ( x ) − f θ ∗ ( x )) 2 ≤ (cid:15) ] ≥ 1 − δ for n = |
Z
| sufﬁciently large.
Function identiﬁcation implies generalization bounds even when the test data comes from a different distribution q .
In particular, we say f θ robustly generalizes if for any q with support on X , we have
[Equation]
where the difference from (3) has been highlighted in red.
It is easy to see that function identiﬁcation implies (4).
Note that the true model f θ ∗ does not change, so there is no label shift.
F UNCTION I DENTIFICATION OF Q UADRATIC N EURAL N ETWORKS
Traditional statistical bounds on parameter identiﬁcation can provide guarantees for arbitrary covariate shift.
In particular, suppose we have a bound of the form
[Equation]
and assume that the model family f θ is K -Lipschitz continuous in θ ; then, we have
[Equation]
with probability at least 1 − δ according to p ( Z ) .
In particular, this bound holds for any covariate distribution q .
Our goal is to extend these techniques to quadratic neural networks, which are over-parameterized so we cannot identify the true parameters θ ∗ —i.e., (5) does not hold.
3. Q UADRATIC N EURAL N ETWORKS
We consider a neural network f θ , where θ ∈ R d × k , with a single hidden layer with k neurons—i.e., f θ ( x ) = (cid:80) kj =1 a j · σ ( (cid:104) θ j , x (cid:105) ) .
We consider the over-parameterization case where k can be much larger than d .
Following prior work (Du & Lee, 2018), we assume that f θ has quadratic activations and output weights equal to one—i.e., σ ( z ) =
z 2 and a j = 1 for each j ∈ [ k ] , so we have
[Equation]
We assume the true (training) loss is the mean-squared error L p ( θ ) =
E p ( x ) [( f θ ( x ) − f θ ∗ ( x )) 2 ] , and we consider a model trained using an empirical estimate of this loss on the training dataset:
[Equation]
where ˆ L (
[Equation]
Z
[Equation]
Now, our goal is to obtain a bound of the form (6); to this end, we assume the following:
Assumption 3.1.
(cid:107) x (cid:107) 2 ≤ x max and (cid:107) θ (cid:107) F ≤ θ max .
Assumption 3.2.
There exists α ∈ R > 0 such that E p ( x ) [( x (cid:62) ∆ x ) 2 ] ≥ α (cid:107) ∆ (cid:107) 2 F for any symmetric ∆ ∈ R d × d .
Our second assumption is standard; in particular, it is closely related to the assumption in linear regression that the minimum eigenvalue of the covariance matrix is lower bounded—i.e., Σ = E p ( x ) [ xx (cid:62) ] (cid:31) 0 .
As an example, when x is i.i.d.
uniform in each component, e.g., p ( x ) =
(cid:81) di =1 Uniform ( x i ; [ − 1 / 2 , 1 / 2]) , then we can take α = 1 / 180 ; we give a proof in Appendix B.1.
3. R OBUST G ENERALIZATION
Our approach leverages the fact that f θ ( x ) = x (cid:62) ( θθ (cid:62) )
x ; thus, f θ resembles a matrix factorization model.
Recent work has leveraged this connection to translate matrix factorization theory to quadratic neural networks (Du & Lee, 2018).
We let g ( θ ) =
θθ (cid:62) and ˜ f
φ ( x ) =
x (cid:62) φx , where φ ∈ Φ ⊆ R d × d , in which case f θ ( x ) =
˜ f
g ( θ ) ( x ) ; in addition, we deﬁne ˜
L p ( φ ) =
E p ( x ) [( ˜ f φ ( x ) − ˜ f
φ ∗ ( x )) 2 ] , where φ ∗ =
g ( θ ∗ ) , and ˆ˜ L ( φ ; Z ) = n − 1 (cid:80) ni
=1 ( ˜ f φ ( x i ) − y i ) 2 , so L p ( θ ) =
˜
L p ( g ( θ )) and ˆ L ( θ ; Z ) =
ˆ˜ L ( g ( θ ); Z ) .
Finally, we assume that (cid:107) φ (cid:107) F ≤ φ max ; in general, we have φ max ≤ θ 2 max by Assumption 3.1.
We begin by stating several lemmas establishing the properties needed for function identiﬁcation.
Our ﬁrst lemma says that the loss is strongly convex in φ .
Lemma 3.3.
Under Assumption 3.2, the loss ˜
L p ( φ ) is 2 α -strongly convex in φ .
We give a proof in Appendix B.2.
Our next lemma says that our model family is Lipschitz in φ .
Lemma 3.4.
Under Assumptions 3.1 & 3.2, ˜ f φ and ˜ L are K -Lipschitz in φ , where K = 4 φ max x 4 max .
We give a proof in Appendix B.3.
Our ﬁnal lemma says that our estimate of the loss function is a uniformly good approximation of the true loss.
Lemma 3.5.
Under Assumptions 3.1 & 3.2, for any δ ∈ R > 0 , we have
[Equation]
where σ ( Z ) = n − 1 (cid:80) ni
=
ξ 2 i , and letting (cid:96) max = 2 x 2 max φ max be an upper bound on | f θ ( x ) − f θ ∗ ( x ) | ,
[Equation]
We give a proof in Appendix B.4.
Note that (cid:15) → 0 as n → ∞ .
Next, we prove our main result, which says that quadratic neural networks can be functionally identiﬁed.
Theorem 3.6.
Under Assumptions 3.1 & 3.2, we have
[Equation]
Proof.
By Lemma 3.3, and since ∇ φ ˜ L ( g ( θ ∗ ))
= 0 , we have
[Equation]
Next, by Lemma 3.5 and the fact that ˆ θ minimizes ˆ L ( θ ; Z ) , we have
[Equation]
with probability at least 1 − δ .
Combining (8) and (9), we have
[Equation]
with probability at least 1 − δ .
Finally, by Lemma 3.4, we have
[Equation]
with probability at least 1 − δ , as claimed.
As a result, we provide a robust generalization error bound for quadratic neural networks with potential distribution shifts.
Corollary 3.7.
Under Assumptions 3.1 & 3.2, for any distribution q ( x ) with support on B 2 (0 , x max ) ,
[Equation]
Finally, we also prove that gradient descent can ﬁnd the global minima of ˆ L ( θ ; Z ) , which ensures that gradient descent can perform function identiﬁcation in practice; we give a proof in Appendix B.5.
Proposition 3.8.
All local minima of ˆ L ( θ ; Z ) are also global minima.
Q UADRATIC N EURAL B ANDITS
A key application of robust generalization bounds is to parametric bandits; this is because, in bandit learning, the distribution of inputs x used to estimate ˆ θ
≈ θ ∗ can differ from the distribution under which f ˆ θ is used.
Thus, generalization bounds based on notions such as Rademacher complexity cannot be used.
Unlike prior literature in bandits, we consider an over-parameterized function that does not admit a unique solution; in contrast, recent work on neural tangent kernel bandits (Zhou et al., 2020) assumes that there is a unique, identiﬁable solution.
Note that this assumption cannot hold for quadratic neural networks because they are invariant to transformations such as rotations.
We consider a standard linear bandit (Rusmevichientong & Tsitsiklis, 2010; Abbasi-Yadkori et al., 2011) with a ﬁxed horizon T ∈ N , but where the expected reward is parameterized by a quadratic neural network instead of a linear function.
At each time step t , the algorithm chooses among a continuum of actions x t ∈ X , and receives a reward
[Equation]
where θ ∗ ∈ R
d × k is an unknown parameter matrix, and ξ t are bounded i.i.d.
random variables.
For simplicity, we assume that X = B 2 (0 , 1) is the unit ball.
Then, our goal is to bound the regret
[Equation]
where
[Equation]
We make the following assumption, which says that φ ∗ =
θ ∗ θ ∗(cid:62) has a gap in its top eigenvalue:
Initialize Z ← ∅
Let m be as in (11) for t ∈ { 1 , ..., m } do
Sample i.i.d.
action x t ∼ p , where p is as in (12)
Take action x t and obtain reward y t as in (10)
Update Z ← Z ∪ { ( x t , y t ) } end for
Compute ˆ θ = arg min θ ˆ L ( θ ; Z ) , where ˆ L ( θ ; Z ) = m − 1 (cid:80) mi
=1 ( f θ ( x i ) − y i )
Compute ˆ x = arg max x ∈X f ˆ θ ( x ) for t ∈ { m + 1 , ..., T } do
Take action x t = ˆ x and obtain reward y t as in (10) end for end procedure
Assumption 4.1.
Let φ ∗ =
θ ∗ θ ∗(cid:62) , and let λ 1 ≥ λ 2 ≥ ... ≥ λ
d be the eigenvalues of φ ∗ .
There exists a constant M ∈ R > 0 such that λ 1 − λ 2 ≥ 4 /M .
This assumption ensures that the eigenvectors of φ ∗ to be stable under perturbations.
The eigenvectors of φ ∗ correspond to the optimal action x ∗ = arg max x ∈X f θ ∗ ( x ) since f θ ∗ ( x ) = x (cid:62) φ ∗ x ; thus, this assumption ensures that if ˆ θ ≈ θ ∗ , then the optimal action ˆ x = arg max x ∈X f ˆ θ ( x ) satisﬁes ˆ x ≈ x ∗ .
Next, we describe our algorithm, summarized in Algorithm 1.
We consider an explore-then-commit strategy for simplicity, since it already achieves the asymptotically optimal regret rate (Rusmevichientong & Tsitsiklis, 2010); our approach can similarly be applied to more sophisticated algorithms such as UCB (Abbasi-Yadkori et al., 2011) and Thompson sampling (Agrawal & Goyal, 2013).
Our algorithm proceeds in two stages: (i) the exploration stage (for t ∈ { 1 , ..., m } ), and (ii) the exploitation stage (for t ∈ { m + 1 , ..., T } ), where
[Equation]
In the exploration stage, we randomly choose actions x t ∼ p , where
[Equation]
Note that (cid:107) x (cid:107) 2 ≤ 1 for x in the support of p , so x t ∈ X .
Following the discussion in Section 3, for this choice of p , Assumption 3.2 holds for the dataset Z with α = 4 / (45 d 2 ) .
Next, we compute an estimate ˆ θ of θ ∗ based on the data Z collected so far, and compute the optimal action ˆ x assuming ˆ θ are the true parameters.
Then, in the exploitation stage, we always use action ˆ x .
The key challenge providing theoretical guarantees using traditional generalization bounds is handling the optimization problem over x ∈ X used to compute ˆ x .
Since ˆ x is not sampled from the distribution p , traditional bounds do not provide any guarantees about the accuracy of f ˆ θ (ˆ x ) compared to f θ ∗ (ˆ x ) .
In contrast, Theorem 3.6 provides a uniform guarantee, and can therefore be used to bound the regret.
Theorem 4.2.
Under Assumptions 3.1, 3.2 & 4.1, the expected regret of Algorithm 1 is
[Equation]
where C 0 and C 1 do not depend on T (see Appendix C).
In particular, we have R ( T ) =
˜ O ( T 2 / 3 ) ; we give a proof in Appendix C. Note that this is worse than the usual ˜ O ( √ T ) regret because Theorem 3.6 only admits a n 1 / 4 convergence rate.
T RANSFER L EARNING OF Q UADRATIC N EURAL N ETWORKS
So far, we have considered shifts in the covariate distribution but not in the label distribution.
Now, we consider a transfer learning problem where there is additionally a small shift in the labels.
In particular, we assume we have proxy data Z p ⊆
X × Y from the source domain of the form y p,i =
f θ ∗ p ( x p,i ) + ξ p,i (for i ∈ [ n p ] ), where θ ∗ p ∈ Θ are the proxy parameters and p ( x p ) is the source covariate distribution, along with gold data Z g ⊆
X × Y from the target domain of the form y g,i =
f θ ∗ g ( x g,i ) + ξ g,i (for i ∈ [ n g ] ), where θ ∗ g ∈ Θ are the gold parameters and q ( x g ) is the target covariate distribution.
We are interested in the setting n p (cid:29) n g , and where (cid:107) θ ∗ g − θ ∗ p (cid:107) F ≤ B is small.
We consider a two-stage estimator (Bastani, 2020) that ﬁrst computes an estimate of the proxy parameters ˆ θ p = arg min θ ∈ Θ ˆ L ( θ ; Z p ) , and then computes an estimate of the gold parameters in a way that is constrained towards the proxy parameters.
First, note that we have
P
[Equation]
Z
[Equation]
where
[Equation]
where we have highlighted the differences from (cid:15) in (7) in red.
Next, we make a technical assumption:
Assumption 5.1.
For some σ 0 ∈ R > 0 , σ min ( θ ∗ p ) ≥ σ 0 , where σ min ( θ ) is the d th singular value of θ .
Equivalently, the minimum eigenvalue of g ( θ ∗ p ) is positive; intuitively, this assumption ensures a good estimate of θ ∗ p θ ∗(cid:62) p implies a good estimate of θ ∗ p (up to an orthogonal transformation).
Then, letting ˆ
[Equation]
=
[Equation]
+
[Equation]
σ
[Equation]
α be an expanded radius to account for error in our estimate of ˆ θ p , we use the estimator
[Equation]
where
[Equation]
Note that we have assume ˆ B is known; in practice, this constraint can be included as an additive regularization term.
Intuitively, this formulation mirrors transfer learning algorithms based on ﬁnetuning—i.e., initializing the parameters to the proxy data θ p and then taking a small number of steps of stochastic gradient descent (SGD) on the gold data θ g .
In particular, SGD can be interpreted as Lregularization on the parameters (Ali et al., 2020), so ﬁne-tuning L 2 -regularizes ˆ θ g towards θ p .
Theorem 5.2.
Under Assumptions 3.1, 3.2 & 5.1, for any q ( x ) with support on B 2 (0 , x max ) ,
[Equation]
where
[Equation]
+ log 4 δ (cid:19) .
Thus, if B is small and n p is large, f ˆ θ g is accurate even if n g is small; we give a proof in Appendix D.
G ENERALIZATION B OUNDS FOR N EURAL M ODULE N ETWORKS
While function identiﬁcation enables robust generalization, many data generating processes are too complex to be identiﬁable.
Neural module networks are designed to break complex prediction problems into smaller tasks that are individually easier to solve.
These models take two kinds of input:
(i) a sequence of tokens w (e
g., word embeddings) indicating the correct composition of modules,
and (ii) the input x to the modules.
Then, the model predicts the sequence of modules j 1 ...
j T based on w , and runs the modules in sequence to obtain output x (cid:48)
=
f j T ( ... ( f j 1 ( x )) ... ) .
We study conditions under which neural module networks can robustly generalize.
Rather than study arbitrary distribution shifts, we consider two separate shifts:
• Module inputs: We assume that the individual modules are identiﬁable;
as a
consequence,
we
assume the shift to the module input x can be arbitrary.
• Module composition: We consider shifts to the token sequence w .
If the model mapping w to j 1 ...
j T is identiﬁable, then the entire model is identiﬁable.
Instead, we show that when this model is not identiﬁable, compositional structure can still aid generalization.
Intuitively, we show that while small shifts in the compositional structure can cause large shifts in the distribution p ( w ) , models that leverage the structure of p ( w ) can still generalize well.
In more detail, consider a simpliﬁed neural module network f , which includes (i) a set of neural modules { f j : X → X} kj =1 , and (ii) a parser g : Z T → [ k ] T , where Z ⊆ R r , with model class g ∈ G .
We assume each component of f j ( x ) is computed by a separate quadratic neural network; we discuss the architecture of g below.
Then, given an input x ∈ X ⊆ R d and w ∈ W =
Z T , the corresponding neural module network f : X × W
→ X is deﬁned by
[Equation]
where
[Equation]
We assume that g has compositional structure—i.e., for some ˜ g :
[ k ] × Z → [ k ] , we have g ( w ) =
j 1 ...j T where j t
[Equation]
where w = z 1 ... z T .
Intuitively, w is a sequence of word vectors; then, the current neural module j t = ˜
g ( z t , j t − 1 ) depends both on the current word vector z t and the previous neural module j t − 1 .
First, we assume that the individual modules have been functionally identiﬁed.
We assume we have fully labeled data we can use to train the neural modules—i.e., for each input x and sequence w , we have both the desired sequence j 1 ...
j T of neural modules, as well as the entire execution x 0 , x 1 , ..., x T , where x 0 = x and x t +1 =
f
j t ( x t ) otherwise.
Thus, we can use supervised learning to train the neural modules; 2 in particular, we can construct labeled examples ( j t − 1 , z t , j t ) used to train the parser ˜ g , and labeled examples ( x t , x t +1 ) to train the modules f j t .
For simplicity, we assume we have a uniform lower bound n on the number of training examples for the parser and for each module.
Then, we have the following straightforward result:
Lemma 6.1.
Under Assumptions 3.1 & 3.2, with probability at least 1 − dkδ , for each j ∈ [ k ] ,
[Equation]
where ˆ f j is the estimated module and f ∗ j is the ground truth module.
This result follows straightforwardly from Theorem 3.6 along with a union bound.
In contrast, we do not assume the parsing model robustly generalizes, but only on distribution.
For the subsequent analysis, we can use any neural network models that satisfy the statement of Lemma 6.1.
Lemma 6.2.
Under Assumptions 3.1 & 3.2, with probability at least 1 − δ , we have
[Equation]
where R n ( G ) is the Rademacher complexity of G (including its loss function) [Equation] where p ( z, j ) =
T − 1 (cid:80) T t =1 p t ( z, j ) [Equation] and where
[Equation]
otherwise .
This result is a standard Rademacher generalization bound (Bartlett & Mendelson, 2002).
Note that we have also assumed that the distribution over token sequences is structured, which is necessary for our compositional implementation of g to generalize, even on distribution.
Intuitively, the distribution over ( z, j ) consists of both a unigram model over the word vectors:
[Equation]
where we deﬁne ˜ p ( z 1 | z 0 )
=
˜ p
( z 1 ) , as well as a unigram model over neural modules:
[Equation]
Next, we consider a shifted distribution ˜ q ( z | z (cid:48) ) , which is close to ˜ p
( z | z (cid:48) ) .
Assumption 6.3.
We have (cid:107) ˜
q
( · | z (cid:48) )
− ˜ p ( · | z (cid:48) )
(cid:107) TV ≤ α .
Importantly, despite this assumption, the shift between the overall distributions p ( z 1 , ..., z T ) and q ( z 1 , ..., z T ) can still be large since it compounds exponentially across
the steps t ∈ [ T ] .
Proposition 6.4.
There exist p and q that satisfy Assumption 6.3, but (cid:107) p − q (cid:107) TV =
2(1 − (1 − α/ 2) T ) .
That is, even if the single step probabilities ˜ p
( z | z (cid:48) ) and ˜ q ( z | z (cid:48) ) have total variation (TV) distance bounded as in Assumption 6.3, the overall distributions p and q can have TV distance exponentially close to the maximum possible distance of 2 in T ; we give a proof in Appendix E.1.
We show that neural module networks generalize since ˆ g leverages the compositional structure of p .
First, we show that under Assumption 6.3, the overall shift in the input distribution of ˆ˜ g is bounded:
Lemma 6.5.
Under Assumptions 3.1, 3.2 & 6.3, we have (cid:107) ˜ q − ˜ p (cid:107) TV ≤
Tα , where ˜ p is deﬁned in
Lemma 6.2 and ˜ q is deﬁned in Assumption 6.3.
That is, while the shift can compound across steps t , it does so only linearly; we give a proof in
Appendix E.2.
Next, we show that as a consequence, the error of ˆ g is bounded.
Lemma 6.6.
Under Assumptions 3.1, 3.2 & 6.3, and assuming that P p ( z,j ) [ˆ˜ g ( z, j ) (cid:54) =
˜
g ∗ ( z, j )] ≤ (cid:15) g , we have that P p ( w ) [ˆ g ( w ) (cid:54) =
g ∗ ( w )] ≤ T(cid:15) g .
We give a proof in Appendix E.3.
Finally, we have our main result.
Theorem 6.7.
Under Assumptions 3.1, 3.2 & 6.3, with probability at least 1 − ( dk + 1) δ , we have [Equation]
[Equation]
We give a proof in Appendix E.4.
Intuitively, Theorem 6.7 says that the error of the neural module network is linear in T as long as K ≤ 1 .
Note that even if there is no distribution shift, its error is
[Equation]
by the same argument as the proof of Theorem 6.7.
The exponential dependence on K is unavoidable since K > 1 says that the modules f j can expand the input, which leads to exponential blowup in the magnitude of the output as a function of T , which also makes the estimation error exponential in T .
Thus, the only cost to the distribution shift from p to q is the additional error probability T 2 α .
C ONCLUSION
We have presented a number of results demonstrating that over-parameterization does not fundamentally harm learning models that are robust to arbitrary distribution shifts.
In particular, even though we can no longer identify the true parameters for quadratic neural networks, we show that we can identify the true function, thereby enabling us to prove new results in bandits and transfer learning.
Finally, we provide preliminary analysis extending these results to neural module networks to handle complex data generating processes.
A limitation of our work is that it is specialized to quadratic neural networks; a key direction for future work is to explore how our results extend to other activation functions and deeper architectures.
A A DDITIONAL R ELATED W ORK
Low-rank matrix factorization.
Our notion of functional identiﬁcation for quadratic neural network is related to the low-rank matrix factorization literature.
However, they impose a low-rank structure on the matrix to recover, and hence typically require extra conditions to identify the matrix—e.g., the restricted isometry property (RIP) (Candes & Plan, 2011; Ge et al., 2017a), or bounded (cid:96) 2 norm of noise vector (Kabanava et al., 2016).
In contrast, we consider a more general case and do not assume any underlying structure of the matrix; in particular, since our goal is to capture over-parameterization of neural networks, our matrix is usually decomposed as φ = θθ (cid:62) , where θ ∈ R d × k and k ≥ d (and φ is not necessarily low-rank).
Also, we study the prediction error of neural networks in the presence of distribution shifts, whereas the goal of the low-rank literature is to recover the true matrix.
Multi-armed bandits.
Prior literature on parameterized bandits has considered a number of functional forms, ranging from linear (Abbasi-Yadkori et al., 2011; Rusmevichientong & Tsitsiklis, 2010) to neural tangent kernels (Zhou et al., 2020).
Most of this work makes a realizability assumption that the model family contains the true model; 3 implicitly, they consider model families where there is a unique , identiﬁable true parameter.
These assumptions are necessary precisely due to the fact that the test and training distributions are different; thus, much of the bandit literature has focused on proving parameter identiﬁcation results to enable learning.
In contrast, the identiﬁability assumption does not hold for quadratic neural networks because they are invariant to parameter transformations.
To the best of our knowledge, we consider the ﬁrst over-parameterized bandit problem that considers a model that is not parameter-identiﬁable; we ﬁnd that similar regret results hold as long as the function represented by the model can be identiﬁed.
Separately, Foster & Rakhlin (2020) makes a general connection between online regression oracles and the regret of a bandit algorithm; however, their approach only provides good guarantees when the regression oracle returns a model that generalizes off-distribution.
Finally, recent work on UCB with neural tangent kernels (Zhou et al., 2020) provides general regret bounds, but their bound is only sublinear under conditions such as the true reward function having small RKHS norm (see Remark 4.8 in their paper), which amounts to assuming they can recover the true parameters.
Novelty . We brieﬂy discuss the novelty of our results compared to existing work. First, the results in
Section 3 are novel.
To the best of our knowledge, the proof strategy in our main result, Theorem 3.6, is novel, though we note that the preceding lemmas are based on standard arguments—e.g., bounding the convexity of ˜ L p ( φ ) (Lemma 3.3) and the Lipschitz constant (Lemma 3.4) of ˜ f φ ; also, Lemma 3. relies on a standard covering number argument. For our applications to bandits and transfer learning, our key novel results are Lemma C.3 for bandits, which proves smoothed bounded response for quadratic neural networks, and Lemma D.1 for transfer learning. Finally, to the best of our knowledge, our arguments in Section 6 are novel.
B P ROOFS FOR S ECTIONB. P ROOF OF M INIMUM E IGENVALUE FOR U NIFORM D ISTRIBUTION
In this section, we prove the claim that Assumption 3.2 holds for the covariate distribution where x i is an i.i.d.
random variable with distribution Uniform ( x i ; [ − 1 / 2 , 1 / 2]) .
To this end, note that
[Equation]
p
[Equation]
x
[Equation]
[Equation]
x
[Equation]
x
[Equation]
as claimed.
B. P ROOF OF L EMMA 3.
We use the notation U : ∇ 2 f ( φ ) : V to denote the matrix inner product (cid:104) U, ∇ 2 f ( φ )( V ) (cid:105) for
U, V ∈ R d × d .
The Hessian ∇ 2 f ( φ ) can be viewed as a d 2 × d 2 matrix.
As everything here is bounded, we can exchange the expectation and differentiation.
Therefore, the Hessian of our loss function has for any symmetric matrix ∆
[Equation]
where the last inequality uses Assumption [Equation]
B. P ROOF OF L EMMA 3.
By our deﬁnition
[Equation]
for any φ
[Equation]
φ (cid:48) ∈ Φ ,
[Equation]
Given our quadratic loss function
[Equation]
we have
[Equation]
Next, the true loss satisﬁes
[Equation]
Finally [Equation] the empirical loss satisﬁes
[Equation]
as claimed.
B. P ROOF OF L EMMA 3.
First, we have the following results:
Lemma B.1 (Covering Number of Euclidean Ball) .
For a Euclidean ball in R n 1 × n 2 with radius R with respect to Frobenius norm, there exists an (cid:15) -net
E such that
[Equation]
Proof.
This claim follows by a direct application of Proposition 4.2.12 in Vershynin (2018) [Equation]
Lemma B.2 (Hoeffding’s Inequality for Subgaussian Random Variables) .
Letting { z i } ni =1 be a set of independent σ -subgaussian random variables [Equation] then for all t ≥ 0 , we have
[Equation]
Proof.
See Proposition 2.1 of Wainwright (2016).
Now [Equation] we prove Lemma 3.5.
Consider an (cid:15)/ (4 K ) -net
E .
Then
[Equation]
for any φ ∈ Φ [Equation] there exists φ (cid:48)
∈ E such that
[Equation]
Therefore [Equation] we have
[Equation]
[Equation]
Now, deﬁning ¯˜ L
[Equation]
φ ; Z
[Equation]
n n [Equation] i = ( ˜ f φ [Equation]
x i ) − ˜ f [Equation] ∗
[Equation] x i ))and ˜ η [Equation] n
[Equation] i = ( ˜ f φ ( x i ) − ˜ f φ ∗ ( x [Equation]
[Equation] ξ i , and recalling that σ ( Z ) = n − 1 (cid:80) n i =1 ξ
i , then we have ˆ˜ L
[Equation]
φ
[Equation]
Z ) = ¯˜ L ( φ ; Z ) + 2˜ η ( φ ; Z ) + σ ( Z ) .
Thus, continuing from (13), we have (cid:88)
[Equation]
For the ﬁrst term in (14), note that | ( ˜ f φ ( x ) − ˜ f
φ ∗ ( x )) 2 | ≤ (cid:96) 2 max , so ( ˜ f φ ( x ) − ˜ f
φ ∗ ( x )) 2 is (cid:96) 2 maxsubgaussian; thus, by Lemma B.2, we have
[Equation]
Next, for the second term in (14), note that | ( ˜ f φ ( x i ) − ˜ f φ ∗ ( x i ))
ξ i | ≤ (cid:96) max ξ max , so ( ˜ f φ ( x i ) − ˜ f
φ ∗ ( x i ))
ξ i is (cid:96) max ξ max -subgaussian; thus, by Lemma B.2, we have
[Equation]
Combining (15) & (16), continuing from (14), we have [Equation]
[Equation]
17) where for the ﬁrst inequality, we have used max { (cid:96) 2 max , ξ 2 max } ≤ (cid:96) 2 max + ξ 2 max , and the second inequality follows since by Lemma B.1, the covering number of the (cid:15) -net
E of Φ satisﬁes
[Equation]
Finally, we choose (cid:15) so that (17) is smaller than δ —in particular, letting (cid:15) =
[Equation]
then continuing (17), we have
[Equation]
≤
[Equation]
as claimed.
B. P ROOF OF P ROPOSITION 3.
ˆ˜ L ( φ ; Z ) is twice differentiable and convex in φ .
Note that the minimization problem of ˆ L ( θ ; Z ) is equivalent to that of ˆ˜ L ( g (ˆ θ ); Z ) .
We consider two cases.
First, consider the case where ˆ θ has rank d .
The ﬁrst order condition ∇ ˆ L ( θ ; Z ) = 0 is the same as ∇ ˆ˜ L ( g (ˆ θ ); Z )
= 0 , which gives
[Equation]
As ˆ θ is of full row rank, there exists a matrix ˆ θ
† ∈ R k × d such that ˆ θ ˆ θ †
=
I (e.g. ˆ θ †
=
ˆ θ (cid:62) (ˆ θ ˆ θ (cid:62) )
− 1 ).
We can right multiply the above equation by ˆ θ † and obtain that
[Equation]
As ˆ˜ L ( φ ; Z ) is convex in φ , the above implies ˆ φ =
g (ˆ θ ) is a global minimum of ˆ˜ L ( φ ; Z ) .
Therefore, ˆ θ is a global minimum of ˆ L ( θ ; Z ) .
Next, consider the case where the rank of ˆ θ is smaller than d .
In this case, we follow the proof strategy in Proposition 4 in Bach et al.
(2008); we provide here for completeness.
In this case, Equation (18) still holds, which implies
[Equation]
The Hessian of ˆ˜ L ( g (ˆ θ ); Z ) has
[Equation]
As ˆ θR is also a local minimum for any orthogonal matrix R (i.e., RR (cid:62)
=
R (cid:62) R
= I )
, we can ﬁnd a ˆ θ with the last column being 0 by right multiplying certain R .
Then, consider any ∆ with the ﬁrst k − 1 columns being 0 and the last column being any u ∈ R d .
With this choice of ∆ and ˆ θ , ˆ θ ∆ (cid:62) = 0 .
Therefore,
[Equation]
Since ˆ θ is a local minimum of ˆ˜ L ( g (ˆ θ ); Z ) , it holds that ∇ 2 ˆ˜ L ( g (ˆ θ ); Z )(∆ , ∆) ≥ 0 , which implies
[Equation]
Equation (19) and (20) together comprise the ﬁrst order conditions of the convex minimization problem min φ (cid:23)
ˆ˜ L ( φ ; Z ) .
Thus, ˆ θ is also a global minimum.
C P ROOFS FOR S ECTION
First, we provide the full statement of Theorem 4.2 (including constants).
Theorem C.1.
The expected regret of Algorithm 1 is
[Equation]
where
[Equation]
Before we prove Theorem C.1, we ﬁrst prove a preliminary result establishing an analog of the smooth best arm response property Rusmevichientong & Tsitsiklis (2010) to our setting.
First, we have the following useful result:
Lemma C.2.
Let φ, φ (cid:48)
∈ R
d × d be symmetric matrices, let x, x
(cid:48) ∈
R d be eigenvectors of φ, φ (cid:48)
corresponding to their top eigenvalue, such that (cid:107) x (cid:107) 2 = (cid:107) x (cid:48) (cid:107) 2 = 1 , and let λ 1 ≥ λ 2 ≥ ... ≥ λ
d be the eigenvalues of φ (cid:48) .
Suppose that (cid:104) x, x (cid:48) (cid:105) ≥ 0 .
Then, we have
[Equation]
Proof.
See Corollary 3 of Yu et al.
Next, let χ : R d × d → 2 X denote the subset of reward-maximizing arms for g ( θ ) = θθ (cid:62) —i.e.,
[Equation]
where the argmax returns the set of all optimal values.
Then, we have the following analog of smooth best arm response:
Lemma C.3.
For any φ ∈ R d × d , there exists x ∈ χ ( φ ) and x ∗ ∈ χ ( φ ∗ ) such that
[Equation]
Proof.
First, note that x, x ∗ are eigenvectors of φ, φ ∗ corresponding to their top eigenvalues, respectively.
Next, note that if x ∗ ∈ χ ( φ ∗ ) , then we also have − x ∗ ∈
χ ( φ ∗ ) ; thus, without loss of generality, we can assume that (cid:104) x ∗ , x (cid:105) ≥ 0 .
Also, note that (cid:107) x (cid:107) 2 = (cid:107) x ∗ (cid:107) 2 = 1 since the optimizer maximizes the magnitude of x .
Thus, we have
[Equation]
where the second inequality follows by by Lemma C.2, and the third inequality follows by Assumption 4.1, as claimed.
Now, we prove Theorem C.1.
The cumulative regret R ( T ) of a horizon of T has that
[Equation]
where ˆ θ is an estimator that minimizes the empirical loss of the ﬁrst m samples, ˆ x ∈ χ ( g (ˆ θ )) maximizes the estimated expected reward f ˆ θ ( x ) , and (cid:104) φ, φ (cid:48) (cid:105)
= (cid:80) di,
j =1 φ ij φ (cid:48) ij is the matrix inner product.
Since ˆ x is a maximizer of f ˆ θ ( x ) = (cid:104) g
(ˆ θ ) , xx (cid:62) (cid:105) , we have (cid:104) g
(ˆ θ ) , x ∗ x ∗(cid:62) − ˆ x ˆ x (cid:62) (cid:105) ≤ 0 .
Thus, continuing from (21), we have
[Equation]
To bound the second term in (22) [Equation] note that
[Equation]
where the last step follows by Lemma C.3.
Next, by Theorem 3. [Equation] we have
[Equation]
with probability at least 1 − δ .
Now, deﬁning the event
[Equation]
letting
[Equation]
log
[Equation]
, and continuing from (22), we have
[Equation]
The third term in inequality (23) is smaller than the second term when
[Equation]
For a choice of m satisfying (24), continuing from (23), we have
[Equation]
Next, we choose m to minimize the upper bound in (24) for sufﬁciently large T —in particular,
[Equation]
With this choice of m , we have
[Equation]
Finally, note that (24) holds under the choice of m in (26) for T satisfying
[Equation]
The claim follows.
D P ROOF OF T HEOREM 5.
First, we have the following key result:
Lemma D.1.
Let θ, θ (cid:48)
∈ R
d × k , and let φ = θθ (cid:62) and φ (cid:48)
= θ (cid:48)
θ (cid:48)(cid:62) .
Assume that (cid:107) φ − φ (cid:48) (cid:107) F ≤ η , and that σ min ( θ ) ≥ σ 0 > 0 , where σ min ( θ ) is the minimum singular value of θ (more precisely, the d th largest singular value).
Then, there exist orthogonal matrices R, R (cid:48)
∈
R k × k such that
[Equation]
Proof.
Consider the SVDs θ = U Σ V (cid:62) and θ (cid:48)
=
U (cid:48) Σ (cid:48)
V (cid:48)(cid:62) , where U, U (cid:48) ∈
R
d × d , Σ , Σ (cid:48) ∈ R d × d , and V, V (cid:48) ∈ R k × d ; then, we have φ =
U Σ 2 U (cid:62) and φ (cid:48) =
U (cid:48) Σ (cid:48)
2 U (cid:48)(cid:62) .
Then, we claim that the choices
R = V U (cid:62) and R (cid:48)
=
V (cid:48) U (cid:48)(cid:62) satisfy (27).
In particular, note that θR =
U Σ U (cid:62) and θ (cid:48) R (cid:48)
=
U (cid:48) Σ (cid:48) U (cid:48)(cid:62) , since V (cid:62) V = V (cid:48)(cid:62) V (cid:48)
=
I d since k ≥ d , where I d ∈ R d × d is the d -dimensional identity matrix.
Thus, it sufﬁces to show that
[Equation]
To this end, note that
[Equation]
where in the last step, we have multiplied the expression inside the Frobenius norm by U (cid:48)(cid:62) on the left and by U on the right, using the fact that the Frobenius norm is invariant under multiplication by orthogonal matrices.
Deﬁning W =
U (cid:48)(cid:62) U , note that
[Equation]
Then, continuing from (29), we have
[Equation]
where on the ﬁrst line, we have used (32) & (33), on the third line we have used Σ jj ≥ σ 0 , on the fourth line we have used (30) & (31), and on the last line we have multiplied on by U (cid:48) on the left
U (cid:62) on the right, again using the fact that the Frobenius norm is invariant under multiplication by orthogonal matrices.
Thus, we have shown (28), so the claim follows.
We note here that our result provides an analog of Lemma 6 in Ge et al.
(2017a) for quadratic neural networks.
Now, we prove Theorem 5.2.
First, by directly applying the arguments in the proof of Theorem 3.6, we have
[Equation]
with probability at least 1 − δ/ 2 .
However, ˆ θ p itself may not be close to θ ∗ p .
Instead, applying
Lemma D.1 with θ =
ˆ θ p and θ (cid:48)
= θ ∗ p , and with η = (cid:112) 2 (cid:15) p /α , there exists an orthogonal matrix
R p =
R (cid:48) R (cid:62) that “aligns” ˆ θ p with θ ∗ p , yielding
[Equation]
where σ 0 is the minimum singular value of θ ∗ p .
Now, let ˜ θ
g
= θ ∗ g R p , and note that this is a global minimizer (i.e., g (˜ θ g )
=
g ( θ ∗ g ) ), since R p is orthogonal.
Then, we have
[Equation]
34) with probability at least 1 − δ/ 2 .
In other words, an alternative global minimizer ˜ θ g exists within a small Frobenius norm of our proxy estimator ˆ θ p , even if ˆ θ p is not close to θ ∗ p .
Finally, on the event that (34) holds, note that for θ ∈ B 2 (ˆ θ p , ˆ B ) , we have the alternative upper bound
[Equation]
where the ﬁrst inequality holds by Lemma 3.4; thus, we can take (cid:96) max = K ˆ B .
Thus, on the event that (34) holds, by Theorem 3.6, we have
[Equation]
so the claim follows by a union bound.
E P ROOFS FOR S ECTIONE. P ROOF OF P ROPOSITION 6.
Suppose that z t ∈ { 0 , 1 } is binary, z 0 = 0 , and
[Equation]
0 otherwise . In particular, since z 0 = 0 , p ( w ) = 1 ( w = w 0 ) places all weight on the zero sequence w 0 = 0 ... 0 .
Next, consider the shifted distribution
[Equation]
otherwise .
Note that (cid:107) ˜ p
( · | z (cid:48) ) − ˜ q ( · | z (cid:48) )
(cid:107) TV ≤ α , so
Assumption 6.3 is satisﬁed.
Note that q ( w 0 ) =
T (cid:89) t = ˜ q (0 | 0)
= (1 − α/ 2) T .
As a consequence, we have (cid:107)
[Equation]
= ( [Equation] [Equation] [Equation]
+ (
[Equation]
[Equation] / 2) T ) = 2( [Equation]
[Equation]
/ 2) T ) , as claimed.
E. P ROOF OF L EMMA 6.
Note that (cid:107) ˜ q t − ˜ p t (cid:107) TV = k (cid:88) j = (cid:90) | ˜ q t ( z, j ) − ˜ p t ( z, j ) |
dz = k (cid:88)
j = k (cid:88) j (cid:48) =
[Equation]
Since q 0 ( z, j ) =
p 0 ( z, j ) for all z ∈ Z and j ∈ [ k ] , by induction, (cid:107) ˜
q t − ˜ p t (cid:107) TV ≤ tα .
Thus, we have
[Equation]
as claimed.
E. P ROOF OF L EMMA 6.
First, we prove the following lemma.
Lemma E.1.
We have
[Equation]
Proof.
For the base case [Equation] we have
[Equation]
as claimed.
For the inductive case [Equation] we have ˜
[Equation]
as claimed.
Now, we prove Lemma 6.6.
First, note that for each t ∈ [ T ] , we have
[Equation]
where the last step follows from Lemma E.1.
Now, note that
[Equation]
as claimed.
E. P ROOF OF T HEOREM 6.
First, we show that P q ( z,j ) [ˆ˜ g ( z, j ) (cid:54) =
˜
g ∗ ( z, j )] ≤ (cid:15) g + Tα .
To this end, note that
P q ( z,j ) [ˆ˜ g ( z, j ) (cid:54) =
˜
g ∗ ( z, j )] =
P p ( z,j ) [ ˆ ˜ g ( z, j ) [Equation]
=
[Equation]
g
[Equation]
( z, j )
[Equation]
P
[Equation]
( z,j )
[Equation]
ˆ
[Equation]
( z, j )
[Equation]
z
[Equation] j )] [Equation] P
[Equation]
z [Equation] j ) [Equation]
ˆ
[Equation]
j
[Equation]
j )]
[Equation]
g +
[Equation]
≤
[Equation]
˜
[Equation]
−
[Equation] p [Equation] TV ≤ (cid:15)
[Equation] + Tα. Next, by Lemma 6.6 with q in place of p and (cid:15) g + Tα in place of (cid:15) g , we have P q ( w ) [ˆ g ( w ) (cid:54) =
g ∗ ( w )] ≤
T(cid:15)
g
+ T 2 α .
Then, assuming that ˆ g ( w ) =
g ∗ ( w ) , we have
[Equation]
max {
[Equation]
[Equation]
The claim follows by a union bound.