O N F EATURE D IVERSITY IN E NERGY - BASED MODELS
@syke
Gabbouj moncef.gabbouj@tuni.fi
A BSTRACT
Energy-based learning is a powerful learning paradigm that encapsulates various discriminative and generative approaches.
An energy-based model (EBM) is typically formed of one (or many) inner-models which learn a combination of the different features to generate an energy mapping for each input conﬁguration.
In this paper, we focus on the diversity of the produced feature set.
We extend the probably approximately correct (PAC) theory of EBMs and analyze the effect of the diversity on the performance of EBMs.
We derive generalization bounds for various learning contexts, i e., regression, classiﬁcation and implicit regression, with different energy functions and we show that indeed increasing the diversity of the feature set can consistently decrease the gap between the true and empirical expectation of the energy and boosts the performance of the model
I NTRODUCTION
The energy-based learning paradigm was ﬁrst proposed by LeCun et al.
(2006) as an alternative to probabilistic graphical models (Koller & Friedman, 2009).
As their name suggests, energy-based models (EBMs) map each input ‘conﬁguration’ to a single scalar, called the ‘energy’.
In the learning phase, the parameters of the model are optimized to associate the desired conﬁgurations with small energy values and the undesired ones with higher energy values (Kumar et al., 2019; Song & Ermon, 2019; Yu et al., 2020; Nash & Durkan, 2019; Meng et al., 2020; Arbel et al., 2021).
In the inference phase, given an incomplete input conﬁguration, the energy surface is explored to ﬁnd the remaining variables which yield the lowest energy.
EBMs encapsulate solutions to several supervised (LeCun et al., 2006; Fang & Liu, 2016) and unsupervised learning problems (Ranzato et al., 2007b; Haarnoja et al., 2017; Parshakova et al., 2019; Deng et al., 2020; Bakhtin et al., 2021) and provide a common theoretical framework for many learning models, including traditional discriminative (Zhai et al., 2016; Grathwohl et al., 2019; Li et al., 2020; LeCun et al., 2006;
Teh et al., 2003) and generative (Zhao et al., 2016; Dai et al., 2017; Ranzato et al., 2007a; Che et al., 2020; Khalifa et al., 2020;
Arbel et al., 2021) approaches.
Formally, let us denote the energy function by E ( W, X , Y ) , where W represents the model parameters to be optimized during training and X , Y are sets of variables.
Figure 1 illustrates how classiﬁcation, regression, and implicit regression can be expressed as EBMs.
In Figure 1 (a), a regression scenario is presented.
The input X , e.g., an image, is transformed using an inner model
G W ( X ) and its distance, D , to the second input Y is computed yielding the energy function.
A valid energy function in this case can be the L 1 or the L 2 distance.
In the binary classiﬁcation case (Figure 1 (b)), the energy can be deﬁned as E ( W, X , Y ) =
− Y G W ( X ) .
In the inference phase, given an input X , the label Y ∗ can be obtained by solving the following optimization problem:
[Equation]
G W ( X )
X
D (G W ( X ), Y )
Y
E (W, X , Y )
G W ( X )
X - Y G W ( X )
Y
E (W, X , Y )
G W1 ( X )
X ||G W1 ( X ) - G W2 ( Y )||
Y
E (W, X , Y )
G W2 ( Y )
An EBM typically relies on an inner model, i.e., G w ( X ) , to generate the desired energy landscape (LeCun et al., 2006).
Depending on the problem at hand, this function can be constructed as a linear projection, a kernel method, or a neural network and its parameters are optimized in a data-driven manner in the training phase.
Formally, G w ( X ) can be written as
[Equation] where { φ 1 ( · ) , · · · , φ D ( · ) } is the feature set, which can be hand-crafted, separately trained from unlabeled data (Zhang & LeCun, 2017), or modeled by a neural network and optimized in the training phase of the EBM model (Du & Mordatch, 2019).
In the rest of the paper, we assume that the inner models G W deﬁned in the energy-based learning system (Figure 1) are obtained as a weighted sum of different features as expressed in equation 2.
In (Zhang, 2013), it was shown that simply minimizing the empirical energy over the training data does not theoretically guarantee the minimization of the expected value of the true energy.
Thus, developing and motivating novel regularization techniques is required (Zhang & LeCun, 2017).
We argue that the quality of this feature set, i.e., { φ 1 ( · ) , · · · , φ D ( · ) } , plays a critical role in the overall performance of the global model.
In this work, we extend the theoretical analysis of (Zhang, 2013) and focus on the ‘diversity’ of this set and its effect on the generalization ability of the EBM models.
Intuitively, it is clear that a less correlated set of intermediate representations is richer and thus able to capture more complex patterns in the input.
Thus, it is important to avoid redundant features for achieving a better performance.
However, a theoretical analysis is missing.
We start by quantifying the diversity of a set.
To this end, we introduce ϑ -diversity:
Deﬁnition 1.
( ϑ -diversity)
A set of feature functions, { φ 1 ( · ) , · · · , φ D ( · ) } is called ϑ -diverse, if there exists a constant ϑ ∈ R , such that for every input X we have
[Equation]
with a high probability τ .
Intuitively, if two feature maps φ i ( · ) and φ j ( · ) are different, then with high probability they have different outputs for the same input.
However, if for example the features are extracted using a neural network with a ReLU activation function, then there is a high probability that some of the features associated with the input will be zero.
Thus, deﬁning a lower bound for the pair-wise diversity directly is impractical.
To this end, we quantify diversity as the lower-bound over the sum of the pair-wise distances of the feature maps as expressed in equation 3.
ϑ measures the diversity of a set.
In machine learning context, diversity has been explored in ensemble learning (Li et al., 2012; Yu et al., 2011), sampling (Derezinski et al., 2019;
Bıyık et al., 2019; Gartrell et al., 2019), ranking (Yang et al., 2019; Gan et al., 2020), pruning (Kondo & Yamauchi, 2014; He et al., 2019; Singh et al., 2020; Lee et al., 2020), and neural networks (Xie et al., 2015; 2017).
In Xie et al.
(2015; 2017), it was shown theoretically and experimentally that employing a diversity strategy over the weights of a neural network using the mutual angles improves the generalization ability of the model.
In this work, we explore a new line of research, where diversity is deﬁned over the feature maps directly, using the ϑ -diversity, in the context of energy-based learning.
We theoretically study the generalization ability of EBMs in different learning contexts, i.e., regression, classiﬁcation, implicit regression, and we derive new generalization bounds using the ϑ -diversity providing theoretical guarantees that a diverse set of features indeed improves the generalization ability of the model.
The contributions of this paper can be summarized as follows:
• We explore a new line of research, where diversity is deﬁned over the features representing the input data and not over the model’s parameters.
To this end, we introduce ϑ -diversity as a quantiﬁcation of the diversity of a given feature set.
• We extend the theoretical analysis (Zhang, 2013) and study the effect of the diversity of the feature set on the generalization of the energy-based models (EBMs).
• We derive approximation bounds for the expectation of the true energy in different learning contexts,
i
e., regression, classiﬁcation, and implicit regression, using different energy functions.
Our analysis consistently shows that increasing the diversity of the feature set can boost the performance of an energy based model.
PAC- LEARNING OF EBM S WITH ϑ - DIVERSITY
In this section, we derive a qualitative justiﬁcation for ϑ -diversity using probably approximately correct (PAC) learning (Valiant, 1984).
The PAC-based theory for standard energy based models has been established in (Zhang, 2013).
Based on the Rademacher complexity (Bartlett & Mendelson, 2002), several EBMs learning guarantees have been shown.
In Lemma 1, we present the principal
PAC-learning bound for energy functions with ﬁnite outputs.
Deﬁnition 2.
(Bartlett & Mendelson, 2002)
For a given dataset with m samples S
= { x i , y i } mi = generated by a distribution D and for a model space F : X → R with a single dimensional output, the empirical Rademacher complexity R m ( F ) of the set F is deﬁned as follows:
[Equation]
where the Rademacher variables σ = { σ 1 , · · · , σ N } are independent uniform random variables in {− 1 , 1 } .
Lemma 1.
(Zhang, 2013)
For a well-deﬁned energy function E ( h, x , y ) over hypothesis class H , input set X and output set Y (LeCun et al., 2006), the following holds for all h in H with a probability of at least 1 − δ
[Equation]
where E is the energy function class deﬁned as E = { E ( h, x , y ) | h ∈ H} , R m ( E ) is its Rademacher complexity, and M is the upper bound of E .
Lemma 1 provides a generalization bound for energy-based models with well-deﬁned (non-negative) and bounded energy.
The expected energy is bounded using the sum of three terms: The ﬁrst term is the empirical expectation of energy over the training data, the second term depends on the
Rademacher complexity of the energy class, and the third term involves the number of the training data m and the upper-bound of the energy function M .
This shows that merely minimizing the empirical expectation of energy, i.e., the ﬁrst term, may not yield a good approximation of the true expectation.
In (Zhang & LeCun, 2017), it has been shown that regularization using unlabeled data reduces the second and third terms, thus, leading to better generalization.
In this work, we express these two terms using the ϑ -diversity and show that employing a diversity strategy may also decrease the gap between the true and empirical expectation of the energy.
In Section 2.1, we consider the special case of regression and derive two bounds relative to two energy functions based on L 1 and
L 2 distances.
In Section 2.2, we derive the bound relative to the binary classiﬁcation task using as energy function E ( W , x , y ) =
− y G W ( x ) (LeCun et al., 2006).
In Section 2.3, we consider the case of implicit regression, which encapsulates different learning problems such as metric learning, generative models, and denoising (LeCun et al., 2006).
For this case, we use the L 2 distance between the inner models as the energy function.
2. R EGRESSION T ASK
Regression can be formulated as an energy-based learning problem (Figure 1 (a)) using the inner model G W ( x ) = (cid:80) Di =1 w i φ i ( x )
= w T Φ( x ) .
We also suppose that the feature set is well-deﬁned over the input domain X , i.e., ∀ x ∈ X
||
Φ( x ) || 2 ≤ A .
The two valid energy functions which can be used for regression are: E 2 ( W , x , y )
=
12 || G W ( x ) − y
||
22 and E 1 ( W , x , y )
=
|| G W ( x ) − y ||(LeCun et al., 2006).
Theorem 1 and Theorem 2 express the special cases of Lemma 1 using the ϑ -diversity of the feature set { φ 1 ( · ) , · · · , φ D ( · ) } .
Theorem 1.
For the energy function E ( h, x , y ) =
12 || G W ( x ) − y
|| 22 , over the input set X ∈ R N , hypothesis class H =
{ G W ( x ) =
(cid:80) Di =1 w i φ i ( x ) =
w T Φ( x ) | Φ ∈ F , ∀ x || Φ( x ) || 2 ≤ A } , and output set Y ⊂ R , if the feature set { φ 1 ( · ) , · · · , φ D ( · ) } is ϑ -diverse with a probability τ , then with a probability of at least (1 − δ ) τ , the following holds for all h in H
[Equation]
where B is the upper-bound of Y , i.e., y ≤ B, ∀ y ∈ Y .
Theorem 2.
For the energy function E ( h, x , y )
=
|| G W ( x ) − y
|| 1 , over the input set X ∈ R N , hypothesis class H =
{ G W ( x ) =
(cid:80) Di =1 w i φ i ( x ) =
w T Φ( x ) | Φ ∈ F , ∀ x || Φ( x ) || 2 ≤ A } , and output set Y ⊂ R , if the feature set { φ 1 ( · ) , · · · , φ D ( · ) } is ϑ -diverse with a probability τ , then with a probability of at least (1 − δ ) τ , the following holds for all h in H
[Equation]
where B is the upper-bound of Y , i.e., y ≤ B, ∀ y ∈ Y .
The proofs are available in the Appendix.
We note that, in Theorem 1 and Theorem 2, we consistently ﬁnd that the bound of the true expectation of the energy is a decreasing function with respect to ϑ .
This proves that that for the regression task employing a diversity strategy can improve the generalization performance of the energy-based model.
2. T WO -C LASS C LASSIFIER
Here, we consider the problem of binary classiﬁcation, as illustrated in Figure 1 (b).
Using the same assumption as in regression for the inner model, i.e., G W ( x ) = (cid:80) Di = [Equation] i [Equation]
x
[Equation] w T Φ( x ) , energy function of E ( W , x , y ) = − y G W ( x ) (LeCun et al., 2006), and the ϑ -diversity of the feature set,we express Lemma 1 for this speciﬁc conﬁguration in Theorem 3.
Theorem 3.
For the energy function E ( h, x , y )
=
− y G W ( x ) , over the input set X ∈ R N , hypothesis class H =
{ G W ( x ) =
(cid:80) Di =1 w i φ i ( x ) =
w T Φ( x ) | Φ ∈ F , ∀ x || Φ( x ) || 2 ≤ A } , and output set Y ⊂ R , if the feature set { φ 1 ( · ) , · · · , φ D ( · ) } is ϑ -diverse with a probability τ , then with a probability of at least (1 − δ ) τ , the following holds for all h in H
[Equation]
The proof is available in the Appendix.
Similar to the regression task, we note that the upper-bound of the true expectation is a decreasing function with respect to the diversity term.
Thus, a more diverse feature set, i.e., higher ϑ , has a lower upper-bound for the true energy.
2. I MPLICIT R EGRESSION
In this section, we consider the problem of implicit regression.
This is a general formulation of a different set of problems such as metric learning, where the goal is to learn a distance function between two domains, image denoising, or object detection as illustrated in (LeCun et al., 2006).
This form of EBM (Figure 1 (c)) has two inner models, G 1 W ( · ) and G 2 W ( · ) , which can be equal or different according to the problem at hand.
Here, we consider the general case, where the two models correspond to two different combinations of different features, i.e., G 1 W ( x ) =
(cid:80) D (1) i
=1 w 1 i φ 1 i ( x ) and
G (2) W ( y ) =
(cid:80) D (2) i
=1 w 2 i φ 2 i ( y ) .
Thus, we have a different ϑ -diversity term for each set.
The ﬁnal result is presented in Theorem 4.
Theorem 4.
For the energy function E ( h, x , y ) =
12 || G (1) W ( x ) − G (2) W ( y ) || 22 , over the input set
X ∈ R N , hypothesis class H =
{ G (1) W ( x ) = (cid:80) D (1) i
=1 w (1) i φ (1) i ( x ) = w (1) T Φ (1) ( x ) , G (2) W ( y ) =
(cid:80) D (2) i
=1 w (2) i φ (2) i ( y ) =
w (2) T Φ (2) ( y ) | Φ (1) ∈
F 1 , Φ (2) ∈
F 2 , ∀ x || Φ (1) ( x ) ||≤
A (1) , ∀ y ||
Φ (2) ( y ) ||
2 ≤ A (2) } , and output set Y ⊂ R N , if the feature set { φ (1)1 ( · ) , · · · , φ (1) D (1) ( · ) } is ϑ (1) -diverse with a probability τ
and the feature set { φ (2)1 ( · ) , · · · , φ (2) D (2) ( · ) } is ϑ (2) -diverse with a probability τ 2 , then with a probability of at least (1 − δ )
τ 1 τ 2 , the following holds for all h in H
E
[Equation]
x ,
[Equation]
where J 1 = || w (1) || 2 ∞ (cid:16) D (1) A (1)2 − ϑ
[Equation]
[Equation]
and J 2 = || w (2) || 2 ∞ (cid:16) D [Equation] [Equation]
A [Equation]
[Equation] (cid:17) .
The proof of Theorem 4 is available in the Appendix.
The upper-bound of the energy model depends on the diversity variable of both feature sets.
Moreover, we note that the bound for the implicit regression decreases proportionally to ϑ 2 , as opposed to the classiﬁcation case for example, where the bound is proportional to ϑ .
We note that the theory developed in our paper (Theorems 1 to 4) is agnostic to the loss function (LeCun et al., 2006) or the optimization strategy used (Kumar et al., 2019; Song & Ermon, 2019; Yu et al., 2020).
We show that increasing the diversity of the features consistently decreases the upperbound of the true expectation of the energy and, thus, can boost the generalization performance of the energy-based model.
We note that our analysis is independent of how the features are obtained, e.g., handcrafted or optimized.
In fact, in the recent state-of-the-art EBMs (Khalifa et al., 2020;
Bakhtin et al., 2021; Nash & Durkan, 2019; Yu et al., 2020), the features are typically parameterized using a deep learning model and optimized during the training.
Thus, our theory suggests the use of a diversity strategy, for example in the form of a regularization as in (Cogswell et al., 2016), to avoid learning redundant features can improve the performance of the model and decrease the gap between the expectation of the true and the empirical energy.
C ONCLUSION
The energy-based learning is a powerful learning paradigm that encapsulates various discriminative and generative systems.
An EBM is typically formed of one (or many) inner models which learn a combination of different features to generate an energy mapping for each input conﬁguration.
In this paper, we introduced the feature diversity concept, i.e., ϑ -diversity, and we used it to extend the PAC theory of EBMs.
We derived different generalization bounds for various learning contexts, i.e., regression, classiﬁcation, and implicit regression, with different energy functions and we consistently found that increasing the diversity of the feature set can improve the approximation error of the true expectation of the energy function.
We also note that our theory is independent of the loss function or the training strategy used to optimize the parameters of the EBM.
Future directions include developing practical strategies to promote the diversity of the feature set in case the features are optimized following a data-driven process, like the training phase of a neural network.
A PPENDIX 4. P ROOF OF T HEOREM
Lemma 2.
With a probability of at least τ , we have
[Equation]
where J = ||
w || 2 ∞ (cid:16)
DA 2 − ϑ 2 (cid:17) and
A = sup x || φ ( x ) || 2 ,
Proof.
h
[Equation]
We have || Φ( x ) || 2 ≤ A .
For the ﬁrst term in equation 11, we have (cid:80) m φ m ( x ) 2 ≤ A 2 .
By using the identity φ m ( x ) φ n ( x ) = 12 (cid:0) φ m ( x ) 2 + φ n ( x ) 2 − ( φ m ( x ) − φ n ( x )) 2 (cid:1) , the second term can be rewritten as
[Equation]
In addition, we have with a probability τ , 12 (cid:80) m
(cid:54) =
n || φ m ( x ) − φ n ( x ) || 2 ≥ ϑ .
Thus, we have with a probability at least τ :
[Equation]
By putting everything back to equation 11, we have with a probability τ ,
[Equation]
Thus, with a probability τ ,
[Equation]
Lemma 3.
With a probability of at least τ , we have
[Equation]
Proof.
We have sup x ,y,h | h ( x ) − y | ≤ 2 sup x ,y,h ( | h ( x ) | + | y | )
= 2( √J + B ) .
Thus sup x,y,h | E ( h ( x ) , y ) | ≤ ( √J + B ) 2 .
Lemma 4.
With a probability of at least τ , we have
[Equation]
[Equation]
Proof.
Using the decomposition property of the Rademacher complexity (if φ is a L -Lipschitz function, then R m ( φ ( A ))
≤ L R m ( A ) ) and given that E ( · , y )
=
|| .
− y
|| 2 is K -Lipschitz with a constant
K = sup x ,y,h
|| h ( x ) − y || ≤ 2( √J + B )
, we have R m ( E ) ≤ K R m ( F ) ≤ 2( √J + B ) R m ( H ) , where H = { G W ( x )
= (cid:80) Di =1 w i φ i ( x )
| ||
w || 1 ≤ D
|| w || ∞ } .
Next, similar to the proof of Theorem 2.10 in (Wolf, 2018), we note that (cid:80) Di =1 w i φ i ( x ) ∈
( D || w || ∞ ) conv ( F + − ( F )) :
= G , where conv denotes the convex hull and F is the set of φ functions.
Thus, R m ( H ) ≤ R m ( G ) =
D || w || ∞
R m ( conv ( F + ( −F ))
=
D || w || ∞
R m ( F + ( −F )) =
2 D || w || ∞
R m ( F ) .
Theorem 1 For the energy function E ( h, x , y )
=
12 || G W ( x ) − y
|| 22 , over the input set X ∈ R N , hypothesis class H =
{ G W ( x ) =
(cid:80) Di =1 w i φ i ( x ) =
w T Φ( x ) | ∀ x || Φ( x ) || 2 ≤ A } , and output set Y ⊂ R , if the feature set { φ 1 ( · ) , · · · , φ D ( · ) } is ϑ -diverse with a probability τ , then with a probability of at least (1 − δ ) τ , the following holds for all h in H
[Equation]
where B is the upper-bound of Y , i.e., y ≤ B, ∀ y ∈ Y ,
Proof.
We replace the variables in Lemma 1 using Lemma 3 and Lemma 4.
P ROOF OF T HEOREM
Lemma 5.
With a probability of at least τ , we have
[Equation]
Proof.
We have sup x ,y,h | h ( x ) − y |
[Equation]
x
[Equation]
y
[Equation]
( |
[Equation] [Equation]
| + | y |
[Equation]
Lemma 6.
With a probability of at least τ , we have
[Equation]
Proof.
| .
| is 1-Lipschitz, Thus R m ( E ) ≤ R m ( H ) .
Theorem 2 For the energy function E ( h, x , y )
=
|| G W ( x ) − y
|| 1 , over the input set X ∈ R N , hypothesis class H =
{ G W ( x ) =
(cid:80) Di =1 w i φ i ( x ) =
w T Φ( x ) | ∀ x || Φ( x ) || 2 ≤ A } , and output set Y ⊂ R , if the feature set { φ 1 ( · ) , · · · , φ D ( · ) } is ϑ -diverse with a probability τ , then with a probability of at least (1 − δ ) τ , the following holds for all h in H
[Equation]
Proof.
We replace the variables in Lemma 1 using Lemma 5 and Lemma 6.
4. P ROOF OF T HEOREM
Lemma 7.
With a probability of at least τ , we have
[Equation]
Proof.
We have sup − y G W ( x ) ≤ sup | G W ( x ) | ≤ √J
Lemma 8.
With a probability of at least τ , we have
[Equation]
Proof.
We note that for y ∈ {− 1 , 1 } , σ and − yσ follow the same distribution.
Thus, we have
R m ( E ) =
R m ( H ) .
Next, we note that R m ( H ) ≤ 2 D || w || ∞ R m ( F )
Theorem 3 For a well-deﬁned energy function E ( h, x , y ) (LeCun et al., 2006), over hypothesis class H , input set X and output set Y , if it has upper-bound M, then with a probability of at least 1 − δ , the following holds for all h in H
[Equation]
Proof.
We replace the variables in Lemma 1 using Lemma 7 and Lemma 8.
4. P ROOF OF T HEOREM
Lemma 9.
With a probability of at least τ 1 τ 2 , we have
[Equation]
Proof.
We have || G (1) W ( x ) − G (2) W ( y ) ||
22 ≤ 2( || G (1) W ( x ) ||
+ || G (2) W ( y ) || 22 ) .
Similar to Theorem 1, we have sup || G (1) W ( x ) ||
22 ≤ || w (1) ||
2 ∞ (cid:16) D (1) A (1)2 − ϑ (1)2 (cid:17) =
J 1 and sup || G (2) W ( y ) ||
22 ≤ || w (2) ||
2 ∞ (cid:16) D (2)
A (2)2 − ϑ
(2)2 (cid:17) =
J
Lemma 10.
With a probability of at least τ 1 τ 2 , we have
[Equation] [Equation]
Proof.
Let f be the square function, i.e., f ( x ) = 12 x 2 and E
= { G (1) W ( x ) − G (2) W ( y ) | x ∈
X , y ∈ Y} .
We have E = f
( E 0 + ( −E 0 )) .
f is Lipschitz over the input space, with a constant L bounded by sup x, W G (1) W ( x ) +
sup y, W G (2) W ( y ) ≤ √J 1 + √J 2 .
Thus, we have
R m ( E ) ≤ ( √J 1 + √J 2 ) R m ( E 0 + ( −E 0 )) ≤ 2( √J 1 + √J 2 )
R m ( E 0 ) .
Next, we note that
R m ( E 0 ) =
R m ( H 1 + ( −H 2 ))
=
R m ( H 1 ) + R m ( H 2 ) .
Using same as technique as in Lemma 4, we have R m
( H 1 ) ≤ 2 D (1) || w (1) ||
∞
R m ( F 1 ) and R m ( H 2 ) ≤ 2 D (2) || w (2) ||
∞
R m ( F 2 )
Theorem 4 For the energy function E ( h, x , y )
=
12 || G (1) W ( x ) − G (2) W ( y ) || 22 , over the input set
X ∈ R N , hypothesis class H =
{ G (1) W ( x ) = (cid:80) D (1) i
=1 w (1) i φ (1) i ( x ) = w (1) T Φ (1) ( x ) , G (2) W ( y ) =
(cid:80) D (2) i
=1 w (2) i φ (2) i ( y ) =
w (2) T Φ (2) ( y ) | Φ (1) ∈
[Equation] Φ
[Equation]
∈
[Equation] ≤
A (1) , ∀ y ||
Φ (2) ( y ) ||
2 ≤ A (2) } , and output set Y ⊂ R N , if the feature set { φ (1)1 ( · ) , · · · , φ (1) D (1) ( · ) } is ϑ (1) -diverse with a probability τ
and the feature set { φ (2)1 ( · ) , · · · , φ (2) D (2) ( · ) } is ϑ (2) -diverse with a probability τ 2 , then with a probability of at least (1 − δ )
τ 1 τ 2 , the following holds for all h in H
[Equation]
where J 1 = || w (1) || 2 ∞ (cid:16) D (1) A (1)2 − ϑ (1)
[Equation]
and J 2 = || w (2) || 2 ∞ (cid:16) D [Equation]
2) A (2)2 − ϑ
(2)2 (cid:17) .
Proof.
We replace the variables in Lemma 1 using Lemma 9 and Lemma 10.