Original: Leveraging Theoretical Tradeoﬀs in Hyperparameter Selection for Improved Empirical Performance
Revised: Leveraging Theoretical Tradeoﬀs in Hyperparameter Selection for Improved Empirical Performance

Original: Abstract
Revised: Abstract

Original: The tradeoﬀs in the excess risk incurred from data-driven learning of a single model has been studied by decomposing the excess risk into approximation, estimation and optimization errors.
Revised: The tradeoﬀs in the excess risk incurred from data-driven learning of a single model has been studied by decomposing the excess risk into approximation, estimation and optimization errors.

Original: In this paper, we focus on the excess risk incurred in data-driven hyperparameter optimization (HPO) and its interaction with approximate empirical risk minimization (ERM) necessitated by large data.
Revised: In this paper, we focus on the excess risk incurred in data-driven hyperparameter optimization (HPO) and its interaction with approximate empirical risk minimization (ERM) necessitated by large data.

Original: We present novel bounds for the excess risk in various common scenarios in HPO.
Revised: We present novel bounds for the excess risk in various common scenarios in HPO.

Original: Based on these results, we propose practical heuristics that allow us to improve performance or reduce computational overhead of data-driven HPO, demonstrating over 2 × speedup with no loss in predictive performance in our preliminary results
Revised: Based on these results, we propose practical heuristics that allow us to improve performance or reduce computational overhead of data-driven HPO, demonstrating over 2 × speedup with no loss in predictive performance in our preliminary results

Original: 1. Excess Risk in Learning
Revised: 1. Excess Risk in Learning

Original: The learning process has various sources of errors.
Revised: The learning process has various sources of errors.

Original: Given data, we choose a model or function class F which corresponds to not just a method (such as Support Vector Machines,
Revised: Given data, we choose a model or function class F which corresponds to not just a method (such as Support Vector Machines,

Original: Generalized Linear Models, Neural Networks, Decision trees) but its speciﬁc hyperparameters (HPs) – these HPs refer to anything that would inﬂuence the predictive performance of the model learned from data.
Revised: Generalized Linear Models, Neural Networks, Decision trees) but its speciﬁc hyperparameters (HPs) – these HPs refer to anything that would inﬂuence the predictive performance of the model learned from data.

Original: Given choice of F , learning searches for the function via (possibly approximate) empirical risk minimization (ERM).
Revised: Given choice of F , learning searches for the function via (possibly approximate) empirical risk minimization (ERM).

Original: We currently have an understanding of the factors aﬀecting the excess risk of this chosen function – (i) the choice of the function class and its capacity to model the process generating the data, (ii) the use of empirical error instead of the generalization error, and (iii) the approximation in the minimization of the empirical error (Vapnik, 2006; Devroye et al., 2013; Bottou and Bousquet, 2008).
Revised: We currently have an understanding of the factors aﬀecting the excess risk of this chosen function – (i) the choice of the function class and its capacity to model the process generating the data, (ii) the use of empirical error instead of the generalization error, and (iii) the approximation in the minimization of the empirical error (Vapnik, 2006; Devroye et al., 2013; Bottou and Bousquet, 2008).

Original: However, in practice, a signiﬁcant part of the whole exercise is the choice of the function class F (method and its HPs).
Revised: However, in practice, a signiﬁcant part of the whole exercise is the choice of the function class F (method and its HPs).

Original: Usually, we consider a (possibly large) set of function classes and select one of them based on the data-driven hyperparameter optimization (HPO) or model selection .
Revised: Usually, we consider a (possibly large) set of function classes and select one of them based on the data-driven hyperparameter optimization (HPO) or model selection .

Original: This search can be done via grid search.
Revised: This search can be done via grid search.

Original: AutoML (automated machine learning) has spurred a lot of research in eﬃcient HPO (Hutter et al., 2011; Shahriari et al., 2016).
Revised: AutoML (automated machine learning) has spurred a lot of research in eﬃcient HPO (Hutter et al., 2011; Shahriari et al., 2016).

Original: The automation allows us to eﬃciently explore larger numbers of HP conﬁgurations for improved performance.
Revised: The automation allows us to eﬃciently explore larger numbers of HP conﬁgurations for improved performance.

Original: HPO has been extended from model conﬁgurations to the conﬁguration of complete ML pipelines with eﬃcient Combined Algorithm Selection and
Revised: HPO has been extended from model conﬁgurations to the conﬁguration of complete ML pipelines with eﬃcient Combined Algorithm Selection and

Original: HPO (or CASH) algorithms (Thornton et al., 2012; Feurer et al., 2015; Kotthoﬀ et al., 2017;
Revised: HPO (or CASH) algorithms (Thornton et al., 2012; Feurer et al., 2015; Kotthoﬀ et al., 2017;

Original: Rakotoarison et al., 2019; Liu et al., 2020; Sarigiannis et al., 2019).
Revised: Rakotoarison et al., 2019; Liu et al., 2020; Sarigiannis et al., 2019).

Original: Penalty-based model selection has been thoroughly studied theoretically, resulting in strong guarantees in the form of “oracle inequalities” – the expected excess risk of the selected model (or HP) is within a multiplicative and additive factor of the best possible excess risk if an oracle provided us with the best HP.
Revised: Penalty-based model selection has been thoroughly studied theoretically, resulting in strong guarantees in the form of “oracle inequalities” – the expected excess risk of the selected model (or HP) is within a multiplicative and additive factor of the best possible excess risk if an oracle provided us with the best HP.

Original: This has been widely studied in (binary) classiﬁcation (Boucheron et al., 2005), (bounded) regression and density estimation (Massart, 2007; Arlot et al., 2010).
Revised: This has been widely studied in (binary) classiﬁcation (Boucheron et al., 2005), (bounded) regression and density estimation (Massart, 2007; Arlot et al., 2010).

Original: However, in practice, penalty-based model selection is not used for data-driven HPO, and we resort to cross-validation (CV) because they are more universally applicable.
Revised: However, in practice, penalty-based model selection is not used for data-driven HPO, and we resort to cross-validation (CV) because they are more universally applicable.

Original: CV schemes have been shown to be theoretically competitive to penalty-based selection at the cost of having less data for learning.
Revised: CV schemes have been shown to be theoretically competitive to penalty-based selection at the cost of having less data for learning.

Original: We focus on CV widely used in the aforementioned HPO.
Revised: We focus on CV widely used in the aforementioned HPO.

Original: While CV based model selection has been studied theoretically, there are various scenarios in practical HPO, which have not been explored in literature.
Revised: While CV based model selection has been studied theoretically, there are various scenarios in practical HPO, which have not been explored in literature.

Original: Here, we focus on such a scenario – the practice of learning a ﬁnal model with the HP selected via CV on the full data, and the role of approximate ERM in HPO and the ﬁnal model training: (cid:73) We study the HPO process under the diﬀerent scenarios and identify new sources of excess risk introduced by this process, and provide upper bounds to the excess risk.
Revised: Here, we focus on such a scenario – the practice of learning a ﬁnal model with the HP selected via CV on the full data, and the role of approximate ERM in HPO and the ﬁnal model training: (cid:73) We study the HPO process under the diﬀerent scenarios and identify new sources of excess risk introduced by this process, and provide upper bounds to the excess risk.

Original: (cid:73) We propose data-driven practical heuristics based on these bounds to improve the predictive performance of selected model and reduce the computational overhead when performing HPO with approximate ERM.
Revised: (cid:73) We propose data-driven practical heuristics based on these bounds to improve the predictive performance of selected model and reduce the computational overhead when performing HPO with approximate ERM.

Original: Outline.
Revised: Outline.

Original: In § 2, we discuss existing excess risk decompositions and describe our HPO setting.
Revised: In § 2, we discuss existing excess risk decompositions and describe our HPO setting.

Original: We study the excess risk resulting from HPO in § 3, providing novel bounds, and deriving practical heuristics from these bounds.
Revised: We study the excess risk resulting from HPO in § 3, providing novel bounds, and deriving practical heuristics from these bounds.

Original: We conclude with future work in § 4.
Revised: We conclude with future work in § 4.

Original: 2. Existing Decompositions & Model Selection
Revised: 2. Existing Decompositions & Model Selection

Original: For a particular model, let F λ denote the function class for some HP λ ∈ Λ in the space of all possible valid HPs Λ .
Revised: For a particular model, let F λ denote the function class for some HP λ ∈ Λ in the space of all possible valid HPs Λ .

Original: For any function f : X → Y with ( X, Y ) , X ⊂ X , Y ⊂ Y generated from a distribution P , and a B -bounded loss function (cid:96) :
Revised: For any function f : X → Y with ( X, Y ) , X ⊂ X , Y ⊂ Y generated from a distribution P , and a B -bounded loss function (cid:96) :

Original: Y × Y , the expected risk E ( f ) and the empirical risk E n ( f ) with n samples { ( x i , y i ) } ni = 1 ∼ P n of the model f is given by
Revised: Y × Y , the expected risk E ( f ) and the empirical risk E n ( f ) with n samples { ( x i , y i ) } ni = 1 ∼ P n of the model f is given by

Original: [Equation]
Revised: [Equation]

Original: We denote the Bayes optimal prediction function as f ∗ such that, for any ( x, y ) ∼ P , f ∗ [Equation]
Revised: We denote the Bayes optimal prediction function as f (cid:63) such that, for any ( x, y ) ∼ P , f (cid:63) ( x ) =

Original: = [Equation]
Revised: arg min ^ y E

Original: [ (cid:96) ( y, ^ y ) | x ] .
Revised: [ (cid:96) ( y, ^ y ) | x ] .

Original: Let f λ = arg min
Revised: Let f λ = arg min

Original: f ∈F λ E ( f ) be the true risk minimizer in F λ , while ^ f n,λ = arg min f ∈F
Revised: f ∈F λ E ( f ) be the true risk minimizer in F λ , while ^ f n,λ = arg min f ∈F

Original: λ E n ( f ) be the ERM solution.
Revised: λ E n ( f ) be the ERM solution.

Original: With ERM over F λ , the excess risk incurred E =
Revised: With ERM over F λ , the excess risk incurred E =

Original: E
Revised: E

Original: (^ f n,λ ) − E ( f ∗ ) decomposes into two terms – (i) the approximation error
Revised: ( ^ f n,λ ) − E ( f (cid:63) ) decomposes into two terms – (i) the approximation error

Original: E app ( λ ) =
Revised: E app ( λ ) =

Original: E ( f λ ) − E ( f ∗ ) , and (ii) the estimation error E est ( λ ) =
Revised: E ( f λ ) − E ( f (cid:63) ) , and (ii) the estimation error E est ( λ ) =

Original: E (^ f n,λ ) − E ( f ) .
Revised: E (^ f n,λ ) − E ( f ) .

Original: For limited number of samples n , there is a tradeoﬀ between E app and E est , where a larger function class F λ usually reduces E app but increases E est (Vapnik, 2006; Devroye et al., 2013).
Revised: For limited number of samples n , there is a tradeoﬀ between E app and E est , where a larger function class F λ usually reduces E app but increases E est (Vapnik, 2006; Devroye et al., 2013).

Original: See § A (and Table 1) for notations, precise deﬁnitions and exact existing decomposition.
Revised: See § A (and Table 1) for notations, precise deﬁnitions and exact existing decomposition.

Original: Bottou and Bousquet (2008) study the tradeoﬀs in a “large-scale” setting where the learning has a computational budget and they categorize the learning setting into two following scales:
Revised: Bottou and Bousquet (2008) study the tradeoﬀs in a “large-scale” setting where the learning has a computational budget and they categorize the learning setting into two following scales:

Original: (i) “small-scale”: the number of samples n is small enough to allow for exact ERM within the compute budget, (ii) “large-scale”: the ERM needs to be approximated given the compute budget.
Revised: (i) “small-scale”: the number of samples n is small enough to allow for exact ERM within the compute budget, (ii) “large-scale”: the ERM needs to be approximated given the compute budget.

Original: They study the excess risk of an approximate ERM solution (cid:101) f n,λ ∈ F λ , introducing the optimization error E opt ( λ ) =
Revised: They study the excess risk of an approximate ERM solution (cid:101) f n,λ ∈ F λ , introducing the optimization error E opt ( λ ) =

Original: E ( (cid:101) f n,λ ) − E (^ f n,λ ) – the excess risk incurred due to approximate ERM – and argue that, in large-scale learning, approximate ERM on all n samples has better excess risk than exact ERM on a subsample of size n (cid:48) ≤ n .
Revised: E ( (cid:101) f n,λ ) − E (^ f n,λ ) – the excess risk incurred due to approximate ERM – and argue that, in large -scale learning, approximate ERM on all n samples has better excess risk than exact ERM on a subsample of size n (cid:48) ≤ n .

Original: We wish to understand the excess risk in HPO over Λ .
Revised: We wish to understand the excess risk in HPO over Λ .

Original: Limited to n samples from the data distribution P , the HPO problem with CV is given by this bilevel optimization:[Equation] min
Revised: Limited to n samples from the data distribution P , the HPO problem with CV is given by this bilevel optimization:

Original: [Equation]
Revised: [Equation]

Original: This results in the selection of the HP ^ λ .
Revised: This results in the selection of the HP ^ λ .

Original: The inner level problem for a ﬁxed HP λ is solved via ERM with E n t ( · ) on n t ≤ n samples, while the outer level problem considers an objective
Revised: The inner level problem for a ﬁxed HP λ is solved via ERM with E n t ( · ) on n t ≤ n samples, while the outer level problem considers an objective

Original: E vn v ( · ) which is evaluated using n v ≤ n samples not used for the inner ERM – while E n t ( · ) and E vn v ( · ) might have the same form, the · v superscript highlights their diﬀerence.
Revised: E vn v ( · ) which is evaluated using n v ≤ n samples not used for the inner ERM – while E n t ( · ) and E vn v ( · ) might have the same form, the · v superscript highlights their diﬀerence.

Original: This formulation incorporates the common practice of splitting the samples into a training set and a held-out validation set (of sizes n t , n v ≤ n with usually n t + n v = n ) and the best
Revised: This formulation incorporates the common practice of splitting the samples into a training set and a held-out validation set (of sizes n t , n v ≤ n with usually n t + n v = n ) and the best

Original: HP is selected based on empirical risk of the solution of ERM (with n t samples) on the heldout set (of size n v ).
Revised: HP is selected based on empirical risk of the solution of ERM (with n t samples) on the heldout set (of size n v ).

Original: When performing k -fold cross-validation , the inner optimization is solved k times for each HP λ (on k diﬀerent sets of n t samples), and the outer optimization averages the objective on k held-out sets (of size n v ) across the k learned models.
Revised: When performing k -fold cross-validation , the inner optimization is solved k times for each HP λ (on k diﬀerent sets of n t samples), and the outer optimization averages the objective on k held-out sets (of size n v ) across the k learned models.

Original: In this paper, we limit our scope to a single training/validation split.
Revised: In this paper, we limit our scope to a single training/validation split.

Original: 3. Excess Risk In HP Selection
Revised: 3. Excess Risk In HP Selection

Original: We ﬁrst analyze HPO with hold-out validation in the “small scale” setting and develop novel results for a common practice in model selection.
Revised: We ﬁrst analyze HPO with hold-out validation in the “small scale” setting and develop novel results for a common practice in model selection.

Original: Using these results, we propose a practical heuristic to obtain improved performance in data-driven small-scale HPO and empirically highlight its utility.
Revised: Using these results, we propose a practical heuristic to obtain improved performance in data-driven small-scale HPO and empirically highlight its utility.

Original: We then build upon these results to provide excess risk bounds in the “large scale” setting necessitating approximate ERM.
Revised: We then build upon these results to provide excess risk bounds in the “large scale” setting necessitating approximate ERM.

Original: Based on these, we propose another heuristic that allows us to enjoy the predictive performance of HPO with exact ERM at reduced computational cost.
Revised: Based on these, we propose another heuristic that allows us to enjoy the predictive performance of HPO with exact ERM at reduced computational cost.

Original: We also demonstrate the practical utility of this heuristic.
Revised: We also demonstrate the practical utility of this heuristic.

Original: In our ensuing presentation, the maximum error discrepancy sup f ∈F λ | E n ( f ) − E ( f ) | of any function class F λ will be appearing multiple times.
Revised: In our ensuing presentation, the maximum error discrepancy sup f ∈F λ | E n ( f ) − E ( f ) | of any function class F λ will be appearing multiple times.

Original: For ease of exposition, we consider as its generic upper-bound ∆ ( F λ , n, δ ) with probability at least 1 − δ , δ > 0 .
Revised: For ease of exposition, we consider as its generic upper-bound ∆ ( F λ , n, δ ) with probability at least 1 − δ , δ > 0 .

Original: Precise deﬁnitions of ∆ ( F λ , n, δ ) can be achieved using covering numbers (Corollary 6 in § B) or Radamacher complexities (Theorem 7 in § B).
Revised: Precise deﬁnitions of ∆ ( F λ , n, δ ) can be achieved using covering numbers (Corollary 6 in § B) or Radamacher complexities (Theorem 7 in § B).

Original: This ∆ ( F λ , n, δ ) is monotonically non-increasing with n and is indicative of the “size” or complexity of the class F λ – a more complex class will have higher maximum error discrepancy.
Revised: This ∆ ( F λ , n, δ ) is monotonically non-increasing with n and is indicative of the “size” or complexity of the class F λ – a more complex class will have higher maximum error discrepancy.

Original: 3.1 Small-scale Learning
Revised: 3.1 Small-scale Learning

Original: In the small-scale learning setup, after the selection of ^ λ by solving problem (2), either ( Case A ) the ERM solution ^ g n t , ^ λ on the training split in the inner problem (2) is utilized as is, or ( Case B ) ERM is performed with all the n samples available on F ^ λ to obtain the
Revised: In the small-scale learning setup, after the selection of ^ λ by solving problem (2), either ( Case A ) the ERM solution ^ g n t , ^ λ on the training split in the inner problem (2) is utilized as is, or ( Case B ) ERM is performed with all the n samples available on F ^ λ to obtain the

Original: ERM solution ^ f n, ^ λ = arg min f ∈F ^ λ E n ( f ) .
Revised: ERM solution ^ f n, ^ λ = arg min f ∈F ^ λ E n ( f ) .

Original: We study the diﬀerent factors aﬀecting the excess risk introduced in both these cases.
Revised: We study the diﬀerent factors aﬀecting the excess risk introduced in both these cases.

Original: See § B.4 for technical details and proofs.
Revised: See § B.4 for technical details and proofs.

Original: Case A (No model discrepancy).
Revised: Case A (No model discrepancy).

Original: This case is commonly used with deep neural networks where the models trained during HP (and architecture) selection are the ones ﬁnally utilized for deployment.
Revised: This case is commonly used with deep neural networks where the models trained during HP (and architecture) selection are the ones ﬁnally utilized for deployment.

Original: In the following theorem, we provide a bound on the excess risk:
Revised: In the following theorem, we provide a bound on the excess risk:

Original: Theorem 1 Let L = 2 | Λ | + 2 .
Revised: Theorem 1 Let L = 2 | Λ | + 2 .

Original: Then, with probability at least 1 − δ for any δ > 0 , the excess risk E = E
Revised: Then, with probability at least 1 − δ for any δ > 0 , the excess risk E = E

Original: (^ g n t , ^ λ ) − E ( f ∗ ) is bounded from above as:
Revised: (^ g n t , ^ λ ) − E ( f (cid:63) ) is bounded from above as:

Original: [Equation]
Revised: [Equation]

Original: We present this simple result for completeness to highlight how we will extend this result to provide bounds for the model selection scenarios we consider in the following.
Revised: We present this simple result for completeness to highlight how we will extend this result to provide bounds for the model selection scenarios we consider in the following.

Original: [Figure]

Original: Case B (Model discrepancy).
Revised: Case B (Model discrepancy).

Original: This case is common in ML where we wish to make the most of the available data and hence train the model for the selected HP ^ λ with the full training set of n samples.
Revised: This case is common in ML where we wish to make the most of the available data and hence train the model for the selected HP ^ λ with the full training set of n samples.

Original: However, this introduces a discrepancy – the learned model ^ g n t , ^ λ that guides the HP selection is diﬀerent from the model ^ f n, ^ λ whose excess risk we wish to understand.
Revised: However, this introduces a discrepancy – the learned model ^ g n t , ^ λ that guides the HP selection is diﬀerent from the model ^ f n, ^ λ whose excess risk we wish to understand.

Original: We provide a novel bound for the excess risk incurred in this process:
Revised: We provide a novel bound for the excess risk incurred in this process:

Original: Theorem 2 Let L 2 = 2 | Λ | + 3 .
Revised: Theorem 2 Let L 2 = 2 | Λ | + 3 .

Original: Let I n,n t , ^ λ = E n (^ g n t , ^ λ ) − E n (^ f n, ^ λ ) denote the “empirical risk improvement” obtained by reﬁtting the model on the full training set.
Revised: Let I n,n t , ^ λ = E n (^ g n t , ^ λ ) − E n ( ^ f n, ^ λ ) denote the “empirical risk improvement” obtained by reﬁtting the model on the full training set.

Original: Then, with probability at least 1 − δ for any δ > 0 , the excess risk E = E
Revised: Then, with probability at least 1 − δ for any δ > 0 , the excess risk E = E

Original: (^ f n, ^ λ ) − E ( f ∗ ) is bounded from above by:
Revised: (^ f n, ^ λ ) − E ( f (cid:63) ) is bounded from above by:

Original: [Equation]
Revised: [Equation]

Original: Note that I n,n t , ^ λ ≥ 0 by deﬁnition of ^ f n, ^ λ being the ERM solution in F ^ λ .
Revised: Note that I n,n t , ^ λ ≥ 0 by deﬁnition of ^ f n, ^ λ being the ERM solution in F ^ λ .

Original: Comparing to Case
Revised: Comparing to Case

Original: A in Theorem 1 to Theorem 2, we see the Case B involves more computation (training ^ f n, ^ λ ) but is only preferable over Case A if the (computable) empirical risk improvement I n,n t , ^ λ is relatively signiﬁcant, reducing the excess risk bound by that quantity.
Revised: A in Theorem 1 to Theorem 2, we see the Case B involves more computation (training ^ f n, ^ λ ) but is only preferable over Case A if the (computable) empirical risk improvement I n,n t , ^ λ is relatively signiﬁcant, reducing the excess risk bound by that quantity.

Original: An interesting aspect of this result is that the statistical cost of the additional ERM for ^ f n, ^ λ is O ( 1/ √ n ) but does not depend on the size of F ^ λ .
Revised: An interesting aspect of this result is that the statistical cost of the additional ERM for ^ f n, ^ λ is O ( 1/ √ n ) but does not depend on the size of F ^ λ .

Original: Based on the results, we propose a practical heuristic to select between ^ f n, ^ λ
Revised: Based on the results, we propose a practical heuristic to select between ^ f n, ^ λ

Original: and ^ g n t , ^ λ
Revised: and ^ g n t , ^ λ

Original: as the ﬁnal model to be deployed for improved excess risk:
Revised: as the ﬁnal model to be deployed for improved excess risk:

Original: Heuristic 1 Let us deﬁne the following data-dependent scalars α, β based on the quantities in Theorems 1 & 2, and we select ^ f n, ^ λ
Revised: Heuristic 1 Let us deﬁne the following data-dependent scalars α, β based on the quantities in Theorems 1 & 2, and we select ^ f n, ^ λ

Original: as the ﬁnal model if α ≥ β , or select ^ g n, ^ λ otherwise:
Revised: as the ﬁnal model if α ≥ β , or select ^ g n, ^ λ otherwise:

Original: [Equation]
Revised: [Equation]

Original: 3.1.1 Empirical validation
Revised: 3.1.1 Empirical validation

Original: For the purposes of empirical validation of our results and proposed heuristic, we consider a
Revised: For the purposes of empirical validation of our results and proposed heuristic, we consider a

Original: HPO problem with 36 neural network conﬁgurations ( | Λ | = 36 ) on a synthetic classiﬁcation data set with Bayes optimal risk E ( f ∗ ) = 0 .
Revised: HPO problem with 36 neural network conﬁgurations ( | Λ | = 36 ) on a synthetic classiﬁcation data set with Bayes optimal risk E ( f ∗ ) = 0 .

Original: We use synthetic data to have control over the experiment and generate large fresh samples to accurately estimate true risks E ( f ) of any model f .
Revised: We use synthetic data to have control over the experiment and generate large fresh samples to accurately estimate true risks E ( f ) of any model f .

Original: The estimation of E app ( λ ) and ∆ ( F λ , n, δ ) is detailed in § B.6.
Revised: The estimation of E app ( λ ) and ∆ ( F λ , n, δ ) is detailed in § B.6.

Original: We consider sample sizes n ∈ [ 2 9 , 2 14 ] and diﬀerent values for n v n ∈ [ 0.1, 0.5 ] with n t = n − n v .
Revised: We consider sample sizes n ∈ [ 2 9 , 2 14 ] and diﬀerent values for n v n ∈ [ 0.1, 0.5 ] with n t = n − n v .

Original: We present a subset of the results in Figure 1 (all results in Figure 3 in § B.6).
Revised: We present a subset of the results in Figure 1 (all results in Figure 3 in § B.6).

Original: The results are averaged over 10 trials.
Revised: The results are averaged over 10 trials.

Original: We set the failure probability δ = 0.05 .
Revised: We set the failure probability δ = 0.05 .

Original: Result (i).
Revised: Result (i).

Original: We compare the excess risk of ^ g n t , ^ λ and ^ f n, ^ λ (solid green & blue respectively) and their bounds from Theorems 1 and 2 (dashed green & blue respectively) in Figure 1a for varying values of n for 2 values of n v n .
Revised: We compare the excess risk of ^ g n t , ^ λ and ^ f n, ^ λ (solid green & blue respectively) and their bounds from Theorems 1 and 2 (dashed green & blue respectively) in Figure 1a for varying values of n for 2 values of n v n .

Original: In one case (Figure 1a left), the excess risk of ^ g n t , ^ λ 
Revised: In one case (Figure 1a left), the excess risk of ^ g n t , ^ λ [Figure]

Original: and ^ f n, ^ λ are close, and so are their respective bounds.
Revised: and ^ f n, ^ λ are close, and so are their respective bounds.

Original: In another case (Figure 1a right), ^ f n, ^ λ has better excess risk than ^ g n t , ^ λ , and that behavior is reﬂected in their respective bounds, with the Theorem 2 bound being slightly lower.
Revised: In another case (Figure 1a right), ^ f n, ^ λ has better excess risk than ^ g n t , ^ λ , and that behavior is reﬂected in their respective bounds, with the Theorem 2 bound being slightly lower.

Original: This result indicates that the common practice of generating ^ f n, ^ λ can sometimes lead to tighter excess risk bounds.
Revised: This result indicates that the common practice of generating ^ f n, ^ λ can sometimes lead to tighter excess risk bounds.

Original: Result (ii).
Revised: Result (ii).

Original: We explore the practical utility of Heuristic 1.
Revised: We explore the practical utility of Heuristic 1.

Original: Figure 1b compares this heuristic “Choice” (dashed red) to ^ f n, ^ λ & ^ g n t , ^ λ (dotted blue & green) – in most cases (diﬀerent values of n , n v n ), Heuristic 1 makes the best choice between ^ f n, ^ λ
Revised: Figure 1b compares this heuristic “Choice” (dashed red) to ^ f n, ^ λ & ^ g n t , ^ λ (dotted blue & green) – in most cases (diﬀerent values of n , n v n ), Heuristic 1 makes the best choice between ^ f n, ^ λ

Original: and ^ g n t , ^ λ .
Revised: and ^ g n t , ^ λ .

Original: 3.2 Large-scale Learning
Revised: 3.2 Large-scale Learning

Original: We now extend our analysis to the “large-scale” setting.
Revised: We now extend our analysis to the “large-scale” setting.

Original: In HPO, all the ERMs are solved approximately – the ERM for the inner problems in (2) during the selection of ^ λ is performed to ρ in tolerance to get approximate ERM solutions (cid:101) g n t ,λ , λ ∈ Λ , where
Revised: In HPO, all the ERMs are solved approximately – the ERM for the inner problems in (2) during the selection of ^ λ is performed to ρ in tolerance to get approximate ERM solutions (cid:101) g n t ,λ , λ ∈ Λ , where

Original: E n t ( (cid:101) g n t ,λ ) − E n t (^ g n t ,λ ) ≤ ρ in ; once ^ λ is selected, the ﬁnal ERM over F ^ λ with all n samples is performed to ρ out tolerance to get (cid:101) f n, ^ λ with E n ( (cid:101) f n, ^ λ ) − E n (^ f n, ^ λ ) ≤ ρ out .
Revised: E n t ( (cid:101) g n t ,λ ) − E n t (^ g n t ,λ ) ≤ ρ in ; once ^ λ is selected, the ﬁnal ERM over F ^ λ with all n samples is performed to ρ out tolerance to get (cid:101) f n, ^ λ with E n ( (cid:101) f n, ^ λ ) − E n (^ f n, ^ λ ) ≤ ρ out .

Original: Then, in the large scale setting, the HPO becomes the following problem:
Revised: Then, in the large scale setting, the HPO becomes the following problem:

Original: [Equation]
Revised: [Equation]

Original: Similar to the “small-scale learning” setting, we can bound the excess risk with and without model discrepancy.
Revised: Similar to the “small-scale learning” setting, we can bound the excess risk with and without model discrepancy.

Original: Here we present the no-model-discrepancy case, with the model-discrepancy case in Theorem 10 (and proofs) in § B.5 for lack of space:
Revised: Here we present the no-model-discrepancy case, with the model-discrepancy case in Theorem 10 (and proofs) in § B.5 for lack of space:

Original: Theorem 3 The excess risk E = E ( (cid:101) g n t , ^ λ ) − E ( f ∗ ) can be bounded from above with probability at least 1 − δ for any δ > 0 and L 3 =
Revised: Theorem 3 The excess risk E = E ( (cid:101) g n t , ^ λ ) − E ( f ∗ ) can be bounded from above with probability at least 1 − δ for any δ > 0 and L 3 =

Original: ( 2 + 3 | Λ | ) :
Revised: ( 2 + 3 | Λ | ) :

Original: [Equation]
Revised: [Equation]

Original: With no model discrepancy (Theorem 3), the approximate ERM in the inner problem (6) introduces an additional ρ in term to the excess risk compared to Theorem 1.
Revised: With no model discrepancy (Theorem 3), the approximate ERM in the inner problem (6) introduces an additional ρ in term to the excess risk compared to Theorem 1.

Original: In the presence of model discrepancy (Theorem 10), the approximation in the ERM shows up in two diﬀerent ways – ρ in from the approximation in the inner problem (6) similar to the no-model-discrepancy case, and ρ out from the approximation in the ﬁnal ERM. 
Revised: In the presence of model discrepancy (Theorem 10), the approximation in the ERM shows up in two diﬀerent ways – ρ in from the approximation in the inner problem (6) similar to the no-model-discrepancy case, and ρ out from the approximation in the ﬁnal ERM. [Figure]

Original: For any problem, ρ in and ρ out are set by the user (possibly based on available compute budget).
Revised: For any problem, ρ in and ρ out are set by the user (possibly based on available compute budget).

Original: However, in HPO with approximate ERM, we can leverage the excess risk bounds to make a more informed choice .
Revised: However, in HPO with approximate ERM, we can leverage the excess risk bounds to make a more informed choice .

Original: Based on Theorem 3, we can see that ρ in needs to be of the same order of magnitude as the other terms in (7) – any smaller ρ in will not improve the excess risk signiﬁcantly; any larger magnitude of ρ in will make this optimization error the dominant term in the excess risk.
Revised: Based on Theorem 3, we can see that ρ in needs to be of the same order of magnitude as the other terms in (7) – any smaller ρ in will not improve the excess risk signiﬁcantly; any larger magnitude of ρ in will make this optimization error the dominant term in the excess risk.

Original: In practice, we cannot compare ρ in and ρ out to the terms E app ( λ ) and ∆ ( F λ , n t , δ ) .
Revised: In practice, we cannot compare ρ in and ρ out to the terms E app ( λ ) and ∆ ( F λ , n t , δ ) .

Original: However, we can compare them to B (cid:112) 2 log ( L 3 /δ )
Revised: However, we can compare them to B (cid:112) 2 log ( L 3 /δ )

Original: /n v – if ρ in (cid:29) B (cid:112) 2 log ( L 3 /δ )
Revised: /n v – if ρ in (cid:29) B (cid:112) 2 log ( L 3 /δ )

Original: /n v , ρ in might dominate the excess risk, and it might be beneﬁcial to allocate more resources (if possible) to reduce ρ in to O ( 1/ √ n v ) ; however, reducing ρ in any[Figure] further will not signiﬁcantly improve the excess risk.
Revised: /n v , ρ in might dominate the excess risk, and it might be beneﬁcial to allocate more resources (if possible) to reduce ρ in to O ( 1/ √ n v ) ; however, reducing ρ in any further will not signiﬁcantly improve the excess risk.

Original: Based on this observation, we propose another heuristic for HPO with approximate ERM with regards to the choice of ρ in :
Revised: Based on this observation, we propose another heuristic for HPO with approximate ERM with regards to the choice of ρ in :

Original: Heuristic 

Revised: Heuristic

Original: Based on the terms deﬁned in Theorem 3 and a scaling parameter γ > 0 , we can set ρ in  = γB (cid:112) 2 log ( L 3 /δ ) 
Revised: Based on the terms deﬁned in Theorem 3, select a scaling parameter γ >and set ρ in as ρ in = γB (cid:112) 2 log ( L 3 /δ ) /n v such that ρ in ∼ o ( B (cid:112) 2 log ( L 3 /δ )

Original: /n v  .
Revised: /n v ) .

Revised: A value of γ = 0.1 suﬃces in our experience.

Original: 3.2.1 Empirical validation
Revised: 3.2.1 Empirical validation

Original: To demonstrate the practical utility of the proposed Heuristic 2, we continue with the aforementioned HPO problem over 36 neural network conﬁgurations on a synthetic classiﬁcation data.
Revised: To demonstrate the practical utility of the proposed Heuristic 2, we continue with the aforementioned HPO problem over 36 neural network conﬁgurations on a synthetic classiﬁcation data.

Original: We consider three choices for ρ in = γ (cid:112) 2 log ( L 3 /δ )
Revised: We consider three choices for ρ in = γ (cid:112) 2 log ( L 3 /δ )

Original: /n v with γ ∈ { 0.1, 1, 10 } .
Revised: /n v with γ ∈ { 0.1, 1, 10 } .

Original: We set δ = 0.05 and consider diﬀerent values of n and n v n (see Figure 4 in § B.6 for all results).
Revised: We set δ = 0.05 and consider diﬀerent values of n and n v n (see Figure 4 in § B.6 for all results).

Original: Figure 2 compares the performance of the exact ERM HPO using ^ g n t ,λ , λ ∈ Λ to approximation ERM HPO for diﬀerent ρ in , using (cid:101) g n t ,λ instead.
Revised: Figure 2 compares the performance of the exact ERM HPO using ^ g n t ,λ , λ ∈ Λ to approximation ERM HPO for diﬀerent ρ in , using (cid:101) g n t ,λ instead.

Original: In Figure 2a, we compare the excess risk incurred from approximate ERM with the data dependent choice of ρ in compared to exact ERM.
Revised: In Figure 2a, we compare the excess risk incurred from approximate ERM with the data dependent choice of ρ in compared to exact ERM.

Original: We see that γ = 0.1 leads to a suﬃciently small ρ in that matches the predictive performance of exact ERM.
Revised: We see that γ = 0.1 leads to a suﬃciently small ρ in that matches the predictive performance of exact ERM.

Original: Any smaller approximation ρ in would not improve the excess risk.
Revised: Any smaller approximation ρ in would not improve the excess risk.

Original: The results also indicate that γ = 10 leads to a ρ in where the optimization error dominates the excess risk, implying that ρ in should be reduced if possible.
Revised: The results also indicate that γ = 10 leads to a ρ in where the optimization error dominates the excess risk, implying that ρ in should be reduced if possible.

Original: Figure 2b presents the computational speedups obtained for the corresponding data-dependent choices of ρ in – we see that we can get a 2 × speedup over exact ERM without any degradation in excess risk with γ = 0.1 while obtaining around 4 − 6 × speedup with slight degradation in performance with γ = 1 .
Revised: Figure 2b presents the computational speedups obtained for the corresponding data-dependent choices of ρ in – we see that we can get a 2 × speedup over exact ERM without any degradation in excess risk with γ = 0.1 while obtaining around 4 − 6 × speedup with slight degradation in performance with γ = 1 .

Original: These results provide empirical evidence for the practical utility of the proposed Heuristic 2 obtained from Theorem 3 – the proposed heuristic provides a data driven way of setting the ERM approximation level in HPO .
Revised: These results provide empirical evidence for the practical utility of the proposed Heuristic 2 obtained from Theorem 3 – the proposed heuristic provides a data driven way of setting the ERM approximation level in HPO .

Original: 4. Conclusion
Revised: 4. Conclusion

Original: We study excess risk in diﬀerent HP selection scenarios.
Revised: We study excess risk in diﬀerent HP selection scenarios.

Original: The results allow us to make more informed decisions regarding choices in HPO.
Revised: The results allow us to make more informed decisions regarding choices in HPO.

Original: As future work, we wish to study similar tradeoﬀs in Bayesian optimization based HPO (Shahriari et al., 2016).
Revised: As future work, we wish to study similar tradeoﬀs in Bayesian optimization based HPO (Shahriari et al., 2016).

Original: We also plan to study multi-ﬁdelity HPO schemes such as Successive Halving (Jamieson and Talwalkar, 2016) where the ERM approximation in HPO is adaptively modiﬁed.
Revised: We also plan to study multi-ﬁdelity HPO schemes such as Successive Halving (Jamieson and Talwalkar, 2016) where the ERM approximation in HPO is adaptively modiﬁed

Original: Appendix A. Detailed notation and current decompositions
Revised: Appendix A. Detailed notation and current decompositions

Original: Consider samples ( x, y ) of covariates x and dependent variable y from a joint distribution
Revised: Consider samples ( x, y ) of covariates x and dependent variable y from a joint distribution

Original: P ( x, y ) over X × Y and loss function (cid:96) :
Revised: P ( x, y ) over X × Y and loss function (cid:96) :

Original: Y × Y → R .
Revised: Y × Y → R .

Original: The conditional distribution P ( y | x ) encodes the relationship between the covariates x and the dependent variable y .
Revised: The conditional distribution P ( y | x ) encodes the relationship between the covariates x and the dependent variable y .

Original: In this setup, the Bayes optimal predictor is given by
Revised: In this setup, the Bayes optimal predictor is given by

Original: [Equation]
Revised: [Equation]

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: For any model/function f : X → Y , the expected risk of the model is given by
Revised: For any model/function f : X → Y , the expected risk of the model is given by

Original: [Equation]
Revised: [Equation]

Original: Given a model class F , the true risk minimizer is f = arg min f ∈F E ( f ) .
Revised: Given a model class F , the true risk minimizer is f = arg min f ∈F E ( f ) .

Original: However, this requires access to the distribution P ( x, y ) .
Revised: However, this requires access to the distribution P ( x, y ) .

Original: In general, we consider n samples { ( x i , y i ) } ni = 1 ∼
Revised: In general, we consider n samples { ( x i , y i ) } ni = 1 ∼

Original: P ( x, y ) .
Revised: P ( x, y ) .

Original: The empirical risk is given by
Revised: The empirical risk is given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: and the empirical risk minimizer is ^ f n = arg min f ∈F E n ( f ) .
Revised: and the empirical risk minimizer is ^ f n = arg min f ∈F E n ( f ) .

Original: The excess risk E incurred by the empirical risk minimizer ^ f F is given by
Revised: The excess risk E incurred by the empirical risk minimizer ^ f F is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The approximation error E app is unknown in general but is assumed to reduce as the “size” of the class F grows.
Revised: The approximation error E app is unknown in general but is assumed to reduce as the “size” of the class F grows.

Original: The estimation error E est is bounded from above by 2∆ ( F , n, δ ) with probability at least 1 − 2δ (see Theorem 9).
Revised: The estimation error E est is bounded from above by 2∆ ( F , n, δ ) with probability at least 1 − 2δ (see Theorem 9).

Original: In practice, we solve the empirical risk minimization to some approximation, and we care about the excess risk incurred by an approximate empirical risk minimizer (cid:101) f n .
Revised: In practice, we solve the empirical risk minimization to some approximation, and we care about the excess risk incurred by an approximate empirical risk minimizer (cid:101) f n .

Original: The excess risk of (cid:101)
Revised: The excess risk of (cid:101)

Original: f n is given by
Revised: f n is given by

Original: [Equation]
Revised: [Equation]

Original: This decomposition of the excess risk includes a new optimization error E opt (Bottou and Bousquet, 2008).
Revised: This decomposition of the excess risk includes a new optimization error E opt (Bottou and Bousquet, 2008).

Original: If the approximate empirical risk minimizer satisﬁes the following for some ρ > 0 with probability at least 1 − δ ,
Revised: If the approximate empirical risk minimizer satisﬁes the following for some ρ > 0 with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Appendix B. Details for SectionB.1 Standard results and deﬁnitions
Revised: Appendix B. Details for SectionB.1 Standard results and deﬁnitions

Original: Deﬁnition 4 Let F be a class of functions X to Y .
Revised: Deﬁnition 4 Let F be a class of functions X to Y .

Original: For any (cid:15) > 0 , the associated (cid:96) ∞ covering number N ( F , (cid:15) ) of F is the minimal possible integer k such that F can be covered by k balls of (cid:96) ∞ radius (cid:15)
Revised: For any (cid:15) > 0 , the associated (cid:96) ∞ covering number N ( F , (cid:15) ) of F is the minimal possible integer k such that F can be covered by k balls of (cid:96) ∞ radius (cid:15)

Original: [Table]
Revised: [Table]

Original: We have the following standard result in statistical learning theory
Revised: We have the following standard result in statistical learning theory

Original: Theorem 5 Let F be a class of functions from X to Y and let E : Y (cid:48)
Revised: Theorem 5 Let F be a class of functions from X to Y and let E : Y (cid:48)

Original: × Y → [ 0, B ] be an
Revised: × Y → [ 0, B ] be an

Original: L -Lipschitz, B -bounded risk function.
Revised: L -Lipschitz, B -bounded risk function.

Original: Then, for any distribution P over X × Y ,
Revised: Then, for any distribution P over X × Y ,

Original: Pr { ( x i ,y i ) } ni = 1 ∼ P n (cid:18) sup f ∈F | E n ( f ) −

Original: [Equation]

Original: ( f

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation]

Original: (cid:19) ≤

Original: [Equation]

Original: N ( F

Original: [Equation]

Original: (cid:15) ) exp (cid:18) − n(cid:15)2B(cid:19) .

Original: Corollary 6 Under conditions of Deﬁnition 4 and Theorem [Table] with probability at least 1 − δ , 
Revised: Corollary 6 Under conditions of Deﬁnition 4 and Theorem 5, with probability at least 1 − δ , sup f ∈F

Original: [Equation]
Revised: [Equation]

Original: Theorem 7 (Bartlett and Mendelson (2002))
Revised: Theorem 7 (Bartlett and Mendelson (2002))

Original: Let P be a distribution over X × Y and let (cid:96) :
Revised: Let P be a distribution over X × Y and let (cid:96) :

Original: Y (cid:48)
Revised: Y (cid:48)

Original: × Y (where Y ⊆ Y (cid:48)
Revised: × Y (where Y ⊆ Y (cid:48)

Original: ⊂ R ) be a B -bounded loss function that is L -Lispschitz in its ﬁrst argument.
Revised: ⊂ R ) be a B -bounded loss function that is L -Lispschitz in its ﬁrst argument.

Original: Let F be a class of functions from X → Y .
Revised: Let F be a class of functions from X → Y .

Original: Then for any δ > 0 , with probability at least 1 − δ (over the random sample draw from P ), the following is true:
Revised: Then for any δ > 0 , with probability at least 1 − δ (over the random sample draw from P ), the following is true:

Original: [Equation]
Revised: [Equation]

Original: where R n ( F ) is the Radamacher complexity of the function class F .
Revised: where R n ( F ) is the Radamacher complexity of the function class F .

Original: B.2 Notation
Revised: B.2 Notation

Original: To concisely and conveniently prove results with somewhat similar arguments, we deﬁne the following notation for providing upper bounds:
Revised: To concisely and conveniently prove results with somewhat similar arguments, we deﬁne the following notation for providing upper bounds:

Original: Deﬁnition 

Revised: Deﬁnition

Original: For any quantity A and B such that A − B (cid:46)(cid:47) (cid:15) with probability at least 1 − δ for some numerical relational operator (cid:46)(cid:47) and δ > 0 , let us deﬁne the following notation:
Revised: For any quantity A and B such that A − B (cid:46)(cid:47) (cid:15) with probability at least 1 − δ for some numerical relational operator (cid:46)(cid:47) and δ > 0 , let us deﬁne the following notation:

Original: [Equation]
Revised: [Equation]

Original: For example, based on Corollary 6, for any function f in some class F and m samples, we can write:
Revised: For example, based on Corollary 6, for any function f in some class F and m samples, we can write:

Original: [Equation]
Revised: [Equation]

Original: where ≤ serves at the numerical relational operator (cid:46)(cid:47) , A = E ( f ) , B =
Revised: where ≤ serves at the numerical relational operator (cid:46)(cid:47) , A = E ( f ) , B =

Original: E m ( f ) , (cid:15) =
Revised: E m ( f ) , (cid:15) =

Original: ∆ ( F , m, δ ) .
Revised: ∆ ( F , m, δ ) .

Original: B.3 Bounds for existing decompositions
Revised: B.3 Bounds for existing decompositions

Original: Theorem 9 For any function class F , the estimation error E est = E (^ f n ) − E ( f ) from the
Revised: Theorem 9 For any function class F , the estimation error E est = E (^ f n ) − E ( f ) from the

Original: ERM solution with n samples is bounded from above by 2∆ ( F , n, δ ) with probability at least 1 − 2δ .
Revised: ERM solution with n samples is bounded from above by 2∆ ( F , n, δ ) with probability at least 1 − 2δ .

Original: Proof We have the following relationship:
Revised: Proof We have the following relationship:

Original: [Equation]
Revised: [Equation]

Original: B.4 Proofs for small-scale learning setup B.4.1 Proof for Theorem
Revised: B.4 Proofs for small-scale learning setup B.4.1 Proof for Theorem

Original: Proof
Revised: Proof

Original: First, consider the E hpo .
Revised: First, consider the E hpo .

Original: =
Revised: =

Original: E (^ g n t , ^ λ ) − E (^ g n t , ¯ λ ) term in E =
Revised: E (^ g n t , ^ λ ) − E (^ g n t , ¯ λ ) term in E =

Original: E
Revised: E

Original: (^ g n t , ^ λ ) − E ( f ∗ ) .
Revised: (^ g n t , ^ λ ) − E ( f (cid:63) ) .

Original: By deﬁnition of F ^ λ and ^ g n t , ^ λ , we have the following relationship for any F λ , λ ∈ Λ with some δ 
Revised: By deﬁnition of F ^ λ and ^ g n t , ^ λ , we have the following relationship for any F λ , λ ∈ Λ with some δ [Equation] > 0 :

Original: [Equation]
Revised: [Equation]

Original: Now for any F λ with corresponding λ , for some failure probability δ (cid:48) > 0 , we have the following relationship: 
Revised: Now for any F λ with corresponding λ , for some failure probability δ (cid:48) > 0 , we have the following relationship: [Equation] →

Original: [Equation]
Revised: [Equation]

Original: Thus, we have E ≤ E hpo + 2∆
Revised: Thus, we have E ≤ E hpo + 2∆

Original: ( F λ , n t , δ ) + E app ( λ ) for all λ ∈ Λ with a failure probability at most ( 2 + 2L ) δ (cid:48) using the union bound.
Revised: ( F λ , n t , δ ) + E app ( λ ) for all λ ∈ Λ with a failure probability at most ( 2 + 2L ) δ (cid:48) using the union bound.

Original: Minimizing the bound over λ and setting δ (cid:48)
Revised: Minimizing the bound over λ and setting δ (cid:48)

Original: =
Revised: =

Original: δ 2 ( 1 + L ) =
Revised: δ 2 ( 1 + L ) =

Original: δL 1 gives us the result in (3).
Revised: δL 1 gives us the result in (3).

Original: B.4.2 Proof for Theorem
Revised: B.4.2 Proof for Theorem

Original: Proof For the selected function class F ^ λ , we have the following relationship given deﬁnition of E ( · ) , E n ( · ) and I n,n t , ^ λ using Hoeﬀding’s inequality for some δ (cid:48) > 0 :
Revised: Proof For the selected function class F ^ λ , we have the following relationship given deﬁnition of E ( · ) , E n ( · ) and I n,n t , ^ λ using Hoeﬀding’s inequality for some δ (cid:48) > 0 :

Original: Let µ : ( X × Y ) n → R be the following function:
Revised: Let µ : ( X × Y ) n → R be the following function:

Original: [Equation]
Revised: [Equation]

Original: then we can show that the function µ satisﬁes the bounded diﬀerences condition
Revised: then we can show that the function µ satisﬁes the bounded diﬀerences condition

Original: [Equation]
Revised: [Equation]

Original: that is, there existing constants
Revised: that is, there existing constants

Original: c
Revised: c

Original: i , i ∈ [ n ] such that:
Revised: i , i ∈ [ n ] such that:

Original: [Equation]
Revised: [Equation]

Original: The bounded diﬀerence inequality tells us
Revised: The bounded diﬀerence inequality tells us

Original: [Equation]
Revised: [Equation]

Original: where C = (cid:80) ni
Revised: where C = (cid:80) ni

Original: = 1 c 2i .
Revised: = 1 c 2i .

Original: Given the deﬁnition of µ ( · ) 
Revised: Given the deﬁnition of µ ( · ) , E n ( · )

Original: [Equation]
Revised: [Equation]

Original: E n ( · ) , we can show that c i ≤ 2B/n and hence C
Revised: we can show that c i ≤ 2B/n and hence C

Original: = 4B 2 /n , giving us
Revised: = 4B 2 /n , giving us

Original: [Equation]
Revised: [Equation]

Original: The above combined with equations (22) & (23) gives us the following for all λ ∈ Λ with a failure probability of at most ( 3 + 2L ) δ (cid:48) for some δ (cid:48) > 0 :
Revised: The above combined with equations (22) & (23) gives us the following for all λ ∈ Λ with a failure probability of at most ( 3 + 2L ) δ (cid:48) for some δ (cid:48) > 0 :

Original: [Equation]
Revised: [Equation]

Original: Setting δ (cid:48)
Revised: Setting δ (cid:48)

Original: =
Revised: =

Original: δ/ ( 3 + 2L )
Revised: δ/ ( 3 + 2L )

Original: =
Revised: =

Original: δ/L 2 and minimizing over λ ∈ Λ gives us the result in (4)
Revised: δ/L 2 and minimizing over λ ∈ Λ gives us the result in (4)

Original: [Equation]
Revised: [Equation]

Original: B.5 Proofs for large scale learning setup B.5.1 Proof for Theorem
Revised: B.5 Proofs for large scale learning setup B.5.1 Proof for Theorem

Original: Proof By deﬁnition of F ^ λ , for any λ ∈ Λ and δ (cid:48) > 0 , we have the following relationships:
Revised: Proof By deﬁnition of F ^ λ , for any λ ∈ Λ and δ (cid:48) > 0 , we have the following relationships:

Original: [Equation]
Revised: [Equation]

Original: For any F λ , λ ∈ Λ
Revised: For any F λ [Equation] λ ∈ Λ

Original: [Equation]
Revised: [Equation]

Original: we have the following relationships by deﬁnition of ∆ ( · , · , · ) and (6):
Revised: we have the following relationships by deﬁnition of ∆ ( · , · , · ) and (6):

Original: [Equation]
Revised: [Equation]

Original: Since the above holds for any λ ∈ Λ , putting (29) and (30) together using the union bound with δ (cid:48)
Revised: Since the above holds for any λ ∈ Λ , putting (29) and (30) together using the union bound with δ (cid:48)

Original: =
Revised: =

Original: δ/ ( 2 + 3 | Λ | )
Revised: δ/ ( 2 + 3 | Λ | )

Original: = δ/L 3 and minimizing over λ ∈ Λ gives us the desired result in (7) with a failure probability of at most δ .
Revised: = δ/L 3 and minimizing over λ ∈ Λ gives us the desired result in (7) with a failure probability of at most δ .

Original: B.5.2 Excess risk for model discrepancy case with approximate ERM
Revised: B.5.2 Excess risk for model discrepancy case with approximate ERM

Original: Theorem 10 Let I n,n t , ^ λ = E n (^ g n t , ^ λ ) − E n (^ f n, ^ λ ) be the “empirical risk improvement” as in
Revised: Theorem 10 Let I n,n t , ^ λ = E n (^ g n t , ^ λ ) − E n (^ f n, ^ λ ) be the “empirical risk improvement” as in

Original: Theorem 2, and let (cid:101) I n,n t , ^ λ = E n ( (cid:101) g n t , ^ λ )− E n ( (cid:101) f n, ^ λ ) be corresponding the “approximate empirical risk improvement”.
Revised: Theorem 2, and let (cid:101) I n,n t , ^ λ = E n ( (cid:101) g n t , ^ λ )− E n ( (cid:101) f n, ^ λ ) be corresponding the “approximate empirical risk improvement”.

Original: The excess risk E = E ( (cid:101) f n, ^ λ ) − E ( f ∗ ) can be bounded as with probability at least 1 − δ for any δ > 0 and L 4 = ( 7 + 3 | Λ | ) :
Revised: The excess risk E = E ( (cid:101) f n, ^ λ ) − E ( f ∗ ) can be bounded as with probability at least 1 − δ for any δ > 0 and L 4 = ( 7 + 3 | Λ | ) :

Original: [Equation]
Revised: [Equation]

Original: where B (cid:48)
Revised: where B (cid:48)

Original: = 2B (cid:113) log L 4 δ .
Revised: = 2B (cid:113) log L 4 δ .

Original: Proof We will begin by bounding the term E ( (cid:101) f n, ^ λ ) − E ( (cid:101) g n t , ^ λ ) using Hoeﬀding’s inequality with failure probability δ (cid:48)
Revised: Proof We will begin by bounding the term E ( (cid:101) f n, ^ λ ) − E ( (cid:101) g n t , ^ λ ) using Hoeﬀding’s inequality with failure probability δ (cid:48)

Original: [Equation]
Revised: [Equation]

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: This gives us the following with failure probability 5δ (cid:48) :
Revised: This gives us the following with failure probability 5δ (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: With B (cid:48)
Revised: With B (cid:48)

Original: = 2B (cid:112) log ( 1/δ (cid:48) ) , we can rewrite above as[Equation] [Equation] I
Revised: = 2B (cid:112) log ( 1/δ (cid:48) ) , we can rewrite above as

Original: [Equation]

Original: ^

Original: [Equation]

Original: (cid:48) √ 2n −

Original: [Equation]

Original: √ 2n t

Original: [Equation]
Revised: [Equation]

Original: 41) with failure probability of 5δ (cid:48) .
Revised: with failure probability of 5δ (cid:48) .

Original: This combined with equations (29) & (30) using the union bound with δ (cid:48)
Revised: This combined with equations (29) & (30) using the union bound with δ (cid:48)

Original: =
Revised: =

Original: δ/ ( 7 + 3 | Λ | ) =
Revised: δ/ ( 7 + 3 | Λ | ) =

Original: δ/L 4 gives us (31) with probability at least 1 − δ .
Revised: δ/L 4 gives us (31) with probability at least 1 − δ .

Original: B.6 Details on Empirical Validation
Revised: B.6 Details on Empirical Validation

Original: Synthetic data.
Revised: Synthetic data.

Original: The binary classiﬁcation data is generated using the make classification function (Guyon, 2003) in the datasets module of scikit-learn (Pedregosa et al., 2011).
Revised: The binary classiﬁcation data is generated using the make classification function (Guyon, 2003) in the datasets module of scikit-learn (Pedregosa et al., 2011).

Original: We ensure that the classes are not overlapping and there is no label noise, ensuring that
Revised: We ensure that the classes are not overlapping and there is no label noise, ensuring that

Original: the
Revised: the

Original: Bayes optimal risk E ( f ∗ ) = 0 .
Revised: Bayes optimal risk E ( f (cid:63) ) = 0 .

Original: HPO search space.
Revised: HPO search space.

Original: We considered 36 conﬁgurations for a fully connected multi-layered perceptron, varying (i) the depth, (ii) the number of neurons in each layer, (iii) the initial learning rate for the SGD optimizer, (iv) the batch size for SGD.
Revised: We considered 36 conﬁgurations for a fully connected multi-layered perceptron, varying (i) the depth, (ii) the number of neurons in each layer, (iii) the initial learning rate for the SGD optimizer, (iv) the batch size for SGD.

Original: The implementation is in
Revised: The implementation is in

Original: PyTorch (Paszke et al., 2019).
Revised: PyTorch (Paszke et al., 2019).

Original: Bound computation.
Revised: Bound computation.

Original: For any HP λ , (i) the approximation error E app ( λ ) is estimated as the best test-set error during ERM on F λ with training sets of diﬀerent sizes and diﬀerent
Revised: For any HP λ , (i) the approximation error E app ( λ ) is estimated as the best test-set error during ERM on F λ with training sets of diﬀerent sizes and diﬀerent

Original: ERM restarts, (ii) the maximum error discrepancy sup f ∈F λ | E n ( f ) − E ( f ) | ≤ ∆ ( F λ , n, δ ) is approximated with the empirical Radamacher complexity for F λ .
Revised: ERM restarts, (ii) the maximum error discrepancy sup f ∈F λ | E n ( f ) − E ( f ) | ≤ ∆ ( F λ , n, δ ) is approximated with the empirical Radamacher complexity for F λ .

Original: Thus, ∆ ( F λ , n, δ ) is deﬁned by the following quantity:
Revised: Thus, ∆ ( F λ , n, δ ) is deﬁned by the following quantity:

Original: [Equation]
Revised: [Equation]

Original: B.6.1 Complete results
Revised: B.6.1 Complete results

Original: The complete results for all values of n v n ∈ [ 0.1, 0.5 ] for Figure 1 and Figure 2 in Figureand Figure 4 respectively.
Revised: The complete results for all values of n v n ∈ [ 0.1, 0.5 ] for Figure 1 and Figure 2 in Figureand Figure 4 respectively.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]