Original: exists, and l < ˜ l, η [Equation]
Revised: Closed interval of solutions (Item 2) We again differentiate whether ˜ l as in the previous step exists or not. If ˜ l does not exist, then the previous step already yields ℓ λ min = { 0 } , which is a closed interval.

Original:  l ) otherwise.
Revised: Next, assume ˜ l exists.

Original: Furthermore, since η v is non-increasing, and η b is decreasing, we have that, in the case where ˜ l existsfor any l , η (˜ l) ≤ η ( l ) for any other l ∈ (cid:2) 0 ,¯ ℓ (cid:3) . Hence, ˜ l ∈ ℓ λ min .
Revised: Then, all l ∈ ℓ λ min satisfy l ≤ ˜ l , since ˜ l ∈ ℓ λ min and η b is strictly increasing. Since further η v is continuous and non-increasing over (cid:104) 0 , ˜ l (cid:105) ⊇ ℓ λ min , ℓ λ min is an interval.

Original: In the case where ˜ l does not exist, wehave that η ( l ) =  η b ( l ) , which is a monotonically increasing function.
Revised: Finally, η ( l ) = max { η b ( l ) , η v ( l ) } is the maximum of two continuous functions, hence itself also continuous.

Original: Hence, it is minimal in 0 .
Revised: Therefore, the minimizers ℓ λ min of η are a closed set.

Original: Thisproves Item 2.
Revised: This concludes that ℓ λ min is a closed interval.

Original: Proof of Item 3 and  Item 4: Item 3 follows directly from the fact that η ( l ) =
Revised: Proof of Items 4 and 5 Item 4 follows straightforwardly from Equation (41): If ˜ l does not exist, then η ( l ) =

Original: η b ( l )  if ˜ l does notexist, or for any l ≥ ˜ l , and ˜ l ∈ ℓ λ min .
Revised: η b ( l ) for all l ∈ (cid:2) 0 , ¯ ℓ (cid:3) . Similarly, if ˜ l exists, η ( l ) = η b ( l ) for all l ≥ ˜ l , in particular for l ≥ ˜ l ∈ ℓ λ min .

Original: Item 4 requiressome additional steps.
Revised: Item 5 requires additional considerations.

Original: In the case where ˜ l does not exist, we havethat the ℓ λ min = { 0 } ,hence the result follows directly.
Revised: In the case where ˜ l does not exist, we have ℓ λ min = { 0 } , and the result follows directly.

Original: Otherwise, let l min = min ℓ λ min : with the same argument as for Item 3, we havethat η ( l ) =
Revised: Otherwise, using Equation (41), we have η ( l ) =

Original: η v ( l ) forany l ≤ l  min  .
Revised: η v ( l ) for any l ≤ l , as l ≤ min( ℓ λ min ) ≤ ˜ l .

Original: The function η v ( l ) alternates steps with derivative 0 , with steps with derivative − 2 /ℓ .
Revised: As shown in the proof of Item 1, η v ( l ) alternates between derivativesand − 2 /ℓ .

Original: We argue that there exists a left neighborhood ofl min  where the derivative is − 2 /ℓ .
Revised: We claim that there exists a left neighborhood of min ( ℓ λ min ) where the derivative is − 2 /ℓ .

Original: By contradiction, assume that this neighborhood does not exist: since the function η v ( l ) alternates between intervals of fixed size with derivative − 2 /ℓ and 0 , it would imply that there exists a left neighborhood  where the derivative is 0 .
Revised: Assume towards a contradiction that no such left neighborhood exists. Then, there must be a left neighborhood of min ( ℓ λ min ) where the derivative is 0 , since η v ( l ) alternates between only two derivatives.

Original: But this would hence mean that within this neighborhood,the function is constant, and all the values η v ( l ) are equal to η min , which contradicts the hypothesis that l  min  =  min  ℓ λ min .
Revised: However, in that left neighborhood, η v ( l ) is constant with all values equal to η min . Hence, there exists l < min ( ℓ λ min ) with η ( l ) = η v ( l ) = η min and thus l ∈ ℓ λ min , which is a contradiction.

Original: Hence, there exist such left neighborhood. Let ε ℓ σ be the diameter of such interval, then, for any l ∈ [ l min  − ε ℓ σ , l min ]
Revised: Thus, there exists a left neighborhood of min ( ℓ λ min ) with diameter ε > 0 throughout which the derivative is − 2 ℓ . Then, for all l ∈ [min ( ℓ λ min ) − ε, min ( ℓ λ min )] ,

Original: [Equation]
Revised: [Equation]

Original: Finally, since the function is non-decreasing, as long as η v ( l ) < η min + 2ε ℓσ ℓ , we havethat l must belarger than l  min  − εℓ σ .
Revised: Finally, since η v ( l ) is non-increasing, as long as η v ( l ) ≤ η min + 2 ℓ ε we have l ≥ min ( ℓ λ min ) − ε .

Original: Taking cℓ σ = 2ε ℓσ ℓ , concludes the proof of Item 4.
Revised: Hence, choosing c = 2 ℓ ε yields the statement of Item 5.

Original: D. P ROOF OF T HEOREM2: T RAINING ERROR
Revised: D. P ROOF OF T HEOREM

Original: The formal version Theorem 2states a bound for the expected training error in two settings: • The optimally regularized model achieves a better risk rate than the interpolating one.
Revised: The informal Theorem 2 in the main text relies on a β ∗ , defined as the intersection of the variance and bias rates from Theorem 1 for the interpolator ˆ f 0 (setting ℓ λ = 0 ).

Original: • The optimally regularized model achieves the same risk rate as the interpolating one.
Revised: Whenever β ∗ is unique, the fact that Bias 2 ( ˆ f 0 ) in Theorem 1 strictly increases as a function of β induces a phase transition: for β > β ∗ , the bias dominates the rate of the risk in Theorem 1, while for β ≤ β ∗ , the variance dominates. In particular, Lemmas 20 and 21 imply that interpolation is harmless whenever the bias dominates, and harmful if the variance dominates. Intuitively, Theorem 2 considers optimally regularized estimators, and varies the inductive bias strength via β . The formal Theorem 4 below presents a different perspective: it considers β fixed, and instead differentiates whether the optimal risk rate in Theorem 1 results from ℓ λ > 0 (harmful interpolation) or ℓ λ = 0 (harmless interpolation).

Original: The quantity β ∗ is defined in the main text as  the intersection between variance and bias. Sinceregularization benefits the  variance and hurts the bias, it is the case that only for values of β > β ∗ , i.e.,where variance dominates bias, that the regularization helps.
Revised: Whenever β ∗ is well-defined and unique, as for example in the setting of Figure 1, the two perspectives coincide. However, one can construct pathological edge cases where variance and bias rates intersect on an interval of β values, or the dominating quantity of the risk rate in Theorem 1 as a function of β alternates between the variance and bias.

Original: This how we state the formal version of Theorem 2 as in Theorem 4.
Revised: While Theorem 2 fails to capture such settings, Theoremstill applies. We hence present Theorem 4 as a more general result.

Original: Theorem 4 (Formal version of Theorem 2) .
Revised: Theorem 4 (Formal version of Theorem 2) .

Original: In thesame setting as Theorem 1, letus fix ℓ > 0 , β ∈(0 , 1) , ℓ σ ≥ 0 , ¯ ℓ = ℓ − 1− β ( L ∗ − 1) .
Revised: In the setting of Theorem 1 using the notation from Appendix D.1, let ℓ > 0 , β ∈ (0 , 1) , ℓ σ ≥ − ¯ ℓ , and L ∗ ∈ (cid:104) 1 , ⌈ ℓ − 1 β ⌉ (cid:105) ∩ N .

Original: Finally, let λ opt be such that Risk ( ˆ f λ opt ) is minimal. If l ∈ [0 , ¯ ℓ{+≥ 
+}
Revised: Let further λ opt = arg min λ {+≥ 
+}

Original: that minimizes η ( l ; ℓ ℓσ , L ∗ , β ) is such that η
Revised: | max { λ, 1 }∈O ( d ¯ ℓ ) Risk ( ˆ f λ ) .

Original: l
Revised: Then, the expected training error behaves as follows:

Original:  ; ℓ, ℓ σ , L ∗ , β ) <
Revised: If all l ∈ arg min l ∈ [0 , ¯ ℓ ] η ( l ; ℓ, ℓ σ , L ∗ , β ) satisfy η ( l ; ℓ, ℓ σ , L ∗ , β ) <

Original: η (0; ℓ, ℓ σ , L ∗ , β ) , then: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) E ϵ (cid:34) n (cid:88) i 
Revised: η (0; ℓ, ℓ σ , L ∗ , β ) , then (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) E ϵ (cid:34) n (cid:88) i ( ˆ f

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:35) [Equation] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: opt

Original: [Equation]
Revised: [Equation]

Original: ( d
Revised: x i ) − y i )

Original: [Equation]
Revised: [Equation]

Original:  ( ˆ f [Equation]
Revised: If all l ∈ (0 , ¯ ℓ ] satisfy η ( l ; ℓ, ℓ σ , L ∗ , β ) > η (0; ℓ, ℓ σ , L ∗

Original: opt )) , w
Revised: , β ) , then

Original: [Equation]
Revised: [Equation]

Original: p [Equation] [Equation]
Revised: for some

Original: min [Equation]
Revised: constant ˜ c <

Original: If, forany l ∈ (0 , ¯ ℓ ] , η ( l ; ℓ, ℓ σ , L ∗ , β ) > η (0; ℓ, ℓ σ , L ∗ , β ) , then: E ϵ (cid:34) n (cid:88) i ( x
Revised: Intuitively, the two cases in Theorem 4 correspond to harmful and harmless interpolation, where the optimal rate in Theorem 1 is for ℓ λ > 0 and ℓ λ = 0 , respectively.

Original: [Equation] y )(cid:35) [Equation]
Revised: Then, Lemma 20 yields with high probability that also ℓ λ opt > 0 and ℓ λ opt = 0 in the first and second case, respectively.

Original: O (cid:16) Risk [Equation] ( ˆ f
Revised: Finally, we remark that Theorem 4 lacks an edge case: if both some ℓ λ > 0 and ℓ λ = 0 minimize the risk rate in

Original: [Equation] opt ) (cid:17) , w. [Equation] 1 − δ } .
Revised: Theorem 1 simultaneously, Lemma 20 fails to differentiate whether ℓ λ opt is zero or positive.

Original: For some constant c < 1 . This theorem covers the case where either regularization is harmful, or it is beneficial. There arestill some edge cases where, for small ℓ λ , the rate of the risk is the same as for the interpolatingmodel, and for large ℓ λ , the performance deteriorate.
Revised: However, that edge case corresponds to either interpolation or very weak regularization.

Original: In this case our theory does not allow discerningweather the expected training matches the noise level or not.
Revised: Hence, we conjecture the corresponding model’s training error to behave similar to the second case in Theorem 4.

Original: Prood of Theorem 4.
Revised: Proof of Theorem 4.

Original: Throughout the proof we will use withhigh probability to mean with probabilityat least 1 − cd − β min { ¯ δ, 1 − ¯ δ }. Thanks to Lemma 20, we know that max { λ opt , 1 } ∈ Θ( d l ) , where lminimizes η ( l ; ℓ, ℓ σ , L ∗ , β ) .
Revised: First, Lemma 20 yields with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } that max { λ opt , 1 } ∈ Θ( d l ) , where l is a minimizer of η ( l ; ℓ, ℓ σ , L ∗ , β ) .

Original: The condition in Item 1 guarantees that the optimal l is positive, and thecondition in Item 1 guarantees that the optimal l is 0 .
Revised: The condition in Item 1 guarantees that all optimal l are positive, while the condition in Item 2 ensures that the optimal l = 0 .

Original: Harmful interpolation setting.
Revised: Harmful interpolation setting (Item 1)

Original: In this setting we have that λ
Revised: In this case, l > 0 , and thus λ

Original: opt ∈ Θ( d l ) , l > 0 .
Revised: opt ∈ Θ( d l ) for d sufficiently large.

Original: We start by applying Theorem 1 with ℓ λ =
Revised: We start by applying Theorem 1 with ℓ λ =

Original: l in this setting.
Revised: l in this setting.

Original: This confirms that, with the same m chosen for the proof, Equation (5) holds with high probability, and  the ground truth is in the span of the first m eigenfunctions.
Revised: Within the proof of Theorem 1 in Section 5.2, we pick a m ∈ N such that for d sufficiently large, with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , m satisfies the conditions of Lemmas 1 to 4 and Theorem 3. For the remainder of this proof, let m be the same as in the proof of Theorem 1 for ℓ λ = l .

Original: Hence, we apply Lemma 22:
Revised: This m satisfies the conditions of Lemma 23, which hence yields λ 2 opt σ

Original: [Equation]
Revised: [Equation]

Original:  Then, by subtracting σ 2 and taking the absolute value we get:
Revised: n, where H : = K + λ I n . Then, using that a ≤ b ≤ a + d implies | b − c | ≤ | a − c | + d , we further get (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Revised: E ϵ (cid:34) n (cid:88)

Revised: i

Original: [Equation]
Revised: [Equation]

Original: ) − y i )(cid:35) − σ(cid:12) (cid:12)(cid:12)(cid:12) (cid:12) ≤σ(cid:12) (cid:12)(cid:12)(cid:12) (cid:12) λ 2 opt n Tr (cid:0)
Revised: (cid:35) [Equation] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: H −(cid:1) −(cid:12)
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12) (cid:12) (cid:124) (cid:123)(cid:122) (cid:125) :
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: = T+(cid:12)(cid:12)(cid:12) (cid:12) λ opt 2 ∥
Revised: [Equation]

Original: D ≤ m − 1 a ∥n(cid:12)(cid:12)(cid:12) (cid:12) (cid:124) (cid:123)(cid:122) (cid:125):
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:124) (cid:123)(cid:122) (cid:125)

Original: = T.
Revised: [Equation] T

Original: To bound T 2 we do:

Original: [Equation]
Revised: [Equation]

Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Original: [Equation]
Revised: [Equation]

Original: iii
Revised: T

Original: [Equation]
Revised: [Equation]

Original: (cid:16) Bias 2 ( ˆ f λ
Revised: We first bound T 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: ) (cid:17) ∈ O (cid:16) Risk 2 ( ˆ f λ opt ) (cid:17) , wherein ( i ) we used that n ∈ Θ( d ℓ ) , λ opt ∈ Θ( d l ) .
Revised: where ( i ) uses n ∈ Θ( d ℓ ) and λ opt ∈ Θ( d l ) , ( ii ) applies Lemma 2 for the rate of ∥ D − 1 ≤ m a ∥ and

Original: In step ( ii ) we used Lemma 2 to have the rate of∥ D ≤ m − 1 a ∥ and Lemma 3 for r 22 , r 21 ∈ Θ(1) .
Revised: Lemma 3 for r 22 , r 21 ∈ Θ(1) , and ( iii ) matches the expression to the rate of the bias in Theorem 1. For T 1 , we will bound Tr [Equation] from above and below using Lemma 22.

Original: In ( iii ) we see that this rate matches the rate of thebias found in Theorem 1.
Revised: We hence introduce the following notation: [Equation]

Original: To bound T 1 we apply Lemma 21, which yields with high probability:
Revised: so that H = H 2 + K − 2 , and where K 1 and K 2 are defined in Appendix C.2. Furthermore, the rank of K − 2 is at most | N \ I 2 | = |{ k ∈ N | | S k | < ¯ L + 2 }| , that is, the number of eigenfunctions that contribute to K − 2 . The rate of | N \ I 2 | is

Revised: [Equation]

Revised: where ( i ) uses the definition of C ( l, q, d ) in Equation (17), ( ii ) applies Lemma 9 with d sufficiently large, and ( iii ) uses the classical bound

Revised: [Equation]

Revised: l

Revised: [Equation]

Revised: 1 as well as the fact that the largest monomial dominates the sum.

Revised: Finally, since n ∈ Θ( d · q ¯ L +¯ δ ) , we have

Revised: [Equation]

Revised: Therefore, for d and hence n ∈ Θ( d ℓ ) sufficiently large, rank( K − 2 ) < n , and Lemma 13 yields c 1 ≤ µ min ( H 2 ) ≤ ∥ H 2 ∥ ≤ c 2 for some constants c 1 , c 2 > 0 with probability at least 1 − c ′ q − (1 − ¯ δ ) .

Revised: We can thus instantiate Lemma 22 with M = H , M 1 = K − 2 , M 2 =

Revised: H 2 .

Revised: This implies that:

Original: [Equation]
Revised: [Equation]

Original: Since c 1 and c 2 are constants, we can bound the absolute value of the inner part as:
Revised: The upper bound on Tr (cid:0) H − [Equation] simply follows from

Revised: [Equation]

Revised: where ( i ) uses the previous lower bound on µ min ( K 2 ) from Lemma 13.

Original: [Equation]
Revised: [Equation]

Revised: yields

Original: where in ( i ) we use the rate of | N \ I 2 | computed in Lemma 21 itself.
Revised: [Equation] Taking absolute values [Equation] a simple case distinction and c 1 < c 2 yields the following bound:

Original: Concluding the proof for thiscase.
Revised: [Equation] where ( i ) uses the rate of rank( K − 2 ) from Equation (44) and λ opt ∈ Θ (cid:0) d l (cid:1) . Combining this bound on T 1 with the bound on T 2 in Equation (43) and collecting all error probabilities concludes the current case.

Original: Harmless interpolation setting.
Revised: Harmless interpolation setting (Item 2)

Original: In this setting we know that the  minimizer of the  rate is  0 , and hence, thanks to Lemma 20, we know that:
Revised: In this setting, the only minimizer of the risk rate is l = 0 , and hence

Original: [Equation]
Revised: [Equation]

Original: with high prob. Like in the other setting, we start by applying Theorem 1 with ℓ λ =l = 0 .
Revised: As for the previous case, let m be the same as in the proof of Theorem 1 for ℓ λ = 0 .

Original: Hence, we apply Lemma 22 and look at  the upper bound:
Revised: With probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , this m again satisfies the conditions of Lemma 23, which yields

Original: [Equation]
Revised: [Equation]

Original: For T 2 we move analogously to the previous case (Equation (31)) in the specific case where l = 0 . This yields:
Revised: Furthermore, we apply the same steps with l = 0 as in the previous case (Equation (43)) to bound T 2 :

Original: [Equation]
Revised: [Equation]

Original: To bound T 1 we apply Lemma 21, which yields with high probability:
Revised: For T 3 , we use the same bound on Tr (cid:0) H −(cid:1) as in Equation (45) with the same probability as follows:

Original: [Equation]
Revised: [Equation]

Original: wherein ( i ) we used that with high probability λ opt ∈ Θ(1) , hence there exists a C > 0 such thatλ opt ≤ C with high probability.
Revised: where ( i ) follows for d sufficiently large from λ opt ∈ O (1) with c ′′ > 0 . Since c 1 > 0 , we have ( c ′′ ) 2 ( c 1 + c ′′ ) 2 < 1 .

Original: Finally, it is clear that C ( c 1 + C ) 2 < 1 , Concluding the proof.
Revised: Hence, combining the bounds on T 2 and T 3 , as well as collecting all probabilities, we get the desired result for this case.

Original: D. T ECHNICAL L EMMAS
Revised: D. T ECHNICAL LEMMAS

Original: Lemma 21 (Trace of the inverse) .
Revised: Lemma 22 (Trace of the inverse) .

Original: In the setting defined in Section 3.1. Then, with high probability, [Equation] for two positive constants c 1 , c 2 .
Revised: Let M , M 1 , M 2 ∈ R n × n be symmetric positive semi-definite matrices with M = M 1 + M 2 .

Original: Furthermore, | N \ I 2 | , I 2 , ¯ δ defined as in Definition 2. Proof. We start by computing | N \ I 2 | .
Revised: Furthermore, assume that µ min ( M 2 ) > 0 and rank( M 1 ) < n .

Original: Computing | N \ I 2 | corresponds to counting how many indexes do not appear in I 2 . Give the definition of the latter, for an index not to be in I 2 , it means that | S k | ≤ ¯ L + 1 . Hence, the total number of such index can be at most:
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where in ( i ) we use the bound the binomial coefficient

Original: [Equation]

Original: l − 1 .

Original: Finally, since n ∈Θ( d · q ¯ L +¯ δ ) , we have that | N \ I 2 | ∈

Original: [-O ( nq − ¯ δ ) . Now we move to the proof of Equation (32). We divide H = K ≤ m + K 1 + K 
 + λ I , where K 1 and-]
Revised: [-O ( nq − ¯ δ ) . Now we move to the proof of Equation (32). We divide H = K ≤ m + K 1 + K 
 + λ I , where K 1 and-]Proof.

Original: K 2 are defined as in Definition 2.

Original: Now, we [-define H 2 = K 
 + λ I . For-] the lower bound it suffices to see:
Revised: First, we [-define H 2 = K 
 + λ I . For-]apply the identity

Original: [Equation]
Revised: [Equation]

Original: with prob. ≥ 1 −
Revised: which holds since M

Original: [Equation]
Revised: [Equation]

Original: , for some constant c 1 . In ( i ) we used Lemma 13 to bound µ min ( K 2 ) . For the upper bound we need a more refined analysis. Let us define the following matrix:
Revised: and thus M

Original: [Equation]
Revised: [Equation]

Original: Notice that K = K − 2 + K 2 , which gives an intuition for the notation. Thanks to Lemma 13, we know that K 2 is non-singular wit probability at least 1 − cd − β (1 − ¯ δ ) . Furthermore, since K − 2 is still positive semi-definite, also K and H and H 2 are non-singular.
Revised: are full rank.

Original: Thus, we can apply the classical linear algebra formula
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: Since K − 2 is the kernel matrix that stems from a kernel with exactly | N \ I 2 | features, it has rank atmost | N \ I 2 | . Since A is defined as the product ofsome matrices includingK − 2 , thus, it also hasrank at most | N \ I 2 | . Notice furthermore, that since | N \ I 2 | ∈ O ( nd − β ¯ δ ) ⊆
Revised: A is a product of matrices including

Original: o ( n ), we have that | N \ I 2 | < n for d sufficiently large.
Revised: M 1 ; hence, the rank of A is bounded by rank( M 1 ) < n .

Original: Let now { v 1 , . .
Revised: Let now { v 1 , . .

Original: . , v | N \I 2 | } be an orthonormal basis of col ( A ), and { v | N \I 2 | +1 , . . . , v n }  an orthonormal basis of col ( A )⊥ . This makes { v 1 , . . . , v n } a full orthonormal basis of R n .
Revised: . , v rank( A ) } be an orthonormal basis of col( A )

Original: Now, since the trace is invariant to similarities, we get that:

Original: [Equation]
Revised: [Equation]

Original: i [Equation] (cid:124)(cid:123)(cid:122)(cid:125) = [Equation]
Revised: and let { v rank( A )+1 , . .

Original: (cid:124)(cid:123)(cid:122)(cid:125) [Equation] (cid:124) [Equation]
Revised: . , v n } be an orthonormal basis of col( A ) ⊥ .

Original:  [Equation] [Equation]
Revised: Thus, { v 1 , . .

Original: c
Revised: . , v n } is an orthonormal basis of R n

Original: [Equation]
Revised: [Equation]

Original: + λ
Revised: and similarity invariance of the trace yields

Original: [Equation]
Revised: [Equation]

Original: with prob. ≥ 1 − cd − β (1 − ¯ δ ) , wherein ( i ) we use the fact that, as long as i > | N \ I 2 | , we have that Av i = v ⊺ i A = 0 ,since v i isorthogonal to the column space.
Revised: where ( i ) uses that, for all i > rank( A ) , v i is orthogonal to the column space of A , and A is symmetric.

Original: In ( ii ) we used the fact that the vectors v i in the orthonormal basishave norm 1 , andin ( iii ) we used Lemma 13 to bound ∥ K 2 ∥ .
Revised: Furthermore, ( ii ) uses that all v i have norm 1 , and ( iii ) that rank( A ) ≤ rank( M 1 ) .

Original: Lemma 22 (Fixed design training error) .
Revised: Lemma 23 (Fixed-design training error) .

Original: In thesame setting as Theorem 3, let m be such that 
Revised: In the setting of Theorem 3, let m ∈ N such that r 1 > 0 ,

Original: Equation (5) holds and the ground truth is in the span of the first m eigenfunctions.
Revised: Equation (5) holds, and the ground truth satisfies f ⋆ ( [Equation] Then,

Original: [Equation]
Revised: [Equation]

Revised: where H :

Revised: =

Revised: K + λ I n .

Original: Proof.
Revised: Proof.

Original: In kernel ridge regression, our estimator is:
Revised: The kernel ridge regression estimator is

Original: [Equation]
Revised: [Equation]

Original: where y :
Revised: where y :

Original: =
Revised: =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y n ] ⊺  and k ( x ∗ ) :
Revised: , y n ] ⊺ with y i = f ⋆ ( x i ) + ϵ i , and k ( x ∗ ) :

Original: =
Revised: =

Original: [ K ( x 1 , x ∗ ) , . .
Revised: [ K ( x 1 , x ∗ ) , . .

Original: , K ( x n , x ∗ )] ⊺, H = K + λ I n .
Revised: , K ( x n , x ∗ )] ⊺ .

Original: Hence, our estimator, evaluated at x 1 , . . .
Revised: Thus, the estimator evaluated at x 1 , . . .

Original: , x n is KH − 1 y . Hence, the  training error is:
Revised: , x n is KH − 1 y , which yields the following training error:

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows fromthe simple calculation:
Revised: where ( i ) follows from

Original: I 
Revised: I n

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Next

Original: [Equation]
Revised: [Equation]

Revised: by the assumptions on the ground truth

Original: [Equation]
Revised: [Equation]

Original: + λ I n − λ I n ) = λ ( K + λ I ) − 1 . Now, y =
Revised: we can write y =

Original: ϵ + Ψ ≤ m a .
Revised: ϵ + Ψ ≤ m a .

Original: And we are interested in the expectation with regard to noise, hence:
Revised: Thus, the expected training error with respect to the noise is

Original: [Equation]
Revised: [Equation]

Original: While T 1 is  already  the desired term
Revised: First, T 2 > 0 since H is positive semi-definite. Therefore, T 1 already yields the desired lower bound on the expected training error. For the upper bound

Original: [Equation]
Revised: [Equation]

Original: for T 2 we do the following:
Revised: we bound T 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: where in ( i ) we use Lemma 6.
Revised: where H >m := K >m + λ I n .

Original: In ( ii ) we use Equation (5) as well as thefact that we recognize term B 1 from the proof of Theorem 3. In ( iii ) we then use thesame bound on B 1 achieved in Theorem 3.
Revised: Step ( i ) follows from Lemma 6, step ( ii ) uses Equation (5) and matches the term B 1 from the proof of Theorem 3, and step ( iii ) applies the bound on B 1 achieved in Theorem 3.

Original: This  concludes the proof.
Revised: This upper-bounds the expected training error and thereby concludes the proof.

Original: E E XPERIMENTAL DETAILS
Revised: E E XPERIMENTAL DETAILS

Original: This section describes our experimental setup and provides additional details.
Revised: This section describes our experimental setup and includes additional details.

Original: Our submission alreadycontains the coderequired to replicate all experiments and plots. We will release a documented andcleaned version of the code with the camera-ready version of the paper.
Revised: We provide the code to replicate all experiments and plots in https://github.com/michaelaerni/ iclr23-InductiveBiasesHarmlessInterpolation .

Original: E. S ETUP FOR FILTER SIZE EXPERIMENTS
Revised: E. S ETUP FOR FILTER SIZE EXPERIMENTS

Original: The following describes the main filter size experiments presented in Section 4.1.
Revised: The following describes the main filter size experiments presented in Section 4.1.

Original: Network architecture We use a special CNN architecture that amplifies the role of filter size asan inductive bias.
Revised: Network architecture We use a special CNN architecture that amplifies the role of filter size as an inductive bias.

Original: Each model of the main filter size experiments in Figure 2 has the followingarchitecture:
Revised: Each model of the main filter size experiments in Figure 2 has the following architecture:

Original: Convolutional layer with 128 filters of varying size and no padding2.
Revised: Convolutional layer with 128 filters of varying size and no padding 2.

Original: Global max pooling over the spatial feature dimensions3.
Revised: Global max pooling over the spatial feature dimensions 3.

Original: ReLU activation4.
Revised: ReLU activation 4.

Original: Linear layer with 256 output features5.
Revised: Linear layer with 256 output features 5.

Original: ReLU activation 
Revised: ReLU activation 6.

Original: Linear layer with 1 output feature
Revised: Linear layer with 1 output feature

Original: All convolutional and linear layers use a bias term.
Revised: All convolutional and linear layers use a bias term.

Original: Since we employ a single convolutional layerbefore global max pooling, the convolutional filter size directly determines the maximum size of aninput patch that can influence the CNN’s output.
Revised: Since we employ a single convolutional layer before global max pooling, the convolutional filter size directly determines the maximum size of an input patch that can influence the CNN’s output.

Original: Note that this architecture reduces to an MLP if thefilter size equals the input image size.
Revised: Note that this architecture reduces to an MLP if the filter size equals the input image size.

Original: Optimization procedure We use the same training procedure for all settings in Figure 2.
Revised: Optimization procedure We use the same training procedure for all settings in Figure 2.

Original: Optimization minimizes the logistic loss for 300 epochs of mini-batch SGD with momentum 0 .
Revised: Optimization minimizes the logistic loss for 300 epochs of mini-batch SGD with momentum 0 .

Original: 9 and batchsize 100 .
Revised: 9 and batch size 100 .

Original: We linearly increase the learning rate from 10 − 6 to a peak value of 0 . 2 during the first 50epochs, and then reduce the learning rate according to an inverse square-root decay every 20 epochs.
Revised: We linearly increase the learning rate from 10 − 6 to a peak value of 0 . 2 during the first [Figure] epochs, and then reduce the learning rate according to an inverse square-root decay every 20 epochs.

Original: For peak learning rate γ 0 , a decay rate L , the inverse square-root decay schedule at epoch t ≥ 0 is 
Revised: For peak learning rate γ 0 , a decay rate L , the inverse square-root decay schedule at epoch t ≥ 0 is [Equation]

Original: [Equation]
Revised: [Equation]

Original: Learning rate warm-up helps to capture the early-stopped test error more precisely.
Revised: Learning rate warm-up helps to capture the early-stopped test error more precisely.

Original: Wheneverpossible, we use deterministic training algorithms, so that our results are as reproducible as possible.
Revised: Whenever possible, we use deterministic training algorithms, so that our results are as reproducible as possible.

Original: We selected all hyperparameters to minimize the training loss of the strongest inductive bias (filtersize 5 ) on noisy training data, with the constraint that all other settings still converge and interpolate.
Revised: We selected all hyperparameters to minimize the training loss of the strongest inductive bias (filter size 5 ) on noisy training data, with the constraint that all other settings still converge and interpolate.

Original: Note that we do not use data augmentation, dropout, or weight decay.
Revised: Note that we do not use data augmentation, dropout, or weight decay.

Original: Evaluation We observed that all models achieved their minimum test error either at the beginningor very end of training.
Revised: Evaluation We observed that all models achieved their minimum test error either at the beginning or very end of training.

Original: Hence, our experiments evaluate the test error every 2 epochs during thefirst 150 epochs, and every 10 epochs afterwards to save computation time.
Revised: Hence, our experiments evaluate the test error every 2 epochs during the first 150 epochs, and every 10 epochs afterwards to save computation time.

Original: We use an oracle, thatis, the true test error, to determine the optimal early stopping epoch in retrospective.
Revised: We use an oracle, that is, the true test error, to determine the optimal early stopping epoch in retrospective.

Original: The optimalearly stopping training error is always over the entire training set (including potential noise) for afixed model, not an average over mini-batches.
Revised: The optimal early stopping training error is always over the entire training set (including potential noise) for a fixed model, not an average over mini-batches.

Original: To mitigate randomness in both the training data andoptimization procedure, we average over multiple dataset and training seeds.
Revised: To mitigate randomness in both the training data and optimization procedure, we average over multiple dataset and training seeds.

Original: More precisely, wesample 5 different pairs of training and test datasets.
Revised: More precisely, we sample 5 different pairs of training and test datasets.

Original: For each dataset, we fit 15 randomly initializedmodels per filter size on the same dataset, and calculate average metrics.
Revised: For each dataset, we fit 15 randomly initialized models per filter size on the same dataset, and calculate average metrics.

Original: The plots then display themean and standard error over the 5 datasets.
Revised: The plots then display the mean and standard error over the 5 datasets.

Original: Dataset All filter size experiments use synthetic images.
Revised: Dataset All filter size experiments use synthetic images.

Original: For a fixed seed, the experiments generate200 training and 100 k test images, both having an equal amount of positive and negative classes.
Revised: For a fixed seed, the experiments generate 200 training and 100 k test images, both having an equal amount of positive and negative classes.

Original: Given a class, the sampling procedure iteratively scatters 10 shapes on an initially black 32 × 32image.
Revised: Given a class, the sampling procedure iteratively scatters 10 shapes on a black 32 × 32 image.

Original: A single shape is either a circle (negative class) or a cross (positive class), has a uniformlyrandom size in [3 , 5] , and a uniformly random center such that all shapes end up completely insidethe target image.
Revised: A single shape is either a circle (negative class) or a cross (positive class), has a uniformly random size in [3 , 5] , and a uniformly random center such that all shapes end up completely inside the target image.

Original: We use a conceptual line width of 0 .
Revised: We use a conceptual line width of 0 .

Original: 5 pixels, but discretize the shapes into a grid.
Revised: 5 pixels, but discretize the shapes into a grid.

Original: See Figure 4 for examples.
Revised: See Figurefor examples.

Original: A single dataset seed fully determines the training data, test data, and allscattered shapes.
Revised: A single dataset seed fully determines the training data, test data, and all scattered shapes.

Original: Noise model
Revised: Noise model

Original: In the noisy case, we select 20% of all training samples uniformly at random withoutreplacement, and flip their label.
Revised: In the noisy case, we select 20% of all training samples uniformly at random without replacement, and flip their label.

Original: The noise is deterministic per dataset seed and does not changebetween different optimization runs.
Revised: The noise is deterministic per dataset seed and does not change between different optimization runs.

Original: Note that we never apply noise to the test data.
Revised: Note that we never apply noise to the test data.

Original: [Figure]

Original: E. S ETUP FOR ROTATIONAL INVARIANCE EXPERIMENTS
Revised: E. S ETUP FOR ROTATIONAL INVARIANCE EXPERIMENTS

Original: The following describes the rotational invariance experiments presented in Section 4.2.
Revised: The following describes the rotational invariance experiments presented in Section 4.2.

Original: Dataset We use the EuroSAT (Helber et al., 2018) training split and subsample it into 7680 rawtraining and 10 k raw test samples in a stratified way.
Revised: Dataset We use the EuroSAT (Helber et al., 2018) training split and subsample it into 7680 raw training and 10 k raw test samples in a stratified way.

Original: For a fixed number of rotations k , we generate atraining dataset as follows:
Revised: For a fixed number of rotations k , we generate a training dataset as follows:

Original: In the noisy case, select a random 20% subset of all training samples without replacement; for each, change the label to one of the other 9 classes uniformly at random.
Revised: In the noisy case, select a random 20% subset of all training samples without replacement; for each, change the label to one of the other 9 classes uniformly at random.

Original: For the i -th training sample ( i ∈ { 1 , . . . , 7680 } ): (a) Determine a random offset angle α i .
Revised: For the i -th training sample ( i ∈ { 1 , . . . , 7680 } ): (a) Determine a random offset angle α i .

Original: (b) Rotate the original image by each angle in { α i + j · (360 ◦ /k )
Revised: (b) Rotate the original image by each angle in { α i + j · (360 ◦ /k )

Original: | j = 0 , . .
Revised: | j = 0 , . .

Original: , k − 1 } .(c)
Revised: , k − 1 } .

Original:  Crop each of the k rotated 64 × 64 images to 44 × 44 so that no network sees black borders from image interpolation.
Revised: (c) Crop each of the k rotated 64 × 64 images to 44 × 44 so that no network sees black borders from image interpolation.

Original: Concatenate all k × 7680 samples into a single training dataset. 
Revised: Concatenate all k × 7680 samples into a single training dataset. 4.

Original: Shuffle this final dataset (at the beginning of training and every epoch).
Revised: Shuffle this final dataset (at the beginning of training and every epoch).

Original: To generate the actual test dataset, we apply a random rotation to each raw test sample independently,and crop the rotated images to the same size as the training samples.
Revised: To generate the actual test dataset, we apply a random rotation to each raw test sample independently, and crop the rotated images to the same size as the training samples.

Original: This procedure rotates everyimage exactly once, and uses random angle offsets to avoid distribution shift effects from imageinterpolation.
Revised: This procedure rotates every image exactly once, and uses random angle offsets to avoid distribution shift effects from image interpolation.

Original: Note that all random rotations are independent of the label noise and the number oftraining rotations.
Revised: Note that all random rotations are independent of the label noise and the number of training rotations.

Original: Hence, all experiments share the same test dataset.
Revised: Hence, all experiments share the same test dataset.

Original: Furthermore, since we applylabel noise before rotating images, all rotations of an image consistently share the same label.
Revised: Furthermore, since we apply label noise before rotating images, all rotations of an image consistently share the same label.

Original: Network architecture All experiments use a Wide Residual Network (Zagoruyko & Komodakis,2016) with 16 layers, widen factor 6 , and default PyTorch weight initialization.
Revised: Network architecture All experiments use a Wide Residual Network (Zagoruyko & Komodakis, 2016) with 16 layers, widen factor 6 , and default PyTorch weight initialization.

Original: We chose the widthand depth such that all networks are sufficiently overparameterized while still being manageable interms of computational cost.
Revised: We chose the width and depth such that all networks are sufficiently overparameterized while still being manageable in terms of computational cost.

Original: Optimization procedure We use the same training procedure for all settings in Figure 3.
Revised: Optimization procedure We use the same training procedure for all settings in Figure 3.

Original: Optimization minimizes the softmax cross-entropy loss using mini-batch SGD with momentum 0 .
Revised: Optimization minimizes the softmax cross-entropy loss using mini-batch SGD with momentum 0 .

Original: 9 and batchsize 128 .
Revised: 9 and batch size 128 .

Original: Since the training set size grows in the number of rotations, all experiments fix the numberof gradient updates to 144 k.
Revised: Since the training set size grows in the number of rotations, all experiments fix the number of gradient updates to 144 k.

Original: This corresponds to 200 epochs over a dataset with 12 rotations.
Revised: This corresponds to 200 epochs over a dataset with 12 rotations.

Original: Similarto the filter size experiments, we linearly increase the learning rate from zero to a peak value of 0 . 15during the first 4800 steps, and then reduce the learning rate according to an inverse square-rootdecay (Equation (33))
Revised: Similar to the filter size experiments, we linearly increase the learning rate from zero to a peak value of 0 .during the first 4800 steps, and then reduce the learning rate according to an inverse square-root decay (Equation (46))

Original: every 960 steps.
Revised: every 960 steps.

Original: Whenever possible, we use deterministic training algorithms,so that our results are as reproducible as possible.
Revised: Whenever possible, we use deterministic training algorithms, so that our results are as reproducible as possible.

Original: We selected all hyperparameters to minimize thetraining loss of the strongest inductive bias ( 12 rotations) on noisy training data, with the constraintthat all other settings still converge and interpolate.
Revised: We selected all hyperparameters to minimize the training loss of the strongest inductive bias ( 12 rotations) on noisy training data, with the constraint that all other settings still converge and interpolate.

Original: As for all experiments in this paper, we do notuse additional data augmentation, dropout, or weight decay.
Revised: As for all experiments in this paper, we do not use additional data augmentation, dropout, or weight decay.

Original: Evaluation Similar to the filter size experiments, we evaluate the test error more frequently duringearly training iterations: every 480 steps for the first 9600 steps, every 1920 steps afterwards.
Revised: Evaluation Similar to the filter size experiments, we evaluate the test error more frequently during early training iterations: every 480 steps for the first 9600 steps, every 1920 steps afterwards.

Original: Theexperiments again use the actual test error to determine the best step for early-stopping, and calculatethe corresponding training error over the entire training dataset, including all rotations and potentialnoise.
Revised: The experiments again use the actual test error to determine the best step for early-stopping, and calculate the corresponding training error over the entire training dataset, including all rotations and potential noise.

Original: Due to the larger training set size and increased computational costs, we only sample a singletraining and test dataset, and report the mean and standard error of all metrics over five training seeds.
Revised: Due to the larger training set size and increased computational costs, we only sample a single training and test dataset, and report the mean and standard error of all metrics over five training seeds.

Original: E. D IFFERENCE TO DOUBLE DESCENT
Revised: E. D IFFERENCE TO DOUBLE DESCENT

Original: As mentioned in Section 4.1, our empirical observations resemble the double descent phenomenon.
Revised: As mentioned in Section 4.1, our empirical observations resemble the double descent phenomenon.

Original: This subsection expands on the discussion and provides additional details about how this paper’sphenomenon differs from double descent.
Revised: This subsection expands on the discussion and provides additional details about how this paper’s phenomenon differs from double descent.

Original: While all models in all experiments interpolate the training data, we observe that both noisy labelsand stronger inductive biases increase the final training loss of an interpolating model: Smaller filtersize results in a decreasing number of model parameters.
Revised: While all models in all experiments interpolate the training data, we observe that both noisy labels and stronger inductive biases increase the final training loss of an interpolating model: Smaller filter size results in a decreasing number of model parameters.

Original: Enforcing invariance to more rotationsrequires a model to interpolate more (correlated) training samples.
Revised: Enforcing invariance to more rotations requires a model to interpolate more (correlated) training samples.

Original: Thus, in both cases, increasinginductive bias strength decreases a model’s overparameterization in relation to the number of trainingsamples — shifting the setting closer to the interpolation threshold.
Revised: Thus, in both cases, increasing inductive bias strength decreases a model’s overparameterization in relation to the number of training samples — shifting the setting closer to the interpolation threshold.

Original: We argue that our choice of architecture and hyperparameter tuning ensures that no model in anyexperiment is close to the corresponding interpolation threshold.
Revised: We argue that our choice of architecture and hyperparameter tuning ensures that no model in any experiment is close to the corresponding interpolation threshold.

Original: If that is the case, then doubledescent predicts that increasing the number of model parameters has a negligible effect on whetherregularization benefits generalization, and does therefore not explain our observations. 
Revised: If that is the case, then double descent predicts that increasing the number of model parameters has a negligible effect on whether regularization benefits generalization, and does therefore not explain our observations. [Figure] t r a i n i n g l o

Original:  In the following, we first describe how our hyperparameter and model selection procedure ensuresthat all models in all experiments are sufficiently overparameterized, so that double descent predictsnegligible effects from increasing the number of parameters.
Revised: [Figure] In the following, we first describe how our hyperparameter and model selection procedure ensures that all models in all experiments are sufficiently overparameterized, so that double descent predicts negligible effects from increasing the number of parameters.

Original: Then, we provide additional experimental evidence that supports our argument: We repeat a subset of the experiments in Section 4 whileupscaling the number of parameters in all models.
Revised: Then, we provide additional experimental evidence that supports our argument: We repeat a subset of the experiments in Section 4 while upscaling the number of parameters in all models.

Original: For a fixed model scale and varying inductive bias,we observe that all phenomena in Section 4 persist.
Revised: For a fixed model scale and varying inductive bias, we observe that all phenomena in Section 4 persist.

Original: For a fixed inductive bias strength, we furthersee that the test error of interpolating models saturates at a value that matches our hypothesis.
Revised: For a fixed inductive bias strength, we further see that the test error of interpolating models saturates at a value that matches our hypothesis.

Original: Inparticular, for strong inductive biases, the gap in test error between interpolating models and theiroptimally early-stopped version — harmful interpolation — persists.
Revised: In particular, for strong inductive biases, the gap in test error between interpolating models and their optimally early-stopped version — harmful interpolation — persists.

Original: Hyperparameter tuning We mitigate differences in model complexity for different  strengthsofinductive biases by tuning all hyperparameters on worst-case settings, that is, maximum inductive biaswith noisy training samples.
Revised: Hyperparameter tuning We mitigate differences in model complexity for different inductive bias strengths by tuning all hyperparameters on worst-case settings, that is, maximum inductive bias with noisy training samples.

Original: To avoid optimizing on test data, we tune on dataset seeds and networkinitializations that differ from the ones used in actual experiments.
Revised: To avoid optimizing on test data, we tune on dataset seeds and network initializations that differ from the ones used in actual experiments.

Original: Figure 5 displays the final trainingloss for all empirical settings in this paper.
Revised: Figure 5 displays the final training loss for all empirical settings in this paper.

Original: While models with  stronger inductive biases exhibit largertraining losses, all values are close to zero, and the numerical difference is small.
Revised: While models with a stronger inductive bias exhibit larger training losses, all values are close to zero, and the numerical difference is small.

Original: Finally, we want tostress again that this discussion is only about the training loss ; all models in all experiments havezero training error and perfectly fit the corresponding training data.
Revised: Finally, we want to stress again that this discussion is only about the training loss ; all models in all experiments have zero training error and perfectly fit the corresponding training data.

Original: Increasing model complexity for varying filter size As additional evidence, we repeat the mainfilter size experiments from Figure 2 in Section 4.1 using the same setup as before (see Appendix E.1),but increase the convolutional layer width to 256 , 512 , 1024 , and 2048 .
Revised: Increasing model complexity for varying filter size As additional evidence, we repeat the main filter size experiments from Figure 2 in Section 4.1 using the same setup as before (see Appendix E.1), but increase the convolutional layer width to 256 , 512 , 1024 , and 2048 .

Original: For computational reasons,we evaluate a reduced number of filter sizes for widths 256 and 512 , and only the smallest filter size5 for widths 1024 and 2048 .
Revised: For computational reasons, we evaluate a reduced number of filter sizes for widths 256 and 512 , and only the smallest filter size 5 for widths 1024 and 2048 .

Original: Since we found the original learning rate 0 .
Revised: Since we found the original learning rate 0 .

Original: 2 to be too unstable for thelarger model sizes, we use a decreased peak learning rate 0 .
Revised: 2 to be too unstable for the larger model sizes, we use a decreased peak learning rate 0 .

Original: 13 for widths 256 and 512 , and 0 .
Revised: 13 for widths 256 and 512 , and 0 .

Original: 1 forwidths 1024 and 2048 .
Revised: 1 for widths 1024 and 2048 .

Original: Figures 6a and 6b show the test errors for 20% and 0% training noise, respectively.
Revised: Figures 6a and 6b show the test errors for 20% and 0% training noise, respectively.

Original: With noisytraining data (Figure 6a), larger interpolating models yield a slightly smaller test error, but the overalltrends remain: the gap in test error between converged and optimally early-stopped models increaseswith inductive bias strength, and the phase transition between harmless and harmful interpolationpersists.
Revised: With noisy training data (Figure 6a), larger interpolating models yield a slightly smaller test error, but the overall trends remain: the gap in test error between converged and optimally early-stopped models increases with inductive bias strength, and the phase transition between harmless and harmful interpolation persists.

Original: In particular, Figure 6a shows strong evidence that the number of model parameters doesnot influence our phenomenon: for example, models with filter size 5 (strong inductive bias) [-andwidth 
 (red line)-] have more parameters than models with filter size 27 (weak inductive bias) [-andwidth 
 (blue line).-]
Revised: In particular, Figure 6a shows strong evidence that the number of model parameters does not influence our phenomenon: for example, models with filter size 5 (strong inductive bias) [-andwidth 
 (red line)-]and width 512 (red) have more parameters than models with filter size 27 (weak inductive bias) [-andwidth 
 (blue line).-]and width 128 (blue).

Original: Nevertheless, models with filter size 5 benefit significantly from early stopping, 
Revised: Nevertheless, models with filter size 5 benefit significantly from early stopping, while interpolation for models with filter size 27 is harmless. In the noiseless case (Figure 6b), interpolating, width: optimally early-stopped, width:    noisy: clean:      23ﬁlter size 0% [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original:  t r a i n i n g l o 
Revised: % t e s t e rr o r

Original: [Figure]
Revised: [Figure]

Original: interpolating, width
Revised: 23ﬁlter size 0% 6% 10% 25% t e s

Original: [Figure]
Revised: [Figure]

Original: t e s t
Revised: e rr [Figure] [Figure] )  23ﬁlter size

Original: e rr o r
Revised: [Figure] [Figure] [Figure] n

Original: [Figure]
Revised: [Figure]

Original: while interpolation for models with filter size 27 is harmless.
Revised: [Figure] m

Original: In the noiseless case (Figure 6b),increasing model complexity does neither harm nor improve generalization, and all models achievetheir optimal performance after interpolating the entire training dataset.
Revised: [Figure] increasing model complexity does neither harm nor improve generalization, and all models achieve their optimal performance after interpolating the entire training dataset.

Original: Similarly, Figure 6c revealsthat the fraction of training noise that optimally early-stopped models fit stays the same for largermodels.
Revised: Similarly, Figure 6c reveals that the fraction of training noise that optimally early-stopped models fit stays the same for larger models.

Original: Finally, for a fixed inductive bias strength, the test errors saturate as model size increases,making a different trend for models with more than 2048 filters unlikely.
Revised: Finally, for a fixed inductive bias strength, the test errors saturate as model size increases, making a different trend for models with more than 2048 filters unlikely.

Original: To increase legibility, wepresent the numerical results for the largest two filter sizes in Table 1. 
Revised: To increase legibility, we present the numerical results for the largest two filter sizes in Table 1. widthwidth

Original:  Increasing model capacity for varying rotational invariance For completeness, we also repeatthe rotation invariance experiments from Figure 3 in Section 4.2 with twice as wide Wide Residual
Revised: [Table] Increasing model capacity for varying rotational invariance For completeness, we also repeat the rotation invariance experiments from Figure 3 in Section 4.2 with twice as wide Wide Residual

Original: Networks on a reduced number of rotations.
Revised: Networks on a reduced number of rotations.

Original: More precisely, we increase the network widen-factorfrom 6 to 12 , and otherwise use the same setting as the main experiments (see Appendix E.2).
Revised: More precisely, we increase the network widen-factor from 6 to 12 , and otherwise use the same setting as the main experiments (see Appendix E.2).

Original: Notethat this corresponds to a parameter increase from around 6 million to around 24 million parameters.[Table] t e s t
Revised: Note that this corresponds to a parameter increase from around 6 million to around 24 million parameters.

Original: e rr o r [Figure] [Figure] [Figure]
Revised: The results in Figure 7 provide additional evidence that our phenomenon is distinct from double descent: both the test error (Figures 7a and 7b) and fraction of fitted noise under optimal early stopping (Figure 7c) exhibit the same trend, despite the significant difference in number of parameters.

Original: [Table]
Revised: [Figure]

Original: t e s t
Revised: t e s t

Original: e rr o r
Revised: e rr o r

Original: [Table]

Original: [Table]

Original: s u b s e t t

Original: r a i n e rr o r

Original: ( o p t i m a l e a r l y s t

Original: o

Original: pp i n g ) (c) Opt

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: The results in Figure 7 provide additional evidence that our phenomenon is distinct from doubledescent: both the test error (Figures 7a and 7b) and fraction of fitted noise under optimal early stopping(Figure 7c) exhibit the same trend, despite the significant difference in number of parameters.