Original: M INI - BATCH k - MEANS TERMINATES WITHIN O ( d/ϵ ) ITERATIONS
Revised: M INI - BATCH k - MEANS TERMINATES WITHIN O ( d/ϵ ) ITERATIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We answer the question: "Does local progress (on batches) imply global progress (on the entire dataset) for mini-batch k -means?".
Revised: We answer the question: "Does local progress (on batches) imply global progress (on the entire dataset) for mini-batch k -means?".

Original: Specifically, we consider minibatch k -means which terminates only when the improvement in the quality of the clustering on the sampled batch is below some threshold.
Revised: Specifically, we consider minibatch k -means which terminates only when the improvement in the quality of the clustering on the sampled batch is below some threshold.

Original: Although at first glance it appears that this algorithm might execute forever, we answer the above question in the affirmative and show that if the batch is of size ˜Ω(( d/ϵ ) 2 ) , it must terminate within O ( d/ϵ ) iterations with high probability, where d is the dimension of the input, and ϵ is a threshold parameter for termination.
Revised: Although at first glance it appears that this algorithm might execute forever, we answer the above question in the affirmative and show that if the batch is of size ˜Ω(( d/ϵ ) 2 ) , it must terminate within O ( d/ϵ ) iterations with high probability, where d is the dimension of the input, and ϵ is a threshold parameter for termination.

Original: This is true regardless of how the centers are initialized.
Revised: This is true regardless of how the centers are initialized.

Revised: When the algorithm is initialized with the k -means++ initialization scheme, it achieves an approximation ratio of O (log k ) (the same as the full-batch version).

Original: Finally, we show the applicability of our results to the mini-batch k -means algorithm implemented in the scikit-learn (sklearn) python library.
Revised: Finally, we show the applicability of our results to the mini-batch k -means algorithm implemented in the scikit-learn (sklearn) python library.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The mini-batch k -means algorithm
Revised: The mini-batch k -means algorithm

Original: (Sculley, 2010) is one of the most popular clustering algorithms used in practice (Pedregosa et al., 2011).
Revised: (Sculley, 2010) is one of the most popular clustering algorithms used in practice (Pedregosa et al., 2011).

Original: However, due to its stochastic nature, it appears that if we do not explicitly bound the number of iterations of the algorithm, then it might never terminate.
Revised: However, due to its stochastic nature, it appears that if we do not explicitly bound the number of iterations of the algorithm, then it might never terminate.

Original: We show that, when the batch size is sufficiently large, using only an "early-stopping" condition, which terminates the algorithm when the local progress observed on a batch is below some threshold, we can guarantee a bound on the number of iterations that the algorithm performs which is independent of input size.
Revised: We show that, when the batch size is sufficiently large, using only an "early-stopping" condition, which terminates the algorithm when the local progress observed on a batch is below some threshold, we can guarantee a bound on the number of iterations that the algorithm performs which is independent of input size.

Original: Problem statement We consider the following optimization problem.
Revised: Problem statement We consider the following optimization problem.

Original: We are given an input (dataset), X = { x i } ni =1 ⊆
Revised: We are given an input (dataset), X = { x i } ni =1 ⊆

Original: [0 , 1] d , of size n of d -dimensional real vectors and a parameter k .
Revised: [0 , 1] d , of size n of d -dimensional real vectors and a parameter k .

Original: Note that the assumption that X ⊆ [0 , 1] d is standard in the literature (Arthur et al., 2011), and is meant to simplify notation (otherwise we would have to introduce a new parameter for the diameter of X ).
Revised: Note that the assumption that X ⊆ [0 , 1] d is standard in the literature (Arthur et al., 2011), and is meant to simplify notation (otherwise we would have to introduce a new parameter for the diameter of X ).

Original: Our goal is to find a set C of k centers (vectors in [0 , 1] d ) such that the following goal function is minimized:
Revised: Our goal is to find a set C of k centers (vectors in [0 , 1] d ) such that the following goal function is minimized:

Original: [Equation]
Revised: [Equation]

Original: Usually, the 1 /n factor does not appear as it does not affect the optimization goal, however, in our case, it will be useful to define it as such.
Revised: Usually, the 1 /n factor does not appear as it does not affect the optimization goal, however, in our case, it will be useful to define it as such.

Original: In this paper we do not deal with the quality of the solution

Original: C , only with the time it takes an algorithm that aims to solve the above problem to terminate.

Original: Lloyd’s algorithm
Revised: Lloyd’s algorithm

Original: The most popular method to solve the above problem is Lloyd’s algorithm (often referred to as the k -means algorithm)
Revised: The most popular method to solve the above problem is Lloyd’s algorithm (often referred to as the k -means algorithm)

Original: (Lloyd, 1982).
Revised: (Lloyd, 1982).

Original: It works by randomly initializing a set of k centers and performing the following two steps: (1) Assign every point in X to the center closest to it.
Revised: It works by randomly initializing a set of k centers and performing the following two steps: (1) Assign every point in X to the center closest to it.

Original: (2) Update every center to be the mean of the points assigned to it.
Revised: (2) Update every center to be the mean of the points assigned to it.

Original: The algorithm terminates when no point is reassigned to a new center.
Revised: The algorithm terminates when no point is reassigned to a new center.

Original: This algorithm is extremely fast in practice but has a worst-case exponential running time (Arthur & Vassilvitskii, 2006; Vattani, 2011).
Revised: This algorithm is extremely fast in practice but has a worst-case exponential running time (Arthur & Vassilvitskii, 2006; Vattani, 2011).

Original: Mini-batch k -means To update the centers, Lloyd’s algorithm must go over the entire input at every iteration.
Revised: Mini-batch k -means To update the centers, Lloyd’s algorithm must go over the entire input at every iteration.

Original: This can be computationally expensive when the input data is extremely large.
Revised: This can be computationally expensive when the input data is extremely large.

Original: To tackle this, the mini-batch k -means method was introduced by Sculley (2010).
Revised: To tackle this, the mini-batch k -means method was introduced by Sculley (2010).

Original: It is similar to Lloyd’s algorithm except that steps (1) and (2) are performed on a batch of b elements sampled uniformly at random with repetitions, and in step (2) the centers are updated slightly differently.
Revised: It is similar to Lloyd’s algorithm except that steps (1) and (2) are performed on a batch of b elements sampled uniformly at random with repetitions, and in step (2) the centers are updated slightly differently.

Original: Specifically, every center is updated to be the weighted average of its current value and the mean of the points (in the batch) assigned to it.
Revised: Specifically, every center is updated to be the weighted average of its current value and the mean of the points (in the batch) assigned to it.

Original: The parameter by which we weigh these values is called the learning rate , and its value differs between centers and iterations.
Revised: The parameter by which we weigh these values is called the learning rate , and its value differs between centers and iterations.

Original: In the original paper by Sculley, there is no stopping condition similar to that of Lloyd’s algorithm, instead, the algorithm is simply executed for t iterations, where t is an input parameter.
Revised: In the original paper by Sculley, there is no stopping condition similar to that of Lloyd’s algorithm, instead, the algorithm is simply executed for t iterations, where t is an input parameter.

Original: In practice (for example in sklearn (Pedregosa et al., 2011)), together with an upper bound on the number of iterations to perform there are several "early stopping" conditions.
Revised: In practice (for example in sklearn (Pedregosa et al., 2011)), together with an upper bound on the number of iterations to perform there are several "early stopping" conditions.

Original: We may terminate the algorithm when the change in the locations of the centers is sufficiently small or when the change in the goal function for several consecutive batches does not improve.
Revised: We may terminate the algorithm when the change in the locations of the centers is sufficiently small or when the change in the goal function for several consecutive batches does not improve.

Original: We note that in both theory (Tang & Monteleoni, 2017; Sculley, 2010) and practice (Pedregosa et al., 2011) the learning rate goes to 0 over time.
Revised: We note that in both theory (Tang & Monteleoni, 2017; Sculley, 2010) and practice (Pedregosa et al., 2011) the learning rate goes to 0 over time.

Original: That is, over time the movement of centers becomes smaller and smaller, which guarantees termination for most reasonable early-stopping conditions at the limit.
Revised: That is, over time the movement of centers becomes smaller and smaller, which guarantees termination for most reasonable early-stopping conditions at the limit.

Original: Our results are the first to show extremely fast termination guarantees for mini-batch k -means with early stopping conditions.
Revised: Our results are the first to show extremely fast termination guarantees for mini-batch k -means with early stopping conditions.

Original: Surprisingly, we need not require the learning rate to go to 0.
Revised: Surprisingly, we need not require the learning rate to go to 0.

Original: Related work Mini-batch k -means was first introduced by Sculley (2010) as a natural generalization to online k -means (Bottou & Bengio, 1994)
Revised: Related work Mini-batch k -means was first introduced by Sculley (2010) as a natural generalization to online k -means (Bottou & Bengio, 1994)

Original: (here the batch is of size 1).
Revised: (here the batch is of size 1).

Original: We are aware only of a single paper that analyzes the convergence rate of mini-batch k -means (Tang & Monteleoni, 2017).
Revised: We are aware only of a single paper that analyzes the convergence rate of mini-batch k -means (Tang & Monteleoni, 2017).

Original: It is claimed in (Tang & Monteleoni, 2017) that under mild assumptions the algorithm has O (1 /t ) convergence rate.
Revised: It is claimed in (Tang & Monteleoni, 2017) that under mild assumptions the algorithm has O (1 /t ) convergence rate.

Original: That is, after t iterations it holds that the current value of the goal function is within an additive O (1 /t ) factor from the value of the goal function in some local optimum of Lloyd’s algorithm.
Revised: That is, after t iterations it holds that the current value of the goal function is within an additive O (1 /t ) factor from the value of the goal function in some local optimum of Lloyd’s algorithm.

Original: However, their asymptotic notation subsumes factors that depend on the size of the input.
Revised: However, their asymptotic notation subsumes factors that depend on the size of the input.

Original: Taking this into account, we get a convergence rate of Ω( n 2 /t ) , which implies, at best, a quadratic bound on the execution time of the algorithm.
Revised: Taking this into account, we get a convergence rate of Ω( n 2 /t ) , which implies, at best, a quadratic bound on the execution time of the algorithm.

Original: This is due to setting the learning rate at iteration t to O (1 / ( n 2 + t )) .
Revised: This is due to setting the learning rate at iteration t to O (1 / ( n 2 + t )) .

Original: Our results do not guarantee convergence to any local-minima, however, they guarantee an exponentially faster runtime bound.
Revised: Our results do not guarantee convergence to any local-minima, however, they guarantee an exponentially faster runtime bound.

Original: Our results We analyze the mini-batch k -means algorithm described above (Sculley, 2010), where the algorithm terminates only when the improvement in the quality of the clustering for the sampled batch is less than some threshold parameter ϵ .
Revised: Our results We analyze the mini-batch k -means algorithm described above (Sculley, 2010), where the algorithm terminates only when the improvement in the quality of the clustering for the sampled batch is less than some threshold parameter ϵ .

Original: That is, we terminate if for some batch the difference in the quality of the clustering before the update and after the update is less than ϵ .
Revised: That is, we terminate if for some batch the difference in the quality of the clustering before the update and after the update is less than ϵ .

Original: Our stopping condition is slightly different than what is used in practice.
Revised: Our stopping condition is slightly different than what is used in practice.

Original: In sklearn termination is determined based on the changes in cluster centers.
Revised: In sklearn termination is determined based on the changes in cluster centers.

Original: In Section 5 we prove that this condition also fits within our framework.
Revised: In Section 5 we prove that this condition also fits within our framework.

Original: Our main goal is to answer the following theoretical question: "Does local progress (on batches) imply global progress (on the entire dataset) for mini-batch k -means, even when the learning rate does not go to 0?".
Revised: Our main goal is to answer the following theoretical question: "Does local progress (on batches) imply global progress (on the entire dataset) for mini-batch k -means, even when the learning rate does not go to 0?".

Original: Intuitively, it is clear that the answer depends on the batch size used by the algorithm.
Revised: Intuitively, it is clear that the answer depends on the batch size used by the algorithm.

Original: If the batch is the entire dataset the claim is trivial and results in a termination guarantee of O ( d/ϵ ) iterations 1 .
Revised: If the batch is the entire dataset the claim is trivial and results in a termination guarantee of O ( d/ϵ ) iterations 1 .

Original: We show that when the batch size exceeds a certain threshold, indeed local progress implies global progress and we achieve the same asymptotic bound on the number of iterations as when the batch is the entire dataset.
Revised: We show that when the batch size exceeds a certain threshold, indeed local progress implies global progress and we achieve the same asymptotic bound on the number of iterations as when the batch is the entire dataset.

Original: We present several results:
Revised: We present several results:

Original: We start with a warm-up in Section 3, showing that when b = ˜Ω( kd 3 ϵ − 2 ) we can guarantee termination within O ( d/ϵ ) iterations 2 w.h.p (with high probability) 3 .
Revised: We start with a warm-up in Section 3, showing that when b = ˜Ω( kd 3 ϵ − 2 ) we can guarantee termination within O ( d/ϵ ) iterations 2 w.h.p (with high probability) 3 .

Original: We require the additional assumption that every real number in the system can be represented using O (1) bits (e.g., 64-bit floats).
Revised: We require the additional assumption that every real number in the system can be represented using O (1) bits (e.g., 64-bit floats).

Original: The above bound holds regardless of how cluster centers are initialized or updated.
Revised: The above bound holds regardless of how cluster centers are initialized or updated.

Original: That is, this bound holds for any center update rule, and not only for the "standard" center update rule described above.
Revised: That is, this bound holds for any center update rule, and not only for the "standard" center update rule described above.

Original: Our proof uses elementary tools and is presented to set the stage for our main result.
Revised: Our proof uses elementary tools and is presented to set the stage for our main result.

Original: In Section 4 we show that using the standard update rule, we can achieve the same termination time with a much smaller batch size.
Revised: In Section 4 we show that using the standard update rule, we can achieve the same termination time with a much smaller batch size.

Original: Specifically, a batch size of Ω(( d/ϵ ) 2 log( nkd/ϵ )) =
Revised: Specifically, a batch size of Ω(( d/ϵ ) 2 log( nkd/ϵ )) =

Original: ˜Ω(( d/ϵ ) 2 ) is sufficient to guarantee termination within O ( d/ϵ ) iterations.
Revised: ˜Ω(( d/ϵ ) 2 ) is sufficient to guarantee termination within O ( d/ϵ ) iterations.

Original: This holds regardless of how centers are initialized and does not require any assumption on the number of bits required to represent real numbers.
Revised: This holds regardless of how centers are initialized and does not require any assumption on the number of bits required to represent real numbers.

Original: Our proof makes use of the fact that the standard update rule adds additional stability to the stochastic process when the learning rate is sufficiently small (but need not go to 0).
Revised: Our proof makes use of the fact that the standard update rule adds additional stability to the stochastic process when the learning rate is sufficiently small (but need not go to 0).

Original: Finally, in Section 5, we show that our main result also holds for the early stopping condition used in sklearn (with our learning rate).
Revised: Finally, in Section 5, we show that our main result also holds for the early stopping condition used in sklearn (with our learning rate).

Original: However, this results in a larger batch size and slower termination.
Revised: However, this results in a larger batch size and slower termination.

Original: Specifically if b = ˜Ω(( d/ϵ ) 3 k ) we terminate within O (( d/ϵ ) 1 . 5 √ k ) iterations w.h.p.
Revised: Specifically if b = ˜Ω(( d/ϵ ) 3 k ) we terminate within O (( d/ϵ ) 1 . 5 √ k ) iterations w.h.p.

Original: Note that for the batch size to be reasonable, we must require that b ≤ n , which implies that ( d/ϵ ) 2 log( nkd/ϵ ) = O ( n ) .
Revised: Note that for the batch size to be reasonable, we must require that b ≤ n , which implies that ( d/ϵ ) 2 log( nkd/ϵ ) = O ( n ) .

Original: Thus, our results only hold for a certain range of values for k, d, ϵ .
Revised: Thus, our results only hold for a certain range of values for k, d, ϵ .

Original: This is reasonable, as in practice it is often the case
Revised: This is reasonable, as in practice it is often the case

Original: that ϵ =
Revised: that ϵ =

Original: O (1) , d ≪ n and the dependence on the rest of the parameters is logarithmic.
Revised: O (1) , d ≪ n and the dependence on the rest of the parameters is logarithmic.

Revised: Solution quality Applying the k -means++ initialization scheme to our results we achieve the same approximation ratio, O (log k ) in expectation, as the full-batch algorithm.

Revised: The approximation guarantee of k -means++ is guaranteed already in the initialization phase (Theorem 3.1 in Arthur &

Revised: Vassilvitskii (2007)), and the execution of Lloyd’s algorithm following initialization can only improve the solution.

Revised: We show that w.h.p the global goal function is decreasing throughout our execution which implies that the approximation guarantee remains the same.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Throughout this paper we work with ordered tuples rather than sets, denoted as Y =
Revised: Throughout this paper we work with ordered tuples rather than sets, denoted as Y =

Original: ( y i ) i ∈
Revised: ( y i ) i ∈

Original: [ ℓ ] , where [ ℓ ] = { 1 , . . . , ℓ } .
Revised: [ ℓ ] , where [ ℓ ] = { 1 , . . . , ℓ } .

Original: To reference the i -th element we either write y i or Y [ i ] .
Revised: To reference the i -th element we either write y i or Y [ i ] .

Original: It will be useful to use set notations for tuples such as x ∈ Y ⇐⇒ ∃ i ∈ [ ℓ ] , x = y i and Y ⊆
Revised: It will be useful to use set notations for tuples such as x ∈ Y ⇐⇒ ∃ i ∈ [ ℓ ] , x = y i and Y ⊆

Original: Z ⇐⇒ ∀ i ∈ [ ℓ ] , y i ∈
Revised: Z ⇐⇒ ∀ i ∈ [ ℓ ] , y i ∈

Original: Z .
Revised: Z .

Original: When summing we often write (cid:80) x ∈ Y g ( x ) which is equivalent to (cid:80) ℓi
Revised: When summing we often write (cid:80) x ∈ Y g ( x ) which is equivalent to (cid:80) ℓi

Original: =1 g
Revised: =1 g

Original: ( Y [ i ]) .
Revised: ( Y [ i ]) .

Original: We borrow the following notation from (Kanungo et al., 2004).
Revised: We borrow the following notation from (Kanungo et al., 2004).

Original: For every x, y ∈ R d let ∆( x, y ) =
Revised: For every x, y ∈ R d let ∆( x, y ) =

Original: ∥ x − y ∥ 2 .
Revised: ∥ x − y ∥ 2 .

Original: For every finite tuple S ⊆ R d and a vector x ∈ R d let ∆( S, x ) =
Revised: For every finite tuple S ⊆ R d and a vector x ∈ R d let ∆( S, x ) =

Original: (cid:80)
Revised: (cid:80)

Original: y ∈ S ∆( y, x ) .
Revised: y ∈ S ∆( y, x ) .

Original: k -means We are given an input X
Revised: k -means We are given an input X

Original: =
Revised: =

Original: ( x i ) ni
Revised: ( x i ) ni

Original: =1 ⊆
Revised: =1 ⊆

Original: [0 , 1] d and a parameter k .
Revised: [0 , 1] d and a parameter k .

Original: Our goal is to find a tuple C ⊆ R d of k centers such that the following goal function is minimized:
Revised: Our goal is to find a tuple C ⊆ R d of k centers such that the following goal function is minimized:

Original: [Equation]
Revised: [Equation]

Original: Let us define for every x ∈ X the function f x : R k · d → R where f x ( C ) =
Revised: Let us define for every x ∈ X the function f x : R k · d → R where f x ( C ) =

Original: min C ∈C ∆( x, C ) .
Revised: min C ∈C ∆( x, C ) .

Original: We can treat R k · d as the set of k -tuples of d -dimensional vectors.
Revised: We can treat R k · d as the set of k -tuples of d -dimensional vectors.

Original: We also define the following function for every tuple
Revised: We also define the following function for every tuple

Original: A =
Revised: A =

Original: ( a i ) ℓi =1 ⊆
Revised: ( a i ) ℓ i =1 ⊆

Original: X :
Revised: X :

Original: [Equation]
Revised: [Equation]

Original: Note that f X is our original goal function.
Revised: Note that f X is our original goal function.

Original: We state the following useful lemma:
Revised: We state the following useful lemma:

Original: Lemma 2.1.
Revised: Lemma 2.1.

Original: For any tuple of k centers C ⊂ [0 , 1] d it holds that f X ( C ) ≤ d .
Revised: For any tuple of k centers C ⊂ [0 , 1] d it holds that f X ( C ) ≤ d .

Original: Proof.
Revised: Proof.

Original: Because X, C ⊂ [0 , 1] d it holds that ∀ x ∈ X, f x ( C ) ≤ max C ∈C ∆( x, C ) ≤ d .
Revised: Because X, C ⊂ [0 , 1] d it holds that ∀ x ∈ X, f x ( C ) ≤ max C ∈C ∆( x, C ) ≤ d .

Original: Therefore f X ( C ) =
Revised: Therefore f X ( C ) =

Original: 1 n (cid:80) x ∈ X f x ( C ) ≤ 1 n · nd = d .
Revised: 1 n (cid:80) x ∈ X f x ( C ) ≤ 1 n · nd = d .

Original: We state the following well known theorems:
Revised: We state the following well known theorems:

Original: Theorem 2.2 (Hoeffding (1963)) .
Revised: Theorem 2.2 (Hoeffding (1963)) .

Original: Let Y 1 , ...,
Revised: Let Y 1 , ...,

Original: Y m be independent random variables such that ∀ 1 ≤ i ≤ m, E
Revised: Y m be independent random variables such that ∀ 1 ≤ i ≤ m, E

Original: [ Y i ]
Revised: [ Y i ]

Original: = µ and Y i ∈
Revised: = µ and Y i ∈

Original: [ a min , a max ] .
Revised: [ a min , a max ] .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.3 (Jensen (1906)) .
Revised: Theorem 2.3 (Jensen (1906)) .

Original: Let ϕ be a convex function, y 1 , . . .
Revised: Let ϕ be a convex function, y 1 , . . .

Original: , y n numbers in its domain and weights a 1 , . .
Revised: , y n numbers in its domain and weights a 1 , . .

Original: . , a n ∈ R + .
Revised: . , a n ∈ R + .

Original: It holds that:
Revised: It holds that:

Original: [Equation]
Revised: [Equation]

Original:  W ARM - UP : A SIMPLE BOUND
Revised:  W ARM - UP : A SIMPLE BOUND

Original: Let us first show a simple convergence guarantee which makes no assumptions about how the centers are updated.
Revised: Let us first show a simple convergence guarantee which makes no assumptions about how the centers are updated.

Original: This will set the stage for our main result in Section 4, where we consider the standard update rule used in mini-batch k -means (Sculley, 2010; Pedregosa et al., 2011).
Revised: This will set the stage for our main result in Section 4, where we consider the standard update rule used in mini-batch k -means (Sculley, 2010; Pedregosa et al., 2011).

Original: Algorithm We analyze a generic variant of the mini-batch k -means algorithm, presented in Algorithm 1.
Revised: Algorithm We analyze a generic variant of the mini-batch k -means algorithm, presented in Algorithm 1.

Original: Note that it a very broad class of algorithms (including the widely used algorithm of Sculley (2010)).
Revised: Note that it a very broad class of algorithms (including the widely used algorithm of Sculley (2010)).

Original: The only assumptions we make are:
Revised: The only assumptions we make are:

Original: The centers remain within [0 , 1] d (the convex hull bounding X ).
Revised: The centers remain within [0 , 1] d (the convex hull bounding X ).

Original: Batches are sampled uniformly at random from X with repetitions.
Revised: Batches are sampled uniformly at random from X with repetitions.

Original: The algorithm terminates when updating the centers does not significantly improve the quality of the solution for the sampled batch.
Revised: The algorithm terminates when updating the centers does not significantly improve the quality of the solution for the sampled batch.

Original: Items (1) and (2) are standard both in theory and practice (Sculley, 2010; Pedregosa et al., 2011; Tang & Monteleoni, 2017).
Revised: Items (1) and (2) are standard both in theory and practice (Sculley, 2010; Pedregosa et al., 2011; Tang & Monteleoni, 2017).

Original: Item (3) is usually referred to as an "early-stopping" condition.
Revised: Item (3) is usually referred to as an "early-stopping" condition.

Original: Early stopping conditions are widely used in practice (for example in sklearn (Pedregosa et al., 2011)), together with a bound on the number of iterations.
Revised: Early stopping conditions are widely used in practice (for example in sklearn (Pedregosa et al., 2011)), together with a bound on the number of iterations.

Original: However, our early-stopping condition is slightly different than the one used in practice.
Revised: However, our early-stopping condition is slightly different than the one used in practice.

Original: We discuss this difference in Section 5.
Revised: We discuss this difference in Section 5.

Original: At first glance, guaranteeing termination for any possible way of updating the centers might seem strange.
Revised: At first glance, guaranteeing termination for any possible way of updating the centers might seem strange.

Original: However, if the update procedure is degenerate, it will make no progress, at which point the algorithm terminates.
Revised: However, if the update procedure is degenerate, it will make no progress, at which point the algorithm terminates.

Original:  Termination guarantees for Algorithm 1 To bound the number of iterations of such a generic algorithm we require the following assumption: every real number in our system can be represented using q = O (1) bits.
Revised: Algorithm 1: Generic Termination guarantees for Algorithm 1 To bound the number of iterations of such a generic algorithm we require the following assumption: every real number in our system can be represented using q = O (1) bits.

Original: This implies that every set of k centers can be represented using qkd bits.
Revised: This implies that every set of k centers can be represented using qkd bits.

Original: This means that the total number of possible solutions is bounded by 2 qkd .
Revised: This means that the total number of possible solutions is bounded by 2 qkd .

Original: This will allow us to show that when the batch is sufficiently large, the sampled batch acts as a sparsifier for the entire dataset.
Revised: This will allow us to show that when the batch is sufficiently large, the sampled batch acts as a sparsifier for the entire dataset.

Original: Specifically, it means that for any tuple of k centers, C , it holds that | f B i ( C )
Revised: Specifically, it means that for any tuple of k centers, C , it holds that | f B i ( C )

Original: − f X ( C ) | < ϵ/ 4 .
Revised: − f X ( C ) | < ϵ/ 4 .

Original: This implies that, for a sufficiently large batch size, simply sampling a single batch and executing
Revised: This implies that, for a sufficiently large batch size, simply sampling a single batch and executing

Original: Lloyd’s algorithm on the batch will be sufficient, and executing mini-batch k -means is unnecessary.
Revised: Lloyd’s algorithm on the batch will be sufficient, and executing mini-batch k -means is unnecessary.

Original: Nevertheless, this serves as a good starting point to showcase our general approach and to highlight the challenges we overcome in Section 4 in order to reduce the required batch size without compromising the running time.
Revised: Nevertheless, this serves as a good starting point to showcase our general approach and to highlight the challenges we overcome in Section 4 in order to reduce the required batch size without compromising the running time.

Original: Let us assume that the algorithm executes for at least t iterations.
Revised: Let us assume that the algorithm executes for at least t iterations.

Original: That is, the termination condition does not hold for the first t iterations.
Revised: That is, the termination condition does not hold for the first t iterations.

Original: Our goal is to upper bound t .
Revised: Our goal is to upper bound t .

Original: Parameter range Let us first define the range of parameter values for which the results for this section hold.
Revised: Parameter range Let us first define the range of parameter values for which the results for this section hold.

Original: Recall that n is the size of the input, k is the number of centers, d is the dimension, ϵ is the termination threshold.
Revised: Recall that n is the size of the input, k is the number of centers, d is the dimension, ϵ is the termination threshold.

Original: For the rest of this section assume that b = Ω(( d/ϵ ) 2 ( kd +log( nt ))) .
Revised: For the rest of this section assume that b = Ω(( d/ϵ ) 2 ( kd +log( nt ))) .

Original: Later we show that t =
Revised: Later we show that t =

Original: O ( d/ϵ ) , which will imply that b = ˜Ω( kd 3 ϵ − 2 ) is sufficient for our termination guarantees to hold.
Revised: O ( d/ϵ ) , which will imply that b = ˜Ω( kd 3 ϵ − 2 ) is sufficient for our termination guarantees to hold.

Original: We state the following useful lemma which guarantees that f B ( C ) is not too far from f X ( C ) when the batch size is sufficiently large and C is fixed (i.e., independent of the choice of B i ).
Revised: We state the following useful lemma which guarantees that f B ( C ) is not too far from f X ( C ) when the batch size is sufficiently large and C is fixed (i.e., independent of the choice of B i ).

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Let B be a tuple of b elements chosen uniformly at random from X with repetitions.
Revised: Let B be a tuple of b elements chosen uniformly at random from X with repetitions.

Original: For any fixed tuple of k centers, C ⊆ [0 , 1] d , it holds that: Pr [ | f B ( C ) − f X
Revised: For any fixed tuple of k centers, C ⊆ [0 , 1] d , it holds that: Pr [ | f B ( C ) − f X

Original: ( C ) | ≥ δ ] ≤ 2 e − 2 bδ 2 /d 2 .
Revised: ( C ) | ≥ δ ] ≤ 2 e − 2 bδ 2 /d 2 .

Original: Proof.
Revised: Proof.

Original: Let us write B =
Revised: Let us write B =

Original: ( y 1 , . .
Revised: ( y 1 , . .

Original: , y b ) , where y i is a random element selected uniformly at random from X with repetitions.
Revised: , y b ) , where y i is a random element selected uniformly at random from X with repetitions.

Original: For every such y i define the random variable Z i =
Revised: For every such y i define the random variable Z i =

Original: f
Revised: f

Original: y i
Revised: y i

Original: ( C ) .
Revised: ( C ) .

Original: These new random variables are IID for any fixed C .
Revised: These new random variables are IID for any fixed C .

Original: It also holds that ∀ i ∈
Revised: It also holds that ∀ i ∈

Original: [ b ] , E [ Z i ] = 1 n (cid:80) x ∈ X f x ( C ) =
Revised: [ b ] , E [ Z i ] = 1 n (cid:80) x ∈ X f x ( C ) =

Original: f X ( C ) and that f B ( C ) =
Revised: f X ( C ) and that f B ( C ) =

Original: 1 b (cid:80) x ∈ B f x ( C ) = 1 b (cid:80) bi =1 Z i .
Revised: 1 b (cid:80) x ∈ B f x ( C ) = 1 b (cid:80) bi =1 Z i .

Original: Applying a Hoeffding bound (Theorem 2.2) with parameters m = b, µ = f X ( C ) , a max − a min ≤
Revised: Applying a Hoeffding bound (Theorem 2.2) with parameters m = b, µ = f X ( C ) , a max − a min ≤

Original: d we get that: Pr [ | f B ( C ) − f X ( C )
Revised: d we get that: Pr [ | f B ( C ) − f X ( C )

Original: | ≥ δ ] ≤ 2 e − 2 bδ 2 /d 2 .
Revised: | ≥ δ ] ≤ 2 e − 2 bδ 2 /d 2 .

Original: Using the above we can show that every B i is a sparsifier for X .
Revised: Using the above we can show that every B i is a sparsifier for X .

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: It holds w.h.p that for every i ∈ [ t ] and for every set of k centers, C ⊂ [0 , 1] d , that | f B i ( C )
Revised: It holds w.h.p that for every i ∈ [ t ] and for every set of k centers, C ⊂ [0 , 1] d , that | f B i ( C )

Original: − f X ( C ) | < ϵ/ 4 .
Revised: − f X ( C ) | < ϵ/ 4 .

Original: Proof.
Revised: Proof.

Original: Using Lemma 3.1, setting δ = ϵ/ 4 and using the fact that b = Ω(( d/ϵ ) 2 ( kd + log( nt ))) , we get: Pr [ | f B ( C ) − f X ( C ) | ≥ δ ] ≤ 2 e − 2 bδ 2 /d 2 = 2 − Θ( bδ 2 /d 2 )
Revised: Using Lemma 3.1, setting δ = ϵ/ 4 and using the fact that b = Ω(( d/ϵ ) 2 ( kd + log( nt ))) , we get: Pr [ | f B ( C ) − f X ( C ) | ≥ δ ] ≤ 2 e − 2 bδ 2 /d 2 = 2 − Θ( bδ 2 /d 2 )

Original: = 2 − Ω( kd +log( nt )) .
Revised: = 2 − Ω( kd +log( nt )) .

Original: Taking a union bound over all t iterations and all 2 qkd configurations of centers, we get that the probability is bounded by 2 − Ω( kd +log( nt )) · 2 qkd · t = O (1 /n ) , for an appropriate constant in the asymptotic notation for b .
Revised: Taking a union bound over all t iterations and all 2 qkd configurations of centers, we get that the probability is bounded by 2 − Ω( kd +log( nt )) · 2 qkd · t = O (1 /n ) , for an appropriate constant in the asymptotic notation for b .

Original: The lemma below guarantees global progress for the algorithm.
Revised: The lemma below guarantees global progress for the algorithm.

Original: Lemma 3.3.
Revised: Lemma 3.3.

Original: It holds w.h.p that ∀ i ∈
Revised: It holds w.h.p that ∀ i ∈

Original: [ t ] , f X ( C i ) − f X
Revised: [ t ] , f X ( C i ) − f X

Original: ( C i +1 ) ≥ ϵ/ 2 .
Revised: ( C i +1 ) ≥ ϵ/ 2 .

Original: Proof.
Revised: Proof.

Original: Let us write (the notation ± x means that we add and subtract x ):
Revised: Let us write (the notation ± x means that we add and subtract x ):

Original: [Equation]
Revised: [Equation]

Original: Due to Lemma 3.2 it holds that w.h.p f X ( C i ) − f B
Revised: Due to Lemma 3.2 it holds that w.h.p f X ( C i ) − f B

Original: i
Revised: i

Original: ( C i ) >
Revised: ( C i ) >

Original: − ϵ/ 4 and f B i ( C i +1 )
Revised: − ϵ/ 4 and f B i ( C i +1 )

Original: − f X ( C i +1 ) >
Revised: − f X ( C i +1 ) >

Original: − ϵ/ 4 .
Revised: − ϵ/ 4 .

Original: Finally due to the termination condition it holds that f B i ( C i ) − f B i ( C i +1 ) ≥ ϵ .
Revised: Finally due to the termination condition it holds that f B i ( C i ) − f B i ( C i +1 ) ≥ ϵ .

Original: This completes the proof.
Revised: This completes the proof.

Original: As f X is upper bounded by d , it holds that we must terminate within O ( d/ϵ ) iterations w.h.p when b = Ω( kd 3 ϵ − 2 log( nd/ϵ )) .
Revised: As f X is upper bounded by d , it holds that we must terminate within O ( d/ϵ ) iterations w.h.p when b = Ω( kd 3 ϵ − 2 log( nd/ϵ )) .

Original: We state our main theorem for this Section.
Revised: We state our main theorem for this Section.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: For b = ˜ Ω
Revised: For b = ˜ Ω

Original: ( kd 3 ϵ − 2 ) , Algorithm 1 terminates within O ( d/ϵ ) iterations w.h.p.
Revised: ( kd 3 ϵ − 2 ) , Algorithm 1 terminates within O ( d/ϵ ) iterations w.h.p.

Original: Towards a smaller batch size Note that the batch size used in this section is about a kd factor larger than what we require in Section 4.
Revised: Towards a smaller batch size Note that the batch size used in this section is about a kd factor larger than what we require in Section 4.

Original: This factor is required for the union bound over all possible sets of k centers in Lemma 3.2.
Revised: This factor is required for the union bound over all possible sets of k centers in Lemma 3.2.

Original: However, when actually applying Lemma 3.2, we only apply it for two centers in iteration i , setting B = B i and C =
Revised: However, when actually applying Lemma 3.2, we only apply it for two centers in iteration i , setting B = B i and C =

Original: C i , C
Revised: C i , C

Original: i +1 .
Revised: i +1 .

Original: A more direct approach would be to apply Lemma 3.1 only for C i , C i +1 , which would get rid of the extra kd factor.
Revised: A more direct approach would be to apply Lemma 3.1 only for C i , C i +1 , which would get rid of the extra kd factor.

Original: This will work when
Revised: This will work when

Original: C = C i as B i is sampled after C i is determined, but will fail for C
Revised: C = C i as B i is sampled after C i is determined, but will fail for C

Original: = C
Revised: = C

Original: i +

Revised: i +

Original: because C i +1 may depend on B i .
Revised: because C i +1 may depend on B i .

Original: In the following section, we show how to use the fact that the learning rate is sufficiently small in order to overcome this challenge.
Revised: In the following section, we show how to use the fact that the learning rate is sufficiently small in order to overcome this challenge.

Original:  M AIN RESULTS
Revised:  M AIN RESULTS

Original: In this section, we show that we can get a much better dependence on the batch size when using the standard center update rule.
Revised: In this section, we show that we can get a much better dependence on the batch size when using the standard center update rule.

Original: Specifically, we show that a batch of size ˜Ω(( d/ϵ ) 2 ) is sufficient to guarantee termination within O ( d/ϵ ) iterations.
Revised: Specifically, we show that a batch of size ˜Ω(( d/ϵ ) 2 ) is sufficient to guarantee termination within O ( d/ϵ ) iterations.

Original: We also do not require any assumption about the number of bits required to represent a real number.
Revised: We also do not require any assumption about the number of bits required to represent a real number.

Original: Section preliminaries Let us define for any finite tuple S ⊂ R d the center of mass of the tuple as cm ( S ) = 1 | S | (cid:80) x ∈ S x .
Revised: Section preliminaries Let us define for any finite tuple S ⊂ R d the center of mass of the tuple as cm ( S ) = 1 | S | (cid:80) x ∈ S x .

Original: For any tuple S ⊂ R d and some tuple of cluster centers C = ( C ℓ ) ℓ ∈ [ k ] it implies a partition ( S ℓ ) ℓ ∈ [ k ] of the points in S .
Revised: For any tuple S ⊂ R d and some tuple of cluster centers C = ( C ℓ ) ℓ ∈ [ k ] it implies a partition ( S ℓ ) ℓ ∈ [ k ] of the points in S .

Original: Specifically, every S ℓ contains the points in S closest to C ℓ and every point in S belongs to a single C ℓ (ties are broken arbitrarily).
Revised: Specifically, every S ℓ contains the points in S closest to C ℓ and every point in S belongs to a single C ℓ (ties are broken arbitrarily).

Original: We state the following useful observation:
Revised: We state the following useful observation:

Original: Observation 4.1.
Revised: Observation 4.1.

Original: Fix some A ⊆ X .
Revised: Fix some A ⊆ X .

Original: Let C be a tuple of k centers, S =
Revised: Let C be a tuple of k centers, S =

Original: ( S ℓ ) ℓ ∈ [ k ] be the partition of A induced by C and S =
Revised: ( S ℓ ) ℓ ∈ [ k ] be the partition of A induced by C and S =

Original: ( S ℓ ) ℓ ∈ [ k ] be any other partition of A .
Revised: ( S ℓ ) ℓ ∈ [ k ] be any other partition of A .

Original: It holds that (cid:80) kj =1 ∆( S j , C j ) ≤ (cid:80) kj =1 ∆( S j , C j ) .
Revised: It holds that (cid:80) kj =1 ∆( S j , C j ) ≤ (cid:80) kj =1 ∆( S j , C j ) .

Original: Let C ji denote the location of the j -th center in the beginning of the i -th iteration.
Revised: Let C ji denote the location of the j -th center in the beginning of the i -th iteration.

Original: Let ( B ℓi ) ℓ ∈ [ k ] be the partition of B i induced by C i and let ( X ℓi ) ℓ ∈ [ k ] be the partition of X induced by C i .
Revised: Let ( B ℓi ) ℓ ∈ [ k ] be the partition of B i induced by C i and let ( X ℓi ) ℓ ∈ [ k ] be the partition of X induced by C i .

Original: We analyze Algorithm 1 when clusters are updated as follows:
Revised: We analyze Algorithm 1 when clusters are updated as follows:

Original: C ji +1 =
Revised: C ji +1 =

Original: (1 − α ji ) C ji + α ji cm ( B ji ) , where α ji is the learning rate .
Revised: (1 − α ji ) C ji + α ji cm ( B ji ) , where α ji is the learning rate .

Original: Note that B ji may be empty in which case cm ( B ji ) is undefined, however, the learning rate is chosen such that α ji = 0 in this case ( C ji +1 =
Revised: Note that B ji may be empty in which case cm ( B ji ) is undefined, however, the learning rate is chosen such that α ji = 0 in this case ( C ji +1 =

Original: C ji ).
Revised: C ji ).

Original: Note that the learning rate may take on different values for different centers, and may change between iterations.
Revised: Note that the learning rate may take on different values for different centers, and may change between iterations.

Original: In the standard mini-batch k -means algorithm (Sculley, 2010; Pedregosa et al., 2011) the learning rate goes to 0 over time.
Revised: In the standard mini-batch k -means algorithm (Sculley, 2010; Pedregosa et al., 2011) the learning rate goes to 0 over time.

Original: This guarantees termination for most reasonable stopping conditions.
Revised: This guarantees termination for most reasonable stopping conditions.

Original: As before, we assume that the algorithm executes for at least t iterations and upper bound t .
Revised: As before, we assume that the algorithm executes for at least t iterations and upper bound t .

Original: We show that the learning rate need not go to 0 to guarantee termination when the batch size is sufficiently large.
Revised: We show that the learning rate need not go to 0 to guarantee termination when the batch size is sufficiently large.

Original: Specifically, we set α ji = (cid:113) b
Revised: Specifically, we set α ji = (cid:113) b

Original: ji /b , where b ji = (cid:12)(cid:12)(cid:12) B ji (cid:12)(cid:12)(cid:12) , and we require that b
Revised: ji /b , where b ji = (cid:12)(cid:12)(cid:12) B ji (cid:12)(cid:12)(cid:12) , and we require that b

Original: = Ω(( d/ϵ ) 2 log( ndtk )) .
Revised: = Ω(( d/ϵ ) 2 log( ndtk )) .

Original: Proof outline In our proof, we use the fact that a sufficiently small learning rate enhances the stability of the algorithm, which in turn allows us to use a much smaller batch size compared to
Revised: Proof outline In our proof, we use the fact that a sufficiently small learning rate enhances the stability of the algorithm, which in turn allows us to use a much smaller batch size compared to

Original: Section 3.
Revised: Section 3.

Original: Let us define the auxiliary value C ji
Revised: Let us define the auxiliary value C ji

Original: +1 =
Revised: +1 =

Original: (1 − α ji ) C ji + α ji cm ( X ji ) .
Revised: (1 − α ji ) C ji + α ji cm ( X ji ) .

Original: This is the j -th center at step i + 1 if we were to use the entire dataset for the update, rather than just a batch.
Revised: This is the j -th center at step i + 1 if we were to use the entire dataset for the update, rather than just a batch.

Original: Note that this is only used in the analysis and not in the algorithm.
Revised: Note that this is only used in the analysis and not in the algorithm.

Original: Recall that in the previous section we required a large batch size because we could not apply
Revised: Recall that in the previous section we required a large batch size because we could not apply

Original: Lemma 3.1 when B = B i and C = C i +1 because C i +1 may depend on B i .
Revised: Lemma 3.1 when B = B i and C = C i +1 because C i +1 may depend on B i .

Original: To overcome this challenge we use C i +1 instead of C
Revised: To overcome this challenge we use C i +1 instead of C

Original: i +1 .
Revised: i +1 .

Original: Note that C
Revised: Note that C

Original: i +1 only depends on C i , X
Revised: i +1 only depends on C i , X

Original: and is independent of B i (i.e., we can fix its value before sampling B i ).
Revised: and is independent of B i (i.e., we can fix its value before sampling B i ).

Original: We show that for our choice of learning rate it holds that C
Revised: We show that for our choice of learning rate it holds that C

Original: i +1 ,
Revised: i +1 ,

Original: C i +1 are sufficiently close, which implies that f X ( C i +1 ) , f X ( C i +1 ) and f B i
Revised: C i +1 are sufficiently close, which implies that f X ( C i +1 ) , f X ( C i +1 ) and f B i

Original: ( C i +1 ) , f B i ( C i +1 ) are also sufficiently close.
Revised: ( C i +1 ) , f B i ( C i +1 ) are also sufficiently close.

Original: This allows us to use a similar proof to that of
Revised: This allows us to use a similar proof to that of

Original: Lemma 3.3 where C i +1 acts as a proxy for C
Revised: Lemma 3.3 where C i +1 acts as a proxy for C

Original: i +1 .
Revised: i +1 .

Original: We formalize this intuition in what follows.
Revised: We formalize this intuition in what follows.

Original: First, we state the following useful lemmas:
Revised: First, we state the following useful lemmas:

Original: Lemma 4.2 (Kanungo et al.
Revised: Lemma 4.2 (Kanungo et al.

Original: For any set S ⊆ R d and any C ∈ R d it holds that ∆( S, C ) =
Revised: For any set S ⊆ R d and any C ∈ R d it holds that ∆( S, C ) =

Original: ∆( S, cm ( S ))
Revised: ∆( S, cm ( S ))

Original: + | S | ∆( C, cm ( S )) .
Revised: + | S | ∆( C, cm ( S )) .

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: For any S ⊆ X and C, C ′ ∈ [0 , 1] d , it holds that: | ∆( S, C ′ )
Revised: For any S ⊆ X and C, C ′ ∈ [0 , 1] d , it holds that: | ∆( S, C ′ )

Original: − ∆( S, C ) | ≤ 2 √ d | S | ∥ C − C ′ ∥ .
Revised: − ∆( S, C ) | ≤ 2 √ d | S | ∥ C − C ′ ∥ .

Original: Proof.
Revised: Proof.

Original: Using Lemma 4.2 we get that ∆( S, C ) =
Revised: Using Lemma 4.2 we get that ∆( S, C ) =

Original: ∆( S, cm ( S ))
Revised: ∆( S, cm ( S ))

Original: + | S | ∆( cm ( S ) , C ) and that ∆( S, C ′ ) =
Revised: + | S | ∆( cm ( S ) , C ) and that ∆( S, C ′ ) =

Original: ∆( S, cm ( S ))
Revised: ∆( S, cm ( S ))

Original: + | S | ∆( cm ( S ) , C ′ ) .
Revised: + | S | ∆( cm ( S ) , C ′ ) .

Original: Thus, it holds that | ∆( S, C ′ ) − ∆( S, C ) | = | S |·| ∆( cm ( S ) , C ′ )
Revised: Thus, it holds that | ∆( S, C ′ ) − ∆( S, C ) | = | S |·| ∆( cm ( S ) , C ′ )

Original: − ∆( cm ( S ) , C ) | .
Revised: − ∆( cm ( S ) , C ) | .

Original: Observe that for two vectors x, y ∈ R d it holds that ∆( x, y )
Revised: Observe that for two vectors x, y ∈ R d it holds that ∆( x, y )

Original: =
Revised: =

Original: ( x − y )
Revised: ( x − y )

Original: · ( x − y ) .
Revised: · ( x − y ) .

Original: Let us switch to vector notation and bound | ∆( cm ( S ) , C ′ )
Revised: Let us switch to vector notation and bound | ∆( cm ( S ) , C ′ )

Original: − ∆( cm ( S ) , C ) | .
Revised: − ∆( cm ( S ) , C ) | .

Original: | ∆( cm ( S ) , C ′ )
Revised: | ∆( cm ( S ) , C ′ )

Original: − ∆( cm ( S ) , C ) | = | ( cm ( S ) − C ′ ) · ( cm ( S ) − C ′ ) − ( cm ( S ) − C ) · ( cm ( S ) − C ) | = |− 2 cm ( S ) · C ′ + C ′ · C ′ + 2 cm ( S ) · C − C · C | = | 2 cm ( S ) · ( C − C ′ )
Revised: − ∆( cm ( S ) , C ) | = | ( cm ( S ) − C ′ ) · ( cm ( S ) − C ′ ) − ( cm ( S ) − C ) · ( cm ( S ) − C ) | = |− 2 cm ( S ) · C ′ + C ′ · C ′ + 2 cm ( S ) · C − C · C | = | 2 cm ( S ) · ( C − C ′ )

Original: + ( C ′ − C ) · ( C ′ + C ) | =
Revised: + ( C ′ − C ) · ( C ′ + C ) | =

Original: | ( C − C ′ ) · (2 cm ( S ) − ( C ′ + C ))
Revised: | ( C − C ′ ) · (2 cm ( S ) − ( C ′ + C ))

Original: | ≤ ∥ C − C ′ ∥∥ 2 cm ( S ) − ( C ′ + C ) ∥ ≤ 2 √ d ∥ C − C ′ ∥
Revised: | ≤ ∥ C − C ′ ∥∥ 2 cm ( S ) − ( C ′ + C ) ∥ ≤ 2 √ d ∥ C − C ′ ∥

Original: Where in the last transition we used the Cauchy-Schwartz inequality.
Revised: Where in the last transition we used the Cauchy-Schwartz inequality.

Original: First, we show that due to our choice of learning rate C ji +1 , C ji +1 are sufficiently close.
Revised: First, we show that due to our choice of learning rate C ji +1 , C ji +1 are sufficiently close.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: For it holds w.h.p that ∀ i ∈ [ t ] , j ∈ [ k ] , ∥C ji +1 − C ji +1 ∥ ≤ [Equation] 10 √ d [Equation]
Revised: For it holds w.h.p that ∀ i ∈ [ t ] , j ∈ [ k ] , ∥C ji +1 − C ji +1 ∥ ≤ ϵ 10 √ d .

Original: Proof.
Revised: Proof.

Original: Note that C ji +1 − C ji +

Revised: Note that C ji +1 − C ji +

Original: =
Revised: =

Original: α ji ( cm ( B ji ) − cm ( X ji )) .
Revised: α ji ( cm ( B ji ) − cm ( X ji )) .

Original: Let us fix some iteration i and center j .
Revised: Let us fix some iteration i and center j .

Original: To simplify notation, let us denote: X ′ =
Revised: To simplify notation, let us denote: X ′ =

Original: X ji , B ′ =
Revised: X ji , B ′ =

Original: B ji , b ′ = b ji , α ′ =
Revised: B ji , b ′ = b ji , α ′ =

Original: α ji .
Revised: α ji .

Original: Although b ′ is a random variable, in what follows we treat it as a fixed value (essentially conditioning on its value).
Revised: Although b ′ is a random variable, in what follows we treat it as a fixed value (essentially conditioning on its value).

Original: As what follows holds for all values of b ′ it also holds without conditioning due to the law of total probabilities.
Revised: As what follows holds for all values of b ′ it also holds without conditioning due to the law of total probabilities.

Original: For the rest of the proof, we assume b ′ > 

Revised: For the rest of the proof, we assume b ′ >

Original: (if b ′ = 

Revised: (if b ′ =

Original: the claim holds trivially).
Revised: the claim holds trivially).

Original: Let us denote by { Y ℓ } b ′ ℓ =1 the sampled points in B ′ .
Revised: Let us denote by { Y ℓ } b ′ ℓ =1 the sampled points in B ′ .

Original: Note that a randomly sampled element from X is in B ′ if and only if it is in X ′ .
Revised: Note that a randomly sampled element from X is in B ′ if and only if it is in X ′ .

Original: As batch elements are sampled uniformly at random with repetitions from X , conditioning on the fact that an element is in B ′ means that it is distributed uniformly over X ′ .
Revised: As batch elements are sampled uniformly at random with repetitions from X , conditioning on the fact that an element is in B ′ means that it is distributed uniformly over X ′ .

Original: Thus, it holds that ∀ ℓ, E [ Y ℓ ] = 1 | X ′ | (cid:80) x
Revised: Thus, it holds that ∀ ℓ, E [ Y ℓ ] = 1 | X ′ | (cid:80) x

Original: ∈ X ′ x
Revised: ∈ X ′ x

Original: = cm ( X ′ ) and E [ cm ( B ′ )]
Revised: = cm ( X ′ ) and E [ cm ( B ′ )]

Original: = 1 b ′ (cid:80) b ′ ℓ =1 E
Revised: = 1 b ′ (cid:80) b ′ ℓ =1 E

Original: [ Y ℓ ] = cm ( X ′ ) .
Revised: [ Y ℓ ] = cm ( X ′ ) .

Original: Our goal is to bound Pr [ ∥ cm ( B ′ ) − cm ( X ′ )
Revised: Our goal is to bound Pr [ ∥ cm ( B ′ ) − cm ( X ′ )

Original: ∥ ≥ ϵ 10 α ′ √ d ] , we note that it is sufficient to bound the deviation of every coordinate by ϵ/ (10 α ′ d ) , as that will guarantee that:
Revised: ∥ ≥ ϵ 10 α ′ √ d ] , we note that it is sufficient to bound the deviation of every coordinate by ϵ/ (10 α ′ d ) , as that will guarantee that:

Original: [Equation]
Revised: [Equation]

Original: We note that for a single coordinate, ℓ , we can apply a Hoeffding bound with parameters µ = cm ( X ′ )
Revised: We note that for a single coordinate, ℓ , we can apply a Hoeffding bound with parameters µ = cm ( X ′ )

Original: [ ℓ ] , a max − a min ≤ 1 and get that:
Revised: [ ℓ ] , a max − a min ≤ 1 and get that:

Original: [Equation]
Revised: [Equation]

Original: Taking a union bound we get that
Revised: Taking a union bound we get that

Original: [Equation]
Revised: [Equation]

Original: Using the fact that α ′ =
Revised: Using the fact that α ′ =

Original: (cid:112) b ′ /b together with the fact that b = Ω(( d/ϵ ) 2 log( ntkd )) (for an appropriate constant) we get that the above is O (1 /ntk ) .
Revised: (cid:112) b ′ /b together with the fact that b = Ω(( d/ϵ ) 2 log( ntkd )) (for an appropriate constant) we get that the above is O (1 /ntk ) .

Original: Finally, taking a union bound over all t iterations and all k centers per iteration completes the proof.
Revised: Finally, taking a union bound over all t iterations and all k centers per iteration completes the proof.

Original: Let us now use the above lemma to bound the goal function when cluster centers are close.
Revised: Let us now use the above lemma to bound the goal function when cluster centers are close.

Original: Lemma 4.5.
Revised: Lemma 4.5.

Original: Fix some A ⊆ X .
Revised: Fix some A ⊆ X .

Original: It holds w.h.p that ∀ i ∈ [ t ] , (cid:12)(cid:12)
Revised: It holds w.h.p that ∀ i ∈ [ t ] , (cid:12)(cid:12)

Original: [Equation]

Original: [Equation] f A 
Revised: f A ( C i +1 )

Original: [Equation] C i +1 ) [Equation] ≤ ϵ/
Revised: − f A ( C i +1 ) (cid:12)(cid:12) ≤ ϵ/

Original: Proof.
Revised: Proof.

Original: Let S =
Revised: Let S =

Original: ( S ℓ ) ℓ ∈ [ k ] , S =
Revised: ( S ℓ ) ℓ ∈ [ k ] , S =

Original: ( S ℓ ) ℓ ∈ [ k ] be the partitions induced by C
Revised: ( S ℓ ) ℓ ∈ [ k ] be the partitions induced by C

Original: i +1 ,
Revised: i +1 ,

Original: C i +1 on A .
Revised: C i +1 on A .

Original: Let us expand the expression:
Revised: Let us expand the expression:

Original: [Equation]
Revised: [Equation]

Original: Where the first inequality is due to Observation 4.1, the second is due Lemma 4.3 and finally we use
Revised: Where the first inequality is due to Observation 4.1, the second is due Lemma 4.3 and finally we use

Original: Lemma 4.4 together with the fact that (cid:80) kj =1 (cid:12)(cid:12)
Revised: Lemma 4.4 together with the fact that (cid:80) kj =1 (cid:12)(cid:12)

Original: S j (cid:12)(cid:12) =
Revised: S j (cid:12)(cid:12) =

Original: | A | .
Revised: | A | .

Original: Using the same argument we also get that f A ( C i +1 ) − f A
Revised: Using the same argument we also get that f A ( C i +1 ) − f A

Original: ( C i +1 ) ≤ ϵ/ 5 , which completes the proof.
Revised: ( C i +1 ) ≤ ϵ/ 5 , which completes the proof.

Original: From here our proof is somewhat similar to that of Section 3.
Revised: From here our proof is somewhat similar to that of Section 3.

Original: Let us state the following useful lemma.
Revised: Let us state the following useful lemma.

Original: Lemma 4.6.
Revised: Lemma 4.6.

Original: It holds w.h.p that for every i ∈ [ t ] :
Revised: It holds w.h.p that for every i ∈ [ t ] :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The first two inequalities follow from Lemma 4.5.
Revised: The first two inequalities follow from Lemma 4.5.

Original: The last two are due to Lemma 3.1 by setting δ = ϵ/ 5 , B
Revised: The last two are due to Lemma 3.1 by setting δ = ϵ/ 5 , B

Original: = B i :
Revised: = B i :

Original: [Equation]
Revised: [Equation]

Original: Where the last inequality is due to the fact that b = Ω(( d/ϵ ) 2 log( nt )) (for an appropriate constant).
Revised: Where the last inequality is due to the fact that b = Ω(( d/ϵ ) 2 log( nt )) (for an appropriate constant).

Original: The above holds for either C = C i or C
Revised: The above holds for either C = C i or C

Original: = C
Revised: = C

Original: i +1 .
Revised: i +1 .

Original: Taking a union bound over all t iterations we get the desired result.
Revised: Taking a union bound over all t iterations we get the desired result.

Original: Putting everything together We wish to lower bound f X ( C i ) − f X
Revised: Putting everything together We wish to lower bound f X ( C i ) − f X

Original: ( C i +1 ) .
Revised: ( C i +1 ) .

Original: We write the following:
Revised: We write the following:

Original: [Equation]
Revised: [Equation]

Original: Where the first inequality is due to inequality (3) in Lemma 4.6 ( f X ( C i ) − f B i ( C i ) ≥ − ϵ/ 5 ), the second is due to the stopping condition of the algorithm ( f B i ( C i ) − f B
Revised: Where the first inequality is due to inequality (3) in Lemma 4.6 ( f X ( C i ) − f B i ( C i ) ≥ − ϵ/ 5 ), the second is due to the stopping condition of the algorithm ( f B i ( C i ) − f B

Original: i
Revised: i

Original: ( C i +1 ) > ϵ ), and the last is due to the remaining inequalities in Lemma 4.6.
Revised: ( C i +1 ) > ϵ ), and the last is due to the remaining inequalities in Lemma 4.6.

Original: The above holds w.h.p over all of the iterations of the algorithms.
Revised: The above holds w.h.p over all of the iterations of the algorithms.

Original: As in Section 3, we conclude that w.h.p it holds that t =
Revised: As in Section 3, we conclude that w.h.p it holds that t =

Original: O ( d/ϵ ) , which implies that b
Revised: O ( d/ϵ ) , which implies that b

Original: = Ω(( d/ϵ ) 2 log( knd/ϵ )) is sufficient.
Revised: = Ω(( d/ϵ ) 2 log( knd/ϵ )) is sufficient.

Original: We state our main theorem.
Revised: We state our main theorem.

Original: Theorem 4.7.
Revised: Theorem 4.7.

Original: For b = ˜Ω(( d/ϵ ) 2 ) and
Revised: For b = ˜Ω(( d/ϵ ) 2 ) and

Original: α ji = (cid:113) b ji /b , Algorithm 1 with center update C ji +1 =
Revised: α ji = (cid:113) b ji /b , Algorithm 1 with center update C ji +1 =

Original: (1 − α ji ) C ji + α ji cm ( B ji ) , terminates within O ( d/ϵ ) iterations w.h.p.
Revised: (1 − α ji ) C ji + α ji cm ( B ji ) , terminates within O ( d/ϵ ) iterations w.h.p.

Original:  A PPLICATION TO SKLEARN
Revised:  A PPLICATION TO SKLEARN

Original: In this section, we show the relevance of our results to the algorithm implementation of sklearn.
Revised: In this section, we show the relevance of our results to the algorithm implementation of sklearn.

Original: The main differences in sklearn are the learning rate and stopping condition.
Revised: The main differences in sklearn are the learning rate and stopping condition.

Original: The termination conditiondepends on the movement of the centers in the iteration, rather than the value of f B i .
Revised: The termination conditiondepends on the movement of the centers in the iteration, rather than the value of f B i .

Original: Specifically, we continue as long as (cid:80) j ∈ [ k ] ∆( C ji +1 , C ji ) ≥ ϵ for some tolerance parameter ϵ .
Revised: Specifically, we continue as long as (cid:80) j ∈ [ k ] ∆( C ji +1 , C ji ) ≥ ϵ for some tolerance parameter ϵ .

Original: The learning rate is set as α ji = b [Equation]
Revised: The learning rate is set as α ji = b ji (cid:80) iℓ =1 b jℓ .

Original: Roughly speaking, this implies that α ji → 0 over time, and guarantees termination of the algorithm in the limit.
Revised: Roughly speaking, this implies that α ji → 0 over time, and guarantees termination of the algorithm in the limit.

Original: However, for our convergence guarantee, we only require α j i = (cid:113)
Revised: However, for our convergence guarantee, we only require α j i = (cid:113)

Original: b
Revised: b

Original: j i /b which need not go toover time.
Revised: j i /b which need not go toover time.

Original: We show that with our learning rate and the termination condition of sklearn, the proof from Section 4 still implies termination, although at a slower rate and requires a larger batch size.
Revised: We show that with our learning rate and the termination condition of sklearn, the proof from Section 4 still implies termination, although at a slower rate and requires a larger batch size.

Original: Specifically, we terminate within O (( d/ϵ ) 1 . 5 √ k ) iterations w.h.p if the batch size is ˜Ω( k ( d/ϵ ) 3 ) .
Revised: Specifically, we terminate within O (( d/ϵ ) 1 . 5 √ k ) iterations w.h.p if the batch size is ˜Ω( k ( d/ϵ ) 3 ) .

Original: Note that this result is not subsumed by the result in Section 3 because the stopping condition is different.
Revised: Note that this result is not subsumed by the result in Section 3 because the stopping condition is different.

Original: Below we show that as long as the termination condition in sklearn does not hold ( (cid:80) j ∈ [ k ] ∆( C ji +1 , C ji ) ≥ ϵ ), our stopping condition also does not hold for an appropriate parameter ( f B i ( C i ) − f B i
Revised: Below we show that as long as the termination condition in sklearn does not hold ( (cid:80) j ∈ [ k ] ∆( C ji +1 , C ji ) ≥ ϵ ), our stopping condition also does not hold for an appropriate parameter ( f B i ( C i ) − f B i

Original: ( C i +1 ) >
Revised: ( C i +1 ) >

Original: ϵ ′ where ϵ ′ =
Revised: ϵ ′ where ϵ ′ =

Original: ϵ 1 . 5 / √ kd ) .
Revised: ϵ 1 . 5 / √ kd ) .

Original: We state the following useful lemma:
Revised: We state the following useful lemma:

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: Let x, y ∈ R d , α ∈ [0 , 1] .
Revised: Let x, y ∈ R d , α ∈ [0 , 1] .

Original: It holds that ∆( x, (1 − α ) x + αy ) =
Revised: It holds that ∆( x, (1 − α ) x + αy ) =

Original: α 2 ∆( x, y ) .
Revised: α 2 ∆( x, y ) .

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: Below is our main lemma for this section:
Revised: Below is our main lemma for this section:

Original: Lemma 5.2.
Revised: Lemma 5.2.

Original: If it holds that (cid:80) j ∈ [ k ] ∆( C ji +1 , C ji ) >
Revised: If it holds that (cid:80) j ∈ [ k ] ∆( C ji +1 , C ji ) >

Original: ϵ then f B i ( C i ) − f B i
Revised: ϵ then f B i ( C i ) − f B i

Original: ( C i +1 ) >
Revised: ( C i +1 ) >

Original: ϵ 1 . 5 √ kd .
Revised: ϵ 1 . 5 √ kd .

Original: Proof.
Revised: Proof.

Original: Recall that C ji +1 =
Revised: Recall that C ji +1 =

Original: (1 − α ji ) C ji + α ji cm ( B ji ) for α ji = (cid:113)
Revised: (1 − α ji ) C ji + α ji cm ( B ji ) for α ji = (cid:113)

Original: b ji /b .
Revised: b ji /b .

Original: Thus, we get:
Revised: Thus, we get:

Original: [Equation]
Revised: [Equation]

Original: Where in the transitions we used Lemma 5.1.
Revised: Where in the transitions we used Lemma 5.1.

Original: Let us fix some j ∈ [ k ] , we can write the following:
Revised: Let us fix some j ∈ [ k ] , we can write the following:

Original: [Equation]
Revised: [Equation]

Original: Where in the first transition we apply Lemma 4.2, and in the last we use the fact that ∆( C ji +1 , cm ( B ji ))
Revised: Where in the first transition we apply Lemma 4.2, and in the last we use the fact that ∆( C ji +1 , cm ( B ji ))

Original: = ( α ji ) 2 ∆( C ji , cm ( B ji )) and the fact that ∀ , α ji ∈ [0 , 1] , 2 α ji − ( α ji ) 2 ≥ α ji .
Revised: = ( α ji ) 2 ∆( C ji , cm ( B ji )) and the fact that ∀ , α ji ∈ [0 , 1] , 2 α ji − ( α ji ) 2 ≥ α ji .

Original: Let us bound f B i ( C i ) − f B i
Revised: Let us bound f B i ( C i ) − f B i

Original: ( C i +1 ) :
Revised: ( C i +1 ) :

Original: [Equation]
Revised: [Equation]

Original: Where the first inequality is due to Observation 4.1, the second is due to the fact that ∀ j ∈ [ k ] , ∆( B ji , C ji ) − ∆( B ji , C ji +1 ) ≥ α ji b ji ∆( C ji , cm ( B ji )) , and in the last equality we simply plug in α ji =
Revised: Where the first inequality is due to Observation 4.1, the second is due to the fact that ∀ j ∈ [ k ] , ∆( B ji , C ji ) − ∆( B ji , C ji +1 ) ≥ α ji b ji ∆( C ji , cm ( B ji )) , and in the last equality we simply plug in α ji =

Original: b ji b combined with.
Revised: b ji b combined with.

Original: We complete the proof by applying Jensen’s inequality, with parameters: ϕ ( x ) = x 1 . 5 , y j = b ji /b and a j = ∆( C ji , cm ( B ji )) , combined with inequality (5).
Revised: We complete the proof by applying Jensen’s inequality, with parameters: ϕ ( x ) = x 1 . 5 , y j = b ji /b and a j = ∆( C ji , cm ( B ji )) , combined with inequality (5).

Original: [Equation]
Revised: [Equation]

Original: Finally, plugging ϵ ′ =
Revised: Finally, plugging ϵ ′ =

Original: ϵ 1 . 5 √ kd into our bounds, we conclude that if b = ˜Ω( ϵ − 3 d 3 k ) then the number of iterations is bounded by O (( d/ϵ ) 1 . 5 √ k ) w.h.p. 
Revised: ϵ 1 . 5 √ kd into our bounds, we conclude that if b = ˜Ω( ϵ − 3 d 3 k ) then the number of iterations is bounded by O (( d/ϵ ) 1 . 5 √ k ) w.h.p. A CKNOWLEDGMENTS

Revised: The author would like to thank Ami Paz, Uri Meir and Giovanni Viglietta for reading preliminary versions of this work.

Revised: This work was supported by JSPS KAKENHI Grant Numbers JP21H05850, JP21K17703,

Revised: JP21KK0204.