A SYNCHRONOUS S TOCHASTIC S UBGRADIENT M ETHODS FOR G ENERAL N ONSMOOTH N ONCONVEX O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Asynchronous distributed methods are a popular way to reduce the communication and synchronization costs of large-scale optimization.
Yet, for all their success, little is known about their convergence guarantees in the challenging case of general non-smooth, non-convex objectives, beyond cases where closed-form proximal operator solutions are available.
This is all the more surprising since these objectives are the ones appearing in the training of deep neural networks.
In this paper, we introduce the ﬁrst convergence analysis covering asynchronous methods in the case of general non-smooth, non-convex objectives.
Our analysis applies to stochastic sub-gradient descent methods both with and without block variable partitioning, and both with and without momentum.
It is phrased in the context of a general probabilistic model of asynchronous scheduling accurately adapted to modern hardware properties.
We validate our analysis experimentally in the context of training deep neural network architectures.
We show their overall successful asymptotic convergence as well as exploring how momentum, synchronization, and partitioning all affect performance.
I NTRODUCTION
Training parameters arising in Deep Neural Net architectures is a difﬁcult problem in several ways (Goodfellow et al., 2016).
First, with multiple layers and nonlinear activation functions such as sigmoid and softmax functions, the ultimate optimization problem is nonconvex.
Second, with ReLU activation functions and max-pooling in convolutional structures, the problem is nonsmooth, i.e., it is not differentiable everywhere, although typically the set of non-differentiable points is a set of measure zero in the space of the parameters.
Finally, in many applications it is unreasonable to load the whole sample size in memory to evaluate the objective function or (sub)gradient, thus samples must be taken, necessitating analysis in a probabilistic framework.
The analysis of parallel optimization algorithms using shared memory architectures, motivated by applications in machine learning, was ushered in by the seminal work of Recht et al.
(2011) (although precursors exist, see the references therein).
Further work reﬁned this analysis, e.g. (Liu & Wright, 2015) and expanded it to nonconvex problems, e.g. (Lian et al., 2015).
However, in all of these results, a very simplistic model of asynchronous computation is presented to analyze the problem.
Notably, it is assumed that every block of the parameter, among the set of blocks of iterates being optimized, has a ﬁxed, equal probability of being chosen at every iteration, with a certain vector of delays that determine how old each block is that is stored in the cache relative to the shared memory.
As one can surmise, this implies complete symmetry with regards to cores reading and computing the different blocks.
This does not correspond to asynchronous computation in practice.
In particular, in the common Non-Uniform Memory Access (NUMA) setting, practical experience has shown that it can be effective for each core to control a set of blocks.
Thus, the choice of blocks will depend on previous iterates, which core was last to update, creating probabilistic dependence between the delay vector and the choice of block.
This exact model is formalized in Cannelli et al., which introduced a new probabilistic model of asynchronous parallel optimization and presented a coordinate-wise updating successive convex approximation algorithm.
In this paper, we are interested in studying parallel asynchronous stochastic subgradient descent for general nonconvex nonsmooth objectives, such as the ones arising in the training of deep neural network architectures.
Currently, there is no work in the literature speciﬁcally addressing this problem.
The closest related work is given by Zhu et al.
and Li & Li (2018), which consider asynchronous proximal gradient methods for solving problems of the form f ( x ) + g ( x ) , where f is smooth and nonconvex, and g ( x ) is nonsmooth, with an easily computable closed form prox expression.
This restriction applies to the case of training a neural network which has no ReLUs or max pooling in the architecture itself, i.e., every activation is a smooth function, and there is an additional regularization term, such as an (cid:96) 1 .
These papers derive expected rates of convergence.
In the general case, where the activations themselves are nonsmooth—for instance in the presence of ReLUs—there is no such additive structure, and no proximal operator exists to handle away the non-smoothness and remove the necessity of computing and using subgradients explicitly in the optimization procedure.
This general problem of nonsmooth nonconvex optimization is already difﬁcult (see, e.g., Bagirov et al.
(2014)), and the introduction of stochastically uncertain iterate updates creates an additional challenge.
Classically, the framework of stochastic approximation, with stochastic estimates of the subgradient approximating elements in a differential inclusion that deﬁnes a ﬂow towards minimization of the objective function, is a standard, effective approach to analyzing algorithms for this class of problems.
Some texts on the framework include Kushner & Yin (2003), which we shall reference extensively in the paper, and Borkar (2008).
See also Ermol’ev & Norkin (1998) and Ruszczy´nski (1987) for some classical results in convergence of stochastic algorithms for nonconvex nonsmooth optimization.
Interest in stochastic approximation has resurfaced recently sparked by the popularity of Deep Neural
Network architectures.
For instance, see the analysis of nonconvex nonsmooth stochastic optimization with an eye towards such models in Davis et al.
(2018) and Majewski et al.
In this paper, we provide the ﬁrst analysis for nonsmooth nonconvex stochastic subgradient methods in a parallel asynchronous setting, in the stochastic approximation framework.
For this, we employ the state of the art model of parallel computation introduced in Cannelli et al., which we map onto the analysis framework of Kushner & Yin (2003).
We prove show that the generic asynchronous stochastic subgradient methods considered are convergent, with probability 1, for nonconvex nonsmooth functions.
This is the ﬁrst result for this class of algorithms, and it combines the state of the art in these two areas, while extending the scope of the results therein.
Furthermore, given the success of momentum methods (see, e.g., Zhang et al. (2017)), we consider the momentum variant of the classical subgradient method, again presenting the ﬁrst convergence analysis for this class of algorithms.
We validate our analysis numerically by demonstrating the performance of asynchronous stochastic subgradient methods of different forms on the problem of ResNet deep network training.
We shall consider variants of asynchronous updating with and without write locks and with and without block variable partitioning, showing the nuances in terms of convergence behavior as depending on these strategies and properties of the computational hardware.
P ROBLEM F ORMULATION
Consider the minimization problem
[Equation]
where f : R n → R is locally Lipschitz continuous (but could be nonconvex and nonsmooth) and furthermore, it is computationally infeasible to evaluate f ( x ) or an element of the Clarke subdifferential ∂f ( x ) .
The problem (1) has many applications in machine learning, including the training of parameters in deep neural networks.
In this setting, f ( x ) is loss function evaluated on some model with x as its parameters, and is dependant on input data A ∈ R n × m and target values
y ∈ R m of high dimension, i.e., f ( x ) =
f ( x ; ( A, y )) , with x a parameter to optimize with respect to the loss function.
In cases of practical interest, f is decomposable in ﬁnite-sum form,
[Equation]
where l : R m × R m → R represents the training loss and { ( A i , y i ) } is a partition of ( A, y ) .
We are concerned with algorithms that solve (1) in a distributed fashion, i.e., using multiple processing cores.
In particular, we are analyzing the following inconsistent read scenario: before computation begins, each core c is allocated a block of variables I c , for which it is responsible to update.
At each iteration the core modiﬁes a block of variables i k , chosen randomly among I c .
Immediately after core c completes its k -th iteration, it updates the shared memory.
A lock is only placed on the shared memory when a core writes to it, thus the process of reading may result in computations of the function evaluated at variable values that never existed in memory, e.g., block 1 is read by core 1 , then core 3 updates block 2 , then core 1 reads block 2 , and now block 1 is operating on a vector with the values in blocks 1 and 2 not simultaneously at their present local values at any point in time in shared memory.
We shall index iterations to indicate when a core writes a new set of values for the variable into memory.
We let d k = { d k c 1 , ..., d k c n } be the vector of delays for each component of the variable used to evaluate the subgradient estimate, thus the j -th component of x that is used in the computation of the update at k is actually not x k c j but x d kcj j .
In this paper, we are interested in applying stochastic approximation methods, of which the classic stochastic gradient descent forms a special case.
Since f in (1) is in general nonsmooth, we will exploit subgradient methods.
Denote by ξ k the set of mini-batches used to compute an element of the subgradient g (( x d kc 1 1 , ..., x d kcn n ); ξ k c ) .
The set of minibatches ξ k c is chosen uniformly at random from ( A, y ) , independently at each iteration.
By the central limit theorem, the error is asymptotically
Gaussian as the total size of the data as well as the size of the mini-batches increases.
2. A LGORITHM D ESCRIPTION
We now recall the stochastic subgradient algorithm under asynchronous updating in Algorithm 1, from the perspective of the individual cores.
The update of the iterate performed by
[Equation]
where m is the momentum constant, required to satisfy 0 < m <
Algorithm
Input: x 0 , core c .
1: while Not converged do 2:
Sample i from the variables I c corresponding to c .
Sample ξ .
Read x k c from the shared memory 4:
Compute a subgradient estimate g k c , local to k c 5:
Write, with a lock, to the shared memory vector partition u i kc =
mu i kc
+ g k c
i kc 6:
Update, with a lock, to the shared memory vector partition x i = x
i − (1 − m ) γ k c u i kc 7: k c =
k c +8: end while
A NALYSIS
For the discrete time probabilistic model of computation introduced in Cannelli et al.
, we must present the basic requirements that must hold across cores.
In particular, it is reasonable to expect that if some core is entirely faulty, or exponentially deccelerates in its computation, convergence should not be expected to be attained.
Otherwise we wish to make the probabilistic assumption governing the asynchronous update scheduling as general as possible in allowing for a variety of possible architectures.
The details of the probabilistic assumptions are technical and left to the Supplementary Material.
It can be veriﬁed that the stochastic approxmation framework discussed in the next section detailing the convergence satisﬁes these assumptions.
We have the standard assumption about the stochastic sub-gradient estimates.
These assumptions hold under the standard stochastic gradient approach wherein one samples some subset ξ ⊆ { 1 , ..., M } of mini-batches uniformly from the set of size | ξ | subsets of { 1 , ..., M } , done independently at each iteration.
This results in independent noise at each iteration being applied to the stochastic subgradient term.
From these mini-batches ξ , a subgradient is taken for each j ∈ ξ and averaged.
Assumption 3.1.
The stochastic subgradient estimates g ( x, ξ ) satisfy,
[Equation]
where β ( x ) deﬁnes a bias term that is zero if f ( · ) is continuously differentiable at x .
We provide some more details on the “global” model of asynchronous stochastic updating in the
Supplementary material.
3. C ONTINUOUS T IME M ODEL AND S TOCHASTIC P ROCESS
In this section, we shall redeﬁne the algorithm and its associated model presented in the previous section in a framework appropriate for analysis from the stochastic approximation perspective.
Consider the Algorithm described as such, for data block i with respective iteration k ,
[Equation]
where Y j,i is the estimate of the partial subgradient with respect to block variables indexed by i at local iteration j .
In the context of Algorithm 1, the step size is deﬁned to be the subsequence { γ k,i } = { γ ν ( c ( i ) ,l ) :
i = i l } where l is the iteration index for the core corresponding to block i .
Thus it takes the subsequence of γ k for which i k
= i is the block of variables being modiﬁed.
The step Y k,i corresponding to g ( xk, ξ ) satisﬁes,
[Equation]
We denote g
i ( x ) to denote a selection of some element of the subgradient, with respect to block i , of f ( x ) .
The quantity δM k,i represents a Martingale difference, satisfying δM k,i =
M k +1 ,i −
M k,i for some Martingale M k , a sequence of random variables which satisﬁes E [ M k,i ] < ∞ and E [ M k +1 ,i | M j,i , j ≤ k ] =
M k,i with probability 1 for all k .
It holds that E [ | M k,i | 2 ] < ∞ and E [
M k +1 ,i − M k,i ]
[ M j +1 ,i − M j,i ] (cid:48)
= 0 .
Finally, it holds that E k,i [ δM k,i ] = 0 .
These are standard conditions implied by the sampling procedure in stochastic gradient methods, introduced by the original Robbins-Monro method (Robbins & Monro, 1985).
In Stochastic Approximation, the standard approach is to formulate a dynamic system or differential inclusion that the sequence of iterates approaches asymptotically.
For this reason, we introduce real time into the model of asynchronous computation, looking at the actual time elapsed between iterations for each block i .
Deﬁne δτ k,i to be the real elapsed time between the k -th and k + 1 -st iteration for block i .
We let
T k,i = (cid:80) k − 1 j =
δτ j,i and deﬁne for σ ≥ 0 , p l ( σ ) =
min { j : T j,i ≥ σ } the ﬁrst iteration at or after σ .
We assume now that the step-size sequence comes from an underlying real function, i.e., γ k,i =
δτ k,i (cid:90) T k,i + δτ k,i
T k,i γ ( s ) ds satisfying (cid:82) ∞
γ [Equation]
s ) ds = ∞ , where 0 < γ ( s ) → 0 as s → ∞ , There are T ( s ) → ∞ as s → ∞ such that lim s →∞
[Equation]
We now deﬁne new σ -algebras F k,i and F + k,i deﬁned to measure the random variables {{ x 0 } , { Y j − 1 ,i : j, i with T j,i <
T k
+1 ,i } , { T j,i : j, i with T j,i ≤ T k
+1 ,i }} , and, {{ x 0 }
, { Y j − 1 ,i : j, i with T j,i ≤ T k +1 ,i } , { T j,i : j, i with T j,i ≤
T k
+1 ,i }} , indicating the set of events up to, and up to and including the computed noisy update at k , respectively.
Note that each of these constructions is still consistent with a core updating different blocks at random, with δτ k,i arising from an underlying distribution for δτ k,
c ( i ) .
Let us relate these σ -algebras to those in the previous section.
Note that this takes subsets of random variables ( i k , d k , ξ k ) for which k is such that i k is i (in the original notation of k ).
The form of Y k,i deﬁned above incorporates the random variable d k and i k , as in which components are updated and the age of the information used by where the subgradient is evaluated, as well as ξ k by the presence of the Martingale difference noise.
For any sequence Z k,i we write Z σk,i =
Z p i ( σ )+ k,i , where p i ( σ ) is the least integer greater than or equal to σ .
Thus, let δτ σk,i denote the inter-update times for block i starting at the ﬁrst update at or after σ , and γ σk,i the associated step sizes.
Now let x σ 0 ,i =
x p i ( σ ) ,i and for k ≥ 0 , x σk +1 ,
i = x σk,i + (1 − m ) γ σk,i (cid:80) kj =
m k − j Y
σj,i .
We consider t σk,i =
(cid:80) k − 1 j =
γ σj,i and τ σk,i =
(cid:80) k − 1 j =0 γ σj,i δτ σj,
i
[Equation]
We introduce piecewise constant interpolations of the vectors in real-time given by,
[Equation]
and τ σi ( t ) =
τ σk,i for t ∈ [ t σk,i , t σk +1 ,i ] .
We also have,
[Equation]
Now we detail the assumptions on the real delay times.
These ensure that the real-time delays do not grow without bound, either on average, or on relevantly substantial probability mass.
Intuitively, this means that it is highly unlikely that any core deccelerates exponentially in its computation speed.
Assumption 3.2.
It holds that { δτ σk,i ; k, i } is uniformly integrable.
Assumption 3.3.
There exists a function u σk +1 ,i and random variables ∆ σ, + k +1 ,i and a random sequence { ψ σk +1 ,i } such that
[Equation]
and there is a ¯ u such that for any compact set A ,
[Equation]
Assumption 3.4.
It holds that,
[Equation]
This assumption holds if, e.g., the set of x such that f ( · ) is not continuously differentiable at x is of measure zero, which is the case for objectives of every DNN architecture the authors are aware of.
3. C ONVERGENCE
As mentioned earlier, the primary goal of the previous section is to deﬁne a stochastic process that approximates some real-time process asymptotically, with this real-time process deﬁned by dynamics for which at the limit the path converges to a stationary point.
In particular, we shall see that the process deﬁned for the iterate time scale approximates the path of a differential inclusion,
[Equation]
and we shall see that this path deﬁnes stationary points of f ( · ) .
We must deﬁne the notion of an invariant set for a differential inclusion (DI).
Deﬁnition 3.1.
A set Λ ⊂ R n is an invariant set for a DI ˙ x ∈ g ( x ) if for all x 0 ∈ Λ , there is a solution x ( t ) ,
−∞ < t < ∞ that lies entirely in Λ and satisﬁes x (0) = x 0 .
Now we state our main result.
Its complete proof can be found in the Supplementary Material.
Theorem 3.1.
Let all the stated Assumptions hold.
Then, the following system of differential inclusions,
[Equation]
holds for any u satisfying 3.3.
On large intervals [0 , T ] , ˆ x σ ( · ) spends nearly all of its time, with the fraction going to one as T → ∞ and σ → ∞ in a small neighborhood of a bounded invariant set of (5) .
This Theorem shows weak convergence.
The extension to convergence with probability one is straightforward and described in the Supplementary material.
3.2. P ROPERTIES OF THE L IMIT P OINT
Finally, we wish to characterize the properties of this invariant set.
From Corollary 5.11 (Davis et al., 2018), we can conclude that problems arising in training of deep neural network architectures, wherein f ( x ) =
l ( y j , a L ) with l ( · ) one of several standard loss functions, including logistic or Hinge loss, and a i =
ρ
i
( V i ( x ) a i − 1 ) or i = 1 , ..., L layers, are activation functions, which are piece-wise deﬁned to be log x , e x , max(0 , x ) or log(1 + e x ) , are such that their set of invariants { x ∗ } for its associated differential inclusion satisﬁes
0 ∈ ∂f ( x ∗ ) , and furthermore the values f ( x k ) for any iterative algorithm generating { x k } such that x k → x ∗ , an invariant of f ( x ) , converge.
Note that the differential inclusions deﬁned above ensure asymptotic convergence to block-wise stationarity, i.e., 0 ∈ ∂ i f ( x ) for all i .
It is clear, however, that every stationary point is also blockwise stationary, i.e., that 0 ∈ ∂f ( x ) implies 0 ∈ ∂ i f ( x ) for all i .
In practice, the set of block-wise stationary points which are not stationary is not large.
One can alternatively consider a variant of the algorithm wherein every core updates the entire vector (thus there is no block partitioning) but locks the shared memory whenever it either reads of writes from it.
The same analysis applies to such a procedure.
In particular, this amounts to i k
= { 1 , ..., n } for all k and every limit of x σ ( t ) as either σ → ∞ or t → ∞ is a critical point of f ( x ) and, with probability one, asymptotically the algorithm converges to a critical point of f ( x ) (i.e., x such that 0 ∈ ∂f ( x ) ).
N UMERICAL R ESULTS
Methods.
We describe an experimental evaluation comparing the following algorithms:
WI ASSM : W rite In consistent A synchronous S tochastic S ubgradient M ethod with lock-free read and updates of x k,i .
This procedure applied to smooth strongly-convex and smooth nonconvex f ( x ) is known as HogWild!
in Recht et al.
(2011) and AsySG-incon in Lian et al.
(2015), respectively, in the literature.
Convergence analysis of HogWild!
and AsySG-incon additionally required sparsity of x .
They have no provable convergence guarantee for nonsmooth nonconvex models.
WC ASSM : W rite C onsistent
A synchronous s tochastic s ubgradient m ethod.
WC ASSM differs from
WI ASSM in its use of locks to update x k,i to make consistent writes.
AsySG-con in Lian et al.
(2015) is its counterpart for smooth nonconvex
f ( x ) and sparse x .
P ASSM :
The presented P artitioned A synchronous S tochastic S ubgradient M ethod.
We read as well as update
x k ,i lock -free asynchronously.
SGD: Sequential S tochastic G radient D escent method
For each of the methods, we adopt a decreasing step size strategy γ k,i = ( α j × γ ) / √ k , where α j > 0 is a constant for the j th processing core.
γ is ﬁxed initially.
In each of the methods we use an
L 2 penalty in form of the weight-decay of 0.0005.
Further, we use a constant momentum of 0.9 in each of them.
In all of the above methods we load the datasets in mini-batches of size 64.
Dataset and Networks.
We used CIFAR10/100 data sets of RGB images Krizhevsky (2009).
It contains 50000 labeled images for training and 10000 labeled images for testing.
We trained three different well known neural network models: VGG19 Simonyan & Zisserman (2015), ResNet34, and
Resnet50 He et al.
VGG19 is a stack of 19 convolution layers out of which 5 are followed by max-pooling.
All hidden layers contain ReLU non-linearity.
ResNets have blocked architecture that build on VGG and thereby contain enough max-pooling and ReLU.
Evidently, training of these neural networks offer general nonsmooth nonconvex optimization problems.
Asynchronous Implementation.
We implemented the asynchronous methods using the open-source
Pytorch library Paszke et al.
(2017) and the multi-processing framework of Python.
We replicate an instance of the neural network to multiple processes, spawned by a parent process, thereby each of the processes, including the parent process has its own local copy to compute the stochastic subgradients on.
The processes synchronize over the copy which the parent process owns by way of reading the model and adding to it the computed stochastic subgradients.
We used the nn.
DataParallel() module of Pytorch library to implement the sequential SGD.
System Speciﬁcation.
We benchmarked the implementations on two different system settings: (a)
M1:
A workstation with a single Intel(R) Xeon(R) CPU E5-1650 v4 containing 6 hardware cores ( logical cores with hyperthreading) running at 3.60 GHz and two Nvidia GeForce RTX 2080 Ti GPUs, and (b) M2: A VM on Google Cloud with a single CPU Intel(R)
Xeon(R) containing 8 cores ( logical cores with HT) running at 2.0 GHz and four Nvidia Tesla P4 GPUs.
On M1, we spawnedprocesses, including the main process, distributed 3 processes each on the two GPUs.
On M2, we spawned 4 processes, including the main one, leading to exclusive utilization of each of the GPUs by the individual process.
For a fair evaluation of scalability with cores, we bind the processes restricting their computations to individual CPU cores.
Each process on M1 uses two logical CPU cores to load the training and testing images whereas on M2 they use four logical cores each.
Both systems are based on Ubuntu 18.04.
Thus, we have fairly diverse asynchronous system settings.
Model Partitioning.
Unlike, P ASSM , the methods WI ASSM , WC ASSM and SGD do not partition the model and compute the stochastic subgradients over the entire computation graph of a neural network via backpropagation provided by the autograd module of Pytorch.
P ASSM partitions the list of leaves of the computation graph into blocks.
While computing the stochastic subgradients with respect to a block, we switch off the requires_grad ﬂag of the leaves corresponding to other blocks during backpropagation thereby obtaining some savings.
Notice that, this is not model parallelization and the stochastic subgradient computation with respect to a leaf depends on the computation path leading to the output.
The multi-GPU-based implementation of SGD has model replication and data partitioning over GPUs.
Thus, the computation of stochastic subgradients happen over each GPU and they are summed and added to the main copy of the neural network.
Experimental Observations.
The experimental results are presented in Figures 1, 2, and 3.
[Figure]
The training loss minimization trajectory of P ASSM almost overlaps that of SGD, ﬁg. 1(a), marginally outperforming it initially.
Whereas, those of WI ASSM and WC ASSM mutually compare at par and outperform the previous two.
Similar behavior is observed in the trajectory of training accuracy.
It shows that the communication cost across GPUs does not dominate the computation cost.
In terms of generalization test error optimization, SGD has clear edge over the other methods, whereas P ASSM underperforms others.
This can be explained as the asynchronous inconsistency added to the variance in training samples accentuates the error in testing the unlearned samples.
Summary.
The block partitioning design of P ASSM has its efﬁciency in the following: 1) it reduces the cost of optimization per node, since the parameter variables are partitioned; note that, in case of neural networks, where backpropagation requires computation of stochastic subgradients over almost the entire model notwithstanding the block partitioning, we have only marginal observable beneﬁt, however it can be signiﬁcantly better if the subgradients with respect to blocks can be computed independently; and 2) reduces memory trafﬁc and potential write conﬂicts between processors.
We
[Figure] [Figure]
and communication costs.
However, now SGD outperforms each of the asynchronous methods showing that the communication cost dominates the computation cost.
The communication cost in multiprocessing based methods is not only across the devices but also across the memory space of the processes, on which SGD implementation has advantage.
Furthermore, P ASSM performs at par with WI ASSM in terms of optimization whereas it offers a saving in terms of the time spent per epoch.
This observation infers that the partitioning method in case of larger models can be competitively attractive. observe this in terms of better convergence per unit time when compared to the asynchronous SGD variant (WI ASSM ) for a large model. And ﬁnally, it is pertinent to mention that we also observed that momentum correction improves the convergence per epoch of the block partitioning approach whose performance was way lower if we did not use it.  D ISCUSSION AND C ONCLUSION
In this paper we analyzed the convergence theory of asynchronous stochastic subgradient descent. We found that the state of the art probabilistic model on asynchronous parallel architecture applied to the stochastic subgradient method, with and without the use of momentum, is consistent with standard theory in stochastic approximation and asymptotic convergence with probability one holds for the method under the most general setting of asynchrony.
We presented numerical results that indicate some possible performance variabilities in three types of asynchrony: block partitioning inconsistent read (for which the above convergence theory applies), full-variable-update consistent write (for which the above convergence theory also applies), and full-variable-update inconsistent read/write (for which no convergence theory exists).
A PPENDIX A: G LOBAL S HARED M EMORY A SYNCHRONOUS S TOCHASTIC S UBGRADIENT M ODEL
Here we give a few more details describing the relation of the probabilistic model of asynchrony to the underlying hardware properties, as modeled in Cannelli et al..
In this section, we present k as a global counter, indicating sequential updates of any block among the variables.
In iteration k , the updated iterate x k +
i k depends on a random vector ζ k (cid:44)
( i k , d k , ξ k ) .
The distribution of ζ k depends on the underlying scheduling or message passing protocol.
We use the following formulation, which applies to a variety of architectures.
Let ζ 0: t (cid:44) ( ζ 0 , ζ 1 , ..., ζ t ) be the stochastic process representing the evolution of the blocks and minibatches used, as well as the iterate delays.
The σ -algebra F is obtained as follows.
Let the cylinder C k ( ζ 0: t ) (cid:44) { ω ∈ Ω : ω 0:
k
= ζ 0: t } and deﬁne F k (cid:44) σ ( C k ) and F (cid:44) σ ( ∪ ∞ t =0 C t ) the cylinder σ -algebra on Ω .
Consider the conditional distribution of ζ k +1 given ζ 0: k ,
[Equation]
) , we have the following assumptions on the probabilities of block selection and the delays,
Assumption 6.1.
The random variables ζ k satisfy,
There exists a δ such that d kj ≤ δ for all j and k .
Thus each d kj ∈ D (cid:44) { 0 , ..., δ } n . 2.
For all i and ζ 0: k − 1 such that p ζ 0: k − 1 ( ζ 0: k − 1 ) > 0 , it holds that, (cid:88) d ∈D P (( i, d, ξ ) | ζ 0: k − 1 ) ≥ p min for some p min > 0 . It holds that, P (cid:18)(cid:26) ζ ∈ Ω : lim inf k →∞
[Equation]
( ζ [Equation]
[Equation]
1 ) > 0 (cid:27)(cid:19) =
The ﬁrst condition indicates that there is some maximum possible delay in the vectors, that each element of x used in the computation of x k +
i
k is not too old.
The second is an irreducibility condition that there is a positive probability for any block or minibatch to be chosen, given any state of previous realizations of { ζ k } .
The last assumption indicates that the set of events in Ω that asymptotically go to zero in conditional probability are of measure zero.
In order to enforce global convergence, we wish to use a diminishing step-size.
However, at the same time, as synchronization is to be avoided, there must not be a global counter indicating the rate of decrease of the step-size.
In particular, each core will have its own local step size γ ν ( c k ,k ) where c k is the core, and, deﬁning the random variable Z k as the component of { 1 , ..., ¯ c } that is active at iteration k , the random variable denoting the number of updates performed by core c k , denoted by ν ( k ) is given by ν ( k ) (cid:44) (cid:80) kj =
I ( Z j = c k ) .
In addition, noting that it has been observed that in practice, partitioning variable blocks across cores is more efﬁcient than allowing every processor to have the ability to choose across every variable block (Liu & Wright, 2015).
Thus we partition the blocks of variables across cores.
We can thus denote c k as being deﬁned uniquely by i k , the block variable index updated at iteration k .
Note that lim inf k →∞ γ ν ( ck,k ) k
= 0 in probability is implied by
[Equation]
for some subsequence, which is antithetical to Assumption 3.1, Part 2.
Thus, note that the stepsizes γ ν ( c k ,k ) satisfy, where the limit of the sequence is taken in probability,
[Equation]
which is an assumption for the analysis of asynchronous parallel algorithms in Borkar (2008).
We are now ready to present Algorithm 2.
This is presented from the "global" iteration counter perspective.
Input: x 0 .
1: while Not converged and k < k max do 2:
Having realized ζ 0: k − 1 , sample { ζ k =
( i k , d k , ξ k ) }| ζ 0: k − 1 } .
Update u i k
= mu
i k
+ g (( x d k 1 1 , x d k 2 2 , ..., x d kn n ) , ξ k ) 4:
Update x k
+
i
k
= x ki k − (1 − m ) γ ν ( k ) u i k 5:
Set k
=
k
+6: end while
A PPENDIX B: P RELIMINARY A SSUMPTIONS AND L EMMAS
Lemma 7.1.
It holds that { Y k,i , Y σk,i ; k, i } is uniformly integrable.
Thus, so is
[Equation]
Proof.
Uniform integrability of { Y k,i , Y σk,i ; k, i } follows from Assumption 3.2, part 3.
The uniform integrability of (cid:110)(cid:80) kj =1 m k − j Y j,i , (cid:80) kj =
m k − j Y
σj,i ; k, i (cid:111) follows from 0 < m < 1 and the fact that a geometric sum of a uniformly integrable sequence is uniformly integrable.
Lemma 7.2.
It holds that, for any K > 0 , and all l ,
[Equation]
in probability as σ → ∞ .
Proof.
As σ → ∞ , by the deﬁnition of γ σk,i , γ σk,i → 0 and since by Assumption 3.1 max d ki ≤ δ , for all k < K , (cid:80) kj = k − [ d ki ] l γ σj,i ≤ δγ σk − δ,i → 0 .
Now we deﬁne some terminology arising in the theory of weak convergence.
We present a result indicating sufﬁcient conditions for a property called tightness.
Theorem 7.1.
(Kushner & Yin, 2003, Theorem 7.3.3) Consider a sequence of processes { A k ( · ) } with paths in D ( −∞ , ∞ ) such that for all δ > 0 and each t in a dense set of ( −∞ , ∞ ) there is a compact set K δ,t such that,
[Equation]
and for any T > 0 ,
[Equation]
then { A n ( · ) } is tight in D ( −∞ , ∞ ) .
If a sequence is tight then every weak sense limit process is also a continuous time process.
We say that A k ( t ) converges weakly to A if,
E [ F ( A k ( t ))]
→ E [ F ( A ( t ))] for any bounded and continuous real-valued function F ( · ) on R n .
Weak convergence is deﬁned in terms of the Skorohod topology, a technical topology weaker than the topology of uniform convergence on bounded intervals, deﬁned in Billingsley (1968).
Convergence of a function f n ( · ) to f ( · ) in the Skorohod topology is equivalent to uniform convergence on each bounded time interval.
We denote by D j [0 , ∞ ) the j -fold product space of real-valued functions on the interval [0 , ∞ ) that are right continuous with left-hand limits, with the Skorohod topology.
It is a complete and separable metric space.
Much of the proof of the main Theorem can be taken from the analagous result in Chapter 12 of Kushner & Yin (2003), which considers a particular model of asynchronous stochastic approximation.
As we introduced a slightly different model from the literature, some of the details of the procedure are now different, and furthermore we introduced momentum to the algorithm, and thus in the next section we indicate how to treat the distinctions in the proof and show that the result still holds.
A PPENDIX C: P ROOF OF T HEOREM
By Theorem 8.6, Chapter 3 in Ethier & Kurtz (2009) a sufﬁcient condition for tightness of a sequence { A n ( · ) } is that for each δ > 0 and each t in a dense set in ( −∞ , ∞ ) , there is a compact set K δ,
t such that inf n P
[ A n ( t ) ∈
K δ,t ] ≥ 1 − δ and for each positive T , lim δ → 0 lim sup n sup | τ |≤ T,s ≤ [Equation] E [ | A n ( τ + s ) − A n ( τ ) | ] = 0 .
Now since Y k,i is uniformly bounded, and Y σk,i ( · ) is its interpolation with jumps only at t being equal to some T k,i , it holds that for all i ,
[Equation]
and so by the deﬁnition of the algorithm,
[Equation]
which implies,
[Equation]
and the same argument implies tightness for { τ σi ( · ) , N σi ( · ) } by the uniform boundedness of { δτ σi,k } and bounded, decreasing γ σk,i and positive u σk,i ( x, ψ σk +1 ,i ) , along with Assumption 3.4.
Lipschitz continuity follows from the properties of the interpolation functions.
Speciﬁcally, the Lipschitz constant of x σi ( · ) is B g .
All of these together imply tightness of ˆ x σi ( · ) as well.
Thus,
[Equation]
is tight in D 4 n [0 , ∞ ) .
This implies the Lipschitz continuity of the subsequence limits with probability one, which exist in the weak sense by Prohorov’s Theorem, Theorems 6.1 and 6.2 (Billingsley, 2013).
As σ → ∞ we denote the weakly convergent subsequence’s weak sense limits by,
[Equation]
Note that,
[Equation]
For more details, see the proof of Kushner & Yin (2003, Theorem 8.2.1) [Equation]
Let,
[Equation]
Now for any bounded continuous and real-valued function h ( · ) , an arbitrary integer p
[Equation]
and t and τ , and s j ≥ t real, we have [Equation] τ ) [Equation]
ˆ
[Equation]
˜ G
[Equation]
( t [Equation] ˜ G [Equation]
( t
[Equation]
Now the term involving M σ equals zero from the Martingale property.
The term involving B σ is zero due to Assumption 3.4.
We now claim that the term involving ˜ G σi goes to zero as well.
Since x σk,i → x σi it holds that, by Lemma 7.2, ( x σk − [ d ki ] 1 , 1 ( t ) , ..., x σk − [ d ki ] j ,j ( t ) , ..., x σk − [ d ki ] N ,N ) →
ˆ x σ ( t ) as well.
By the upper semicontinuity of the subgradient, it holds that there exists a g
i
(ˆ x σi ( t ))
∈
∂
i f (ˆ x σi ( t ))
such that
[Equation]
as σ → ∞ .
Thus each term in the sum converges to g i (ˆ x σk − j ( t )) .
Now, given j , as k → ∞ , the boundedness assumptions and stepsize rules imply that g i (ˆ x σk − j ( t ))
→ g i (ˆ x σk ( t )) .
On the other hand as k → ∞ and j → ∞ , m j g i (ˆ x σk − j ( t )) → 0 .
Thus (cid:80) kj =
m j g
i
(ˆ x σk − j ( t ))
→ 1 − m k 1 − m g i (ˆ x σk ( t ))
→ 1 1 − m g i (ˆ x σk ( t )) , and the claim has been shown.
Thus the weak sense limit of lim σ →∞ W σ i ( · ) =
W i ( · ) satisﬁes
[Equation]
and thus by Kushner & Yin (2003, Theorem 7.4.1) is a martingale and is furthermore a constant with probability one by the Lipschitz continuity of x by Kushner & Yin (2003, Theorem 4.1.1).
Thus,
[Equation]
where g (ˆ x ( s )) ∈ ∂f (ˆ x ( s )) , and the conclusion holds.
C ONVERGENCE W ITH P ROBABILITY O NE
The previous Theorem showed that under the conditions described for the algorithm, there is a weakly convergent subsequence to an invariant set.
We can now use the results in Dupuis & Kushner (1989) to infer from weak convergence, probability one convergence of the sequence of iterates.
For this, we shall use the machinery developed in Dupuis & Kushner (1989), which establishes conditions for which a weakly convergent stochastic approximation algorithm approximating a continuous ODE converges with probability one, under certain conditions.
One can study the proof structure to quickly reveal that with minor modiﬁcations the results carry through.
In particular, when ˙ b appears in the proof, one can replace it with an element of the differential inclusion, and the limit point is replaced by the invariant set.
Assumption 2.1 in Dupuis & Kushner (1989) is now associated with a set-valued map S ( x, T, φ ) , and by the noise structure of the assumptions, it can easily be seen that ¯ L exists for all possible values of x , T and φ in the notation of the paper.
One can see that the uniqueness appears once in the beginning of the proof of Theorem 3.1 with the existence of this Tsuch that the trajectory lies in a speciﬁc ball around the limit point for t ≥ T 1 .
This can be replaced by the trajectory lying in this ball around the invariant set, for T 1 deﬁned as the supremum of such ˆ
T 1 associated with every possible subgradient, i.e., element of the DI.
Since the subgradient is a compact set and is upper semicontinuous, this supremum exists.
Finally, note that Assumption 3.2 is as Assumption 4.1 in Dupuis & Kushner (1989) and thus similarly implies Theorem 4.1 and Theorem 5.3.
This proves that as σ → ∞ , w.p.1 x σ ( · ) converges to an invariant set of the differential inclusion.