Elastic Bulk Synchronous Parallel Model for Distributed Deep Learning The bulk synchronous parallel (BSP) is a celebrated synchronization model for generalpurpose parallel computing that has successfully been employed for distributed training of machine learning models. A prevalent shortcoming of the BSP is that it requires workers to wait for the straggler at every iteration. To ameliorate this shortcoming of classic BSP, we propose ELASTICBSP a model that aims to relax its strict synchronization requirement. The proposed model offers more flexibility and adaptability during the training phase, without sacrificing on the accuracy of the trained model. We also propose an efficient method that materializes the model, named ZIPLINE. The algorithm is tunable and can effectively balance the tradeoff between quality of convergence and iteration throughput, in order to accommodate different environments or applications. A thorough experimental evaluation demonstrates that our proposed ELASTICBSP model converges faster and to a higher accuracy than the classic BSP. It also achieves comparable (if not higher) accuracy than the other sensible synchronization models.

Processing graphs with barrierless asynchronous parallel model on sharedmemory systems Abstract   Graph analytics systems rely on parallel models to realize parallel processing of largescale graphs, in which the most extensively used models are the bulk synchronous parallel (BSP) and the asynchronous parallel (AP) model. The BSP model is simple and deterministic but has difficulties in expressing some graph processing algorithms, while the AP model is flexible to handle various kinds of algorithms but behaves inefficiently on memoryintensive algorithms due to higher scheduling overhead and internal global barriers. Additionally, although the matrix computation has demonstrated its highperformance characteristic in graph processing, most matrixbased systems only support the BSP model. To improve efficiency of the AP model for graph analytics, this paper proposes a barrierless AP model which uses matrix operations to process graphs and removes the global barrier in each superstep of the AP model to accelerate the execution progress of algorithms. The proposed model is evaluated by comparing it with both AP and BSPbased systems including GraphLab, GRACE, and GraphMat. Experimental results show that our model behaves better than the asynchronous systems, and even outperforms the BSPbased system on some memoryintensive graph algorithms as well as computationintensive graph algorithms.