signADAM: Learning Confidences for Deep Neural Networks In this paper, we propose a new firstorder gradientbased algorithm to train deep neural networks. We first introduce the sign operation of stochastic gradients (as in signbased methods, e.g., SIGNSGD) into ADAM, which is called as signADAM. Moreover, in order to make the rate of fitting each feature closer, we define a confidence function to distinguish different components of gradients and apply it to our algorithm. It can generate more sparse gradients than existing algorithms do. We call this new algorithm signADAM. In particular, both our algorithms are easy to implement and can speed up training of various deep neural networks. The motivation of signADAM is preferably learning features from the most different samples by updating large and useful gradients regardless of useless information in stochastic gradients. We also establish theoretical convergence guarantees for our algorithms. Empirical results on various datasets and models show that our algorithms yield much better performance than many stateoftheart algorithms including SIGNSGD, SIGNUM and ADAM. We also analyze the performance from multiple perspectives including the loss landscape and develop an adaptive method to further improve generalization. The source code is available at https:github.comDongWanginxdusignADAMLearnbyConfidence.

A Weighted Aggregating SGD for Scalable Parallelization in Deep Learning We investigate the stochastic optimization problem and develop a scalable parallel computing algorithm for deep learning tasks. The key of our study involves a reformation of the objective function for the stochastic optimization in neural network models. We propose a novel update rule, named weighted aggregating stochastic gradient decent, after theoretically analyzing the characteristics of the newly formalized objective function. The new rule introduces a weighted aggregation scheme based on the performance of local workers and does not require a center variable. It assesses the relative importance of local workers and accepts them according to their contributions. Our new rule also allows the implementation of both synchronous and asynchronous parallelization and can result in varying convergence rates. For method evaluation, we benchmark our schemes against the mainstream algorithms, including the elastic averaging SGD in training deep neural networks for classification tasks. We conduct extensive experiments on several classic datasets, and the results confirm the strength of our scheme in accelerating the training of deep architecture and scalable parallelization.