On the Convergence Speed of AMSGRAD and Beyond In ICLRu0027s (2018) best paper On the Convergence of Adam and Beyond, the author points out the shortcomings in Adamu0027s convergence proof, proposes an AMSGRAD algorithm that can guarantee convergence as the number of iterations increases. However, through some comparative experiments, this paper finds that there are two problems in the convergence process of AMSGRAD algorithm. Firstly, the AMSGRAD algorithm is easy to oscillate; Secondly, the AMSGRAD algorithm converges slowly. After analysis, the above two problems can be solved by the following ways. When gt1gt u003e 0, this paper adds the momentum term in Momentum algorithm to the AMSGRAD algorithm to accelerate convergence. When gt1gt xe2x89xa4 0, this paper use SGD algorithm instead of AMSGRAD algorithm to update the model weights. In order to eliminate some negative effects of the previous parameter gradient on the current parameter gradient and reduce the oscillation amplitude of the objective function, the firstorder and secondorder moment estimations of the parameter gradient are recalculated when gt1gt xe2x89xa4 0. Therefore, this paper proposes the ACADG algorithm, which not only can improve the convergence speed, suppress the oscillation amplitude of the objective function, but also can improve the accuracy of training and test data sets.

Fast Classification Algorithms via Distributed Accelerated Alternating Direction Method of Multipliers Distributed machine learning has gained lots of attention due to the rapid growth of data. In this paper, we focus regularized empirical risk minimization problems, and propose two novel Distributed Accelerated Alternating Direction Method of Multipliers (DA2DM2) algorithms for distributed classification. Based on the framework of Alternating Direction Method of Multipliers (ADMM), we decentralize the distributed classification problem as a global consensus optimization problem with a series of subproblems. In DA2DM2, we exploit ADMM with variance reduction for subproblem optimization in parallel. Taking global update and local update into consideration respectively, we propose two acceleration mechanisms in the framework of DA2DM2. In particular, inspired by Nesterovu0027s accelerated gradient descent, we utilize it for global update to further improve time efficiency. Moreover, we also introduce Nesterovu0027s acceleration for local update, and develop the corrected local update and symmetric dual update to accelerate the convergence with only a little change in the computational effort. Theoretically, DA2DM2 has a linear convergence rate. Empirically, experimental results show that DA2DM2 converge faster than existing distributed ADMMbased classification, and could be a highly efficient algorithm for practical use.