Paper:
Parallel Direction Method of Multipliers
We consider the problem of minimizing block-separable (non-smooth) convex functions subject to linear constraints. While the Alternating Direction Method of Multipliers (ADMM) for two-block linear constraints has been intensively studied both theoretically and empirically, in spite of some preliminary work, effective generalizations of ADMM to multiple blocks is still unclear. In this paper, we propose a parallel randomized block coordinate method named Parallel Direction Method of Multipliers (PDMM) to solve optimization problems with multi-block linear constraints. At each iteration, PDMM randomly updates some blocks in parallel, behaving like parallel randomized block coordinate descent. We establish the global convergence and the iteration complexity for PDMM with constant step size. We also show that PDMM can do randomized block coordinate descent on overlapping blocks. Experimental results show that PDMM performs better than state-of-the-arts methods in two applications, robust principal component analysis and overlapping group lasso.

Review:
This paper considers the problem of minimizing block-separable functions subject to linear constraints. A variant of ADMM (called PDMM) is proposed to deal with the multiple blocks case in this paper: it randomly selects K blocks of the primal variable to update and uses a backward step to refine the dual variable; the process to update the primal variable can be implemented in parallel. The sublinear convergence rate is proved for the proposed method. This method generalized several variants of ADMM.This paper is well written and easy to follow. I am positive to accept this paper. My questions and comments are given below:1) The algorithm description in (5) and (6) is not clear enough. Do you update all coordinates of y^{t+1} and \hat{y}^{t+1} or just a single block "i"? My understanding is that you only update a block of the dual variable "y", but I did not find how to select "i". This should be clarified in the revision.2) In Theorem 2, I believe that "\sum_{i=1}^I" was missing behind of "{" from your proof.3) More discussion about Theorem should be included. Theorem 2 (the main result in this paper) provides the convergence rate of the proposed method PDMM. First, PDMM generalizes several variants of ADMM. A natural question is if Theorem 2 (by properly choose parameters like K, J) is consistent with the convergence rates of those variants. Second, since you split the transformation matrix A into multiple blocks, what is the optimal way to split it from your theorem?4) I am curious of the comparison on the tensor completion problem ("tensor completion for estimating missing values in visual data, 2012"), which also has the multiple block structure. Do you have any clue which variant of ADMM is optimal?    Minors:1) In (9), a space is missing behind of "min"2) Line 217, remove the space before "Section"