Paper:
A Posteriori Error Bounds for Joint Matrix Decomposition Problems
Joint matrix triangularization is often used for estimating the joint eigenstructure of a set M of matrices, with applications in signal processing and machine learning. We consider the problem of approximate joint matrix triangularization when the matrices in M are jointly diagonalizable and real, but we only observe a set M' of noise perturbed versions of the matrices in M. Our main result is a first-order upper bound on the distance between any approximate joint triangularizer of the matrices in M' and any exact joint triangularizer of the matrices in M. The bound depends only on the observable matrices in M' and the noise level. In particular, it does not depend on optimization specific properties of the triangularizer, such as its proximity to critical points, that are typical of existing bounds in the literature. To our knowledge, this is the first a posteriori bound for joint matrix decomposition. We demonstrate the bound on synthetic data for which the ground truth is known.

Review:
The paper addresses the problem of approximate non-orthogonal joint diagonalization (NOJD) of several matrices, where each matrix is of the form $V_o \Lambda_n V_o^{-1}$ plus some noise. Such problem appears, e.g., when performing estimation in some latent variable models via the method of moments. The authors approach this NOJD problem via related simultaneous Schur decomposition (SSD), which is formulated as a non-convex optimization problem (OP).  The main result of this paper is an a posteriori bound (which depends solely on a posteriori quantities). The only a priori quantity appearing in the bound is the noise parameter $\sigma$, which the authors assume to be known (see lines 57-59). For the obtained bound, an approximate joint triangularizer (the output of an algorithm for OP) is not required to be neither a global solution nor a critical point and the bound depends on the respective objective value. This bound is the first a posteriori bound for the SSD problem and, more generally, for joint matrix decomposition problems.
Update of the review: I read the rebuttal and other reviews and argue for acceptance of the paper. I would ask the authors to pay attention to the application of the Weyl's theorem which holds true only for Hermitian matrices.  The main result of the paper -- an a posteriori bound for the SSD problem -- and the techniques used to obtain the result are novel both in the context of the SSD problem and in the context of joint matrix decomposition problems. Since joint matrix decomposition methods are known to work well in practice for computing tensor factorizations, this result can be a step forward in understanding this empirical observation. Therefore, the contributions of this paper are important and can be of interest to a certain community.   The paper, however, is not written in a reader friendly way. I especially disagree with the way the authors motivate the problem since it looks like confusing several different matrix joint decomposition problems (see the detailed explanation in the next paragraph), which (a) can easily be interpreted in a wrong way and (b) can look like hiding some important assumptions. Therefore, I would insist on rewriting the introductory section in a clearer way. Moreover, some part of the proof of Theorem 1 (the main result; see the detailed discussion below) are subtle and have to be clarified. Given the authors successfully fix the mentioned problems, I would support the acceptance of the paper.  The citations are not in accordance with the NIPS template (Authors, year format vs [number]).  ---  *Introduction.* One of my points of criticism is how the problem is motivated, e.g., in lines 24-26. Basically, there are two different types of simultaneous diagonalization problems: problem (1) in lines 19-20 and joint diagonalization by congruence (see comment #1). The latter is one of several techniques to obtain *symmetric* tensor CP decomposition, which, e.g., naturally arises in estimation for some latent variable models (as in the cited papers by Anandkumar et al, Balle et al, Kuleshov et al where the considered matrices are tensor contractions/projections or slices). The former naturally arises, e.g., in (De Lethauwer et al, 2004; section 5) as a technique to obtain *non-symmetric* tensor CP decomposition. One can transform the matrices in joint diagonalization by congruence to problem (1) (see comment #2). This, however, requires some additional assumptions (invertibility of $V_o$?) and poses some additional questions, e.g., whether this transformation simplifies the original problem or instead makes it more complicated. These differences should be explicitly addressed in the paper given the authors want to motivate the problem with applications discussed in Anandkumar, et al, Balle et al, Kuleshov et al. As an alternative or in addition, the authors could motivate the problem with applications which boil down to a non-symmetric CP decomposition (as, e.g., discussed in De Lethauwer et al, 2004; [4] would be a more recent machine learning reference). In any case, the differences, assumptions, and connections between the problems have to be explicitly addressed.  Comments: #1: There are two types of simultaneous diagonalization problems: the problem in (1) and joint diagonalization by congruence, where the matrices have the form $V_o \Lambda_n V_o^{\top}$, i.e. the last matrix goes with the *transpose* instead of the *inverse*. The two problems are equivalent only if $V_o$ is orthogonal and then it is the problem of orthogonal joint diagonalization considered, e.g., by Cardoso and Souloumiac (1996). The setting where $V_o$ is orthogonal is, however, not the main setting in the paper being reviewed.  #2: In joint diagonalization by congruence the matrices are in the form $V_o \Lambda_n V_o^{\top}$. Assuming $V_o$ and say j-th matrix are invertible, multiply i-th matrix by the inverse of the j-th matrix to obtain a matrix $V_o \Lambda_i \Lambda_j^{-1} V_o^{-1}$ in the form suitable for problem (1) (see, e.g., De Lethauwer et al, 2004, section 5).  This being said, we recall that the perturbation analysis is only obtained for the SSD problem, but not for related tensor factorization problems. Do the results of the analysis easily extend to this case?  ---  *The proof of Theorem 1.* I have several questions regarding the proof: - The definition (line 212) of the quantity $T_{\beta}$ is not clear and, possibly wrong (see comment #4). This definition does not influence the overall result significantly, but this definition has to be clarified. - Is it obvious that min and max can be exchanged in (29)? (see comment #5) - Lemma 2 is provided without proof and I can not check whether the result is correct, esp., given the uncertainty in the definition of $T_{\beta}$. - The expansion in Eq. (32) does not obviously follow from the Weyl's theorem applied to the singular values of the matrices in line 224. The expression is in therms of the condition numbers ($kappa$?) of $V$ and $\hat{V}$ and not the singular values of $M$ and $\hat{M}$.   Comments: #4: In definition of the quantity $T_{\beta}$ in line 212, it not clear what Low means. Assuming the first Low refers to the lower triangular part of a matrix, the LHS of (25) is not equal to the LHS of (24). E.g., $low( M_{\beta} \otimes 1 ) vec(X)$ is actually the zero matrix and $low( 1 \otimes M_{\beta}^{\top} ) vec(X)$ is not equal to neither $vec( low( low(X) M_{\beta} ) )$ nor $vec( low( M_{\beta} low(X) ) )$. Obvious modifications of this result do not seem to help. Clearly, there is a matrix $T_{\beat}$ can be expressed via $M_{\beta}$, but the question is how simple / difficult this expression is? #5: The problem is that the sets in min and max are not convex and compact. Is it obvious to show that these constraints can be relaxed to some convex and compact sets such that the solution is tight? I can see that the min problem can be reformulated as minimization over the simplex, but the objective function is not linear in $\beta$ anymore (an upper envelope of piecewise linear functions) and, hence, it does not seem to be straightforward to relax the unit sphere to the unit ball for $\beta$. Is there a different explanation? If not, does it help that by weak duality the first equality in (29) can always be replaced by the less or equal sign.  ---  Other less important questions and comments:  I cannot check the correctness of Lemma 1 since the proof was not provided. However, the results of the lemma are not used in other proofs, so I see it as a complimentary result. Lemma 1 provides a kind of identifiability results for the SSD problem. Is there some intuition behind these results (e.g., analogy to permutation and scaling unidentifiability in the joint diagonalization by congruence case)?  After introducing problem (1), the authors may consider mentioning that this problem was already addressed in the literature, e.g., in [1,2,3] (see below) or by De Lethauwer et al (2004). However, papers [1,2,3] do not provide theoretical analysis and the first order perturbation analysis is from section 6 of De Lethauwer et al (2004) is weaker than the results of this paper. One could also add an explanation of these differences.  In lines 38-39, reference to Cardoso and Souloumiac (1996) is not appropriate since their optimization is also over the manifold of orthogonal matrices. The related optimization problems which do not involve the manifold of orthogonal matrices are, e.g., the ones by Afsari (2006) or Kuleshov et al (2015) or [9]. Moreover, the statement in lines 38-39 is a bit vague since the problems have different objective functions.  In lines 40-41, one could add references, e.g., to De Lethauwer et al (2004) and [6].  The known in the literature bounds (Cardoso, 1994; Afsari, 2008; Kuleshov et al, 2015) are obtained for joint diagonalization by congruence and, therefore, are formally different from the SSD problem considered in this paper. On the contrary, an a priori bound for the SSD problem was recently obtained [6] and the authors may consider to cite this.   List of typos/comments: 1) One should define the set GL(d) in equation (5). 2) It could be useful to specify which matrix norm is used, esp. in the proof of Theorem 1. 3) In the transition between equations (22) and (23), it worth mentioning that the expansion $e^{\alpha X} = 1 + \alpha X + O(\alpha^2)$ is used. 4) In equations (24), (25) and further the term $O( (\alpha + \sigma)^2 )$ is lost (either explain why it is omitted or put it back in). 5) In equation (28), the rightmost $\sigma$ has to be squared. 6) ``R'' is missing in the sum sign in line 223. 7) It would be useful to define $\kappa(\cdot)$ and $\sigma_{max/min}(\cdot)$ appearing in the proof - the condition number and the singular values? 8) In line 230, "min" and "max" are probably mixed up. 9) Is there some intuition behind the identifiability results in Lemma 1 (like permutation and scaling)? 10) Comment to line 132: there is only one way to obtain a slice of a tensor, but the matrices which are to be diagonalized can be constructed in different algebraic form. 11) In lines 158=159, the whitening procedure does not necessary lead to poor performance (and this is what the cited paper says). Moreover, it can be (and often is) used for dimensionality reduction.   ---  References: [1] ``Joint eigenvalue decomposition using polar matrix factorization'' by Luciani and Albera from 2010;  [2] ``A new Jacobi-like method for joint diagonalization of arbitrary non-defective matrices'' by Iferroudjene, Abed Meraim, and Belouchrani from 2009;  [3] ``Simultaneous diagonalization with similarity transformation for non-defective matrices'' by Fu and Gao from 2006; [4] ``Beyond CCA: moment matching for multi-view models'' by Podosinnikova, Bach, and Jacoste-Julien from 2016; [5] ``Nonorthogonal joint diagonalization by combining Givens and hyperbolic rotations'' by Souloumiac from 2009; [6] ``Tensor decomposition via joint matrix Schur decomposition'' by Colombo and Vlassis from 2016;