Paper:
Orthogonal NMF through Subspace Exploration
Orthogonal Nonnegative Matrix Factorization (ONMF) aims to approximate a nonnegative matrix as the product of two k-dimensional nonnegative factors, one of which has orthonormal columns. It yields potentially useful data representations as superposition of disjoint parts, while it has been shown to work well for clustering tasks where traditional methods underperform. Existing algorithms rely mostly on heuristics, which despite their good empirical performance, lack provable performance guarantees. We present a new ONMF algorithm with provable approximation guarantees. For any constant dimension k, we obtain an additive EPTAS without any assumptions on the input. Our algorithm relies on a novel approximation to the related Nonnegative Principal Component Analysis (NNPCA) problem; given an arbitrary data matrix, NNPCA seeks k nonnegative components that jointly capture most of the variance. Our NNPCA algorithm is of independent interest and generalizes previous work that could only obtain guarantees for a single component. We evaluate our algorithms on several real and synthetic datasets and show that their performance matches or outperforms the state of the art.

Review:
The paper proposes a subspace exploration algorithm for the problems of orthogonal NMF and nonnegative PCA (which are tightly related). According to the authors, this is the first algorithm that comes with performance guarantees that are characterized by a tradeoff between the approximation error and the computational complexity. Although the complexity scales exponentially with some user controlled parameters (order of low rank approximation, and target rank), simulations indicate good performance in practice.Quality: The paper is technically sound, with proofs about the main theorems, discussion about their shortcomings (exponential complexity), and simulations to establish state of the art results in practice. Clarity: The paper is mostly clearly written. I think one clarifications that is important but is missing is the following: Algorithm 3 LocalOptW runs with complexity O(2^k*k*m). Naively, for Algorithm 1, algorithm 3 has to be run for every choice of C. But the number of choices for C is also exponentially large (the authors should state explicitly how large it is). This would make the whole algorithm even more computationally hard. However the authors argue (footnote 2) that when the two algorithms are combined (as in the application shown in this paper), Alg 3 can be simplified into a O(k*m) procedure. While intuitive, the authors should make explicitly clear how this is done, and perhaps re-write algs 1 and 3 to demonstrate that. Originality: To the best of my knowledge the approach is original.Significance: The paper establishes both theoretical bounds (for the first time), and state of the art results through simulations. I'd like to see a comparison (or a discussion) of the various different methods, also with respect to the run-time of the various algorithms. The complexity bound O((1/\epsilon)^(1/\epsilon)) can grow rapidly, so it'd be nice to see that the gained performance does not come at a terrible computational cost. Since ONMF and NNPCA are largely equivalent, why does the proposed method perform better than EM for NNPCA (table 1) but worse than EM for ONMF (table 2) for the same dataset MFEAT PIX? Is there something that I miss here?Typo: last paragraph of the results section, Table 1 should be Table 2 (I think)