Paper: Sharp results for NIEP and NMF (ICLR-2024)

Reviewer comment (Theory):
Based on my understanding of Schur decomposition, it seems that a nonnegative matrix \( A \) with a given spectrum can be expressed as \( A = U(\Lambda+V)U^\top \), where \( U \) is orthogonal, and \( V \) is upper triangular.

Author response:
**Response to your main point**.   We thank you for a very positive feedback for our paper.  We are especially 
glad that you think we have made a significant advancement for the NIEP problem,  particularly in challenging cases where the size of the matrix is large. You have raised the following main point. 
- I apologize for my limited familiarity with the solvability of the NIEP and NMF problems. I have a straightforward question: Based on my understanding of Schur decomposition, it seems that a nonnegative matrix  with a given spectrum can be expressed as 
$A = U (\\Lambda + V) U'$,  where $U$ 
 is orthogonal, and $V$ 
 is upper triangular. If this is the case, does the solvability of the nonlinear system 
 $U (\\Lambda + V) U' \\geq 0$ 
 offer a potential approach to address the NIEP problem from an optimization perspective?

Here is our response.  Yes, you are right, but this is a non-convex problem: the set of 
$K \times K$ orthogonal matrices (which is called the Stiefel manifold in the literature) 
is non-convex, and the optimization problem is also non-convex, so computationally, 
this is a challenging problem. For example, we may put an $\\epsilon$-net on the Stiefel manifold (so that for any point on the Stiefel manifold, there is a point in the net within a distance of $\\epsilon$) and only 
check your inequality when $U$ is a net-point.   However, such an $\\epsilon$-net will have at least 
$C(1/\\epsilon)^{O(K^2)}$ points, so this approach is computationally infeasible for relatively large $K$.

Reviewer comment (Writing):
Very little exposition is given to clearly stating the result the paper shows (for example, in a clean theorem statement).

Author response:
Response to your main point 2. Thanks for your valuable comment.  We have revised several places in Section 1 to improve literature review, overview of our results, and comparison with existing literature.  To further clarify, 
our main results are as follows. 
- The orthodox NIEP concerns the following problem:  given $K$ numbers $\sigma_1,\sigma_2,\ldots, \sigma_K$, when do we have a $K \times K$ non-negative matrix $M$ with $\sigma_1, \sigma_2, \ldots, \sigma_K$ as its eigenvalues?   
The existing literature on the NIEP have been largely focused on general $(\sigma_1, \sigma_2 \ldots, \sigma_K)$ but 
small $K$ (e.g., $K \leq 4$) (e.g., see the two survey papers we added  in the first paragraph of 
the introduction). Especially, the orthodox NIEP and all of the subproblems we mentioned in the first 
paragraph of our paper remain unsolved for $K \geq 5$.  See Page 1 of the paper for details.  
- Since the orthodox NIEP is too hard (and people can only crack the case for $K \leq 4$ in the past  70 years),  it is of interest to focus on some special cases (especially those motivated by real applications),  where the results may be more fruitful.  
- Motivated by the NMF problem and network modeling, we  focus on a special case where 
$(\sigma_1,\sigma_2, \ldots, \sigma_K) = (1 + a_{K, m}, 1, \ldots, 1, -1, \ldots, -1)$  but $K$ is arbitrary. Here,  $1 \leq m \leq (K-1)$, and $a_{K,m} = \max\{0, 2m-K\}$. 
We show that for $9$ different cases (S1)-(S9), the NIEP is solvable.  For example, one of these cases is $K$ is a multiple of $4$ and $m$ is arbitrary; we show that in this case the NIEP is always solvable.   We also identify two cases (N1)-(N2)  and proved that the NIEP is not solvable in these cases. Table 1 in Section 3 summarizes our results for all pairs of $(K, m)$ with $K \leq 20$ (but 
our theorems are for general $(K, m)$).  
- Moreover, using the results on NIEP and the connection  between NMF and NIEP in Section 1.1,  we derive sharp results for NMF in Section 4. There is a rich literature on solving general NMF problems of the form $V \approx W H$, e.g. [1] and the ensuing work, but they cannot be directly applied to our NMF problem (2), which has a more constrained factorization. We will add these discussion to the paper.

Reviewer comment (Novelty):
Very little exposition is given to explaining the connection of the result to related or relevant works to this paper.

Author response:
Response to your main point 2. Thanks for your valuable comment.  We have revised several places in Section 1 to improve literature review, overview of our results, and comparison with existing literature.  To further clarify, 
our main results are as follows. 
- The orthodox NIEP concerns the following problem:  given $K$ numbers $\sigma_1,\sigma_2,\ldots, \sigma_K$, when do we have a $K \times K$ non-negative matrix $M$ with $\sigma_1, \sigma_2, \ldots, \sigma_K$ as its eigenvalues?   
The existing literature on the NIEP have been largely focused on general $(\sigma_1, \sigma_2 \ldots, \sigma_K)$ but 
small $K$ (e.g., $K \leq 4$) (e.g., see the two survey papers we added  in the first paragraph of 
the introduction). Especially, the orthodox NIEP and all of the subproblems we mentioned in the first 
paragraph of our paper remain unsolved for $K \geq 5$.  See Page 1 of the paper for details.  
- Since the orthodox NIEP is too hard (and people can only crack the case for $K \leq 4$ in the past  70 years),  it is of interest to focus on some special cases (especially those motivated by real applications),  where the results may be more fruitful.  
- Motivated by the NMF problem and network modeling, we  focus on a special case where 
$(\sigma_1,\sigma_2, \ldots, \sigma_K) = (1 + a_{K, m}, 1, \ldots, 1, -1, \ldots, -1)$  but $K$ is arbitrary. Here,  $1 \leq m \leq (K-1)$, and $a_{K,m} = \max\{0, 2m-K\}$. 
We show that for $9$ different cases (S1)-(S9), the NIEP is solvable.  For example, one of these cases is $K$ is a multiple of $4$ and $m$ is arbitrary; we show that in this case the NIEP is always solvable.   We also identify two cases (N1)-(N2)  and proved that the NIEP is not solvable in these cases. Table 1 in Section 3 summarizes our results for all pairs of $(K, m)$ with $K \leq 20$ (but 
our theorems are for general $(K, m)$).  
- Moreover, using the results on NIEP and the connection  between NMF and NIEP in Section 1.1,  we derive sharp results for NMF in Section 4. There is a rich literature on solving general NMF problems of the form $V \approx W H$, e.g. [1] and the ensuing work, but they cannot be directly applied to our NMF problem (2), which has a more constrained factorization. We will add these discussion to the paper.

Reviewer comment (Novelty):
What is the result that this paper shows and what are already known results?

Author response:
Response to your main point 2. Thanks for your valuable comment.  We have revised several places in Section 1 to improve literature review, overview of our results, and comparison with existing literature.  To further clarify, 
our main results are as follows. 
- The orthodox NIEP concerns the following problem:  given $K$ numbers $\sigma_1,\sigma_2,\ldots, \sigma_K$, when do we have a $K \times K$ non-negative matrix $M$ with $\sigma_1, \sigma_2, \ldots, \sigma_K$ as its eigenvalues?   
The existing literature on the NIEP have been largely focused on general $(\sigma_1, \sigma_2 \ldots, \sigma_K)$ but 
small $K$ (e.g., $K \leq 4$) (e.g., see the two survey papers we added  in the first paragraph of 
the introduction). Especially, the orthodox NIEP and all of the subproblems we mentioned in the first 
paragraph of our paper remain unsolved for $K \geq 5$.  See Page 1 of the paper for details.  
- Since the orthodox NIEP is too hard (and people can only crack the case for $K \leq 4$ in the past  70 years),  it is of interest to focus on some special cases (especially those motivated by real applications),  where the results may be more fruitful.  
- Motivated by the NMF problem and network modeling, we  focus on a special case where 
$(\sigma_1,\sigma_2, \ldots, \sigma_K) = (1 + a_{K, m}, 1, \ldots, 1, -1, \ldots, -1)$  but $K$ is arbitrary. Here,  $1 \leq m \leq (K-1)$, and $a_{K,m} = \max\{0, 2m-K\}$. 
We show that for $9$ different cases (S1)-(S9), the NIEP is solvable.  For example, one of these cases is $K$ is a multiple of $4$ and $m$ is arbitrary; we show that in this case the NIEP is always solvable.   We also identify two cases (N1)-(N2)  and proved that the NIEP is not solvable in these cases. Table 1 in Section 3 summarizes our results for all pairs of $(K, m)$ with $K \leq 20$ (but 
our theorems are for general $(K, m)$).  
- Moreover, using the results on NIEP and the connection  between NMF and NIEP in Section 1.1,  we derive sharp results for NMF in Section 4. There is a rich literature on solving general NMF problems of the form $V \approx W H$, e.g. [1] and the ensuing work, but they cannot be directly applied to our NMF problem (2), which has a more constrained factorization. We will add these discussion to the paper.

Reviewer comment (Experiments):
This paper analyzes the solvability of the special NIEP and provides the algorithms for NMF, but does not give the time complexity of the proposed method, which lacks comparisons with some prior methods with respect to time efficiency.

Author response:
Response to your main point 2. Time efficiency of the proposed algorithm: with explicit construction of the $Q$ matrix, the dominating computation in both Algorithms 1 and 2 are computing the top $K$ Eigen value and vector pairs of a $n$-by-$n$ real symmetric matrix, which can be solved in O($Kn^2$) time by the Lanczos method or power iteration, and there exist quite a few numerically stable and well optimized software implementations. Standard methods for NMF such as the multiplicative update method of [1] are iterative optimization, and usually run in $O(T K n^2)$ time, where $T$ is the number of iterations. Hence our proposed method with explicit construction of the $Q$ matrix has similar time complexity to existing methods. However, [1] and other optimization-based NMF methods yield factorizations of the form $V \approx W H$, where $W$ and $H$ are allowed to be different. To solve our NMF problem in (2), these techniques need to be modified to enforce extra constraints and may run slower. When constructing $Q$ via numerical optimization as in Lemma 13, the time complexity can be significantly higher. We will add these discussions to the paper.

Reviewer comment (Writing):
Typo on Page 7, Section 3, paragraph 1, line 3: "Table 2.5" should be "Section 2.5".

Author response:
**Response to your minor points**.   We thank you for pointing out these typos,   and we have  fixed them in the revision.

Reviewer comment (Writing):
Typo on Page 8, the line above Algorithm 2: "develop an more" should be "develop a more".

Author response:
**Response to your minor points**.   We thank you for pointing out these typos,   and we have  fixed them in the revision.

Reviewer comment (Writing):
Typo on Page 9, Section 5, line 3: delete "analysis of" in "analysis of network analysis".

Author response:
**Response to your minor points**.   We thank you for pointing out these typos,   and we have  fixed them in the revision.

Reviewer comment (Presentation):
It would be great if the authors could make more efforts in organizing and presenting the technical in a way more accessible to general ML/AI audience.

Author response:
**Response to your main point**. You have made the following main point. 

- The paper contains ton of information and is written in such a way that is not easy to follow. It would be great if the authors could make more efforts in organizing and presenting the technical in a way more accessible to general ML/AI audience.

Thanks  for a very valuable comment.  To respond, we have made the following changes. 
- We have greatly revised Section 2 (Page 4 and Page 5-6, marked in blue) to 
discuss the key ideas underlying our proofs. 
- We have added Section L in the supplement to explain the connection between the NMF problem and the social network modeling problem.  Especially, we discuss the impact our results 
on modeling $5$ well-known networks: weblog, dolphin, karate, polbooks, and UKFacutly.

Reviewer comment (Writing):
Section 1.2: "We also show that in 2 two different cases, (N1)-(N2), Problem (1) is not solvable": redundancy here?

Author response:
**Response to your other points**. You have made the following other points. 
- Section 1.2: ``We also show that in 2 two different cases, (N1)-(N2), Problem (1) is not solvable``: redundancy here?
-  Section 2:  ``Theorems 2-4 are proved in the supplement`` perhaps say also in which specific section of the supplement?
-  After Theorem 9: For Case (d))
-  After Lemma 13: ``This motivates the following algorithm``, could the algorithm be put into an algorithm environment here? or is it just not necessary?
- More generally, the authors propose to tackle Problem (1) using a construction approach by exploiting Haar and DFT basis. To me this choice of basis may not be unique, could the authors comment on the possible extension of the proposed construction?
 
Thanks for your very careful read and valuable comments. For 1-3: we have revised as suggested. 
For 4: we can but we didn't do it due to the space constraint. We will try to revise the paper to accommodate that. For 5: this is a very interesting point and for some of the cases, we have indeed proposed different approaches.  For example, consider the case of $K = 2^d$, where $d \geq 1$ is an integer.  In this case, we can use the Haar construction as in Theorem 6, and we can also use the Haar-DFT combined construction as in Theorem 10. How to extend and combine the proposed approach  to broader cases  is a very interesting research problem for the future work.

Reviewer comment (Presentation):
Section 2: "Theorems 2-4 are proved in the supplement" perhaps say also in which specific section of the supplement?

Author response:
**Response to your other points**. You have made the following other points. 
- Section 1.2: ``We also show that in 2 two different cases, (N1)-(N2), Problem (1) is not solvable``: redundancy here?
-  Section 2:  ``Theorems 2-4 are proved in the supplement`` perhaps say also in which specific section of the supplement?
-  After Theorem 9: For Case (d))
-  After Lemma 13: ``This motivates the following algorithm``, could the algorithm be put into an algorithm environment here? or is it just not necessary?
- More generally, the authors propose to tackle Problem (1) using a construction approach by exploiting Haar and DFT basis. To me this choice of basis may not be unique, could the authors comment on the possible extension of the proposed construction?
 
Thanks for your very careful read and valuable comments. For 1-3: we have revised as suggested. 
For 4: we can but we didn't do it due to the space constraint. We will try to revise the paper to accommodate that. For 5: this is a very interesting point and for some of the cases, we have indeed proposed different approaches.  For example, consider the case of $K = 2^d$, where $d \geq 1$ is an integer.  In this case, we can use the Haar construction as in Theorem 6, and we can also use the Haar-DFT combined construction as in Theorem 10. How to extend and combine the proposed approach  to broader cases  is a very interesting research problem for the future work.

Reviewer comment (Theory):
After Theorem 9: For Case (d))

Author response:
**Response to your other points**. You have made the following other points. 
- Section 1.2: ``We also show that in 2 two different cases, (N1)-(N2), Problem (1) is not solvable``: redundancy here?
-  Section 2:  ``Theorems 2-4 are proved in the supplement`` perhaps say also in which specific section of the supplement?
-  After Theorem 9: For Case (d))
-  After Lemma 13: ``This motivates the following algorithm``, could the algorithm be put into an algorithm environment here? or is it just not necessary?
- More generally, the authors propose to tackle Problem (1) using a construction approach by exploiting Haar and DFT basis. To me this choice of basis may not be unique, could the authors comment on the possible extension of the proposed construction?
 
Thanks for your very careful read and valuable comments. For 1-3: we have revised as suggested. 
For 4: we can but we didn't do it due to the space constraint. We will try to revise the paper to accommodate that. For 5: this is a very interesting point and for some of the cases, we have indeed proposed different approaches.  For example, consider the case of $K = 2^d$, where $d \geq 1$ is an integer.  In this case, we can use the Haar construction as in Theorem 6, and we can also use the Haar-DFT combined construction as in Theorem 10. How to extend and combine the proposed approach  to broader cases  is a very interesting research problem for the future work.

Reviewer comment (Novelty):
More generally, the authors propose to tackle Problem (1) using a construction approach by exploiting Haar and DFT basis. To me this choice of basis may not be unique, could the authors comment on the possible extension of the proposed construction?

Author response:
**Response to your other points**. You have made the following other points. 
- Section 1.2: ``We also show that in 2 two different cases, (N1)-(N2), Problem (1) is not solvable``: redundancy here?
-  Section 2:  ``Theorems 2-4 are proved in the supplement`` perhaps say also in which specific section of the supplement?
-  After Theorem 9: For Case (d))
-  After Lemma 13: ``This motivates the following algorithm``, could the algorithm be put into an algorithm environment here? or is it just not necessary?
- More generally, the authors propose to tackle Problem (1) using a construction approach by exploiting Haar and DFT basis. To me this choice of basis may not be unique, could the authors comment on the possible extension of the proposed construction?
 
Thanks for your very careful read and valuable comments. For 1-3: we have revised as suggested. 
For 4: we can but we didn't do it due to the space constraint. We will try to revise the paper to accommodate that. For 5: this is a very interesting point and for some of the cases, we have indeed proposed different approaches.  For example, consider the case of $K = 2^d$, where $d \geq 1$ is an integer.  In this case, we can use the Haar construction as in Theorem 6, and we can also use the Haar-DFT combined construction as in Theorem 10. How to extend and combine the proposed approach  to broader cases  is a very interesting research problem for the future work.

Reviewer comment (Writing):
The paper lacks a clear discussion in its own right that the NIEP and NMF problems are well motivated. There are exactly 2 sentences that motivate the problem studied. Instead of actually motivating the problem, these sentences instead say that someone else wrote down a good motivation. This is vastly insufficient evidence, and the paper should motivate its own problem clearly.

Author response:
For your main point 1, it seems that there is a major misunderstanding. The NIEP problem is motivated by an NMF problem in equation (2), and the NMF problem is motivated by the problem of social network modeling:  

Network modeling -->  The NMF problem in equation (2) --> NIEP, 

In Section 1.1, we have used a whole page to explain how the NIEP is motivated by the 
NMF problem in equation (2).  The two sentences you mentioned is for explaining how the NMF problem in equation (2) 
is motivated by network modeling. For the latter, we have now added a more detailed explanation, but for space limit, we put it in the appendix (Section L, the last page), where we also discuss modeling for $5$ real social networks.

Reviewer comment (Theory):
The body of the paper contains many constructions for \(Q\) matrices that solve the NIEP for various \((K,m)\) pairs, but it does not justify at all why these constructions should work, or provide any intuition. For a theoretical paper, this is devastating as it leaves no confidence in the correctness of the results, and the appendix is out of scope for a typical review. It would be much better to describe constructions in the appendix and intuitively explain the ideas behind constructions in the body of the paper.

Author response:
For your main point 2: thanks and your comments have greatly helped us in revising our paper. In particular,  
- we have rewritten the first part of Section 2 (marked in blue) on page 4, 
explaining a key idea in our proofs. We also used the key idea to explain the proof for Theorem 2.  
- we have rewritten Section 2.2 (marked in blue) on Page 5-6, explaining the key idea 
underlying Lemma 7 and Theorem 8 there. 

We encourage you to read the changes. We hope these changes have explained some of the key ideas in our proofs  and assure you that our proofs are correct and rigorous.

Reviewer comment (Theory):
On page 3, in the first paragraph of section 1.2, it is stated that four different techniques are used to design \(Q\) matrices, but no explanation is given for why these methods work, no discussion of which similar methods might or might not work, and no lessons to be learned.

Author response:
For your main point 2: thanks and your comments have greatly helped us in revising our paper. In particular,  
- we have rewritten the first part of Section 2 (marked in blue) on page 4, 
explaining a key idea in our proofs. We also used the key idea to explain the proof for Theorem 2.  
- we have rewritten Section 2.2 (marked in blue) on Page 5-6, explaining the key idea 
underlying Lemma 7 and Theorem 8 there. 

We encourage you to read the changes. We hope these changes have explained some of the key ideas in our proofs  and assure you that our proofs are correct and rigorous.

Reviewer comment (Theory):
On page 4, Theorems 2, 3, and 4 are all stated without any justification or argument whatsoever — they are simply stated.

Author response:
For your main point 2: thanks and your comments have greatly helped us in revising our paper. In particular,  
- we have rewritten the first part of Section 2 (marked in blue) on page 4, 
explaining a key idea in our proofs. We also used the key idea to explain the proof for Theorem 2.  
- we have rewritten Section 2.2 (marked in blue) on Page 5-6, explaining the key idea 
underlying Lemma 7 and Theorem 8 there. 

We encourage you to read the changes. We hope these changes have explained some of the key ideas in our proofs  and assure you that our proofs are correct and rigorous.

Reviewer comment (Novelty):
It is unclear if this paper is a good fit for ICLR or any other ML conference. The body of the paper feels like pure math with a superficial connection to ML. The NIEP itself does not feel ML-like, and without strong motivation, the work appears unmoored from ML practice. A stronger connection to the practice of ML would help substantially.

Author response:
For your main point (3),  we think being diverse, inclusive, and interdisciplinary is one of the main reasons why ML conferences become so successful in the past decades. 
- It is a win-win to publish papers of this kind in ML conferences: on one hand, we can  attract talented mathematicians to tackle ML problems in areas such as NMF and network analysis,   and on the other hand,  we can enrich mathematics by providing mathematicians with new ideas and new problems.  
- Both NMF and network analysis are important ML research topics, and 
in the past decades, ML conferences have published many interesting papers in these areas. 
We have spent almost a whole page (Page 2)  explaining how the problem (i.e., NIEP)  grows out from NMF problem in equation (2), 
and in the appendix (Section L),  we have added discussion on how the NMF problem (2) is motivated by network modeling. 
See our response to your main point (1) above.