Paper: The Phase Transition Phenomenon of Shuffled Regression (ICLR-2024)

Reviewer comment (Theory):
I would like the authors to discuss the connections to phase transitions studied in other problems, which could provide more insight on this phenomenon.

Author response:
Moreover, let us answer your question concerning the connections to phase transitions studied in other problems. 
The most related work are (Montanari & Mezard, 2009) and (Semerjian et al, 2020). In (Montanari & Mezard, 2009), 
all edge weights are i.i.d. samples and there is no phase transition behavior. 
Whereas, (Semerjian et al, 2020) observe the phase transition behavior when switching the independent identical distribution (i.i.d.) assumption to independent non-identical distributions. Hence, we conjecture the phase transition is caused by the distribution discrepancy among different edges.

Reviewer comment (Writing):
The clarity and pedagogy of the presentation of the paper seems rather insufficient. Section 2 is a review of the message-passing approach to the linear assignment problem, but the relation to the present problem is not motivated beforehand. Moreover, the relation between the purely random and the planted version is not clear, and it does not seem possible to understand the reasoning in Sec. 2.3 that leads to theorem 1—on which the rest of the results of the paper relies—without reading the original paper [Semerjian et al. (2020)]. Appendix A, which seems intended to clarify this derivation, remains quite obscure.

Author response:
**1. Explanation of the reasoning in Section 2.3.**

First, we explain the reasoning in Section 2.3 and the role of Theorem 1. Section 2.3 basically is to analyze the impact of SNR on the convergence behaviors of the message passing (c.f. Equation (6)). As it is very difficult to track all message flows, we view the message flows to be i.i.d. samples. Here, we regard the message flow $h_{i\rightarrow (i, \pi^{\natural}(i))}$ and $h_{j\rightarrow (i, \pi^{\natural}(i))}$ along the correct correspondence edge to be i.i.d. samples (represented by the random variable $H$). Similarly, we regard the message flows along the rest edges as i.i.d. samples from a distribution denoted by $\hat{H}$. In this way, we only need to track two scalars ($H$ and $\hat{H}$) instead of all message flows. The iterative equation is Equation (6), which can be further simplified to Equation (8) and can be analyzed by Theorem 1. When $\inf_{\theta} 1/\theta \log\left(\sum_{i=1}^n \mathbb{E} e^{-\theta \Xi_i}\right) > 0$, we have $H^{t}$ converge to a large positive value; whereas $\inf_{\theta} 1/\theta \log\left(\sum_{i=1}^n \mathbb{E} e^{-\theta \Xi_i}\right) < 0$ leads to a large negative value. Hence, we conclude there is a sudden behavior change, namely, phase transition happens when $\inf_{\theta} 1/\theta \log\left(\sum_{i=1}^n \mathbb{E} e^{-\theta \Xi_i}\right) = 0$.

Reviewer comment (Theory):
There is a concern about the use of theorem 1 in a setting where it does not apply. In [Semerjian et al. (2020)] the weights on the edges of the graph are independent by construction, which allowed for writing the recursive distributional equations (7). Here, the weight matrix E = X B has strong correlations in the edge weights, and it is not clear why these correlations should be neglected. The manuscript states on page 8 that in the "non-oracle case," "Unlike the oracle case, we notice the edge weights E_ij are strongly correlated," but it is questionable whether these correlations are already present in the oracle case as well.

Author response:
**1. Justifications of weak correlations of matrix $E$ for the oracle case.**

First, we can calculate the correlations between entries in $E$. We can find most of these correlations are zero. There are only two scenarios with non-zero correlations, (i) $E_{\pi(i), \pi(i)}$ $E_{\pi(j)\pi(j)}$ and (ii) $E_{i \pi(j)}$ $E_{j \pi(i)}$, which is with $O(n)$ number while the total edge number is $O(n^2)$. In this sense, we can say $E$ is only weakly correlated. 
In addition, the numerical evidence in Table 1 and Table 2 also suggests that we can safely ignore these weak correlations and predict
the phase transition points to a good extent.

Reviewer comment (Theory):
There are imprecisions in the use of the term "phase transition," since there is no clear description of the qualitative differences between the two phases that the transition should separate, and a study of the effect of the system size on the sharpness of the transition is not presented in the main text.

Author response:
**2. Meaning of phase transition.**

We use the term phase transition to denote the phenomenon such that a small change in the parameter will lead to a sudden change in the reconstruction performance. That is, the error rate drops from one to zero drastically when SNR exceeds the threshold. The meaning of phase transition can be observed in Table 1, Table 2, and Figure 2, where we specifically study the change in error rates.

Reviewer comment (Presentation):
In the upper panels of Fig. 2, the SNR axis extends to negative values. What is the meaning of the part of the curve at negative SNR?

Author response:
**2. Meaning of negative SNR.** 
The negative SNR denotes the place of the potential phase transition points. As SNR is non-negative by nature, we can conclude there is no phase transition in this region, which means the permutation reconstruction cannot be ever reconstructed.

Reviewer comment (Theory):
On page 6, "As a mitigation, we resort to approximating ... by its lower-bound" — could you explain which inequality is used to derive this expression?

Author response:
**3. Lower Approximation of Page 6. We use the standard inequality $\log(1+x)\leq x$.**

Reviewer comment (Theory):
On page 8, it is stated that "this estimator can reach the statistical optimality in a broad range of parameters." Evidence for this statement is requested, as it is surprising: from the beginning of the Section one understands that the "statistically optimal" estimator would involve solving a QAP, which cannot be done efficiently. How can one assert that the LAP reaches statistical optimality if the latter is not known?

Author response:
Dear Reviewer, we would like to first thank you for recognizing our method to be original and promising. 
Please allow us to clarify the statistical optimality of LAP/QAP in the non-oracle case, where there seems to be a misunderstanding. 

First, whether the QAP will reach the statistical optimality still remains an open problem. To the best of our knowledge, the only related work is (Zhang, Slawski & Li, 2022) but they only prove the optimality holds for a restricted region. More specifically, they require the $\textup{rank}(B) \leq c\log n/n$. 

Second, the optimality of LAP over a wide range is answered by the cited work (Zhang & Li, 2020) (c.f. one line above  Equation (19)). Work (Zhang & Li, 2020) show that the mini-max lower bound of SNR requirement, i.e., no estimator can obtain the correct permutation matrix provided that $\log SNR \leq c_0 \log n/\textup{stable-rank}(B)$; and the LAP in Equation (19) can obtain the correct permutation matrix when $\log SNR \geq c_1 \log n/\textup{stable-rank}(B)$ ($\textup{stable-rank}(B)=|B|_F^2/|B|^2$ is the stable rank). 

Interestingly, based on the current literature, the LAP estimator in Equation (19) can actually reach statistical optimality in a wider range than the QAP estimator.

Reviewer comment (Reproducibility):
On page 14, in the description of the numerical procedure, it is not clear that the dichotomy search is what needs further explanation, but rather what corresponds to lines 5 and 6 of Algorithm 1. What is meant by "run experiments" — the resolution of the LAP? Is this done with an exact algorithm or via message passing? Also, could the definition of the error rate be specified: is it the fraction of samples on which the inferred permutation fails to coincide with the planted one, or the average fraction of the permutation that is correctly inferred? This same question applies to the y-axis of the lower panels of Fig. 2.

Author response:
**4. Presentation of Algorithm 1 and detailed explanation of Lines 5 and 6**

Thanks for the suggestion in presenting Algorithm 1 that we may need to give more focus on Lines 5 and 6 rather than the 
binary search. It is interesting that some other reviewers suggested that the binary search method 
needs more clarification since many readers might be unfamiliar with it. We will try to balance these suggestions.

Regarding Line 5, it corresponds to the estimators to solve the permutation matrix, that is equation (9) for the oracle case and equation (19) for the non-oracle estimator.

As for Line 6, the error rate refers to the fraction of samples on which the inferred permutation fails to coincide with the planted one, as we state in the second paragraph in Appendix C "we calculate the error rate of the permutation matrix (full recovery)".  Its mathematical definition is $L^{-1}\sum_{i=1}^L 1(\Pi \neq \hat{\Pi})$, where $1(\cdot)$ is the indicator function, $L$ is the number of experiments and is set as $100$. We explained this in the main text. Perhaps we can make it more explicit. Thanks for the suggestion.

Again, thanks for your review of our work. Please kindly let us know if there are further questions. Thank you.

Reviewer comment (Theory):
Some steps in the derivation are mathematically rigorous, while others are approximations and/or intuitions without strong theoretical evidence.

Author response:
First, we can assure you that **all theorems/propositions are rigorously derivated and can be confidently reused**. Regarding the approximations, they mainly exist in establishing the connections between the phase transition point and the SNR. To be more specific, it primarily involves two aspects (i) the process of linking the phase transition point and Equation (11) and (ii) approximating the solution of (11). The conclusions in Theorem 1, Proposition 1, and Theorem 2, are rock solid. 

Again, we would like to mention that all theorems/propositions (Theorem 1, Proposition 1, and Theorem 2) are rigorously derived. The non-rigorous part is mostly about the dependence between the phase transition points and the expectation $\mathbb{E} \exp(-\theta\Xi)$ and its calculations.

Reviewer comment (Evaluation):
It remains unclear which part of the results can be confidently called for future use, and which part cannot.

Author response:
We feel your major concern is on whether our results can be reused or not.  

First, we can assure you that **all theorems/propositions are rigorously derivated and can be confidently reused**. Regarding the approximations, they mainly exist in establishing the connections between the phase transition point and the SNR. To be more specific, it primarily involves two aspects (i) the process of linking the phase transition point and Equation (11) and (ii) approximating the solution of (11). The conclusions in Theorem 1, Proposition 1, and Theorem 2, are rock solid. 

Again, we would like to mention that all theorems/propositions (Theorem 1, Proposition 1, and Theorem 2) are rigorously derived. The non-rigorous part is mostly about the dependence between the phase transition points and the expectation $\mathbb{E} \exp(-\theta\Xi)$ and its calculations.

Reviewer comment (Theory):
The derivation of the phase transition threshold appears to rely on the approximation $\log( \mathbb{E}[e^{-\theta \Xi}] )$ by its lower bound.

Author response:
**1. Approximation of $\log \mathbb{E}e^{-\theta \Xi}$.**

 This is due to the relation $\log(1+x)\leq x$.

Reviewer comment (Theory):
The derivation of the phase transition threshold appears to rely on a Gaussian approximation of the random variable $\Xi$.

Author response:
**2. Justification of Gaussian approximation.**

The technique of Gaussian approximation is widely applied in analyzing the message-passing algorithm. One of the most successful its applications is in designing the LDPC code (c.f. 'Analysis of sum-product decoding of low-density parity-check codes using a Gaussian approximation' by Sae-Young Chung, T.J. Richard, and R.L. Urbanke). Its success is mostly verified by empirical experiments and the theoretical justifications are still awaiting to be given.

Reviewer comment (Theory):
Further theoretical arguments and/or discussions should be provided to support these approximations; otherwise, it is not clear why the derived phase transition is a reasonable approximate solution, and the main results should be stated as "conjectures".

Author response:
Again, we would like to mention that all theorems/propositions (Theorem 1, Proposition 1, and Theorem 2) are rigorously derived. The non-rigorous part is mostly about the dependence between the phase transition points and the expectation $\mathbb{E} \exp(-\theta\Xi)$ and its calculations. 

**1. Approximation of $\log \mathbb{E}e^{-\theta \Xi}$.**

 This is due to the relation $\log(1+x)\leq x$. 

**2. Justification of Gaussian approximation.**

The technique of Gaussian approximation is widely applied in analyzing the message-passing algorithm. One of the most successful its applications is in designing the LDPC code (c.f. 'Analysis of sum-product decoding of low-density parity-check codes using a Gaussian approximation' by Sae-Young Chung, T.J. Richard, and R.L. Urbanke). Its success is mostly verified by empirical experiments and the theoretical justifications are still awaiting to be given.

Reviewer comment (Theory):
It is unclear whether Proposition 1 is proven somewhere.

Author response:
**3. Position of the Proof of Proposition 1.**

It can be found in Appendix B.

Reviewer comment (Theory):
**Specific Errors:** The representation in equation (3) on page 3 seems incorrect. A direct comparison to the original definition from Mezard & Montanari (2009) indicates a mistake. Specifically, the equation should be \(1 = \sum_j \Pi_{i,j}\) or \(1 \leq \sum_j \Pi_{i,j}\) instead of \(1 - \sum_j \Pi_{i,j}\). Such inaccuracies can mislead readers and detract from the paper's overall quality.

Author response:
First, there seems to be a misunderstanding on Equation (3), please allow us to clarify. Our notations with (Montanari & Mezard, 2009) are in fact equivalent, so Equation (3) is correct. In both our work and (Montanari & Mezard, 2009), the terms with the indicator function are one when $\sum_{j}\Pi_{ij}  = 1$ (and $\sum_i \Pi_{ij} = 1$), and zero otherwise. They are both to enforce the constraint such that $\Pi$ is a permutation matrix.

Reviewer comment (Writing):
The paper's motivation could benefit from a more explicit and clear description. While the challenges of the problem are well-defined, a paragraph highlighting the practical applications of shuffled regression would greatly enhance the paper's appeal. Providing real-world scenarios where this problem arises would place it within a broader context of applicability and increase its relevance.

Author response:
In addition, we appreciate your suggestions on the paper organization such as adding more discussion on the practical applications. As our work a theoretical paper focusing on bridging the message passing with the analysis of permuted linear regression, we limit the content to avoid distraction. The problem of permuted regression has many useful applications, for example:

(A) **Record linkage**.  
Given multiple databases containing information about the same entities, the objective is to merge them into one comprehensive database. However, these data may not be well-aligned due to the data formatting or data quality issues. Modeling the mismatches as a permutation is investigated as a mitigation strategy. 

(B) **Data de-anonymization.** 
This task can be regarded as the opposite side of privacy protection. The intruders aim to infer the hidden identities/labels in certain private networks with public information. One commonly used method is to compare the correlation between this information and to find matching pairs with maximum correlation sum. Here, the permuted linear regression arises as a natural generalization. 

For more applications, we recommend (Unnikrishnan et al., 2015; Pananjady et al., 2018; Slawski & Ben-David, 2019; Pananjady et al., 2017; Slawski et al., 2020; Zhang & Li, 2020), which are a good starting point. We are happy to add more references on the applications if the reviewer suggests.

Reviewer comment (Theory):
The derivation of the lower bound for $\log \mathbb{E}e^{-\theta\Xi}$ on page 6 is not clearly explained. Providing a more detailed and comprehensible explanation of how this lower bound is derived would greatly enhance the paper's clarity.

Author response:
**1. Lower bound on Equation (6).**
The lower bound is obtained with the relation $\log(1 + x) \leq x$.

Reviewer comment (Theory):
Given the significance of the Gaussian approximation in obtaining closed-form expressions for $\text{snr}$, it would be valuable to provide an intuitive explanation of why this approximation is effective.

Author response:
**2. Intuition of Gaussian approximation.**
The choice of Gaussian approximation in the message passing (MP) is mainly due to its past success in analyzing the MP algorithm. One of the most successful examples is (c.f. 'Analysis of sum-product decoding of low-density parity-check codes using a Gaussian approximation' by Sae-Young Chung, T.J. Richard, and R.L. Urbanke), a famous paper in coding theory with more than 1600+ citations.

Reviewer comment (Theory):
The paper imposes a condition on the ratio of the singular values for matrix $B$ when computing $\text{snr}_{\text{non-oracle}}$. An explanation for the rationale behind this condition would be beneficial.

Author response:
**3. Role of the assumption such that all singular values of the matrix $B$ are of the same range in computing $snr_{\textup{non-oracle}}$.**

First, this assumption can greatly simplify the expression of $snr_{\textup{non-oracle}}$. Otherwise,  $\textup{snr}_{\textup{non-oracle}}$ will be the solution of a quartic equation (fourth order) rather than a quadratic equation (second order). Second, this assumption can preserve the generality of our theorem since this assumption can cover a wide family of matrices, say sub-Gaussian random matrices. It is well known that all their eigenvalues are within the region $[1-c_0\sqrt{m/p},~1 + c_0 \sqrt{m/p}]$ (after column normalization), and satisfy our assumption that all eigenvalues are of the same range.