Paper: Efficient Differentiable Approximation of the Generalized Low-rank Regularization (ICLR-2024)

Reviewer comment (Reproducibility):
There are still many specific details that are missing and that I would like to obtain (refer to the questions section).

Author response:
Thank you for the interest in our work! Our code will be made publicly accessible on github after anonymous review.  
In our implementation, we use Mathematica to compute the required coefficients in (6) and (7), which are then input to the python code. We refer the reviewer to article [1], which shows how to compute the coefficients in the Laguerre expansions. Our code calculates coefficients of the Taylor expansion and the Laguerre expansions in a similar way. Besides, the computation is rather efficient, talking only several seconds on a personal laptop.

Reviewer comment (Reproducibility):
How do you truncate the Taylor and Laguerre series?

Author response:
In our experiments, the series of Taylor and Laguerre expansions are truncated up to the 5th degree. We also tested with higher degrees, but no further improvements can be observed.

Reviewer comment (Reproducibility):
I would find it useful to have the precise algorithm you use in practice to compute (6) with the different approximations and the different parameters.

Author response:
Thank you for the interest in our work! Our code will be made publicly accessible on github after anonymous review.  
In our implementation, we use Mathematica to compute the required coefficients in (6) and (7), which are then input to the python code. We refer the reviewer to article [1], which shows how to compute the coefficients in the Laguerre expansions. Our code calculates coefficients of the Taylor expansion and the Laguerre expansions in a similar way. Besides, the computation is rather efficient, talking only several seconds on a personal laptop.

Reviewer comment (Theory):
Have the authors considered efficient backward pass via implicit differentiation of the fixed-point iterations?

Author response:
We thank the reviewer for the suggestion. Though currently our iterative approach for calculating fix-points is sufficient for practical applications, it is certainly an interesting idea to further improve its efficiency with implicit differentiation. We will leave this exploration as a future work.

Reviewer comment (Experiments):
Do you train a different model for each sigma parameter, or do you use sigma as a parameter of the model?

Author response:
We trained a separate model for each noise level, following the previous experiment pipeline.

Reviewer comment (Theory):
Why is the low-rank structure only applicable to the entire image?

Author response:
The application of low-rank structure in image tasks is based on the observation that, most singular values of natural images are concentrated in a few of the largest ones [2]. This phenomenon becomes less evident when the patch becomes too small and the distribution deviates from that of natural images.

Reviewer comment (Reproducibility):
For the denoising applications, which regularizer do you use?

Author response:
The results reported in the paper are obtained by using the basic nuclear norm regularization (i.e., Proposition 4). We also tested with other relaxation functions, but the difference is not evident in this particular task.

Reviewer comment (Experiments):
It would have been nice to see some results on the approximation errors that one gets as one truncates the infinite sums in equations 6 and 7. This may help the user to choose the truncation point N.

Author response:
In our experiments, the series of Taylor and Laguerre expansions are truncated up to the 5th degree. We also tested with higher degrees, but no further improvements can be observed.

Reviewer comment (Theory):
This paper introduces a smooth approximation technique for the rank function, which is a well-explored area in the literature, featuring numerous convex and nonconvex approximation methods. However, the paper lacks an optimality analysis of the proposed technique, leaving it unclear why this particular strategy is necessary.

Author response:
Though various convex and nonconvex application methods exist in the literature, our method possesses several unique advantages that distinguished from other works: 1) Our method can be generalized to various convex and nonconvex relaxations, while each previous work focuses on its particular form of low-rank regularization without a unified treatment. 2) Our method makes the low-rank regularization differentiable, which is compatible to modern deep learning models and libraries. 3) Our method is rather convenient to apply. The regularization term can be added in a plug-and-play fashion, and only a few lines of codes are needed to adapt to a new problem. 4) Our method is GPU-friendly and takes advantages of the efficient parallel computation, since the proposed method solely depends on matrix multiplication.

Though it remains a theoretical challenge for the optimality and convergence analysis of our method, our paper presents the first low-rank approximation technique that simultaneously possess the above characteristics. Besides, the superior effectiveness and efficiency of our method have been thoroughly demonstrated in the experiments.

Reviewer comment (Reproducibility):
The authors suggest employing Laguerre expansion to handle the low-rank regularization function, with $\alpha_{k,p}$ representing the polynomial coefficients. The authors mention that these necessary coefficients can be computed using readily available tools like Mathematica; however, the complexity of computing these coefficients remains unclear. Furthermore, the paper does not discuss specific ranges for $k$ and $p$.

Author response:
We apologize for the unclarity. Implementation details of the Laguerre expansion are presented as follows:

1.	In Mathematica, the coefficients are computed by numerically integrating a 1d function, which is a very efficient procedure. 
2.	Particularly, it takes only several seconds to complete the computation on a personal laptop. Since the coefficients only need to be computed once, the overall computational overhead is negligible.
3.	The Mathematica code of our implementation will be publicly available in our repository. We refer the reviewer to article [2], which is similar to our implementation.
4.	In our experiments we set the ranges of k and p to be 5. In other words, the series of Taylor and Laguerre expansions are truncated up to the 5th degree. We also tested with higher degrees, but no further improvements can be observed.

[1] Lin, Tsung-Yu, and Subhransu Maji. "Improved Bilinear Pooling with CNNs." Proceedings of the British Machine Vision Conference, 2017.

[2] Li, Peihua, et al. "Towards faster training of global covariance pooling networks by iterative matrix square root normalization." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[3] MATHEMATICA TUTORIAL Part 2.5: Laguerre expansions (https://www.cfm.brown.edu/people/dobrush/am34/Mathematica/ch5/laguerre.html).

Reviewer comment (Writing):
The writing of this paper is crude. The authors should present the detailed algorithms as well as convergence analysis when using the proposed model for specific applications such as matrix completion as well as video foreground and background separation.

Author response:
We thank the reviewer for the suggestions. We will add a detailed algorithm description in the Appendix in the revised version. 
Rigorous theoretical convergence analysis is challenging in our case. Since our method novely applies several approximations for computing matrix root and inversion, existing analysis techniques cannot be directly applied. Nevertheless, our method converges quite well in practice. One representative example is the convergence result shown in Appendix A.2.1.

Reviewer comment (Writing):
Some of the presentation is not correct, such as “…key problem of the matrix factorization is that it demands strong prior knowledge on the matrix rank…”. To my knowledge, the variational representation of the nuclear norm as well as the non-convex regularization do not need to know the exact rank, such as [1,2,4,5].

Author response:
Though these methods do not need to know the exact rank, they need to know “the upper bound of the exact rank”: most factorization methods assume $A=BC$ where $A\in R^{m\times n}$, $B\in R^{m\times k}$, and $C\in R^{k\times n}$. Here the choice of $k$ requires strong prior knowledge, and many previous works show that an inappropriate choice will deteriorate the performance.

Reviewer comment (Theory):
The authors argue that the SVD is not differentiable, please refer to [6] for more information.

Author response:
We sincerely thank the reviewer for pointing out the result unknown to us before. Indeed, it presents a method for computing the gradients of SVD. The related discussions in our manuscript will be modified accordingly. However, though the result shows that differentiating SVD is possible, it is far from being practical. The prohibitive $O(N^4)$ complexity for computing gradients is unacceptable in our setting.

Reviewer comment (Evaluation):
Some of the experimental results are vague. Specifically, in Table 1, the running time of MSS is slower than IRNN. As for as I know, MSS is a representative factorization-based model which is much faster than the IRNN. Besides, the proposed model provides an equivalent representation of the nuclear norm and non-convex rank regularization function, the overall mathematic model is the same as the low rank regularization function implied on the original full matrix. Why does the performance of the proposed method present in Table 1 is much better than existing methods with the same mathematical model? Where does the performance gain come from?

Author response:
About running time: For all the compared baselines, we use the codes from public repositories and locally evaluated their performance with the same environment. Particularly, the code of MSS is taken from [6], and the code of IRNN is from [7]. The exact running times may vary between implementations. 

About performance gain: There are two major reasons that explains the superior performance of our method:

1.	Our method benefits from the non-convex relaxation of the rank function. Many of the compared baselines use the nuclear norm or Schatten-p quasi-norms with p<1, while our method utilizes more sophisticated relaxation.
2.	Even the non-convex relaxation function and the mathematical models are the same, different methods have unequal convergence behaviors. We also note that while many related works have presented convergence analysis, these results typically characterize the worst-case performance, and may not fully explain the empirical performance.

[6] https://github.com/xuchen314/MSS/tree/master

[7] https://github.com/canyilu/Iteratively-Reweighted-Nuclear-Norm-Minimization