Paper: On Socially Fair Regression and Low-Rank Approximation (ICLR-2024)

Reviewer comment (Writing):
The paper uses many existing results from randomized numerical linear algebra and other literature for the main technical results. Although I don't consider this by itself a weakness (in fact the ideas are combined nicely), it does have the unintended effect of making the paper less accessible to readers unfamiliar with the areas. It would be helpful if the writeup following theorems 1.5 and 1.6 is modified to make it more accessible.

Author response:
We apologize that the previous version may have been inaccessible to general audiences. We have added a new Appendix A.1 that describes high-level intuition for a number of common techniques, to provide additional guiding exposition intuition for the statements following Theorems 1.5 and 1.6, which we intend to incorporate into the main body in a full version of the paper.

Reviewer comment (Theory):
The question might sound a little basic. However, correct me if I am wrong, when you talk about the exponential time, you are only talking about time in terms of $k$ and not $n$ right? If yes please make it clearer, else it may create confusion regarding theorems 1.4 and 1.5.

Author response:
Yes, we are referring to exponential time in $k$. We have updated the discussion surrounding Theorems 1.4 and 1.5 in the revised version to clarify this point.

Reviewer comment (Experiments):
I believe here the groups are non-overlapping. Can you comment on the results were the groups overlapping?

Author response:
In general, socially fair algorithms do not seem well-defined when the groups are overlapping, i.e., if an individual can belong to multiple sub-populations. However, we agree with the reviewer that it could make sense for an individual to be able to contribute to multiple groups through some fractional weighting mechanism, which would be an interesting direction to explore.

Reviewer comment (Experiments):
There is no real-world application or numerical evaluation of the proposed fair low-rank approximation method, which I think is more important than the evaluation of the fair regression method shown in Section 4.

Author response:
Our lower bounds show that under a common hardness conjecture, i.e., Exponential Time Hypothesis, exponential time is required for socially fair low-rank approximation algorithms. Thus, it does not seem feasible to implement proposed algorithms on large-scale real-world datasets. On the other hand, in the appendix, we provide experiments on synthetic datasets show

Reviewer comment (Experiments):
There is no comparison between the proposed methods and other methods in the existing literature. For example, for fair regression, the authors may want to compare their convex solver method to Abernethy et al. (2022).

Author response:
Yes, we remark that for socially fair regression, the main observation is that the problem is convex, similar to Abernethy et. al. (2022). Whereas Abernethy then uses a different convex solver method, we reference the interior point methods for a full end-to-end theoretical guarantee. On the other hand, for our experiments, we use the standard convex solver packages in Python. We believe empirical comparisons between the standard packages and the convex solvers of Abernethy et. al. (2022) are more a question about convex optimization benchmarks rather than regression benchmarks. 

In any case, we do not view the socially fair regression as our main result, but rather a simple warm-up that in conjunction our main socially fair low-rank approximation results, exhibits a somewhat surprising juxtaposition for the complexities of the two problems, given the abundance of shared techniques often used to approach these problems.

Reviewer comment (Novelty):
The contribution to socially fair regression seems limited since the method is simple and based on the observation in Abernethy et al. (2022).

Author response:
Yes, we remark that for socially fair regression, the main observation is that the problem is convex, similar to Abernethy et. al. (2022). Whereas Abernethy then uses a different convex solver method, we reference the interior point methods for a full end-to-end theoretical guarantee. On the other hand, for our experiments, we use the standard convex solver packages in Python. We believe empirical comparisons between the standard packages and the convex solvers of Abernethy et. al. (2022) are more a question about convex optimization benchmarks rather than regression benchmarks. 

In any case, we do not view the socially fair regression as our main result, but rather a simple warm-up that in conjunction our main socially fair low-rank approximation results, exhibits a somewhat surprising juxtaposition for the complexities of the two problems, given the abundance of shared techniques often used to approach these problems.

Reviewer comment (Presentation):
This paper is not well-organized. Section 1.1 about the contributions is overwhelmingly long (about 2.5 pages), most of which is about the fair low-rank approximation which I think can be put in Section 3, especially since some parts in Section 1.1 and Section 3 are overlapped.

Author response:
We remark that the purpose of Section 1.1 is both to highlight our contributions but also to provide a technical overview/intuition for our results. We have renamed Section 1.1 to emphasize this intention.

Reviewer comment (Writing):
Page 26. 'paragraphSynthetic dataset' looks strange to me.

Author response:
Thanks for pointing out this typos. We have addressed these points in the revised version.

Reviewer comment (Theory):
For the approximate fair low-rank approximation, is it possible to have corresponding lower bounds for Theorems 1.5 and 1.6?

Author response:
Note that in some sense, Theorem 1.3 and Theorem 1.4 are lower bounds that correspond to Theorems 1.5 and 1.6. Theorem 1.3 notes that any one-sided constant-factor approximation in polynomial is unlikely under the assumption that P!=NP, which motivates the study of the bicriteria approximation in Theorem 1.6. Similarly, Theorem 1.4 shows that under the stronger Exponential Time Hypothesis, runtime that is exponential in $k$ is necessary, and Theorem 1.5 achieves runtime that has exponential dependnecies on $k$ (though admittedly not linear in the exponential). We agree that it would be an interesting question to resolve this remaining gap.

Reviewer comment (Theory):
Can we replace the probability value 2/3 in Theorems 1.5 and 1.6 with some value $\xi$ which can be as close to 1 as possible or converge to 1 w.r.t. n? Otherwise, it seems like finding the approximate solution is still not guaranteed with high confidence.

Author response:
Yes, the success probability can be boosted to an arbitrarily high $1-\delta$ by running $O\left(\log\frac{1}{\delta}\right)$ independent instances of the algorithm and taking the best solution. In this case, the runtime will also increase by a factor of $O\left(\log\frac{1}{\delta}\right)$.

Reviewer comment (Theory):
The equation $\max \\|x\\|_\infty = (1\pm \epsilon)\\|x\\|_p$ in the beginning of Section 3.2 is surprising to me. Is there a typo?

Author response:
The maximum is somewhat redundant, but the reason that the $L_\infty$ norm is close to the $L_p$ for large $p$ is that the $p$-th power of the large entries are significantly larger than the $p$-th power of the small entries, and thus $\|x\|_\infty=(1\pm\epsilon)\|x\|_p$ for $p=O\left(\log\frac{n}{\epsilon}\right)$, where $n$ is the dimension of $x$. See the proof of Lemma C.10 in the appendix for a formal proof.

Reviewer comment (Experiments):
k-means clustering can also be viewed as constraint low-rank approximations. Can the socially fair low-rank approximation be used to solve the socially fair k-means clustering problem?

Author response:
Although the view of $k$-means clustering as a constrained low-rank approximation problem has been useful in results such as dimensionality reduction, it is not immediately clear to us how to use the socially fair low-rank approximation problem to solve the socially fair $k$-means clustering problem, in part due to the fact that the linear combination of the low-rank factors can be both fractional and negative. However, we do think that in general the relationships between $k$-means clusteirng and low-rank approximation are interesting to explore.

Reviewer comment (Writing):
Notations are not well defined and difficult to follow, eg alpha is well defined in the context where ever it has been used.

Author response:
We first introduce and define $\alpha$ in the discussion after Theorem 1.5. We again introduce and define $\alpha$ is the discussion at the beginning of Section 3.1. Finally, we define $\alpha$ in Line 1 of Algorithm 4. Regardless, we have made a pass to clarify notations and variables, including repeating the definition of a variable if it has not been recently discussed in the paper. We hope this improves reader accessibility.