Paper: Federated Binary Matrix Factorization using Proximal Optimization (ICLR-2024)

Reviewer comment (Theory):
The non-trivial part of the paper is the aggregation procedure; however, it is a heuristic and also increases the time complexity for the server.

Author response:
We acknowledge that our proximal aggregation increases the work required from our server. However, the proximal operator is not inherently costly to compute. In fact it can be implemented as a single element-wise matrix operation as a GPU kernel. Computational complexity speaking, it is therefore cheaper than matrix multiplication, and will not raise the computational complexity class neither of our server nor of our clients .

Additionally, aggregation does not occur after every update. Given that the proximal operator is applied to the aggregate, the associated increase in complexity remains independent of the number of clients. Consequently, our approach demonstrates scalability with the number of clients, as illustrated in Figures 3 and 4 of the manuscript.

Reviewer comment (Theory):
The paper does not theoretically show the convergence rate of the algorithm.

Author response:
The convergence rate trivially follows from our proof of Proposition 1, which will yield a rate of $\mathcal{O}\left(\frac1{\lambda_T}\right)$. That is, setting $\lambda_t = t$ yields a linear convergence rate of $\mathcal{O}\left(T^{-1}\right)$. Note that, by choosing a large lambda (such as $t^2$) we can yield faster convergence, but in practice might lead to a reduction in the relative reconstruction loss.

We thank the reviewer for the interest in this aspect of our algorithm. We will clarify the implications in the final version of our manuscript.

Reviewer comment (Theory):
The analysis relies on a strong assumption of the loss function.

Author response:
We make use of the standard squared error as loss function and furthermore only make the standard assumption that gradients are bounded. Could you please point out which assumption you deem strong?

Reviewer comment (Writing):
The writing of this paper could be improved. For example, more intuition for the proximal operator should be provided.

Author response:
To improve our presentation and raise the accessibility of our work, we will include more intuitive descriptions in the main body of our paper, regarding—for example—the interplay of gradient descent, proximal operator, regularization, and aggregation.

Reviewer comment (Writing):
The current manuscript is not self-contained, and it could be difficult for people outside of this field to understand.

Author response:
To make our paper self-contained, we will expand the details and a complete derivation of our proximal algorithms in the appendix.

Reviewer comment (Writing):
Eq. (2): why \(A_{ij}\) or \([U\circ V]_{ij}\)? It seems to be xor.

Author response:
We deemed this to be more readable, but will revisit the notation.

Reviewer comment (Writing):
Eq. (4): it is unclear what it means for \(x\in X\). Is \(X\) a set of points, and \(x\) is a vector? Or is \(x\) a number? And what is \(x-1\)?

Author response:
$x$ is a single real-valued matrix cell from one of the factor matrices. For example $x = U_{ij}$.

Reviewer comment (Theory):
Page 4: “Without the knowledge of \(U_i\), the server cannot estimate specific attributes of individual users (assuming sufficiently large client datasets).” Can the server learn \(U\) from \(A\) and \(\hat{V}\)? Is it proved in this paper to rule out such an attack? If so, a reference should be added here.

Author response:
Our point is that the server cannot trivially estimate specific attributes about users (i.e., learn $U$ from $A$ and $V$, since factorizations are not unique (nonnegative factorization are not unique, even in case of infinitesimal rigidity [1], and Boolean factorizations can be shown to be not unique by a simple application of the pigeonhole principle, see [2] for further details). We apologize for the imprecision. As stated in Sec. 4.2, we are well aware that a curious server could in principle obtain information about local data from V - one could imagine approaches similar to gradient leakage or membership inference attacks. This is the whole motivation for the differential privacy mechanism in Sec. 4.2 and our empirical evaluation of it in Sec. 5.1.3.

Reviewer comment (Writing):
Proposition 1: the symbol \(R\) should be replaced by some other symbol, as it has been used to denote the regularizer.

Author response:
Indeed, this is a mistake. We will replace it with another symbol. Thank you!

Reviewer comment (Writing):
Page 5, line 4: is it a typo for \(\nabla_U\nabla_U\)?

Author response:
Good catch!

Reviewer comment (Theory):
Proposition 1: even if the gradient tends to zero, the loss function may not tend to zero without further assumption on the loss function. Is the loss function considered here convex? And how do we justify the assumption on the bounded gradient?

Author response:
It is correct that the loss function does not necessarily converge to $0$. Indeed, it is rather unlikely that it does, since it would imply a perfect factorization. Proposition 1 does not assume this. The assumption of a bounded gradient is ubiquitous in the optimization literature and holds, e.g., for Lipschitz continuous loss functions and data from a bounded domain.

Reviewer comment (Theory):
Section 4.2: the differential privacy claims should have a formal statement with proof.

Author response:
Thank you for this observation. Having a formal statement would indeed highlight the differential privacy claims. Due to space limitations and since the results can be easily derived from (BalleandWang,2018), respectively (Nobleetal.,2022), we decided, however, to keep it simple. We do appreciate your point, though, and will therefore include formal statements and their derivation in the appendix.

Reviewer comment (Writing):
Page 7: the index of the summation should be stated explicitly.

Author response:
Thank you, we will adjust this equation in the final submission.

Reviewer comment (Novelty):
(Chen et al. 2022) considers the symmetric and sparse version of BMF and shows a combinatorial algorithm for the average-case instances. It is interesting to see whether the method in this paper can be generalized to this regime.

Author response:
This is a great question and the suggested work is interesting. Our work is based on clients with access to a vertical part of the input matrix. That is, rows and columns are treated differently. If the matrix is symmetric, it will ‘break’ this setting. In fact, we made sure that our experimental data will not by accident include symmetric data. Nevertheless, symmetric settings are relevant in general, and hence we will certainly add a discussion of Chen et al.to related work.

Reviewer comment (Novelty):
The contribution of the paper seems limited, as it is a straightforward extension of the method of Dalleiger et al. to the federated setting. Each client applies the same method, then the $V_i$ updates are aggregated on the server. The main algorithmic novelty appears to be in how aggregation is done, specifically in applying the proximal operator (the same one used by Dalleiger et al.) to the mean of the updates.

Author response:
Although we make use of work from Dalleiger et al. on Boolean matrix factorization, our improvements are significant. First, we contribute a novel regularization scheme that clearly outperforms Dalleiger et al.’s approach. Second, our approaches are the first and only truly federated Boolean matrix factorization algorithm in existence, filling an important research gap and addressing real-world needs. Third, we provide rigorous guarantees in terms of convergence.

Reviewer comment (Theory):
The theoretical result (convergence of the iterates in Proposition 4.1) has several issues. First, the proof is wrong: it argues that the gradients tend to 0, thus the iterates converge, which is not true (e.g., the sequence $x_k = 1/k$).

Author response:
We respectfully disagree with the comment that the proof is incorrect. The proof is, in fact, correct. There seems to be a misunderstanding in what approaches zero. It is not the gradient itself, as the reviewer seems to have understood, but rather, we assume that the gradient is limited. It is the gradient component, as opposed to the regularizer, that diminishes towards zero. The proof demonstrates that the gradient is bounded by $R$, and since $\eta_t\propto \lambda_t^-1\rightarrow 0$ for $t\rightarrow \infty$, the limited gradient multiplied by a decreasing learning rate tends to zero. We apologize if the wording caused any confusion, and will rewrite this part to avoid this for any future reader.

Reviewer comment (Theory):
Even if the proof in Proposition 4.1 were to be corrected, the result remains very weak. It only claims that the process converges (to a binary solution) without providing any guarantee about the utility of the limit (e.g., its proximity to the actual solution). Such a result could be obtained by a trivial algorithm, such as one that uses a step size of 0 after a finite number of steps.

Author response:
We also respectfully disagree that a convergence result is without value if the limit’s utility is not demonstrated. In non-convex optimization, such as for deep learning or federated deep learning, it is common practice to illustrate that $\mathbb{E}[\nabla_w F(w)]$ approaches zero, signifying the attainment of a local minimum or saddle point. These convergence results are highly valuable, although they do not imply the usefulness of that limit.

Reviewer comment (Novelty):
The motivation to add binary constraints in recommender systems is unclear, as this does not seem to be a common practice. Relatedly, some of the recommendation datasets used in experiments were employed in prior work on private matrix completion without binary constraints, and comparison would be beneficial.

Author response:
Boolean data is ubiquitous in real-world settings, such as gene mutation data.  
In many recommender applications, the goal is to distinguish between good or bad recommendations. Factorizing a binarized data matrix with two Boolean factor matrices,  will result in sparse (Miettinen et al. 2006), informative, and easily interpretable factorizations by default.  

Our datasets are derived from commonly-used and well-known benchmark datasets.  
However, as we require Boolean inputs, we binarized our datasets first.  
Comparing aforementioned matrix completion algorithms with our method, would not only compare the performance of different tasks, but would also introduce a bias in favor of our algorithm, as—unlike our approaches—the completion algorithms are not specialized towards Boolean data.

Reviewer comment (Theory):
In section 4, the claim that "Without the knowledge of $U_i$, the server cannot estimate specific attributes of individual users" is not true. Access to gradients can leak information about client data, as shown in reconstruction attacks (e.g., [6], which studied matrix completion specifically).

Author response:
The claim is that the server cannot trivially estimate specific attributes about users, since factorizations are not unique (nonnegative factorization are not unique, even in case of infinitesimal rigidity [1], and Boolean factorizations can be shown to be not unique by a simple application of the pigeonhole principle, see [2] for further details). We apologize for the imprecision. As stated in Sec. 4.2, we are well aware that a curious server could in principle obtain information about local data from V - one could imagine approaches similar to gradient leakage or membership inference attacks. This is the whole motivation for the differential privacy mechanism in Sec. 4.2 and our empirical evaluation of it in Sec. 5.1.3.

Reviewer comment (Experiments):
The privacy-related experiments show that for epsilon up to roughly 10, the loss exceeds 1 (which, under the loss definition, implies performance worse than the trivial all-0 solution). This is surprising and essentially means learning is impossible except for very large epsilon, suggesting the method may not have been properly tuned. The authors should revisit these experiments.

Author response:
The relative loss can exceed $1$, if the reconstruction consists of more $1$s than the input.

Reviewer comment (Theory):
In the appendix, sensitivity is estimated by the maximum number of 1s in the input data. This value is private and should be estimated privately (not computed exactly), or a maximum should be set with per-client subsampling to this number.

Author response:
Thank you for this great observation. Indeed, we have used a very straight-forward estimation of sensitivity in our experiments, that reveals the maximum number of $1$s in the dataset. Using a differential privacy mechanism to share local maxima is a great idea. Setting a global maximum is a neat practical trick, but requires some domain knowledge to set it right - too low and we have to distort data substantially by subsampling, too large and we decrease the quality due to excessive noise.

Reviewer comment (Reproducibility):
Although the authors commendably included their code in the supplementary material, the lack of comments or instructions makes reproduction of results nearly impossible. At minimum, a README with instructions on running the methods and acquiring the data should be included.

Author response:
Of course, we will submit a complete, fully documented, and easily usable reproducibility package, that comes as an archive with DOI and a publicly available and maintained GitHub repository.

Reviewer comment (Evaluation):
Regarding relative loss, a value exceeding 1 implies the solution is essentially useless, as the trivial 0 solution would achieve a loss of 1. This suggests that the private method learns little except for very large epsilon. The authors are encouraged to revisit these experiments and investigate this poor performance.

Author response:
The relative loss can exceed $1$, if the reconstruction consists of more $1$s than the input.

Reviewer comment (Theory):
The potential disadvantages of having a proximal operator is not being discussed.

Author response:
**W2) advantages of a proximal operator**

Approaching binary matrix factorization in terms of a proximal algorithm has similar benefits as approaching NMF with proximal gradient descent. That is, by separating loss and binary regularization, we can take a much longer “unregularized” gradient step than the steps taken by e.g., multiplicative update rules (MUR). This will usually result in significantly faster convergence rates in comparison to MUR.

The downside is that we need to compute the proximal step. This, however, is done by a single element-wise matrix operation via an efficient GPU kernel in O(n*k). We will clarify the advantages of our proximal approach in the updated version of the manuscript.

Reviewer comment (Reproducibility):
The methods compared could be discussed further, for instance in Figure 6. the loss values for most of the models seem to be stuck at 1. What is the reason for that? What are the parameters used for these methods? How are they implemented? It might be possible that this is a bug. Similarly the recall values are zero. This is skeptical.

Author response:
**W3) competitors, parameters**

We share the desire for having a rigorous and extensive evaluation. We do believe that our evaluation fits this standard well.

**W3.1) Regarding reproducibility**
- The parameters can be found and are discussed in the “Reproducibility” section in the appendix.
- The implementations for the competing BMF algorithms are publicly available from their respective authors.

**W3.2) Regarding results**

- We consider a _relative_ loss, normalized by the Frobenius norm of the input matrix, rather than an absolute loss, as this allows for more insightful comparison across datasets.
- During the empirical evaluation, we extensively studied the performance, looked into reconstructions, and validated the correctness of all methods, including the “competing” algorithms. On synthetic data, we observe a different trend in the performance of our “competitors” on synthetic data, which is evidence for this. On real-world data, those algorithms do not allow for an accurate reconstruction in terms of shared coefficient (component) matrices. Because the algorithms perform as expected locally, we discovered that widely-used aggregation strategies fail to address the problem well. 
We therefore proposed our two methods to address this problem.

Reviewer comment (Evaluation):
Most of the recent research shows that preserving privacy inevitably brings a reduction in accuracy. The authors claim that the accuracy of the method is even higher than the methods that do not have a privacy component. What is the trade-off? computational complexity? scalability? time complexity? difficulty in implementation?

Author response:
**Q2) privacy**

The enhanced privacy of occluding observation-specific factors not only reduces the computational complexity on the server-side but also the amount of data that needs to be transmitted. To guarantee differential privacy, we use an additive noise mechanism, which _reduces_ the accuracy—as expected—with regards to having no additive noise.

Reviewer comment (Theory):
What are the advantages that the proximal gradients bring to the problem?

Author response:
**Q3) Advantages of proximal gradients**

By separating the optimization into a (non-regularized) gradient step and a proximal projection, we take significantly larger gradient steps that will result in faster convergence rates (in comparison to multiplicative update rules)
Furthermore, this separation allows for computing the gradients (per factor matrix) using a  known closed form solution, which leads to speedups and performance gains in comparison to, e.g. auto differentiation.

Reviewer comment (Experiments):
Did the authors do any comparison with the new federated matrix factorization (Du et al., 2021) and federated non-negative matrix factorization (Li et al., 2021)?

Author response:
**Q4) NMF**

We haven’t compared against Federated NMF, because (i) we are after factorizing Boolean matrices and (ii) we are interested in a specific privacy model that keeps the first factor matrix private.

Reviewer comment (Presentation):
In Figure 6, is there a logical order of the datasets?

Author response:
**Q5) Order of datasets**

We are happy to consider a reorganization of our partial dataset ordering in our camera ready version.

Reviewer comment (Reproducibility):
Is the lambda different for u and v? (Algorithm 1, line 4 and 5)

Author response:
**Q7) Different lambdas per factor**

For brevity, we choose the same lambda to parameterize all matrices. This can easily be adapted if the need arises.