From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: Paper studies Federated Learning (FL) and proposes using 0th-order estimators for approximating client gradients through only forward passes instead of exactly computing them through memory-wise costly auto-differentiation. Authors have refreshingly interesting motivations for doing this: (a) compute and memory constraints on edge client devices, (b) communication throughput constraints, (c) prevent data inversion attacks. The proposed 0th-order method for estimating the gradients from function value evaluations is pretty standard. It is notable that the author fail to properly discuss the vast literature on zeroth-order optimization, and in particular acknowledge similar algorithms in zeroth-order FL literature [1, 2]. Theorem 1 seems standard in the literature and it is not clear why they are not attributed properly to past work. 

Authors provide a lot of experiments and ablation studies which is commendable. Especially the memory comparison is very useful and promising. However, these results fail to fully justify the assumptions and motivations for this work. The local and global computational complexity (number of floating point operations per communication round) is not compared with the baselines. This is important since abstract claims "executing backpropagation on them incurs computational...overhead". The experiments to prove robustness to attacks is not quantitative and doesn't seem to be properly designed, specifically it doesn't compare the privacy leakage with standard FL algorithms. From the writing, it is really hard to know which experiments use FedSGD and which use FedAvg ("Ground Truth" is confusing). In practice FedAvg is almost always used, so it is not clear why FedSGD is chosen as a baseline. It is not clear how FedAvg+BAFFLE would work since they seems to require sending the pseduo-gradients (Tensors) from clients to servers instead of just function values like FedSGD+BAFFLE. In that case, the "#4 x K Bytes" annotation in the pseudocode is misleading. It would be great addition to the paper, if the authors can provide a graph plotting accuracy vs number of bytes of data communicated for different algorithms. Authors also do not compare their work with state-of-the-art algorithms eg: SCAFFOLD. Paper considers only cases with a small number of clients (up to C=100). Authors are highly encouraged to improve the experimental design and writing in above aspects.

[1] Li, Z., & Chen, L. (2021, October). Communication-efficient decentralized zeroth-order method on heterogeneous data. In 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP) (pp. 1-6). IEEE.
[2] Fang, W., Yu, Z., Jiang, Y., Shi, Y., Jones, C. N., & Zhou, Y. (2022). Communication-efficient stochastic zeroth-order optimization for federated learning. arXiv preprint arXiv:2201.09531.
justification_for_why_not_higher_score: Failure to discuss vast related work literature. Inadequate or unclear experimental evalaution.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Any new comments before the final recommendation
comment: Dear Reviewers, 

Thank you again for your valuable comments, which significantly aid in improving our work.

In summary, BAFFLE provides a communication- and memory-efficient framework that is underexplored in the FL literature. Because BAFFLE only performs forward propagation, its computation cost could be greatly reduced by various inference-only optimizations at both hardware and software levels in practical deployment.

Since the deadline for the final recommendation is approaching, please feel free to give any new comments, and we will do our best to address them.

Best,

The Authors

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: About additional computation
comment: Thank you for the new comments.

We must clarify that $K=1000$ *does not* result in X1000 times more computation, as in BAFFLE, only forward propagation is performed. As described in Section 4.4, the additional computation cost in our code is about $\frac{K}{5}-1$. For example, in Table 3, $K=200$ results in $\\sim 3\\%$ drop in accuracy on MobileNet, and the additional computation is approximately X39 ($\frac{200}{5}-1$).

Please note that the factor $5$ only applies to our code implementation, as specially optimized implementations can result in significantly higher factor values (i.e., much faster forward propagation).

Regarding the application of low memory usage, TEEs, whose dynamic memory is typically less than 100MB, are a typical example. BAFFLE is a promising approach to resolving the difficulty of incorporating BP-based training into a TEE.

From: ICLR.cc/2023/Conference/Paper1664/Reviewer_wGpR
title: Regarding memory
comment: Thank you for your update.

I understand the improvement in memory and that it is a big plus, but this comes with the cost of running the network X1000 times more. In problems where you need low memory usage, you also want to save on computing and energy uses. I have a hard time thinking about a case where saving 95% on memory would be so important that you are willing to suffer an X1000 running time.

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Thank you!
comment: Thank you for your support, we really appreciate it! We will further polish the paper in the final revision.

From: ICLR.cc/2023/Conference/Paper1664/Reviewer_7JBh
title: Thank you for the comment
comment: I thank the authors for clarifying the points that I and other reviewers raised. I believe that the paper now is more comprehensive. 
Following the authors comments I decided to raise the score.

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Any further feedback is welcomed
comment: Dear Reviewer 7JBh,

Thank you again for your encouraging comments and insightful suggestions. We have uploaded revised papers containing new empirical findings (e.g., Tables 2, 3, and 4) as well as refined analyses and writing. Since the deadline for rebuttals is quickly approaching, we welcome any additional feedback and will do our best to address it in the final days.

Best,

The Authors

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Sufficiency for publication
comment: Dear Reviewer wGpR,

In our most recent responses, we demonstrated the memory efficiency of BAFFLE, which could be useful for a wide range of FL deployments on edge devices. We hope that these new results will persuade you that our work is of value to the community and sufficient (might requires further polishing in the final revision) for publication.

Best,

The Authors

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Looking forward to feedback
comment: Dear Reviewer EJwi,

Thank you for your insightful comments and suggestions. We have uploaded revised papers containing new empirical findings (e.g., Tables 2, 3, and 4) as well as refined analyses and writing. Since the deadline for rebuttals is quickly approaching, we would appreciate your feedback on whether our responses address your concerns. In the final days, we will make every effort to address any remaining questions.

Best,

The Authors

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Full results of memory efficiency
comment: In the new revision, we add full results of memory efficiency to Table 4, along with a more precise memory cost calculation. Even though our code is not specially optimized, the memory cost ratio between BP-based training and BAFFLE is still consistently less than 10\% across different architectures and datasets.

In practice, memory efficiency makes BAFFLE a competitive option. For instance, BAFFLE **reduces memory costs by approximately 95\%** (from 5271MB to 289MB, as reported in Table 4) while **sacrificing only 1\%$\\sim$3\% accuracy** (as reported in Table 3). This should be qualified as an important gain from the application of BAFFLE.

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Thank you for your response
comment: We greatly appreciate your timely response. BAFFLE is a communication-efficient protocol, but indeed convergence may require additional training iterations.

However, we must clarify that the most important characteristic of BAFFLE is **memory efficiency** for edge devices:

- **Static memory efficiency.** Executing BP for deep networks uses auto-differentiation in ML packages (e.g., TF, PyTorch or JAX). Installing any of them takes more than 1GB (typically 2$\\sim$3GB) of static memory, which is a significant burden for edge devices such as IoT sensors. BAFFLE, on the other hand, only requires forward propagation, which could be executed by build-in matrix computing.

- **Dynamic memory efficiency.** Executing BP for deep networks requires the restoration of intermediate states in dynamic memory. Using our code, BP-based training for WRN-10-2 takes *2432MB* GPU memory and MobileNet takes *8934MB*; in contrast, BAFFLE training for WRN-10-2 takes *117MB* and MobileNet takes *252MB*. As can be seen, BAFFLE's dynamic memory cost is only 3\%$\\sim$5\% of that of BP-based training. Please note that we DID NOT specifically optimize memory efficiency in our code, even though the memory cost ratio could ideally be as low as 1\%.

Furthermore, many practically deployed models are quantized (e.g., 8-bit or 4-bit) and optimized for inference-only propose. These significantly accelerate forward propagation and reduce the computational time required by BAFFLE.

From: ICLR.cc/2023/Conference/Paper1664/Reviewer_wGpR
title: Response to authors 
comment: Thank you for your detailed response and updates to the paper. I think the paper is better now and raised its score, but I  still don't think it is sufficient for publication. 

I do think it is an application of method A to domain B and as such has limited novelty. This isn't a reason to reject a paper, but in my mind for such a paper to get accepted it should show some important gain from the application of the said method. 

As it stands, your method is useful in a very narrow domain - when saving in communication is much more important than running time on the client and you are willing to sacrifice performance for it.

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Looking forward to further feedback
comment: Dear Reviewers,

Thank you again for your valuable comments and suggestions, which are really helpful for us. We have uploaded a revision (with new empirical results) and posted responses to the proposed concerns.

We totally understand that this is quite a busy period, since the reviewers may be preparing rebuttal for their own submissions or rushing for the deadline of recent conferences.

We deeply appreciate it if you could take some time to return further feedback on whether our responses solve your concerns. If there are any other comments, we will try our best to address them.

Best,

The Authors

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Thank you for your supportive review
comment: Thank you for your supportive review and suggestions. We have uploaded a revision of our paper and highlighted the revised parts in purple.

***Question 1: Noticeable drop in the model accuracy***

Thank you for your understanding, there is indeed accuracy drop compared to conventional FL. To extend our empirical exploration of BAFFLE, in the updated revision we add experiments as

- Update results of training LeNet and WRN-10-2 from scratch on MNIST and CIFAR-10/100 in the iid scenario (Table 1)
- Add results of training LeNet and WRN-10-2 from scratch CIFAR-10/100 in the label non-iid scenario (Table 2)
- Add results of finetuning ImageNet-pretrained MobileNet onto the OfficeHome dataset in the feature non-iid scenario (Table 3)

***Question 2: Some information is missing***

In the first paragraph of Section 4, we add more details on the iid participation, label non-iid participation, and feature non-iid participation. We split the data into a train set and a validation set with the ratio $80\\%/20\\%$, then we adjust hyperparameters on the validation set. In Section 4.1, we provide specifics regarding the selection of the perturbation scale $\sigma$ and number $K$, as well as four empirical guidelines for enhancing the accuracy-efficiency balance in BAFFLE.

As to the Top-1 accuracy on CIFAR-100, for example, in Table 1 with $K=1000$, LeNet achieves $28.55\\%$ accuracy and the ground-truth is $35.49\\%$; WRN-10-2 achieves $31.17\\%$ accuracy and the ground-truth is $40.84\\%$. We will document the full Top-1 accuracy of our experiments and update them in the final revision.

***Question 3: Softplus as an alternative to ReLU***

Actually, we have evaluated four activation functions as alternatives to ReLU, including SELU, Hardswish, Softplus, and GELU. The performance rank is Hardswish $\approx$ Softplus $\approx$ GELU $>$ ReLU $\gg$ SELU. We choose Hardswish because it does not need to compute $\exp,\log$ and is easy to implement on edge devices.

***Question 4: Performance personalized FL and comparison with more recent baselines***

Thank you for the suggestion. BAFFLE focuses on the computation of gradients, and FedAvg is its most direct baseline. Conceptually, BAFFLE is compatible with more recent FL methods and personalized FL, i.e., we may simply substitute BP-based modules with BAFFLE. We did not explore these intriguing scenarios further in this paper, but will do so in future research.

***Question 5: Performance under 100 or 500 clients***

In the updated revision, we add the experiments under non-iid scenarios with the number of clients $C=100$ on CIFAR-10/100 (Table 2) and $C=40$ on ImageNet-to-OfficeHome (Table 3).

***Question 6: Computation gain***

In Section 4.4, we add discussions on the computation efficiency and memory efficiency of BAFFLE compared to conventional FL. Briefly, BAFFLE results in approximately $(\frac{K}{5}-1)$ times extra computational cost but reduce the memory cost to approximately $1\\%$.

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Thank you for your valuable review
comment: Thank you for your valuable review and suggestions. We have uploaded a revision of our paper and highlighted the revised parts in purple.

***Question 1: Limited novelty***

To the best of our knowledge, we are the first work to 1) discuss the promising properties of BP-free FL; 2) propose a prototype of BP-free FL (BAFFLE) in Algorithm 1 with the detailed communication pipeline and secure aggregation; and 3) provide empirical results (updated in the revision) showing the potential of BP-free FL.

We assert that our work is NOT merely a casual application of method A to domain B. Instead, we are motivated by the memory-efficient and easy-to-implement properties of zero-order optimization, which are well-suited for edge devices but underexplored in the FL literature.

We did not expect BP-free FL to completely supplant conventional FL. Instead, BP-free FL could be an optional choice in practical situations, such as fine-tuning an ImageNet-pretrained Mobilenet onto the multi-domain dataset (e.g., OfficeHome) using low-memory edge devices, as our updated experiments demonstrate in Table 3.

***Question 2: Limited experimentation***

In the updated revision, we experiment on three network architectures: 1) **LeNet** with two layers ($2.7\times 10^4$ parameters); 2) **WRN-10-2** with $\text{depth}=10$ and $\text{width}=2$ ($3.0\times 10^5$ parameters) and 3) **MobileNet** as the industrial model ($1.3\times 10^7$ parameters) that works on ImageNet.

We present the empirical results of training LeNet and WRN-10-2 from scratch on MNIST and CIFAR-10/100, as well as finetuning ImageNet-pretrained MobileNet onto OfficeHome.

Using the FedLab participation tools, we simulate three popular FL scenarios: 1) **iid participations**; 2) **label non-iid participations** and 3) **feature non-iid participations**. We further tune our parameter setups and provide updated results in Tables 1-3 and Figures 2-3.

***Question 3: Inexact/misleading theoretical results***

As you suggested, in Section 3.3 we update Theorem 2 with dimension factor $n$, where the empirical covariance converges to the identity matrix (in the sense of operator 2-norm) at a rate of $\mathcal{O}(\sqrt{\frac{n}{K}})$.

***Question 4: Exaggerated claims***

In the revised abstract and introduction, we tone down the language and eliminate words such as comparable. We clarify that BAFFLE’s accuracy is suboptimal compared to conventional FL, but could be acceptable as an initial exploration in this direction.

Existing inference attacks typically require accurate inference results such as confidence values or classification scores for successful evasion. Given that BAFFLE provides stochastic loss differences $\Delta \mathcal{L}(\mathbf{W},\delta;\mathbb{D})$ associated with the random perturbation $\delta$, we compare the distribution of $\Delta \mathcal{L}$ between validation data and random noise, and demonstrate in Figure 4 that it is challenging for off-the-shelf inference attacks to obtain useful information from BAFFLE's outputs. However, we clarify that **adaptively designed attacking strategies are possible to evade BAFFLE**.

***Question 5: Unclear use of TEE***

TEE is part of our motivation for using BP-free FL. Specifically, TEE can isolate models into a black-box secure area and avoid white-box vulnerability; however, TEE is memory-constrained, whereas BP requires high memory to restore intermediate states. BP-free FL is memory-efficient and could be more compatible with TEE infrastructure for this reason. We provide more details in Appendix C and will further clarify this part in the revision.

From: ICLR.cc/2023/Conference/Paper1664/Authors
title: Thank you for your valuable review
comment: Thank you for your valuable review and suggestions. We have uploaded a revision of our paper and highlighted the revised parts in purple.

***Question 1: Lack of novelty***

To the best of our knowledge, we are the first work to 1) discuss the promising properties of BP-free FL; 2) propose a prototype of BP-free FL (BAFFLE) in Algorithm 1 with the detailed communication pipeline and secure aggregation; and 3) provide empirical results (updated in the revision) showing the potential of BP-free FL.

We assert that our work is NOT merely a casual application of method A to domain B. Instead, we are motivated by the memory-efficient and easy-to-implement properties of zero-order optimization, which are well-suited for edge devices but underexplored in the FL literature.

We did not expect BP-free FL to completely supplant conventional FL. Instead, BP-free FL could be an optional choice in practical situations, such as fine-tuning an ImageNet-pretrained Mobilenet onto the multi-domain dataset (e.g., OfficeHome) using low-memory edge devices, as our updated experiments demonstrate in Table 3.

***Question 2: Lack of convergence analysis***

In Section 3.3, we update convergence analysis, where the empirical covariance converges to the identity matrix at a rate of $\mathcal{O}(\sqrt{\frac{n}{K}})$. According to previous research, a noisy estimation of the gradient can still be used to train a generalizable model in practice.

***Question 3: Lack of guidance on parameter choices and computational comparison***

In Section 4.1, we describe parameter selection details for the perturbation scale $\sigma$ and number $K$. In experiments, we assess the effect of $K$ over a wide range, from $100$ to $5000$. We propose four empirical guidelines for improving the accuracy-efficiency balance in BAFFLE and conduct ablation studies as shown in Table 1 and Figure 3.

***Question 4: Limited experiments: shallow networks, iid environment***

In the updated revision, we experiment on three network architectures: 1) **LeNet** with two layers ($2.7\times 10^4$ parameters); 2) **WRN-10-2** with $\text{depth}=10$ and $\text{width}=2$ ($3.0\times 10^5$ parameters) and 3) **MobileNet** as the industrial model ($1.3\times 10^7$ parameters) that works on ImageNet.

We present the empirical results of training LeNet and WRN-10-2 from scratch on MNIST and CIFAR-10/100, as well as finetuning ImageNet-pretrained MobileNet onto OfficeHome.

Using the FedLab participation tools, we simulate three popular FL scenarios: 1) **iid participations**; 2) **label non-iid participations** and 3) **feature non-iid participations**. We further tune our parameter setups and provide updated results in Tables 1-3 and Figures 2-3.

From: ICLR.cc/2023/Conference/Paper1664/Reviewer_EJwi
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a zeroth order Federated Learning method BAFFLE using forward only computations to replace back-propagation. The method intends to reduce memory storage and strengthen privacy against white-box vulnerability. The soundness of the finite difference method is justified by theory, experiments show that the method achieve comparable performance to conventional FL methods.    
strength_and_weaknesses: Strength: 
- clear presentation

Weakness: 
- Lack of novelty 
- Lack of convergence analysis 
- Lack of guidance on parameter choices and computational comparison 
- Limited experiments: shallow networks, iid environment 
clarity,_quality,_novelty_and_reproducibility: The paper is presented clearly and it is easy to follow.
summary_of_the_review: The paper proposes a zeroth order Federated Learning algorithm which is back propagation free. The main idea is to replace the gradient evaluation by zeroth order finite difference. This is very well known in zeroth order optimization literature. The more challenging part is to provide guidance on how to set the discretization stepsize and the number of samplings where the paper fail to provide any theoretical guidance. In particular, the more samples we draw, the more accurate the gradient estimation is, but the computational expenses will increase. The balance between the precision on the gradient estimation and the computational cost is a key for zeroth order method, which is a major omission of the paper. 

Besides the lack of convergence analysis, the experiment is also insufficient. Only shallow network are considered (two layers), it is expected that when the network goes deeper the gradient is more difficult to be estimated by zeroth order method (noise level increase), hence it will not be as competitive as gradient base method. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1664/Reviewer_wGpR
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper presents a method to train federated learning using zero-order optimization. They use a gradient-free optimization approach and evaluate it on several benchmarks.
strength_and_weaknesses: Strengths:
- The problem the paper addresses is important. A common use case for FL is with clients with limited computing. As such, the reduction in memory by avoiding backdrop can be significant.
- The paper is clear and easy to follow

Weaknesses:
- Limited novelty: Mostly applies an existing zero-order optimization algorithm to FL
- Limited experimentation: Only uses LeNet, which has very bad performance on cifar10/100 even for the ground truth model and WRN only on MNIST. Not enough to be convincing.
- Inexact/misleading theoretical results: The authors ignore the dimension factor in both theorem 1&2 which is a critical factor especially as it is much larger than $\sqrt{k}$. Also, in theorem 2 what we care about is the operator 2 norm of $\hat{\Sigma}$, so just the fact that each element has $1/\sqrt{K}$ std around the desired value isn't enough (again will depend on the dimension). Ignoring the dimension is misleading, as there is no theoretical justification to use this approach with the given K values. The approach may empirically work well, but the theory doesn't support it.
- Exaggerated claims: 
   The authors in several cases raise claims that, in my opinion, are not supported by the empirical data. For example, they claim that 
   "BAFFLE achieves comparable performance to conventional FL using a relatively small value of K". In CIFAR10/100 we see about 7% 
   drop in accuracy even when using K=500, i.e. 1000 forward passes. Running 1000 forward passes on low resource compute is a 
   considerable amount and 7% drop is a considerable degradation in performance. 

   They also "assert that it is difficult to perform inference attack on BAFFLE". The support for that is very limited at best. They show the 
   distribution of $\Delta \mathcal{L}$ is the same between val data and random noise, but I do not agree this shows robustness to 
   inference attacks. The model is trained during many update steps and has to acquire information regarding the task to succeed. The 
   authors need to show that reasonable attacks fail to claim such robustness. 
- Unclear use of TEE. TEE from my understanding is part of the infrastructure to run FL and is agnostic to the exact algorithm. As such, it is not clear why is it important to include as part of the work. If the TEE details are relevant, it should be explained better.

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, although I would claim has misleading statements, and easy to follow. The quality of the experimental part is low and the novelty is small. The work is reproducible.
summary_of_the_review: The authors do now show empirical evidence to support their claims and as such I do not think this paper should be accepted.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1664/Reviewer_7JBh
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose BAFFLE, backpropagation-free federated learning (FL) framework that uses multiple forward processes instead
of backpropagation. The main idea is to use Monte-Carlo approximation for stein's identity to approximate the analytical gradient. BAFFLE is mainly designed for two specific setups: low-resource edge devices, and FL systems with malicious clients. The authors prove convergence analysis for their estimator and show the effectiveness of their method in several FL setups (standard classification, robustness to attacks, and communication efficiency).
strength_and_weaknesses: Strength:
* First, it is written clearly and relatively easy to follow. 
* Second, the goal of the paper (addressing cases of low-resource end devices and robustness to malicious clients) is important and refreshing. 
* Third, the method, which is relatively simple, is constructed in a thoughtful way to adjust for FL setups. It seems to work in a descent way compared to the standard FedAvg. 
* Forth, although not mandatory in my opinion, Thm 1 & 2 are a nice addition. 

Weaknesses/Questions:
* As the authors stated, there is a noticeable drop in the model accuracy compared to FedAvg, but in my opinion it is fine for a first paper of this kind.
* Some information is missing, how did you perform the data split between the clients? There are multiple strategies for that. How did you optimized the hyper-parameters of the model? Was there a validation set? The authors should present the Top-1 results for CIFAR-100 for completeness (even though they are probably not so good). 
* Did you consider Softplus as a continuously differentiable alternative to Relu?
* Did you test how will the model perform in Personalized FL? It is especially interesting to show plots similar to those in Fig. 4 for real datasets that come from a different class distribution (as in PFL).
* How will the model perform under a more realistic FL setup such as having 100 or 500 clients?
* Can you quantify the computation gain (in terms of FLOPS or wall-clock time) compared to standard backpropagation?
* A comparison to more recent baselines other than FedAvg is missing in my opinion.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, has novel contributions and the results are reproducible (code was provided).
summary_of_the_review: In my opinion this is a nice paper that deals with an interesting learning setup. The proposed solution is elegant and seems to be relatively efficient.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper