From: NeurIPS.cc/2022/Workshop/Federated_Learning/Program_Chairs
title: Paper Decision
decision: Accept (Poster)

From: NeurIPS.cc/2022/Workshop/Federated_Learning/Paper71/Reviewer_Mc6e
title: The paper proposes FedorAS, a resource-aware Neural Architecture Search framework for developing models in a cross-device Federated Learning system. The clients are split into tiers based on their resource availability. They employ One-shot neural architecture search on each client where the sub-paths are sampled based on the resource availability of the clients and limiting the number of operations updated based on communication limits.
review: Strengths - 
1.	Using NAS to find good models in a heterogeneous FL setting is a good research direction.
2.	The paper is well written and easy to read.

Weaknesses - 
1.	The paper does not seem to have new contributions or observations.
2.	Many explanations are lacking.
3.	The evaluation section is not convincing.

Details - 
1.	Based on the description of the work, it seems the only novel addition compared to other state-of-the-art is Equation 1. The rest has been seen in one form or another in other works (e.g. OPA is just weighted FedAvg with sub-sampled operations).
2.	The idea of using Tiers/clusters based on resource availability is not new in FL and has been explored quite extensively, but the authors do not cite any of the works.
3.	Please explain LDA and what varying it denotes.
4.	Why are the state-of-the-art comparisons not consistent? For example, why do Tables 2 and 3 not have FJORD comparison?
5.	Also, how are the FJORD tiers set?
6.	Some discussions on why the overall accuracies are significantly better than the state-of-the-art would be good to have. It is unclear what part of the FedorAS causes this increase.
7.	Why are there no comparisons with state-of-the-art in the data heterogeneity experiments?
8.	Apart from FLOPs, one important aspect is how long the NAS takes to achieve a good model. This is especially important for cross-device FL since the cost is borne by the user devices. The longer they are used, the more cost is incurred and this must be kept at a minimum. How does FedoraAS compare to state-of-the-art in this aspect?
9.	Overall, the idea is interesting but the evaluation results are not conclusive. Additionally, the authors need to add more explanations as to why their system is distinguishable from others.

From: NeurIPS.cc/2022/Workshop/Federated_Learning/Paper71/Reviewer_HadS
title: Good idea and concrete results.  
review: Summary: 
This paper propose a NAS approach for resource-constrained and heterogeneity-aware federated learning. The idea involves a two-level sampling both on server and device sides to reduce the search computation complexity. The results demonstrate that the proposed approach can improve the accuracy of local models with less computation complexity and communication overhead. 

Strengths: 
- This is good work which proposes to optimize both communication and computation costs.
- The paper provides sufficient results. 

Weaknesses: 
- The authors need to improve the writing quality to make the approach and results sections more clear. 
- The authors mention that there’s a global validation set; however, in practice, the creation of such a dataset my bring some privacy concerns.