Paper:
Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures. Our experiments indicate that FedMA outperforms popular state-of-theart federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, while improving the communication efficiency.

Review:
Post Rebuttal Summary
---------------------------------
I have nudged my score up to an "Accept", based on my comments to the rebuttal below. I hope the authors continue to improve the readability of Sec. 2.1
Review Summary
--------------
Overall I think this is almost above the bar to be accepted, and I could be persuaded with a strong rebuttal.  The strengths here are the extensive experiments and the easy-to-implement method. The primary weakness of this paper is that it is a "straightfoward" way to extend the BBP-MAP method to CNNs and RNNs, so the methodological novelty is weak relative to the BBP-MAP past work (Yurochkin et al. ICML 2019). Other technical weaknesses limit the ability to use this method on clients with diverse class distributions, which will be common in real deployments.
Paper Summary
-------------
This paper addresses the problem of federated learning, where J separate "clients" with disjoint datasets each train a neural network model for a supervised problem, and then try to aggregate all J individual client models into one "global model" in a coherent way. The natural problem is that due to hidden units being permutable within one network, naively taking parameter averages across two client models will lead to bad accuracy without first coming up with a consistent ordering of the units in each layer. 
Previous work (Yurochkin et al. ICML 2019) has developed a Bayesian nonparametric model based on the Beta-Bernoulli Process (BBP) for the case of federated learning of multi-layer perceptrons. However, the extension to convolutional layers or recurrent layers has yet to be solved, which is the focus of this paper. 
This paper's algorithm (Federated Matched Averaging (FedMA), see Alg 1), proceeds by iteratively stepping thru the CNN or RNN layer by layer greedily from input to output. At each layer, we first solve a BBP-MAP optimization (bipartite matching using a BBP maximum a-posteriori objective as cost function, a subprocedure taken direclty from Yurochkin et al.). This obtains a consistent low-cost permutation for each client model. Then, the global model weights for that layer is the average of the aligned client weights. After the current layer update, each client keeps training, keeping all layers up to the current frozen but revising later layers. This layer-by-layer training can be applied to both CNNs and RNNs.
The proposed approach is compared to FedAvg and FedProx on MNIST and CIFAR image classification tasks with CNNs, and Shakespeare text classification tasks with RNNs. Later experiments explore the effect of communication efficiency (MB transfered between client and master), effect of local training epochs, handling biased class distributions, and interpretabilty.

Novelty & Significance
-----------------------
Solving federated learning problems is of increasing practical importance, and certainly trying to do so for CNNs and RNNs (more than just large MLPs) is important. So I like where the paper is going.
Although the method is "new", it is more or less a straightforward extension of work by Yurochkin et al. (ICML 2019) to CNNs and RNNs. If you read the last few sentences of Yurochkin et al., you'll see "Finally, it is of interest to extend our model-ing framework to other architectures such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The permutation invariance necessitating matching inference also arises in CNNs since any permutation of the filters results in the same output, however additional bookkeeping is needed due to the pooling operations." I view this paper as a well-executed implementation of this "bookkeeping". Certainly not trivial, but to some readers perhaps not clearly "above the bar" for a top conference like ICLR.
Technical Concerns
------------------
## Concern 1: Client models will not always be alignable after permutation
My first concern is that there will not always be a one-to-one permutation of the neurons learned by two client models with different class distributions. Given fixed capacity at each layer, some clients may learn a filter for "horse hooves" (esp. if horse images are common to that client), while other clients may learn a filter for "snake skin" (if snakes are more common to that client). I wonder if we can quantify how well the aligned filters match in practice, and if there is any benefit to revising the alignment to allow some client-specific customizations (e.g. by having the global model can learn more units than the client model). 
## Concern 2: Use of the BBP-MAP subprocedure poorly motivated
The paper prioritizes a clean and easy-to-implement algorithm to resolve practical alignment issues between client CNN and RNN models. However, I was a bit underwhelmed that the BBP-MAP solution used by Yurochkin et al. was treated as a black-box subprocedure without much justification. I could see 2 preferable alternatives to the current use of BBP-MAP. Either a simpler approach using Eq. 2 with a squared error cost and the Munkres algorithm to solve bipartitite matching to obtain the permutation (which seems more in spirit of the rest of the paper). Or, a more sophisticated probabilistic approach (taking a Bayesian hierarchical model from Yurochkin et al. seriously and forming the estimated global weights from a weighted sums that includes both the clients (weighted by dataset size) and the assumed prior). As it is, I feel the BBP-MAP subprocedure in the current Algorithm 1 is poorly motivated for the task at hand.
Experimental Evaluation
-----------------------
Overall the experiments were extensive and demonstrated several apparent advantages (reduced need to transfer large memory during communication, etc.). 
Minor Presentation Concerns
---------------------
Before Eq. 2, you should introduce the "\theta" notation
I'm a bit confused about how "FedMA" differs from "FedMA with communication", even after reading Sec. 2.3. How exactly are communicate costs kept down? What are you sending from master to client at beginning of every "round" if not the full global model (all weights of the CNN)?