Paper:
Federated learning is a distributed, privacy-aware learning scenario which trains a single model on data belonging to several clients. Each client trains a local model on its data and the local models are then aggregated by a central party. Current federated learning methods struggle in cases with heterogeneous client-side data distributions which can quickly lead to divergent local models and a collapse in performance. Careful hyper-parameter tuning is particularly important in these cases but traditional automated hyper-parameter tuning methods would require several training trials which is often impractical in a federated learning setting. We describe a two-pronged solution to the issues of robustness and hyper-parameter tuning in federated learning settings. We propose a novel representation matching scheme that reduces the divergence of local models by ensuring the feature representations in the global (aggregate) model can be derived from the locally learned representations. We also propose an online hyper-parameter tuning scheme which uses an online version of the REINFORCE algorithm to find a hyper-parameter distribution that maximizes the expected improvements in training loss. We show on several benchmarks that our two-part scheme of local representation matching and global adaptive hyper-parameters significantly improves performance and training robustness.

Review:
This manuscript proposes two strategies to improve both the robustness and accuracy of local agents under the setting of federated learning. Specifically, online reinforcement learning is used to perform adaptive hyperparameter search in order to maximize the utility of local models. This is quite an interesting idea since traditional hyperparameter tuning techniques, including random search or Bayesian optimization, usually requires access to monolithic dataset, which is clearly impractical under federated learning. The second contribution is an idea on using local distribution matching in order to synchronize the learning trajectories of different local models. This again is a novel and interesting idea. Overall the paper is well-written and clear to follow, which is a plus. Detailed comments and questions follow:
-   I understand that change of hyperparameter in local model affects the global model during in model aggregation stage. However, under the federated learning setting, if the number of local models is huge, then the influence of a single local model should be small. Hence instead of using online reinforcement learning for hyperparameter search, which is notoriously data-inefficient, why not framing the problem as a pure online learning problem? This helps to increase the data efficiency and also increases the stability of learning. 
-   In the design of matching network, is there any intuition why performance significantly improves if the activations of
one layer are instead derived from the activations of the next layer of interest above it? Furthermore, why we need to have additional model with parameters $\theta_i$ for alignment? Intuitively if we want the new model $M$ to be close to the original one $M^F$, shouldn't we just use some distance measure, e.g., $\ell_2$ norm, to measure the distance of feature activations in the corresponding layers directly? 
-   The experiments are illustrative, but might be too toyish under the federated learning setting. I appreciate the ablation studies the authors performed to show the relative impact of different strategies, which makes the relative contributions more clear. However, even in the case of MNIST and CIFAR, the improvement over baseline FA is not very significant. In this case, it would be better if the authors could also report the computational overhead in terms of running time.