Paper:
Parameter-transfer is a well-known and versatile approach for meta-learning, with applications including few-shot learning, federated learning, and reinforcement learning. However, parameter-transfer algorithms often require sharing models that have been trained on the samples from specific tasks, thus leaving the task-owners susceptible to breaches of privacy. We conduct the first formal study of privacy in this setting and formalize the notion of task-global differential privacy as a practical relaxation of more commonly studied threat models. We then propose a new differentially private algorithm for gradient-based parameter transfer that not only satisfies this privacy requirement but also retains provable transfer learning guarantees in convex settings. Empirically, we apply our analysis to the problems of federated learning with personalization and few-shot classification, showing that allowing the relaxation to task-global privacy from the more commonly studied notion of local privacy leads to dramatically increased performance in recurrent neural language modeling and image classification.

Review:
This paper considers the problem of achieving formal privacy guarantees in the context of parameter-sharing meta learning. This is a problem that has been studied recently, although generally not under the exact record-level task-privacy model studied in this paper (what they call global task privacy). The problem is well-motivated: since in meta learning we want to leverage information from similar tasks to increate data efficiency, there may be privacy concerns for each of the task owners about both other task owners and the aggregator (meta-learner). 
Their approach is conceptually simple; they use standard private stochastic gradient descent within each task to provide task level privacy. In combination with post-processing and composition guarantees this gives privacy for the overall mechanism. They are able to show theoretical guarantees via the standard accuracy guarantees of private SGD and no-regret
guarantees of OCO. They show a bound on the expected transfer risk:  
O(V/sqrt(m) + 1/Tsqrt(m)) + o(1/sqrt(m)) which is close to the non-private bound from Denevi (2019).
They evaluate the empirical performance of these models via a transfer learning setting  where they are training a deep RNN for next word prediction on two large corpi, and the tasks correspond to individual articles. They show that the private variate of Reptile is competitive with the non-private variant in these settings. 
A few comments 
-	It is odd to just fix epsilon = 9.2 instead of showing a Pareto curve. Why this particular value?
-	Simpler (convex) experiments to illustrate the theoretical guarantees would improve the paper
Overall I like the motivation and the theory results are solid, if not a bit obvious. However, due to the lack of novelty in any of the applied techniques, and the fact that the experiments could be expanded, I recommend a weak accept.