Original: P RIVATE M ULTI -T ASK L EARNING : F ORMULATION AND A PPLICATIONS TO F EDERATED L EARNING Anonymous authors
Revised: P RIVATE M ULTI -T ASK L EARNING : F ORMULATION AND A PPLICATIONS TO F EDERATED L EARNING Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many problems in machine learning rely on multi-task learning (MTL) , in which the goal is to solve multiple related machine learning tasks simultaneously.
Revised: Many problems in machine learning rely on multi-task learning (MTL) , in which the goal is to solve multiple related machine learning tasks simultaneously.

Original: MTL is particularly relevant for privacy-sensitive applications in areas such as healthcare, ﬁnance, and IoT computing, where sensitive data from multiple, varied sources are shared for the purpose of learning.
Revised: MTL is particularly relevant for privacy-sensitive applications in areas such as healthcare, ﬁnance, and IoT computing, where sensitive data from multiple, varied sources are shared for the purpose of learning.

Original: In this work, we formalize notions of task-level privacy for MTL via joint differential privacy (JDP), a relaxation of differential privacy for mechanism design and distributed optimization.
Revised: In this work, we formalize notions of task-level privacy for MTL via joint differential privacy (JDP), a relaxation of differential privacy for mechanism design and distributed optimization.

Original: We then propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP.
Revised: We then propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP.

Original: We analyze our objective and solver, providing certiﬁable guarantees on both privacy and utility.
Revised: We analyze our objective and solver, providing certiﬁable guarantees on both privacy and utility.

Original: Empirically, we ﬁnd that our method allows for improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks.
Revised: Empirically, we ﬁnd that our method allows for improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-task learning (MTL) aims to solve multiple learning tasks simultaneously while exploiting similarities/differences across tasks (Caruana, 1997).
Revised: Multi-task learning (MTL) aims to solve multiple learning tasks simultaneously while exploiting similarities/differences across tasks (Caruana, 1997).

Original: Multi-task learning is commonly used in applications that warrant strong privacy guarantees.
Revised: Multi-task learning is commonly used in applications that warrant strong privacy guarantees.

Original: For example, MTL has been used in healthcare, as a way to learn over diverse populations or between multiple institutions (Baytas et al., 2016;
Revised: For example, MTL has been used in healthcare, as a way to learn over diverse populations or between multiple institutions (Baytas et al., 2016;

Original: Suresh et al., 2018; Harutyunyan et al., 2019); in ﬁnancial forecasting, to combine knowledge from multiple indicators or across organizations (Ghosn & Bengio, 1997; Cheng et al., 2020); and in IoT computing, as an approach for learning in federated networks of heterogeneous devices (Smith et al., 2017;
Revised: Suresh et al., 2018; Harutyunyan et al., 2019); in ﬁnancial forecasting, to combine knowledge from multiple indicators or across organizations (Ghosn & Bengio, 1997; Cheng et al., 2020); and in IoT computing, as an approach for learning in federated networks of heterogeneous devices (Smith et al., 2017;

Original: Hanzely & Richt ´ arik, 2020; Hanzely et al., 2020; Ghosh et al., 2020; Sattler et al., 2020;
Revised: Hanzely & Richt ´ arik, 2020; Hanzely et al., 2020; Ghosh et al., 2020; Sattler et al., 2020;

Original: Deng et al., 2020; Mansour et al., 2020).
Revised: Deng et al., 2020; Mansour et al., 2020).

Original: While MTL can signiﬁcantly improve accuracy when learning in these applications, there is a dearth of work studying the privacy implications of multi-task learning.
Revised: While MTL can signiﬁcantly improve accuracy when learning in these applications, there is a dearth of work studying the privacy implications of multi-task learning.

Original: In this work, we develop and theoretically analyze methods for MTL with formal privacy guarantees.
Revised: In this work, we develop and theoretically analyze methods for MTL with formal privacy guarantees.

Original: Motivated by applications in federated learning, we aim to provide task-level privacy 1 , where each task corresponds to a client/device/data silo, and the goal is to protect the sensitive information in each task’s data (McMahan et al., 2018).
Revised: Motivated by applications in federated learning, we aim to provide task-level privacy 1 , where each task corresponds to a client/device/data silo, and the goal is to protect the sensitive information in each task’s data (McMahan et al., 2018).

Original: We focus on incorporating differential privacy (DP) (Dwork et al., 2006), which (informally) requires an algorithm’s output to be insensitive to the change of any single entity’s data.
Revised: We focus on incorporating differential privacy (DP) (Dwork et al., 2006), which (informally) requires an algorithm’s output to be insensitive to the change of any single entity’s data.

Original: For MTL, using task-level DP directly would require the entire set of predictive models across all tasks to be insensitive to changes in the private data of any single task.
Revised: For MTL, using task-level DP directly would require the entire set of predictive models across all tasks to be insensitive to changes in the private data of any single task.

Original: This requirement is too stringent for most applications, as it implies that the predictive model for task k must have little dependence on the training data for task k , thus preventing the usefulness of the model (see Figure 1).
Revised: This requirement is too stringent for most applications, as it implies that the predictive model for task k must have little dependence on the training data for task k , thus preventing the usefulness of the model (see Figure 1).

Original: To circumvent this limitation, we leverage a meaningful relaxation of DP known as joint differential privacy (JDP) (Kearns et al., 2014), which requires that for each task k , the set of output predictive models for all other tasks except k is insensitive to k ’s private data.
Revised: To circumvent this limitation, we leverage a meaningful relaxation of DP known as joint differential privacy (JDP) (Kearns et al., 2014), which requires that for each task k , the set of output predictive models for all other tasks except k is insensitive to k ’s private data.

Original: As a consequence, the client’s private data in task k is protected even if all other clients/tasks collude and share their private data and output models (as long as client k keeps their data private).
Revised: As a consequence, the client’s private data in task k is protected even if all other clients/tasks collude and share their private data and output models (as long as client k keeps their data private).

Original: In contrast to standard DP, JDP allows the predictive model for task k
Revised: In contrast to standard DP, JDP allows the predictive model for task k

Original: to depend on k ’s private data, helping to preserve the task’s utility.
Revised: to depend on k ’s private data, helping to preserve the task’s utility.

Original: Using JDP, we then develop new learning algorithms for MTL with rigorous privacy and utility guarantees.
Revised: Using JDP, we then develop new learning algorithms for MTL with rigorous privacy and utility guarantees.

Original: Speciﬁcally, we propose Private Mean-Regularized MTL, a simple framework for learning
Revised: Speciﬁcally, we propose Private Mean-Regularized MTL, a simple framework for learning

Original: Algorithm
Revised: Algorithm

Original: Dwww m
Revised: Dwww m

Original: D
Revised: D

Original: D m
Revised: D m

Original: Algorithm
Revised: Algorithm

Original: D ′w ′
Revised: D ′w ′

Original: 1 ≈ w 1 w ′ 

Revised: 1 ≈ w 1 w ′

Original: ≈ ww ′ m ≈ w m
Revised: ≈ ww ′ m ≈ w m

Original: D
Revised: D

Original: D m
Revised: D m

Original: Algorithm
Revised: Algorithm

Original: D ′w ′ 1 ≉ w 1 w ′ 

Revised: D ′w ′ 1 ≉ w 1 w ′

Original: ≈ ww ′ m ≈ w m
Revised: ≈ ww ′ m ≈ w m

Original: D
Revised: D

Original: D m multiple tasks while ensuring task-level privacy.
Revised: D m multiple tasks while ensuring task-level privacy.

Original: We show that our method achieves ( (cid:15), δ ) -JDP.
Revised: We show that our method achieves ( (cid:15), δ ) -JDP.

Original: Our scalable solver builds on FedAvg (McMahan et al., 2017), a common method for communicationefﬁcient federated optimization.
Revised: Our scalable solver builds on FedAvg (McMahan et al., 2017), a common method for communicationefﬁcient federated optimization.

Original: We analyze the convergence of our solver on both nonconvex and convex objectives, demonstrating a tradeoff between privacy and utility, and evaluate this trade-off empirically on multiple federated learning benchmarks.
Revised: We analyze the convergence of our solver on both nonconvex and convex objectives, demonstrating a tradeoff between privacy and utility, and evaluate this trade-off empirically on multiple federated learning benchmarks.

Original: We summarize our contributions below:
Revised: We summarize our contributions below:

Original: • Our work is the ﬁrst we are aware of to provide formal deﬁnitions of task-level differential privacy for multi-task learning objectives (Section 3).
Revised: • Our work is the ﬁrst we are aware of to provide formal deﬁnitions of task-level differential privacy for multi-task learning objectives (Section 3).

Original: Our deﬁnitions rely on joint differential privacy and are applicable to commonly-used multi-task relationship learning objectives.
Revised: Our deﬁnitions rely on joint differential privacy and are applicable to commonly-used multi-task relationship learning objectives.

Original: • Using our privacy deﬁnitions, we propose Private Mean-Regularized MTL, a simple MTL framework that provides task-level privacy (Section 4).
Revised: • Using our privacy deﬁnitions, we propose Private Mean-Regularized MTL, a simple MTL framework that provides task-level privacy (Section 4).

Original: We prove that our method achieves ( (cid:15), δ ) -JDP, and we analyze the convergence of our communication-efﬁcient solver on convex and nonconvex objectives.
Revised: We prove that our method achieves ( (cid:15), δ ) -JDP, and we analyze the convergence of our communication-efﬁcient solver on convex and nonconvex objectives.

Original: Our convergence analysis extends to non-private settings with partial participation, which may be of independent interest for problems in cross-device federated learning.
Revised: Our convergence analysis extends to non-private settings with partial participation, which may be of independent interest for problems in cross-device federated learning.

Original: Finally, we explore the performance of our approach on common federated learning benchmarks (Section 5).
Revised: Finally, we explore the performance of our approach on common federated learning benchmarks (Section 5).

Original: Our results show that it is possible to retain the accuracy beneﬁts of MTL in these settings relative to global baselines while still providing meaningful privacy guarantees.
Revised: Our results show that it is possible to retain the accuracy beneﬁts of MTL in these settings relative to global baselines while still providing meaningful privacy guarantees.

Original: Further, even in cases where the MTL objective achieves similar accuracy to the global objective, we ﬁnd that privacy/utility beneﬁts exist when employing the private MTL formulation.
Revised: Further, even in cases where the MTL objective achieves similar accuracy to the global objective, we ﬁnd that privacy/utility beneﬁts exist when employing the private MTL formulation.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Multi-task learning.
Revised: Multi-task learning.

Original: Multi-task learning considers jointly solving multiple related ML tasks.
Revised: Multi-task learning considers jointly solving multiple related ML tasks.

Original: Our work focuses on the general and widely-used formulation of multi-task relationship learning (Zhang &
Revised: Our work focuses on the general and widely-used formulation of multi-task relationship learning (Zhang &

Original: Yeung, 2010), as discussed in Section 3.
Revised: Yeung, 2010), as discussed in Section 3.

Original: This form of MTL is particularly useful in privacy-sensitive applications where datasets are shared among multiple heterogeneous entities (Baytas et al., 2016;
Revised: This form of MTL is particularly useful in privacy-sensitive applications where datasets are shared among multiple heterogeneous entities (Baytas et al., 2016;

Original: Smith et al., 2017; Ghosn & Bengio, 1997).
Revised: Smith et al., 2017; Ghosn & Bengio, 1997).

Original: In these cases, it is natural to view each data source (e.g., ﬁnancial institution, hospital, mobile phone) as a separate ‘task’ that is learned in unison with the other tasks.
Revised: In these cases, it is natural to view each data source (e.g., ﬁnancial institution, hospital, mobile phone) as a separate ‘task’ that is learned in unison with the other tasks.

Original: This allows data to be shared, but the models to be personalized to each data silo.
Revised: This allows data to be shared, but the models to be personalized to each data silo.

Original: For example, in the setting of cross-device federated learning, MTL is commonly used to train a personalized model for each device in a distributed network (Smith et al., 2017; Liu et al., 2017).
Revised: For example, in the setting of cross-device federated learning, MTL is commonly used to train a personalized model for each device in a distributed network (Smith et al., 2017; Liu et al., 2017).

Original: Federated learning.
Revised: Federated learning.

Original: A motivation for our work is the application of federated learning (FL), in which the goal is to collaboratively learn from data that has been generated by, and resides on, a number of private data silos, such as remote devices or servers (McMahan et al., 2017; Kairouz et al., 2019; Li et al., 2020a).
Revised: A motivation for our work is the application of federated learning (FL), in which the goal is to collaboratively learn from data that has been generated by, and resides on, a number of private data silos, such as remote devices or servers (McMahan et al., 2017; Kairouz et al., 2019; Li et al., 2020a).

Original: To ensure client-level differential privacy in FL, a common technique is to learn one global model across the distributed data and then add noise to the aggregated model to sufﬁciently mask any speciﬁc client’s update (Kairouz et al., 2019; McMahan et al., 2018;
Revised: To ensure client-level differential privacy in FL, a common technique is to learn one global model across the distributed data and then add noise to the aggregated model to sufﬁciently mask any speciﬁc client’s update (Kairouz et al., 2019; McMahan et al., 2018;

Original: Geyer et al., 2017).
Revised: Geyer et al., 2017).

Original: However, a deﬁning characteristic of federated learning is that the distributed data are likely to be heterogeneous, i.e., each client may generate data via a distinct data distribution (Kairouz et al., 2019; Li et al., 2020a).
Revised: However, a deﬁning characteristic of federated learning is that the distributed data are likely to be heterogeneous, i.e., each client may generate data via a distinct data distribution (Kairouz et al., 2019; Li et al., 2020a).

Original: To model the (possibly) varying data distributions on each client, it is natural to instead consider learning a separate model for each client’s local dataset.
Revised: To model the (possibly) varying data distributions on each client, it is natural to instead consider learning a separate model for each client’s local dataset.

Original: To this end, a number of recent works have explored multi-task learning as a way to improve the accuracy of learning in federated networks (Smith et al., 2017;
Revised: To this end, a number of recent works have explored multi-task learning as a way to improve the accuracy of learning in federated networks (Smith et al., 2017;

Original: Hanzely & Richt ´ arik, 2020; Hanzely et al., 2020; Ghosh et al., 2020; Sattler et al., 2020;
Revised: Hanzely & Richt ´ arik, 2020; Hanzely et al., 2020; Ghosh et al., 2020; Sattler et al., 2020;

Original: Deng et al., 2020; Mansour et al., 2020).
Revised: Deng et al., 2020; Mansour et al., 2020).

Original: Despite the prevalence of multi-task federated learning, we are unaware of any work that has explored task-level privacy for commonly-used multi-task relationship models (Section 3) in federated settings.
Revised: Despite the prevalence of multi-task federated learning, we are unaware of any work that has explored task-level privacy for commonly-used multi-task relationship models (Section 3) in federated settings.

Original: Differentially private MTL.
Revised: Differentially private MTL.

Original: Prior work in private MTL differs from our own either in terms of the privacy formulation or MTL objective.
Revised: Prior work in private MTL differs from our own either in terms of the privacy formulation or MTL objective.

Original: For example, Wu et al.
Revised: For example, Wu et al.

Original: (2020) explore a speciﬁc MTL setting where a feature representation shared by all tasks is ﬁrst learned, followed by task-speciﬁc models on top of this private representation.
Revised: (2020) explore a speciﬁc MTL setting where a feature representation shared by all tasks is ﬁrst learned, followed by task-speciﬁc models on top of this private representation.

Original: We instead study multi-task relationship learning (Section 3), which is a general and widely-used MTL framework, particularly in federated learning (Smith et al., 2017).
Revised: We instead study multi-task relationship learning (Section 3), which is a general and widely-used MTL framework, particularly in federated learning (Smith et al., 2017).

Original: While our work focuses on task-level privacy, there has been work on data-level privacy for MTL, which aims to protect any single piece of local data rather than protecting the entire local dataset.
Revised: While our work focuses on task-level privacy, there has been work on data-level privacy for MTL, which aims to protect any single piece of local data rather than protecting the entire local dataset.

Original: For example, Xie et al.
Revised: For example, Xie et al.

Original: (2017) propose a method for data-level privacy by representing the model for each task as a sum of a public, shared weight and a task-speciﬁc weight that is only updated locally, and Gupta et al.
Revised: (2017) propose a method for data-level privacy by representing the model for each task as a sum of a public, shared weight and a task-speciﬁc weight that is only updated locally, and Gupta et al.

Original: (2016) study data-level privacy for a mean estimation MTL problem.
Revised: (2016) study data-level privacy for a mean estimation MTL problem.

Original: Finally, Li et al.
Revised: Finally, Li et al.

Original: (2019) studies multiple notions of differential privacy for meta-learning.
Revised: (2019) studies multiple notions of differential privacy for meta-learning.

Original: Although similarly motivated by personalization, their framework does not cover the multi-task setting, where there exists a separate model for each task.
Revised: Although similarly motivated by personalization, their framework does not cover the multi-task setting, where there exists a separate model for each task.

Original:  M ULTI -T ASK L EARNING AND P RIVACY F ORMULATION
Revised:  M ULTI -T ASK L EARNING AND P RIVACY F ORMULATION

Original: In this section, we ﬁrst formalize our multi-task learning objective, which is a form of meanregularized multi-task learning (Section 3.1), and then provide our privacy formulation (Section 3.2).
Revised: In this section, we ﬁrst formalize our multi-task learning objective, which is a form of meanregularized multi-task learning (Section 3.1), and then provide our privacy formulation (Section 3.2).

Original: 3. P ROBLEM S ETUP
Revised: 3. P ROBLEM S ETUP

Original: In the classical setting of multi-task relationship learning (Zhang & Yang, 2017; Zhang & Yeung, 2010), there are m different task learners with their own task-speciﬁc data.
Revised: In the classical setting of multi-task relationship learning (Zhang & Yang, 2017; Zhang & Yeung, 2010), there are m different task learners with their own task-speciﬁc data.

Original: The aim is to solve:
Revised: The aim is to solve:

Original: [Equation]
Revised: [Equation]

Original: where w k is model for task learner
Revised: where w k is model for task learner

Original: k ; { x 1 , . .
Revised: k ; { x 1 , . .

Original: , x n k } is the local data for the k th task; l k ( · ) is the empirical loss for task k ; W
Revised: , x n k } is the local data for the k th task; l k ( · ) is the empirical loss for task k ; W

Original: =
Revised: =

Original: [ w 1 ; · · · ; w m ] ; and Ω ∈ R m × m characterizes the relationship between every pair of task learners.
Revised: [ w 1 ; · · · ; w m ] ; and Ω ∈ R m × m characterizes the relationship between every pair of task learners.

Original: A common choice for setting the regularization term R ( W, Ω) in previous works (Zhang & Yeung, 2010; Smith et al., 2017) is:
Revised: A common choice for setting the regularization term R ( W, Ω) in previous works (Zhang & Yeung, 2010; Smith et al., 2017) is:

Original: [Equation]
Revised: [Equation]

Original: where Ω can be viewed as a covariance matrix, used to learn/encode positive, negative, or unrelated task relationships (Zhang & Yeung, 2010).
Revised: where Ω can be viewed as a covariance matrix, used to learn/encode positive, negative, or unrelated task relationships (Zhang & Yeung, 2010).

Original: In this paper, we focus on studying the mean-regularized multi-task learning objective (Evgeniou & Pontil, 2004): a special case of (1) where Ω =
Revised: In this paper, we focus on studying the mean-regularized multi-task learning objective (Evgeniou & Pontil, 2004): a special case of (1) where Ω =

Original: ( I m × m − 1m 1 m 1 Tm ) 2 is ﬁxed.
Revised: ( I m × m − 1m 1 m 1 Tm ) 2 is ﬁxed.

Original: Here I m ×
Revised: Here I m ×

Original: m is the identity matrix of size m × m and 1 m ∈
Revised: m is the identity matrix of size m × m and 1 m ∈

Original: R m is the vector with all entries equal to 1.
Revised: R m is the vector with all entries equal to 1.

Original: By picking λ 1 = λ 2 , we can rewrite the objective as:
Revised: By picking λ 1 = λ 2 , we can rewrite the objective as:

Original: [Equation]
Revised: [Equation]

Original: where ¯ w is the average of task-speciﬁc models: ¯ w = 1 m (cid:80) mi
Revised: where ¯ w is the average of task-speciﬁc models: ¯ w = 1 m (cid:80) mi

Original: =1 w k .
Revised: =1 w k .

Original: Note that ¯ w is shared across all tasks, and each w k is kept locally for task learner k .
Revised: Note that ¯ w is shared across all tasks, and each w k is kept locally for task learner k .

Original: During optimization, each task learner k solves:
Revised: During optimization, each task learner k solves:

Original: [Equation]
Revised: [Equation]

Original: Despite the prevalence of this simple form of multi-task learning and its recent use in applications such as federated learning with strong privacy motivations (e.g., Hanzely & Richt´arik, 2020; Hanzely et al., 2020; Dinh et al., 2020), we are unaware of prior work that has formalized task-level differential privacy in the context of solving Objective (2).
Revised: Despite the prevalence of this simple form of multi-task learning and its recent use in applications such as federated learning with strong privacy motivations (e.g., Hanzely & Richt´arik, 2020; Hanzely et al., 2020; Dinh et al., 2020), we are unaware of prior work that has formalized task-level differential privacy in the context of solving Objective (2).

Original: 3. P RIVACY F ORMULATION
Revised: 3. P RIVACY F ORMULATION

Original: We start by introducing the deﬁnition of differential privacy (DP) before discussing its generalization to joint differential privacy (JDP) .
Revised: We start by introducing the deﬁnition of differential privacy (DP) before discussing its generalization to joint differential privacy (JDP) .

Original: In the context of multi-task learning, each of the m task learners owns a private dataset D i ∈
Revised: In the context of multi-task learning, each of the m task learners owns a private dataset D i ∈

Original: U i ⊂
Revised: U i ⊂

Original: U .
Revised: U .

Original: We deﬁne D
Revised: We deﬁne D

Original: = { D 1 , · · · , D m } and D (cid:48)
Revised: = { D 1 , · · · , D m } and D (cid:48)

Original: = { D (cid:48) 1 , · · · , D (cid:48) m } .
Revised: = { D (cid:48) 1 , · · · , D (cid:48) m } .

Original: We call two sets D, D (cid:48) neighboring sets if they only differ on the index i , i.e., D j = D (cid:48)
Revised: We call two sets D, D (cid:48) neighboring sets if they only differ on the index i , i.e., D j = D (cid:48)

Original: j for all j except i .
Revised: j for all j except i .

Original: With this setup in mind, we deﬁne differential privacy more formally below.
Revised: With this setup in mind, we deﬁne differential privacy more formally below.

Original: Deﬁnition 1 (Differential Privacy (DP) for MTL (Dwork et al., 2006)) .
Revised: Deﬁnition 1 (Differential Privacy (DP) for MTL (Dwork et al., 2006)) .

Original: A randomized algorithm
Revised: A randomized algorithm

Original: M : U m → R m is ( (cid:15), δ ) -differentially private if for every pair of neighboring sets that only differ in arbitrary index i : D, D (cid:48)
Revised: M : U m → R m is ( (cid:15), δ ) -differentially private if for every pair of neighboring sets that only differ in arbitrary index i : D, D (cid:48)

Original: ∈ U and for every set of subsets of outputs S ⊂ R ,
Revised: ∈ U and for every set of subsets of outputs S ⊂ R ,

Original: [Equation]
Revised: [Equation]

Original: In the context of MTL, a learning algorithm outputs one model for every task learner.
Revised: In the context of MTL, a learning algorithm outputs one model for every task learner.

Original: As mentioned previously, since the output of MTL is a collection of models , traditional DP would require that all the models produced by an MTL learning algorithm are insensitive to changes that happen to the private dataset of any single task.
Revised: As mentioned previously, since the output of MTL is a collection of models , traditional DP would require that all the models produced by an MTL learning algorithm are insensitive to changes that happen to the private dataset of any single task.

Original: In this work we are interested in studying task-level privacy, where the purpose is to protect one task learner’s data from leakage to any other task learners.
Revised: In this work we are interested in studying task-level privacy, where the purpose is to protect one task learner’s data from leakage to any other task learners.

Original: In this setting, DP incurs an additional restriction that the model of any task learner should also be insensitive to changes in its own data , which would render each of the models useless.
Revised: In this setting, DP incurs an additional restriction that the model of any task learner should also be insensitive to changes in its own data , which would render each of the models useless.

Original: To overcome this limitation of DP, we suggest employing joint differential privacy (JDP) (Kearns et al., 2014), a relaxed notion of DP, to formalize the guarantee that an MTL algorithm should provide in order to protect task-level privacy.
Revised: To overcome this limitation of DP, we suggest employing joint differential privacy (JDP) (Kearns et al., 2014), a relaxed notion of DP, to formalize the guarantee that an MTL algorithm should provide in order to protect task-level privacy.

Original: Intuitively,
Revised: Intuitively,

Original: JDP requires that for each task k , the set of output predictive models for all other tasks except k is insensitive to k ’s private data.
Revised: JDP requires that for each task k , the set of output predictive models for all other tasks except k is insensitive to k ’s private data.

Original: We provide a formal deﬁnition below.
Revised: We provide a formal deﬁnition below.

Original: Deﬁnition 2 (Joint Differential Privacy (JDP) (Kearns et al., 2014)) .
Revised: Deﬁnition 2 (Joint Differential Privacy (JDP) (Kearns et al., 2014)) .

Original: A randomized algorithm
Revised: A randomized algorithm

Original: M : U m → R m is ( (cid:15), δ ) -joint differentially private if for every i , for every pair of neighboring datasets that only differ in index i : D, D (cid:48) ∈
Revised: M : U m → R m is ( (cid:15), δ ) -joint differentially private if for every i , for every pair of neighboring datasets that only differ in index i : D, D (cid:48) ∈

Original: U m and for every set of subsets of outputs S ⊂ R m ,
Revised: U m and for every set of subsets of outputs S ⊂ R m ,

Original: [Equation]
Revised: [Equation]

Original: where M ( D ) − i represents the vector M ( D ) with the i -th entry removed.
Revised: where M ( D ) − i represents the vector M ( D ) with the i -th entry removed.

Original: JDP allows the predictive model for task k
Revised: JDP allows the predictive model for task k

Original: to depend on the private data of k , while still providing a strong guarantee: even if all the clients from all the other tasks collude and share their information, they still will not be able to learn much about the private data in the task k .
Revised: to depend on the private data of k , while still providing a strong guarantee: even if all the clients from all the other tasks collude and share their information, they still will not be able to learn much about the private data in the task k .

Original: JDP has mostly been used in applications related to mechanism design (Hsu et al., 2016a; Kannan et al., 2015; Hsu et al., 2016b;
Revised: JDP has mostly been used in applications related to mechanism design (Hsu et al., 2016a; Kannan et al., 2015; Hsu et al., 2016b;

Original: Cummings et al., 2015; Rogers & Roth, 2014; Kearns et al., 2014).
Revised: Cummings et al., 2015; Rogers & Roth, 2014; Kearns et al., 2014).

Original: Although it is a natural choice for achieving task-level privacy in MTL, we are unaware of any work that studies MTL subject to JDP.
Revised: Although it is a natural choice for achieving task-level privacy in MTL, we are unaware of any work that studies MTL subject to JDP.

Original: We also note that we can naturally connect joint differential privacy to standard differential privacy.
Revised: We also note that we can naturally connect joint differential privacy to standard differential privacy.

Original: Informally, if we take the output of a differentially private process and run some algorithm on top of that locally for each task learner without communicating to the global learner or other task learners, this whole process can be shown to be joint differentially private.
Revised: Informally, if we take the output of a differentially private process and run some algorithm on top of that locally for each task learner without communicating to the global learner or other task learners, this whole process can be shown to be joint differentially private.

Original: This is formalized as the Billboard
Revised: This is formalized as the Billboard

Original: Lemma (Kearns et al., 2014), presented in Lemma 1 below.
Revised: Lemma (Kearns et al., 2014), presented in Lemma 1 below.

Original: Lemma 1 (Billboard Lemma) .
Revised: Lemma 1 (Billboard Lemma) .

Original: Suppose M : D → W is ( (cid:15), δ ) -differentially private.
Revised: Suppose M : D → W is ( (cid:15), δ ) -differentially private.

Original: Consider any set of functions:
Revised: Consider any set of functions:

Original: f i : D i ×W → W (cid:48) .
Revised: f i : D i ×W → W (cid:48) .

Original: The composition { f i (Π i D , M ( D )) } is ( (cid:15), δ ) -joint differentially private, where Π i : D → D i is the projection of D onto D i .
Revised: The composition { f i (Π i D , M ( D )) } is ( (cid:15), δ ) -joint differentially private, where Π i : D → D i is the projection of D onto D i .

Original: With the Billboard Lemma , we are able to obtain joint differential privacy by ﬁrst training a differentially private model with data from all tasks, and then ﬁnetuning on each task with its local data.
Revised: With the Billboard Lemma , we are able to obtain joint differential privacy by ﬁrst training a differentially private model with data from all tasks, and then ﬁnetuning on each task with its local data.

Original: We formally introduce our algorithm and corresponding JDP guarantee by using Lemma 1 in Section 4.
Revised: We formally introduce our algorithm and corresponding JDP guarantee by using Lemma 1 in Section 4.

Original: Finally, note that our privacy formulation itself is not limited to the multi-task relationship learning framework.
Revised: Finally, note that our privacy formulation itself is not limited to the multi-task relationship learning framework.

Original: For any form of multi-task learning where each task-speciﬁc model is obtained by training a combination of global component and local component(e.g.
Revised: For any form of multi-task learning where each task-speciﬁc model is obtained by training a combination of global component and local component(e.g.

Original: Li et al.
Revised: Li et al.

Original: , we can provide a JDP guarantee for the MTL training process by using a differentially private global component.
Revised: , we can provide a JDP guarantee for the MTL training process by using a differentially private global component.

Original:  PMTL: P RIVATE M ULTI -T ASK L EARNING
Revised:  PMTL: P RIVATE M ULTI -T ASK L EARNING

Original: We now present PMTL, a method for performing joint differentially-private MTL (Section 4.1).
Revised: We now present PMTL, a method for performing joint differentially-private MTL (Section 4.1).

Original: We provide both a privacy guarantee (Section 4.2) and utility guarantee (Section 4.3) for our approach.
Revised: We provide both a privacy guarantee (Section 4.2) and utility guarantee (Section 4.3) for our approach.

Original: 1: Input: m , T , λ , η , { w 01 , · · · , w 0 m } , (cid:101) w 

Revised: 1: Input: m , T , λ , η , { w 01 , · · · , w 0 m } , (cid:101) w

Original: = 1 m (cid:80) mk
Revised: = 1 m (cid:80) mk

Original: =1 w 0 k 2: for t = 0 , · · · , T − 1 do 3:
Revised: =1 w 0 k 2: for t = 0 , · · · , T − 1 do 3:

Original: Global Learner randomly selects a set of tasks S t and broadcasts the mean weight (cid:101) w t 4: for k ∈ S t in parallel do 5:
Revised: Global Learner randomly selects a set of tasks S t and broadcasts the mean weight (cid:101) w t 4: for k ∈ S t in parallel do 5:

Original: Each task updates its weight w k for E iterations, o k is the last iteration task k is selected w t
Revised: Each task updates its weight w k for E iterations, o k is the last iteration task k is selected w t

Original: +1 k
Revised: +1 k

Original: = ClientUpdate ( w o k k ) 6:
Revised: = ClientUpdate ( w o k k ) 6:

Original: Each task sends g t +1 k
Revised: Each task sends g t +1 k

Original: = w t
Revised: = w t

Original: +1 k − w tk back to the global learner.
Revised: +1 k − w tk back to the global learner.

Original: 7: end for 8:
Revised: 7: end for 8:

Original: Global Learner computes a noisy aggregator of the weights (cid:101) w t
Revised: Global Learner computes a noisy aggregator of the weights (cid:101) w t

Original: +1 = (cid:101) w t + 1 | S t | (cid:88)
Revised: +1 = (cid:101) w t + 1 | S t | (cid:88)

Original: k ∈ S t g t +1 k min (cid:18) 1 , γ (cid:107) g t +1 k
Revised: k ∈ S t g t +1 k min (cid:18) 1 , γ (cid:107) g t +1 k

Original: (cid:107)(cid:19)
Revised: (cid:107)(cid:19)

Original: + N (0 , σ 2 I d × d ) 9: end for 10: return w 1 , · · · , w m as differentially private personalized models 11:
Revised: + N (0 , σ 2 I d × d ) 9: end for 10: return w 1 , · · · , w m as differentially private personalized models 11:

Original: ClientUpdate (w) 12: for j = 0 , · · · , E − 1 do 13:
Revised: ClientUpdate (w) 12: for j = 0 , · · · , E − 1 do 13:

Original: Task learner performs SGD locally
Revised: Task learner performs SGD locally

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:101)
Revised: (cid:101)

Original: [Equation]
Revised: [Equation]

Original: ) 14: end for
Revised: ) 14: end for

Original: 4. A LGORITHM
Revised: 4. A LGORITHM

Original: We summarize our solver for private multi-task learning in Algorithm 1.
Revised: We summarize our solver for private multi-task learning in Algorithm 1.

Original: Our method is based off of
Revised: Our method is based off of

Original: FedAvg (McMahan et al., 2017), a communication-efﬁcient method widely used in federated learning.
Revised: FedAvg (McMahan et al., 2017), a communication-efﬁcient method widely used in federated learning.

Original: FedAvg alternates between two steps: (i) each task learner selected at one communication round solves its own local objective by running stochastic gradient descent for E iterations and sending the updated model to the global learner; (ii) the global learner aggregates the local updates and broadcasts the aggregated mean.
Revised: FedAvg alternates between two steps: (i) each task learner selected at one communication round solves its own local objective by running stochastic gradient descent for E iterations and sending the updated model to the global learner; (ii) the global learner aggregates the local updates and broadcasts the aggregated mean.

Original: By performing local updating in this manner, FedAvg has been shown to empirically reduce the total number of communication rounds needed for convergence in federated settings relative to baselines such as mini-batch FedSGD (McMahan et al., 2017).
Revised: By performing local updating in this manner, FedAvg has been shown to empirically reduce the total number of communication rounds needed for convergence in federated settings relative to baselines such as mini-batch FedSGD (McMahan et al., 2017).

Original: Our private MTL algorithm differs from FedAvg in that: (i) instead of learning a single global model, all task learners collaboratively learn separate, personalized models for each task; (ii) each task learner solves the local objective with the mean-regularization term; (iii) individual model updates are clipped and random Gaussian noise is added to the aggregated model updates to ensure task-level privacy.
Revised: Our private MTL algorithm differs from FedAvg in that: (i) instead of learning a single global model, all task learners collaboratively learn separate, personalized models for each task; (ii) each task learner solves the local objective with the mean-regularization term; (iii) individual model updates are clipped and random Gaussian noise is added to the aggregated model updates to ensure task-level privacy.

Original: To aggregate updates from each task, we assume that we have access to a trusted global learner, i.e., it is safe for some global entity to observe/collect the individual model updates from each task.
Revised: To aggregate updates from each task, we assume that we have access to a trusted global learner, i.e., it is safe for some global entity to observe/collect the individual model updates from each task.

Original: This is a standard assumption in federated learning, where access to a trusted central server is assumed in order to collect client updates (Kairouz et al., 2019).
Revised: This is a standard assumption in federated learning, where access to a trusted central server is assumed in order to collect client updates (Kairouz et al., 2019).

Original: However, even with this assumption, note that it is possible for any single task learner to infer information about other tasks from the global model, since it is a linear combination of all task speciﬁc models and is shared among all task learners.
Revised: However, even with this assumption, note that it is possible for any single task learner to infer information about other tasks from the global model, since it is a linear combination of all task speciﬁc models and is shared among all task learners.

Original: There are several ways to overcome this privacy risk and thus achieve ( (cid:15), δ ) -differential privacy.
Revised: There are several ways to overcome this privacy risk and thus achieve ( (cid:15), δ ) -differential privacy.

Original: In this paper, we use the Gaussian Mechanism (Dwork & Roth, 2014) during global aggregation as a simple yet effective method, highlighted in the red portion of line 8 in Algorithm 1.
Revised: In this paper, we use the Gaussian Mechanism (Dwork & Roth, 2014) during global aggregation as a simple yet effective method, highlighted in the red portion of line 8 in Algorithm 1.

Original: In this case, each task learner receives a noisy aggregated global model, making it difﬁcult for any task to leak private information to the others.
Revised: In this case, each task learner receives a noisy aggregated global model, making it difﬁcult for any task to leak private information to the others.

Original: To apply the Gaussian mechanism, we need to bound the (cid:96) 2 -sensitivity of each local model update that is communicated to lie in B = { ∆ w |(cid:107) ∆ w (cid:107) 2 ≤ γ } , as highlighted in the blue part of line 8 in Algorithm 1.
Revised: To apply the Gaussian mechanism, we need to bound the (cid:96) 2 -sensitivity of each local model update that is communicated to lie in B = { ∆ w |(cid:107) ∆ w (cid:107) 2 ≤ γ } , as highlighted in the blue part of line 8 in Algorithm 1.

Original: Hence, at each communication round, the global learner receives the model updates from each task, and clips the model updates to B before aggregation.
Revised: Hence, at each communication round, the global learner receives the model updates from each task, and clips the model updates to B before aggregation.

Original: Note that different from DPSGD (Abadi et al., 2016), when we solve the local objective for each selected task at each communication round, our algorithm doesn’t clip and perturb the gradient used to update the task-speciﬁc model.
Revised: Note that different from DPSGD (Abadi et al., 2016), when we solve the local objective for each selected task at each communication round, our algorithm doesn’t clip and perturb the gradient used to update the task-speciﬁc model.

Original: Instead, since the purpose is to protect task or client-level privacy in multi-task learning, we perform standard SGD locally for each task and only clip and perturb the model update that is sent to the global learner.
Revised: Instead, since the purpose is to protect task or client-level privacy in multi-task learning, we perform standard SGD locally for each task and only clip and perturb the model update that is sent to the global learner.

Original: We formalize the privacy guarantee of Algorithm 1 in Section 4.2.
Revised: We formalize the privacy guarantee of Algorithm 1 in Section 4.2.

Original: 4. P RIVACY A NALYSIS
Revised: 4. P RIVACY A NALYSIS

Original: We now rigorously explore the privacy guarantee provided by Algorithm 1.
Revised: We now rigorously explore the privacy guarantee provided by Algorithm 1.

Original: In our optimization scheme, for each task k , at the end of each communication round, a shared global model is received.
Revised: In our optimization scheme, for each task k , at the end of each communication round, a shared global model is received.

Original: After that the task speciﬁc model is updated by optimizing the local objective.
Revised: After that the task speciﬁc model is updated by optimizing the local objective.

Original: We formalize this local task learning process as h k : D k × W → W .
Revised: We formalize this local task learning process as h k : D k × W → W .

Original: Here we simply assume W ⊂ R
Revised: Here we simply assume W ⊂ R

Original: d is closed.
Revised: d is closed.

Original: Deﬁne the mechanism for communication round t to be
Revised: Deﬁne the mechanism for communication round t to be

Original: [Equation]
Revised: [Equation]

Original: where β t ∼ N (0 , σ 2 I d × d ) .
Revised: where β t ∼ N (0 , σ 2 I d × d ) .

Original: Note that M t characterizes a Sampled Gaussian Mechanism given (cid:101) w t as a ﬁxed model rather than the output of a composition of M j for j < t .
Revised: Note that M t characterizes a Sampled Gaussian Mechanism given (cid:101) w t as a ﬁxed model rather than the output of a composition of M j for j < t .

Original: To analyze the privacy guarantee of Algorithm 1 over T communication rounds, we deﬁne the composition of M 1 to M T recursively as M 1: T = M T ( { D i } , { h i ( · ) } , M T − 1 , σ ) .
Revised: To analyze the privacy guarantee of Algorithm 1 over T communication rounds, we deﬁne the composition of M 1 to M T recursively as M 1: T = M T ( { D i } , { h i ( · ) } , M T − 1 , σ ) .

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume | S t | =
Revised: Assume | S t | =

Original: q for all t and the total number of communication rounds is T .
Revised: q for all t and the total number of communication rounds is T .

Original: There exists constants c 1 , c 2 such that for any (cid:15) <
Revised: There exists constants c 1 , c 2 such that for any (cid:15) <

Original: c 1 q 2 m 2 T , the mechanism M 1: T is ( (cid:15), δ ) -differentially private for any δ > 0 if we choose σ ≥ c 2 γ
Revised: c 1 q 2 m 2 T , the mechanism M 1: T is ( (cid:15), δ ) -differentially private for any δ > 0 if we choose σ ≥ c 2 γ

Original: [Equation]
Revised: [Equation]

Original: When q = m , M 1: T is ( (cid:15), δ ) -differentially private if we choose σ = 4 γ
Revised: When q = m , M 1: T is ( (cid:15), δ ) -differentially private if we choose σ = 4 γ

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 provides a provable privacy guarantee on the learned global model.
Revised: Theorem 1 provides a provable privacy guarantee on the learned global model.

Original: When all tasks participate in every communication round, i.e. q = m , the global aggregation step in Algorithmreduce to applying Gaussian Mechanism without sampling rather than Sampled Gaussian Mechanism on the average model updates.
Revised: When all tasks participate in every communication round, i.e. q = m , the global aggregation step in Algorithmreduce to applying Gaussian Mechanism without sampling rather than Sampled Gaussian Mechanism on the average model updates.

Original: We provide a detailed proof of Theorem 1 in Appendix A.1.
Revised: We provide a detailed proof of Theorem 1 in Appendix A.1.

Original: Note that Theorem 1 doesn’t rely on how task learners optimize their local objective.
Revised: Note that Theorem 1 doesn’t rely on how task learners optimize their local objective.

Original: Hence, Theorem 1 is not limited to Algorithm 1 and could be generalized to other local objectives and other global aggregation methods that produce a single model aggregate.
Revised: Hence, Theorem 1 is not limited to Algorithm 1 and could be generalized to other local objectives and other global aggregation methods that produce a single model aggregate.

Original: Now we show that Algorithm 1, which outputs m separate models, satisﬁes joint differential privacy.
Revised: Now we show that Algorithm 1, which outputs m separate models, satisﬁes joint differential privacy.

Original: Given (cid:101) w t for any t ≤ T , we formally deﬁne the process that each task learner
Revised: Given (cid:101) w t for any t ≤ T , we formally deﬁne the process that each task learner

Original: k optimize its local objective to be h (cid:48)
Revised: k optimize its local objective to be h (cid:48)

Original: k : D k × W → W .
Revised: k : D k × W → W .

Original: Note that h (cid:48) k is not restricted to be h k and could represent the optimization process for any local objective.
Revised: Note that h (cid:48) k is not restricted to be h k and could represent the optimization process for any local objective.

Original: In order to show that Algorithm 1 satisﬁes JDP, we would apply the Billboard Lemma introduced in Section 3.
Revised: In order to show that Algorithm 1 satisﬁes JDP, we would apply the Billboard Lemma introduced in Section 3.

Original: In our case, the average model that is broadcast by the global learner at every communication round is the output of a differentially private learning process.
Revised: In our case, the average model that is broadcast by the global learner at every communication round is the output of a differentially private learning process.

Original: Task learners then individually train their task speciﬁc models on the respective private data to obtain personalized models.
Revised: Task learners then individually train their task speciﬁc models on the respective private data to obtain personalized models.

Original: We now present our main theorem of the JDP guarantee provided by Algorithm 1:
Revised: We now present our main theorem of the JDP guarantee provided by Algorithm 1:

Original: Theorem 2.
Revised: Theorem 2.

Original: There exists constants c 1 , c 2 , for any 0 < (cid:15) <
Revised: There exists constants c 1 , c 2 , for any 0 < (cid:15) <

Original: c 1 q 2 m 2 T and δ > 0 , let σ ≥ c 2 γ √ T log(1 /δ )
Revised: c 1 q 2 m 2 T and δ > 0 , let σ ≥ c 2 γ √ T log(1 /δ )

Original: (cid:15)m .
Revised: (cid:15)m .

Original: Algorithm 1 that outputs h (cid:48) k
Revised: Algorithm 1 that outputs h (cid:48) k

Original: ( D k , M 1: T ) for each task is ( (cid:15), δ ) -joint differentially private.
Revised: ( D k , M 1: T ) for each task is ( (cid:15), δ ) -joint differentially private.

Original: From Theorem 2, for any ﬁxed δ , the more tasks involved in the learning process, the smaller σ we need in order to keep the privacy parameter (cid:15) the same.
Revised: From Theorem 2, for any ﬁxed δ , the more tasks involved in the learning process, the smaller σ we need in order to keep the privacy parameter (cid:15) the same.

Original: In other words, less noise is required to keep the task-speciﬁc data private.
Revised: In other words, less noise is required to keep the task-speciﬁc data private.

Original: When we have inﬁnitely many tasks ( m → ∞ ), we have σ → 0 , in which case only a negligible amount of noise is needed to add to the model aggregates to make the global model private to all tasks.
Revised: When we have inﬁnitely many tasks ( m → ∞ ), we have σ → 0 , in which case only a negligible amount of noise is needed to add to the model aggregates to make the global model private to all tasks.

Original: We provide a detailed proof in Appendix A.1.
Revised: We provide a detailed proof in Appendix A.1.

Original: Remark.
Revised: Remark.

Original: Note that privacy guarantee provided by our Theorem 2 is not limited to mean-regularized multi-task learning.
Revised: Note that privacy guarantee provided by our Theorem 2 is not limited to mean-regularized multi-task learning.

Original: For any form of multi-task relationship learning with ﬁxed relationship matrix Ω , as long as we ﬁx the (cid:96) 2 -sensitivity of model updates and the noise scale of the Gaussian mechanism applied to the statistics broadcast to all task learners, the privacy guarantee induced by this aggregation step is ﬁxed, regardless of the local objective being optimized.
Revised: For any form of multi-task relationship learning with ﬁxed relationship matrix Ω , as long as we ﬁx the (cid:96) 2 -sensitivity of model updates and the noise scale of the Gaussian mechanism applied to the statistics broadcast to all task learners, the privacy guarantee induced by this aggregation step is ﬁxed, regardless of the local objective being optimized.

Original: For example, as a natural extension of our mean-regularized MTL objective, consider the case where task learners are partitioned into ﬁxed clusters and optimize the mean-regularized MTL objective within each cluster, as in Evgeniou et al.
Revised: For example, as a natural extension of our mean-regularized MTL objective, consider the case where task learners are partitioned into ﬁxed clusters and optimize the mean-regularized MTL objective within each cluster, as in Evgeniou et al.

Original: In this scenario, Theorem 2 directly applies to the algorithm run on each cluster.
Revised: In this scenario, Theorem 2 directly applies to the algorithm run on each cluster.

Original: 4. C ONVERGENCE A NALYSIS
Revised: 4. C ONVERGENCE A NALYSIS

Original: As discussed in Section 3, we are interested in the following task-speciﬁc objective:
Revised: As discussed in Section 3, we are interested in the following task-speciﬁc objective:

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) w is an estimate for the average model w ; l k ( w k ) is the empirical loss for task k ; w k ∈ R d .
Revised: where (cid:101) w is an estimate for the average model w ; l k ( w k ) is the empirical loss for task k ; w k ∈ R d .

Original: Here, we analyze the convergence behavior in the setting where a set S t of q tasks participate in the optimization process at every communication round.
Revised: Here, we analyze the convergence behavior in the setting where a set S t of q tasks participate in the optimization process at every communication round.

Original: Further, we assume the number of local optimization steps E = 1 .
Revised: Further, we assume the number of local optimization steps E = 1 .

Original: We present the following convergence result:
Revised: We present the following convergence result:

Original: Theorem 3 (Convergence under nonconvex loss) .
Revised: Theorem 3 (Convergence under nonconvex loss) .

Original: Let f k be ( L + λ ) -smooth.
Revised: Let f k be ( L + λ ) -smooth.

Original: Assume γ is sufﬁciently large such that γ ≥ max k,t (cid:107)∇ w tk f k ( w tk ; (cid:101) w t ) (cid:107) 2 .
Revised: Assume γ is sufﬁciently large such that γ ≥ max k,t (cid:107)∇ w tk f k ( w tk ; (cid:101) w t ) (cid:107) 2 .

Original: Further let f ∗ k = min w, ¯ w f k ( w ; ¯ w ) and p = qm .
Revised: Further let f ∗ k = min w, ¯ w f k ( w ; ¯ w ) and p = qm .

Original: If we use a ﬁxed learning rate η t = η =
Revised: If we use a ﬁxed learning rate η t = η =

Original: [Equation]
Revised: [Equation]

Original:  Algorithm 1 satisﬁes:
Revised: , Algorithm 1 satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Let σ chosen as we set in Theorem 2.
Revised: Let σ chosen as we set in Theorem 2.

Original: Take T
Revised: Take T

Original: = O (cid:16) mλdγ 2 (cid:17) , the right hand side is bounded by λdγ
Revised: = O (cid:16) mλdγ 2 (cid:17) , the right hand side is bounded by λdγ

Original: [Equation]
Revised: [Equation]

Original: We provide a formal statement and full proof of Theorem 3 in Appendix A.2.
Revised: We provide a formal statement and full proof of Theorem 3 in Appendix A.2.

Original: The upper bound in (30) consists of two parts: error induced by the gradient descent algorithm and error induced by the
Revised: The upper bound in (30) consists of two parts: error induced by the gradient descent algorithm and error induced by the

Original: Gaussian Mechanism.
Revised: Gaussian Mechanism.

Original: When σ = 0 , Algorithm 1 recovers a non-private mean-regularized multi-task learning solver.
Revised: When σ = 0 , Algorithm 1 recovers a non-private mean-regularized multi-task learning solver.

Original: Corollary 4.
Revised: Corollary 4.

Original: When σ = 0 , Algorithm 1 with ( L + λ ) -smooth and nonconvex f k satisﬁes
Revised: When σ = 0 , Algorithm 1 with ( L + λ ) -smooth and nonconvex f k satisﬁes

Original: [Equation]
Revised: [Equation]

Original: By Theorem 2, given ﬁxed (cid:15) , σ 2 grows linearly with respect to T .
Revised: By Theorem 2, given ﬁxed (cid:15) , σ 2 grows linearly with respect to T .

Original: Hence, given the same privacy guarantee, larger noise is required if the algorithm is run for more communication rounds.
Revised: Hence, given the same privacy guarantee, larger noise is required if the algorithm is run for more communication rounds.

Original: Note that in
Revised: Note that in

Original: Theorem 3, the upper bound consists of O ( 1 m(cid:15) 2 ) , which means when there are more tasks, the upper bound becomes smaller while the privacy parameter remains the same.
Revised: Theorem 3, the upper bound consists of O ( 1 m(cid:15) 2 ) , which means when there are more tasks, the upper bound becomes smaller while the privacy parameter remains the same.

Original: On the other hand, Theorem 3 also shows a privacy-utility tradeoff using our Algorithm 1: the upper bound grows inversely proportional to the privacy parameter (cid:15) .
Revised: On the other hand, Theorem 3 also shows a privacy-utility tradeoff using our Algorithm 1: the upper bound grows inversely proportional to the privacy parameter (cid:15) .

Original: We also provide a convergence analysis of Algorithm 1 with strongly-convex losses in Theorem 5 below (formal statement and proof in Appendix A.3).
Revised: We also provide a convergence analysis of Algorithm 1 with strongly-convex losses in Theorem 5 below (formal statement and proof in Appendix A.3).

Original: Theorem 5 (Convergence under strongly-convex loss) .
Revised: Theorem 5 (Convergence under strongly-convex loss) .

Original: Let f k be ( L + λ ) -smooth and ( µ + λ ) - strongly convex.
Revised: Let f k be ( L + λ ) -smooth and ( µ + λ ) - strongly convex.

Original: Assume γ is sufﬁciently large such that γ ≥ max k,t (cid:107)∇ w tk f k ( w tk ; (cid:101) w t ) (cid:107) 2 .
Revised: Assume γ is sufﬁciently large such that γ ≥ max k,t (cid:107)∇ w tk f k ( w tk ; (cid:101) w t ) (cid:107) 2 .

Original: Further let w ∗ k = arg min w f k ( w ; ¯ w ∗ ) , where ¯ w ∗ = 1 m (cid:80) mk
Revised: Further let w ∗ k = arg min w f k ( w ; ¯ w ∗ ) , where ¯ w ∗ = 1 m (cid:80) mk

Original: =1 w ∗ k and p = qm .
Revised: =1 w ∗ k and p = qm .

Original: If we use a ﬁxed learning rate η t = η = c L − 2 p + λp for some constant c such that 0 ≤ ηp ( c − 2)( µ + λ ) ≤ 1 , Algorithm 1 satisﬁes:
Revised: If we use a ﬁxed learning rate η t = η = c L − 2 p + λp for some constant c such that 0 ≤ ηp ( c − 2)( µ + λ ) ≤ 1 , Algorithm 1 satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: where ∆ [Equation] =
Revised: where ∆ t =

Original: [Equation]
Revised: [Equation]

Original: Let σ be chosen as in Theorem 2, then there exists T
Revised: Let σ be chosen as in Theorem 2, then there exists T

Original: = O (cid:18) m ( c 2 − 2 c )( µ + λ ) λ (cid:16)[Equation]
Revised: = O (cid:18) m ( c 2 − 2 c )( µ + λ ) λ (cid:16)

Original:  such that
Revised: L −p 2 + λ (cid:17) dγ(cid:19) such that

Original: [Equation]
Revised: [Equation]

Original: As with Corollary 4, we recover the bound of the non-private mean-regularized MTL solver for σ = 0 .
Revised: As with Corollary 4, we recover the bound of the non-private mean-regularized MTL solver for σ = 0 .

Original: Corollary 6.
Revised: Corollary 6.

Original: When σ = 0 , Algorithm 1 with ( L + λ ) -smooth and ( µ + λ ) -strongly convex f k satisﬁes[Equation] T [Equation]
Revised: When σ = 0 , Algorithm 1 with ( L + λ ) -smooth and ( µ + λ ) -strongly convex f k satisﬁes

Original: ∆ 0 − mB

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section [Figure] we empirically evaluate our private MTL solver on several federated learning benchmarks.Speciﬁcally
Revised: In this section, we empirically evaluate our private MTL solver on several federated learning benchmarks.

Original: [Figure] we demonstrate the privacy-utility trade-off of training a MTL objective compared with training a global model.
Revised: Speciﬁcally, we demonstrate the privacy-utility trade-off of training a MTL objective compared with training a global model.

Original: We then compare results of performing local ﬁnetuning after learning an MTL objective with local ﬁnetuning after learning a global objective.
Revised: We then compare results of performing local ﬁnetuning after learning an MTL objective with local ﬁnetuning after learning a global objective.

Original: 5. S ETUP
Revised: 5. S ETUP

Original: For all experiments [Figure] we evaluate the test accuracy and privacy parameter of our private MTL solver given a ﬁxed clipping bound γ , variance of Gaussian noise σ 2 , and communication rounds T .
Revised: For all experiments, we evaluate the test accuracy and privacy parameter of our private MTL solver given a ﬁxed clipping bound γ , variance of Gaussian noise σ 2 , and communication rounds T .

Original: All experiments are performed on common federated learning benchmarks as a natural application of multi-task learning.
Revised: All experiments are performed on common federated learning benchmarks as a natural application of multi-task learning.

Original: We provide a detailed description of datasets and models in Appendix A.4.
Revised: We provide a detailed description of datasets and models in Appendix A.4.

Original: Each dataset is naturally partitioned among m different clients.
Revised: Each dataset is naturally partitioned among m different clients.

Original: Under such a scenario, each client can be viewed as a task and the data that a client generates is only visible to the local task learner
Revised: Under such a scenario, each client can be viewed as a task and the data that a client generates is only visible to the local task learner.

Original: [Figure]
Revised: [Figure]

Original: 5. P RIVACY -U TILITY T RADE - OFF OF PMTL
Revised: v a c y p [Figure] r [Figure]

Revised: e t e r ( )

Revised: [Figure]

Revised: [Figure]

Revised: L o ss e s

Revised: P r i v a c y p a r a m e t e r ( ) =0.02, = (c) CelebA

Revised: 5. P RIVACY -U TILITY T RADE - OFF OF PMTL

Original: We ﬁrst explore the training loss ( orange ) and privacy parameter (cid:15) ( blue ) as a function of communication rounds across three datasets (Figure 2).
Revised: We ﬁrst explore the training loss ( orange ) and privacy parameter (cid:15) ( blue ) as a function of communication rounds across three datasets (Figure 2).

Original: Speciﬁcally, we evaluate the average loss for all the tasks and (cid:15) given a ﬁxed δ after each round, where δ is set to be 1 m for all experiments.
Revised: Speciﬁcally, we evaluate the average loss for all the tasks and (cid:15) given a ﬁxed δ after each round, where δ is set to be 1 m for all experiments.

Original: In general, for a ﬁxed clipping bound γ and σ , we see that the method converges fairly quickly with respect to the resulting privacy, but that privacy guarantees may be sacriﬁced in order to achieve very small losses.
Revised: In general, for a ﬁxed clipping bound γ and σ , we see that the method converges fairly quickly with respect to the resulting privacy, but that privacy guarantees may be sacriﬁced in order to achieve very small losses.

Original: To put these results in context, we also compare the test performance of our private MTL solver with that of training a global model.
Revised: To put these results in context, we also compare the test performance of our private MTL solver with that of training a global model.

Original: In particular, we use FedAvg (McMahan et al., 2017) to train a global model.
Revised: In particular, we use FedAvg (McMahan et al., 2017) to train a global model.

Original: At each communication round, task learners solve their local objective individually.
Revised: At each communication round, task learners solve their local objective individually.

Original: Assuming the global learner is trustworthy, while aggregating the model updates from all tasks, the global learner applies a Gaussian Mechanism and sends the noisy aggregation back to the task learners.
Revised: Assuming the global learner is trustworthy, while aggregating the model updates from all tasks, the global learner applies a Gaussian Mechanism and sends the noisy aggregation back to the task learners.

Original: As a result, private FedAvg differs from our private MTL solver in the following two places: (i) the MTL objective solved locally by each task learner has an additional mean-regularized term; (ii) the MTL method evaluates on one task-speciﬁc model for every task while the global method evaluates all tasks on one global model.
Revised: As a result, private FedAvg differs from our private MTL solver in the following two places: (i) the MTL objective solved locally by each task learner has an additional mean-regularized term; (ii) the MTL method evaluates on one task-speciﬁc model for every task while the global method evaluates all tasks on one global model.

Original: For each dataset, we select privacy parameter (cid:15) ∈ [0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 , 1 . 6 , 2 . 0 , 4 . 0] .
Revised: For each dataset, we select privacy parameter (cid:15) ∈ [0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 , 1 . 6 , 2 . 0 , 4 . 0] .

Original: For each (cid:15) , we select the γ , σ , and T that result in the best validation accuracy for a given (cid:15) and record the test accuracy.
Revised: For each (cid:15) , we select the γ , σ , and T that result in the best validation accuracy for a given (cid:15) and record the test accuracy.

Original: A detailed description of hyperparameters is listed in Appendix A.5.
Revised: A detailed description of hyperparameters is listed in Appendix A.5.

Original: We plot the test accuracy with respect to the highest validation accuracy given one (cid:15) for both private MTL model and private global model.
Revised: We plot the test accuracy with respect to the highest validation accuracy given one (cid:15) for both private MTL model and private global model.

Original: The results are shown in Figure 3.
Revised: The results are shown in Figure 3.

Original: In all three datasets, our private MTL solver achieves higher test accuracy compared with training a private global model with FedAvg given the same (cid:15) .
Revised: In all three datasets, our private MTL solver achieves higher test accuracy compared with training a private global model with FedAvg given the same (cid:15) .

Original: Moreover, the proposed mean regularized MTL solver is able to retain an advantage over global model even with noisy aggregation.
Revised: Moreover, the proposed mean regularized MTL solver is able to retain an advantage over global model even with noisy aggregation.

Original: In particular, for small (cid:15) < 1 , adding random Gaussian noise during global aggregation ampliﬁes the test accuracy  difference between our MTL solver and FedAvg.
Revised: In particular, for small (cid:15) < 1 , adding random Gaussian noise during global aggregation ampliﬁes the test accuracy [Figure] difference between our MTL solver and FedAvg.

Original: Under the StackOverﬂow task, both methods obtain test accuracy close to the non private baseline for large (cid:15) .
Revised: Under the StackOverﬂow task, both methods obtain test accuracy close to the non private baseline for large (cid:15) .

Original: To demonstrate that applying private MTL has an advantage over private global training more generally, we also compared our PMTL method with private FedProx (Li et al., 2020b).
Revised: To demonstrate that applying private MTL has an advantage over private global training more generally, we also compared our PMTL method with private FedProx (Li et al., 2020b).

Original: The results (which mirror Figure 3) are in Appendix A.6.[Figure]
Revised: The results (which mirror Figure 3) are in Appendix A.6.

Original: (cid:15)

Original: = ∞ corresponds to the case where no noise and clipping happened, i.e., training non-privately.
Revised: = ∞ corresponds to the case where no noise and clipping happened, i.e., training non-privately

Original: The higher accuracy between MTL and Global given the same (cid:15) and ﬁnetuning method is bolded .

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 5. PMTL WITH LOCAL FINETUNING
Revised: 5. PMTL WITH LOCAL FINETUNING

Original: Finally, in federated learning, previous works have shown local ﬁnetuning with different objectives is helpful for improving utility while training a differentially private global model (Yu et al., 2020).
Revised: Finally, in federated learning, previous works have shown local ﬁnetuning with different objectives is helpful for improving utility while training a differentially private global model (Yu et al., 2020).

Original: In this section, after obtaining a private global model, we explore locally ﬁnetuning the task speciﬁc models by optimizing different local objective functions.
Revised: In this section, after obtaining a private global model, we explore locally ﬁnetuning the task speciﬁc models by optimizing different local objective functions.

Original: In particular, we use common objectives which (i) naively optimize the local empirical risk (Vanilla Finetuning), or (ii) encourage minimizing the distance between local and global model under different distance metrics (Mean-regularization,
Revised: In particular, we use common objectives which (i) naively optimize the local empirical risk (Vanilla Finetuning), or (ii) encourage minimizing the distance between local and global model under different distance metrics (Mean-regularization,

Original: Symmetrized KL, EWC (Kirkpatrick et al., 2017; Yu et al., 2020)).
Revised: Symmetrized KL, EWC (Kirkpatrick et al., 2017; Yu et al., 2020)).

Original: The results are listed in Table 1.
Revised: The results are listed in Table 1.

Original: When (cid:15) =
Revised: When (cid:15) =

Original: ∞ (the non-private setting), global with mean-regularization ﬁnetuning outperforms all
Revised: ∞ (the non-private setting), global with mean-regularization ﬁnetuning outperforms all

Original: MTL+ﬁnetuning methods.
Revised: MTL+ﬁnetuning methods.

Original: However, when we add privacy to both methods, private MTL+ﬁnetuning has an advantage over global with ﬁnetuning on different ﬁnetuning objectives.
Revised: However, when we add privacy to both methods, private MTL+ﬁnetuning has an advantage over global with ﬁnetuning on different ﬁnetuning objectives.

Original: In some cases, e.g. using Symmetrized KL as the ﬁnetuning objective, the test accuracy gap between private MTL with ﬁntuning and private global with ﬁnetuning is ampliﬁed when (cid:15) is small compared to the case where no privacy is added during training.
Revised: In some cases, e.g. using Symmetrized KL as the ﬁnetuning objective, the test accuracy gap between private MTL with ﬁntuning and private global with ﬁnetuning is ampliﬁed when (cid:15) is small compared to the case where no privacy is added during training.

Original:  C ONCLUSION AND F UTURE WORK
Revised:  C ONCLUSION AND F UTURE WORK

Original: In this work, we deﬁne notions of task-level privacy for multi-task learning and propose a simple method for differentially private mean-regularized MTL.
Revised: In this work, we deﬁne notions of task-level privacy for multi-task learning and propose a simple method for differentially private mean-regularized MTL.

Original: Theoretically, we provide both privacy and utility guarantees for our approach.
Revised: Theoretically, we provide both privacy and utility guarantees for our approach.

Original: Empirically, we show that private mean-regularized MTL retains advantages over training a private global on common federated learning benchmarks.
Revised: Empirically, we show that private mean-regularized MTL retains advantages over training a private global on common federated learning benchmarks.

Original: In future work, we are interested in building on our results to explore privacy for more general forms of MTL e g., the family of objectives in (1) with arbitrary matrix Ω .
Revised: In future work, we are interested in building on our results to explore privacy for more general forms of MTL, e.g., the family of objectives in (1) with arbitrary matrix Ω .

Original: We are also interested in studying how task-level privacy relates to algorithmic fairness in the MTL setting
Revised: We are also interested in studying how task-level privacy relates to algorithmic fairness in the MTL setting.

Original: Ethics Statement.
Revised: Ethics Statement.

Original: In this work we aim to raise awareness about privacy issues that potentially exist when performing multi-task learning, which is commonly used to model heterogeneous data in areas such as healthcare, ﬁnance, and IoT computing.
Revised: In this work we aim to raise awareness about privacy issues that potentially exist when performing multi-task learning, which is commonly used to model heterogeneous data in areas such as healthcare, ﬁnance, and IoT computing.

Original: We are particularly motivated by the application of cross-device federated learning, where numerous forms of multi-task learning have been proposed to improve accuracy, but the privacy implications remain unclear.
Revised: We are particularly motivated by the application of cross-device federated learning, where numerous forms of multi-task learning have been proposed to improve accuracy, but the privacy implications remain unclear.

Original: To ensure common notions of task or client-level privacy in this setting while retaining the demonstrated accuracy beneﬁts of MTL, we have provided formal privacy deﬁnitions for multi-task learning.
Revised: To ensure common notions of task or client-level privacy in this setting while retaining the demonstrated accuracy beneﬁts of MTL, we have provided formal privacy deﬁnitions for multi-task learning.

Original: Building on these deﬁnitions, we proposed a practical algorithm and studied its performance both theoretically and empirically for the special case of mean-regularized multi-task learning objective.
Revised: Building on these deﬁnitions, we proposed a practical algorithm and studied its performance both theoretically and empirically for the special case of mean-regularized multi-task learning objective.

Original: We hope this work will be a starting point for future extensions that study the issue of privacy in multi-task learning
Revised: We hope this work will be a starting point for future extensions that study the issue of privacy in multi-task learning

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: To enable reproducibility in our theoretical and empirical results, we provide:
Revised: To enable reproducibility in our theoretical and empirical results, we provide:

Original: • Proofs of Theorem 1, 2, 3, 5 in Appendix A.1, A.2, A.3.
Revised: • Proofs of Theorem 1, 2, 3, 5 in Appendix A.1, A.2, A.3.

Original: • Dataset and model details in Appendix A.4.
Revised: • Dataset and model details in Appendix A.4.

Original: • Hyperparameters for tuning in Appendix A.5.
Revised: • Hyperparameters for tuning in Appendix A.5.

Original: • Code used for running experiments in Supplementary Material.
Revised: • Code used for running experiments in Supplementary Material.