Original: W EIGHTED T RAINING FOR C ROSS -T ASK L EARNING
Revised: W EIGHTED T RAINING FOR C ROSS -T ASK L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we introduce T arget-
Revised: In this paper we introduce T arget-

Original: A ware W eighted T raining ( TAWT ), a weighted training algorithm for cross-task learning based on minimizing a representationbased task distance between the source and target tasks.
Revised: A ware W eighted T raining ( TAWT ), a weighted training algorithm for cross-task learning based on minimizing a representationbased task distance between the source and target tasks.

Original: We show that TAWT is easy to implement, is computationally efﬁcient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees.
Revised: We show that TAWT is easy to implement, is computationally efﬁcient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees.

Original: The effectiveness of
Revised: The effectiveness of

Original: TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER).
Revised: TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER).

Original: As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of ﬁne-tuning.
Revised: As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of ﬁne-tuning

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The state-of-the-art (SOTA) models in real-world applications rely increasingly on the usage of weak supervision signals (Pennington et al., 2014; Devlin et al., 2019; Liu et al., 2019).
Revised: The state-of-the-art (SOTA) models in real-world applications rely increasingly on the usage of weak supervision signals (Pennington et al., 2014; Devlin et al., 2019; Liu et al., 2019).

Original: Among these, cross-task signals are one of the most widely-used weak signals (Zamir et al., 2018; McCann et al., 2018).
Revised: Among these, cross-task signals are one of the most widely-used weak signals (Zamir et al., 2018; McCann et al., 2018).

Original: Despite their popularity, the beneﬁts of cross-task signals are not well understood from a theoretical point of view, especially in the context of deep learning (He et al., 2020; Neyshabur et al., 2020), hence impeding the efﬁcient usage of those signals.
Revised: Despite their popularity, the beneﬁts of cross-task signals are not well understood from a theoretical point of view, especially in the context of deep learning (He et al., 2021; Neyshabur et al., 2020), hence impeding the efﬁcient usage of those signals.

Original: Previous work has adopted representation learning as a framework to understand the beneﬁts of cross-task signals, where knowledge transfer is achieved by learning a representation shared across different tasks (Baxter, 2000; Maurer et al., 2016;
Revised: Previous work has adopted representation learning as a framework to understand the beneﬁts of cross-task signals, where knowledge transfer is achieved by learning a representation shared across different tasks (Baxter, 2000; Maurer et al., 2016;

Original: Tripuraneni et al., 2020; 2021; Du et al., 2021).
Revised: Tripuraneni et al., 2020; 2021; Du et al., 2021).

Original: However, the existence of a shared representation is often too strong an assumption in practice.
Revised: However, the existence of a shared representation is often too strong an assumption in practice.

Original: Such an assumption also makes it difﬁcult to reason about several critical aspects of cross-task learning, such as the quantiﬁcation of the value of the source data and the impact of ﬁne-tuning (Kalan & Fabian, 2020;
Revised: Such an assumption also makes it difﬁcult to reason about several critical aspects of cross-task learning, such as the quantiﬁcation of the value of the source data and the impact of ﬁne-tuning (Kalan & Fabian, 2020;

Original: Chua et al., 2021).
Revised: Chua et al., 2021).

Original: In this paper, we propose T arget-
Revised: In this paper, we propose T arget-

Original: A ware W eighted T raining ( TAWT ), a weighted training algorithm for efﬁcient cross-task learning.
Revised: A ware W eighted T raining ( TAWT ), a weighted training algorithm for efﬁcient cross-task learning.

Original: The algorithm can be easily applied to existing cross-task learning paradigms, such as pre-training and joint training, to boost their sample efﬁciency by assigning adaptive (i.e., trainable) weights on the source tasks or source samples.
Revised: The algorithm can be easily applied to existing cross-task learning paradigms, such as pre-training and joint training, to boost their sample efﬁciency by assigning adaptive (i.e., trainable) weights on the source tasks or source samples.

Original: The weights are determined in a theoretically principled way by minimizing a representation-based task distance between the source and target tasks.
Revised: The weights are determined in a theoretically principled way by minimizing a representation-based task distance between the source and target tasks.

Original: Such a strategy is in sharp contrast to other weighting schemes common in machine learning, such as importance sampling in domain adaptation (Shimodaira, 2000; Cortes et al., 2010; Jiang & Zhai, 2007).
Revised: Such a strategy is in sharp contrast to other weighting schemes common in machine learning, such as importance sampling in domain adaptation (Shimodaira, 2000; Cortes et al., 2010; Jiang & Zhai, 2007).

Original: The effectiveness of TAWT is veriﬁed via both theoretical analyses and empirical experiments.
Revised: The effectiveness of TAWT is veriﬁed via both theoretical analyses and empirical experiments.

Original: Using empirical process theory, we prove a non-asymptotic generalization bound for TAWT .
Revised: Using empirical process theory, we prove a non-asymptotic generalization bound for TAWT .

Original: The bound is a superposition of two vanishing terms and a term depending on the task distance, the latter of which is potentially negligible due to the re-weighting operation.
Revised: The bound is a superposition of two vanishing terms and a term depending on the task distance, the latter of which is potentially negligible due to the re-weighting operation.

Original: We then conduct comprehensive experiments on four sequence tagging tasks in NLP: part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER).
Revised: We then conduct comprehensive experiments on four sequence tagging tasks in NLP: part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER).

Original: We demonstrate that TAWT further improves the performance of
Revised: We demonstrate that TAWT further improves the performance of

Original: BERT (Devlin et al., 2019) in both pre-training and joint training for cross-task learning with limited target data, achieving an average absolute improvement of 3 . 1% on the performance.
Revised: BERT (Devlin et al., 2019) in both pre-training and joint training for cross-task learning with limited target data, achieving an average absolute improvement of 3 . 1% on the performance.

Original: As a byproduct, we propose a representation-based task distance that depends on the quality of representations for each task, respectively, instead of assuming the existence of a single shared representation among all tasks.
Revised: As a byproduct, we propose a representation-based task distance that depends on the quality of representations for each task, respectively, instead of assuming the existence of a single shared representation among all tasks.

Original: This ﬁner-grained notion of task distance enables a better understanding of cross-task signals.
Revised: This ﬁner-grained notion of task distance enables a better understanding of cross-task signals.

Original: For example, the representation-based task distance gives an interpretable measure of the value of the source data on the target task based on the discrepancy between their optimal representations.
Revised: For example, the representation-based task distance gives an interpretable measure of the value of the source data on the target task based on the discrepancy between their optimal representations.

Original: Such a measure is more informative than measuring the difference between tasks via the discrepancy of their task-speciﬁc functions (e.g. linear functions) as done in previous theoretical frameworks (Tripuraneni et al., 2020).
Revised: Such a measure is more informative than measuring the difference between tasks via the discrepancy of their task-speciﬁc functions (e.g. linear functions) as done in previous theoretical frameworks (Tripuraneni et al., 2020).

Original: Furthermore, the representation-based task distance clearly conveys the necessity of ﬁne-tuning: if this distance is non-zero, then ﬁne-tuning the representation becomes necessary as the representation learned from the source data does not converge to the optimal target representation.
Revised: Furthermore, the representation-based task distance clearly conveys the necessity of ﬁne-tuning: if this distance is non-zero, then ﬁne-tuning the representation becomes necessary as the representation learned from the source data does not converge to the optimal target representation.

Original: Finally, we compare our work with some recent attempts in similar directions.
Revised: Finally, we compare our work with some recent attempts in similar directions.

Original: Liu et al.
Revised: Liu et al.

Original: (2020) analyze the beneﬁts of transfer learning by distinguishing source-speciﬁc features and transferable features in the source data.
Revised: (2020) analyze the beneﬁts of transfer learning by distinguishing source-speciﬁc features and transferable features in the source data.

Original: Based on the two types of features, they further propose a meta representation learning algorithm to encourage learning transferable and generalizable features.
Revised: Based on the two types of features, they further propose a meta representation learning algorithm to encourage learning transferable and generalizable features.

Original: Instead of focusing on the distinction between two types of features, our algorithm and analyses are based on the representation-based task distance and are thus different.
Revised: Instead of focusing on the distinction between two types of features, our algorithm and analyses are based on the representation-based task distance and are thus different.

Original: Chua et al.
Revised: Chua et al.

Original: (2021) present a theoretical framework for analyzing representations derived from model agnostic meta-learning (Finn et al., 2017), assuming all the tasks use approximately the same underlying representation.
Revised: (2021) present a theoretical framework for analyzing representations derived from model agnostic meta-learning (Finn et al., 2017), assuming all the tasks use approximately the same underlying representation.

Original: In contrast, we do not impose any a priori assumption on the proximity among source and target representations, and our algorithm seeks for a weighting scheme to maximize the proximity.
Revised: In contrast, we do not impose any a priori assumption on the proximity among source and target representations, and our algorithm seeks for a weighting scheme to maximize the proximity.

Revised: Our work is also different from task weighting in curriculum learning.

Revised: This line of work tends to learn suitable weights in the stochastic policy to decide which task to study next in curriculum learning (Graves et al., 2017), while TAWT aims to learn better representations by assigning more suitable weights on source tasks.

Revised: Compared to heuristic weighting strategies in multi-task learning (Gong et al., 2019; Zhang & Yang, 2021), we aim to design a practical algorithm with theoretical guarantees for cross-task learning.

Original:  TAWT : T ARGET -A WARE W EIGHTED T RAINING 2. P RELIMINARIES
Revised:  TAWT : T ARGET -A WARE W EIGHTED T RAINING 2. P RELIMINARIES

Original: Suppose we have T source tasks, represented by a collection of probability distributions {D t } Tt = on the sample space X × Y ,
Revised: Suppose we have T source tasks, represented by a collection of probability distributions {D t } Tt = on the sample space X × Y ,

Original: where
Revised: where

Original: X ⊆ R
Revised: X ⊆ R

Original: d is the feature space and Y ⊆
Revised: d is the feature space and Y ⊆

Original: R is the label space 1 .
Revised: R is the label space.

Revised: For classiﬁcation problems, we take Y to be a ﬁnite subset of R .

Original: We have a single target task, whose probability distribution is denoted as D 0 .
Revised: We have a single target task, whose probability distribution is denoted as D 0 .

Original: For the t -th task, where t = 0 , 1 , . . .
Revised: For the t -th task, where t = 0 , 1 , . . .

Original: , T , we observe n t i.i.d.
Revised: , T , we observe n t i.i.d.

Original: samples S t = { ( x ti , y ti ) } n t
Revised: samples S t = { ( x ti , y ti ) } n t

Original: i
Revised: i

Original: =1 from D t .
Revised: =1 from D t .

Original: Typically, the number of samples from the target task, n 0 , is much smaller than the samples from the source tasks, and the goal is to use samples from source tasks to aid the learning of the target task.
Revised: Typically, the number of samples from the target task, n 0 , is much smaller than the samples from the source tasks, and the goal is to use samples from source tasks to aid the learning of the target task.

Original: Let Φ be a collection of representations from the feature space X to some latent space Z ⊆ R r .
Revised: Let Φ be a collection of representations from the feature space X to some latent space Z ⊆ R r .

Original: We refer to Φ as the representation class .
Revised: We refer to Φ as the representation class .

Original: Let F be a collection of task-speciﬁc functions from the latent space Z to the label space Y .
Revised: Let F be a collection of task-speciﬁc functions from the latent space Z to the label space Y .

Original: The complexity of the representation class Φ is usually much larger (i.e., more expressive) than that of the task-speciﬁc function class F .
Revised: The complexity of the representation class Φ is usually much larger (i.e., more expressive) than that of the task-speciﬁc function class F .

Original: Given a bounded loss function (cid:96) :
Revised: Given a bounded loss function (cid:96) :

Original: Y ×Y → [0 , 1] , the optimal pair of representation and task-speciﬁc function of the t -th task is given by
Revised: Y ×Y → [0 , 1] , the optimal pair of representation and task-speciﬁc function of the t -th task is given by

Original: [Equation]
Revised: [Equation]

Original: Note that in general, the optimal representations of different tasks are different.
Revised: Note that in general, the optimal representations of different tasks are different.

Original: For brevity, all proofs for the theory part are deferred to the Appx.
Revised: For brevity, all proofs for the theory part are deferred to the Appx.

Original: A.
Revised: A.

Original: 2. D ERIVATION OF TAWT
Revised: 2. D ERIVATION OF TAWT

Original: Under the assumption that the optimal representations { φ (cid:63)t } Tt =0 are similar, a representation learned using samples only from the source tasks would perform reasonably well on the target task.
Revised: Under the assumption that the optimal representations { φ (cid:63)t } Tt =0 are similar, a representation learned using samples only from the source tasks would perform reasonably well on the target task.

Original: Consequently, we can devote n 0 samples from the target task to learn only the task speciﬁc function.
Revised: Consequently, we can devote n 0 samples from the target task to learn only the task speciﬁc function.

Original: This is a much easier task, since the complexity of F is typically much smaller than that of Φ .
Revised: This is a much easier task, since the complexity of F is typically much smaller than that of Φ .

Original: This discussion leads to a simple yet immensely popular two-step procedure as follows (Tripuraneni et al., 2020; Du et al., 2021).
Revised: This discussion leads to a simple yet immensely popular two-step procedure as follows (Tripuraneni et al., 2020; Du et al., 2021).

Original: First, we solve a weighted empirical risk minimization problem with respect to the source tasks:
Revised: First, we solve a weighted empirical risk minimization problem with respect to the source tasks:

Original: [Equation]
Revised: [Equation]

Original: where ω ∈ ∆ T − 1 is a user-speciﬁed vector lying in the T -dimensional probability simplex (i.e., (cid:80)
Revised: where ω ∈ ∆ T − 1 is a user-speciﬁed vector lying in the T -dimensional probability simplex (i.e., (cid:80)

Original: Tt =1 ω t = 1 and ω t ≥ 0 , ∀ 1 ≤ t ≤ T ).
Revised: Tt =1 ω t = 1 and ω t ≥ 0 , ∀ 1 ≤ t ≤ T ).

Original: In the second stage, we freeze the representation (cid:98) φ , andwe seek the task-speciﬁc function that minimizes the empirical risk with respect to the target task:
Revised: In the second stage, we freeze the representation (cid:98) φ , and seek the task-speciﬁc function that minimizes the empirical risk with respect to the target task:

Original: [Equation]
Revised: [Equation]

Original: In practice, we can allow (cid:98) φ to slightly vary (e.g., via ﬁne-tuning) to get a performance boost.
Revised: In practice, we can allow (cid:98) φ to slightly vary (e.g., via ﬁne-tuning) to get a performance boost.

Original: In the two-step procedure (2.2)–(2.3), the weight vector ω is usually taken to be a hyperparameter and is ﬁxed during training.
Revised: In the two-step procedure (2.2)–(2.3), the weight vector ω is usually taken to be a hyperparameter and is ﬁxed during training.

Original: Popular choices include the uniform weights (i.e., ω t = 1 /T ) or weights proportional to the sample sizes (i.e., ω t = n t / (cid:80) Tt [Equation]
Revised: Popular choices include the uniform weights (i.e., ω t = 1 /T ) or weights proportional to the sample sizes (i.e., ω t = n t / (cid:80) Tt (cid:48)

Original: =1 n t [Equation]
Revised: =1 n t (cid:48) )

Original: ) (Liu et al., 2019; Johnson & Khoshgoftaar, 2019).
Revised: (Liu et al., 2019; Johnson & Khoshgoftaar, 2019).

Original: This reveals the target-agnostic nature of the two-step procedure (2.2)–(2.3): the weights stay the same regardless the level of proximity between the source tasks and the target task.
Revised: This reveals the target-agnostic nature of the two-step procedure (2.2)–(2.3): the weights stay the same regardless the level of proximity between the source tasks and the target task.

Original: Consider the following thought experiment: if we know a priori that the ﬁrst source task D 1 is closer (compared to other source tasks) to the target task D 0 , then we would expect a better performance by raising the importance of D 1 , i.e., make ω 1 larger.
Revised: Consider the following thought experiment: if we know a priori that the ﬁrst source task D 1 is closer (compared to other source tasks) to the target task D 0 , then we would expect a better performance by raising the importance of D 1 , i.e., make ω 1 larger.

Original: This thought experiment motivates a target-aware procedure that adaptively adjusts the weights based on the proximity of source tasks to the target.
Revised: This thought experiment motivates a target-aware procedure that adaptively adjusts the weights based on the proximity of source tasks to the target.

Original: A natural attempt for developing such a task-aware procedure is a follows:
Revised: A natural attempt for developing such a task-aware procedure is a follows:

Original: [Equation]
Revised: [Equation]

Original: That is, we seek for the best weights ω such that solving (2.2) with this choice of ω would lead to the lowest training error when we subsequently solve (2.3).
Revised: That is, we seek for the best weights ω such that solving (2.2) with this choice of ω would lead to the lowest training error when we subsequently solve (2.3).

Original: Despite its conceptual simplicity, the formulation (OPT1) is a complicated constrained optimization problem.
Revised: Despite its conceptual simplicity, the formulation (OPT1) is a complicated constrained optimization problem.

Original: Nevertheless, we demonstrate that it is possible to transform it into an unconstrained form for which a customized gradient-based optimizer could be applied.
Revised: Nevertheless, we demonstrate that it is possible to transform it into an unconstrained form for which a customized gradient-based optimizer could be applied.

Original: To do so, we let ( φ ω , { f ω t } ) be any representation and task-speciﬁc functions that minimizes (cid:80) Tt =1 ω t (cid:98)
Revised: To do so, we let ( φ ω , { f ω t } ) be any representation and task-speciﬁc functions that minimizes (cid:80) Tt =1 ω t (cid:98)

Original: L t ( φ, f t ) over φ ∈ Φ and { f t } ⊂ F .
Revised: L t ( φ, f t ) over φ ∈ Φ and { f t } ⊂ F .

Original: Equivalently, φ ω minimizes (cid:80) Tt =1 ω t min f t ∈F (cid:98) 
Revised: Equivalently, φ ω minimizes (cid:80) Tt =1 ω t [Equation] (cid:98) L [Equation]

Revised: ( φ

Revised: [Equation]

Original: L t ( φ, f t ) over φ ∈ Φ .
Revised: f [Equation] ) over φ ∈ Φ .

Original: With such notations, we can re-write (OPT1) as
Revised: With such notations, we can re-write (OPT1) as

Original: [Equation]
Revised: [Equation]

Original: The gradient of the above objective with respect to the task-speciﬁc function, ∇ f (cid:98) L 0 ( φ ω , f 0 ) , is easy to calculate via back-propagation.
Revised: The gradient of the above objective with respect to the task-speciﬁc function, ∇ f (cid:98) L 0 ( φ ω , f 0 ) , is easy to calculate via back-propagation.

Original: The calculation of the gradient with respect to the weights requires more work, as φ ω is an implicit function of ω .
Revised: The calculation of the gradient with respect to the weights requires more work, as φ ω is an implicit function of ω .

Original: By the chain rule, we have ∂∂ω t (cid:98) L 0 ( φ ω , f 0 ) =
Revised: By the chain rule, we have ∂∂ω t (cid:98) L 0 ( φ ω , f 0 ) =

Original: [ ∇ φ (cid:98) L 0 ( φ ω , f 0 )] (cid:62) ∂∂ω t φ ω .
Revised: [ ∇ φ (cid:98) L 0 ( φ ω , f 0 )] (cid:62) ∂ ∂ω t φ ω .

Original: Since φ ω is a minimizer of φ (cid:55)→ (cid:80) Tt =1 ω t min f t ∈F (cid:98) L t ( φ, f t ) , we haveF ( φ ω , ω ) = 0 , ∀ ω ∈ ∆ T − 1 , F ( [Equation]
Revised: Since φ ω is a minimizer of φ (cid:55)→ (cid:80) Tt =1 ω t min f t ∈F (cid:98) L t ( φ, f t ) , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:88) [Equation] [Equation] By implicit function theorem, if F ( · , · ) is everywhere differentiable and the matrix ∂F ( φ, ω ) /∂φ is invertible for any ( φ, ω ) near some ( (cid:101) φ, (cid:101) ω ) satisfying F ( (cid:101) φ, (cid:101) ω )
Revised: By implicit function theorem, if F ( · , · ) is everywhere differentiable and the matrix ∂F ( φ, ω ) /∂φ is invertible for any ( φ, ω ) near some ( (cid:101) φ, (cid:101) ω ) satisfying F ( (cid:101) φ, (cid:101) ω )

Original: = 0 , then we can conclude that the map ω (cid:55)→ φ ω is a locally well-deﬁned function near (cid:101) ω , and the derivative of this map is given by
Revised: = 0 , then we can conclude that the map ω (cid:55)→ φ ω is a locally well-deﬁned function near (cid:101) ω , and the derivative of this map is given by

Original: [Equation]
Revised: [Equation]

Original: To simplify the above expression, note that under regularity conditions, we can regard ∇ φ (cid:98) L t ( φ, f ω t ) as a sub-gradient of the map φ (cid:55)→ min f t ∈F (cid:98) L t ( φ, f t ) .
Revised: To simplify the above expression, note that under regularity conditions, we can regard ∇ φ (cid:98) L t ( φ, f ω t ) as a sub-gradient of the map φ (cid:55)→ min f t ∈F (cid:98) L t ( φ, f t ) .

Original: This means that we can write F ( φ ω , ω )
Revised: This means that we can write F ( φ ω , ω )

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: φ (cid:98) L t ( φω , f ω t ) . Plugging this expression back to (2.5) and recalling the expression for ∂ (cid:98) L
Revised: t ∇ φ

Original: [Equation]
Revised: [Equation]

Original:  ( φ ω , f 0 ) /∂ω
Revised: L t ( φ ω , f ω t ) .

Revised: Plugging this expression back to (2.5) and recalling the expression for ∂ (cid:98) L 0 ( φ ω , f 0 ) /∂ω

Original: t derived via the chain rule, we get
Revised: t derived via the chain rule, we get

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] Now that we have the expressions for the gradients of (cid:98) L 0 ( φ ω , f 0 ) with respect to f 0 and ω , we can solve (OPT2) via a combination of alternating minimization and mirror descent.
Revised: Now that we have the expressions for the gradients of (cid:98) L 0 ( φ ω , f 0 ) with respect to f 0 and ω , we can solve (OPT2) via a combination of alternating minimization and mirror descent.

Original: To be more speciﬁc, suppose that at iteration k , the current weights, representation, and task-speciﬁc functions are ω k , φ k , and { f kt } T t =0 , respectively.
Revised: To be more speciﬁc, suppose that at iteration k , the current weights, representation, and task-speciﬁc functions are ω k , φ k , and { f kt } Tt =0 , respectively.

Original: At this iteration, we conduct the following three steps:
Revised: At this iteration, we conduct the following three steps:

Original: Freeze ω k .
Revised: Freeze ω k .

Original: Starting from ( φ k , { f kt } Tt =1 ) , run a few steps of SGD on the objective function φ
Revised: Starting from ( φ k , { f kt } Tt =1 ) , run a few steps of SGD on the objective function ( φ, { f t } Tt =1 )

Original: 1 ω kt 
Revised: (cid:55)→ (cid:80) Tt =1 ω kt (cid:98)

Original: L t ( φ, f t ) to get ( φ k +1 , { f k + t } Tt =1 ) , which is regarded as an approximation of ( φ ω k , { f ω k t } Tt =1 ) ; 2.
Revised: L t ( φ, f t ) to get ( φ k +1 , { f k +1 t } Tt =1 ) , which is regarded as an approximation of ( φ ω k , { f ω k t } Tt =1 ) ; 2.

Original: Freeze ( φ k +1 , { f k +1 t } Tt =1 ) .
Revised: Freeze ( φ k +1 , { f k +1 t } Tt =1 ) .

Original: Approximate the gradient ∇ f (cid:98) L 0 ( φ ω k , f 0 ) by ∇ f (cid:98) L 0 ( φ k +1 , f 0 ) .
Revised: Approximate the gradient ∇ f (cid:98) L 0 ( φ ω k , f 0 ) by ∇ f (cid:98) L 0 ( φ k +1 , f 0 ) .

Original: Using this approximate gradient, run a few steps of SGD from f k 0 to get f k +1;3.
Revised: Using this approximate gradient, run a few steps of SGD from f k 0 to get f k +1;

Original: Freeze ( φ k +1 , { f k +1 t } Tt =0 ) .
Revised: Freeze ( φ k +1 , { f k +1 t } Tt =0 ) .

Original: Approximate the partial derivative ∂ (cid:98) 
Revised: Approximate the partial derivative ∂ (cid:98) L 0 ( φ ω k , f k +1) /∂ω t by g k t := − [ ∇ φ (cid:98) L

Original: [Equation]
Revised: [Equation]

Original: by{+( φ k +1 , f k +
+}
Revised: {+( φ k +1 , f k +
+}

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then run one step of mirror descent (with step size η k ) from ω k to get ω k +1 :
Revised: Then run one step of mirror descent (with step size η k ) from ω k to get ω k +1 :

Original: [Equation]
Revised: [Equation]

Original: We use mirror descent in (2.8), as it is a canonical generalization of Euclidean gradient descent to gradient descent on the probability simplex (Beck & Teboulle, 2003).
Revised: We use mirror descent in (2.8), as it is a canonical generalization of Euclidean gradient descent to gradient descent on the probability simplex (Beck & Teboulle, 2003).

Revised: Note that other optimization methods, such as projected gradient descent, can also be used here.

Original: The update rule (2.8) has a rather intuitive explanation.
Revised: The update rule (2.8) has a rather intuitive explanation.

Original: Note that g kt is a weighted dissimilarity measure between the gradients ∇ φ (cid:98) L 0 and ∇ φ (cid:98) L t .
Revised: Note that g kt is a weighted dissimilarity measure between the gradients ∇ φ (cid:98) L 0 and ∇ φ (cid:98) L t .

Original: This can further be regarded as a crude dissimilarity measure between the optimal representations of the target task and the t -th source task.
Revised: This can further be regarded as a crude dissimilarity measure between the optimal representations of the target task and the t -th source task.

Original: The mirror descent updates ω t along the direction where the target task and the t -th source task are more similar.
Revised: The mirror descent updates ω t along the direction where the target task and the t -th source task are more similar.

Original: The overall procedure is summarized in Algorithm 1.
Revised: The overall procedure is summarized in Algorithm 1.

Original: A faithful implementation of the above steps would require a costly evaluation of the inverse of the Hessian matrix (cid:80) T t =1 ω t ∇ 2 φ (cid:98)
Revised: A faithful implementation of the above steps would require a costly evaluation of the inverse of the Hessian matrix (cid:80) Tt =1 ω t ∇ 2 φ (cid:98)

Original: L t ( φ k +1 , f k +1 t ) ∈
Revised: L t ( φ k +1 , f k +1 t ) ∈

Original: R r × r .
Revised: R r × r .

Original: In practice, we can bypass this step by replacing 2 the Hessian-inverse-weighted dissimilarity measure (2.7) with a consine-similarity-based dissimilarity measure (see Section 4 for details).
Revised: In practice, we can bypass this step by replacing 2 the Hessian-inverse-weighted dissimilarity measure (2.7) with a consine-similarity-based dissimilarity measure (see Section 4 for details).

Original: The previous derivation has focused on weighted pre-training, i.e., the target data is not used when deﬁning the constrained set in (OPT1).
Revised: The previous derivation has focused on weighted pre-training, i.e., the target data is not used when deﬁning the constrained set in (OPT1).

Original: It can be modiﬁed, mutatis mutandis , to handle weighted joint-training, where we change (OPT1) to
Revised: It can be modiﬁed, mutatis mutandis , to handle weighted joint-training, where we change (OPT1) to

Original: [Equation]
Revised: [Equation]

Original: Compared to (OPT1), we now also use the data from the target task when learning the representation φ , and thus there is an extra weight ω 0 on the target task.
Revised: Compared to (OPT1), we now also use the data from the target task when learning the representation φ , and thus there is an extra weight ω 0 on the target task.

Original: The algorithm can also be easily extended to handle multiple target tasks or to put weights on samples (as opposed to putting weights on tasks).
Revised: The algorithm can also be easily extended to handle multiple target tasks or to put weights on samples (as opposed to putting weights on tasks).

Original: The algorithm could also be applied to improve the efﬁciency of learning from cross-domain and cross-lingual signals, and we postpone such explorations for future work.
Revised: The algorithm could also be applied to improve the efﬁciency of learning from cross-domain and cross-lingual signals, and we postpone such explorations for future work.

Original:  T HEORETICAL G UARANTEES 3. A R EPRESENTATION -B ASED T ASK D ISTANCE
Revised:  T HEORETICAL G UARANTEES 3. A R EPRESENTATION -B ASED T ASK D ISTANCE

Original: In this subsection, we introduce a representation-based task distance, which will be crucial in the theoretical understanding of TAWT .
Revised: In this subsection, we introduce a representation-based task distance, which will be crucial in the theoretical understanding of TAWT .

Original: To start with, let us deﬁne the representation and task-speciﬁc functions that are optimal in an “ ω -weighted sense” as follows:
Revised: To start with, let us deﬁne the representation and task-speciﬁc functions that are optimal in an “ ω -weighted sense” as follows:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, (¯ φ ω , { ¯ f ω t } ) are optimal on the ω -weighted source tasks when only a single representation is used.
Revised: Intuitively, (¯ φ ω , { ¯ f ω t } ) are optimal on the ω -weighted source tasks when only a single representation is used.

Original: Since ¯ φ ω may not be unique, we introduce the function space ¯Φ ω ⊂
Revised: Since ¯ φ ω may not be unique, we introduce the function space ¯Φ ω ⊂

Original: Φ to collect all ¯ φ ω s that satisfy (3.1).
Revised: Φ to collect all ¯ φ ω s that satisfy (3.1).

Original: To further simplify the notation, we let L (cid:63)t ( φ ) :
Revised: To further simplify the notation, we let L (cid:63)t ( φ ) :

Original: = min f t ∈F L t ( φ, f t ) , which stands for the risk incurred by the representation φ on the t -th task.
Revised: = min f t ∈F L t ( φ, f t ) , which stands for the risk incurred by the representation φ on the t -th task.

Original: With the foregoing notation, we can write ¯ φ ω ∈ argmin φ ∈ Φ (cid:80)
Revised: With the foregoing notation, we can write ¯ φ ω ∈ argmin φ ∈ Φ (cid:80)

Original: Tt =1 ω t L (cid:63)t ( φ ) .
Revised: Tt =1 ω t L (cid:63)t ( φ ) .

Original: The deﬁnition of the task distance is given below.
Revised: The deﬁnition of the task distance is given below.

Original: Deﬁnition 3.1 (Representation-based task distance) .
Revised: Deﬁnition 3.1 (Representation-based task distance) .

Original: The representation-based task distance between the ω -weighted source tasks and the target task is deﬁned as
Revised: The representation-based task distance between the ω -weighted source tasks and the target task is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where the supremum is taken over any ¯ φ ω satisfying (3.1), and φ (cid:63) 0 is the optimal target representation.
Revised: where the supremum is taken over any ¯ φ ω satisfying (3.1), and φ (cid:63) 0 is the optimal target representation.

Original: If all the tasks share the same optimal representation, then above distance becomes exactly zero.
Revised: If all the tasks share the same optimal representation, then above distance becomes exactly zero.

Original: Under such an assumption, the only source of discrepancy among tasks arises from the difference in their task-speciﬁc functions.
Revised: Under such an assumption, the only source of discrepancy among tasks arises from the difference in their task-speciﬁc functions.

Original: This can be problematic in practice, as the task-speciﬁc functions alone are usually not expressive enough to describe the intrinsic difference among tasks.
Revised: This can be problematic in practice, as the task-speciﬁc functions alone are usually not expressive enough to describe the intrinsic difference among tasks.

Original: In contrast, we relax the shared representation assumption substantially by allowing the optimal representations to differ and the distance to be non-zero.
Revised: In contrast, we relax the shared representation assumption substantially by allowing the optimal representations to differ and the distance to be non-zero.

Original: The above notion of task distance also allows us to reason about certain important aspects of cross-task learning.
Revised: The above notion of task distance also allows us to reason about certain important aspects of cross-task learning.

Original: For example, this task distance is asymmetric, capturing the asymmetric nature of cross-task learning.
Revised: For example, this task distance is asymmetric, capturing the asymmetric nature of cross-task learning.

Original: Moreover, if the task distance is non-zero, then ﬁne-tuning the representation becomes necessary, because solving (2.2) alone would gives a representation that does not converge to the correct target φ (cid:63) 0 .
Revised: Moreover, if the task distance is non-zero, then ﬁne-tuning the representation becomes necessary, because solving (2.2) alone would gives a representation that does not converge to the correct target φ (cid:63) 0 .

Original: In addition, an empirical illustration of the representation-based task distance can be found in Fig.
Revised: In addition, an empirical illustration of the representation-based task distance can be found in Fig.

Original: 3 in Appx.
Revised: 3 in Appx.

Original: B.
Revised: B.

Original: This task distance can be naturally estimated from the data by replacing all population quantities with their empirical version.
Revised: This task distance can be naturally estimated from the data by replacing all population quantities with their empirical version.

Original: Indeed, minimizing the estimated task distance over the weights is equivalent to the optimization formulation (OPT1) of our algorithm (see Appx.
Revised: Indeed, minimizing the estimated task distance over the weights is equivalent to the optimization formulation (OPT1) of our algorithm (see Appx.

Original: A.1 for a detailed derivation).
Revised: A.1 for a detailed derivation).

Original: This observation gives an alternative and theoretically principled derivation of TAWT .
Revised: This observation gives an alternative and theoretically principled derivation of TAWT .

Original: 3. P ERFORMANCE G UARANTEES FOR TAWT
Revised: 3. P ERFORMANCE G UARANTEES FOR TAWT

Original: To give theoretical guarantees for TAWT , we need a few standard technical assumptions.
Revised: To give theoretical guarantees for TAWT , we need a few standard technical assumptions.

Original: The ﬁrst one concerns the Lipschitzness of the function classes, and the second one controls the complexity of the function classes via uniform entropy (Wellner & van der Vaart, 2013) as follows.
Revised: The ﬁrst one concerns the Lipschitzness of the function classes, and the second one controls the complexity of the function classes via uniform entropy (Wellner & van der Vaart, 2013) as follows.

Original: Assumption A (Lipschitzness) .
Revised: Assumption A (Lipschitzness) .

Original: The loss function (cid:96) : Y × Y → [0 , 1] is L (cid:96) -Lipschitz in the ﬁrst argument, uniformly over the second argument.
Revised: The loss function (cid:96) : Y × Y → [0 , 1] is L (cid:96) -Lipschitz in the ﬁrst argument, uniformly over the second argument.

Original: Any f ∈ F is L F -Lipschitz w.r.t.
Revised: Any f ∈ F is L F -Lipschitz w.r.t.

Original: the (cid:96) 2 norm.
Revised: the (cid:96) 2 norm.

Original: Assumption B (Uniform entropy control of function classes) .
Revised: Assumption B (Uniform entropy control of function classes) .

Original: There exist C Φ > 0 , ν Φ > 0 , such that for any probability measure Q X on X ⊆ R d , we have
Revised: There exist C Φ > 0 , ν Φ > 0 , such that for any probability measure Q X on X ⊆ R d , we have

Original: [Equation]
Revised: [Equation]

Original: ε > [Equation] [Equation] .3) where N (Φ; L 2 ( Q X ); ε ) is the L 2 ( Q X ) covering number of Φ (i.e., the minimum number of L 2 ( Q X ) balls 3 with radius ε required to cover Φ ).
Revised: where N (Φ; L 2 ( Q X ); ε ) is the L 2 ( Q X ) covering number of Φ (i.e., the minimum number of L 2 ( Q X ) balls 3 with radius ε required to cover Φ ).

Original: In parallel, there exist C F > 0 , ν F > 0 , such that for any probability measure Q Z on Z ⊆ R r , we have
Revised: In parallel, there exist C F > 0 , ν F > 0 , such that for any probability measure Q Z on Z ⊆ R r , we have

Original: [Equation]
Revised: [Equation]

Original: where N ( F ; L 2 ( Q Z )
Revised: where N ( F ; L 2 ( Q Z )

Original: ; ε ) is the L 2 ( Q Z ) covering number of F .
Revised: ; ε ) is the L 2 ( Q Z ) covering number of F .

Original: Uniform entropy generalizes the notion of Vapnik-Chervonenkis dimension (Vapnik, 2013) and allows us to give a uniﬁed treatment of regression and classiﬁcation problems.
Revised: Uniform entropy generalizes the notion of Vapnik-Chervonenkis dimension (Vapnik, 2013) and allows us to give a uniﬁed treatment of regression and classiﬁcation problems.

Original: For this reason, function classes satisfying the above assumption are also referred to as “VC-type classes” in the literature (Koltchinskii, 2006).
Revised: For this reason, function classes satisfying the above assumption are also referred to as “VC-type classes” in the literature (Koltchinskii, 2006).

Original: In particular, if each coordinate of Φ has VC-subgraph dimension c (Φ) , then (3.3) is satisﬁed with ν Φ
Revised: In particular, if each coordinate of Φ has VC-subgraph dimension c (Φ) , then (3.3) is satisﬁed with ν Φ

Original: =
Revised: =

Original: Θ( r · c (Φ)) (recall that r is the dimension of the latent space
Revised: Θ( r · c (Φ)) (recall that r is the dimension of the latent space

Original: Z ).
Revised: Z ).

Original: Similarly, if F has VC-subgraph dimension c ( F ) , then (3.4) is satisﬁed with ν F = Θ( c ( F )) .
Revised: Similarly, if F has VC-subgraph dimension c ( F ) , then (3.4) is satisﬁed with ν F = Θ( c ( F )) .

Original: The following deﬁnition characterizes how “transferable” a representation φ is from the ω -weighted source tasks to the target task.
Revised: The following deﬁnition characterizes how “transferable” a representation φ is from the ω -weighted source tasks to the target task.

Original: Deﬁnition 3.2 (Transferability) .
Revised: Deﬁnition 3.2 (Transferability) .

Original: A representation φ ∈ Φ is ( ρ, C ρ ) -transferable from ω -weighted source tasks to the target task, if there exists ρ > 0 , C ρ > 0 such that for any ¯ φ ω ∈ ¯Φ ω , we have
Revised: A representation φ ∈ Φ is ( ρ, C ρ ) -transferable from ω -weighted source tasks to the target task, if there exists ρ > 0 , C ρ > 0 such that for any ¯ φ ω ∈ ¯Φ ω , we have

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the above deﬁnition says that relative to ¯ φ ω , the risk of φ on the target task can be controlled by a polynomial of the average risk of φ on the source tasks.
Revised: Intuitively, the above deﬁnition says that relative to ¯ φ ω , the risk of φ on the target task can be controlled by a polynomial of the average risk of φ on the source tasks.

Original: This can be regarded as an adaptation of the notions of transfer exponent and relative signal exponent (Hanneke & Kpotufe, 2019; Cai & Wei, 2021) originated from the transfer learning literature to the representation learning setting.
Revised: This can be regarded as an adaptation of the notions of transfer exponent and relative signal exponent (Hanneke & Kpotufe, 2019; Cai & Wei, 2021) originated from the transfer learning literature to the representation learning setting.

Original: This can also be seen as a generalization of the task diversity assumption (Tripuraneni et al., 2020; Du et al., 2021) to the case when the optimal representations { [Equation]
Revised: This can also be seen as a generalization of the task diversity assumption (Tripuraneni et al., 2020; Du et al., 2021) to the case when the optimal representations { φ (cid:63)t } Tt =1 do not coincide.

Original: not coincide.
Revised: In addition, Tripuraneni et al. (2020) and Du et al. (2021) proved that ρ = 1 in certain simple models and under some simpliﬁed setting.

Original: In this part, we prove a non-asymptotic generalization bound for TAWT .
Revised: In this part, we prove a non-asymptotic generalization bound for TAWT .

Original: The fact that the weights are learned from the data substantially complicates the analysis.
Revised: The fact that the weights are learned from the data substantially complicates the analysis.

Original: To proceed further, we make a few simplifying assumptions.
Revised: To proceed further, we make a few simplifying assumptions.

Original: First, we assume that the sample sizes of the source data are relatively balanced: there exists an integer n such that n t =
Revised: First, we assume that the sample sizes of the source data are relatively balanced: there exists an integer n such that n t =

Original: Θ( n ) for any t ∈ { 1 , . .
Revised: Θ( n ) for any t ∈ { 1 , . .

Original: , T } .
Revised: , T } .

Original: Meanwhile, instead of directly analyzing (OPT1), we focus on its sample split version.
Revised: Meanwhile, instead of directly analyzing (OPT1), we focus on its sample split version.

Original: In particular, we let B 1 ∪ B 2 be a partition of { 1 , . . . , n 0 } , where | B 1 | =
Revised: In particular, we let B 1 ∪ B 2 be a partition of { 1 , . . . , n 0 } , where | B 1 | =

Original: Θ( | B 2 | 
Revised: Θ( | B 2 | ) .

Revised: Deﬁne (cid:98) L (1)0 ( φ, f 0 ) :

Original:  [Equation]
Revised: = [Equation]

Original: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst solve (OPT1) restricted to the ﬁrst part of the data:
Revised: We ﬁrst solve (OPT1) restricted to the ﬁrst part of the data:

Original: [Equation]
Revised: [Equation]

Revised: argmin

Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Revised: ∈ argmin

Revised: [Equation]

Revised: [Equation]

Original: Then, we proceed by solving
Revised: Then, we proceed by solving

Original: [Equation]
Revised: [Equation]

Original: Such a sample splitting ensures the independence of (cid:98) φ and the second part of target data B 2 , hence allowing for more transparent theoretical analyses 4 .
Revised: Such a sample splitting ensures the independence of (cid:98) φ and the second part of target data B 2 , hence allowing for more transparent theoretical analyses.

Revised: Such strategies are common in statistics and econometrics literature when the algorithm has delicate dependence structures (see, e.g., (Hansen, 2000; Chernozhukov et al., 2018)).

Original: We emphasize that sample splitting is conducted only for theoretical convenience and is not used in practice.
Revised: We emphasize that sample splitting is conducted only for theoretical convenience and is not used in practice.

Original: The following theorem gives performance guarantees for the sample split version of TAWT .
Revised: The following theorem gives performance guarantees for the sample split version of TAWT .

Original: Theorem 3.1 (Performance of TAWT with sample splitting) .
Revised: Theorem 3.1 (Performance of TAWT with sample splitting) .

Original: Let ( (cid:98) φ, (cid:98) f 0 ) be obtained via solving (3.6)–(3.7).
Revised: Let ( (cid:98) φ, (cid:98) f 0 ) be obtained via solving (3.6)–(3.7).

Original: Let Assumptions A and B hold.
Revised: Let Assumptions A and B hold.

Original: In addition, assume that the learned weights satisfy (cid:98) ω ∈ W β := { ω ∈ ∆ T − 1 : β − 1 ≤ ω t /ω t (cid:48) ≤ β, ∀ t (cid:54) =
Revised: In addition, assume that the learned weights satisfy (cid:98) ω [Equation] , ∀ [Equation]

Original:  t (cid:48) } , where β ≥ 1 is an absolute constant.
Revised: (cid:54) = t (cid:48) } , where β ≥ 1 is an absolute constant.

Original: Fix δ ∈ (0 , 1) .
Revised: Fix δ ∈ (0 , 1) .

Original: There exists a constant C = C
Revised: There exists a constant C = C

Original: ( L (cid:96) , L F , C Φ , C F ) > 

Revised: ( L (cid:96) , L F , C Φ , C F ) >

Original: such that the following holds: if for any weights ω ∈ W β and any representation φ in a Cβ · (cid:112) ( ν Φ log δ − 1 ) /nT
Revised: such that the following holds: if for any weights ω ∈ W β and any representation φ in a Cβ · (cid:112) ( ν Φ log δ − 1 ) /nT

Original: + ( ν F + log T ) /nneighborhood of ¯Φ ω 5 , there exists a speciﬁc ¯ φ ω ∈ ¯Φ ω
Revised: + ( ν F + log T ) /nneighborhood of ¯Φ ω 4 , there exists a speciﬁc ¯ φ ω ∈ ¯Φ ω

Original: such that ¯ φ ω is ( ρ, C ρ ) -transferable, then there exists another C (cid:48)
Revised: such that ¯ φ ω is ( ρ, C ρ ) -transferable, then there exists another C (cid:48)

Original: =
Revised: =

Original: C (cid:48)
Revised: C (cid:48)

Original: ( L (cid:96) , L F , C Φ , C F , C ρ , ρ ) such that with probability at least 1 − δ , we have[Equation] [Table]
Revised: ( L (cid:96) , L F , C Φ , C F , C ρ , ρ ) such that with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: The upper bound in (3.8) is a superposition of three terms.
Revised: The upper bound in (3.8) is a superposition of three terms.

Original: Let us disregard the log(1 /δ ) terms for now and focus on the dependence on the sample sizes and problem dimensions.
Revised: Let us disregard the log(1 /δ ) terms for now and focus on the dependence on the sample sizes and problem dimensions.

Original: The ﬁrst term, which scales with (cid:112) ν F /n 0 , corresponds to the error of learning the task-speciﬁc function in the target task.
Revised: The ﬁrst term, which scales with (cid:112) ν F /n 0 , corresponds to the error of learning the task-speciﬁc function in the target task.

Original: This is unavoidable even if the optimal representation φ (cid:63) 0 is known.
Revised: This is unavoidable even if the optimal representation φ (cid:63) 0 is known.

Original: The second term that scales with [( ν Φ + Tν F ) / ( nT )] 1 / 2 ρ characterizes the error of learning the imperfect representation ¯ φ ω from the source datasets and transferring the knowledge to the target task.
Revised: The second term that scales with [( ν Φ + Tν F ) / ( nT )] 1 / 2 ρ characterizes the error of learning the imperfect representation ¯ φ ω from the source datasets and transferring the knowledge to the target task.

Original: Note that this term is typically much smaller than (cid:112) ( ν Φ + ν F ) /n 0 , the error that would have been incurred when learning only from the target data, thus illustrating the potential beneﬁts of representation learning.
Revised: Note that this term is typically much smaller than (cid:112) ( ν Φ + ν F ) /n 0 , the error that would have been incurred when learning only from the target data, thus illustrating the potential beneﬁts of representation learning.

Original: This happens because nT is typically much larger than n 0 .
Revised: This happens because nT is typically much larger than n 0 .

Original: The third term is precisely the task distance introduced in Deﬁnition 3.1.
Revised: The third term is precisely the task distance introduced in Deﬁnition 3.1.

Original: The form of the task distance immediately demonstrates the possibility of “matching” ¯ φ ω ≈ φ (cid:63) 0 via varying the weights ω , under which case the third term would be negligible compared to the former two terms.
Revised: The form of the task distance immediately demonstrates the possibility of “matching” ¯ φ ω ≈ φ (cid:63) 0 via varying the weights ω , under which case the third term would be negligible compared to the former two terms.

Original: For example, in
Revised: For example, in

Original: Appx.
Revised: Appx.

Original: A.2, we give a sufﬁcient condition for exactly matching ¯ φ ω = φ (cid:63) 0 .
Revised: A.2, we give a sufﬁcient condition for exactly matching ¯ φ ω = φ (cid:63) 0 .

Original: The proof of Theorem 3.1 is based on empirical process theory.
Revised: The proof of Theorem 3.1 is based on empirical process theory.

Original: Along the way, we also establish an interesting result on multi-task learning, where the goal is to improve the average performance for all tasks instead of target tasks (see Lemma A.1 in Appx.
Revised: Along the way, we also establish an interesting result on multi-task learning, where the goal is to improve the average performance for all tasks instead of target tasks (see Lemma A.1 in Appx.

Original: A).
Revised: A).

Original: The current analysis can also be extended to cases where multiple target tasks are present.
Revised: The current analysis can also be extended to cases where multiple target tasks are present.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we verify the effectiveness of TAWT in extensive experiments using four NLP tasks, including PoS tagging, chunking, predicate detection, and NER.
Revised: In this section, we verify the effectiveness of TAWT in extensive experiments using four NLP tasks, including PoS tagging, chunking, predicate detection, and NER.

Original: More details are in Appx.
Revised: More details are in Appx.

Original: B.
Revised: B.

Original: Experimental settings.
Revised: Experimental settings.

Original: In our experiments, we mainly use two widely-used NLP datasets, Ontontes 5.0 (Hovy et al., 2006) and CoNLL-2000 (Tjong Kim Sang & Buchholz, 2000).
Revised: In our experiments, we mainly use two widely-used NLP datasets, Ontontes 5.0 (Hovy et al., 2006) and CoNLL-2000 (Tjong Kim Sang & Buchholz, 2000).

Original: Ontonotes 5. contains annotations for PoS tagging, predicate detection, and NER, and CoNLL-2000 is a shared task for chunking.
Revised: Ontonotes 5. contains annotations for PoS tagging, predicate detection, and NER, and CoNLL-2000 is a shared task for chunking.

Original: There are about 116 K sentences, 16 K sentences, and 12 K sentences in the training, development, and test sets for tasks in Ontonotes 5.0.
Revised: There are about 116 K sentences, 16 K sentences, and 12 K sentences in the training, development, and test sets for tasks in Ontonotes 5.0.

Original: As for CoNLL-2000, there are  about 9 K sentences and 2 K sentences in the training and test sets.
Revised: As for CoNLL-2000, there are [Table] about 9 K sentences and 2 K sentences in the training and test sets.

Original: As for the evaluation metric, we use accuracy for PoS tagging, span-level F1 for chunking, word-level F1 for predicate detection, and span-level F1 for NER.
Revised: As for the evaluation metric, we use accuracy for PoS tagging, span-level F1 for chunking, word-level F1 for predicate detection, and span-level F1 for NER.

Original: We use BERT 6 (Devlin et al., 2019) as our basic model in our main experiments.
Revised: We use BERT 5 (Devlin et al., 2019) as our basic model in our main experiments.

Original: Speciﬁcally, we use the pre-trained case-sensitive BERT-base PyTorch implementation (Wolf et al., 2020), and the common hyperparameters for BERT.
Revised: Speciﬁcally, we use the pre-trained case-sensitive BERT-base PyTorch implementation (Wolf et al., 2020), and the common hyperparameters for BERT.

Original: In the BERT, the task-speciﬁc function is the last-layer linear classiﬁer, and the representation model is the remaining part.
Revised: In the BERT, the task-speciﬁc function is the last-layer linear classiﬁer, and the representation model is the remaining part.

Original: As for cross-task learning paradigms, we consider two popular learning paradigms, pre-training, and joint training.
Revised: As for cross-task learning paradigms, we consider two popular learning paradigms, pre-training, and joint training.

Original: Pre-training ﬁrst pre-train the representation part on the source data and then ﬁne-tune the whole target model on the target data.
Revised: Pre-training ﬁrst pre-train the representation part on the source data and then ﬁne-tune the whole target model on the target data.

Original: Joint training uses both source and target data to train the shared representation model and task-speciﬁc functions for both source and target tasks at the same time.
Revised: Joint training uses both source and target data to train the shared representation model and task-speciﬁc functions for both source and target tasks at the same time.

Original: As for the multi-task learning part in both pre-training and joint training, we adopt the same multi-task learning algorithm as in MT-DNN (Liu et al., 2019).
Revised: As for the multi-task learning part in both pre-training and joint training, we adopt the same multi-task learning algorithm as in MT-DNN (Liu et al., 2019).

Original: More explanation on the choice of experimental settings can be found in Appx.
Revised: More explanation on the choice of experimental settings can be found in Appx.

Original: B.1.
Revised: C.

Original: Settings for weighted training.
Revised: Settings for weighted training.

Original: For scalability, in all the experiments, we approximate g kt in Eq.
Revised: For scalability, in all the experiments, we approximate g kt in Eq.

Original: (2.7) by − c × sim ( ∇ φ (cid:98) L 0 ( φ k +1 , f k +1) , ∇ φ (cid:98) L t ( φ k +1 , f k +1 t )) , where sim ( · , · ) denotes the cosine similarity between two vectors.
Revised: (2.7) by − c × sim ( ∇ φ (cid:98) L 0 ( φ k +1 , f k +1) , ∇ φ (cid:98) L t ( φ k +1 , f k +1 t )) , where sim ( · , · ) denotes the cosine similarity between two vectors.

Original: Note that this type of approximation is common and almost necessary, such as MAML (Finn et al., 2017).
Revised: Note that this type of approximation is common and almost necessary, such as MAML (Finn et al., 2017).

Original: For weighted joint training, we choose c = 1 .
Revised: For weighted joint training, we choose c = 1 .

Original: For weighted pre-training, we choose the best c among [0 . 3 , 1 , 10 , 30 , 100] , because the cosine similarity between the pre-training tasks and the target task is small in general.
Revised: For weighted pre-training, we choose the best c among [0 . 3 , 1 , 10 , 30 , 100] , because the cosine similarity between the pre-training tasks and the target task is small in general.

Original: In practice, we further simplify the computation of ∇ φ (cid:98) L t ( φ k +1 , f k +1 t ) ( t = 0 , 1 , . . .
Revised: In practice, we further simplify the computation of ∇ φ (cid:98) L t ( φ k +1 , f k +1 t ) ( t = 0 , 1 , . . .

Original: , T ) in Eq.
Revised: , T ) in Eq.

Original: (2.7) by computing the gradient over the average loss of a randomly sampled subset of the training set instead of the average loss of the [Table] [Table] [Table]
Revised: (2.7) by computing the gradient over the average loss of a randomly sampled subset of the training set instead of the average loss of the whole training set as in Eq.

Original: the three tasks in Ontonotes 5.0, i.e., PoS tagging, predicate detection, and NER.

Original: For each setting, we choose one task as the target task and the remaining two tasks as source tasks.

Original: We randomly choose 20 K training sentences for each source task respectively.

Original: As for the target task, we randomly choose 500 , 600 , 1000 training sentences for PoS tagging, predicate detection, and NER respectively.

Original: whole training set as in Eq.

Original: In our experiments, we simply set the size of the randomly sampled subset of the training set as 64 , though a larger size is more beneﬁcial in general.
Revised: In our experiments, we simply set the size of the randomly sampled subset of the training set as 64 , though a larger size is more beneﬁcial in general.

Original: In our experiments, we choose η k = 1 . 0 in the mirror descent update (2.8).
Revised: In our experiments, we choose η k = 1 . 0 in the mirror descent update (2.8).

Original: It is worthwhile to note that there is no need to tune any extra hyperparameters for weighted joint training, though we believe that the performance of TAWT can be further improved by tuning extra hyper parameters, such as the learning rate η k .
Revised: It is worthwhile to note that there is no need to tune any extra hyperparameters for weighted joint training, though we believe that the performance of TAWT can be further improved by tuning extra hyper parameters, such as the learning rate η k .

Original: Results.
Revised: Results.

Original: The effectiveness of TAWT is ﬁrst demonstrated for both pre-training and joint training on four tasks with quite a few training examples in the target data, as shown in Table 1. After that, we experiment with more  training examples  in the target data for three tasks in Ontonotes 5.0.
Revised: The effectiveness of TAWT is ﬁrst demonstrated for both pre-training and joint training on four tasks with quite a few training examples in the target data, as shown in Table 1. Experiments with more target training examples and some additional experiments can be found in Table 2 and

Original: Without chunking, we also use more training examples in the source tasks. As shown in Table 2, we can see that TAWT is still beneﬁcial even with more training examples, though the relative improvement of TAWT is smaller compared to that with fewer training examples.
Revised: Table 3 in Appx.

Original:  Finally, we note that TAWT can be easily extended from putting weights on tasks to putting weights on samples (see Fig. 2 in Appx.
Revised: B. Finally, we note that TAWT can be easily extended from putting weights on tasks to putting weights on samples (see Fig. 2 in Appx.

Original: B).
Revised: B).

Original: Some additional experiments can also be found in Appx.

Original: B.

Original: Normalized joint training.
Revised: Normalized joint training.

Original: Inspired by the ﬁnal task weights learned by TAWT (see Table 7 in
Revised: Inspired by the ﬁnal task weights learned by TAWT (see Table 6 in

Original: Appx.
Revised: Appx.

Original: B), we also experiment with normalized joint training.
Revised: B), we also experiment with normalized joint training.

Original: The difference between joint training and normalized joint training lies in the initialization of task weights.
Revised: The difference between joint training and normalized joint training lies in the initialization of task weights.

Original: For (weighted) joint training, the weights on tasks are initialized to be ω t = n t / (cid:80) Tt (cid:48)
Revised: For (weighted) joint training, the weights on tasks are initialized to be ω t = n t / (cid:80) Tt (cid:48)

Original: =
Revised: =

Original: 1 n t (cid:48) (i.e, the weights on the loss of each  example will be uniform), whereas for the (weighted) normalized joint training, the weights on tasks are initialized to be ω t = 1 /T
Revised: 1 n t (cid:48) (i.e, the weights on the loss of each [Figure] example will be uniform), whereas for the (weighted) normalized joint training, the weights on tasks are initialized to be ω t = 1 /T

Original: (i.e., the weight on the loss of each example will be normalized by the sample sizes).
Revised: (i e., the weight on the loss of each example will be normalized by the sample sizes).

Original: Note that the loss for each task is already normalized by the sample size in our theoretical analysis (see Eq. 2.2).
Revised: Note that the loss for each task is already normalized by the sample size in our theoretical analysis (see Eq. 2.2).

Original: As a byproduct, we ﬁnd that normalized joint training is much better than the widely used joint training when the source data is much larger than the target data.
Revised: As a byproduct, we ﬁnd that normalized joint training is much better than the widely used joint training when the source data is much larger than the target data.

Original: In addition, TAWT can still be used to further improve normalized joint training.
Revised: In addition, TAWT can still be used to further improve normalized joint training.

Original: The corresponding results can be found in Table 3.
Revised: The corresponding results can be found in Table 1.

Original: In addition, we ﬁnd that dynamic weights might be a better choice in weighted training compared to ﬁxed weights, and more discussion can be found in Appx.B. [Figure] [Table]
Revised: In addition, we ﬁnd that dynamic weights might be a better choice in weighted training compared to ﬁxed weights (see Table 7 in Appx.

Original: [Figure] on a log scale. The corresponding improvement from the normalized joint training to the weighted normalized joint training is shown.
Revised: B).

Original: Analysis.
Revised: Analysis.

Original: Furthermore, we analyze two crucial factors (i.e., the ratio between the training sizes of the source and target datasets, and the training size of the target dataset) that affect the improvement of TAWT in Fig.
Revised: Furthermore, we analyze two crucial factors (i.e., the ratio between the training sizes of the source and target datasets, and the training size of the target dataset) that affect the improvement of TAWT in Fig.

Original: In general, we ﬁnd that TAWT is more beneﬁcial when the performance of the base model is poorer, either because of a smaller-sized target data or due to a smaller ratio between the source data size and the target data size. 
Revised: In general, we ﬁnd that TAWT is more beneﬁcial when the performance of the base model is poorer, either because of a smaller-sized target data or due to a smaller ratio between the source data size and the target data size. More details are in Table 4 and Table 5 in Appx. B.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we propose a new weighted training algorithm, TAWT , to improve the sample efﬁciency of learning from cross-task signals.
Revised: In this paper, we propose a new weighted training algorithm, TAWT , to improve the sample efﬁciency of learning from cross-task signals.

Original: TAWT adaptively assigns weights on tasks or samples in the source data to minimize the representation-based task distance between source and target tasks.
Revised: TAWT adaptively assigns weights on tasks or samples in the source data to minimize the representation-based task distance between source and target tasks.

Original: The algorithm is an easy-to-use plugin that can be applied to existing cross-task learning paradigms, such as pre-training and joint training, without introducing too much computational overhead and hyperparameters tuning.
Revised: The algorithm is an easy-to-use plugin that can be applied to existing cross-task learning paradigms, such as pre-training and joint training, without introducing too much computational overhead and hyperparameters tuning.

Original: The effectiveness of TAWT is further corroborated through theoretical analyses and empirical experiments.
Revised: The effectiveness of TAWT is further corroborated through theoretical analyses and empirical experiments.

Original: To the best of our knowledge, TAWT is the ﬁrst weighted algorithm for cross-task learning with theoretical guarantees, and the proposed representation-based task distance also sheds light on many critical aspects of cross-task learning.
Revised: To the best of our knowledge, TAWT is the ﬁrst weighted algorithm for cross-task learning with theoretical guarantees, and the proposed representation-based task distance also sheds light on many critical aspects of cross-task learning.

Original: Limitations and Future Work.
Revised: Limitations and Future Work.

Original: There are two main limitations in our work.
Revised: There are two main limitations in our work.

Original: First, although we gave an efﬁcient implementation of task-weighted version of TAWT , an efﬁcient implementation of sample-weighted version of TAWT is still lacking, and we leave it for future work.
Revised: First, although we gave an efﬁcient implementation of task-weighted version of TAWT , an efﬁcient implementation of sample-weighted version of TAWT is still lacking, and we leave it for future work.

Original: Second, we are not aware of any method that can efﬁciently estimate the representation-based task distance without training the model, and we plan to work more on this direction.
Revised: Second, we are not aware of any method that can efﬁciently estimate the representation-based task distance without training the model, and we plan to work more on this direction.

Original: In addition, we also plan to evaluate
Revised: In addition, we also plan to evaluate

Original: TAWT in other settings, such as cross-domain and cross-lingual settings; in more general cases, such as multiple target tasks in the target data; and in other tasks, such as  question answering, textual entailment, image classiﬁcation, and object detection.
Revised: TAWT in other settings, such as cross-domain and cross-lingual settings; in more general cases, such as multiple target tasks in the target data; and in other tasks, such as language modeling, question answering, sentiment analysis, image classiﬁcation, and object detection.

Revised: A CKNOWLEDGMENTS

Revised: This material is based upon work supported by the US Defense Advanced Research Projects Agency (DARPA) under contracts FA8750-19-2-0201 and W911NF-20-1-0080, NSF through CAREER

Revised: DMS-1847415 and an Alfred Sloan Research Fellowship.

Revised: The views expressed are those of the authors and do not reﬂect the ofﬁcial policy or position of the Department of Defense or the U

Original:  A O MITTED P ROOFS
Revised: S. Government. A O MITTED P ROOFS

Original: We start by introducing some notations.
Revised: We start by introducing some notations.

Original: For a set S , we let 1 S be its indicator function and we use # S and | S | interchangeably to denote its cardinality.
Revised: For a set S , we let 1 S be its indicator function and we use # S and | S | interchangeably to denote its cardinality.

Original: For two positive sequences { a n } and { b n } , we write a n (cid:46) b n or a n = O ( b n ) to denote lim sup a n /b n < ∞ , and we let a n (cid:38) b n or a n = Ω( b n ) to denote b n (cid:46) a n .
Revised: For two positive sequences { a n } and { b n } , we write a n (cid:46) b n or a n = O ( b n ) to denote lim sup a n /b n < ∞ , and we let a n (cid:38) b n or a n = Ω( b n ) to denote b n (cid:46) a n .

Original: Meanwhile, the notation a n (cid:16) b n or a n = Θ( b n ) means a n (cid:46) b n and a n (cid:38) b n simultaneously.
Revised: Meanwhile, the notation a n (cid:16) b n or a n = Θ( b n ) means a n (cid:46) b n and a n (cid:38) b n simultaneously.

Original: For a vector x , we let (cid:107) x (cid:107) denote its (cid:96) 2 norm.
Revised: For a vector x , we let (cid:107) x (cid:107) denote its (cid:96) 2 norm.

Original: In this section, we treat L (cid:96) , L F , C Φ , C F as absolute constants and we hide the dependence on those parameters in our theoretical results.
Revised: In this section, we treat L (cid:96) , L F , C Φ , C F as absolute constants and we hide the dependence on those parameters in our theoretical results.

Original: The exact dependence on those parameters can be easily traced from the proofs.
Revised: The exact dependence on those parameters can be easily traced from the proofs.

Original: A. TAWT AND TASK DISTANCE MINIMIZATION
Revised: A. TAWT AND TASK DISTANCE MINIMIZATION

Original: If we estimate L (cid:63) 0 ( φ ) = min f 0 ∈F L 0 ( φ, f 0 ) by min f 0 ∈F (cid:98) L 0 ( φ, f 0 ) and ¯ φ ω ∈ argmin φ ∈ Φ (cid:80) Tt =1 ω t · min f t ∈F L t ( φ, f t ) by the argmin of (cid:80) Tt =1 ω t · min f t ∈F (cid:98) L t ( φ, f t ) , then overall, the quantity L (cid:63) 0 (¯ φ ω ) can be estimated by
Revised: If we estimate L (cid:63) 0 ( φ ) = min f 0 ∈F L 0 ( φ, f 0 ) by min f 0 ∈F (cid:98) L 0 ( φ, f 0 ) and ¯ φ ω ∈ argmin φ ∈ Φ (cid:80) Tt =1 ω t · min f t ∈F L t ( φ, f t ) by the argmin of (cid:80) Tt =1 ω t · min f t ∈F (cid:98) L t ( φ, f t ) , then overall, the quantity L (cid:63) 0 (¯ φ ω ) can be estimated by

Original: [Equation]
Revised: [Equation]

Original: subject to φ ∈ argmin ψ ∈ Φ
Revised: subject to [Equation] argmin [Equation]

Original: T (cid:88) t =
Revised: T (cid:88) [Equation]

Original: [Equation]
Revised: [Equation]

Original: t (cid:98) L t ( ψ
Revised: f

Original: [Equation]
Revised: [Equation]

Original: f t ) . Thus, minimizing the estimated task distance over the weights (note that L (cid:63) 0 ( φ (cid:63) ) is a constant) is equivalent to the optimization formulation (OPT1) of our algorithm.
Revised: Thus, minimizing the estimated task distance over the weights (note that L (cid:63) 0 ( φ (cid:63) ) is a constant) is equivalent to the optimization formulation (OPT1) of our algorithm.

Original: To further relate TAWT with task distance minimization, we provided an analysis of the two-step procedure (2.2)–(2.3) with ﬁxed weights.
Revised: To further relate TAWT with task distance minimization, we provided an analysis of the two-step procedure (2.2)–(2.3) with ﬁxed weights.

Original: The next theorem gives the corresponding performance guarantees.
Revised: The next theorem gives the corresponding performance guarantees.

Original: Theorem A.1 (Performance of the two-step procedure with ﬁxed weights) .
Revised: Theorem A.1 (Performance of the two-step procedure with ﬁxed weights) .

Original: Let ( (cid:98) φ, (cid:98) f 0 ) be obtained via solving (2.2)–(2.3) with ﬁxed ω .
Revised: Let ( (cid:98) φ, (cid:98) f 0 ) be obtained via solving (2.2)–(2.3) with ﬁxed ω .

Original: Let Assumptions A and B hold.
Revised: Let Assumptions A and B hold.

Original: Fix any δ ∈ (0 , 1) and deﬁne
Revised: Fix any δ ∈ (0 , 1) and deﬁne

Original: N ω
Revised: N ω

Original: = (
Revised: = (

Original: [Equation]
Revised: [Equation]

Original: Tt =
Revised: Tt =

Original: [Equation]
Revised: [Equation]

Original: 2 t /n t ) − 1 .
Revised: 2 t /n t ) − 1 .

Original: There exists a constant C = C
Revised: There exists a constant C = C

Original: ( L (cid:96) , L F , C Φ , C F ) > 

Revised: ( L (cid:96) , L F , C Φ , C F ) >

Original: such that the following holds: if for any representation φ in a C (cid:112) ( ν Φ + Tν F + log δ − 1 ) /N
Revised: such that the following holds: if for any representation φ in a C (cid:112) ( ν Φ + Tν F + log δ − 1 ) /N

Original: ω -neighborhood of ¯Φ ω 7 , there exists a speciﬁc ¯ φ ω ∈ ¯Φ ω such that φ is ( ρ, C ρ )
Revised: ω -neighborhood of ¯Φ ω 6 , there exists a speciﬁc ¯ φ ω ∈ ¯Φ ω such that φ is ( ρ, C ρ )

Original: -transferable, then there exists another
Revised: -transferable, then there exists another

Original: C (cid:48)
Revised: C (cid:48)

Original: =
Revised: =

Original: C (cid:48)
Revised: C (cid:48)

Original: ( L (cid:96) , L F , C Φ , C F , C ρ , ρ ) such that with probability at least 1 − δ , we have
Revised: ( L (cid:96) , L F , C Φ , C F , C ρ , ρ ) such that with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix A.3.
Revised: See Appendix A.3.

Original: Note that the task distance naturally appears in the upper bound above.
Revised: Note that the task distance naturally appears in the upper bound above.

Original: This theorem can be regarded as a predecessor of Theorem 3.1, and we refer readers to Section 3.2 for a detailed exposition on the meaning of each term in the upper bound.
Revised: This theorem can be regarded as a predecessor of Theorem 3.1, and we refer readers to Section 3.2 for a detailed exposition on the meaning of each term in the upper bound.

Original: A. A S UFFICIENT C ONDITION FOR E XACTLY M ATCHING ¯ φ ω = φ (cid:63)
Revised: A. A S UFFICIENT C ONDITION FOR E XACTLY M ATCHING ¯ φ ω = φ (cid:63)

Original: Proposition A.1 (Weighting and task distance minimization) .
Revised: Proposition A.1 (Weighting and task distance minimization) .

Original: Suppose that for every φ ∈ Φ , there exists two source tasks 1 ≤ t 1 , t 2 ≤ T such that L (cid:63)t 1 ( φ ) ≤ L (cid:63) 0 ( φ ) ≤ L (cid:63)t 2 ( φ ) .
Revised: Suppose that for every φ ∈ Φ , there exists two source tasks 1 ≤ t 1 , t 2 ≤ T such that L (cid:63)t 1 ( φ ) ≤ L (cid:63) 0 ( φ ) ≤ L (cid:63)t 2 ( φ ) .

Original: Then there exists a choice of weights ω , possibly depending on φ , such that dist ( (cid:80)
Revised: Then there exists a choice of weights ω , possibly depending on φ , such that dist ( (cid:80)

Original: Tt =1 ω t D t , D 0 )
Revised: Tt =1 ω t D t , D 0 )

Original: = 0 .
Revised: = 0 .

Original: Proof.
Revised: Proof.

Original: By construction, it sufﬁces to show the existence of some ω such that
Revised: By construction, it sufﬁces to show the existence of some ω such that

Original: [Equation]
Revised: [Equation]

Original: By assumption, ﬁx any φ ∈ Φ , we can ﬁnd 1 ≤ t 1 , t 2 ≤ T such that L (cid:63)t 1 ( φ ) ≤ L (cid:63) 0 ( φ ) ≤ L (cid:63)t 2 ( φ ) .
Revised: By assumption, ﬁx any φ ∈ Φ , we can ﬁnd 1 ≤ t 1 , t 2 ≤ T such that L (cid:63)t 1 ( φ ) ≤ L (cid:63) 0 ( φ ) ≤ L (cid:63)t 2 ( φ ) .

Original: We set ω t = 0 for any t / ∈ { t 1 , t 2 } .
Revised: We set ω t = 0 for any t / ∈ { t 1 , t 2 } .

Original: If L (cid:63)t 1 ( φ ) =
Revised: If L (cid:63)t 1 ( φ ) =

Original: L (cid:63) 

Revised: L (cid:63)

Original: ( φ ) =
Revised: ( φ ) =

Original: L (cid:63)t 2 ( φ ) , then any choice of ω t 1 , ω t 2 will sufﬁce.
Revised: L (cid:63)t 2 ( φ ) , then any choice of ω t 1 , ω t 2 will sufﬁce.

Original: Otherwise, we let
Revised: Otherwise, we let

Original: [Equation]
Revised: [Equation]

Original: It is straightforward to check that such a choice of ω indeed ensures (A.2).
Revised: It is straightforward to check that such a choice of ω indeed ensures (A.2).

Original: The proof is concluded.
Revised: The proof is concluded.

Original: A. P ROOF OF T HEOREM A. 
Revised: A. P ROOF OF T HEOREM A.

Original: We start by stating two useful lemmas.
Revised: We start by stating two useful lemmas.

Original: Lemma A.1 (Error for learning the imperfect representation from source data) .
Revised: Lemma A.1 (Error for learning the imperfect representation from source data) .

Original: Under the setup of
Revised: Under the setup of

Original: Theorem A.1, there exists a constant C 1 = C 1 ( L (cid:96) , L F , C Φ , C F ) > 0 such that for any δ ∈ (0 , 1) , we have
Revised: Theorem A.1, there exists a constant C 1 = C 1 ( L (cid:96) , L F , C Φ , C F ) > 0 such that for any δ ∈ (0 , 1) , we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Proof.
Revised: Proof.

Original: See Appendix A.3.1.
Revised: See Appendix A.3.1.

Original: Lemma A.2 (Error for learning the task-speciﬁc function from target data) .
Revised: Lemma A.2 (Error for learning the task-speciﬁc function from target data) .

Original: Under the setup of
Revised: Under the setup of

Original: Theorem A.1, there exists
Revised: Theorem A.1, there exists

Original: a constant C 2 =
Revised: a constant C 2 =

Original: C 2 ( L (cid:96) , C F ) > 0 such that for any δ ∈ (0 , 1) , we have
Revised: C 2 ( L (cid:96) , C F ) > 0 such that for any δ ∈ (0 , 1) , we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Proof.
Revised: Proof.

Original: See Appendix A.3.2.
Revised: See Appendix A.3.2.

Original: To prove Theorem A.1, we start by writing
Revised: To prove Theorem A.1, we start by writing

Original: [Equation]
Revised: [Equation]

Original: Suppose the two high probability events in Lemmas A.1 and A.2 hold.
Revised: Suppose the two high probability events in Lemmas A.1 and A.2 hold.

Original: Then we can bound L 0 ( (cid:98) φ, (cid:98) f 0 ) −
Revised: Then we can bound L 0 ( (cid:98) φ, (cid:98) f 0 ) −

Original: L (cid:63) 

Revised: L (cid:63)

Original: ( (cid:98) φ ) by (A.4).
Revised: ( (cid:98) φ ) by (A.4).

Original: Meanwhile, since (cid:98) φ is in a C (cid:112) ( ν Φ + Tν F + log δ − 1 ) /N
Revised: Meanwhile, since (cid:98) φ is in a C (cid:112) ( ν Φ + Tν F + log δ − 1 ) /N

Original: ω -neighborhood of ¯Φ ω , we can invoke the transferability assumption to get
Revised: ω -neighborhood of ¯Φ ω , we can invoke the transferability assumption to get

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have the trivial bound that L (cid:63) 

Revised: Moreover, we have the trivial bound that L (cid:63)

Original: (¯ φ ω ) −L (cid:63) 

Revised: (¯ φ ω ) −L (cid:63)

Original: ( φ (cid:63) 0 ) ≤ sup ¯ φ ω L (cid:63) 0 (¯ φ ω ) −L (cid:63) 

Revised: ( φ (cid:63) 0 ) ≤ sup ¯ φ ω L (cid:63) 0 (¯ φ ω ) −L (cid:63)

Original: ( φ (cid:63) 0 ) .
Revised: ( φ (cid:63) 0 ) .

Original: Assembling the three bounds above gives the desired result.
Revised: Assembling the three bounds above gives the desired result.

Original: A.3. P ROOF OF L EMMA A. 
Revised: A.3. P ROOF OF L EMMA A.

Original: We start by writing
Revised: We start by writing

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the ﬁrst inequality is by L (cid:63)t ( · )
Revised: where the ﬁrst inequality is by L (cid:63)t ( · )

Original: =
Revised: =

Original: min f t ∈F L t ( · , f t ) and the second inequality is by the fact that ( (cid:98) φ, { (cid:98) f t } Tt =1 ) is a minimizer of (2.2).
Revised: min f t ∈F L t ( · , f t ) and the second inequality is by the fact that ( (cid:98) φ, { (cid:98) f t } Tt =1 ) is a minimizer of (2.2).

Original: To simplify notations, let z ti =
Revised: To simplify notations, let z ti =

Original: ( x ti , y ti ) and let the right-hand side above be G ( { z ti } ) .
Revised: ( x ti , y ti ) and let the right-hand side above be G ( { z ti } ) .

Original: Fix two indices 1 ≤ t (cid:48) ≤ T , 1 ≤ i t (cid:48) ≤ n t , and let { (cid:101)
Revised: Fix two indices 1 ≤ t (cid:48) ≤ T , 1 ≤ i t (cid:48) ≤ n t , and let { (cid:101)

Original: z ti } be the source datasets formed by replacing z t (cid:48) ,i t (cid:48) with some (cid:101) z t
Revised: z ti } be the source datasets formed by replacing z t (cid:48) ,i t [Equation] with some (cid:101) z t

Original: [Equation]
Revised: [Equation]

Original: ,i t
Revised: ,i t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: X × Y .
Revised: X × Y .

Original: Since { z ti } and { (cid:101) z ti } differ by only one example
Revised: Since { z ti } and { (cid:101) z ti } differ by only one example

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by the fact that the loss function is bounded in [
Revised: where the last inequality is by the fact that the loss function is bounded in [

Original: [Equation]
Revised: [Equation]

Original: Taking the supremum over φ ∈ Φ , { f t } ⊂ F at both sides, we get G ( { z ti } ) − G 
Revised: Taking the supremum over φ ∈ Φ , { f t } ⊂ F at both sides, we get G ( { z ti } ) − G (

Original: [Equation]
Revised: [Equation]

Original: (cid:48) [Equation] n t (cid:48) . A symmetric argument shows that the reverse inequality, namely G ( { (cid:101) z ti } ) − G ( { z ti } ) ≤ 4 ω t (cid:48) /n
Revised: A symmetric argument shows that the reverse inequality, namely G ( { (cid:101) z ti } ) − G ( { z ti } ) ≤ 4 ω t (cid:48) /n

Original: t (cid:48) , is also true.
Revised: t (cid:48) , is also true.

Original: That is, we have shown
Revised: That is, we have shown

Original: [Equation]
Revised: [Equation]

Original: This means that we can invoke McDiarmid’s inequality to get
Revised: This means that we can invoke McDiarmid’s inequality to get

Original: [Equation]
Revised: [Equation]

Original: for any ε > 0 , or equivalently
Revised: for any ε > 0 , or equivalently

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ for any δ ∈ (0 , 1) .
Revised: with probability at least 1 − δ for any δ ∈ (0 , 1) .

Original: To bound the expectation term, we use a standard symmetrization argument (see, e.g., Lemma 11.4 in (Boucheron et al., 2013)) to get(cid:112)
Revised: To bound the expectation term, we use a standard symmetrization argument (see, e.g., Lemma 11.4 in (Boucheron et al., 2013)) to get

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the randomness in both the source datasets { z ti } and the i.i.d.
Revised: where the expectation is taken over the randomness in both the source datasets { z ti } and the i.i.d.

Original: symmetric Rademacher random variables { ε ti } . Consider the function space G :
Revised: symmetric Rademacher random variables { ε ti } . Consider the function space G :

Original: = { ( φ, { f t } Tt =1 ) : φ ∈ Φ , { f t } ⊂ F} .
Revised: = { ( φ, { f t } Tt =1 ) : φ ∈ Φ , { f t } ⊂ F} .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: be the empirical process indexed by the function space G .
Revised: be the empirical process indexed by the function space G .

Original: Conditional on the randomness in the data { z ti } , this is a Rademacher process with sub-Gaussian increments:
Revised: Conditional on the randomness in the data { z ti } , this is a Rademacher process with sub-Gaussian increments:

Original: [Equation]
Revised: [Equation]

Original: where the pseudometric
Revised: where the pseudometric

Original: [Equation]
Revised: [Equation]

Original: Thus, we can invoke Dudley’s entropy integral inequality (see, e.g., Corollary 13.2 in (Boucheron et al., 2013)) to get
Revised: Thus, we can invoke Dudley’s entropy integral inequality (see, e.g., Corollary 13.2 in (Boucheron et al., 2013)) to get

Original: [Equation]
Revised: [Equation]

Original: where ¯ g =
Revised: where ¯ g =

Original: (¯ φ ω , { ¯ f ω t } ) , and N ( G ; d ; ε ) is the ε -covering number of G with respect to the pseudometric d .
Revised: (¯ φ ω , { ¯ f ω t } ) , and N ( G ; d ; ε ) is the ε -covering number of G with respect to the pseudometric d .

Original: Taking expectation over the randomness in { z ti } , we get
Revised: Taking expectation over the randomness in { z ti } , we get

Original: [Equation]
Revised: [Equation]

Original: We now bound the covering number of G .
Revised: We now bound the covering number of G .

Original: To do so, we deﬁne
Revised: To do so, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where δ x ti is a point mass at x ti .
Revised: where δ x ti is a point mass at x ti .

Original: Let { φ (1) , . . .
Revised: Let { φ (1) , . . .

Original: , φ ( N ε ) } ⊂ Φ be an ε -covering of Φ with respect to
Revised: , φ ( N ε ) } ⊂ Φ be an ε -covering of Φ with respect to

Original: L 2 ( Q ) , where N ε = N (Φ; L 2 ( Q ); ε ) .
Revised: L 2 ( Q ) , where N ε = N (Φ; L 2 ( Q ); ε ) .

Original: This means that for any φ ∈ Φ , there exists j ∈ { 1 , . . .
Revised: This means that for any φ ∈ Φ , there exists j ∈ { 1 , . . .

Original: , N ε } such that
Revised: , N ε } such that

Original: [Equation]
Revised: [Equation]

Original: Now for each j ∈ { 1 , . . .
Revised: Now for each j ∈ { 1 , . . .

Original: , N ε } and t ∈ { 1 , . . .
Revised: , N ε } and t ∈ { 1 , . . .

Original: , T } , let { f ( j, 1) t , . . .
Revised: , T } , let { f ( j, 1) t , . . .

Original: , f ( j,N ( j ) ε ) t } be an ε -covering of
Revised: , f ( j,N ( j ) ε ) t } be an ε -covering of

Original: F with respect to L 2 ( φ ( j ) # Q t ) , where φ ( j ) # Q t is the pushforward of Q t by φ ( j ) , and N ( j ) ε =
Revised: F with respect to L 2 ( φ ( j ) # Q t ) , where φ ( j ) # Q t is the pushforward of Q t by φ ( j ) , and N ( j ) ε =

Original: N ( F ; L 2 ( φ ( j ) # Q t ); ε ) has no dependence on t due to the uniform entropy control from Assumption
Revised: N ( F ; L 2 ( φ ( j ) # Q t ); ε ) has no dependence on t due to the uniform entropy control from Assumption

Original: B. This means that for any f t ∈ F , j ∈ { 1 , . . . , N ε } , there exists k ∈ { 1 , . . .
Revised: B. This means that for any f t ∈ F , j ∈ { 1 , . . . , N ε } , there exists k ∈ { 1 , . . .

Original: , N ( j ) ε } such that
Revised: , N ( j ) ε } such that

Original: [Equation]
Revised: [Equation]

Original: j,k ) t
Revised: ,k

Original: [Equation]
Revised: [Equation]

Original: φ
Revised: t

Original: [Equation]
Revised: [Equation]

Original: ≤ ε
Revised: ≤ ε

Original: [Equation]
Revised: [Equation]

Original: Now, let us ﬁx ( φ, { f t } Tt =1 ) ∈ G .
Revised: Now, let us ﬁx ( φ, { f t } Tt =1 ) ∈ G .

Original: By construction, we can ﬁnd j ∈ { 1 , . . .
Revised: By construction, we can ﬁnd j ∈ { 1 , . . .

Original: , N ε } and k t ∈ { 1 , . . .
Revised: , N ε } and k t ∈ { 1 , . . .

Original: , N ( j ) ε } for any 1 ≤ t ≤ T such that (cid:107)
Revised: , N ( j ) ε } for any 1 ≤ t ≤ T such that (cid:107)

Original: [Equation]
Revised: [Equation]

Original: L
Revised: −

Original: [Equation]
Revised: [Equation]

Original:  Q ) ≤ ε,
Revised: L 2 ( Q ) ≤ ε,

Original: [Equation]
Revised: [Equation]

Original: φ
Revised: − f

Original: [Equation]
Revised: [Equation]

Original: # Q
Revised: ,k

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) ≤ ε
Revised: φ

Original: [Equation]
Revised: [Equation]

Original: ∀
Revised: # Q

Original: [Equation]
Revised: [Equation]

Original:  ≤ t ≤
Revised: ) ≤ ε

Original: [Equation]
Revised: [Equation]

Revised: ∀ 1 ≤ t ≤ T.

Original: (A.8)
Revised: (A.8)

Original: Thus, we have d 2 (cid:18) ( φ, { f t } Tt =1 ) , ( φ ( j ) , { f ( j,k t ) t } Tt =1 ) (cid:19) =
Revised: Thus, we have d 2 (cid:18) ( φ, { f t } Tt =1 ) , ( φ ( j ) , { f ( j,k t ) t } Tt =1 ) (cid:19) =

Original: N ω ·
Revised: N ω ·

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = n t (cid:88) i =
Revised: = n t (cid:88) i =

Original: ω 2 t n 2 t (cid:18) (cid:96) ( f t ◦ φ ( x ti ) , y ti ) − (cid:96) ( f ( j,k t ) t ◦ φ ( j ) ( x ti ) , y ti ) (cid:19)
Revised: ω 2 t n 2 t (cid:18) (cid:96) ( f t ◦ φ ( x ti ) , y ti ) − (cid:96) ( f ( j,k t ) t ◦ φ ( j ) ( x ti ) , y ti ) (cid:19)

Original: ≤ L 2 (cid:96) N ω ·
Revised: ≤ L 2 (cid:96) N ω ·

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = n t (cid:88) i =
Revised: = n t (cid:88) i =

Original: ω 2 t n 2 t (cid:18)
Revised: ω 2 t n 2 t (cid:18)

Original: f t ◦ φ ( x ti ) − f ( j,k t ) t ◦ φ ( j ) ( x ti ) (cid:19)= L 2 (cid:96) N ω ·
Revised: f t ◦ φ ( x ti ) − f ( j,k t ) t ◦ φ ( j ) ( x ti ) (cid:19)= L 2 (cid:96) N ω ·

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = n t (cid:88) i =
Revised: = n t (cid:88) i =

Original: ω 2 t n 2 t (cid:18)
Revised: ω 2 t n 2 t (cid:18)

Original: f t ◦ φ ( x ti ) − f t ◦ φ ( j ) ( x ti ) + f t ◦ φ ( j ) ( x ti ) − f ( j,k t ) t ◦ φ ( j ) ( x ti ) (cid:19)
Revised: f t ◦ φ ( x ti ) − f t ◦ φ ( j ) ( x ti ) + f t ◦ φ ( j ) ( x ti ) − f ( j,k t ) t ◦ φ ( j ) ( x ti ) (cid:19)

Original: [Equation]
Revised: [Equation]

Original: This yields
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: from which we get
Revised: from which we get

Original: [Equation]
Revised: [Equation]

Original: Plugging the above inequality to (A.7), we get
Revised: Plugging the above inequality to (A.7), we get

Original: [Equation]
Revised: [Equation]

Original: The proof is concluded by plugging the above inequality to (A.6).
Revised: The proof is concluded by plugging the above inequality to (A.6).

Original: A.3. P ROOF OF L EMMA A. 
Revised: A.3. P ROOF OF L EMMA A.

Original: Since (cid:98) φ is obtained from the source datasets { S t } Tt =1 , it is independent of the target data S 0 .
Revised: Since (cid:98) φ is obtained from the source datasets { S t } Tt =1 , it is independent of the target data S 0 .

Original: Throughout the proof, we condition on the randomness in the source datasets, thus effectively treating (cid:98) φ as ﬁxed.
Revised: Throughout the proof, we condition on the randomness in the source datasets, thus effectively treating (cid:98) φ as ﬁxed.

Original: Let f 0 , (cid:98) φ ∈ argmin f 0 ∈F L 0 ( (cid:98) φ, f ) .
Revised: Let f 0 , (cid:98) φ ∈ argmin f 0 ∈F L 0 ( (cid:98) φ, f ) .

Original: We start by writing
Revised: We start by writing

Original: L 0 ( (cid:98) φ, (cid:98) f 0 ) − L (cid:63) 

Revised: L 0 ( (cid:98) φ, (cid:98) f 0 ) − L (cid:63)

Original: ( (cid:98) φ )=
Revised: ( (cid:98) φ )

Original: [Equation]
Revised: =

Original:  0 ( (cid:98) φ [Equation] (cid:98) f 0 ) − (cid:98) L 0 ( (cid:98) φ, (cid:98) f 0 ) + (cid:98) 
Revised: L 0 ( (cid:98) φ, (cid:98) f 0 ) − (cid:98) L 0 ( (cid:98) φ, (cid:98) f 0 ) + (cid:98) L 0 ( (cid:98) φ, (cid:98) f 0 ) − (cid:98)

Original: [Equation] 0 ( (cid:98) φ, (cid:98) f 0 ) − (cid:98) L 0 ( (cid:98) φ, f 0 , (cid:98) φ ) + (cid:98) L 0 ( (cid:98) φ, f 0 , (cid:98) φ ) − L 0 ( (cid:98) φ, f [Equation] (cid:98) φ )≤
Revised: L 0 ( (cid:98) φ, f 0 , (cid:98) φ ) + (cid:98) L 0 ( (cid:98) φ, f 0 , (cid:98) φ ) − L 0 ( (cid:98) φ, f 0 , (cid:98) φ )

Original: [Equation]
Revised: [Equation]

Original: The right-hand side above is an empirical process indexed by f 0 ∈ F .
Revised: The right-hand side above is an empirical process indexed by f 0 ∈ F .

Original: Using similar arguments as those appeared in the proof of Lemma A.1, we have
Revised: Using similar arguments as those appeared in the proof of Lemma A.1, we have

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ , which is exactly the desired result.
Revised: with probability at least 1 − δ , which is exactly the desired result.

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: The proof bears similarities to the proof of Theorem A.1, with additional complications in ensuring a uniform control over the learned weights.
Revised: The proof bears similarities to the proof of Theorem A.1, with additional complications in ensuring a uniform control over the learned weights.

Original: Let f 0 , (cid:98) φ ∈ argmin f 0 ∈F L 0 ( (cid:98) φ, f 0 ) .
Revised: Let f 0 , (cid:98) φ ∈ argmin f 0 ∈F L 0 ( (cid:98) φ, f 0 ) .

Original: Note that
Revised: Note that

Original: L 0 ( (cid:98) φ, (cid:98) f 0 ) − L 0 ( φ (cid:63) 0 , f (cid:63) 0 ) =
Revised: L 0 ( (cid:98) φ, (cid:98) f 0 ) − L 0 ( φ (cid:63) 0 , f (cid:63) 0 ) =

Original: L 0 ( (cid:98) φ, (cid:98) f 0 ) − (cid:98) L (2)0 ( (cid:98) φ, (cid:98) f 0 ) + (cid:98) L (2)0 ( (cid:98) φ, (cid:98) f
Revised: L 0 ( (cid:98) φ, (cid:98) f 0 ) − (cid:98) L (2)0 ( (cid:98) φ, (cid:98) f 0 ) + (cid:98) L (2)0 ( (cid:98) φ, (cid:98) f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let (cid:98) f t ∈ argmin
Revised: Let (cid:98) f t ∈ argmin

Original: [Equation]
Revised: [Equation]

Original: t ∈F (cid:98) L t ( (cid:98) φ, f t ) .
Revised: (cid:98) L t ( (cid:98) φ, f t ) .

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: Let z ti =
Revised: Let z ti =

Original: ( x ti , y ti ) , and let { (cid:101)
Revised: ( x ti , y ti ) , and let { (cid:101)

Original: z ti } be the source datasets formed by replacing z t (cid:48) ,i t (cid:48) with (cid:101)
Revised: z ti } be the source datasets formed by replacing z t (cid:48) ,i t (cid:48) with (cid:101)

Original: z t (cid:48) ,i t (cid:48) .
Revised: z t (cid:48) ,i t (cid:48) .

Original: Let
Revised: Let

Original: G ( { z ti } ) := sup
Revised: G ( { z ti } ) := sup

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then conducting a similar calculation to what led to (A.5), we get
Revised: Then conducting a similar calculation to what led to (A.5), we get

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by the fact that ω ∈ W β implies ω t ≤ β/T for any 1 ≤ t ≤ T .
Revised: where the last inequality is by the fact that ω ∈ W β implies ω t ≤ β/T for any 1 ≤ t ≤ T .

Original: Now, invoking McDiarmid’s inequality, we get
Revised: Now, invoking McDiarmid’s inequality, we get

Original: [Equation]
Revised: [Equation]

Original: nT
Revised: nT

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A.10) with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Now, a standard symmetrization argument plus an application of
Revised: Now, a standard symmetrization argument plus an application of

Original: Dudley’s entropy integral bound (similar to what led to (A.7)) gives
Revised: Dudley’s entropy integral bound (similar to what led to (A.7)) gives

Original: [Equation]
Revised: [Equation]

Original: where now G := { ( φ, { f t } Tt =1 , ω ) : φ ∈ Φ , { f t } ⊂ F , ω ∈ W β } , and d 
Revised: where now G := { ( φ, { f t } Tt =1 , ω ) : φ ∈ Φ , { f t } ⊂ F , ω ∈ W β } , and d [Equation] [Equation] { f

Original: [Equation]
Revised: [Equation]

Original:  nT 
Revised: } Tt [Equation] [Equation] ) (cid:19) (cid:16) nT β

Revised: T

Original: [Equation]
Revised: [Equation]

Revised: t = n t (cid:88)

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: ) (cid:19)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: nT β
Revised: nT β

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = n t (cid:88) i [Equation]
Revised: = n t (cid:88) i =

Original: [Equation] t n [Equation] (cid:18) 
Revised: ω 2 t n 2 t (cid:18) (cid:96) ( f

Original: [Equation]
Revised: [Equation]

Original:  ti) (cid:19)
Revised: ◦ φ ( x ti

Original: [Equation]
Revised: [Equation]

Original: (cid:88) t [Equation]
Revised: y ti ) − (cid:96) ( (cid:101)

Original: n
Revised: f t ◦ (cid:101) φ ( x ti ) (cid:19)+ nT β

Original: [Equation]
Revised: [Equation]

Original:  (cid:88) i =  n
Revised: T (cid:88) t

Revised: = n t (cid:88) i =  n 2 t ( ω t −

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: nT

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by n t (cid:16) n, ω t ≤ β/T for any 1 ≤ t ≤ T and β ≥ 1 .
Revised: where the last inequality is by n t (cid:16) n, ω t ≤ β/T for any 1 ≤ t ≤ T and β ≥ 1 .

Original: This means that we can construct a Cε -covering of G by the following two steps (where C is an absolute constant only depending on L (cid:96) and L F ): (1) cover the space { ( φ, { f t } T t =1 ) } by the same construction as (A.8);
Revised: This means that we can construct a Cε -covering of G by the following two steps (where C is an absolute constant only depending on L (cid:96) and L F ): (1) cover the space { ( φ, { f t } T t =1 ) } by the same construction as (A.8);

Original: (2) construct an ε/T -covering of W β with at most ( cT/ε ) T many points, where c is an absolute constant.
Revised: (2) construct an ε/T -covering of W β with at most ( cT/ε ) T many points, where c is an absolute constant.

Original: Overall, we can construct a Cε covering G with ( C Φ /ε )
Revised: Overall, we can construct a Cε covering G with ( C Φ /ε )

Original: ν
Revised: ν

Original: Φ · ( C F /ε )
Revised: Φ · ( C F /ε )

Original: Tν F · ( cT/ε ) T many points.
Revised: Tν F · ( cT/ε ) T many points.

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by ν F ≥ 1 .
Revised: where the last inequality is by ν F ≥ 1 .

Original: Plugging this inequality back to (A.10) and (A.11), we get
Revised: Plugging this inequality back to (A.10) and (A.11), we get

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: This means that under this high probability event, we can invoke the transferability assumption to conclude the existence of a speciﬁc ¯ φ (cid:98) ω with
Revised: This means that under this high probability event, we can invoke the transferability assumption to conclude the existence of a speciﬁc ¯ φ (cid:98) ω with

Original: [Equation]
Revised: [Equation]

Original: Recalling (A.9), we arrive at
Revised: Recalling (A.9) [Equation] we arrive at

Original: [Equation]
Revised: [Equation]

Original: with probability 1 − δ .
Revised: with probability 1 − δ .

Original: Since (cid:98) φ is independent of the second batch of target data { ( x 0 i , y 0 i ) : i ∈ B 2 } and | B 2 | (cid:16) n [Equation] a nearly identical argument as that appeared in the proof of Lemma A
Revised: Since (cid:98) φ is independent of the second batch of target data { ( x 0 i , y 0 i ) : i ∈ B 2 } and | B 2 | (cid:16) n 0 , a nearly identical argument as that appeared in the proof of Lemma A.2 gives

Original: [Equation]
Revised: [Equation]

Original: 2 gives

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: nwith probability at least 1 − δ .
Revised: nwith probability at least 1 − δ .

Original: We conclude the proof by invoking a union bound.
Revised: We conclude the proof by invoking a union bound.

Original: B E XPERIMENTAL D ETAILS AND A DDITIONAL R ESULTS 
Revised: B E XPERIMENTAL D ETAILS AND A DDITIONAL R ESULTS Learning Paradigm

Original: [Table]
Revised: [Table]

Original: In this section, we brieﬂy highlight some important settings in our experiments.
Revised: In this section, we brieﬂy highlight some important settings in our experiments.

Original: More details can be found in our submitted code.
Revised: More details can be found in our released code.

Original: It usually costs about half an hour to run the experiment for each setting (e.g. one number in Table 1) on one GeForce RTX 2080 GPU. 
Revised: It usually costs about half an hour to run the experiment for each setting (e.g. one number in Table 1) on one GeForce RTX 2080 GPU. [Table] [Table]

Revised: that used in the experiments in Table 1, so that the performance of single-task learning for the same setting can be a little different.

Original: Data.
Revised: Data.

Original: In our experiments, we mainly use two widely-used NLP datasets, Ontontes 5.0 (Hovy et al., 2006) and CoNLL-2000 (Tjong Kim Sang & Buchholz, 2000).
Revised: In our experiments, we mainly use two widely-used NLP datasets, Ontontes 5.0 (Hovy et al., 2006) and CoNLL-2000 (Tjong Kim Sang & Buchholz, 2000).

Original: Ontonotes 5.0 is a large multilingual richly annotated corpus covering a lot of NLP tasks, and we use the corresponding English annotations [Table] [Table] [Table]
Revised: Ontonotes 5.0 is a large multilingual richly annotated corpus covering a lot of NLP tasks, and we use the corresponding English annotations for three sequence tagging tasks, PoS tagging, predicate detection, and NER.

Original: for three sequence tagging tasks, PoS tagging, predicate detection, and NER.

Original: CoNLL-2000 is a shared task for another sequence tagging task, chunking.
Revised: CoNLL-2000 is a shared task for another sequence tagging task, chunking.

Original: There are about 116 K sentences, 16 K sentences, and 12 K sentences in the training, development, and test sets for tasks in Ontonotes 5.0.
Revised: There are about 116 K sentences, 16 K sentences, and 12 K sentences in the training, development, and test sets for tasks in Ontonotes 5.0.

Original: The average sentence length for sentences in Ontonotes 5.0 is about 19 .
Revised: The average sentence length for sentences in Ontonotes 5.0 is about 19 .

Original: As for CoNLL-2000, there are about 9 K sentences and 2 K sentences in the training and test sets.
Revised: As for CoNLL-2000, there are about 9 K sentences and 2 K sentences in the training and test sets.

Original: The corresponding average sentence length is about 24 .
Revised: The corresponding average sentence length is about 24 .

Original: Tasks.
Revised: Tasks.

Original: We consider four common sequence tagging tasks in NLP, PoS tagging, chunking, predicate detection, and NER.
Revised: We consider four common sequence tagging tasks in NLP, PoS tagging, chunking, predicate detection, and NER.

Original: PoS tagging aims to assign a particular part of speech, such as nouns and verbs, for each word in the sentence.
Revised: PoS tagging aims to assign a particular part of speech, such as nouns and verbs, for each word in the sentence.

Original: Chunking divides a sentence into syntactically related nonoverlapping groups of words and assigns them with speciﬁc types, such as noun phrases and verb phrases.
Revised: Chunking divides a sentence into syntactically related nonoverlapping groups of words and assigns them with speciﬁc types, such as noun phrases and verb phrases.

Original: Predicate detection aims to ﬁnd the corresponding verbal or nominal predicates for each sentence.
Revised: Predicate detection aims to ﬁnd the corresponding verbal or nominal predicates for each sentence.

Original: NER seeks to locate and classify named entities mentioned in each sentence into pre-deﬁned categories such as person names, organizations, and locations.
Revised: NER seeks to locate and classify named entities mentioned in each sentence into pre-deﬁned categories such as person names, organizations, and locations.

Original: Based on the above datasets, there are 50 labels in PoS tagging, 23 labels in chunking, 2 labels in predicate detection, and 37 labels in
Revised: Based on the above datasets, there are 50 labels in PoS tagging, 23 labels in chunking, 2 labels in predicate detection, and 37 labels in

Original: NER.
Revised: NER.

Original: As for the evaluation metric, we use accuracy for PoS tagging, span-level F1 for chunking, token-level F1 for predicate detection, and span-level F1 for NER.[Table] [Table]
Revised: As for the evaluation metric, we use accuracy for PoS tagging, span-level F1 for chunking, token-level F1 for predicate detection, and span-level F1 for NER.

Original: The model.
Revised: The model.

Original: We use BERT as our basic model in our main experiments.
Revised: We use BERT as our basic model in our main experiments.

Original: Speciﬁcally, we use the pretrained case-sensitive BERT-base PyTorch implementation (Wolf et al., 2020).
Revised: Speciﬁcally, we use the pretrained case-sensitive BERT-base PyTorch implementation (Wolf et al., 2020).

Original: We use the common parameter settings for BERT.
Revised: We use the common parameter settings for BERT.

Original: Speciﬁcally, the max length is 128 , the batch size is 32 , the epoch number is 4 , and the learning rate is 5 e − 5 .
Revised: Speciﬁcally, the max length is 128 , the batch size is 32 , the epoch number is 4 , and the learning rate is 5 e − 5 .

Original: In the BERT model, the task-speciﬁc function is the last-layer linear classiﬁer, and the representation model is the remaining part. 
Revised: In the BERT model, the task-speciﬁc function is the last-layer linear classiﬁer, and the representation model is the remaining part. [Table] [Table]

Original: Cross-task learning paradigms.
Revised: Cross-task learning paradigms.

Original: In our experiments, we consider two cross-task learning paradigms, pre-training, and joint training.
Revised: In our experiments, we consider two cross-task learning paradigms, pre-training, and joint training.

Original: Pre-training ﬁrst pre-train the representation part on the source data and then ﬁne-tune the whole target model on the target data.
Revised: Pre-training ﬁrst pre-trains the representation part on the source data and then ﬁne-tunes the whole target model on the target data.

Original: Joint training uses both source and target data to train the shared representation model and task-speciﬁc functions for both source and target tasks at the same time.
Revised: Joint training uses both source and target data to train the shared representation model and task-speciﬁc functions for both source and target tasks at the same time.

Original: As for the multi-task learning part in both pre-training and joint training, we adopt the same multi-task learning algorithm as in MT-DNN (Liu et al., 2019).
Revised: As for the multi-task learning part in both pre-training and joint training, we adopt the same multi-task learning algorithm as in MT-DNN (Liu et al., 2019).

Revised: Experiments with more target training examples.

Revised: Compared to the main experiments in Sec. 4, we further experiment with more training examples in the target data for three tasks in Ontonotes 5.0.

Revised: Without chunking, we also use more training examples in the source tasks.

Revised: As shown in Table 2, we can see that TAWT is still beneﬁcial even with more training examples, though the relative improvement of TAWT is smaller compared to that with fewer training examples.

Original: Additional experiments to illustrate the beneﬁts of TAWT .
Revised: Additional experiments to illustrate the beneﬁts of TAWT .

Original: In this part, we conduct additional experiments with TAWT on the joint training with fewer source tasks compared to the main experiments in Sec. 4.
Revised: In this part, we conduct additional experiments with TAWT on the joint training with fewer source tasks compared to the main experiments in Sec. 4.

Original: For simplicity, we make the target task bold to distinguish it from source tasks for each setting.
Revised: For simplicity, we make the target task bold to distinguish it from source tasks for each setting.

Original: For example, in the setting of PoS + Chunking + NER , NER is the target task and the other two tasks are source tasks.
Revised: For example, in the setting of PoS + Chunking + NER , NER is the target task and the other two tasks are source tasks.

Original: For PoS + NER , we randomly sample 20 K training sentences for PoS tagging and 2 K training sentences for NER.
Revised: For PoS + NER , we randomly sample 20 K training sentences for PoS tagging and 2 K training sentences for NER.

Original: For PoS + Chunking + NER , we randomly sample 20 K , 9 K 9 and 2 K training sentences for PoS tagging, chunking, and NER respectively.
Revised: For PoS + Chunking + NER , we randomly sample 20 K , 9 K 8 and 2 K training sentences for PoS tagging, chunking, and NER respectively.

Original: As for PoS + Chunking + Predicate Detection , we randomly sample 20 K , 9 K , and 2 K training sentences for
Revised: As for PoS + Chunking + Predicate Detection , we randomly sample 20 K , 9 K , and 2 K training sentences for

Original: PoS tagging, chunking, and predicate detection respectively.
Revised: PoS tagging, chunking, and predicate detection respectively.

Original: As shown in Table 4, we can see that
Revised: As shown in Table 3, we can see that

Original: TAWT is still beneﬁcial under these diverse settings, which is a complement for our main experiments in Sec. 4. 
Revised: TAWT is still beneﬁcial under these diverse settings, which is a complement for our main experiments in Sec. 4. [Figure] [Table] [Table]

Revised: [Figure]

Original:  Analysis of some crucial factors.
Revised: [Table] [Table] [Figure] Analysis of some crucial factors.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, we analyze two crucial factors that affect the improvement of TAWT .
Revised: 1, we analyze two crucial factors that affect the improvement of TAWT .

Original: First, in general, we ﬁnd that the improvement of TAWT in normalized joint training is larger when the ratio between the training sizes of the source and target dataset is smaller, though the largest improvement may not be achieved at the smallest ratio.
Revised: First, in general, we ﬁnd that the improvement of TAWT in normalized joint training is larger when the ratio between the training sizes of the source and target dataset is smaller, though the largest improvement may not be achieved at the smallest ratio.

Original: Note that the improvement vanishes with a large ratio mainly because the baseline (normalized joint training) is already good enough with a large ratio.
Revised: Note that the improvement vanishes with a large ratio mainly because the baseline (normalized joint training) is already good enough with a large ratio.

Original: Second, the improvement from TAWT will decrease with the increase of the training size of the target data.
Revised: Second, the improvement from TAWT will decrease with the increase of the training size of the target data.

Original: In a word, TAWT is more beneﬁcial when the performance of the base model is poorer, either with a smaller target data or with a smaller ratio.
Revised: In a word, TAWT is more beneﬁcial when the performance of the base model is poorer, either with a smaller target data or with a smaller ratio.

Original: The speciﬁc performance for experiments in our analysis on two crucial factors for TAWT , the ratio between the training sizes of P e r f
Revised: The speciﬁc performance for experiments in our analysis on two crucial factors for TAWT , the ratio between the training sizes of the source and target datasets, and the training size of the target dataset, can be found in Table 4 and

Original: o r m an c e ( % )

Original: Single-Task Learning Joint Training

Original: Weighted-Sample Joint Training

Original: P e

Original: r

Original: f

Original: o r m an c e ( % )

Original: [Figure]

Original: [Figure]

Original: The training size of the target task

Original: [Figure]

Original: In this part, we keep the training size of the source task as 10000 and change the training size of the target task from 10 to 10000 in a log scale.

Original: STL denotes the single-task learning only with the target data.

Original: LTL denotes the learning to learn paradigm, where we ﬁrst learn the representations in the source data and then learn the task-speciﬁc function in the target data.

Original: [-For the learning to learn paradigm, we consider the source task with different ﬂip rates from 0 . 
-]
Revised: [-For the learning to learn paradigm, we consider the source task with different ﬂip rates from 0 . 
-]

Original: to 1 . 0 , where the ﬂip rate is an important factor in generating the source data and lower ﬂip rate indicates a smaller task distance between the source data and the target data.

Original: the source and target datasets, and the training size of the target dataset, can be found in Table 5 and

Original: Table 6 respectively.
Revised: Table 5 respectively.

Original: Dynamic weights analysis.
Revised: Dynamic weights analysis.

Original: As for experiments in Table 1, there are four tasks in total, PoS tagging, chunking, predicate detection, and NER.
Revised: As for experiments in Table 1, there are four tasks in total, PoS tagging, chunking, predicate detection, and NER.

Original: For each setting, we choose one task as the target task and the remaining three tasks are source tasks.
Revised: For each setting, we choose one task as the target task and the remaining three tasks are source tasks.

Original: The learned ﬁnal weights on tasks are shown in Table 7.
Revised: The learned ﬁnal weights on tasks are shown in Table 6.

Original: To better understand our algorithm, we compare TAWT with dynamic weights and TAWT with ﬁxed ﬁnal weights.
Revised: To better understand our algorithm, we compare TAWT with dynamic weights and TAWT with ﬁxed ﬁnal weights.

Original: TAWT with dynamic weights is the default setting for our algorithm, where weights on tasks for each epoch are different.
Revised: TAWT with dynamic weights is the default setting for our algorithm, where weights on tasks for each epoch are different.

Original: As for TAWT with ﬁxed weights, we simply initialize the weights as the ﬁnal weights learned by our algorithm and ﬁx the weights during the training.
Revised: As for TAWT with ﬁxed weights, we simply initialize the weights as the ﬁnal weights learned by our algorithm and ﬁx the weights during the training.

Original: As shown in Table 8, we ﬁnd that TAWT with dynamic weights (the default one) is slightly better than TAWT with ﬁnal ﬁxed weights.
Revised: As shown in Table 7, we ﬁnd that TAWT with dynamic weights (the default one) is slightly better than TAWT with ﬁnal ﬁxed weights.

Original: It indicates that ﬁxed weights might not be a good choice for weighted training, because the importance of different source tasks may change during the training.
Revised: It indicates that ﬁxed weights might not be a good choice for weighted training, because the importance of different source tasks may change during the training.

Original: In other words, it might be better to choose the weighted training with dynamic weights, where the weights are automatically adjusted based on the state of the trained model.
Revised: In other words, it might be better to choose the weighted training with dynamic weights, where the weights are automatically adjusted based on the state of the trained model.

Original: Extension to the weighted-sample training.
Revised: Extension to the weighted-sample training.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, TAWT can be easily extended from the weights on tasks to the weights on samples.
Revised: 2, TAWT can be easily extended from the weights on tasks to the weights on samples.

Original: We can see that TAWT based on weighted-sample training is also beneﬁcial for both pre-training and joint training.
Revised: We can see that TAWT based on weighted-sample training is also beneﬁcial for both pre-training and joint training.

Original: Because the weighted-sample training is much more costly compared to the weighted-task training, we choose a much simpler setting here.
Revised: Because the weighted-sample training is much more costly compared to the weighted-task training, we choose a much simpler [Figure]

Original: In this part, we use the PoS tagging on entity words as the source taskand named entity classiﬁcation on entity words as  the target task.{+task from 10 to 10000 in a log scale. STL denotes the single-task learning only with the target data. LTL denotes the learning to learn paradigm, where we ﬁrst learn the representations in the source data and then learn the task-speciﬁc function in the target data. For the learning to learn paradigm, we consider the source task with different ﬂip rates from 0 . 
+}
Revised: In this part, we keep the training size of the source task as 10000 and change the training size of the target {+task from 10 to 10000 in a log scale. STL denotes the single-task learning only with the target data. LTL denotes the learning to learn paradigm, where we ﬁrst learn the representations in the source data and then learn the task-speciﬁc function in the target data. For the learning to learn paradigm, we consider the source task with different ﬂip rates from 0 . 
+}

Revised: to 1 . 0 , where the ﬂip rate is an important factor in generating the source data and lower ﬂip rate indicates a smaller task distance between the source data and the target data.

Revised: setting here.

Revised: In this part, we use the PoS tagging on entity words as the source task and named entity classiﬁcation on entity words as the target task.

Original: Note that both tasks we considered here are word-level classiﬁcation tasks, i.e., predicting the label for a given word in a named entity.
Revised: Note that both tasks we considered here are word-level classiﬁcation tasks, i.e., predicting the label for a given word in a named entity.

Original: We still use the Ontontes 5.0 (Hovy et al., 2006) as our dataset.
Revised: We still use the Ontontes 5.0 (Hovy et al., 2006) as our dataset.

Original: There are 50 labels in PoS tagging on entity words, and 18 labels 10 in named entity classiﬁcation on entity words.
Revised: There are 50 labels in PoS tagging on entity words, and 18 labels 9 in named entity classiﬁcation on entity words.

Original: There are 37534 examples in the development set and 23325 examples in the test set for both source and target tasks.
Revised: There are 37534 examples in the development set and 23325 examples in the test set for both source and target tasks.

Original: As for the training set, we randomly sample 1000 examples for the source task and 100 examples for the target task.
Revised: As for the training set, we randomly sample 1000 examples for the source task and 100 examples for the target task.

Original: As for the model, we use two-layer NNs with 5 -gram features.
Revised: As for the model, we use two-layer NNs with 5 -gram features.

Original: The two-layer NNs have a hidden size of 4096 , ReLU non-linear activation, and cross-entropy loss.
Revised: The two-layer NNs have a hidden size of 4096 , ReLU non-linear activation, and cross-entropy loss.

Original: As for the embeddings, we use 50 dimensional Glove embeddings (Pennington et al., 2014).
Revised: As for the embeddings, we use 50 dimensional Glove embeddings (Pennington et al., 2014).

Original: The majority baseline for named entity classiﬁcation on entity words on the test data is 20 .
Revised: The majority baseline for named entity classiﬁcation on entity words on the test data is 20 .

Original: As for training models, the size of the training batch is 128 , the optimizer is Adam (Kingma & Ba, 2015) with a learning rate 3 e − 4 , and the number of training epochs is 200 .
Revised: As for training models, the size of the training batch is 128 , the optimizer is Adam (Kingma & Ba, 2015) with a learning rate 3 e − 4 , and the number of training epochs is 200 .

Original: As for updating weights on samples, we choose to update the weights every 5 epoch with the mirror descent in Eq.
Revised: As for updating weights on samples, we choose to update the weights every 5 epoch with the mirror descent in Eq.

Original: 2.8 and thus 40 updates on weights in total.
Revised: 2.8 and thus 40 updates on weights in total.

Original: In this part, we approximate the inverse of the Hessian matrix ( (cid:104) (cid:80)
Revised: In this part, we approximate the inverse of the Hessian matrix ( (cid:104) (cid:80)

Original: Tt =1 ω t ∇ 2 φ (cid:98)
Revised: Tt =1 ω t ∇ 2 φ (cid:98)

Original: L t ( φ k +1 , f k +1 t ) (cid:105) − 1 ) in
Revised: L t ( φ k +1 , f k +1 t ) (cid:105) − 1 ) in

Original: Eq.
Revised: Eq.

Original: 2.7 by a constant multiple of the identity matrix as in MAML (Finn et al., 2017), and choose the constant as 5 .
Revised: 2.7 by a constant multiple of the identity matrix as in MAML (Finn et al., 2017), and choose the constant as 5 .

Original: According to our experiments, the results of this approximation are similar to those of the approximation that we used in Sec. 4.
Revised: According to our experiments, the results of this approximation are similar to those of the approximation that we used in Sec. 4.

Original: The corresponding results are shown in Fig.
Revised: The corresponding results are shown in Fig.

Original: In the future, we also plan to group the instances and give each group a weight rather than each sample.
Revised: In the future, we also plan to group the instances and give each group a weight rather than each sample.

Original: Settings for simulations on task distance.
Revised: Settings for simulations on task distance.

Original: In this part, we ﬁrst randomly generate 10 K examples
Revised: In this part, we ﬁrst randomly generate 10 K examples

Original: D , where the dimension of the inputs are 1000 and the corresponding labels are in 0 − 9 ( 10 classes).
Revised: D , where the dimension of the inputs are 1000 and the corresponding labels are in 0 − 9 ( 10 classes).

Original: For each dimension of the input, it is randomly sampled from a uniform distribution over [ − 0 . 5 , 0 . 5] .
Revised: For each dimension of the input, it is randomly sampled from a uniform distribution over [ − 0 . 5 , 0 . 5] .

Original: For each ﬂip rate q % ∈ [0 , 1] , we randomly choose q % of the data and replace their labels with a random label uniformly sampled from the 10 classes.
Revised: For each ﬂip rate q % ∈ [0 , 1] , we randomly choose q % of the data and replace their labels with a random label uniformly sampled from the 10 classes.

Original: The data with ﬂip rate q % is denoted as D q % (including D 0 . 

Revised: The data with ﬂip rate q % is denoted as D q % (including D 0 .

Original: for the original dataset).
Revised: for the original dataset).

Original: For each dataset D q % , we train a 2-layer NNs M q % with 100% accuracy and almost zero training loss on the dataset.
Revised: For each dataset D q % , we train a 2-layer NNs M q % with 100% accuracy and almost zero training loss on the dataset.

Original: The 2-layer NNs can then be used to generatedata for the task
Revised: The 2-layer NNs can then be used to generate the source task.

Original: T q % .

Original: [-We use the T 0 . 
-]
Revised: [-We use the T 0 . 
-]

Original: [-as the target task and change q % from 0 . to 1 . 
 as the source task.-] Based on the process of the data generation, we can see that the ﬂip rate q % plays an important role in the intrinsic difference between the source task T q % and the target task T 0 . 0 .
Revised: [-as the target task and change q % from 0 . to 1 . 
 as the source task.-] Based on the process of the data generation, we can see that the ﬂip rate q % plays an important role in the intrinsic difference between the source task T q % and the target task T 0 . 0 .

Original: In general, we can expect that a smaller ﬂip rate q % indicates a smaller task distance between the source task T q % and the target task T 0 . 0 .
Revised: In general, we can expect that a smaller ﬂip rate q % indicates a smaller task distance between the source task T q % and the target task T 0 . 0 .

Original: This data generation process is inspired by the teacher-student network in (Hinton et al., 2015).
Revised: This data generation process is inspired by the teacher-student network in (Hinton et al., 2015).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, we keep the training size of the source task as 10000 and change the training size of the target task from 10 to 10000 in a log scale.
Revised: 3, we keep the training size of the source task as 10000 and change the training size of the target task from 10 to 10000 in a log scale.

Original: For simplicity, we use the same architectures for both the teacher network that we used for generating data and the student network that we used for learning the data, i.e., a two-layer neural network with a hidden size of 4096 , an input size of 1000 , and an output size of 10 .
Revised: For simplicity, we use the same architectures for both the teacher network that we used for generating data and the student network that we used for learning the data, i.e., a two-layer neural network with a hidden size of 4096 , an input size of 1000 , and an output size of 10 .

Original: As for training, the size of the training batch is 100 , the optimizer is Adam (Kingma & Ba, 2015) with learning rate 3 e − 4 , and the number of training epochs is 100 .
Revised: As for training, the size of the training batch is 100 , the optimizer is Adam (Kingma & Ba, 2015) with learning rate 3 e − 4 , and the number of training epochs is 100 .

Original: Details on using existing assets.
Revised: Details on using existing assets.

Original: As for the corresponding code and pre-trained models for BERT, we directly download it from the Github of huggingface whose license is Apache-2.0, and more details can be found in https://github.com/huggingface/transformers .
Revised: As for the corresponding code and pre-trained models for BERT, we directly download it from the Github of huggingface whose license is Apache-2.0, and more details can be found in https://github.com/huggingface/transformers .

Original: As for the
Revised: As for the

Original: Ontonotes 5.0, we obtain the data from LDC.
Revised: Ontonotes 5.0, we obtain the data from LDC.

Original: The corresponding license and other details can be found in https://catalog.ldc.upenn.edu/LDC2013T19 .
Revised: The corresponding license and other details can be found in https://catalog.ldc.upenn.edu/LDC2013T19 .

Original: As for the CoNLL- shared task, we download the data directly from the website and more details are in https: //www.clips.uantwerpen.be/conll2000/chunking .
Revised: As for the CoNLLshared task, we download the data directly from the website and more details are in https: //www.clips.uantwerpen.be/conll2000/chunking .

Original: B. O N THE C HOICE OF E XPERIMENTAL S ETTINGS
Revised: C O N THE C HOICE OF E XPERIMENTAL S ETTINGS

Original: In this part, we clarify the choice of experimental settings based on the following perspectives.
Revised: In this part, we clarify the choice of experimental settings based on the following perspectives.

Original: Signal selection: TAWT can also be applied to cross-domain or cross-lingual signals.
Revised: Signal selection: TAWT can also be applied to cross-domain or cross-lingual signals.

Original: Compared to the above two types of signals, we think cross-task signals are more widespread and more difﬁcult to be used efﬁciently.
Revised: Compared to the above two types of signals, we think cross-task signals are more widespread and more difﬁcult to be used efﬁciently.

Original: Therefore, in our experiments, we choose the most difﬁcult and crucial signals, cross-task signals, to verify the effectiveness of TAWT .
Revised: Therefore, in our experiments, we choose the most difﬁcult and crucial signals, cross-task signals, to verify the effectiveness of TAWT .

Original: Another reason is that we didn’t ﬁnd any existing weighted training algorithms with theoretical guarantees for cross-task learning, while weighted training is common for domain adaptation, such as importance sampling.
Revised: Another reason is that we didn’t ﬁnd any existing weighted training algorithms with theoretical guarantees for cross-task learning, while weighted training is common for domain adaptation, such as importance sampling.

Original: The choice of similar domains: We choose to select tasks with similar domains.
Revised: The choice of similar domains: We choose to select tasks with similar domains.

Original: Ideally, TAWT can be used for cross-task signals in different domains or languages.
Revised: Ideally, TAWT can be used for cross-task signals in different domains or languages.

Original: However, the gap between different domains or different languages can make cross-task learning more complicated.
Revised: However, the gap between different domains or different languages can make cross-task learning more complicated.

Original: We instead consider cross-task learning in similar domains of English to simplify the settings.
Revised: We instead consider cross-task learning in similar domains of English to simplify the settings.

Original: But similar domains didn’t mean that datasets overlap because we randomly sample sentences for different tasks independently without repetition, causing a small ratio of overlap in our case.
Revised: But similar domains didn’t mean that datasets overlap because we randomly sample sentences for different tasks independently without repetition, causing a small ratio of overlap in our case.

Original: Area selection: We choose to experiment with NLP tasks because we think cross-task signals are more common in the NLP area and NLP tasks are more diverse compared to tasks in other areas.
Revised: Area selection: We choose to experiment with NLP tasks because we think cross-task signals are more common in the NLP area and NLP tasks are more diverse compared to tasks in other areas.

Original: Dataset selection: We choose Ontonotes 5.0 as our main dataset because it is widely used, and provides large-scale expert annotations (on 2.9 million words) for a wide range of NLP tasks.
Revised: Dataset selection: We choose Ontonotes 5.0 as our main dataset because it is widely used, and provides large-scale expert annotations (on 2.9 million words) for a wide range of NLP tasks.

Original: This enables us to focus on learning with various tasks in similar domains.
Revised: This enables us to focus on learning with various tasks in similar domains.

Original: As for CoNLL-2000, we add it mainly because we want to analyze the impact of chunking on NER.
Revised: As for CoNLL-2000, we add it mainly because we want to analyze the impact of chunking on NER.

Original: Task selection:
Revised: Task selection:

Original: On the one hand, sequence tagging is more challenging than classiﬁcation tasks.
Revised: On the one hand, sequence tagging is more challenging than classiﬁcation tasks.

Original: On the other hand, the evaluation of sequence tagging is more reliable, compared with generative tasks.
Revised: On the other hand, the evaluation of sequence tagging is more reliable, compared with generative tasks.

Original: Another reason is that Ontonotes 5.0 mainly cover sequence tagging tasks.
Revised: Another reason is that Ontonotes 5.0 mainly cover sequence tagging tasks.

Original: Model selection: We choose to use BERT in our main experiments because BERT is widely used and all SOTA models are slight improvements of BERT.
Revised: Model selection: We choose to use BERT in our main experiments because BERT is widely used and all SOTA models are slight improvements of BERT.

Original: For weighted-sample training, we instead choose two-layer NNs with 5-gram features because it is simple and fast.
Revised: For weighted-sample training, we instead choose two-layer NNs with 5-gram features because it is simple and fast.

Original: Similar results could be shown even if the model is more complex, but exhaustive experimentation is not our goal.
Revised: Similar results could be shown even if the model is more complex, but exhaustive experimentation is not our goal.

Original: The choice of the low-resource setting: There is an intrinsic trade-off between the base performance of single-task learning and the relevant improvement of cross-task learning.
Revised: The choice of the low-resource setting: There is an intrinsic trade-off between the base performance of single-task learning and the relevant improvement of cross-task learning.

Original: Speciﬁcally, if the base performance of single-task learning is good enough, adding cross-task signals can introduce extra noise compared to its information.
Revised: Speciﬁcally, if the base performance of single-task learning is good enough, adding cross-task signals can introduce extra noise compared to its information.

Original: In our experiments, we simply consider a simple low-resource setting where few target examples are available.
Revised: In our experiments, we simply consider a simple low-resource setting where few target examples are available.

Original: Actually, we can still see the effectiveness of TAWT in cross-task learning as long as the base performance is not too high, but the relative improvement of
Revised: Actually, we can still see the effectiveness of TAWT in cross-task learning as long as the base performance is not too high, but the relative improvement of

Original: TAWT will be smaller compared to that in the low-resource setting.
Revised: TAWT will be smaller compared to that in the low-resource setting.