Original: S HOULD W E B E Pre - TRAINING ? A C ASE S TUDY OF E ND -T ASK A WARE T RAINING I N L IEU OF C ONTINUED P RE - TRAINING
Revised: S HOULD WE BE Pre -T RAINING ? E XPLORING E ND -T ASK A WARE T RAINING I N L IEU OF C ONTINUED P RE - TRAINING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Pre-training, where models are trained on an auxiliary objective with abundant data before being ﬁne-tuned on data from the downstream task, is now the dominant paradigm in NLP. In general, the pre-training step relies on little to no direct knowledge of the task on which the model will be ﬁne-tuned, even when the endtask is known in advance.
Revised: In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks. However, widely used methods for leveraging auxiliary data like pre-training and its continuedpretraining variant are end-task agnostic: they rarely, if ever, exploit knowledge of the target task.

Original: Our work challenges the status-quo of task-agnostic continued pre-training as a step towards the larger goal of investigating the place of pre -training when the  end-task is known in advance.
Revised: We study replacing end-task agnostic continued training of pretrained language models with end-task aware training of said models. We argue that for sufﬁciently important end-tasks, the beneﬁts of leveraging auxiliary data in a task-aware fashion can justify forgoing the traditional approach of obtaining generic, end-task agnostic representations as with (continued) pre-training.

Original: First, on three different low-resource NLP tasks from two domains, we demonstrate that multi-tasking the end-task and auxiliary objectives results in signiﬁcantly better downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan et al.
Revised: On three different low-resource NLP tasks from two domains, we demonstrate that multi-tasking the end-task and auxiliary objectives results in signiﬁcantly better downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan et al.

Original: We next introduce an online meta-learning algorithm that learns a set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on end task performance and data efﬁciency.
Revised: We next introduce an online meta-learning algorithm that learns a set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on endtask performance and data efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: optimizes both T aux and T simultaneously and can ﬁnd better minima since optimization is informed by the end-task.
Revised: better minima since optimization is informed by the end-task.

Original: The increasingly popular pre-training paradigm (Dai &
Revised: The increasingly popular pre-training paradigm (Dai &

Original: Le, 2015; Devlin et al., 2018; Gururangan et al., 2020) involves ﬁrst training a generalist model on copious amounts of easy-to-obtain data, e.g. raw text data in
Revised: Le, 2015; Devlin et al., 2018; Gururangan et al., 2020) involves ﬁrst training a generalist model on copious amounts of easy-to-obtain data, e.g. raw text data in

Original: NLP, and then using this model to initialize training on a wide swath of downstream tasks.
Revised: NLP, and then using this model to initialize training on a wide swath of downstream tasks.

Original: Generalist models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and GPT-3 (Brown et al., 2020) have a strong appeal; a few institutions with signiﬁcant resources incur the cost of training these large models whilst the rest of the research community enjoys a signiﬁcant performance improvement at minimal computational overhead.
Revised: Generalist models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and GPT-3 (Brown et al., 2020) have a strong appeal; a few institutions with signiﬁcant resources incur the cost of training these large models whilst the rest of the research community enjoys a signiﬁcant performance improvement at minimal computational overhead.

Original: However, the advantages of initializing a downstream task from a generalist model are not guaranteed.
Revised: However, the advantages of initializing a downstream task from a generalist model are not guaranteed.

Original: Previous work has shown that the beneﬁts of pre-training depend heavily on the degree of domain overlap between the end task data and the massive, heterogenous data on which the generalist model was trained (Beltagy et al., 2019; Gururangan et al.,2020; Lee et al., 2020).
Revised: Previous work has shown that the beneﬁts of pre-training depend heavily on the degree of domain overlap between the end-task data and the massive, heterogenous data on which the generalist model was trained (Beltagy et al., 2019; Gururangan et al., 2020).

Original: Notably, Gururangan et al.
Revised: Notably, Gururangan et al.

Original: (2020) have demonstrated the beneﬁts of continued pre-training of generalist models using data that is similar to that of the end task.
Revised: (2020) have demonstrated the beneﬁts of continued pre-training of generalist models using data that is similar to that of the end-task.

Original: Their approach is formalized into two classes:
Revised: Their approach is formalized into two classes:

Original: Domain Adaptive Pre-training (DAPT) and Task Adaptive Pretraining (TAPT) where further stages of pre-training of generalist models are conducted on domain- and task-speciﬁc data, respectively.
Revised: Domain Adaptive Pre-training (DAPT) and Task Adaptive Pretraining (TAPT) where further stages of pre-training of generalist models are conducted on domain- and task-speciﬁc data, respectively.

Original: DAPT and TAPT exploit the fact that we often know the end task beforehand , and so we can make speciﬁc choices about our pre-training regimen to improve end-task performance.
Revised: DAPT and TAPT exploit the fact that we often know the end-task beforehand , and so we can make speciﬁc choices about our pre-training regimen to improve end-task performance.

Original: However, in both pre-training for generalist models and continued pre-training, the training procedure itself does not explicitly incorporate the end task objective function .
Revised: However, in both pre-training for generalist models and continued pre-training, the training procedure itself does not explicitly incorporate the end-task objective function .

Original: Because of this, practi- tioners have to be careful with their choice of auxiliary tasks, the order in which they are trained on, and the early-stopping criteria for each pre-training stage so as to actually achieve good downstream end-task performance (Gururangan et al., 2020; Dery et al., 2021).
Revised: Because of this, practitioners have to be careful with their choice of auxiliary tasks, the order in which they are trained on, and the early-stopping criteria for each pre-training stage so as to actually achieve good downstream end-task performance (Gururangan et al., 2020; Dery et al., 2021).

Original: In the absence of principled criteria to make these difﬁcult design choices, it is common to instead resort to the computationally demanding heuristic of pre-training on as much data as possible for as long as possible.
Revised: In the absence of principled criteria to make these difﬁcult design choices, it is common to instead resort to the computationally demanding heuristic of pre-training on as much data as possible for as long as possible.

Original: In this paper, we raise the following question: “ In settings where we have a particular end task in mind, should we be pre- training at all? ”.
Revised: In this paper, we raise the following question: “ In settings where we have a particular end-task in mind, should we be pre- training at all? ”.

Original:  As a ﬁrst step in answering this question, we focus on themore computationally friendly and ubiquitous continued pre-training setting where we question the wisdom of having disjoint further pre -training and ﬁne-tuning steps on a generalist model.
Revised: We deﬁne pre -training as any form of task-agnostic training that a model undergoes before it is ﬁnally ﬁne-tuned on the end-task of interest. As a ﬁrst milestone in addressing the larger question posed above, we explore the ubiquitous continued pretraining setting (Gururangan et al., 2020; Aghajanyan et al., 2021). Speciﬁcally, our paper questions the wisdom of having disjoint further pre -training then ﬁne-tuning steps on a generalist model.

Original: In response, we advocate for an alternative approach in which we directly introduce the end task objective of interest into the learning process.
Revised: In response, we advocate for an alternative approach in which we directly introduce the end-task objective of interest into the learning process.

Original: This results in a suite of end-task aware methods called TARTAN (end- T ask A wa R e T r A ini N g).
Revised: This results in a suite of end-task aware methods called TARTAN (end- T ask A wa R e T r A ini N g).

Original: Our formulations incorporate both unsupervised auxiliary objectives traditionally used in NLP pre-training (such as masked language modeling as in Devlin et al.
Revised: Our formulations incorporate both unsupervised auxiliary objectives traditionally used in NLP pre-training (such as masked language modeling as in Devlin et al.

Original: and the end-task objective, followed by an optional ﬁne-tuning step on the end task.
Revised: and the end-task objective, followed by an optional ﬁne-tuning step on the end-task.

Original: We motivate TARTAN experimentally in the continued pre-training setting and based on this, we make the following contributions to the literature on leveraging auxiliary tasks and data:
Revised: We motivate TARTAN experimentally in the continued pre-training setting and based on this, we make the following contributions to the literature on leveraging auxiliary tasks and data:

Original: • In lieu of standard end-task
Revised: • In lieu of standard end-task

Original: agnostic continued pre-training, we suggest introducing the end task objective into the training process via multi-task learning (Caruana, 1997; Ruder, 2017).
Revised: agnostic continued pre-training, we suggest introducing the end-task objective into the training process via multi-task learning (Caruana, 1997; Ruder, 2017).

Original: We call this procedure M ulti- T asking end- T ask A wa R e T r A ini N g
Revised: We call this procedure M ulti- T asking end- T ask A wa R e T r A ini N g

Original: (MT-TARTAN)
Revised: (MT-TARTAN)

Original: (Section 3.1).
Revised: (Section 3.1).

Original: MTTARTAN is a simple yet surprisingly effective alternative to task-agnostic pre-training.
Revised: MTTARTAN is a simple yet surprisingly effective alternative to task-agnostic pre-training.

Original: In Section 5, we demonstrate that MT-TARTAN signiﬁcantly improves performance, convergence time, and data efﬁciency over Gururangan et al.
Revised: In Section 5, we demonstrate that MT-TARTAN signiﬁcantly improves performance and data efﬁciency over Gururangan et al.

Original: (2020)’s results.
Revised: (2020)’s results.

Original: It also obviates the need for ﬁckle hyperparameter tuning through direct optimization of validation performance.
Revised: It also obviates the need for ﬁckle hyper-parameter tuning through direct optimization of validation performance.

Original: • To allow more ﬁne-grained control of the end task over the auxiliary tasks, in Section 3.2, we present an online meta-learning algorithm that learns adaptive multi-task weights with the aim of improving ﬁnal end task performance.
Revised: • To allow more ﬁne-grained control of the end-task over the auxiliary tasks, in Section 3.2, we present an online meta-learning algorithm that learns adaptive multi-task weights with the aim of improving ﬁnal end-task performance.

Original: Our META -learning end- T ask A wa R e T r A ini N g
Revised: Our META -learning end- T ask A wa R e T r A ini N g

Original: (METATARTAN) allows us to robustly modulate between multiple objectives and further improves performance over MT-TARTAN .
Revised: (METATARTAN) allows us to robustly modulate between multiple objectives and further improves performance over MT-TARTAN .

Original: • A naive implementation of META-TARTAN based on ﬁrst-order meta-learning analysis results in a sub-optimal algorithm that ignores all tasks except the end-task.
Revised: • A naive implementation of META-TARTAN based on ﬁrst-order meta-learning analysis results in a sub-optimal algorithm that ignores all tasks except the end-task.

Original: We trace this problem to the use of a single model training head for computing both the end-task training loss and meta-objective (end-task validation loss).
Revised: We trace this problem to the use of a single model training head for computing both the end-task training loss and meta-objective (end-task validation loss).

Original: To guard against this pathological solution, we introduce a separate model head for computing the meta-objective.
Revised: To guard against this pathological solution, we introduce a separate model head for computing the meta-objective.

Original: In Section 3.3, we justify this simple-to-implement ﬁx and validate its practical efﬁcacy in Section 5.
Revised: In Section 3.3, we justify this simple-to-implement ﬁx and validate its practical efﬁcacy in Section 5.

Original: Our results suggest that TARTAN may be an attractive alternative to the continued pre-training paradigm, andfurther research into the place of pre -training in end-task aware settings is warranted.
Revised: Our results suggest that TARTAN may be an attractive alternative to the continued pre-training paradigm , and

Revised: further research into the place of pre -training in end-task aware settings is warranted.

Original:  F ORMALIZING P RE - TRAINING AND C ONTINUED P RE - TRAINING
Revised:  F ORMALIZING P RE - TRAINING AND C ONTINUED P RE - TRAINING

Original: Consider a dataset D
Revised: Consider a dataset D

Original: = { ( x i , y i ) i ∈
Revised: = { ( x i , y i ) i ∈

Original: [ m ] } consisting of m labelled examples.
Revised: [ m ] } consisting of m labelled examples.

Original: We deﬁne a task as an objective function and dataset pair: T =
Revised: We deﬁne a task as an objective function and dataset pair: T =

Original: {L ( · ) , D } .
Revised: {L ( · ) , D } .

Original: Consider M θ , a model parameterized by θ .
Revised: M θ is a model parameterized by θ .

Original: The objective function L ( y i , M θ ( x i )) evaluates how well a model prediction M
Revised: The objective function L ( y i , M θ ( x i )) evaluates how well a model prediction M

Original: θ
Revised: θ

Original: x i ) ﬁts the true label
Revised: x i ) ﬁts the true label

Original: y i , such as cross-entropy loss in the case of classiﬁcation.
Revised: y i , such as cross-entropy loss in the case of classiﬁcation.

Original: Note that the task dataset, D , is typically decomposed into the sets ( D train , D val , D test ) .
Revised: Note that the task dataset, D , is typically decomposed into the sets ( D train , D val , D test ) .

Original: D train is the set of examples used for model training whilst D test is used for ﬁnal task evaluation.
Revised: D train is the set of examples used for model training whilst D test is used for ﬁnal task evaluation.

Original: The validation set, D val , is typically used for model selection but it is also frequently used in meta-learning to deﬁne the meta-objective – L valT ∗ .
Revised: The validation set, D val , is typically used for model selection but it is also frequently used in meta-learning to deﬁne the meta-objective – L val .

Original: Given a speciﬁc end-task T ∗ , our aim is to improve performance on T ∗ (as measured by the model loss on D test T ∗ ) by leveraging auxiliary tasks
Revised: Given a speciﬁc end-task T ∗ , our aim is to improve performance on T ∗ (as measured by the model loss on D test T ∗ ) by leveraging auxiliary tasks

Original: T aux =
Revised: T aux =

Original: { T 1 , . . .
Revised: { T 1 , . . .

Original: , T n } .
Revised: , T n } .

Original: Note that we do not particularly care about the performance of any of the tasks in T aux .
Revised: Note that we do not particularly care about the performance of any of the tasks in T aux .

Original: We are willing to sacriﬁce performance on T aux if it improves performance on T ∗ .
Revised: We are willing to sacriﬁce performance on T aux if it improves performance on T ∗ .

Original: From the perspective of model architecture, there are several ways to leverage T aux .
Revised: From the perspective of model architecture, there are several ways to leverage T aux .

Original: We focus on the simple but ubiquitous parameter sharing setting.
Revised: We focus on the simple but widely-used parameter sharing setting.

Original: Here, all tasks share a model body θ body but each task T i has its own head φ i for prediction.
Revised: Here, all tasks share a model body θ body but each task T i has its own head φ i for prediction.

Original: We denote the head belonging to T ∗ as φ (cid:48) .
Revised: We denote the head belonging to T ∗ as φ (cid:48) .

Original: Thus θ = (cid:2) θ body ; (cid:0) φ 1 , . . .
Revised: Thus θ = (cid:2) θ body ; (cid:0) φ 1 , . . .

Original: , φ n , φ (cid:48) (cid:1)(cid:3)
Revised: , φ n , φ (cid:48) (cid:1)(cid:3)

Original: and θ body is reusable across new tasks.
Revised: and θ body is reusable across new tasks.

Original: 2. P RE - TRAINING
Revised: 2. P RE - TRAINING

Original: Pre-training is when a model is ﬁrst trained on T aux before performing a ﬁnal ﬁne-tuning phase on T ∗ .
Revised: Pre-training is when a model is ﬁrst trained on T aux before performing a ﬁnal ﬁne-tuning phase on T ∗ .

Original: The motivation behind pre-training is that learning T aux ﬁrst hopefully captures relevant information that can be utilized during training of T ∗ .
Revised: The motivation behind pre-training is that learning T aux ﬁrst hopefully captures relevant information that can be utilized during training of T ∗ .

Original: This desire has led to the proliferation of generalist pre-trained models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and GPT3 (Brown et al., 2020) that have been trained on copious amounts of data.
Revised: This desire has led to the proliferation of generalist pre-trained models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and GPT3 (Brown et al., 2020) that have been trained on copious amounts of data.

Original: Generalist models have been widely successful at improving downstream task performance when used as initilization.
Revised: Generalist models have been widely successful at improving downstream task performance when used as initilization.

Original: We can formalize the pre-training procedure as follows:
Revised: We can formalize the pre-training procedure as follows:

Original: [Equation]
Revised: [Equation]

Original: In Equation 1, we seek a point θ 0 that achieves minimal loss on the tasks in T aux .
Revised: In Equation 1, we seek a point θ 0 that achieves minimal loss on the tasks in T aux .

Original: We hope that θwill be a good starting point for gradient descent on T ∗ .
Revised: We hope that θwill be a good starting point for gradient descent on T ∗ .

Original: Let g ( θ 0 ) represent the set of end-points of stochastic gradient descent on an initialization, θ 0 .
Revised: Let g ( θ 0 ) represent the set of end-points of stochastic gradient descent on an initialization, θ 0 .

Original: Stochastic gradient descent from the same initialization can produce different end-points due to differences in hyper-parameters like learning rate, batch size and order, as well as regularization strength.
Revised: Stochastic gradient descent from the same initialization can produce different end-points due to differences in hyper-parameters like learning rate, batch size and order, as well as regularization strength.

Original: We can write the ﬁne-tuning phase as:
Revised: We can write the ﬁne-tuning phase as:

Original: [Equation]
Revised: [Equation]

Original: Note that pre-training is end-task agnostic : the pre-training Equation 1 occurs entirely before training on the end-task Equation 2, and does not explicitly incorporate the end-task objective, T ∗ .
Revised: Note that pre-training is end-task agnostic : the pre-training Equation 1 occurs entirely before training on the end-task Equation 2, and does not explicitly incorporate the end-task objective, T ∗ .

Original: Since there is no awareness of the end-task during pre-training it is important to carefully choose T aux so that pre-training actually results in improved performance on T ∗ (Wang et al., 2018a).
Revised: Since there is no awareness of the end-task during pre-training it is important to carefully choose T aux so that pre-training actually results in improved performance on T ∗ (Wang et al., 2018a).

Original: For text data, past work has found left-to-right language modeling (Peters et al., 2017) and masked language modeling (MLM) (Devlin et al., 2018) to be good choices to include in T aux .
Revised: For text data, past work has found left-to-right language modeling (Peters et al., 2017) and masked language modeling (MLM) (Devlin et al., 2018) to be good choices to include in T aux .

Original: 2. C ONTINUED P RE - TRAINING
Revised: 2. C ONTINUED P RE - TRAINING

Original: Recent work (Beltagy et al., 2019; Gururangan et al., 2020; Lee et al., 2020) showed that downstream performance on T ∗ can be improved by further adapting generalist models via continued pre-training on a more relevant set of auxiliary tasks.
Revised: Recent work (Beltagy et al., 2019; Gururangan et al., 2020; Lee et al., 2020) showed that downstream performance on T ∗ can be improved by further adapting generalist models via continued pre-training on a more relevant set of auxiliary tasks.

Original: This is equivalent to sequentially performing multiple steps of Equation 1, with different T aux , before ﬁnally performing Equation 2 on T ∗ .
Revised: This is equivalent to sequentially performing multiple steps of Equation 1, with different T aux , before ﬁnally performing Equation 2 on T ∗ .

Original: Domain and Task Adaptive Pre-training Gururangan et al.
Revised: Domain and Task Adaptive Pre-training Gururangan et al.

Original: (2020) present Domain Adaptive PreTraining (DAPT) and Task Adaptive Pre-Training (TAPT) as methods for continued pre-training.
Revised: (2020) present Domain Adaptive PreTraining (DAPT) and Task Adaptive Pre-Training (TAPT) as methods for continued pre-training.

Original: During DAPT, a generalist model is further pre-trained on an unsupervised objective with large amounts of data from the same domain as the end-task.
Revised: During DAPT, a generalist model is further pre-trained on an unsupervised objective with large amounts of data from the same domain as the end-task.

Original: TAPT also pre-trains with the same unsupervised objective as DAPT, but on the actual dataset of the end task.
Revised: TAPT also pre-trains with the same unsupervised objective as DAPT, but on the actual dataset of the end-task.

Original: Gururangan et al.
Revised: Gururangan et al.

Original: ﬁnd that performance can be further improved by chaining objectives, DAPT ﬁrst, followed by TAPT.
Revised: ﬁnd that performance can be further improved by chaining objectives, DAPT ﬁrst, followed by TAPT.

Original: Though TAPT and DAPT do not directly incorporate the end-task objective during training, it still indirectly informs both the choice of pre-training data and the order in which the pre-training tasks are trained on.
Revised: Though TAPT and DAPT do not directly incorporate the end-task objective during training, it still indirectly informs both the choice of pre-training data and the order in which the pre-training tasks are trained on.

Original: Below, we explore stronger versions of this inﬂuence.
Revised: Below, we explore stronger versions of this inﬂuence.

Original:  E ND -T ASK A WARE T RAINING (TARTAN)
Revised:  E ND -T ASK A WARE T RAINING (TARTAN)

Original: In this section, we argue for the end-task to be added directly into the training process to create explicit interactions between T ∗ and T aux .
Revised: In this section, we argue for the end-task to be added directly into the training process to create explicit interactions between T ∗ and T aux .

Original: 3. E ND - TASK A WARE T RAINING VIA M ULTI - TASKING (MT-TARTAN)
Revised: 3. E ND - TASK A WARE T RAINING VIA M ULTI - TASKING (MT-TARTAN)

Original: We propose to directly incorporate knowledge of the end-task by multi-tasking T ∗ together with T aux , before optionally ﬁne-tuning on T ∗ exclusively.
Revised: We propose to directly incorporate knowledge of the end-task by multi-tasking T ∗ together with T aux , before optionally ﬁne-tuning on T ∗ exclusively.

Original: To this end, we introduce a set of task weights w = ( w ∗ , w 1 , · · · , w | T aux | ) satisfying w ∗
Revised: To this end, we introduce a set of task weights w = ( w ∗ , w 1 , · · · , w | T aux | ) satisfying w ∗

Original: + (cid:80) i
Revised: + (cid:80) i

Original: w i
Revised: w i

Original: = 1 , to modulate between the different losses.
Revised: = 1 , to modulate between the different losses.

Original: Our new formulation is:
Revised: Our new formulation is:

Original: [Equation]
Revised: [Equation]

Original: Here, Equation 3 replaces Equation 1 and can be followed by the optional ﬁne-tuning stage of
Revised: Here, Equation 3 replaces Equation 1 and can be followed by the optional ﬁne-tuning stage of

Original: Equation 2.
Revised: Equation 2.

Original: Note that this formulation ﬁxes the tasks weights w throughout the training process.
Revised: Note that this formulation ﬁxes the tasks weights w throughout the training process.

Original: We call this formulation End-task Aware Training via Multi-tasking (MT-TARTAN) because we introduce the end-task directly into the training procedure, and do so by multi-tasking it with T aux .
Revised: We call this formulation End-task Aware Training via Multi-tasking (MT-TARTAN) because we introduce the end-task directly into the training procedure, and do so by multi-tasking it with T aux .

Original: MT-TARTAN allows us to prioritize performance on T ∗ in several ways.
Revised: MT-TARTAN allows us to prioritize performance on T ∗ in several ways.

Original: First, we can weight the end-task higher than all the other auxiliary tasks.
Revised: First, we can weight the end-task higher than all the other auxiliary tasks.

Original: Also, during training, we can monitor L T ∗ on the end-task validation set and early stop when it plateaus; even if the auxiliary tasks have not yet converged.
Revised: Also, during training, we can monitor L T ∗ on the end-task validation set and early stop when it plateaus; even if the auxiliary tasks have not yet converged.

Original: This is not possible during standard pre-training because we do not train T ∗
Revised: This is not possible during standard pre-training because we do not train T ∗

Original: and so it performs at random before we actually start ﬁne-tuning.
Revised: and so it performs at random before we actually start ﬁne-tuning.

Original: Early stopping on T ∗ can represent signiﬁcant computational savings over end-task agnostic pre-training when the savings in dataefﬁciency supercede the extra overhead of end-task aware gradient descent steps.
Revised: Early stopping on T ∗ can represent significant computational savings over end-task agnostic pre -training when the savings in data-efﬁciency supercede the extra overhead of end-task aware gradient descent steps.

Original: 3. E ND - TASK A WARE T RAINING VIA M ETA - LEARNING (META-TARTAN)
Revised: 3. E ND - TASK A WARE T RAINING VIA M ETA - LEARNING (META-TARTAN)

Original: MT-TARTAN, DAPT and TAPT, all share the same drawback: they implicitly assume that the auxiliary tasks have static importance to the end-task over the lifetime of its training, either by being end-task agnostic (DAPT and TAPT) or by having static task weights (MT-TARTAN).
Revised: MT-TARTAN, DAPT and TAPT, all share the same drawback: they implicitly assume that the auxiliary tasks have static importance to the end-task over the lifetime of its training, either by being end-task agnostic (DAPT and TAPT) or by having static task weights (MT-TARTAN).

Original: With MTTARTAN, an additional drawback noted by Wang et al.
Revised: With MTTARTAN, an additional drawback noted by Wang et al.

Original: ; Yu et al.
Revised: ; Yu et al.

Original: (2020) is that multi-tasking can negatively impact task performance compared to isolated training.
Revised: (2020) is that multi-tasking can negatively impact task performance compared to isolated training.

Original: These shortcomings motivate the formulation of an adaptive algorithm that can mitigate the negative inﬂuence of some tasks whilst responding to the changing relevance of auxiliary tasks over the lifetime of end-task training.
Revised: These shortcomings motivate the formulation of an adaptive algorithm that can mitigate the negative inﬂuence of some tasks whilst responding to the changing relevance of auxiliary tasks over the lifetime of end-task training.

Original: As they stand, the pre-training equation pair (Equations 1, 2) and the MT-TARTAN pair (Equations 2, 3) are decoupled.
Revised: As they stand, the pre-training equation pair (Equations 1, 2) and the MT-TARTAN pair (Equations 2, 3) are decoupled.

Original: The inner-level variables of the pre-training phase do not depend on the outerlevel variables of the ﬁne-tuning phase.
Revised: The inner-level variables of the pre-training phase do not depend on the outerlevel variables of the ﬁne-tuning phase.

Original: Thus the equation pairs are typically solved sequentially.
Revised: Thus the equation pairs are typically solved sequentially.

Original: We propose to tightly couple Equations 2 and 3 by formulating jointly learning w and θ 0 as a bi-level optimization problem.
Revised: We propose to tightly couple Equations 2 and 3 by formulating jointly learning w and θ 0 as a bi-level optimization problem.

Original: A bi-level formulation allows us to leverage meta-learning (Schmidhuber, 1995) techniques to learn adaptive task weights which capture variable auxiliary task importance whilst mitigating the contribution of harmful tasks.
Revised: A bi-level formulation allows us to leverage meta-learning (Schmidhuber, 1995) techniques to learn adaptive task weights which capture variable auxiliary task importance whilst mitigating the contribution of harmful tasks.

Original: We propose a meta-learning algorithm in the mold of Model Agnostic Meta-Learning (MAML)
Revised: We propose a meta-learning algorithm in the mold of Model Agnostic Meta-Learning (MAML)

Original: (Finn et al., 2017) to learn task weights.
Revised: (Finn et al., 2017) to learn task weights.

Original: As a bi-level problem, this can be formulated as :
Revised: As a bi-level problem, this can be formulated as :

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We want to jointly learn w , with θ 0 , such that taking a gradient descent step modulated by w leads to improvement in end-task generalization.
Revised: We want to jointly learn w , with θ 0 , such that taking a gradient descent step modulated by w leads to improvement in end-task generalization.

Original: We use performance on the end-task validation set ( D val T ∗ ) as a meta-objective to train w .
Revised: We use performance on the end-task validation set ( D val T ∗ ) as a meta-objective to train w .

Original: Performance on D val T ∗ serves as a stand-in for end-task generalization performance whilst also naturally capturing the asymmetrical importance of T ∗ .
Revised: Performance on D val T ∗ serves as a stand-in for end-task generalization performance whilst also naturally capturing the asymmetrical importance of T ∗ .

Original: Our joint descent algorithm proceeds as follows.
Revised: Our joint descent algorithm proceeds as follows.

Original: At each timestep t , we hold the task weights ﬁxed and update θ t based on ∇ θ L total ( θ t , w ) .
Revised: At each timestep t , we hold the task weights ﬁxed and update θ t based on ∇ θ L total ( θ t , w ) .

Original: We then proceed to update w via gradient descent on the end-task validation loss at θ t +1 .
Revised: We then proceed to update w via gradient descent on the end-task validation loss at θ t +1 .

Original: For this, we derive an approximation for ∇ w L valT ∗ ( θ t +1 , w ) below:
Revised: For this, we derive an approximation for ∇ w L val T ∗ ( θ t +1 , w ) below:

Original: [Equation]
Revised: [Equation]

Original: We can take the gradient of the above ﬁrst-order approximation w.r.t an individual weight w i .
Revised: We can take the gradient of the above ﬁrst-order approximation w.r.t an individual weight w i .

Original: This tells us how to update w i to improve the meta-objective.
Revised: This tells us how to update w i to improve the meta-objective.

Original: [Equation]
Revised: [Equation]

Original: In Equation 6, we explicitly specify (cid:2) θ body , φ (cid:48) (cid:3) t because computing losses on T ∗ depend on only these parameters.
Revised: In Equation 6, we explicitly specify (cid:2) θ body , φ (cid:48) (cid:3) t because computing losses on T ∗ depend on only these parameters.

Original: L T i depends solely on (cid:2) θ body , φ i (cid:3) t
Revised: L T i depends solely on (cid:2) θ body , φ i (cid:3) t

Original: but we leave this out to avoid notation clutter.
Revised: but we leave this out to avoid notation clutter.

Original: Our analysis above is similar to that of Lin et al.
Revised: Our analysis above is similar to that of Lin et al.

Original: (2019) with one key difference: we learn a weighting for the main task w ∗ too.
Revised: (2019) with one key difference: we learn a weighting for the main task w ∗ too.

Original: This ability to directly modulate T ∗ allows us to capture the fact that at certain stages in training, auxiliary tasks may have greater impact on end-task generalization than the end-task’s own training data.
Revised: This ability to directly modulate T ∗ allows us to capture the fact that at certain stages in training, auxiliary tasks may have greater impact on end-task generalization than the end-task’s own training data.

Original: This choice also allows us to control for over-ﬁtting and the inﬂuence of bad (mislabelled or noisy) training data.
Revised: This choice also allows us to control for over-ﬁtting and the inﬂuence of bad (mislabelled or noisy) training data.

Original: 3. I NTRODUCING A SEPARATE CLASSIFICATION HEAD FOR META - LEARNING
Revised: 3. I NTRODUCING A SEPARATE CLASSIFICATION HEAD FOR META - LEARNING

Original: Observe that from Equation 6, updates for w (cid:54) =
Revised: Observe that from Equation 6, updates for w (cid:54) =

Original: w ∗ involve gradients computed from different model heads φ i and φ (cid:48) whilst for w ∗ , we are taking the dot product of gradients from the same end-task head φ (cid:48) .
Revised: w ∗ involve gradients computed from different model heads φ i and φ (cid:48) whilst for w ∗ , we are taking the dot product of gradients from the same end-task head φ (cid:48) .

Original: As we will show empirically in Section 5.4, computing weight updates this way creates a strong bias towards the primary task, causing w ∗ to rail towards 1 whilst the other weights dampen to 0, which may be sub-optimal in the long run.
Revised: As we will show empirically in Section 5.4, computing weight updates this way creates a strong bias towards the primary task, causing w ∗ to rail towards 1 whilst the other weights dampen to 0, which may be sub-optimal in the long run.

Original: Intuitively, this short-horizon (greedy)  behavior makes sense: the quickest way to make short-term progress (improve L valT ∗ ( θ t +1 ) ) is to descend solely on T ∗ .
Revised: Intuitively, this short-horizon (greedy) (Wu et al., 2018) behavior makes sense: the quickest way to make short-term progress (improve L valT ∗ ( θ t +1 ) ) is to descend solely on T ∗ .

Original: More formally, the greedy approach arises because we derive ∇ w i L valT ∗ ( θ t +1 ) in Equation 6 as a proxy for the gradient at θ ∗ , the outerloop end-point in Equation 4.
Revised: More formally, the greedy approach arises because we derive ∇ w i L valT ∗ ( θ t +1 ) in Equation 6 as a proxy for the gradient at θ ∗ , the outer-loop end-point in Equation 4.

Original: Variations of this substitution are common in the meta-learning literature (Finn et al., 2017;
Revised: Variations of this substitution are common in the meta-learning literature (Finn et al., 2017;

Original: Liu et al., 2018; Nichol et al., 2018) because it is computationally infeasible to train a model to convergence every time we wish to compute ∇ w i L valT ∗ ( θ ∗ ) .
Revised: Liu et al., 2018; Nichol et al., 2018) because it is computationally infeasible to train a model to convergence every time we wish to compute ∇ w i L valT ∗ ( θ ∗ ) .

Original: To remedy the greedy solution, instead of estimating ∇ θ L T ∗ and ∇ θ L valT ∗ from the same classiﬁcation head (Equation 6), we introduce a special head φ ∗ for computing the meta-objective.
Revised: To remedy the greedy solution, instead of estimating ∇ θ L T ∗ and ∇ θ L valT ∗ from the same classiﬁcation head (Equation 6), we introduce a special head φ ∗ for computing the meta-objective.

Original: Specifically, instead of trying to compute θ ∗ , we approximate it by ﬁxing the body of the network θ body and training the randomly initialized head φ ∗ to convergence on a subset of the end-task training data.
Revised: Specifically, instead of trying to compute θ ∗ , we approximate it by ﬁxing the body of the network θ body and training the randomly initialized head φ ∗ to convergence on a subset of the end-task training data.

Original: We do this every time we wish to estimate ∇ w i L valT ∗ ( θ ∗ ) .
Revised: We do this every time we wish to estimate ∇ w i L valT ∗ ( θ ∗ ) .

Original: Introducing φ ∗ eliminates the strong positive bias on w ∗ and enables us to compute a better proxy for the meta-gradient at θ ∗ :
Revised: Introducing φ ∗ eliminates the strong positive bias on w ∗ and enables us to compute a better proxy for the meta-gradient at θ ∗ :

Original: [Equation]
Revised: [Equation]

Original: Equation 7 represents a simple-to-implement alternative to Equation 6.
Revised: Equation 7 represents a simple-to-implement alternative to Equation 6.

Original: We provide a more detailed justiﬁcation for Equation 7 in Appendix A.1.
Revised: We provide a more detailed justiﬁcation for Equation 7 in Appendix A.1.

Original: In Section 5.4, we empirically validate that the transition from Equation 6 to 7 improves performance whilst mitigating pathological solutions.
Revised: In Section 5.4, we empirically validate that the transition from Equation 6 to 7 improves performance whilst mitigating pathological solutions.

Original: Our approach of creating φ ∗ for approximating the meta-objective (down-stream validation performance) is inspired by Metz et al.
Revised: Our approach of creating φ ∗ for approximating the meta-objective (down-stream validation performance) is inspired by Metz et al.

Original: (2018), who use a similar technique to construct a meta-objective for evaluating the quality of unsupervised representations.
Revised: (2018), who use a similar technique to construct a meta-objective for evaluating the quality of unsupervised representations.

Original: See Algorithm 1 in Appendix A.3 for details about META-TARTAN.
Revised: Please see Algorithm 1 in Appendix A.3 for details about META-TARTAN.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Revised: Setting 1 Though our algorithms and methodology can be directly applied to both continued pretraining (Section 2.2) and pre-training from scratch (Section 2.1) of generalist models, we focus on the former scenario.

Revised: This is because the continued pre-training setting is more common amongst everyday practitioners as it is less computationally demanding.

Revised: It thus lends itself more easily to exploration under a realistic computational budget.

Revised: In Appendix A.4, we show that end-task aware training from scratch is viable by studying a simple computer vision setting.

Original:  In keeping with previous work (Devlin et al., 2018; Gururangan et al.,2020; Lee et al., 2020), we focus on T aux as a set of MLM tasks on varied datasets.
Revised: Concurrent work by Yao et al. (2021) shows that from-scratch end-task aware training for NLP problems is viable. In keeping with previous work (Devlin et al., 2018; Gururangan et al., 2020), we focus on T aux as a set of MLM tasks on varied datasets.

Original: In the case of DAPT and our end-task aware variants of it, T aux is an MLM task with data from the domain of the end-task.
Revised: In the case of DAPT and our end-task aware variants of it, T aux is an MLM task with data from the domain of the end-task.

Original: For TAPT, T aux is an MLM task with data from the end-task itself.
Revised: For TAPT, T aux is an MLM task with data from the end-task itself.

Original: DAPT, TAPT and DAPT+TAPT (chained pre-training with DAPT followed by TAPT) will serve as our baseline  pre-training approaches.
Revised: DAPT, TAPT and DAPT+TAPT (chained pre-training with DAPT followed by TAPT) will serve as our baseline continued pre-training approaches.

Original: We will compare these baselines to their end-task aware variants that use MT-TARTAN and META-TARTAN.
Revised: We will compare these baselines to their end-task aware variants that use MT-TARTAN and META-TARTAN.

Original: Our choice of baselines reﬂect our focus on the continued pre-training setting (Section 2.2) instead of pre-training from scratch as with generalist models (Section 2.1).

Original: The continued pre-training setting is more ubiquitous amongst everyday practitioners as it is less computationally demanding.

Original: It thus lends itself more easily to exploration.

Original: We hope that our initial inquiry into the continuedpretraining setting will spur future investigation into end-task-aware training from scratch.

Original:  Our experiments focus on two domains: computer science (CS) papers and biomedical (BIOMED) papers.
Revised: Datasets Our experiments focus on two domains: computer science (CS) papers and biomedical (BIOMED) papers.

Original: We follow Gururangan et al.
Revised: We follow Gururangan et al.

Original: and build our CS and BIOMED domain data from the S2ORC dataset (Lo et al., 2019).
Revised: and build our CS and BIOMED domain data from the S2ORC dataset (Lo et al., 2019).

Original: We extract 1.49M full text articles to construct our CS corpus and 2.71M for our BIOMED corpus.
Revised: We extract 1.49M full text articles to construct our CS corpus and 2.71M for our BIOMED corpus.

Original: Under both domains, our end-tasks are low-resource classiﬁcation tasks.
Revised: Under both domains, our end-tasks are low-resource classiﬁcation tasks.

Original: Using low-resource tasks allows us to explore a setting where pre-training can have a signiﬁcant impact.
Revised: Using low-resource tasks allows us to explore a setting where pre-training can have a signiﬁcant impact.

Original: Under the CS domain, we consider two tasks: ACL-ARC (Jurgens et al., 2018) and SCIERC (Luan et al., 2018).
Revised: Under the CS domain, we consider two tasks: ACL-ARC (Jurgens et al., 2018) and SCIERC (Luan et al., 2018).

Original: ACL-ARC is a 6-way citation intent classiﬁcation task with 1688 labelled training examples.
Revised: ACL-ARC is a 6-way citation intent classiﬁcation task with 1688 labelled training examples.

Original: For SCIERC, the task is to classify the relations between entities in scientiﬁc articles.
Revised: For SCIERC, the task is to classify the relations between entities in scientiﬁc articles.

Original: This task has 3219 labelled examples as training data.
Revised: This task has 3219 labelled examples as training data.

Original: We choose CHEMPROT (Kringelum et al., 2016) as the classiﬁcation task from the BIOMED domain.
Revised: We choose CHEMPROT (Kringelum et al., 2016) as the classiﬁcation task from the BIOMED domain.

Original: This task has 4169 labelled training examples and the goal is to classify chemical-protein interactions.
Revised: This task haslabelled training examples and the goal is to classify chemical-protein interactions.

Original: More details of these datasets can be found in Table 2 of Gururangan et al.
Revised: More details of these datasets can be found in Table 2 of Gururangan et al.

Original: Gururangan et al.
Revised: Gururangan et al.

Original: (2020) evaluate against all 3 tasks and their available code served as a basis on which we built MT-TARTAN and META-TARTAN.
Revised: (2020) evaluate against all 3 tasks and their available code served as a basis on which we built MT-TARTAN and META-TARTAN.

Original: Model Details We use a pre-trained RoBERTa base (Liu et al., 2019) as the shared model base and implement each task as a separate multi-layer perceptron (MLP) head on top of this pre-trained base.
Revised: Model Details We use a pre-trained RoBERTa base (Liu et al., 2019) as the shared model base and implement each task as a separate multi-layer perceptron (MLP) head on top of this pre-trained base.

Original: As in Devlin et al.
Revised: As in Devlin et al.

Original: (2018), we pass the [CLS] token embedding from RoBERTa base to the MLP for classiﬁcation.
Revised: (2018), we pass the [CLS] token embedding from RoBERTa base to the MLP for classiﬁcation.

Original: Training Details For DAPT and TAPT, we download the available pre-trained model bases provided by Gururangan et al.
Revised: Training Details For DAPT and TAPT, we download the available pre-trained model bases provided by Gururangan et al.

Original: To train thier corresponding classiﬁcation heads, we follow the experimental setup described in Appendix B of Gururangan et al.
Revised: To train thier corresponding classiﬁcation heads, we follow the experimental setup described in Appendix B of Gururangan et al.

Original: Performing end-task aware training introduces a few extra hyper-parameters.
Revised: Performing end-task aware training introduces a few extra hyper-parameters.

Original: We ﬁx the other hyperparameters to those used in Gururangan et al.
Revised: We ﬁx the other hyperparameters to those used in Gururangan et al.

Original: MT-TARTAN and META-TARTAN introduce joint training of a classiﬁcation head for the end-task T ∗ .
Revised: MT-TARTAN and META-TARTAN introduce joint training of a classiﬁcation head for the end-task T ∗ .

Original: We experiment with batch sizes of 128, 256 and 512 for training this head.
Revised: We experiment with batch sizes of 128, 256 and 512 for training this head.

Original: We try out learning rates in the set { 10 − 3 , 10 − 4 , 10 − 5 } and drop out rates of { 0 . 1 , 0 . 3 } .
Revised: We try out learning rates in the set { 10 − 3 , 10 − 4 , 10 − 5 } and drop out rates of { 0 . 1 , 0 . 3 } .

Original: For META-TARTAN since we are now learning the task weights, w , we test out task weight learning rates in { 10 − 1 , 5 × 10 − 2 , 3 × 10 − 2 , 10 − 2 } .
Revised: For META-TARTAN since we are now learning the task weights, w , we test out task weight learning rates in { 10 − 1 , 5 × 10 − 2 , 3 × 10 − 2 , 10 − 2 } .

Original: Note that for all MTTARTAN experiments we use equalized task weights 1 | T aux | +1 .
Revised: Note that for all MTTARTAN experiments we use equalized task weights 1 | T aux | +1 .

Original: A small grid-search over a handful of weight conﬁgurations did not yield signiﬁcant improvement over the uniform task weighting.
Revised: A small grid-search over a handful of weight conﬁgurations did not yield signiﬁcant improvement over the uniform task weighting.

Original: We use the Adam optimizer (Kingma & Ba, 2014) for all experiments.
Revised: We use the Adam optimizer (Kingma & Ba, 2014) for all experiments.

Original: As mentioned in section 3.3, we train as separate meta-classiﬁcation head, φ ∗ , to estimate the validation meta-gradients.
Revised: As mentioned in section 3.3, we train as separate meta-classiﬁcation head, φ ∗ , to estimate the validation meta-gradients.

Original: To estimate φ ∗ , we use batch sizes of { 16 , 32 } samples from T ∗ ’s train set.
Revised: To estimate φ ∗ , we use batch sizes of { 16 , 32 } samples from T ∗ ’s train set.

Original: We regularize the meta-head with l 2 weight decay and set the decay constant to 0 .
Revised: We regularize the meta-head with l 2 weight decay and set the decay constant to 0 .

Original: We use a learning rate 10 − 3 to learn the meta-head.
Revised: We use a learning rate 10 − 3 to learn the meta-head.

Original: We stop training φ ∗ after 10 gradient descent steps.
Revised: We stop training φ ∗ after 10 gradient descent steps.

Original:  R ESULTS AND D ISCUSSION
Revised:  R ESULTS AND D ISCUSSION

Original: In this section, we will discuss the results of comparing our models against DAPT and TAPT baselines.
Revised: In this section, we will discuss the results of comparing our models against DAPT and TAPT baselines.

Original: 1 Broadly, we demonstrate the effectiveness of end-task awareness as improving both performance and data-efﬁciency.
Revised: 2 Broadly, we demonstrate the effectiveness of end-task awareness as improving both performance and data-efﬁciency.

Original: [Table]
Revised: [Table]

Original: 0 . 05 ), is boldfaced.

Original: See A.2,A.5 for more details about this table

Original: E ND - TASK AWARENESS IMPROVES OVER TASK - AGNOSTIC PRE - TRAINING
Revised: E ND - TASK AWARENESS IMPROVES OVER TASK - AGNOSTIC PRE - TRAINING

Original: Table 1 compares TAPT to its end-task aware variants.
Revised: Table 1 compares TAPT to its end-task aware variants.

Original: Just as in Gururangan et al.
Revised: As in Gururangan et al.

Original: (2020), we observe that performing task adaptive pre-training improves upon just ﬁne-tuning RoBERTa.
Revised: (2020), we observe that performing task adaptive pre-training improves upon just ﬁne-tuning RoBERTa.

Original: However, note that introducing the end-task byway of multi-tasking with the TAPT MLM objective leads to a signiﬁcant improvement in performance.
Revised: However, note that introducing the end-task by multi-tasking with the TAPT MLM objective leads to a signiﬁcant improvement in performance.

Original: This improvement is consistent across the 3 tasks we evaluate against.
Revised: This improvement is consistent across the 3 tasks we evaluate against.

Original: We ﬁnd that both MT-TARTAN and META-TARTAN achieve similar results in this setting.
Revised: We ﬁnd that both MT-TARTAN and META-TARTAN achieve similar results in this setting.

Original: In Appendix A.4, we show that the gains from end-task awareness are not restricted to only text data; we observe similar trends for image classiﬁcation.

Original: 5. E ND - TASK AWARENESS IMPROVES DATA - EFFICIENCY
Revised: 5. E ND - TASK AWARENESS IMPROVES DATA - EFFICIENCY

Original: Gururangan et al.
Revised: Gururangan et al.

Original: (2020) train DAPT on large amounts of in-domain datain order to achieve resultsthat are competitive with TAPT.
Revised: (2020) train DAPT on large amounts of in-domain data to achieve results competitive with TAPT.

Original: They use 7.55 billion tokens in the case of the BIOMED domain and 8.10 billiontokens for the CS domain.
Revised: They use 7.55 billion tokens for the BIOMED domain and 8.10 billion for the CS domain.

Original: This is on average over 10 4 × the size of the training data of our end-tasks of interest.
Revised: This is on average over 10 4 × the size of the training data of our end-tasks of interest.

Original: The large amount of data required to train a competitive DAPT model represents a signiﬁcant computational burden to the every-day practitioner.
Revised: The large amount of data required to train a competitive DAPT model represents a signiﬁcant computational burden to the every-day practitioner.

Original: This begets the question: are such large amounts of auxiliary data necessary for achieving good downstream performance?
Revised: This begets the question: are such large amounts of auxiliary data necessary for achieving good downstream performance?

Original: To answer this, we train DAPT and its TARTAN version on variable amounts of data for both SCIERC and ACL-ARC tasks.
Revised: To answer this, we train DAPT and its TARTAN version on variable amounts of data for both SCIERC and ACL-ARC tasks.

Original: [Figure]
Revised: [Figure]

Original: TARTAN is more data-efﬁcient than DAPT In Figure 2, we focus on training on a small fraction of available domain data n = { 10 0 , 10 1 } × | Train | for the DAPT auxiliary task.
Revised: TARTAN is more data-efﬁcient than DAPT In Figure 2, we focus on training on a small fraction of available domain data n = { 10 0 , 10 1 } × | Train | for the DAPT auxiliary task.

Original: Full domain data is n (cid:48)
Revised: Full domain data is n (cid:48)

Original: ≈ 10 4 × | Train | .
Revised: ≈ 10 4 ×| Train | .

Original: This relatively low auxiliary data regime represents a realistic setting that is akin to those encountered by everyday practitioners who are likely to be computationally constrained.
Revised: This relatively low auxiliary data regime represents a realistic setting that is akin to those encountered by everyday practitioners who are likely to be computationally constrained.

Original: As can be seen in Figure 2, on the ACL-ARC task, META-TARTAN matches the performance of DAPT when the sizes of the domain data and end-task data are of the same order ( 10 0 ).
Revised: As can be seen in Figure 2, on the ACL-ARC task, META-TARTAN matches the performance of DAPT when the sizes of the domain data and end-task data are of the same order ( 10 0 ).

Original: At this data size, META-TARTAN supersedes DAPT on the SCIERC task.
Revised: At this data size, META-TARTAN supersedes DAPT on the SCIERC task.

Original: When trained on 10 × more auxiliary data, META-TARTAN supersedes DAPT in performance on both tasks.META-TARTAN establishes a new state-of-the-art result ( 82 . 08 1 . 19 ) on the SCIERC task.
Revised: When trained on 10 × more auxiliary data, META-TARTAN supersedes DAPT in performance on both tasks.

Original: On the ACL-ARC task, META-TARTAN achieves 71 .
Revised: On the ACL-ARC task, METATARTAN achieves 71 .

Original: 19 4 . 88 , which is close to DAPT’s performance of 72 . 49 3 .
Revised: 19 4 . 88 , which is close to DAPT’s performance of 72 . 49 3 .

Original: 28 using more than 10 3 × auxiliary data.
Revised: 28 using more than 10 3 × auxiliary data.

Original: These results indicate that end-task awareness can improve data-efﬁciency and in this case, improvements are on the order of 1000 × . 
Revised: These results indicate that end-task awareness can improve data-efﬁciency and in this case, improvements are on the order of 1000 × . [Figure]

Original: [Table]
Revised: [Table]

Original: Statistically performance 
Revised: performance is See

Original: TARTAN is more data-efﬁcient than DAPT+TAPT Table 2 compares DAPT and DAPT+TAPT (DAPT followed by TAPT) to *-TARTAN which multi-task DAPT, TAPT and the end-task.
Revised: TARTAN is more data-efﬁcient than DAPT+TAPT Table 2 compares DAPT and DAPT+TAPT (DAPT followed by TAPT) to *-TARTAN which multi-task DAPT, TAPT and the end-task.

Original: MTTARTAN and META-TARTAN signiﬁcantly outperform DAPT and DAPT+TAPT in 2 of the tasks whilst giving higher average performance in the ACL-ARC task.
Revised: MTTARTAN and META-TARTAN signiﬁcantly outperform DAPT and DAPT+TAPT in 2 of the tasks whilst giving higher average performance in the ACL-ARC task.

Original: We thus conclude that end-task awareness allows us to get a greater performance boost out of the same amount of data .
Revised: We thus conclude that end-task awareness allows us to get a greater performance boost out of the same amount of data .

Original: We explore the data efﬁciency of TARTAN methods even further by comparing the relatively data-poor versions of MT-TARTAN and META-TARTAN above ( n = 10 × | Train | ) to the DAPT and DAPT+TAPT variants trained on all the available domain data ( n (cid:48)
Revised: We explore the data efﬁciency of TARTAN methods even further by comparing the relatively data-poor versions of MT-TARTAN and META-TARTAN above ( n = 10 × | Train | ) to the DAPT and DAPT+TAPT variants trained on all the available domain data ( n (cid:48)

Original: ≈ 10 4 × | Train | ).
Revised: ≈ 10 4 × | Train | ).

Original: We can see from Table 3 that for the CS domain, our end-task aware variants come close to (ACL-ARC) and even supersede (SCIERC) the end-task agnostic variants though trained with ≈ 1000 × less data.
Revised: We can see from Table 3 that for the CS domain, our end-task aware variants come close to (ACL-ARC) and even supersede (SCIERC) the end-task agnostic variants though trained with ≈ 1000 × less data.

Original: For BIOMED domain (CHEMPROT task), increasing the amount of data drastically improves the performance of end-task agnostic variants compared to MT-TARTAN and META-TARTAN trained on much less data.
Revised: For BIOMED domain (CHEMPROT task), increasing the amount of data drastically improves the performance of end-task agnostic variants compared to MT-TARTAN and META-TARTAN trained on much less data.

Original: [Table]
Revised: [Table]

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020) show that different tasks exhibit sigmoid-like curves in terms of how much pretraining data is required to achieve good results before performance levels off.
Revised: (2020) show that different tasks exhibit sigmoid-like curves in terms of how much pretraining data is required to achieve good results before performance levels off.

Original: We contextualize Tables 2 and 3 within said work and posit that the
Revised: We contextualize Tables 2 and 3 within said work and posit that the

Original: CHEMPROT task intrinsically requires much more data (compared to our other tasks) before performance begins to improve appreciably.
Revised: CHEMPROT task intrinsically requires much more data (compared to our other tasks) before performance begins to improve appreciably.

Original: 5. META-TARTAN MORE EFFECTIVELY UTILIZES OUT - OF 
Revised: 5. META-TARTAN MORE EFFECTIVELY UTILIZES OUT - OF - DISTRIBUTION AUXILIARY DATA OVER MT-TARTAN

Original: [Table] DISTRIBUTION AUXILIARY DATA OVER [Table]

Original: TARTAN
Revised: TARTAN

Original: [Table]
Revised: [Table]

Original: We  n 
Revised: We use n = 10 × | Train | . We report averages acrossrandom seeds.

Original: So far we have seen that leveraging TAPT (Table 1 uses TAPT only, 2 uses TAPT+DAPT) leads MT-TARTAN and
Revised: Best average task performance is bolded. We have seen that leveraging TAPT (Table 1 and 2) leads MT-TARTAN and

Original: META-TARTAN to perform similarly.
Revised: META-TARTAN to perform similarly.

Original: The advantage of learning adaptive weights becomesmore pronounced in the DAPT only setting.
Revised: The advantage of learning adaptive weights becomes pronounced in the DAPT only setting.

Original: Note that whilst

Original:  TAPT uses the end-task’s own training data for masked language modelling, DAPT uses heterogeneous domain data whose impact on the end-task performance is less clear.
Revised: Whilst TAPT uses the end-task’s own training data for masked language modelling, DAPT uses heterogeneous domain data whose impact on the end-task performance is less clear.

Original: Notice from Table 4 that when required to rely solely on domain data for auxiliary tasking, META-TARTAN improves performance over MT-TARTAN.
Revised: Notice from Table 4 that when required to rely solely on domain data for auxiliary tasking, META-TARTAN improves performance over MT-TARTAN.

Original: We attribute META-TARTAN’s improvement over MT-TARTAN on its ability to more ﬂexibly adapt to incoming data of variable utility to the end-task.
Revised: We attribute META-TARTAN’s improvement over MTTARTAN to its ability to more ﬂexibly adapt to incoming data of variable utility to the end-task.

Original: 5. T ASK WEIGHTING STRATEGIES DISCOVERED BY META - LEARNING
Revised: 5. T ASK WEIGHTING STRATEGIES DISCOVERED BY META - LEARNING

Original: [Figure]
Revised: [Figure]

Original:  To illustrate the importance of the separate classiﬁcation head φ ∗ for computing the meta-signal for the task weights (described in Section 3.3), we run META-TARTAN experiments with ACL-ARC as the end-task and DAPT as the auxiliary task.
Revised: auxiliary tasks. To illustrate the importance of the separate classiﬁcation head φ ∗ for computing the meta-signal for the task weights (described in Section 3.3), we run META-TARTAN experiments with ACL-ARC as the end-task and DAPT as the auxiliary task.

Original: We compare using either a separate ( φ ∗ ) or the same ( φ (cid:48) ) classiﬁcation head for calculating the meta-gradient.
Revised: We compare using either a separate ( φ ∗ ) or the same ( φ (cid:48) ) classiﬁcation head for calculating the meta-gradient.

Original: Figure 3 plots the task weightings learned in each setting during training.
Revised: Figure 3 plots the task weightings learned in each setting during training.

Original: We can clearly see that using a separate head counteracts the pathological solution of down-weighting all tasks that are not T ∗ and as a result, improves performance: a delta of 1.7 F1 points in this case .
Revised: We can clearly see that using a separate head counteracts the pathological solution of down-weighting all tasks that are not T ∗ and as a result, improves performance: a delta of 1.7 F1 points in this case .

Original: The strategy discovered by META-TARTAN presents an interesting contrast to classical pre-training: whilst the initial phase of classical pre-training involves  solely the auxiliary task, early in training, META-TARTAN up-weights the auxiliary task but does not fully zero out the end-task.
Revised: The strategy discovered by META-TARTAN presents an interesting contrast to classical pre-training: whilst the initial phase of classical pre-training involves [Figure] solely the auxiliary task, early in training, META-TARTAN up-weights the auxiliary task but does not fully zero out the end-task.

Original: Later in training, we see leveling off of weights instead of railing the end-task to 1 as in classical pre-training.
Revised: Later in training, we see leveling off of weights instead of railing the end-task to 1 as in classical pre-training.

Original: Next, we plot a similar graph for using both DAPT and TAPT across our three tasks in Figure 4.
Revised: Next, we plot a similar graph for using both DAPT and TAPT across our three tasks in Figure 4.

Original: From the ﬁgure, it is apparent that META-TARTAN discovers similar task-weighting strategies[Figure] across different end-tasks.
Revised: From the ﬁgure, it is apparent that META-TARTAN discovers similar task-weighting strategies across different end-tasks.

Original: This suggests that the MLM objective and META-TARTAN’s strategy for learning task weights are generic enough to induce similar behaviours across tasks.
Revised: This suggests that the MLM objective and META-TARTAN’s strategy for learning task weights are generic enough to induce similar behaviours across tasks.

Original: In general,
Revised: In general,

Original: DAPT is signiﬁcantly up-weighted compared to the end-task and TAPT.
Revised: DAPT is signiﬁcantly up-weighted compared to the end-task and TAPT.

Original: Note that the TAPT +
Revised: Note that the TAPT +

Original: ACL-ARC task weights (Figure 4) has the same approximate trajectory as ACL-ARC task weight in
Revised: ACL-ARC task weights (Figure 4) has the same approximate trajectory as ACL-ARC task weight in

Original: Figure 3.
Revised: Figure 3.

Original: It seems important to assign high weight to the task data (Figure 3) but not necessarily all of it needs to go to the actual task loss (Figure 4).
Revised: It seems important to assign high weight to the task data (Figure 3) but not necessarily all of it needs to go to the actual task loss (Figure 4).

Original: We hypothesize that the diversity in the domain data counteracts overﬁtting to the end-task data(all our tasks are low-resource) and results in DAPT being up-weighted.
Revised: We hypothesize that the diversity in the domain data counteracts overﬁtting to the end-task data and results in DAPT being up-weighted.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-task learning can be traced back to seminal work by Caruana (1995), Caruana (1997), and has since been the subject of a ﬂourishing literature, recent surveys of which can be found in Ruder (2017) or Zhang & Yang (2021).
Revised: Multi-task learning can be traced back to seminal work by Caruana (1995), Caruana (1997), and has since been the subject of a ﬂourishing literature, recent surveys of which can be found in Ruder (2017) or Zhang & Yang (2021).

Original: In NLP, while initial work from Collobert & Weston (2008) already showed the beneﬁts of multi-task learning, it has only recently become a central topic in the ﬁeld, with the advent of multi-task benchmarks (Wang et al., 2018b; McCann et al., 2018).
Revised: In NLP, while initial work from Collobert & Weston (2008) already showed the beneﬁts of multi-task learning, it has only recently become a central topic in the ﬁeld, with the advent of multi-task benchmarks (Wang et al., 2018b; McCann et al., 2018).

Original: Pre-training isa variation on the multi-task learning paradigm where a machine learning model is ﬁrst trained on a generic (generally more data-rich) task, before being ﬁne-tuned on an end task.
Revised: Pre-training is where a machine learning model is ﬁrst trained on a generic, data-rich task before being ﬁne-tuned on an end-task.

Original: In NLP this practice dates back to the use of pre-trained word embeddings (Turian et al., 2010; Mikolov et al., 2013) and later pre-trained encoders (Kiros et al., 2015; Dai & Le, 2015).
Revised: In NLP this practice dates back to the use of pre-trained word embeddings (Turian et al., 2010; Mikolov et al., 2013) and later pre-trained encoders (Kiros et al., 2015; Dai & Le, 2015).

Original: Peters et al.
Revised: Peters et al.

Original: and Howard & Ruder (2018) heralded a renaissance of pre-training before BERT (Devlin et al., 2018) and its many offshoots (Liu et al., 2019; Yang et al., 2019; Lewis et al., 2019) cemented it as the de facto standard for modern NLP.
Revised: and Howard & Ruder (2018) heralded a renaissance of pre-training before BERT (Devlin et al., 2018) and its many offshoots (Liu et al., 2019; Yang et al., 2019; Lewis et al., 2019) cemented it as the de facto standard for modern NLP.

Original: As an idea, meta-learning, or “learning to learn”, dates back to early work from Schmidhuber (1995); Thrun (1998).
Revised: Meta-learning dates back to early work from Schmidhuber (1995); Thrun (1998).

Original: More relevant to our workhowever isa modern strand of the literature concerned with gradient-based meta-learning for solving bi-level optimization problems, ﬁrst popularized by Finn et al.
Revised: More relevant to our work is gradient-based meta-learning for solving bi-level optimization problems, ﬁrst popularized by Finn et al.

Original: and followup work (Nichol et al., 2018; Rajeswaran et al., 2019) for few-shot learning.
Revised: and followup work (Nichol et al., 2018; Rajeswaran et al., 2019) for few-shot learning.

Original: This method has transferred to a variety of applications such as architecture search (Liu et al., 2018), multilingual data weighting (Wang et al., 2020) and model poisoning (Kurita et al., 2020).
Revised: This method has transferred to a variety of applications such as architecture search (Liu et al., 2018) and model poisoning (Kurita et al., 2020).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we advocate for a paradigm shift in the way we approach pre-training.
Revised: We have advocated for a paradigm shift in the way we approach pre-training.

Original: We have motivated making pre-training more end-task aware when the end task is known in advance.
Revised: We have motivated making pre-training more end-task aware when the end-task is known in advance.

Original: Our work introduced two novel end-task aware training algorithms: End-task Aware Training via Multitasking (MT-TARTAN) and End-task Aware Training via Meta-learning (META-TARTAN).
Revised: Our work introduced two novel end-task aware training algorithms: End-task Aware Training via Multitasking (MT-TARTAN) and End-task Aware Training via Meta-learning (META-TARTAN).

Original: In Section 5,through experiments on 3 different datasets, we demonstrated the ability of our proposed algorithms to improve performance and data-efﬁciency over their end-task agnostic counterparts.
Revised: In Section 5, we demonstrated the ability of our proposed algorithms to improve performance and dataefﬁciency over their end-task agnostic counterparts.

Original: This work suggests several promising directions for future work.
Revised: This work suggests several promising directions for future work.

Original: First, it would be interesting to see if end-task aware training of generalist models like BERT and RoBERTa from scratch result in improved accuracy and efﬁciency; both in terms of data and computation.

Original: Also, instead of learning coarse task level weights, can further performance improvements be achieved via ﬁner-grained example level weighting as in Wang et al.
Revised: Instead of learning coarse task level weights, can further performance improvements be achieved via ﬁner-grained example level weighting as in Wang et al.

Original: Can meta-learning algorithms like METATARTAN enable more effective utilization of previously discarded (Aroca-Ouellette & Rudzicz, 2020) pre-training auxiliary tasks like Next Sentence Prediction (NSP) (Devlin et al., 2018)?
Revised: Can meta-learning algorithms like META-TARTAN enable more effective utilization of previously discarded (Aroca-Ouellette & Rudzicz, 2020) pre-training auxiliary tasks like Next Sentence Prediction (NSP) (Devlin et al., 2018)?

Original: We hope this work spurs conversation around these questions and many more.
Revised: We hope this work spurs conversation around these questions and many more.

Original:  E THICS S TATEMENT
Revised:  A CKNOWLEDGEMENTS

Revised: This work was supported in part by DSO National Laboratories, an ENS-CFM Data Science

Revised: Chair, DARPA FA875017C0141, the National Science Foundation grants IIS1705121, IIS1838017,

Revised: IIS2046613 and IIS-2112471, an Amazon Web Services Award, a Facebook Faculty Research

Revised: Award, funding from Booz Allen Hamilton Inc., and a Block Center Grant.

Revised: Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of any of these funding agencies.

Revised:  E THICS S TATEMENT

Original: Our work introduces new algorithms but leverages pre-existing datasets and models.
Revised: Our work introduces new algorithms but leverages pre-existing datasets and models.

Original: Overall, this work inherits some of the risk of original work upon which it is implemented.
Revised: Overall, this work inherits some of the risk of original work upon which it is implemented.

Original: Algorithms for continued training such as TAPT and DAPT necessitate per-task training of unsupervised objectives which result in corresponding green-house emissions due to energy consumption (Strubell et al., 2019).
Revised: Algorithms for continued training such as TAPT and DAPT necessitate per-task training of unsupervised objectives which result in corresponding green-house emissions due to energy consumption (Strubell et al., 2019).

Original: However, as shown in Sections 3 and 5, our new compute-efﬁcient algorithms greatly increase the data efﬁciency of these algorithms, reducing these harms as well as the various harms associated with labor for data-collection (Jo & Gebru, 2020).
Revised: However, as shown in Sections 3 and 5, our new compute-efﬁcient algorithms greatly increase the data efﬁciency of these algorithms, reducing these harms as well as the various harms associated with labor for data-collection (Jo & Gebru, 2020).

Original: Also, since our work is set in the context of pre-existing datasets and models (Section 4), we recognize that any ethical issues that have been revealed in these (such as bias (Bender et al., 2021) or privacy leakage (Carlini et al., 2021)) may also propagate to models trained using our work, and mitigation strategies such as Schick et al.
Revised: Also, since our work is set in the context of pre-existing datasets and models (Section 4), we recognize that any ethical issues that have been revealed in these (such as bias (Bender et al., 2021) or privacy leakage (Carlini et al., 2021)) may also propagate to models trained using our work, and mitigation strategies such as Schick et al.

Original: Liang et al.
Revised: Liang et al.

Original: (2021) may be necessary.
Revised: (2021) may be necessary.

Original: Finally, there is a potential risk in META-TARTAN that leveraging a validation set for deﬁning the meta-objective could amplifying bias that exists in this data split, although this is done indirectly through task weighting and hence we believe that this risk is small.
Revised: Finally, there is a potential risk in META-TARTAN that leveraging a validation set for deﬁning the meta-objective could amplifying bias that exists in this data split, although this is done indirectly through task weighting and hence we believe that this risk is small.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We pledge to release the source-code for this project to improve the ease of reproducibility of our results by the NLP and machine learning communities.
Revised: We pledge to release the source-code for this project to improve the ease of reproducibility of our results by the NLP and machine learning communities.

Original: In Section 4, we have speciﬁed details about datasets, training regimes and models to allow anyone who wishes to reproduce our results without our original source code to do so.
Revised: In Section 4, we have speciﬁed details about datasets, training regimes and models to allow anyone who wishes to reproduce our results without our original source code to do so.

Original: Our discussion of the algorithmic and evaluation details can be found in Appendices A.1, A.3 and A.2.
Revised: Our discussion of the algorithmic and evaluation details can be found in Appendices A.1, A.3 and A.2.

Original: As we noted in 4, we build off of Gururangan et al.
Revised: As we noted in 4, we build off of Gururangan et al.

Original: (2020)’s implementations which can be found at https://github.com/allenai/dont-stop-pretraining.
Revised: (2020)’s implementations which can be found at https://github.com/allenai/dont-stop-pretraining.

Original: A A PPENDIX A. J USTIFYING THE INTRODUCTION OF A M ETA -H EAD
Revised: A. J USTIFYING THE INTRODUCTION OF A M ETA -H EAD

Original: Proof.
Revised: Proof.

Original: To arrive at Equation 7 we start with the closed form solution for ∇ w i L valT ∗ ( θ ∗ ) and then introduce approximations in order to produce Equation 7.
Revised: To arrive at Equation 7 we start with the closed form solution for ∇ w i L valT ∗ ( θ ∗ ) and then introduce approximations in order to produce Equation 7.

Original: First, note that :
Revised: First, note that :

Original: [Equation]
Revised: [Equation]

Original: To get ∇ w i θ ∗ ( w ) we invoke the Cauchy Implicit Function Theorem (IFT) as with Lorraine et al.
Revised: To get ∇ w i θ ∗ ( w ) we invoke the Cauchy Implicit Function Theorem (IFT) as with Lorraine et al.

Original: (2020); Navon et al.
Revised: (2020); Navon et al.

Original: ; Liao et al.
Revised: ; Liao et al.

Original: [Equation]
Revised: [Equation]

Original: Only terms with w i survive
Revised: Only terms with w i survive

Original: [Equation]
Revised: [Equation]

Original: Bringing it all together
Revised: Bringing it all together

Original: [Equation]
Revised: [Equation]

Original: we get :
Revised: we get :

Original: [Equation]
Revised: [Equation]

Original: Computing ∇ w i L valT ∗ ( θ ∗ ) from Equation 9 is computationally unwieldy since we would not only have to optimize θ to convergence for every step of w i
Revised: Computing ∇ w i L valT ∗ ( θ ∗ ) from Equation 9 is computationally unwieldy since we would not only have to optimize θ to convergence for every step of w i

Original: but we would also have to invert the Hessian of a typically large model.
Revised: but we would also have to invert the Hessian of a typically large model.

Original: Our middle ground between Equations 9 and 6 (Equation 7) makes use of the following approximations:
Revised: Our middle ground between Equations 9 and 6 (Equation 7) makes use of the following approximations:

Original: • We approximate the inverse Hessian with the identity.
Revised: • We approximate the inverse Hessian with the identity.

Original: This approximation is not new; we follow previous work like Lorraine et al.
Revised: This approximation is not new; we follow previous work like Lorraine et al.

Original: (2020)(Table 3) who explore the use of this approximation because of computational efﬁciency.
Revised: (2020)(Table 3) who explore the use of this approximation because of computational efﬁciency.

Original: [Equation]
Revised: [Equation]

Original: We are assuming the contribution of terms with i > 0 are negligible.
Revised: We are assuming the contribution of terms with i > 0 are negligible.

Original: • Instead of training the whole network to convergence, at each time-step, we ﬁx the body of the network and train a special head φ ∗ to convergence on a small batch of end-task training data.
Revised: • Instead of training the whole network to convergence, at each time-step, we ﬁx the body of the network and train a special head φ ∗ to convergence on a small batch of end-task training data.

Original: We then use [ θ body ; φ ∗ ] as a proxy for θ ∗ .
Revised: We then use [ θ body ; φ ∗ ] as a proxy for θ ∗ .

Original: This is a computationally feasible workaround to training all of θ to convergence to get a single step gradient estimate.
Revised: This is a computationally feasible workaround to training all of θ to convergence to get a single step gradient estimate.

Original: Especially in the continued pre-training setting where a pre-trained generalist model like BERT is used as θ body , this approximation is reasonable.
Revised: Especially in the continued pre-training setting where a pre-trained generalist model like BERT is used as θ body , this approximation is reasonable.

Original: To our knowledge, we are the ﬁrst to suggest this approximation.
Revised: To our knowledge, we are the ﬁrst to suggest this approximation.

Original: [Equation]
Revised: [Equation]

Original: • Above, we have approximated θ ∗ =
Revised: • Above, we have approximated θ ∗ =

Original: [ θ body ; φ ∗ ] .
Revised: [ θ body ; φ ∗ ] .

Original: Since φ ∗ is only used to evaluate end-task ( T ∗ ) validation data, it means θ remains unchanged with respect to the training data for task T i .
Revised: Since φ ∗ is only used to evaluate end-task ( T ∗ ) validation data, it means θ remains unchanged with respect to the training data for task T i .

Original: Thus ∇ θ L T i ([ θ body ; (cid:0) φ ∗ , . .
Revised: Thus ∇ θ L T i ([ θ body ; (cid:0) φ ∗ , . .

Original:  , φ i 
Revised: . , φ i (cid:1)

Original: [Equation]
Revised: [Equation]

Original: Bringing it all together
Revised: Bringing it all together

Original: [Equation]
Revised: [Equation]

Original: we get Equation 7, repeated here:
Revised: we get Equation 7, repeated here:

Original: [Equation]
Revised: [Equation]

Original: A. C ALCULATING P - VALUES FROM P ERMUTATION T EST
Revised: A. C ALCULATING P - VALUES FROM P ERMUTATION T EST

Original: We used the permutation test (Good, 2005; Dror et al., 2018) to test for statistical signiﬁcance.
Revised: We used the permutation test (Good, 2005; Dror et al., 2018) to test for statistical signiﬁcance.

Original: For each test, we generate 10000 permutations to calculate signiﬁcance level.
Revised: For each test, we generate 10000 permutations to calculate signiﬁcance level.

Original: This is sufﬁcient to converge to a stable p-value without being a computational burden.
Revised: This is sufﬁcient to converge to a stable p-value without being a computational burden.

Original: We chose this over the common student t-test because :
Revised: We chose this over the common student t-test because :

Original: We have only 10 runs per algorithm and permutation tests are more robust at low sample size 2.
Revised: We have only 10 runs per algorithm and permutation tests are more robust at low sample size 2.

Original: Permutation test is assumption free.
Revised: Permutation test is assumption free.

Original: Student t-tests assume that the samples are normally distributed 3.
Revised: Student t-tests assume that the samples are normally distributed 3.

Original: Permutation test is robust to variance in the samples, so even though error-bars can overlap, we still establish signiﬁcant differences in the samples.
Revised: Permutation test is robust to variance in the samples, so even though error-bars can overlap, we still establish signiﬁcant differences in the samples.

Original: Variance in our results is expected due to small dataset sizes of end-tasks.
Revised: Variance in our results is expected due to small dataset sizes of end-tasks.

Original: A. A LGORITHM FOR META-TARTAN
Revised: A. A LGORITHM FOR META-TARTAN

Revised: Algorithm

Original: Require: T ∗ , T aux : End-task, Set of auxiliary pre-training tasks
Revised: Require: T ∗ , T aux : End-task, Set of auxiliary pre-training tasks

Original: Require: η, β 1 , β 2 : Step size hyper-parameters
Revised: Require: η, β 1 , β 2 : Step size hyper-parameters

Original: Initialize :
Revised: Initialize :

Original: Pre-trained RoBERTa as shared network body, θ body
Revised: Pre-trained RoBERTa as shared network body, θ body

Original: Task weightings: w ∗ , w i = 1 | T aux | +1 Randomly initialize : end-task head as φ (cid:48) meta head for end-task as φ ∗ task head, φ i , for each T i ∈ T aux while not done do
Revised: Task weightings: w ∗ , w i = 1 | T aux | +1 Randomly initialize : end-task head as φ (cid:48) meta head for end-task as φ ∗ task head, φ i , for each T i ∈ T aux while not done do

Original: B ∗ tr ∼ T ∗ train //
Revised: B ∗ tr ∼ T ∗ train //

Original: Sample a batch from end-task 
Revised: Sample a batch from end-task g

Original: [Table]
Revised: [Table]

Revised: g

Original: [Table]
Revised: [Table]

Original:  n ] , B i ∼ T i // Learn a new meta head φ ∗ ← estimate meta head ( B ∗ tr , β 2 , θ, φ ∗ ) //
Revised: [Table] (cid:19) / [Table] ∀ i ∈ [ n ] , B i ∼ T i // Learn a new meta head φ ∗ ← estimate meta head ( B ∗ tr , β 2 , θ, φ ∗ ) //

Original: B ∗ tr ∼ T ∗ train g ∗ meta ← ∇ θ L T ∗ ( θ, φ ∗ , B ∗ val ) //
Revised: B ∗ tr ∼ T ∗ train g ∗ meta ← ∇ θ L T ∗ ( θ, φ ∗ , B ∗ val ) //

Original: B ∗ val ∼ T ∗ val // Update task weightings w ∗ ← w ∗ + η cos( g ∗ meta , g ∗ θ ) w i ← w i + η cos( g ∗ meta , g i θ ) // Update task parameters α ∗ , α 1 , . .
Revised: B ∗ val ∼ T ∗ val // Update task weightings w ∗ ← w ∗ + η cos( g ∗ meta , g ∗ θ ) w i ← w i + η cos( g ∗ meta , g i θ ) // Update task parameters α ∗ , α 1 , . .

Original: . , α | T aux | = softmax( w ∗ , w 1 , . .
Revised: . , α | T aux | = softmax( w ∗ , w 1 , . .

Original: , w | T aux | )
Revised: , w | T aux | )

Original: Update
Revised: Update

Original: [Table]
Revised: [Table]

Original: +
Revised: endResult :

Original: [Table]

Original: [Table]
Revised: [Table]

Original: endResult : θ, φ

Original: [Table]
Revised: [Table]

Original: A. V ISION E XPERIMENTS
Revised: A. V ISION E XPERIMENTS

Original: We validate that the gains from end-task Aware Training are not siloed to only learning from text.
Revised: We validate that the gains from end-task Aware Training are not siloed to only learning from text.

Original: We conduct an experiment comparing end-task aware training on images to its end-task agnostic variant.
Revised: We conduct an experiment comparing end-task aware training on images to its end-task agnostic variant.

Original: We use the Cifar100 dataset (Krizhevsky et al., 2009).
Revised: We use the Cifar100 dataset (Krizhevsky et al., 2009).

Original: We use the Medium-Sized Mammals superclass (one of the 20 coarse labels) as our main task whilst the other 19 super classes are used as auxiliary data.
Revised: We use the Medium-Sized Mammals superclass (one of the 20 coarse labels) as our main task whilst the other 19 super classes are used as auxiliary data.

Original: Our primary task is thus a 5-way classiﬁcation task of images different types of
Revised: Our primary task is thus a 5-way classiﬁcation task of images different types of

Original: [Table]
Revised: [Table]

Original: As can be seen from Table [Table] being end-task aware improves over task agnostic pre-training.
Revised: As can be seen from Table 5, being end-task aware improves over task agnostic pre-training.

Original: We ﬁnd that, again, when our auxiliary task consist of solely domain data and no task data, METATARTAN performs better than MT-TARTAN (as measured by averaged performance).
Revised: We ﬁnd that, again, when our auxiliary task consist of solely domain data and no task data, METATARTAN performs better than MT-TARTAN (as measured by averaged performance).

Original: A. F ULL TAPT T ABLE WITH S IGNIFICANCE LEVELS
Revised: A. F ULL TAPT T ABLE WITH S IGNIFICANCE LEVELS

Original: We repeat Table 1 and provide details about levels of statistical signifance. 
Revised: We repeat Table 1 and provide details about levels of statistical signifance. [Table]

Original: [Table]
Revised: [Table]

Original: A. F ULL DAPT/DAPT+TAPT T ABLE
Revised: A. F ULL DAPT/DAPT+TAPT T ABLE

Original: We repeat Table 3 and provide details about levels of statistical signifance.
Revised: We repeat Table 3 and provide details about levels of statistical signifance.

Original: [Table]
Revised: [Table]

Revised: A. FAQ

Revised: What settings are TARTAN algorithms designed for?

Revised: TARTAN algorithms specialize auxiliary objectives to a particular end-task.

Revised: This comes at a risk of losing the generic representations afforded by generalist pre-trained models.

Revised: Thus if a practitioner has a sufﬁciently important end-task where obtaining improved end-task performance is paramount over generic representations, then TARTAN is a viable option.

Revised: When do we get computational savings from META-TARTAN?

Revised: MT-TARTAN does not add any extra overhead compared to pre-train then ﬁne-tune approaches.

Revised: META-TARTAN however, adds extra overhead per gradient descent step due to computing meta-gradients.

Revised: However, as shown in Section 5 we are able to get several orders of magnitude improvement in data-efﬁciency from applying the method.

Revised: In general,

Revised: for the tasks we experimented with, we ﬁnd that the savings in data-efﬁciency superseded the extra per-timestep meta-learning overhead.

Revised: When should we use META-TARTAN over MT-TARTAN?

Revised: In +TAPT settings (Tables 1, 3), we observe that META-TARTAN and MT-TARTAN perform similarly.

Revised: We attribute this to the strength of TAPT-MLM objective.

Revised: We were pleasantly surprised that the two methods performed comparatively in this setting but in hindsight, we appreciate the insight that went into designing TAPT-MLM as an objective which makes it a strong baseline.

Revised: In other settings with less carefully designed auxiliary objectives and data (which can potentially be detrimental to the end-task) we expect METATARTAN to perform better.

Revised: Section 5.3 provides evidence of this.