Original: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks
Revised: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

Original: Anonymous ACL-IJCNLP submission
Revised: DeepMind ruder google

Original: Abstract
Revised: Abstract

Original: State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model.
Revised: State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model.

Original: However, such modules are trained separately for each task and thus do not enable sharing information across tasks.
Revised: However, such modules are trained separately for each task and thus do not enable sharing information across tasks.

Original: In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position and layer id in a transformer model.
Revised: In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model.

Original: This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters.
Revised: This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters.

Original: Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0 .
Revised: Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0 .

Original: 29% parameters per task.
Revised: 29% parameters per task.

Original: We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks.
Revised: We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks.

Revised: Our code is publicly available in https://github.com/ rabeehk/hyperformer .

Original:  Introduction
Revised:  Introduction

Original: Transfer learning from pretrained large-scale language models yields state-of-the-art results in a variety of tasks (Devlin et al., 2019;
Revised: Transfer learning from pretrained large-scale language models yields state-of-the-art results in a variety of tasks (Devlin et al., 2019;

Original: Radford et al., 2018; Liu et al., 2019b).
Revised: Radford et al., 2018; Liu et al., 2019b).

Original: As a highly expressive and abstract framework, Raffel et al.
Revised: As a highly expressive and abstract framework, Raffel et al.

Original: (2020) explored the landscape of transfer learning by converting text-based natural language processing (NLP) problems into a sequence-to-sequence format to train a unified model on several tasks simultaneously.
Revised: (2020) explored the landscape of transfer learning by converting text-based natural language processing (NLP) problems into a sequence-to-sequence format to train a unified model on several tasks simultaneously.

Original: Multi-task learning with pretrained language models (Ruder, 2017) is appealing for multiple reasons: 1) Training individual models per task results in higher computational costs, which hinders deployment and maintenance.
Revised: Multi-task learning with pretrained language models (Ruder, 2017) is appealing for multiple reasons: 1) Training individual models per task results in higher computational costs, which hinders deployment and maintenance.

Original: These costs are substantially reduced by training a singlemodel.
Revised: These costs are substantially reduced by training a single

Original: 2) Fine-tuning the model across multiple tasks allows sharing information between the different tasks and positive transfer to other related tasks.

Original: Specifically, when target datasets have limited training

Original: [Figure]
Revised: [Figure]

Original: Following Houlsby et al.
Revised: Following Houlsby et al.

Original: (2019), we include adapter modules after the two feed-forward layers.
Revised: (2019), we include adapter modules after the two feed-forward layers.

Original: The Adapter hypernetwork h lA produces the weights ( U lτ and D lτ ) for task-specific adapter modules conditioned on an input task embedding I τ .
Revised: The Adapter hypernetwork h lA produces the weights ( U lτ and D lτ ) for task-specific adapter modules conditioned on an input task embedding I τ .

Original: Similarly, the layer normalization hypernetwork h lLN generates the conditional layer normalization parameters ( β τ and γ τ ).
Revised: Similarly, the layer normalization hypernetwork h lLN generates the conditional layer normalization parameters ( β τ and γ τ ).

Original: During training, we only update layer normalizations in T5, hypernetworks, and task embeddings.
Revised: During training, we only update layer normalizations in T5, hypernetworks, and task embeddings.

Original: The compact H YPER F ORMER ++ sharesdata, multi-task learning improves the performance compared to individually trained models (Liu et al., 2019a;
Revised: The compact H YPER F ORMER ++ shares the same hypernetworks across all layers and tasks and computes the task embedding based on task, layer id, and position of the adapter module (§2.4).

Original:  Ratner et al., 2018).
Revised: model. 2) Fine-tuning the model across multiple tasks allows sharing information between the different tasks and positive transfer to other related tasks. Specifically, when target datasets have limited training data, multi-task learning improves the performance compared to individually trained models (Liu et al., 2019a; Ratner et al., 2018).

Original: However, multi-task fine-tuning can result in models underperforming on high-resource tasks due to constrained capacity (Arivazhagan et al., 2019; McCann et al., 2018).
Revised: However, multi-task fine-tuning can result in models underperforming on high-resource tasks due to constrained capacity (Arivazhagan et al., 2019; McCann et al., 2018).

Original: An additional issue with multi-task fine-tuning is the potential for task interference or negative transfer , where achieving good performance on one task can hinder performance on another (Wang et al., 2019c).
Revised: An additional issue with multi-task fine-tuning is the potential for task interference or negative transfer , where achieving good performance on one task can hinder performance on another (Wang et al., 2019c).

Original: As an alternative to fine-tuning (Howard and Ruder, 2018), adapter layers (Houlsby et al., 2019) insert a small number of additional parameters per task into the model.
Revised: As an alternative to fine-tuning (Howard and Ruder, 2018), adapter layers (Houlsby et al., 2019) insert a small number of additional parameters per task into the model.

Original: During fine-tuning, only the adapter modules, layer normalizations, and parameters of the final classification layer are updated, while the original pretrained model parameters remain frozen.
Revised: During fine-tuning, only the adapter modules, layer normalizations, and parameters of the final classification layer are updated, while the original pretrained model parameters remain frozen.

Original: Such task-specific adapters eliminate negative task interference by encapsulating task-specific information (Pfeiffer et al., 2020).
Revised: Such task-specific adapters eliminate negative task interference by encapsulating task-specific information (Pfeiffer et al., 2020).

Original: However, so far there has not been an effective and parameter-efficient way to share information across multiple adapters to enable positive transfer to low-resource and related tasks.
Revised: However, so far there has not been an effective and parameter-efficient way to share information across multiple adapters to enable positive transfer to low-resource and related tasks.

Original: To address this problem and to enable sharing information across tasks while reaping the benefits of adapter layers, as depicted in Figure 1, we propose
Revised: To address this problem and to enable sharing information across tasks while reaping the benefits of adapter layers, as depicted in Figure 1, we propose

Original: H YPER F ORMER ++ , which employs a compact hypernetwork (Ha et al., 2017; Oswald et al., 2020) shared across tasks and layers.
Revised: H YPER F ORMER ++ , which employs a compact hypernetwork (Ha et al., 2017; Oswald et al., 2020) shared across tasks and layers.

Original: The hypernetwork learns to generate task and layer-specific adapter parameters, conditioned on task and layer id embeddings.
Revised: The hypernetwork learns to generate task and layer-specific adapter parameters, conditioned on task and layer id embeddings.

Original: The hypernetwork is jointly learned between all tasks and is thus able to share information across them, while negative interference is minimized by generating separate adapter layers for each task.
Revised: The hypernetwork is jointly learned between all tasks and is thus able to share information across them, while negative interference is minimized by generating separate adapter layers for each task.

Original: For each new task, our model only requires learning an additional task embedding, reducing the number of trained parameters.
Revised: For each new task, our model only requires learning an additional task embedding, reducing the number of trained parameters.

Original: We use the encoder-decoder T5 model (Raffel et al., 2020) as the underlying model for our experiments and evaluate on the standard GLUE benchmark (Wang et al., 2019b).
Revised: We use the encoder-decoder T5 model (Raffel et al., 2020) as the underlying model for our experiments and evaluate on the standard GLUE benchmark (Wang et al., 2019b).

Original: We achieve strong gains over both the T5 BASE model as well as adapters (Houlsby et al., 2019).
Revised: We achieve strong gains over both the T5 BASE model as well as adapters (Houlsby et al., 2019).

Original: To our knowledge, this is the first time that adapters have been successfully integrated into a stateof-the-art encoder-decoder model beyond machine translation (Bapna and Firat, 2019), demonstrating that our method effectively balances sharing information across tasks while minimizing negative transfer.
Revised: To our knowledge, this is the first time that adapters have been successfully integrated into a stateof-the-art encoder-decoder model beyond machine translation (Bapna and Firat, 2019), demonstrating that our method effectively balances sharing information across tasks while minimizing negative transfer.

Original: In summary, we make the following contributions: (1) We propose a parameter-efficient method for multitask fine-tuning based on hypernetworks and adapter layers.
Revised: In summary, we make the following contributions: (1) We propose a parameter-efficient method for multitask fine-tuning based on hypernetworks and adapter layers.

Original: (2) We demonstrate that our method scales more efficiently than prior work.
Revised: (2) We demonstrate that our method scales more efficiently than prior work.

Original: (3) We provide empirical results on GLUE demonstrating the effectiveness of the proposed method on multi-task learning.
Revised: (3) We provide empirical results on GLUE demonstrating the effectiveness of the proposed method on multi-task learning.

Original: ) We perform extensive few-shot domain transfer experiments, which reveal that the captured shared knowledge can positively transfer to unseen in-domain tasks.
Revised: (4) We perform extensive few-shot domain transfer experiments, which reveal that the captured shared knowledge can positively transfer to unseen in-domain tasks.

Original: We release our code to facilitate future work.
Revised: We release our code to facilitate future work.

Original: H YPER F ORMER
Revised: H YPER F ORMER

Original: In this section, we present our H YPER F ORMER model, which integrates hyper network-based adapter layers into a multi-task trans former model.
Revised: In this section, we present our H YPER F ORMER model, which integrates hyper network-based adapter layers into a multi-task trans former model.

Original: In §2.4, we introduce a parameter-efficient variant of this model, called H YPER F ORMER ++ .
Revised: In §2.4, we introduce a parameter-efficient variant of this model, called H YPER F ORMER ++ .

Original: Problem formulation:
Revised: Problem formulation:

Original: We consider a general multi-task learning problem, where we are given the data from a set of tasks {D τ } Tτ =1 , where T is the total number of tasks and D τ = { ( x iτ ,y iτ ) }
Revised: We consider a general multi-task learning problem, where we are given the data from a set of tasks {D τ } Tτ =1 , where T is the total number of tasks and D τ = { ( x iτ ,y iτ ) }

Original: N τ i =1 shows the training data for τ -th task with N τ samples.
Revised: N τ i =1 shows the training data for τ -th task with N τ samples.

Original: We assume we are also given a large-scale pretrained language model f θ ( . ) parameterized by θ that computes the output for input x iτ .
Revised: We assume we are also given a large-scale pretrained language model f θ ( . ) parameterized by θ that computes the output for input x iτ .

Original: Standard multi-task fine-tuning minimizes the following loss on the training set:
Revised: Standard multi-task fine-tuning minimizes the following loss on the training set:

Original: [Equation]
Revised: [Equation]

Original: where l is typically the cross-entropy loss, and w τ shows the sampling weight for τ -th task.
Revised: where l is typically the cross-entropy loss, and w τ shows the sampling weight for τ -th task.

Original: Our goal is to finetune the pretrained model in a multi-task learning setup efficiently, while allowing sharing information across tasks and at the same time, enabling the model to adapt to each individual task.
Revised: Our goal is to finetune the pretrained model in a multi-task learning setup efficiently, while allowing sharing information across tasks and at the same time, enabling the model to adapt to each individual task.

Original: The key idea of our approach, depicted in Figure 1, is to learn a parametric task embedding { I τ } Tτ = for each task, and then feed these task embeddings to hypernetworks parameterized by ν that generate the task-specific adapter layers (Houlsby et al., 2019).
Revised: The key idea of our approach, depicted in Figure 1, is to learn a parametric task embedding { I τ } Tτ = for each task, and then feed these task embeddings to hypernetworks parameterized by ν that generate the task-specific adapter layers (Houlsby et al., 2019).

Original: We insert adapter modules within the layers of a pretrained model, making the final model of
Revised: We insert adapter modules within the layers of a pretrained model, making the final model of

Original: X ν ( x iτ , θ , I τ ) parameterized by ν that computes the output for input x iτ .
Revised: X ν ( x iτ , θ , I τ ) parameterized by ν that computes the output for input x iτ .

Original: During training, we only train hypernetwork parameters ν , task embeddings { I τ } Tτ =1 , and layer normalizations in f θ ( . ) , while the rest of the pretrained model parameters θ are fixed:
Revised: During training, we only train hypernetwork parameters ν , task embeddings { I τ } Tτ =1 , and layer normalizations in f θ ( . ) , while the rest of the pretrained model parameters θ are fixed:

Original: [Equation]
Revised: [Equation]

Original: The hypernetworks capture the shared information across tasks in a multi-task learning model enabling positive transfer between related domains and transferable tasks, while adapters are reducing negative interference, encapsulating task-specific information.
Revised: The hypernetworks capture the shared information across tasks in a multi-task learning model enabling positive transfer between related domains and transferable tasks, while adapters are reducing negative interference, encapsulating task-specific information.

Original: Base model:
Revised: Base model:

Original: All of our models are built on top of the state-of-the-art T5 transformer model (Raffel et al., 2020).
Revised: All of our models are built on top of the state-of-the-art T5 transformer model (Raffel et al., 2020).

Original: This model frames text-based language tasks as sequence-to-sequence problems.
Revised: This model frames text-based language tasks as sequence-to-sequence problems.

Original: T5 consists of an encoder-decoder Transformer (Vaswani et al., 2017) with minor modifications (Raffel et al., 2020).
Revised: T5 consists of an encoder-decoder Transformer (Vaswani et al., 2017) with minor modifications (Raffel et al., 2020).

Original: The model is trained simultaneously on multiple tasks, obtaining state-of-the-art performance across a diverse set of tasks.
Revised: The model is trained simultaneously on multiple tasks, obtaining state-of-the-art performance across a diverse set of tasks.

Original: We use the T5 framework as it enables training a universal model that interfaces with many language tasks.
Revised: We use the T5 framework as it enables training a universal model that interfaces with many language tasks.

Original: Our model has three main components: 1) task conditional adapter layers; 2) task conditional layer normalizations ; and 3) hypernetworks that generate task-specific parameters.
Revised: Our model has three main components: 1) task conditional adapter layers; 2) task conditional layer normalizations; and 3) hypernetworks that generate task-specific parameters.

Original: We next describe these components.
Revised: We next describe these components.

Original: 2. Task Conditional Adapter Layers
Revised: 2. Task Conditional Adapter Layers

Original: Prior work has shown that fine-tuning all parameters of the model can result in a sub-optimal solution, particularly for resource-limited datasets (Peters et al., 2019).
Revised: Prior work has shown that fine-tuning all parameters of the model can result in a sub-optimal solution, particularly for resource-limited datasets (Peters et al., 2019).

Original: As an alternative to fine-tuning all the model’s parameters, prior work (Houlsby et al., 2019; Rebuffi et al., 2018; Stickland and Murray, 2019) inserted small modules called adapter layers within layers of a pretrained model, as shown in Figure 1.
Revised: As an alternative to fine-tuning all the model’s parameters, prior work (Houlsby et al., 2019; Rebuffi et al., 2018; Stickland and Murray, 2019) inserted small modules called adapter layers within layers of a pretrained model, as shown in Figure 1.

Original: Adapters introduce no change to the structure or parameters of the original model.
Revised: Adapters introduce no change to the structure or parameters of the original model.

Original: In this work, we propose conditional adapter modules, in which we generate the adapters weights based on input task embeddings using shared hypernetworks (Ha et al., 2017), which capture information across tasks that can be used to positively transfer to other relevant tasks.
Revised: In this work, we propose conditional adapter modules, in which we generate the adapters weights based on input task embeddings using shared hypernetworks (Ha et al., 2017), which capture information across tasks that can be used to positively transfer to other relevant tasks.

Original: Each layer of a transformer model consists of an attention block and a feed-forward block, each followed by a skip connection.
Revised: Each layer of a transformer model consists of an attention block and a feed-forward block, each followed by a skip connection.

Original: Following Houlsby et al.
Revised: Following Houlsby et al.

Original: (2019), as depicted in Figure 1, we introduce a conditional adapter layer after each block before the skip connection.
Revised: (2019), as depicted in Figure 1, we introduce a conditional adapter layer after each block before the skip connection.

Original: The conditional adapter layer A lτ for layer l consists of a down-projection, D lτ ∈ R h × d ,
Revised: The conditional adapter layer A lτ for layer l consists of a down-projection, D lτ ∈ R h × d ,

Original: GeLU non-linearity
Revised: GeLU non-linearity

Original: Hendrycks and Gimpel, 2016), and up-projection U lτ ∈ R d × h , where h is the input dimension, and d is the bottleneck dimension for the adapter layer, mathematically defined as:
Revised: Hendrycks and Gimpel, 2016), and up-projection U lτ ∈ R d × h , where h is the input dimension, and d is the bottleneck dimension for the adapter layer, mathematically defined as:

Original: [Equation]
Revised: [Equation]

Original: where x is the input hidden state and LN lτ is the conditional layer norm defined in the next section.
Revised: where x is the input hidden state and LN lτ is the conditional layer norm defined in the next section.

Original: We generate adapter weights ( U lτ , D lτ ) through a hypernetwork described in §2.3.
Revised: We generate adapter weights ( U lτ , D lτ ) through a hypernetwork described in §2.3.

Original: 2. Task Conditional Layer Normalization
Revised: 2. Task Conditional Layer Normalization

Original: Conventional layer normalization (Ba et al., 2016) is defined as:
Revised: Conventional layer normalization (Ba et al., 2016) is defined as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is the element-wise multiplication between two vectors, and γ lτ and β lτ are learnable parameters with the same dimension as x iτ .
Revised: where (cid:12) is the element-wise multiplication between two vectors, and γ lτ and β lτ are learnable parameters with the same dimension as x iτ .

Original: Values of µ τ and σ τ show the mean and standard deviation of training data for τ -th task.
Revised: Values of µ τ and σ τ show the mean and standard deviation of training data for τ -th task.

Original: To allow the layer normalization inside adapters to adapt to each task, inspired by Perez et al.
Revised: To allow the layer normalization inside adapters to adapt to each task, inspired by Perez et al.

Original: De Vries et al.
Revised: De Vries et al.

Original: (2017), we generate γ lτ , β lτ via a hypernetwork as a function of task embeddings (§2.3).
Revised: (2017), we generate γ lτ , β lτ via a hypernetwork as a function of task embeddings (§2.3).

Original: 2. Task Conditioned Hypernetworks
Revised: 2. Task Conditioned Hypernetworks

Original: In order to have a model that can share information while being able to adapt to each individual task, we generate the parameters of task conditional adapter layers and layer normalization using hypernetworks.
Revised: In order to have a model that can share information while being able to adapt to each individual task, we generate the parameters of task conditional adapter layers and layer normalization using hypernetworks.

Original: A hypernetwork is a network that generates the weights of another network (Ha et al., 2017).
Revised: A hypernetwork is a network that generates the weights of another network (Ha et al., 2017).

Original: The hypernetworks capture the shared information, while the generated task conditional adapters and layer normalization allow the model to adapt to each individual task to reduce negative task interference.
Revised: The hypernetworks capture the shared information, while the generated task conditional adapters and layer normalization allow the model to adapt to each individual task to reduce negative task interference.

Original: Learned task embedding:
Revised: Learned task embedding:

Original: We first compute a task embedding I τ ∈ R t for each individual task using a task projector network
Revised: We first compute a task embedding I τ ∈ R t for each individual task using a task projector network

Original: h
Revised: h

Original: I ( . ) , which is a multi-layer perceptron consisting of two feed-forward layers and a ReLU non-linearity:
Revised: I ( . ) , which is a multi-layer perceptron consisting of two feed-forward layers and a ReLU non-linearity:

Original: [Equation]
Revised: [Equation]

Original: (5) where z τ ∈ R t (cid:48) can be a learnable parameter or any pretrained task features (Vu et al., 2020), and the task projector network h I ( . ) learns a suitable compressed task embedding from input task features.
Revised: where z τ ∈ R t (cid:48) can be a learnable parameter or any pretrained task features (Vu et al., 2020), and the task projector network h I ( . ) learns a suitable compressed task embedding from input task features.

Original: In this work, we consider a parametric z τ to allow end-to-end training which is convenient in practice.
Revised: In this work, we consider a parametric z τ to allow end-to-end training which is convenient in practice.

Original: Removing task prefixes:
Revised: Removing task prefixes:

Original: The T5 model prepends task-specific prefixes to the input sequence for conditioning.
Revised: The T5 model prepends task-specific prefixes to the input sequence for conditioning.

Original: For instance, when training on
Revised: For instance, when training on

Original: CoLA (Warstadt et al., 2019), cola sentence: is prepended to each sample.
Revised: CoLA (Warstadt et al., 2019), cola sentence: is prepended to each sample.

Original: Instead, we remove task prefixes and use task embeddings for conditioning.
Revised: Instead, we remove task prefixes and use task embeddings for conditioning.

Original: Task conditioned hypernetworks
Revised: Task conditioned hypernetworks

Original: We consider simple linear layers as hypernetworks that are functions of input task embeddings I τ .
Revised: We consider simple linear layers as hypernetworks that are functions of input task embeddings I τ .

Original: We introduce these hypernetworks in each layer of the transformer.
Revised: We introduce these hypernetworks in each layer of the transformer.

Original: We define hypernetwork h lA ( . ) that generates task conditional adapter weights ( U lτ , D lτ ):[Equation]
Revised: We define hypernetwork h lA ( . ) that generates task conditional adapter weights ( U lτ , D lτ ):

Original: [Equation]
Revised: [Equation]

Original: where W U l ∈ R ( d × h ) × t and W D l ∈ R ( h × d ) × t are the respective hypernetwork parameters.
Revised: where W U l ∈ R ( d × h ) × t and W D l ∈ R ( h × d ) × t are the respective hypernetwork parameters.

Original: We additionally define the hypernetwork h lLN ( . ) that computes the layer normalization parameters:
Revised: We additionally define the hypernetwork h lLN ( . ) that computes the layer normalization parameters:

Original: [Equation]
Revised: [Equation]

Original: where W γ l ∈ R h × t and W β l ∈ R h × t .
Revised: where W γ [Equation] ∈ R h [Equation]

Revised: W

Original: [Equation]
Revised: [Equation]

Original:  A downside of introducing a separate hypernetwork in each layer of the Transformer is that it increases the overall number of parameters.
Revised: ∈ R h × t [Equation] H YPER F ORMER ++ A downside of introducing a separate hypernetwork in each layer of the Transformer is that it increases the overall number of parameters.

Original: We, therefore, propose to share hypernetworks across transformer layers.
Revised: We, therefore, propose to share hypernetworks across transformer layers.

Original: By having a shared hypernetwork that is reusable, this strategy results in a substantial reduction in the number of parameters.
Revised: By having a shared hypernetwork that is reusable, this strategy results in a substantial reduction in the number of parameters.

Original: However, reapplying the same hypernetwork across all the layers introduces weight sharing across target parameters, which may not be desirable.
Revised: However, reapplying the same hypernetwork across all the layers introduces weight sharing across target parameters, which may not be desirable.

Original: To allow for a flexible parameterization of task conditional adapters/layer normalization, for a transformer of L layers, we introduce a set of layer id embeddings I = { l i } Li =1 , and adapter position embeddings P
Revised: To allow for a flexible parameterization of task conditional adapters/layer normalization, for a transformer of L layers, we introduce a set of layer id embeddings I = { l i } Li =1 , and adapter position embeddings P

Original: = { p j } 2 j =1 , which specify the position of adapter layers in each transformer block (after the attention layer or feed-forward layer), which are used as additional inputs to the hypernetworks.
Revised: = { p j } 2 j =1 , which specify the position of adapter layers in each transformer block (after the attention layer or feed-forward layer), which are used as additional inputs to the hypernetworks.

Original: For simplicity, we consider l i ∈ R t , p j ∈ R t , and z τ ∈ R t .
Revised: For simplicity, we consider l i ∈ R t , p j ∈ R t , and z τ ∈ R t .

Original: We feed a concatenation of ( z τ , l i , p j ) to a similar task projector network h (cid:48)
Revised: We feed a concatenation of ( z τ , l i , p j ) to a similar task projector network h (cid:48)

Original: I as in Eq.
Revised: I as in Eq.

Original: [Equation]

Original: I

Original: [Equation]
Revised: [Equation]

Original: which is then followed by a shared layer normalization to compute final task embeddings I τ ∈ R t to the hypernetwork.
Revised: which is then followed by a shared layer normalization to compute final task embeddings I τ ∈ R t to the hypernetwork.

Original: This way, the hypernetwork is able to produce distinct weights for each task, adapter position, and layer of a transformer.
Revised: This way, the hypernetwork is able to produce distinct weights for each task, adapter position, and layer of a transformer.

Original: Furthermore, layer id and adapter position embeddings are parameters that are learned via back-propagation, allowing us to train the whole model end-to-end conveniently.
Revised: Furthermore, layer id and adapter position embeddings are parameters that are learned via back-propagation, allowing us to train the whole model end-to-end conveniently.

Original:  Experiments
Revised:  Experiments

Original: Datasets:
Revised: Datasets:

Original: Following Raffel et al.
Revised: Following Raffel et al.

Original: (2020), we evaluate the performance of the models on the GLUE benchmark (Wang et al., 2019b).
Revised: (2020), we evaluate the performance of the models on the GLUE benchmark (Wang et al., 2019b).

Original: This benchmark covers multiple tasks of paraphrase detection (MRPC,
Revised: This benchmark covers multiple tasks of paraphrase detection (MRPC,

Original: QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI), and linguistic acceptability (CoLA).
Revised: QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI), and linguistic acceptability (CoLA).

Original: The original test sets are not publicly available, and following Zhang et al.
Revised: The original test sets are not publicly available, and following Zhang et al.

Original: (2021), for datasets fewer than 10K samples (RTE, MRPC, STS-B, CoLA), we divide the original validation set in half, using one half for validation and the other for the test.
Revised: (2021), for datasets fewer than 10K samples (RTE, MRPC, STS-B, CoLA), we divide the original validation set in half, using one half for validation and the other for the test.

Original: For the other larger datasets, we split 1k samples from the training set as our validation data and test on the original validation set.
Revised: For the other larger datasets, we split 1k samples from the training set as our validation data and test on the original validation set.

Original: Experimental details:
Revised: Experimental details:

Original: We use the HuggingFace implementation (Wolf et al., 2020a) of the T model (Raffel et al., 2020).
Revised: We use the HuggingFace implementation (Wolf et al., 2020a) of the T model (Raffel et al., 2020).

Original: We fine-tune all models with a constant learning rate of 0 .
Revised: We fine-tune all models with a constant learning rate of 0 .

Original: 0003 and following Raffel et al.
Revised: 0003 and following Raffel et al.

Original: (2020), we use 2 18 = steps in all experiments.
Revised: (2020), we use 2 18 = steps in all experiments.

Original: We save a checkpoint every 1000 steps for all models (see also §A).
Revised: We save a checkpoint every 1000 steps for all models (see also §A).

Original: Raffel et al.
Revised: Raffel et al.

Original: (2020) report the results based on the best checkpoint for each task independently.
Revised: (2020) report the results based on the best checkpoint for each task independently.

Original: In contrast, we focus on the more realistic setting where we report the results on a single checkpoint with the highest average validation performance across all tasks.
Revised: In contrast, we focus on the more realistic setting where we report the results on a single checkpoint with the highest average validation performance across all tasks.

Revised: The hyperparameters are selected in the same manner.

Original: In contrast to prior work (Houlsby et al., 2019), we do not learn a separate output layer for each task but instead share a frozen output layer for all the tasks, which makes our setting more parameter-efficient than prior work and is an advantage of multi-task learning with encoder-decoder models.
Revised: In contrast to prior work (Houlsby et al., 2019), we do not learn a separate output layer for each task but instead share a frozen output layer for all the tasks, which makes our setting more parameter-efficient than prior work and is an advantage of multi-task learning with encoder-decoder models.

Original: Baselines:
Revised: Baselines:

Original: We compare to the strong adapter baseline (Houlsby et al., 2019).
Revised: We compare to the strong adapter baseline (Houlsby et al., 2019).

Original: Following Houlsby et al.
Revised: Following Houlsby et al.

Original: (2019), we add adapters modules for each task after the two feed-forward modules in each transformer block of the T5 model.
Revised: (2019), we add adapters modules for each task after the two feed-forward modules in each transformer block of the T5 model.

Original: As suggested in Houlsby et al.
Revised: As suggested in Houlsby et al.

Original: (2019), we train the layer normalization parameters inside the T5 model, per task.
Revised: (2019), we train the layer normalization parameters inside the T5 model, per task.

Original: We refer to this method as Adapters .
Revised: We refer to this method as Adapters .

Original: We additionally propose a variant of this model, in which we share all layer normalization parameters (T5 and adapters) across all tasks.
Revised: We additionally propose a variant of this model, in which we share all layer normalization parameters (T5 and adapters) across all tasks.

Original: We refer to this model as Adapters † .
Revised: We refer to this model as Adapters † .

Original: We compare our models to the state-of-the-art T5 model, in which we fine-tune all parameters of the model on all tasks.
Revised: We compare our models to the state-of-the-art T5 model, in which we fine-tune all parameters of the model on all tasks.

Original: We refer to this method as T5 SMALL /T5 BASE in experiments.
Revised: We refer to this method as T5 SMALL /T5 BASE in experiments.

Original:  tasks:
Revised: Sampling tasks:

Original: During training, we sample tasks with conventional temperature-based sampling with temperature T =10 for all methods.
Revised: During training, we sample tasks with conventional temperature-based sampling with temperature T =10 for all methods.

Original: We sample different tasks proportional to p 1 /Tτ where p τ =
Revised: We sample different tasks proportional to p 1 /Tτ where p τ =

Original: N τ (cid:80) Ti =1 Nτ
Revised: N [Equation]

Original: and N τ is the number of training samples for the τth task.
Revised: and N τ is the number of training samples for the τth task.

Original: We did not experiment with more complex sampling strategies (Raffel et al., 2020) or tuning of T . 
Revised: We did not experiment with more complex sampling strategies (Raffel et al., 2020) or tuning of T . [Table]

Original: 3. Results on the GLUE Benchmark
Revised: 3. Results on the GLUE Benchmark

Original: Table 1 shows the results on GLUE for single-task and multi-task training.
Revised: Table 1 shows the results on GLUE for single-task and multi-task training.

Original: We experiment with reduc-

Original: [Table]

Original: [Table] [Table] [Table] tion factors of r = { 8 , 16 , 32 } for all adapter-based methods, where r = hd .
Revised: We experiment with reduction factors of r = { 8 , 16 , 32 } for all adapter-based methods, where r = hd .

Original: We report the results both with T5 SMALL (6 layers and 60M parameters) and
Revised: We report the results both with T5 SMALL (6 layers and 60M parameters) and

Original: T5 BASE models (12 layers and 222M parameters).
Revised: T5 BASE models (12 layers and 222M parameters).

Original: Overall, our proposed H YPER F ORMER ++ obtains strong gains over Adapters (82.51 versus 79.53 for
Revised: Overall, our proposed H YPER F ORMER ++ obtains strong gains over Adapters (82.51 versus 79.53 for

Original: T5 SMALL and 86.48 versus 84.88 for T5 BASE ) while being more parameter-efficient.
Revised: T5 SMALL and 86.48 versus 84.88 for T5 BASE ) while being more parameter-efficient.

Original: Our variant of Adapters † , which shares layer norms across tasks, outperforms prior work (Houlsby et al., 2019), which does not share such information (80. versus 79.53 for T5 SMALL and 85.83 versus 84.88 for
Revised: Our variant of Adapters † , which shares layer norms across tasks, outperforms prior work (Houlsby et al., 2019), which does not share such information (80. versus 79.53 for T5 SMALL and 85.83 versus 84.88 for

Original: T5 BASE ).
Revised: T5 BASE ).

Original: This demonstrates that in encoder-decoder models such as T5 more sharing of information across tasks is beneficial.
Revised: This demonstrates that in encoder-decoder models such as T5 more sharing of information across tasks is beneficial.

Original: Our proposed H YPER F ORMER obtains consistent improvement over our proposed Adapters † method.
Revised: Our proposed H YPER F ORMER obtains consistent improvement over our proposed Adapters † method.

Original: We attribute this improvement to the ability to learn the shared information across tasks through our hypernetworks.
Revised: We attribute this improvement to the ability to learn the shared information across tasks through our hypernetworks.

Original: Interestingly, H YPER F ORMER ++ obtains similar performance as H YPER F ORMER while being more than an order of magnitude more parameterefficient.
Revised: Interestingly, H YPER F ORMER ++ obtains similar performance as H YPER F ORMER while being more than an order of magnitude more parameterefficient.

Original: Adapter modules thus seem to be similar enough so that much of their information can be modeled by a single, appropriately conditioned network.
Revised: Adapter modules thus seem to be similar enough so that much of their information can be modeled by a single, appropriately conditioned network.

Original: Compared to single-task fine-tuning of all parameters, our methods on average improve the results by 0.45 for T5 SMALL and 1.81 for T5 BASE with substantial improvement on low-resource datasets like CoLA (63.73 versus 54.85) and RTE (75.36 versus 67.39) due to shared hypernetworks that capture the shared information and enable positive transfer effects.
Revised: Compared to single-task fine-tuning of all parameters, our methods on average improve the results by 0.45 for T5 SMALL and 1.81 for T5 BASE with substantial improvement on low-resource datasets like CoLA (63.73 versus 54.85) and RTE (75.36 versus 67.39) due to shared hypernetworks that capture the shared information and enable positive transfer effects.

Original: We also report the total number of parameters and trainable parameters for all methods in Table 1.
Revised: We also report the total number of parameters and trainable parameters for all methods in Table 1.

Original: For adapter-based methods, the number of parameters varies based on the adapter size (we report all numbers with r =32 ).
Revised: For adapter-based methods, the number of parameters varies based on the adapter size (we report all numbers with r =32 ).

Original: The multiple in terms of the number of parameters of H YPER F ORMER ++ BASE
Revised: The multiple in terms of the number of parameters of H YPER F ORMER ++ BASE

Original: with regard to
Revised: with regard to

Original: T5 BASE is 1 .
Revised: T5 BASE is 1 .

Original: 02 × with only 0 .
Revised: 02 × with only 0 .

Original: 29% trainable parameters per task.
Revised: 29% trainable parameters per task.

Original: Note that by keeping the output layer frozen for Adapters SMALL and Adapters BASE , they require 5 .
Revised: Note that by keeping the output layer frozen for Adapters SMALL and Adapters BASE , they require 5 .

Original: 51 × and 2 .
Revised: 51 × and 2 .

Original: 53 × fewer parameters respectively compared to a direct application of prior work (Houlsby et al., 2019).
Revised: 53 × fewer parameters respectively compared to a direct application of prior work (Houlsby et al., 2019).

Original: Despite using more efficient baselines, compared to Adapters BASE , H YPER F ORMER ++ BASE requires 3 × fewer trainable parameters.
Revised: Despite using more efficient baselines, compared to Adapters BASE , H YPER F ORMER ++ BASE requires 3 × fewer trainable parameters.

Original: 3. Few-shot Domain Transfer
Revised: 3. Few-shot Domain Transfer

Original: Finally, we assess how well a trained H YPER F ORMER can generalize to new tasks.
Revised: Finally, we assess how well a trained H YPER F ORMER can generalize to new tasks.

Original: We evaluate performance on 5 tasks and 7 datasets.
Revised: We evaluate performance on 5 tasks and 7 datasets.

Original: In particular, we consider
Revised: In particular, we consider

Original: 1) the natural language inference (NLI) datasets SciTail (Khot et al., 2018), and CB (De Marneffe et al., 2019) from SuperGLUE (Wang et al., 2019a) 2) the question answering (QA) dataset BoolQ (Clarket al.
Revised: 1) the natural language inference (NLI) datasets SciTail (Khot et al., 2018), and CB (De Marneffe et al., 2019) from SuperGLUE (Wang et al., 2019a) 2) the question answering (QA) dataset BoolQ (Clark

Original: 2019a) ; 3) the sentiment analysis datasets
Revised: et al., 2019a); 3) the sentiment analysis datasets

Original: IMDB (Maas et al., 2011) and Yelp Polarity (Zhang et al., 2015); and 4) the paraphrase detection dataset
Revised: IMDB (Maas et al., 2011) and Yelp Polarity (Zhang et al., 2015); and 4) the paraphrase detection dataset

Original: PAWS (Baldridge et al., 2019); 5) the question classification dataset TREC (Li and Roth, 2002).
Revised: PAWS (Baldridge et al., 2019); 5) the question classification dataset TREC (Li and Roth, 2002).

Original: For CB and BoolQ, since test sets are not available, we divide the validation sets in half, using one half for validation and the other for testing.
Revised: For CB and BoolQ, since test sets are not available, we divide the validation sets in half, using one half for validation and the other for testing.

Original: For Yelp polarity, TREC, and IMDB, since validation sets are not available, we similarly divide the test sets to form validation sets.
Revised: For Yelp polarity, TREC, and IMDB, since validation sets are not available, we similarly divide the test sets to form validation sets.

Original: For the rest, we report on the original test sets.
Revised: For the rest, we report on the original test sets.

Original: We consider the models trained on GLUE reported in Table 1 and evaluate them on the test set after the few-shot fine-tuning on each target training data.
Revised: We consider the models trained on GLUE reported in Table 1 and evaluate them on the test set after the few-shot fine-tuning on each target training data.

Original: For
Revised: For

Original: Adapters † and our method, we use the adapter and the task embedding respectively trained on the most similar GLUE task for initialization, i.e. MNLI for NLI,
Revised: Adapters † and our method, we use the adapter and the task embedding respectively trained on the most similar GLUE task for initialization, i.e. MNLI for NLI,

Original: QNLI for QA, SST-2 for sentiment analysis, and QQP for paraphrase detection.
Revised: QNLI for QA, SST-2 for sentiment analysis, and QQP for paraphrase detection.

Original: Following prior evidence of positive transfer from NLI to other tasks (Conneau and Kiela, 2018; Yin et al., 2020; Phang et al., 2018), we initialize the out-of-domain TREC from MNLI.
Revised: Following prior evidence of positive transfer from NLI to other tasks (Conneau and Kiela, 2018; Yin et al., 2020; Phang et al., 2018), we initialize the out-of-domain TREC from MNLI.

Original: We show the results of full fine-tuning of all model’s parameters, Adapters † , and H YPER F ORMER ++in Table 2.
Revised: We show the results of full fine-tuning of all model’s parameters, Adapters † , and H YPER F ORMER ++in Table 2.

Original: Our method significantly surpasses the baselines on the majority of settings.
Revised: Our method significantly surpasses the baselines on the majority of settings.

Original: 3. Low-resource Fine-tuning
Revised: 3. Low-resource Fine-tuning

Original:  Given that our model H YPER F ORMER ++ BASE has substantially fewer trainable parameters than T5 BASE , we investigate whether it generalizes better in a low-resource setting.
Revised: [Figure] Given that our model H YPER F ORMER ++ BASE has substantially fewer trainable parameters than T5 BASE , we investigate whether it generalizes better in a low-resource setting.

Original: We subsample each individual task in GLUE for varying training sizes.
Revised: We subsample each individual task in GLUE for varying training sizes.

Original: We train the models for 15,000 steps, which we found to be sufficient to allow them to converge.
Revised: We train the models for 15,000 steps, which we found to be [Table] Sentiment Analysis

Original: Figure 2 shows the results.
Revised: [Table] 86.01 ± 0. 86.57 ± 0. 86.60 ± 0.

Revised: Yelp polarity 76.85 ± 14.3 81.37 ± 13.1 90.25 ± 1.

Revised: 87.84 ± 1.

Revised: 91.08 ± 0.

Revised: 90.36 ± 1.

Revised: 89.22 ± 0.

Revised: 91.09 ± 0.

Revised: 91.15 ± 0.

Revised: 90.19 ± 0.

Revised: 90.15 ± 0.

Revised: 91.06 ± 0.

Revised: 90.92 ± 0.

Revised: 91.52 ± 0.

Revised: 92.09 ± 0.

Revised: 91.32 ± 0.

Revised: 92.26 ± 0.

Revised: 92.50 ± 0.

Revised: 91.68 ± 0.

Revised: 92.36 ± 0.

Revised: 92.70 ± 0.

Revised: Paraphrase Detection

Revised: PAWS 53.89 ± 3.

Revised: 55.69 ± 9.

Revised: 55.58 ± 7.

Revised: 54.18 ± 1.

Revised: 63.38 ± 5.

Revised: 72.71 ± 1.

Revised: 55.23 ± 3.

Revised: 68.78 ± 1.

Revised: 73.39 ± 2.

Revised: 71.51 ± 2.

Revised: 73.82 ± 1.

Revised: 78.24 ± 2.

Revised: 82.81 ± 1.

Revised: 85.36 ± 0.

Original: H YPER F ORMER ++ BASE substantially improves results with limited training data, indicating more effective fine-tuning in this regime.
Revised: 86.3 ± 1. 85.67 ± 0. 87.89 ± 0. 89.12 ± 0.

Original: [Figure]
Revised: 88.33 ± 0.

Original: [Table]
Revised: 90.41 ± 0.

Original: Sentiment
Revised: 90.87 ± 0.

Original: [Table]
Revised: sufficient to allow them to converge. Figure 2 shows the results. H YPER F ORMER ++ BASE substantially improves results with limited training data, indicating more effective fine-tuning in this regime.

Original:  Analysis 4. Parameter Efficiency
Revised:  Analysis 4. Parameter Efficiency

Original: In this section, we compare the number of parameters of H YPER F ORMER ++ with Adapters.
Revised: In this section, we compare the number of parameters of H YPER F ORMER ++ with Adapters.

Original: Adapters parameters:
Revised: parameters:

Original: The standard setting (Houlsby et al., 2019) employs two adapters per layer for each task.
Revised: The standard setting (Houlsby et al., 2019) employs two adapters per layer for each task.

Original: Each adapter layer has 2 hd parameters for projection matrices ( U lτ and D lτ ) and 2 h parameters for the layer normalization.
Revised: Each adapter layer has 2 hd parameters for projection matrices ( U lτ and D lτ ) and 2 h parameters for the layer normalization.

Original: The total number of parameters for Adapters for L Transformer layers in both an encoder and a decoder across T tasks is, therefore, 4 TL (2 hd +2 h ) , which scales linearly with the number of tasks times the number of layers.
Revised: The total number of parameters for Adapters for L Transformer layers in both an encoder and a decoder across T tasks is, therefore, 4 TL (2 hd +2 h ) , which scales linearly with the number of tasks times the number of layers.

Original: H YPER F ORMER ++ parameters: 
Revised: H YPER F ORMER ++ parameters: Our approach

Original: Our approach learns a task feature embedding per task, consisting of Tt parameters.
Revised: learns a task feature embedding per task, consisting of Tt parameters.

Original: We additionally employ layer id and adapter position embeddings in the encoder and decoder, which require 2(2+ L ) t parameters, with a fixed embedding size of t for all these feature embeddings.
Revised: We additionally employ layer id and adapter position embeddings in the encoder and decoder, which require 2(2+ L ) t parameters, with a fixed embedding size of t for all these feature embeddings.

Original: We consider a separate task projector networks h (cid:48)
Revised: We consider a separate task projector networks h (cid:48)

Original: I for encoder and decoder, which is in both cases a two-layer MLP, consisting of a total of 2(3 te + et ) parameters, where e = 128 is the hidden dimension for the task-projector network.
Revised: I for encoder and decoder, which is in both cases a two-layer MLP, consisting of a total of 2(3 te + et ) parameters, where e = 128 is the hidden dimension for the task-projector network.

Original: Our hypernetwork for adapters in encoder/decoder consists of 2(2 thd ) parameters and our layer normalization hypernetwork consists of 2(2 th ) parameters.
Revised: Our hypernetwork for adapters in encoder/decoder consists of 2(2 thd ) parameters and our layer normalization hypernetwork consists of 2(2 th ) parameters.

Original: In total, this results in t ( T +4+2 L ) 
Revised: In total, this results in t ( T +4+2 L ) (cid:124) (cid:123)(cid:122) (cid:125)

Original: [Equation]
Revised: Task features + 8 te +2 t (2 hd +2 h ) (cid:124) (cid:123)(cid:122) (cid:125) Hypernetworks parameters.

Original: The total number of parameters for hypernetworks remains constant, while the task feature parameters scale with the number of tasks or layers times t , where t =64 in our experiments.
Revised: The total number of parameters for hypernetworks remains constant, while the task feature parameters scale with the number of tasks or layers times t , where t =64 in our experiments.

Original: In settings with a large number of layers and a large number of tasks, since t (cid:28) 2 hd +2 h and T + L (cid:28) TL , our method is much more parameter-efficient compared to Adapters.
Revised: In settings with a large number of layers and a large number of tasks, since t (cid:28) 2 hd +2 h and T + L (cid:28) TL , our method is much more parameter-efficient compared to Adapters.

Original: In the current setting, the term hd is the largest term, and the factor 2 TL for Adapters is larger than the factor t for H YPER F ORMER
Revised: In the current setting, the term hd is the largest term, and the factor 2 TL for Adapters is larger than the factor t for H YPER F ORMER

Original: ++ .
Revised: ++ .

Original: 4. Do Extra Parameters Make a Difference?
Revised: Do Extra Parameters Make a Difference?

Original: While our H YPER F ORMER ++ is more parameterefficient than the baselines, the number of parameters of H YPER F ORMER per task is higher compared to
Revised: While our H YPER F ORMER ++ is more parameterefficient than the baselines, the number of parameters of H YPER F ORMER per task is higher compared to

Original: Adapters † .
Revised: Adapters † .

Original: To confirm that the improvements of
Revised: To confirm that the improvements of

Original:  H YPER F ORMER are due to its capability of sharing information across tasks and not the number of parameters, as an ablation, we run the Adapters † with r = { 2 , 4 } and choose the model performing the best on the validation set.
Revised: [Table] [Table] H YPER F ORMER are due to its capability of sharing information across tasks and not the number of parameters, as an ablation, we run the Adapters † with r = { 2 , 4 } and choose the model performing the best on the validation set.

Original: This allows Adapters †[Table] [Table] to have a higher number of parameters compared to
Revised: This allows Adapters † to have a higher number of parameters compared to

Original: H YPER F ORMER .
Revised: H YPER F ORMER .

Original: We report the results in Tableand compare them with results of H YPER F ORMER in Table 1.
Revised: We report the results in Tableand compare them with results of H YPER F ORMER in Table 1.

Original: The results demonstrate that even with an increased number of parameters, Adapters † is not able to reach the performance of H YPER F ORMER , and H YPER F ORMER performs substantially better.
Revised: The results demonstrate that even with an increased number of parameters, Adapters † is not able to reach the performance of H YPER F ORMER , and H YPER F ORMER performs substantially better.

Original: 4. Impact of the Framework Components
Revised: 4. Impact of the Framework Components

Original: We investigate the impact of the components of our framework including: (1) task conditional adapter blocks; (2) task conditional layer normalization; (3) task projection network; (4) fine-tuning of layer normalizations in the T5 model; (5) task conditional layer normalization in adapter modules and fine-tuning of layer normalizations inside the T model.
Revised: We investigate the impact of the components of our framework including: (1) task conditional adapter blocks; (2) task conditional layer normalization; (3) task projection network; (4) fine-tuning of layer normalizations in the T5 model; (5) task conditional layer normalization in adapter modules and fine-tuning of layer normalizations inside the T model.

Original: We consider our small model of Table 1 and train different variants of it.
Revised: We consider our small model of Table 1 and train different variants of it.

Original: Table 4 shows the results on GLUE, demonstrating that each component of the model contributes positively to its final performance.
Revised: Table 4 shows the results on GLUE, demonstrating that each component of the model contributes positively to its final performance.

Original: 4. Visualization of Task Embeddings
Revised: 4. Visualization of Task Embeddings

Original: To analyze what H YPER F ORMER ++ BASE has learned about the relations between different tasks, we visualize the learned task embeddings for the models trained with the largest number of samples in Table 1 and 2. 
Revised: To analyze what H YPER F ORMER ++ BASE has learned about the relations between different tasks, we visualize the learned task embeddings for the models trained with the largest number of samples in Table 1 and 2. Figure 3 illustrates the 2D vector projections of task co st ss yeim

Revised: mnsc cbrt mr qqpa qnbo tr co: CoLA st: STS-B ss: SSTye: Yelp polarity im: IMDB mn: MNLI sc:

Original: Figure 3 illustrates the 2D vector projections of task embeddings using PCA (Wold et al., 1987).
Revised: SciTail cb: CB rt: RTE [Figure] embeddings using PCA (Wold et al., 1987).

Original: Interestingly, the observed groupings correspond to similar tasks.
Revised: Interestingly, the observed groupings correspond to similar tasks.

Original: This shows that learned task embeddings by
Revised: This shows that learned task embeddings by

Original: H YPER F ORMER ++ BASE are meaningful.
Revised: H YPER F ORMER ++ BASE are meaningful.

Original: For CB, an
Revised: For CB, an

Original: NLI dataset despite being initialized from MNLI, af- [Figure] ter few-shot training the task embedding is closest to RTE, another NLI dataset.
Revised: NLI dataset despite being initialized from MNLI, after few-shot training the task embedding is closest to RTE, another NLI dataset.

Original: This is plausible as premises and hypotheses in both the discourse-based
Revised: This is plausible as premises and hypotheses in both the discourse-based

Original: CB and the news and Wikipedia-based RTE are more complex compared to MNLI.
Revised: CB and the news and Wikipedia-based RTE are more complex compared to MNLI.

Original: The sentence similarity dataset STS-B is grouped close to the MRPC paraphrase dataset.
Revised: The sentence similarity dataset STS-B is grouped close to the MRPC paraphrase dataset.

Original: CoLA, which focuses on linguistic acceptability is very different from other tasks and is not grouped with any of the observed task embeddings.
Revised: CoLA, which focuses on linguistic acceptability is very different from other tasks and is not grouped with any of the observed task embeddings.

Original: In addition, the task embeddings for 1) all the sentiment analysis datasets namely SST-2, Yelp polarity, and IMDB; 2) the two large-scale NLI datasets namely
Revised: In addition, the task embeddings for 1) all the sentiment analysis datasets namely SST-2, Yelp polarity, and IMDB; 2) the two large-scale NLI datasets namely

Original: MNLI and SciTail; 3) question answering datasets, i.e.
Revised: MNLI and SciTail; 3) question answering datasets, i.e.

Original: BoolQ and QNLI; and 4) paraphrase datasets namely
Revised: BoolQ and QNLI; and 4) paraphrase datasets namely

Original: QQP and PAWS are each grouped together.
Revised: QQP and PAWS are each grouped together.

Original:  Related Work
Revised:  Related Work

Original: Multi-task learning:
Revised: Multi-task learning:

Original: Multi-task learning, i.e., learning a unified model to perform well on multiple different tasks, is a challenging problem in NLP.
Revised: Multi-task learning, i.e., learning a unified model to perform well on multiple different tasks, is a challenging problem in NLP.

Original: It requires addressing multiple challenges such as catastrophic forgetting, and handling disproportionate task sizes resulting in a model overfitting in lowresource tasks while underfitting in high-resource ones (Arivazhagan et al., 2019).
Revised: It requires addressing multiple challenges such as catastrophic forgetting, and handling disproportionate task sizes resulting in a model overfitting in lowresource tasks while underfitting in high-resource ones (Arivazhagan et al., 2019).

Original: Liu et al. (2019a) proposed Multi-Task Deep Neural Network (MTDNN) for learning from multiple NLU tasks.
Revised: Liu et al. (2019a) proposed Multi-Task Deep Neural Network (MTDNN) for learning from multiple NLU tasks.

Original: Although
Revised: Although

Original: MTDNN obtains impressive results on GLUE, it applies multi-task learning as a form of pretraining followed by task-specific fine-tuning.
Revised: MTDNN obtains impressive results on GLUE, it applies multi-task learning as a form of pretraining followed by task-specific fine-tuning.

Original: In another line of research, Clark et al. (2019b) proposed to learn multi-task models with knowledge distillation.
Revised: In another line of research, Clark et al. (2019b) proposed to learn multi-task models with knowledge distillation.

Original: Houlsby et al.
Revised: Houlsby et al.

Original: (2019) trained adapters for each task separately, keeping the model fixed.
Revised: (2019) trained adapters for each task separately, keeping the model fixed.

Original: Stickland and
Revised: Stickland and

Original: Murray (2019) share the model parameters across tasks and introduce task-specific adapter parameters, which is more parameter-inefficient than our method.
Revised: Murray (2019) share the model parameters across tasks and introduce task-specific adapter parameters, which is more parameter-inefficient than our method.

Original: Hypernetworks and contextual parameter generation:
Revised: Hypernetworks and contextual parameter generation:

Original: Our work is closely related to hypernetworks (Ha et al., 2017).
Revised: Our work is closely related to hypernetworks (Ha et al., 2017).

Original: In a continual learning setup, where tasks are learned sequentially, Oswald et al.
Revised: In a continual learning setup, where tasks are learned sequentially, Oswald et al.

Original: (2020) proposed a task-conditioned hypernetwork to generate all the weights of the target model.
Revised: (2020) proposed a task-conditioned hypernetwork to generate all the weights of the target model.

Original: Our method is substantially more efficient as we do not generate all the weights of the target model but a very small number of parameters for adapter modules to allow the model to adapt to each individual task efficiently.
Revised: Our method is substantially more efficient as we do not generate all the weights of the target model but a very small number of parameters for adapter modules to allow the model to adapt to each individual task efficiently.

Original: Similarly, Jin et al.
Revised: Similarly, Jin et al.

Original: (2020) generate the full model from task-specific descriptions in different domains whereas we efficiently generate only small adapter modules for each task. 
Revised: (2020) generate the full model from task-specific descriptions in different domains whereas we efficiently generate only small adapter modules for each task. .

Original: Prior work also proposed meta-learning or
Revised: Prior work also proposed meta-learning or

Original: Bayesian approaches to generate softmax layer parameters for new settings (Bansal et al., 2020;
Revised: Bayesian approaches to generate softmax layer parameters for new settings (Bansal et al., 2020;

Original: Ponti et al., 2020).
Revised: Ponti et al., 2020).

Original: Meta-learning approaches are notoriously slow to train.
Revised: Meta-learning approaches are notoriously slow to train.

Original: In addition, generating softmax parameters requires a substantially higher number of parameters, leaves the method unable to adapt the lower layers of the model, and restricts their application to classification tasks.
Revised: In addition, generating softmax parameters requires a substantially higher number of parameters, leaves the method unable to adapt the lower layers of the model, and restricts their application to classification tasks.

Original: In contemporaneous work, ¨Ust¨un et al.
Revised: In contemporaneous work, ¨Ust¨un et al.

Original: (2020) proposed a multilingual dependency parsing method based on adapters and contextual parameter generator networks (Platanios et al., 2018) where they generate adapter parameters conditioned on trained input language embeddings.
Revised: (2020) proposed a multilingual dependency parsing method based on adapters and contextual parameter generator networks (Platanios et al., 2018) where they generate adapter parameters conditioned on trained input language embeddings.

Original: Their study is limited to multilingual dependency parsing, while our work studies multi-task learning and applies to several tasks thanks to the general sequence-to-sequence nature of our model.
Revised: Their study is limited to multilingual dependency parsing, while our work studies multi-task learning and applies to several tasks thanks to the general sequence-to-sequence nature of our model.

Original: Moreover, their number of trainable parameters is 2 . 88 × larger than their base model since they employ a contextual parameter generator in each layer.
Revised: Moreover, their number of trainable parameters is 2 . 88 × larger than their base model since they employ a contextual parameter generator in each layer.

Original: In contrast, we use a single compact hypernetwork allowing us to efficiently condition on multiple tasks and layers of a transformer model.
Revised: In contrast, we use a single compact hypernetwork allowing us to efficiently condition on multiple tasks and layers of a transformer model.

Original:  Conclusion
Revised:  Conclusion

Original: We propose a parameter-efficient method for multi-task fine-tuning.
Revised: We propose a parameter-efficient method for multi-task fine-tuning.

Original: Our approach is to train shared hypernetworks to generate task-specific adapters conditioned on the task, layer id, and adapter position embeddings.
Revised: Our approach is to train shared hypernetworks to generate task-specific adapters conditioned on the task, layer id, and adapter position embeddings.

Original: The shared hypernetworks capture the knowledge across tasks and enable positive transfer to low-resource and related tasks, while task-specific layers allow the model to adapt to each individual task.
Revised: The shared hypernetworks capture the knowledge across tasks and enable positive transfer to low-resource and related tasks, while task-specific layers allow the model to adapt to each individual task.

Original: Extensive experiments show that our method obtains strong improvement over multi-task learning on the GLUE benchmark, and substantially improves the in-domain task generalization. 
Revised: Extensive experiments show that our method obtains strong improvement over multi-task learning on the GLUE benchmark, and substantially improves the in-domain task generalization. Acknowledgments

Revised: We are grateful to Dani Yogatama, Neil Houlsby and

Revised: Colin Raffel for feedback on a draft of this paper.

Revised: We would like to also thank Adam Paszke, Jamie

Revised: Kiros and George Dahl for useful comments and discussions

Original: A Experimental DetailsHyperparameters
Revised: A Experimental Details

Revised: Hyperparameters:

Original: We use a batch size of 64 for
Revised: We use a batch size of 64 for

Original: T5 SMALL and 32 for T5 BASE to fit the GPU memory.
Revised: T5 SMALL and 32 for T5 BASE to fit the GPU memory.

Original: We set the dimension of the task feature embedding ( z τ ) to t (cid:48)
Revised: We set the dimension of the task feature embedding ( z τ ) to t (cid:48)

Original: =512 , and the dimension of the task embedding ( I τ ) to t =64 .
Revised: =512 , and the dimension of the task embedding ( I τ ) to t =64 .

Original: For low-resource fine-tuning in §3.3, we use reduction factors of { 16 , 32 , 64 } .
Revised: For low-resource fine-tuning in §3.3, we use reduction factors of { 16 , 32 , 64 } .

Original: Data pre-processing:
Revised: Data pre-processing:

Original: We download all datasets from the HuggingFace Datasets library (Wolf et al., 2020b).
Revised: We download all datasets from the HuggingFace Datasets library (Wolf et al., 2020b).

Original: Following Raffel et al.
Revised: Following Raffel et al.

Original: , we cast all datasets into a sequence-to-sequence format, and recast STS-B as a 21-class classification task by rounding its target scores to their nearest increment of 0.2.
Revised: , we cast all datasets into a sequence-to-sequence format, and recast STS-B as a 21-class classification task by rounding its target scores to their nearest increment of 0.2.

Original: Computing infrastructure:
Revised: Computing infrastructure:

Original: We run the experiments in Table 1 on 4 GPUs, and the rest of the experiments on 1 GPU on a heterogeneous cluster with Tesla
Revised: We run the experiments in Table 1 on 4 GPUs, and the rest of the experiments on 1 GPU on a heterogeneous cluster with Tesla

Original: V100, Tesla A100, Tesla P4, and GTX1080ti GPUs.
Revised: V100, Tesla A100, Tesla P4, and GTX1080ti GPUs.

Original: Performance evaluation:
Revised: Performance evaluation:

Original: Table 5 and 6 present the efficiency evaluation in terms of memory, and time for all the methods measured on the GLUE benchmark.
Revised: Table 5 and 6 present the efficiency evaluation in terms of memory, and time for all the methods measured on the GLUE benchmark.

Original: We report the time for 1000 training steps.
Revised: We report the time for 1000 training steps.

Original: Our approach has several attractive properties.
Revised: Our approach has several attractive properties.

Original: Our
Revised: Our

Original: H YPER F ORMER ++ BASE approach offers a much better memory usage with low-overhead, while H YPERF ORMER BASE and T5 BASE cause substantial memory overhead.
Revised: H YPER F ORMER ++ BASE approach offers a much better memory usage with low-overhead, while H YPERF ORMER BASE and T5 BASE cause substantial memory overhead.

Original: In dealing with large-scale transformer models like T5, efficient memory usage is of paramount importance.
Revised: In dealing with large-scale transformer models like T5, efficient memory usage is of paramount importance.

Original: Second, in terms of training time, our method is much faster than Adapters † BASE .
Revised: Second, in terms of training time, our method is much faster than Adapters † BASE .

Original: Relative to
Revised: Relative to

Original: T5 BASE , H YPER F ORMER ++ BASE increases the training time by 30.49%, while Adapters † BASE causes the substantial training time overhead of 84.93%. 
Revised: T5 BASE , H YPER F ORMER ++ BASE increases the training time by 30.49%, while Adapters † BASE causes the substantial training time overhead of 84.93%. Model Memory ∆ %

Revised: T5 BASE 7.76 (GB) -

Revised: Adapters † BASE 5.95 (GB) -23.32%

Revised: H YPER F ORMER BASE 7.60 (GB)

Original: [Table]
Revised: [Table]

Revised: H YPER F ORMER ++ BASE

Revised: [Table]

Revised: [Table]

Revised: H YPER F ORMER BASE

Original: [Table]
Revised: [Table]

Revised: H YPER F ORMER ++ BASE

Revised: [Table]

Revised: Impact of adapter’s bottleneck size on the performance

Revised: Similar to (Houlsby et al., 2019), adapter’s reduction factor needs to be set per dataset.

Revised: Table 7 shows the validation performance of H YPERF ORMER ++ on the GLUE tasks for different adapters’ reduction factors.

Revised: While the pattern may not be always consistent, generally, smaller datasets seem to benefit more from larger bottleneck size, i.e., less parameters for adapters, while the opposite is the case for larger datasets, which require more modeling capacity.

Revised: [Table]