Original: P ARAMETER -E FFICIENT F INE -T UNING D ESIGN S PACES
Revised: P ARAMETER -E FFICIENT F INE - TUNING D ESIGN S PACES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Parameter-efficient fine-tuning (PEFT) aims to achieve performance  comparable to fine-tuning, using fewer trainable parameters.
Revised: The aim of parameter-efficient fine-tuning is to achieve performance that is comparable to fine-tuning, but with fewer trainable parameters.

Original: Several strategies (e.g., Adapters, Prefix Tuning, BitFit, and LoRA) have been proposed. However, their designs are hand-crafted separately, and it remains unclear whether certain design patterns exist for PEFT.
Revised: Several hand-crafted strategies, such as Adapters, Prefix Tuning, BitFit, and LoRA, have been proposed, but it remains unclear whether there are underlying design patterns.

Original: Thus, we present a PEFT design paradigm and discover design patterns that are applicable to different experimental settings.
Revised: Thus, we present a parameter-efficient design paradigm and identify design patterns that are applicable to various experimental settings.

Original: Instead of focusing on designing another individual tuning strategy, we introducePEFT design spaces characterized by four components: layer grouping, trainable parameter allocation, tunable groups, and strategy assignment.
Revised: Instead of developing another individual tuning strategy, we introduce design spaces that parameterize tuning structures and strategies. These design spaces consist of four components: layer grouping, trainable parameter allocation, tunable groups, and strategy assignment.

Original: Starting from an initial design space, we progressively refine the space based on the model quality of each design choice and make greedy selection at each stage over these four components.

Original: We discover the following design patterns: (i) group layers in a spindle pattern; (ii) allocatethe number of trainable parameters to layers uniformly; (iii) tune all the groups; (iv) assign proper tuning strategies to different groups.
Revised: Our experiments reveal the following design patterns: (i) group layers in a spindle pattern, (ii) allocate trainable parameters evenly among layers, (iii) tune all groups, and (iv) assign appropriate tuning strategies to each group.

Original: Thesedesign patterns result in new PEFT methods. We show experimentallythat these methods consistently and significantly outperform investigated PEFT strategies across different backbone models and different tasksin natural language processing 1 .
Revised: These patterns lead to new methods for parameter-efficient fine-tuning, which we show experimentally outperform existing strategies across various backbone models and NLP tasks 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large pretrained models have achieved the state-of-the-art performances across a wide variety of downstream natural language processing tasks through fine-tuning on task-specific labeled data (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Joshi et al., 2019; Sun et al., 2019; Clark et al., 2019; Lewis et al., 2020a;Bao et al., 2020;
Revised: Large pre-trained models have shown to achieve state-of-the-art results in many downstream natural language processing tasks, by fine-tuning on task-specific labeled data (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Joshi et al., 2019; Sun et al., 2019; Clark et al., 2019; Lewis et al., 2020a;

Original:  He et al., 2020; Raffel et al., 2020; Ziems et al., 2022).
Revised: Bao et al., 2020; He et al., 2020; Raffel et al., 2020; Ziems et al., 2022).

Original: However,fine-tuning all the  parameters and storing them separately for different tasks is expensive in terms of computation and storage overhead (e.g., 355 M parameters for RoBERTa (Liu et al., 2019) and 175 B parameters for GPT- 3 (Brown et al., 2020)).
Revised: However, the cost of finetuning all parameters and storing them separately for each task is high in terms of computational and storage resources, e.g., 355 million parameters for RoBERTa (Liu et al., 2019) and 175 billion parameters for GPT-3 (Brown et al., 2020).

Original: This makes it difficult to deploy in real-world natural language processing (NLP) systems composed of multiple tasks.
Revised: This makes it challenging to deploy in real-world natural language processing (NLP) systems that handle multiple tasks.

Original: To adapt general knowledge in pretrained modelsto specific down-stream tasks in a more parameterefficient way, various strategies have been proposed where only  a small number of (extra) parametersare learned while  the remaining pretrained parameters are frozen (Houlsby et al., 2019b;
Revised: To make pretrained models more efficient for specific downstream tasks, various strategies have been proposed that only learn a small number of extra parameters while keeping the rest frozen (Houlsby et al., 2019b;

Original: Pfeiffer et al., 2021; Li & Liang, 2021; Brown et al., 2020; Lester et al., 2021b; Schick & Sch¨utze, 2021; Ziems et al., 2022).
Revised: Pfeiffer et al., 2021; Li & Liang, 2021; Brown et al., 2020; Lester et al., 2021b; Schick & Sch¨utze, 2021; Ziems et al., 2022).

Original: Adapter tuning (Houlsby et al., 2019b) is among the earliest strategies to steer pretrained models with a limited number of parameters. It inserts adapters (small neural modules) to each layer of the pretrained network and only  the adapters are trained at the fine-tuning time.
Revised: One such strategy is adapter tuning (Houlsby et al., 2019b), which adds small neural modules (adapters) to each layer of the pretrained network, and only trains the adapters during fine-tuning.

Original: Inspired by the success of prompting methods that control pretrainedlanguage models through textual prompts (Brown et al., 2020), Prefix tuning (Li & Liang, 2021) and Prompt tuning (Lester et al., 2021a) prependadditional tunable tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks.
Revised: Other methods, such as prefix tuning (Li & Liang, 2021) and prompt tuning (Lester et al., 2021a), have been inspired by the success of controlling pretrained models through textual prompts (Brown et al., 2020). These methods prepend tunable tokens to the input or hidden layers, and only train these tokens during fine-tuning.

Original: BitFit (Zaken et al., 2021) updates the bias terms in pretrained models while freezing the remaining parameters. LoRA (Hu et al., 2021) decomposes attention weight gradients into low-rank matrices to reduce the number of trainable parameters.
Revised: BitFit (Zaken et al., 2021) updates the bias terms of pretrained models while freezing the rest, while LoRA (Hu et al., 2021) decomposes attention weight gradients into low-rank matrices to reduce the number of trainable parameters.

Original: With promising results from such research, He et al.
Revised: He et al.

Original: (2022) proposed a unified view of these ex-
Revised: (2022) proposed a unified view of these strategies, illustrating their differences and connections, but like its predecessors, the method is still equally applied to different layers of the pretrained network.

Original: [Figure]
Revised: Most current fine-tuning strategies to adapt pretrained models to specific tasks are effective, but they are often developed through manual design processes without considering potential design patterns

Original: P
Revised: P

Revised: P

Original: PL PL
Revised: PL PL

Original: ABL
Revised: ABL

Original: ABL …
Revised: ABL …

Original: [Figure]
Revised: [Figure]

Original: p ⇥ p isting strategies and illustrated differences and connections among them.
Revised: p ⇥ p across these strategies, different backbone models, and downstream tasks.

Original: Like its antecedents, the resulting method is still equally assigned to different pretrained layers.

Original: Despite being effective, most parameter-efficient fine-tuning (PEFT) strategies have been developed via manual design processes, without much consideration of whether design patterns exist across these different strategies and how such patterns might apply to different backbone models and downstream tasks.

Original: Moreover, different strategies  are usually applied separately; thus, it is unclear which strategy works best when and where (Mao et al., 2022), as well as how these different strategies reinforce or complement each other.
Revised: The effectiveness of different strategies is also unclear as they are usually applied separately, and it’s unknown how they reinforce or complement each other (Mao et al., 2022).

Original: In this light, our goal is to understand the PEFT design in amore comprehensive view and discover design patterns that are both interpretable and  applicable across different experimental settings.
Revised: Our aim is to gain a comprehensive understanding of the fine-tuning design and uncover interpretable and widely applicable design patterns.

Original: Instead of designing yet anotherindividual strategy that is equally applied  to different pretrained layers, we introduce PEFT design spaces that parameterize both tuning structures and strategies.
Revised: Instead of creating yet another strategy to be applied uniformly to various pretrained layers, we present parameter-efficient fine-tuning design spaces that allow customization of both tuning structures and strategies.

Original: More concretely, any of these design spaces is characterized by four major components as shown in
Revised: These design spaces are comprised of four main components, as illustrated in

Original: Starting from a relatively unconstrainedPEFT design space, we progressively refine the space by comparing the overall quality of models randomly sampled from design spaces enforced with different constraints (e.g., each group has the same number of layers).
Revised: We start our journey towards parameter-efficient fine-tuning design using a relatively unconstrained design space. We then narrow this space through successive rounds of comparison, using random sampling and while enforcing constraints such as equal layer grouping.

Original: Throughout the experimental process, we discover several  design patterns for PEFT, such as group layers in a spindle pattern, allocate the number of trainable parameters to layers uniformly, tune allthe groups, and assign proper tuning strategies to different groups. We further introduce new PEFT methods that adopt all these discovered design patterns.
Revised: Through this process, we discover several key design patterns, including layer grouping in a spindle pattern, uniform allocation of trainable parameters, tuning all groups, and appropriate strategy assignments.

Original: Extensive experiments show that our methodsconsistently outperform investigated PEFT strategies.
Revised: Our new methods outperform existing parameter-efficient fine-tuning strategies.

Original: Although we use T5 (Raffel et al., 2020) and classification tasks as the working example, we find that our methods with all these discovered design patters are applicable to other backbones (e.g., RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020b) and XLNet (Yang et al., 2019)) and different natural language processing tasks (e.g., summarization, machine translation, and eight SuperGLUE datasets).
Revised: We demonstrate the effectiveness of our approach using T5 (Raffel et al., 2020) and classification tasks, but find that the discovered design patterns are applicable to other backbones (such as RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020b) and XLNet (Yang et al., 2019)), and NLP tasks (e.g., summarization, machine translation, and eight SuperGLUE datasets).

Original: Our contributions can be summarized as follows: (i)
Revised: Our contributions are: (i)

Original: We introduce PEFT design spaces.
Revised: The introduction of parameter-efficient fine-tuning design spaces.

Original: (ii)
Revised: (ii)

Original: Based on these design spaces, we discover several design patterns in PEFT via comprehensive experiments.
Revised: The discovery of several design patterns in parameter-efficient fine-tuning through comprehensive experiments.

Original: (iii)
Revised: (iii)

Original: Our discovered design patterns lead to PEFT methods, consistently outperforming investigated PEFT strategies across different backbone models anddifferent NLP tasks.
Revised: The creation of parameter-efficient fine-tuning methods based on the discovered design patterns, which outperform existing strategies on various backbone models and NLP tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work is closely related to and built upon the research aboutthe network design spaces and PEFT.
Revised: Our work is closely related to and builds on work about network design spaces and parameterefficient fine-tuning.

Original: We discuss the connections and differences below.
Revised: We discuss the connections and differences below.

Original: Network Design Spaces
Revised: Network Design Spaces.

Original: A lot of works designed neural network models via an ad-hoc discovery of new design choices that improve performances (Radosavovic et al., 2019), such as the use of deeper architectures or residuals.
Revised: Many works designed neural network models via an ad-hoc discovery of new design choices that improve performance (Radosavovic et al., 2019), such as the use of deeper architectures or residual connections.

Original: Recently, there have been works (Radosavovic et al., 2020; You et al., 2020; Radosavovic et al., 2019) performing at the design spacelevel to discover new design principles for convolutional neural networks (Radosavovic et al., 2020) and graph neural networks (You et al., 2020).
Revised: Recent work (Radosavovic et al., 2020; You et al., 2020; Radosavovic et al., 2019) focuses on the design space to discover new design principles for convolutional neural networks (Radosavovic et al., 2020) and graph neural networks (You et al., 2020).

Original: Inspired by this line of research, we focus on the design space perspective to rethink PEFT, with the goal of discovering design patterns that are applicable to differentexperimental settings.
Revised: Inspired by this work we focus on the design spaces to rethink parameter-efficient finetuning, with the goal of discovering design patterns that are applicable to different settings.

Original: PEFT for NLP As pretrained models grow in size, storingfine-tuned models becomes exceedingly expensive, and fine-tuning becomes infeasible for those without extremely high compute resources.
Revised: Parameter-Efficient Fine-Tuning for NLP. As pretrained models increase in size, storing and finetuning them becomes increasingly expensive and unfeasible for those without ample computational resources.

Original: A growing body of research has been devoted to findingparameter-efficient alternatives for adapting large-scalepretrained models with reduced memory and storage costs.
Revised: A growing body of research is aimed at finding alternatives to fine-tuning large-scale models that reduce memory and storage costs.

Original: Houlsby et al. (2019a) proposedto adapt large models using bottleneck layers (with skip-connections) between each layer. This idea has been extended in many domains (Stickland & Murray, 2019; Pfeiffer et al., 2020; Rebuffi et al., 2017; Lin et al., 2020).
Revised: Some researchers have proposed using bottleneck layers with skip-connections to adapt large models, as seen in works such as Houlsby et al. (2019a), (Stickland & Murray, 2019), (Pfeiffer et al., 2020), and (Rebuffi et al., 2017).

Original: Other works have aimed to avoid introducing additional parameters by identifying and training only a subset of all model parameters (Zhao et al., 2020; Guo et al., 2020; Mallya et al., 2018; Radiya-Dixit & Wang, 2020; Sung et al., 2021;
Revised: Other works focus on identifying and training only a subset of all model parameters, such as (Zhao et al., 2020) and (Guo et al., 2020).

Original: Zaken et al., 2021). Recent works also explored the idea of rank decomposition (Zhang et al., 2021) and injecting trainable rank decomposition matrices into each layer (Hu et al., 2021; Karimi Mahabadi et al., 2021).
Revised: More recent research explores low-rank decomposition (Zhang et al., 2021) and the injection of trainable low-rank decomposition matrices into each layer (Hu et al., 2021; Karimi Mahabadi et al., 2021).

Original: Li & Liang (2021) introduced prefix-tuning that prepends a set of prefixes  to autoregressive language models orprepends prefixes for both encoders and decoders.
Revised: Li & Liang (2021) introduced prefix-tuning, where a set of prefixes is added to autoregressive language models or both encoders and decoders, while Lester et al.

Original: The prefix parameters are updated while the pretrained parameters are fixed. Lester et al. (2021b) proposed a similar method, but only added virtual tokens at the embedding layer of large-scale models rather than discrete prompts (Deng et al., 2022; Zhong et al., 2022).
Revised: (2021b) proposed adding virtual tokens to the embedding layer.

Original: Sung et al. (2022) used side-tuning to be more parameterefficient. Recently,
Revised: Another approach, side-tuning, was introduced in (Sung et al., 2022).

Original: He et al.
Revised: He et al.

Original: (2022) and Ding et al.
Revised: (2022) and Ding et al.

Original: (2022) proposed a unified view ofthe existing PEFT strategies and illustrated the difference and connections among them.
Revised: They proposed a unified view of existing parameter-efficient fine-tuning strategies.

Original:  Mao et al.
Revised: In yet another approach, Mao et al.

Original: (2022)also introduced a unified framework to combine different methods through mixture-of-experts.
Revised: (2022) introduced a unified framework to combine various methods through mixture-of-experts.

Original: In contrast to these aforementioned works that assign their individual method equally to different pretrained layers, we focus on more general design spaces of PEFT. This could provide a more comprehensive view of PEFT in terms of both the tuning structures and tuning strategies.
Revised: Our research focuses on the general design spaces of parameter-efficient fine-tuning, providing a more comprehensive view of this method.

Original: Through experiments where we progressively refine design spaces, we  discover design patterns for PEFT.
Revised: By experimenting and refining design spaces, we aim to discover design patterns for parameter-efficient fine-tuning.

Original:  C OMPONENTS OF D ESIGN S PACES
Revised:  C OMPONENTS OF D ESIGN S PACES

Original: When defining design spaces of PEFT, we aim to cover key design components and provide a representative set of choices in each design component. Note that our goal is not to enumerate all possible design spaces, but to demonstrate howthe use of design spaces can help inform PEFT research. 
Revised: Our goal is not to list all possible design choices, but to show how design spaces can guide parameter-efficient fine-tuning research. As such, we pick a representative subset for each of the following four components: (i) layer grouping, (ii) trainable parameter allocation, (iii) tunable groups, and (iv) strategy assignment.

Original: Concretely, in our work, the PEFT design spaces are formed by  a representative set of choices in
Revised: Figure 1 provides an example. Given these choices, we sample from a distribution over them, then pick a subset that performs the best, narrowing down the set of choices. Given that more restrictive set, we repeat the procedure by sampling and picking a now even more restrictive subset until we arrive at a concise description of the design space. Quite understandably, this is very costly when dealing with Large Language

Original: PEFT, which consists of the following four components: (i) layer grouping, (ii) trainable parameter allocation, (iii) tunable groups, and (iv) strategy assignment.
Revised: Models.

Original: Following the illustrated design space example in Figure 1, we describe these four design components in detail below and  will explore their design choices in Section 4.
Revised: Taking a leaf out of (Radosavovic et al., 2020) we perform our experiments using a sufficiently cheap model, in this case T5-base and T5-3b, unless stated otherwise. Further details will be discussed in the next section. For now let’s review the set of choices available.

Original: Layer Grouping Different layers in pretrained models capture different information and behave differently.
Revised: Layer Grouping. Different layers in pre-trained models capture varying information and behave differently.

Original: For example, Jawahar et al. (2019) found that the{ 3 , 4 , 5 , 6 , 7 , 9 , 12 } -th layers have the most representation power in BERT and every layer captures a different type of information ranging from the surface, syntactic, tothe semantic level representation of text.
Revised: For example, the authors of (Jawahar et al., 2019) found that the 3 , 4 , 5 , 6 , 7 , 9 , 12 th layers have the most representation power in BERT and each layer captures a different type of information, ranging from surface to syntactic to semantic level representation of text.

Original: For instance, the 9th layer has predictive power for semantic tasks such as checking random swapping of coordinated clausal conjuncts, while the 3rd layer performs best in surface tasks like predicting sentence length.
Revised: For instance, the 9th layer performs well in semantic tasks such as checking random swaps of coordinated clauses, while the 3rd layer is best suited for surface tasks like predicting sentence length.

Original: Therefore when adapting these pretrained models to downstream tasks, how to group layers with similar behaviors together is critical to the design andapplication of proper PEFT strategies.
Revised: When adapting these pre-trained models for downstream tasks, it’s crucial to group layers with similar behaviors together. This is critical to the design and proper implementation of parameter-efficient fine-tuning strategies.

Original: For this design component, we studythe patterns of how to group consecutive layers in pretrained models (e.g., transformer layers in T5) during the fine-tuning process.
Revised: In this design component, we study patterns of how to group consecutive layers in pre-trained models (e.g., transformer layers in T5) during the fine-tuning process.

Original: Trainable Parameter Allocation
Revised: Trainable Parameter Allocation.

Original: In PEFT, the total number of trainable parameters is usually preset, such as a small portion of the totalnumber of parameters in the pretrained models.
Revised: In parameter-efficient fine-tuning, the total number of trainable parameters is usually set to a small portion of the total parameters in the pretrained model.

Original: We will study  different design choices for how to allocate a predefined number of trainable parameters to  layers.
Revised: Our study will explore different ways to allocate the predefined number of trainable parameters to the layers.

Original: Tunable Groups Zaken et al.
Revised: Tunable Groups.

Original: (2021) found that not all the parameters  need to be tuned during fine-tuning on the downstream tasks.
Revised: Not all the parameters of a pretrained model need to be updated during fine-tuning for downstream tasks.

Original: For instance, BitFit (Zaken et al., 2021) only updates the bias parametersin pretrained models while freezing the remaining parameters.
Revised: For example, BitFit (Zaken et al., 2021) only updates the bias parameters while freezing the rest.

Original: Thus, we study which groups  need to be learned during PEFT to attain better performances.
Revised: As a result, we explore which groups of parameters need to be learned during parameter-efficient fine-tuning to achieve better performance.

Original: Strategy Assignment
Revised: Strategy Assignment.

Original: In order to improve the parameter efficiency, different sets of strategies (Li & Liang, 2021; Lester et al., 2021b;
Revised: In order to improve the parameter efficiency, different sets of strategies (Li & Liang, 2021; Lester et al., 2021b;

Original: Houlsby et al., 2019b; Hu et al., 2021) have been proposed where only a small number of (extra) parameters are tuned and the remaining parameters in these pretrained models are frozen to adapt their general knowledge to specific down-stream tasks.
Revised: Houlsby et al., 2019b; Hu et al., 2021) were proposed, where only a small number of (extra) parameters are tuned and the remaining parameters in these pretrained models are frozen to adapt their general knowledge to specific down-stream tasks.

Original: We hypothesize that different groups might benefit from different proper strategies (or combinations) for capturing different types of information.
Revised: We hypothesize that different groups might benefit from different proper strategies (or combinations) for capturing different types of information.

Original: More formally, given a set of individual strategies A for assignment, for any group G i , assign a subset U i
Revised: More formally, given a set of individual strategies A for assignment, for any group G i , assign a subset U i

Original: ⊂
Revised: ⊂

Original: A to each layer in G i .
Revised: A to each layer in G i .

Original:  D ISCOVERING D ESIGN P ATTERNS
Revised:  D ISCOVERING D ESIGN P ATTERNS

Original: Building on these four different design components of PEFT design spaces, we will start from a relatively unconstrained design space and progressively discover the design patterns

Original: . 4. D ESIGN S PACE E XPERIMENTAL S ETUP

Original: We first describe our experimental setup for discovering the design patterns.

Original: Note that our process is generic for other tasks and future pretrained backbone models.

Original: Datasets Our process for discovering design patterns of PEFT is based on the average performances on the widely-used GLUE benchmark (Wang et al., 2018).

Original: It covers a wide range of natural language understanding tasks. First, single-sentence tasks include (i) Stanford Sentiment Treebank (SST-2) and (ii) Corpus of Linguistic Acceptability (CoLA). Second, similarity and paraphrase tasks include (i) Quora Question Pairs (QQP), (ii) Semantic Textual Similarity Benchmark (STS-B), and (iii) Microsoft Research Paraphrase Corpus (MRPC). Third, inference tasks include (i) MultiGenre Natural Language Inference (MNLI), (ii) Question Natural Language Inference (QNLI), and (iii)
Revised: Each design space, denoted as S i , consists of a set of models ( S i -models) that satisfy the constraints characterizing the space with respect to layer grouping, trainable parameter allocation, tunable groups, and strategy assignment.

Original: Recognizing Textual Entailment (RTE). To compare performances, the Matthews correlation is measured for CoLA; the Spearman correlation is used for STS-B, and accuracy is measured for the rest GLUE tasks. Pretrained Backbone Models and Model Settings We use T5-base/3b (Raffel et al., 2020) as the main pretrained backbone models for discovering design patterns via our PEFT design spaces.
Revised: To discover design patterns, we start from a relatively unconstrained parameter-efficient fine-tuning design space S 0 .

Original: We use Hugging Face 2 for our implementations and follow the default settings. During the exploration, we set the total number of trainable parameters (in the percentage of that in the backbone model) to 0.5% by following He et al. 4. D ISCOVERING D ESIGN P ATTERNS U SING T5- BASE In this subsection, we describe the empirical process for discovering the design patterns using T5base (pretrained backbone model) as the working example.
Revised: We progressively refine it via S 1 , . . .

Original: Each PEFT design space (denoted as S i ) consists of a set of models ( S i -models) that satisfy constraints characterizing the space with respect to layer grouping, trainable parameter allocation, tunable groups, and strategy assignment. To discover design patterns, we start from a relatively unconstrained PEFT design space ( S 0 ). Then we progressively refine design spaces (from S 0 to S 1:4 ) by comparing  overall quality of models in de- sign spaces enforced with different constraints (e.g., each group has the same number of layers).
Revised: S 4 by comparing the overall quality of models in design spaces enforced with different constraints (e.g., each group has the same number of layers).

Original: To quantify the overall quality of models in any design space S i with a low-compute, low-epoch regime (Radosavovic et al., 2020), we randomly sample 100 models from S i , fine-tune with  3 epochs 3 , and compute the average of the GLUE average performances.
Revised: To quantify the overall quality of models in any design space S i with a low-compute, low-epoch regime (Radosavovic et al., 2020), we randomly sample 100 models from S i , fine-tune with only 3 epochs 2 , and compute the average of the GLUE average performance.

Revised: Using such a low number of epochs is sufficient to obtain a sufficiently representative score to draw consistent conclusions (see Table 7 in the Appendix) that extend to a full training run.

Original: We emphasize that our goal is to demonstrate how the perspective of design spaces can help inform PEFT research, rather than to find out the “best” design space or method.
Revised: We emphasize that our goal is to demonstrate how the perspective of design spaces can help inform parameter-efficient fine-tuning research, rather than to find out the “best” design space or method.

Original: For computational efficiency, it is beyond the scope of this work to enumerate all possible constraints with respect to the design space components (Section 3).
Revised: For computational efficiency, it is beyond the scope of this work to enumerate all possible constraints with respect to the design space components (Section 3).

Revised: For efficiency, we use T5-base (pretrained backbone model) as it’s both representative and also sufficiently small to make experimentation with many options computationally affordable.

Revised: In this work, we follow the discovery sequence of “grouping patterns – trainable parameter allocation – tunable groups – strategy assignment”: (1) To explore and understand the design patterns in all the layers in large pre-trained models in scale, it is necessary and more efficient to study the layers in the unit of groups.

Revised: So we start with the grouping patterns.

Revised: (2) Once figuring out the optimal grouping patterns, it is then important to explore how to allocate the trainable parameters to these different groups in order to study more subtle designs with fair comparisons (e.g., this would allow comparing different patterns of strategy assignments without the impact from different trainable parameters.).

Original: 4.2. T HE I NITIALS 0 D ESIGN S PACE
Revised: (3) Next, it becomes influential to examine which groups need to be learned during fine-tuning before we dig into the strategy assignment patterns. Because it is only meaningful to study assigning strategies to different groups after we figure out which groups need to be learned. (4) Finally, we study the tuning strategy assignment, which is the most subtle design. 4. S 0 — T HE I NITIAL D ESIGN S PACE

Original: The initial relatively unconstrained design space S 0 consists of all models without constraints on the design space components (Section 3).
Revised: The initial relatively unconstrained design space S 0 consists of all models without constraints on the design space components.

Original: Individual PEFT strategies consist of Adapter,Prefix,
Revised: Individual parameter-efficient fine-tuning strategies consist of Adapter,

Original:  BitFit, and LoRA.
Revised: Prefix, BitFit, and LoRA.

Original: One can think of this S 0 design space as a set of random models ( S 0 -models) with random design patterns.

Original: Specifically, without grouping constraints, each layer of the pretrained layer has a half chance to be tuned: if tuned,  random strategies (or combinations) with a random amount of trainable parameters are assigned to that layer.
Revised: Specifically, without grouping constraints, each layer of the pretrained layer has a probability of 0 . 5 to be tuned. If tuned, a random strategy, or combinations thereof, with a random amount of trainable parameters are assigned to that layer.

Original: Before comparing more subtle design patterns such ashow to properly assign tunable strategies among Adapter, Prefix, BitFit, and LoRA, we begin with exploring how to group layers and how to allocate the total number of trainable parameters to layers.
Revised: Before comparing more subtle design patterns such as to which tuning strategy among Adapter, Prefix, BitFit, and LoRA to pick, we begin by exploring how to group layers and how to allocate the total number of trainable parameters to layers.

Original: 4.2. T HE S 1 D ESIGN S PACE WITH A DDITIONAL G ROUPING C ONSTRAINTS
Revised: 4. S 1 — A PPLYING G ROUPING C ONSTRAINTS

Revised: Transformers are quite deep by now.

Revised: This makes it impractical to pick a different tuning strategy for each layer.

Revised: As such, the first question to ask is how to assemble the layers into groups that will be tuned using the same strategy.

Original: Inspired by Radosavovic et al.
Revised: Inspired by Radosavovic et al.

Original: (2020), wealso consider 4 groups ( G 1 , . .
Revised: (2020), we consider 4 groups, G 1 , . .

Original: , G 4 , in the order of forward pass) in the experiments 4 . Denote by N i the number of layers in G i .
Revised: , G 4 , in the order of forward pass, in the experiments 3 Denote by N i the number of layers in G i .

Original: As illustrated in Figure 2, we compare the following layer grouping patterns:(i)
Revised: As illustrated in Figure 2, we compare the following layer grouping patterns:

Revised: Increasing

Revised: ( N i +1 > N i ): the number of layers in groups gradually increases; 2 We set the low epoch by observing whether it is enough for models to obtain stable performances to draw consistent conclusions (See Table 7 in the Appendix).

Original: Increasing ( N i +1 > N i ): the number of layers in groups gradually increases; (ii) Uniform ( N i +

Revised: The experimental results with 8 groups are shown in the Table 16 in the Appendix. [Table] [Table] Uniform ( N i +

Original: =
Revised: =

Original: N i ): the number of layers in groups is the same;(iii) Decreasing ( N i +1 <
Revised: N i ): the number of layers in groups is the same; Decreasing ( N i +1 <

Original: N i ): the number of layers in groups gradually decreases;(iv)
Revised: N i ): the number of layers in groups gradually decreases;

Original: Spindle ( N 1 < N 2 = N 3 > N 4 ): the numbers of layers in groups at both ends are smaller;and (v)
Revised: Spindle ( N 1 < N 2 = N 3 > N 4 ): the numbers of layers in groups at both ends are smaller;

Original: Bottleneck ( N 1 > N 2 = N 3 < N 4 ): the numbers of layers in groups at both ends are bigger.
Revised: Bottleneck ( N 1 > N 2 = N 3 < N 4 ): the numbers of layers in groups at both ends are bigger.

Original: These layer grouping patterns lead to 5 different design spaces.
Revised: These layer grouping patterns lead to 5 possible design choices.

Original: Any of these 5 design spaces consists of all models in the S 0 design space that satisfy one of these grouping pattern constraints.
Revised: They consist of all models in the S 0 design space that satisfy one of these grouping pattern constraints.

Original: To compare the overall model qualities of different design spaces, we (i) randomly sample 100 models from the S 0 design space that satisfy each grouping pattern constraint (Figure 2); (ii) fine-tune with 3 epochs; and (iii) compute the average performances for each design space.
Revised: To compare the overall model qualities of different design spaces, we (i) randomly sample 100 models from the S 0 design space that satisfy each grouping pattern constraint (Figure 2); (ii) fine-tune with 3 epochs; and (iii) compute the average performance for each design space.

Original: We will follow this procedure as we progressively add new constraints later.
Revised: We will follow this procedure as we progressively add new constraints later.

Original: The averaged performances are shown in Table 1 5 .
Revised: The average performance is shown in Table 1 4 .

Original: We find that models from the design space with the spindle grouping pattern (Figure 2) consistently outperform those from the other design spaces across all the 8 GLUE tasks.
Revised: We find that models from the design space with the spindle grouping pattern (Figure 2) consistently outperform those from the other design spaces across all the 8 GLUE tasks.

Original: This may be due to the complexities ofinformation captured in different layers of large pretrained models, which favor information adaptation in the discovered layer grouping pattern. From now on, we will group layers in a spindle pattern.
Revised: In other words, we find that fine-tuning works better if we treat a small number of layers close to the input and close to the output as special, and furthermore, if we divide up the bulk of the network into two blocks, each with their own design choices.

Original: We refer to S 0 with this additional design pattern as the new  S 1 design space. [Table] [Table]
Revised: Applying the spindle grouping partitioning to S 0 yields the new design space S 1 .

Original: 4.2. T HE S 2 D ESIGN S PACE WITH A DDITIONAL P ARAMETER C ONSTRAINTS
Revised: 4. S 2 — V ARYING THE N UMBER OF T RAINABLE P ARAMETERS PER L AYER

Original: We continue to explore design patterns in trainable parameter allocation to refine the S 1 design space. Denote by n i  the number of trainable parameters for the i-th layer of the pretrained backbone model, we compare the following design patterns: (i)
Revised: Now that we know how to group the layers we need to establish how to allocate the parameters n i within the layers i of each group. In particular, we consider the following options:

Original: Increasing ( n i +1 ≥ n i ):the number of trainable parameters in every layergradually increases (or remains the [-same); (ii) Uniform ( n i +
-]
Revised: Increasing ( n i +1 ≥ n i ): number of trainable parameters per layer increases or remains the [-same); (ii) Uniform ( n i +
-]same.

Original: = n i ): the number of trainable parameters in every layer is the same; and (iii) Decreasing ( n i +1 ≤ n i ): the number of trainable parameters in every layer gradually decreases (or remains the same).
Revised: [Table]

Original: Following the procedure described in Section 4.2.2, we obtain 100 models for each of these 3 new design spaces.
Revised: As above, we obtain 100 models for each of these 3 new design spaces.

Original: Table 2 reports the average performances of these 3 design spaces.
Revised: Table 2 reports the average performance of these 3 design spaces.

Original: The uniform allocation design pattern obtains the highest GLUE average performance, making this relatively simple, interpretable design pattern favorable.
Revised: The uniform allocation design pattern obtains the highest GLUE average performance, making this relatively simple, interpretable design pattern favorable.

Original: We will allocate the number of trainable parameters to layers uniformly. We refer to S 1 with this additional design pattern as the new  S 2 design space.
Revised: Allocating the number of trainable parameters to layers uniformly yields the new design space S 2 .

Original: 4.2. T HE S 3 D ESIGN S PACE WITH A DDITIONAL T UNABLE G ROUP C ONSTRAINTS
Revised: 4. S 3 — S ELECTING THE G ROUPS

Original: Before digging into the strategy assignment design patterns, it is necessary to examine which groups need to be tuned. After all, it isonly meaningful to study assigning strategies to different groups after we find out which groups  need to be fine-tuned.
Revised: Given that we established how to partition layers into groups, and how to allocate parameters per group, the next step is to assess whether all groups actually need tuning.

Original: As shown in Table 3, we explore various design patterns in tunable groups tofurther constrain the S 2 design space.
Revised: Rather than exploring the 2 4 − 1 = 15 combinatorial choices we limit ourselves to the 4(4 + 1) / 2 = 10 options with the exception of ( G 2 , G 3 ) , since focusing on interior groups only does not yield good results (this is consistent with our findings in Table 3).

Original: Based on the GLUE average performances, we find that all the groups need to be tuned to obtain the best performances.
Revised: Based on the GLUE average performance, we find that all the groups need to be tuned to obtain the best results.

Original: This suggests that all the groups of pretrained layers have captured useful information that should be adapted to the downstream tasks.
Revised: This suggests that all the groups of pretrained layers have captured useful information that should be adapted to the downstream tasks.

Original: We will tune all the groups. We refer to S 2 with this additional design pattern as the new  S 3 design space. [Equation] [Equation]
Revised: Tuning all the groups yields the new design space S 3 .

Original:  S 4 D ESIGN S PACE WITH A DDITIONAL S TRATEGY C ONSTRAINTS
Revised: 4. S 4 — S ELECTING S TRATEGIES PER G ROUP

Original: Finally, we study the subtle design pattern with respect to assigning proper strategies by further constraining the derived S 3 design space.
Revised: So far the structure we’ve been exploring is fairly trivial: S 4 amounts to a uniform distribution of parameters over the layers of the groups and to tuning all groups. This belies the fact that we still have significant freedom of design in picking specific fine-tuning approaches.

Original: Specifically, each design space consists of models that assign a subset of { Adapter (A), Prefix (P), BitFit (B), andLoRA (L) } to all layers of any group G i ( i = 1 , . .
Revised: Specifically, each design space consists of models that assign a subset of { Adapter (A), Prefix (P), BitFit (B), and

Original: We begin by adding different G 1 strategy assignment constraints to the S 3 space. [Table] ) pretrained backbone models, respectively.
Revised: LoRA (L) } to the layers of each group G i for i ∈ { 1 . . . 4 } .

Original: The results are averaged over 20 random runs (with standard deviations as subscripts). The S 4 -model and the S 4 -3b-model perform significantly better than the second-best PEFT methods in all the eight datasets at the significance level p < 0 . 05( ∗ ) or even p < 0 . 01( ∗∗ )
Revised: This is quite a large space of options.

Original: [Table] Following the same pattern discovery procedure (Section 4.2.2), we discover strategy assignment patterns for G 1 . Then we progressively add G i ( i > 1 ) strategy assignment constraints together with the discovered strategy assignment patterns forall Gj ( j = 1 , . . , i − 1 ) to the S 3 space.
Revised: To make some headway, we determine the ideal choice progressively by first reviewing strategies for G 1 , then G 2 up to G 4 .

Original: Due to space limit, we present results of this process in the Appendix ( G 1 in Table 8, G 2 Table 9, Gin Table 10, and G 4 in Table 11), which suggests strategy assignment of G 1 -(A, L) – G 2 -(A, P) –
Revised: Due to space constraints the details of this procedure are relegated to the appendix( G 1 in Table 8, G 2 Table 9, G 3 in Table 10, and G 4 in Table 11).

Original: G 3 -(A, P, B) – G 4 -(P, B, L) for the T5-base pretrained backbone model.
Revised: We arrive at the following strategy assignment for the T5-base pretrained backbone:

Original: We will assign the discovered proper tuning strategies to groups.
Revised: [Equation] For example, Adapter is more recommended in groups closer to input, while BitFit is more recommended in groups closer to the output.

Original: We refer to S 3 with this additional design pattern asthe new S 4 design space, which consists of the final S 4 -model.
Revised: The resulting design space will be referred to as S 4 .

Original: 4. D ISCOVERING D ESIGN P ATTERNS U SING T5-3 B
Revised: 4. V ERIFICATION OF THE D ESIGN C HOICES ON T5-3 B

Revised: So far our results have led to a competent fine-tuning strategy for T5-base.

Original: We then repeat the above process on T5-3b toexamine if the  design patternswe discovered using smaller models (T5-base) still apply when we use larger models.
Revised: To assess whether we actually discovered some useful strategies that have validity beyond T5-base, we need to apply it to other models, too. For convenience we pick T5-3b. As before, the detailed results are relegated to the appendix (Tables 12, 13, 14 and 15). We observe that the following design patterns still apply:

Original: The results are shown in Table(layer grouping),
Revised: grouping layers in a spindle pattern (Table 12) 2.

Original: Table 13 (trainable parameter allocation), Table 14 (tunable groups) and Table(strategy assignment) in the Appendix.
Revised: uniformly allocating the number of trainable parameters to layers (Table 13) 3. tuning all the groups (Table 14) 4.

Original: We observe that the design patterns still apply when larger models like T5-3b are used: (i) grouping layers in a spindle pattern (Table 12), (ii) uniformly allocating the number of trainable parameters to layers (Table 13), (iii) tuning all the groups (Table 14), and (iv) tuning different groups with proper strategies (Table 15).
Revised: tuning different groups with proper strategies (Table 15) Note that for T5-3b (with final design space S 4 -3b), the discovered proper strategy assignment is slightly different [Equation]

Original: For T5-3b, the discovered proper strategy assignment is G 1 -(P, L) – G 2 -(A, L) – G 3 -(P, B, L) – G 4 -(A, P, B).
Revised: Datasets. Our process is based on the average performance on the widely-used GLUE benchmark (Wang et al., 2018). It covers a wide range of natural language understanding tasks.

Original: We refer to the final design space as S 4 -3b and the final model in this space as S 4 -3b-model.
Revised: First, single-sentence tasks include (i) Stanford Sentiment Treebank (SST-2) and (ii) Corpus of Linguistic Acceptability (CoLA). Second, similarity and paraphrase tasks include (i) Quora Question Pairs (QQP), (ii) Semantic Textual Similarity Benchmark (STS-B), and (iii) Microsoft Research Paraphrase Corpus (MRPC). Third, inference tasks include (i) Multi-Genre Natural Language Inference (MNLI), (ii) Question Natural Language Inference (QNLI), and (iii)

Revised: Recognizing Textual Entailment (RTE).

Revised: To compare performance, the Matthews correlation is measured for CoLA; the Spearman correlation is used for STS-B, and accuracy is measured for the remaining GLUE tasks.

Revised: Pretrained Backbone Models and Model Settings

Original: [Table]
Revised: We use T5-base/3b (Raffel et al., 2020) as the main pretrained backbone models for discovering design patterns via our parameter-efficient finetuning design spaces. We use HuggingFace Transformers for our implementations and follow the default settings. During the exploration, we fix the total number of trainable parameters (in the percentage of that in the backbone model) by following He et al. By limiting ourselves to a rather concise parameter space and a small number of parameters within that parameter space that we allow to be fine-tuned we ensure that exploration remains computationally feasible.

Revised: Obviously, this exploration would be pointless if the discovered insights were not portable.

Revised: Hence, we need to evaluate how well the strategies perform on new models and new architectures.

Original:  E VALUATION
Revised:  E VALUATION

Original: The S 4 -model (Section 4.2.5) and S 4 -3b-model (Section 4.3) adoptall the design patternsthat have been discovered by using T5-base and T5-3b [Table] respectively.As a result
Revised: The S 4 model (Section 4.5) and S 4 -3b model (Section 4.6) adopt the design patterns discovered from T5-base and T5-3b, respectively.

Original: [Table]

Original: they are both new methods of PEFT.

Original: We will evaluate their effectiveness when applied to different pretrained backbone models and different NLP tasks.
Revised: We will evaluate their effectiveness when applied to different pretrained backbones and different NLP tasks.

Original: 5. E XPERIMENTAL S ETUPDatasets
Revised: 5. E XPERIMENTAL S ETUP

Revised: Dataset.

Original: Besides the GLUE datasets (Wang et al., 2018)
Revised: Besides the GLUE datasets (Wang et al., 2018)

Original: (Section 4.1), wefurther evaluate our methods on two generation tasks used by He et al.
Revised: (Section 4.7), we evaluate our methods on two generation tasks used by He et al.

Original: (2022): (i) Abstractive Summarization using XSum (Narayan et al., 2018), and(ii) Machine Translation using the WMT 2016 en-ro dataset (Bojar et al., 2016).
Revised: Abstractive Summarization using XSum (Narayan et al., 2018), and Machine Translation using the WMT 2016 en-ro dataset (Bojar et al., 2016).

Original: We report ROUGE scores (Lin, 2004) on the XSum test set, and BLEU scores (Papineni et al., 2002) on the en-ro test set.
Revised: We report ROUGE scores (Lin, 2004) on the XSum test set, and BLEU scores (Papineni et al., 2002) on the en-ro test set.

Original: Models and Model Settings
Revised: Models and Model Settings.

Original: We mainly compare our methods with the following baselines: (i)
Revised: We mainly compare our methods with the following baselines: (i)

Original: Full Fine-tuning (full): it fine-tunes all the model parameters in the pretrained models; (ii) Adapter (Houlsby et al., 2019b): it adds adapter modules to each transformer layer;
Revised: Full Fine-tuning (full): it fine-tunes all the model parameters in the pretrained models; (ii) Adapter (Houlsby et al., 2019b): it adds adapter modules to each transformer layer;

Original: (iii) Prefix (Li & Liang, 2021): it optimizes a set of small continuous vectors prepended to transformer layers; (iv) BitFit (Zaken et al., 2021): it only updates the bias terms in pretrained models; (v) LoRA (Hu et al., 2021): it decomposes the attention weight into low-rank matrices to reduce the number of trainable parameters.
Revised: (iii) Prefix (Li & Liang, 2021): it optimizes a set of small continuous vectors prepended to transformer layers; (iv) BitFit (Zaken et al., 2021): it only updates the bias terms in pretrained models; (v) LoRA (Hu et al., 2021): it decomposes the attention weight into low-rank matrices to reduce the number of trainable parameters.

Original: Besides T5 (Raffel et al., 2020), we additionally apply our methods to other backbone models including RoBERTa-base/large (Liu et al., 2019) and BART-base/large (Lewis et al., 2020a).
Revised: Besides T5 (Raffel et al., 2020), we additionally apply our methods to other backbone models including RoBERTa-base/large (Liu et al., 2019) and BART-base/large (Lewis et al., 2020a).

Original: We use the default settings.
Revised: We use the default settings.

Original: We set the total number of trainable parameters (in the percentage of  that in the backbone model) by following He et al.
Revised: We set the total number of trainable parameters (in the percentage of [Table] [Table] that in the backbone model) by following He et al.

Original: Specifically, this value is set to 0.5% for
Revised: Specifically, this value is set to 0.5% for

Original: Adapter, Prefix, LoRA, and our methods, and 0.1% for BitFit.
Revised: Adapter, Prefix, LoRA, and our methods, and 0.1% for BitFit.

Original: For all the experiments, we followed Liu et al.
Revised: For all the experiments, we followed Liu et al.

Original: (2019) to set the linear decay scheduler with a warmup ratio of 0.06 for training.
Revised: (2019) to set the linear decay scheduler with a warmup ratio of 0.06 for training.

Original: The batch size was 128 for base models and 64 for large models.
Revised: The batch size was 128 for base models and 64 for large models.

Original: The maximum learning rate was 5 e − 5 and the maximum number of training epochs was set to be either 5 or 10 .
Revised: The maximum learning rate was 5 e − 5 and the maximum number of training epochs was set to be either 5 or 10 .

Original: All the experiments were performed 
Revised: All the experiments were performed using 8 A100 GPUs.

Original: 5. E FFECTIVENESS ON GLUE WITH T5 B ACKBONES
Revised: 5. E FFECTIVENESS ON D IFFERENT B ACKBONES

Original: [Table]
Revised: GLUE with T5 Backbone.

Original: With our discovered design patterns [Table] we fine-tune T5-base ( S 4 -model) and T5-3b ( S 4 -3b-model) on GLUE and compare them with all the baseline methods.
Revised: With our discovered design patterns, we fine-tune T5-base ( S 4 -model) and T5-3b ( S 4 -3b-model) on GLUE and compare them with all the baseline methods.

Original: The results are shown in Table 
Revised: The results are shown in Table 4, where the key measure is the GLUE average performance (last column). We

Original: [Table]
Revised: [Table]

Original: where the key measure is the GLUE average performance (last column).

Original: We find that our S 4model and S 4 -3b-model consistently outperform the investigated methods in the key measure.
Revised: find that our S 4 -model and S 4 -3b-model consistently outperform the investigated methods in the key measure.

Original: By tuning only 0 .
Revised: By tuning only 0 .

Original: 5% parameters, our methods even outperform the full fine-tuning baseline where all the parameters are tuned, indicating the effectiveness of our discovered PEFT design patterns.
Revised: 5% parameters [Table] our methods even outperform the full fine-tuning baseline where all the parameters are tuned, indicating the effectiveness of our discovered parameterefficient fine-tuning design patterns.

Original: 5. G ENERAL E FFECTIVENESS ON GLUE WITH R O BERT A B ACKBONES
Revised: GLUE with RoBERTa Backbone.

Original: We directly apply the S 4 -model and S 4 - 3b-model (adopting design patterns discovered using T5-base and T5-3b) to finetune the RoBERTa-base and RoBERTalarge pretrained backbone models (with no extra discovery process), respectively.
Revised: We directly apply the S 4 -model and S 4 -3b-model (adopting design patterns discovered using T5-base and T5-3b) to fine-tune the RoBERTa-base and RoBERTalarge pretrained backbone models, respectively.

Original: We keep all the other settings the same and evaluate them on GLUE datasets.
Revised: We keep all the other settings the same and evaluate them on GLUE datasets.

Original: We also compare with variant methods randomly sampled from two design spaces: (i) S 0 -model , where all the designs are randomly selected for RoBERTa as in S 0 ; (ii) S 3 -model , where strategies are randomly assigned to different RoBERTa layer groups as in S 3 .
Revised: We also compare with variant methods randomly sampled from two design spaces: (i) S 0 -model , where all the designs are randomly selected for RoBERTa as in S 0 ; (ii) S 3 - model , where strategies are randomly assigned to different RoBERTa layer groups as in S 3 .

Original: Table 5 shows that (i) the design patterns (adopted by S 4 -model and S 4 -3b-model) discovered using T5 models are applicable to the RoBERTa backbone models and outperform the investigated methods in GLUE average performances with no extra discovery process;(ii) improved performances fromS 0 -models,
Revised: Table 5 shows that (i) the design patterns (adopted by S 4 -model and S 4 -3b-model) discovered using T5 models are applicable to the RoBERTa backbone models and outperform the investigated methods in GLUE average performance with no extra discovery process 5 ; (ii) improved performance from

Original: S  3 -models, to S 4 -(3b)-models support adding more constraints in the pattern discovery process (Section 4).
Revised: S 0 -models, S 3 -models, to S 4 -(3b)-models support adding more constraints in the pattern discovery process (Section 4).

Original: 5. G ENERAL E FFECTIVENESS ON G ENERATION T ASKS WITH BART B ACKBONES
Revised: SuperGLUE with XLNet Backbone. We also directly use the S 4 -model and S 4 -3b-model (adopting design patterns discovered using T5-base and T5-3b) to fine-tune the XLNet-base and XLNetlarge pretrained backbone models without any extra discovery process. We keep all the other settings the same and evaluate them on SuperGLUE datasets. Table 17 (In the Appendix) reiterates the fact that our PEFT design patterns discovered from T5 models are generelizable to the XLNet backbone models and outperform the investigated methods in other tasks (SuperGLUE) with no additional discovery process.

Original: Like in Section 5.3, we furtherdirectly apply the S 4 -model and S 4 -3b-model (adopting design patterns discovered using T5-base and T5-3b) to fine-tune the BART-base and BART-large pretrained backbone models (without additional discovery process.), respectively.
Revised: Generation Tasks with BART Backbone. We further apply the S 4 -model and S 4 -3b-model (adopting design patterns discovered using T5-base and T5-3b) to fine-tune the BART-base and BARTlarge pretrained backbone models, respectively.

Original: We evaluate the models on two generation tasks: summarization (XSUM) and machine translation (en-ro) following He et al.
Revised: We evaluate the models on two generation tasks: summarization (XSUM) and machine translation (en-ro) following He et al.

Original: We also compare with PA (parallel adapter) using the same number of trainable parameters (He et al., 2022).
Revised: We also compare with PA (parallel adapter) using the same number of trainable parameters (He et al., 2022).

Original: Table 6 shows that our methods, although adopting design patterns discovered from classification tasks using T5, still outperform investigated PEFT strategies on generation tasks with different BART backbones.
Revised: Table 6 shows that our methods, although adopting design patterns discovered from classification tasks using T5, still outperform investigated parameter-efficient fine-tuning strategies on generation tasks with different BART backbones.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: PEFT adapts knowledge in pretrained models to down-stream tasks in a more parameter-efficient fashion.
Revised: Parameter-efficient fine-tuning adapts knowledge in pretrained models to down-stream tasks in a more parameter-efficient fashion.

Original: Instead of focusing on designing another strategy in the first place, we introduced PEFT design spaces.
Revised: Instead of focusing on designing another strategy in the first place, we introduced parameter-efficient fine-tuning design spaces.

Original: We empirically discovered several design patterns in PEFT.
Revised: We empirically discovered several design patterns in parameter-efficient fine-tuning.

Original: These design patterns led to new PEFT methods.
Revised: These design patterns led to new parameter-efficient fine-tuning methods.

Original: Experiments showed that these methods consistently outperform investigated PEFT strategies across different backbone models and different tasks in natural language processing.
Revised: Experiments showed that these methods consistently outperform investigated parameter-efficient fine-tuning strategies across different backbone models and different tasks in natural language processing.

Original: A M ORE E XPERIMENTAL R ESULTS
Revised: A M ORE E XPERIMENTAL R ESULTS

Original: [Table] [Table] epochs Increasing
Revised: Grouping

Original: [Table]
Revised: [Table]

Original:  Uniform 
Revised: 70.8 Uniform 84.

Original: [Table] Decreasing [Table] [Table]
Revised: 72.1 Decreasing 81.

Original: 9 Spindle [Table]
Revised: 69.9 Spindle 86.

Original: Bottleneck 
Revised: Bottleneck 84.

Original: [Table]
Revised: 4 epochs

Original: Increasing 
Revised: Increasing 88.

Original: [Table]
Revised: [Table]

Original:  Uniform 
Revised: 74.4 Uniform 88.

Original: [Table]
Revised: [Table]

Original: Decreasing

Original: [Table]
Revised: [Table]

Original: Spindle

Original: [Table]
Revised: [Table]

Original: Bottleneck
Revised: 75.4 Decreasing 87. 75.1 Spindle 89.

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: Bottleneck 86.

Original: [Table]
Revised: [Table]

Original: 20 epochs
Revised: 20 epochs

Original: Increasing 
Revised: Increasing 92.

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 79.1 Uniform 
Revised: 79.1 Uniform 92.

Original: [Table]

Original: [Table] Decreasing [Table]
Revised: 79.9 Decreasing 91.

Original:  Spindle 
Revised: 78.1 Spindle 93.

Original: [Table]
Revised: [Table]

Original: B G ENERAL E FFECTIVENESS ON S UPER GLUE WITH XLN ET B ACKBONES
Revised: B G ENERAL E FFECTIVENESS ON S UPER GLUE WITH XLN ET B ACKBONES

Original: We also directly use the S 4 -model and S 4 -3b-model (adopting design patterns discovered using T5base and T5-3b) to fine-tune the XLNet-base and XLNet-large pretrained backbone models without any extra discovery process.
Revised: We also directly use the S 4 -model and S 4 -3b-model (adopting design patterns discovered using T5base and T5-3b) to fine-tune the XLNet-base and XLNet-large pretrained backbone models without any extra discovery process.

Original: We keep all the other settings the same and evaluate them on SuperGLUE datasets.
Revised: We keep all the other settings the same and evaluate them on SuperGLUE datasets.

Original: Table 17 reiterates the fact that our PEFT design patterns discovered from T models are generelizable to the XLNet backbone models and outperform the investigated methods in other tasks (SuperGLUE (Wang et al., 2019)) with no additional discovery process.
Revised: Table 17 reiterates the fact that our PEFT design patterns discovered from T models are generelizable to the XLNet backbone models and outperform the investigated methods in other tasks (SuperGLUE) with no additional discovery process.

Original: C O N THE D ISCOVERY S EQUENCE

Original: In this work, we follow the discovery sequence of “grouping patterns – trainable parameter allocation – tunable groups – strategy assignment”:

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: -(A, P, L) 
Revised: 79.3 G 3 -(P, L) 93.

Original: [Table]
Revised: 79.0 G 3 -(A, P) 92. 79.3 G 3 -(A, L) 92. 79.4 G 3 -(A, P, L) 92. 79.6 G 3 -(P, B, L) 92.

Original: -(P [Table]
Revised: 79.2 G 3 -(A, P, B) 93.

Original:  B, L) 
Revised: 79.8 G 3 -(A, B, L) 93. 79.4 G 3 -(A, P, B, L) 93.

Original: [Table]
Revised: [Table]

Original: 2 -(A, P, B)

Original: [Table]
Revised: [Table]

Original: 2 -(A, B, L)
Revised: the T5-3b pretrained backbone model. We compare adding different layer parameter constraints to the S 1 design space

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: (A [Table]
Revised: [Table] [Table] [Table]

Original: B, L)

Original: [Table]
Revised: [Table]

Original: To explore and understand the design patterns in all the layers in large pre-trained models in scale, it is necessary and more efficient to study the layers in the unit of groups.

Original: So we start with the grouping patterns. Once figuring out the optimal grouping patterns, it is then important to explore how to allocate the trainable parameters to these different groups in order to study more subtle designs with fair comparisons (e.g., this would allow comparing different patterns of strategy assignments without the impact from different trainable parameters.). Next, it becomes influential to examine which groups need to be learned during fine-tuning before we dig into the strategy assignment patterns. Because it is only meaningful to study assigning strategies to different groups after we figure out which groups need to be learned.
Revised: [Table]

Original: Finally, we study the tuning strategy assignment, which is the most subtle design.

Original: [Table][Table] [Table] 76.8 Decreasing 83.
Revised: [Table]

Original: 74.5 Spindle 88.

Original: Bottleneck 86.

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: Prefix

Original: [Table][Table]
Revised: [Table]

Original: BitFit

Original: [Table][Table]
Revised: [Table]

Original: LoRA

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: BitFit

Original: [Table]

Original: [Table]
Revised: [Table]

Original: LoRA
Revised: LoRA

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]