From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This work introduces a method for parameter-efficient fine-tuning of pre-trained language models. The authors consider two reparametrizations of learnable weights for this method. The work builds on soft prompt-tuning (or p-tuning). Reviewers found the manuscript clear, valued the methodological novelty and the effort to make results reproducible.

However, several reviewers expressed concerns that stronger baselines had not been considered in the experimental validation to the point that two reviewers found the paper misleading. The authors revised the paper accordingly and compared their method to missing baselines. The results showed that the proposed method is close to  state of the art methods. Besides the fact that a small number of concerns remained after the discussion, the large number of changes (new experiments, revised conclusion, etc.) made to the manuscript during the rebuttal/discussion suggest this work is not ready for publication and should be resubmitted with a better positioning of the current work and discussion of the results.
justification_for_why_not_higher_score: Only one reviewer voted for acceptance. He initially gave a score of 8 without motivating his rating. After asking to justify the score, he lowered to 6, but again without providing a good justification. The reviewer did not participate to the discussions.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Response
comment: Thank you for the response.

1) We would like to note that for larger models, the performance gap is negligible (0.3 points in macro score), while this gap is primarily produced by a single task (i.e., CB). Though, for 3 of 7 tasks, AoT P-Tuning is performing better. As we claim in the paper, our main contribution is that one could fine-tune a model with a simple method that performs faster than other baselines and obtain comparable performance. There is also a novelty in that we could get comparable results with other baselines with such a simple scheme, which is contr-intuitive at first sight. From such a perspective, there is a huge amount of various proposed methods that others could develop to improve its performance further.

2) It is important to note that one may want to be able to perform multi-task inference not only because it allows evaluating several tasks within one sequence but also because it allows not replicating a model several times. I.e., even if evaluating a model with batch size equal to $1$, we could store a single backbone model for all workers instead of having $n$ each with a specific model. 
Doing so simplifies a model's serving and balancing the load on it. E.g., if we can't perform multi-task inference, then one of the workers for some tasks could be poorly utilized, while others could be over-utilized depending on the time of the day or other factors.

From: ICLR.cc/2023/Conference/Paper1701/Reviewer_ozTx
title: Updated Review
comment: Thank the authors for providing the rebuttal. After reading the rebuttal and the other reviewers' comments, I decided to maintain my current rating due to the following reasons:

1. The proposed method does not seem to outperform existing work like LoRA/Adapter (actually sometimes underperforming) in terms of accuracy. 
2. The major advantage is multi-task inference in a single batch. Firstly, I am not sure if it is highly required in real-world scenarios: for online settings, people usually run an inference with a single input (batch size =1); for offline settings, people can group the inputs by different tasks. Secondly, the advantage compared to (unfused) LoRA/adapter seems to be only a smaller computation overhead, which could be negligible compared to the LLM itself (see Figure 9, in many cases, Adaptor latency is on-par compared to the proposed method). I think the gap will be even smaller as the model gets larger.

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Response
comment: > To the best of my understanding, the proposed counterargument (1) focuses on LoRA, i.e. only one of existing parameter-efficient fine-tuning methods and (2) relies heavily on the requirement of absolutely no additional computation.

We focus on LoRA since the described by you scheme for performing a multi-task inference must work in specific cases. However, in others, it could require significant engineering efforts in practice. 
We did not presuppose an absence of additional computation. We instead speak of negligible overhead and non-negligible overhead on computations.

> i am not convinced that there a practitioner would see significant different between zero overhead and the negligible overhead introduced by, for instance, IA3 (see [1] in my original comment), or even LoRA with a sufficiently low rank.

Claiming that LoRA adds negligible computational cost is only partially correct. As we noted in the counterargument, it is zero-cost when fused, but for the non-fused case, it adds computational overhead, which a practitioner could find significant. For example, in the most practical setup, with batch size equal to $1$ and long sequence length (Figure 4g), the use of LoRA adds $15$% overhead even for the smallest possible factorization rank (i.e., $r=2$). We highlight that $15$% overhead is significant for real-world applications with high-load inference.

The gemm kernel still implies that we have to introduce computational overhead. Even if we had a specific kernel that evaluates the $WX + ABX$ operation, it would still introduce an overhead compared to the $WX$ operation. Though, it is not what the gemm kernel evaluates, so the overhead is even more significant than for such a hypothetical kernel.

Also, we tried to find an implementation of LoRA, which used the gemm kernel for inference. It is premature to claim that LoRA will have a negligible overhead with this kernel, as there seems to be no evidence for so (but we would be happy to see one!). At the same time, in our experiments, we provided evidence that the general implementation of non-fused LoRA adds computational overhead.

> IA^3

We omitted IA^3 from our experiments since it uses an entirely different experimental design than any others methods.
First, T-Few implies pre-training of IA^3 weights on an unsupervised task, while ablations without the such pre-training show that IA^3 starts performing worse. 
Second, there are only experiments available with a T0 backbone; thus, it needs to be clarified (or supported by evidence) that this method would produce similar results with other backbones. Furthermore, T-Few used in experiments utilized LM head to give predictions, and some training techniques (e.g., unlikelihood loss) could not be directly applied to the case when no verbalization is used (i.e., predictions are only produced with classification head), which is further concerns on the applicability of this method.

While we could have evaluated them, understanding the limits of IA^3 is beyond the scope of our paper. Instead, we concentrated on conventional baselines in our paper.

> Again, I admit that not having this (small) overhead is still nice to have, but this does not justify disregarding the more powerful methods for parameter-efficient fine-tuning. As such, I would argue that my concern remains unresolved.

We have not claimed in our paper that one should disregard other fine-tuning methods. We argue that AoT P-Tuning is comparable to them while being faster. 

For a practitioner with a specific amount of computational power and particular requirements for the latency of a model during the inference, it is essential to obtain as much accuracy by spending as little as possible on computations. When the computation resources are enormous or the required latency is big enough, one could choose a model with the best possible accuracy. But if these conditions do not hold, one must sacrifice accuracy for speed. From such a perspective, even a $15$% overhead for LoRA with batch size equal to $1$ could be vital since this $15$% could be a boundary between "we can deploy this model" and "we can't afford this". Our paper proposes a method that reduces this overhead for a little cost at accuracy ($0.3$ macro score for DeBERTa-XL).

From: ICLR.cc/2023/Conference/Paper1701/Reviewer_YHHD
title: Response
comment: To the best of my understanding, the proposed counterargument (1) focuses on LoRA, i.e. only one of existing parameter-efficient fine-tuning methods and (2) relies heavily on the requirement of absolutely no additional computation.

While this (zero overhead) is nice to have, i am not convinced that there a practitioner would see significant different between zero overhead and the negligible overhead introduced by, for instance, IA3 (see [1] in my original comment), or even LoRA with a sufficiently low rank. For the reference IA3 introduces only element-wise multiplications per transformer layer, while "non-fused LoRA" requires a low-rank matrix multiplication that is fused with the gemm kernel (in efficient implementations).

Again, I admit that not having this (small) overhead is still nice to have, but this does not justify disregarding the more powerful methods for parameter-efficient fine-tuning. As such, I would argue that my concern remains unresolved.

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Rebuttal Revision of the Paper 2
comment: Dear Reviewers,

Thank you for your time and effort in reviewing our work. We have provided detailed clarification and additional experiments to address the issues raised in your comments. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

If you have any additional questions or comments, we would be happy to have further discussions.

Thank you, Authors

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Rebuttal Version of the Paper
comment: Thank you for the review. We have uploaded a rebuttal version of the paper based on all the provided reviews. Please see this version. 
We also provided a detailed description of the changes in a general comment on top of all reviews; we kindly ask you to see it.

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Rebuttal Version of the Paper
comment: Thank you for the review. We have uploaded a rebuttal version of the paper based on all the provided reviews. Please see this version. 
We also provided a detailed description of the changes in a general comment on top of all reviews; we kindly ask you to see it.

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Rebuttal Version of the Paper
comment: Thank you for the review. We have uploaded a rebuttal version of the paper based on all the provided reviews. Please see this version. 
We also provided a detailed description of the changes in a general comment on top of all reviews; we kindly ask you to see it.

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Rebuttal Version of the Paper
comment: Thank you for the review. We have uploaded a rebuttal version of the paper based on all the provided reviews. Please see this version. 
We also provided a detailed description of the changes in a general comment on top of all reviews; we kindly ask you to see it.

From: ICLR.cc/2023/Conference/Paper1701/Authors
title: Rebuttal Revision of the Paper
comment: Thank all reviewers for the provided feedback on the paper. We have considered it and improved the paper in the latest rebuttal currently available.

These changes include:
1) All reviewers were concerned about the lack of baselines used in our experiments. We have added experiments with LoRA, Adapters, and BitFit to the paper. We have observed that these methods outperform AoT P-Tuning for small models, while the gap in performance for larger ones becomes negligible. We have changed conclusions based on these results.

2) Reviewer YHHD noted that Adapters and LoRA could be parallelized for multi-task inference mode with batched matmul operation. We have missed this fact and improved the paper based on it (See Section 3.1). 

 Although it is straightforward to perform multi-task inference with Adapters, it is not easy for LoRA. If we fuse attention weights after fine-tuning, it is not well-tractable to perform multi-task inference with them. If we would like to replicate weights for each sequence in a batch, stack them, and place them in the GPU memory, this will create a significant memory consumption footprint (we covered this scenario in the new version of the paper). One could perform replication, stacking, and placing the weights into the GPU while evaluating the model weight by weight to reduce memory footprint. However, such a scheme will create a computational footprint (large matrices must be copied several times to the GPU memory). Furthermore, such a method does not allow one to use specific inference engines such as TensorRT, which is crucial for good performance.
Because of this, we argue that to perform multi-task inference with LoRA, it is necessary to perform inference without fusing weights. Passing replicated and stacked weights of low-rank addition with batch will be tractable in such a scenario.
From such a perspective, Adapters and "Not Fused LoRA" could be used for multi-task inference mode but create a computational footprint, which is covered in Sections 3.1 and 4.3. At the same time, "Fused LoRA" allows zero-cost inference and does not enable multi-task inference.

3) Reviewer ozTx is concerned about inference speed measurements and proposed to add experiments with batch size equal to $1$ and longer sequence lengths (e.g., $1024$). We have improved the section on inference speed experiment based on this review. Although, all back-bone models used in our experiment imply having sequence lengths of less than $512$ tokens. Furthermore, P-Tuning v1/v2 in its original implementation reduces the longest possible sequence length with $512 - p$ tokens, where $p$ is the prompt length. While this limitation could be easily reduced (e.g., remove positional embeddings), such a setup is unfair since it is not an implied setup to use P-Tuning v1/v2 and RoBERTa/DeBERTa models. Thus, we limited the sequence length to $384$ tokens. 

 Within the improved section of inference speed experiments, we observed that AoT P-Tuning performed with negligible overhead in most of the used setups while adding overhead for the setup when we simultaneously have the smallest back-bone model, shortest sequence length, and smallest batch size (though still being faster than other baseline methods).

From: ICLR.cc/2023/Conference/Paper1701/Reviewer_BA6Y
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes a new method to improve the inference efficiency of a family of parameter-efficient fine-tuning methods. The authors propose to add input-dependent biases to the transformer's weight matrices (Q, K, V) instead of concatenating prefixes which results in decreasing the dimensions of matrices needed to evaluate the transformer attention. Further, the authors propose two reparameterization tricks to make the calculation of these matrices tractable. Evaluation on GLUE and SuperGLUE benchmark datasets shows that the proposed method performs on par (sometimes better) with previously proposed parameter-efficient fine-tuning methods while being faster at inference. 
strength_and_weaknesses: Strengths:

+ The motivation for this work is well-founded.

Weaknesses / Questions

- Authors decide to exclude recent parameter efficient tuning methods which seem unfair. Their rationale for skipping those methods is that those methods do not work with multi-task inference. Even if that is the case, authors should still compare their method to the recent baselines.
- Without the above evaluation, it is hard to comment on the efficacy of the proposed method.
- According to table 2, the std for AoT P-Tuning is generally considerably higher than baseline models. This suggests that the model might not be stable.
clarity,_quality,_novelty_and_reproducibility: The proposed method is novel but the work ignores some of the recent developments in parameter-efficient fine-tuning methods. The writing of the paper is clear and the ideas are presented well.
summary_of_the_review: In this work, the authors propose a new method for improving the inference time of parameter-efficient methods. The writing of this paper is clear and the method is well presented. Although the idea is novel and works well when compared to P-Tuning, authors miss out on some of the most important baselines. Rationale provided by the authors to skip those baselines is unjust and thus I cannot recommend accepting this paper and thus I vote to reject this paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1701/Reviewer_ozTx
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: In this paper, the authors proposed ahead-of-time (AoT) P-Tuning, a new prompt-tuning method that adds an input-dependent offset to each layer's activation (based on the vocabulary index). The proposed method does not increase the sequence length, leading to now significant computation increase. It achieves competitive transfer learning accuracy vs. cost trade-off. 
strength_and_weaknesses: Strength:
1. Firstly, the paper is generally well-written and easy to follow.
2. The proposed method does not increase the sequence length, which does not lead to significant computation overhead for the tuned model.
3. The proposed method achieves competitive tuning accuracy compared to existing P-tuning work. 

Weakness:
1. Compared to P-tuning, the work is more similar to adapter-based tuning (since the offset is input-dependent; packing multiple tasks in a batch does not sound like a more important factor). However, the authors did not compare the proposed method with existing adapter-based tuning methods, which also does not significantly increase computation.
2. The latency measurement setting (batch size 256, sequence length 128) seems unfair. The sequence length is smaller than usual, and the batch size is larger than usual, which all favors the proposed method, leading to a smaller average overhead. Can the authors also compare the latency with P-Tuning using a batch size 1, sequence length 1024, and discuss if the overhead is still smaller than P-tuning?
3. The comparison also left out some recent state-of-the-art lite tuning methods that also do not increase sequence length like LoRA. I think it is not fair to exclude LoRA and adapter tuning in experiments, since in many cases, people would perform inference with single batch size.  
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility are good. 
The proposed method introduces a new P-tuning alike design that does not increase the sequence length, which has a certain technical novelty. 
summary_of_the_review: Please refer to the strength and weaknesses section for details. I would like to hear from authors' feedback for the final decision. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1701/Reviewer_YHHD
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper studies the problem of parameter-efficient fine-tuning for large transformer-based models. More specifically, authors consider a subset of parameter-efficient fine-tuning methods that support multi-task inference.
Authors extend the popular soft prompt-tuning (or p-tuning) technique, resulting in a technique called "Ahead-of-Time P-tuning". Authors demonstrate that their method can match or outperform prefix tuning (p-tuning v2) on standard GLUE / SuperGLUE benchmarks on popular MLM transformers. The paper also emphasizes that Ahead-of-time p-tuning allows more efficient inference than standard prefix tuning.
strength_and_weaknesses: Preface: I strongly believe that __the current manuscript can mislead the reader__ by unjustly omitting SoTA parameter-efficient methods. I make my case below and suggest several options to correct that. I am also open to discussion if authors disagree with my assessment.

__Strengths:__ 
- the paper is generally well-written and easy to follow, apart from minor typos
- authors work on a real-world problem, optimizing for the types of models that are often used in NLP applications
- the proposed method is evaluated in a fairly standard setup, making it easier to understand the reported statistics
- authors make considerable effort to make their work reproducible.

Regardless of my case against the paper, I deeply respect authors' intent to specify the exact versions and even providing a dockerfile that refers to pytorch/cuda in use. It would also be nice - but not critically required - to include some human-readable instructions on running the code, e.g. readme.

__Weakness:__

As authors note in the paper, P-tuning v1 and v2 have been largely surpassed by methods such as LoRA. However, authors decide to exclude these methods from experiments because "Adapters do not imply performing multi-task inference" (see full quote below‡).

To the best of my understanding, the only way this statement _could_ be true is when simultaneously inferencing multiple tasks in one batch - since non-simultaneous multui-task inference with adapters can be done by simply swapping adapters between batches. Disregarding the fact that this is a fairly niche scenario, it is simply wrong: __both LoRA and simpler adapters (Houlsby et al. 2019) support simultaneous multi-task inference,__ and actively run it in many - though not most - applications.

For Houlsby et al (2019) adapters, if every sequence in a batch uses its own adapter weights, you can run multi-task inference using batched GEMM, (e.g. bmm in PyTorch). This way, each sequence can be processed with its own adapter. For simplicity, I provide a simple code snippet to illustrate this principle later at the end of this section†.
LoRA adapters are parallelized the same way, except that there is no nonlinearlity, and the adapter outputs are applied to Q/K/V matrices separately.

On top of that, industrial frameworks for inferencing with adapters typically optimize this process by (1) grouping together samples for the same task and (2) writing custom kernel that runs BMM with _few_ adapters per batch, instead of one adapters per sample. However, even when this technique is not used, it is trivial to run the common parts of the transformer model in parallel for many tasks, then use a for loop to apply task-specific bits.

Finally, there are other methods for parameter-efficient fine-tuning that belong to neither adapter nor p-tuning category. Notably,

1. IA3 (Liu et al, 2022; [1]) can run multi-task inference out-of-the-box and claims to outperform p-tuning in low-resource settings at no extra computation cost
2. BitFit (Zaken et al, 2021 [2]) can run multi-task inference out-of-the-box and claims to be competitive with p-tuning at no extra inference cost
3. FISH (Sung et al, 2021 [3]) can run multi-task inference with sparse batch GEMM, similar to adapters

In its current state, the paper may mislead a non-experienced reader into thinking that if they want to run multi-task inference, they would be restricted to P-tuning variants and not allowed to use other parameter-efficient fine-tuning methods - and thus may end up with an inferior solution. I believe that this is a major concern that outweighs several positive aspects listed in the "Strengths" section.

I suggest two ways this could be potentially alleviated:

1. comparing against all the relevant baselines for parameter-efficient fine-tuning (adapters, BitFit / IA3, FISH, etc), which may yet reveal that AOT p-tuning outperforms them in some applications;
2. stating, _throughout the paper,_ that this is a specific optimization for prompt tuning - and still showing how it compares to other parameter-efficient prompt-tuning methods, even if unfavorably;

While I give authors the benefit of the doubt, it may be difficult to address this concern in the time-frame for author response. For performance comparisons, if authors are not willing to implement efficient multi-task inference for each method, their performance can be upper-bounded by single-task code with popular inference frameworks (e.g. by extending FasterTransformer[4]).


‡ - full quotes, for the reference

> One may note that the proposed method is more similar to Adapters Tuning (Houlsby et al., 2019) than P-Tuning. Although, Adapters do not imply performing multi-task inference, thus we refer to the proposed method as a variant of P-Tuning, rather than a special case of Adapters.

> Based on this experimental design choice, we exclude experiments with Adapters (Houlsby et al., 2019; He et al., 2022), as well as with LoRA (Hu et al., 2022). While a wide range of efficient fine-tuning methods could be similar to the proposed method (Ding et al., 2022; He et al., 2022), they do not allow to perform multi-task inference, which is the motivation to use AoT P-Tuning.


† - sample code for inferencing adapters for multiple tasks in parallel

```python
import torch   # tested with torch 1.10.1, no cuda required to run this test
batch, seq_length, d_model, d_adapter = 3, 5, 1024, 8

# pre-adapter activations, e.g. outputs of previous layer
inputs = torch.randn(batch, seq_length, d_model)

# adapter weights, one for each sequence in a batch
adapters_weight1 = torch.randn(batch, d_model, d_adapter)
adapters_weight2 = torch.randn(batch, d_adapter, d_model)

# apply adapters in parallel via batched GEMM
adapters_hidden = torch.bmm(inputs, adapters_weight1)
adapters_nonlinearity = torch.relu(adapters_hidden)
adapters_out = torch.bmm(adapters_nonlinearity, adapters_weight2)

# verify that this is equivalent to non-simultaneous computation
for i in range(batch):
    weight1, weight2 = adapters_weight1[i], adapters_weight2[i]
    reference = torch.matmul(inputs[i], weight1).relu().matmul(weight2)
    assert torch.allclose(adapters_out[i], reference)
```


[1] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel , https://arxiv.org/abs/2205.05638

[2] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg

[3] Training Neural Networks with Fixed Sparse Masks, Yi-Lin Sung, Varun Nair, Colin Raffel , https://arxiv.org/abs/2111.09839

[4] https://github.com/NVIDIA/FasterTransformer
clarity,_quality,_novelty_and_reproducibility: To reiterate, the paper is generally well-written and easy to follow, apart from minor typos.
Authors make considerable effort to make their work reproducible, including the supplementary code with pinned dependency versions and a Dockerfile that can be used to release a public docker image. The only thing I found lacking is the precise scripts for reproducing the experiments - or a README explaining how to run them - which can be easily fixed in a revised version of the paper.


Typos:

- End of page 3: "transforms into O((n + p)2)" - extra closing ")"
- Page 4: "lockup of xj-th prompt embedding" -  this may be a typo of "lookup" -- or a deep metaphor that i failed to understand.



summary_of_the_review: This is a well-written, easily reproducible paper that has a fundamental flaw: it unfairly ignores many advanced baseline methods based on - to the best of my knowledge - an incorrect assumption about their multi-task capabilities. Above, I made an argument (and a code snippet) showing that the more advanced baselines can in fact support multi-task inference in several ways. If my argument is correct, the paper is fundamentally flawed and will mislead readers into disregarding important results in parameter-efficient fine-tuning. As such, I will only recommend accepting the paper if either (1) my argument is formally proven incorrect, with minor revisions as requested or (2) my argument is accepted, in which case the paper will need a major revision / repositioning that may be difficult to review in one round.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1701/Reviewer_8wRe
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a new parameter-efficient method for fine-tuning a transformer model, called AoT P-tuning. Experiments show that it performs at least as well as P-tuning v2 while being 1.3x faster. As P-tuning, AoT P-tuning enables efficient multi-task inference.
strength_and_weaknesses: This paper introduces a new variant of P-tuning and shows its effectiveness (especially in terms of inference speed). The paper is well written and I don't see any particular weaknesses beside the set of experiments being somewhat limited (as pointed out by the other reviewers).
clarity,_quality,_novelty_and_reproducibility: The paper is overall very clear. The method introduced is relatively simple but novel. A reproducibility section is provided to make it possible to reproduce the results (although I haven't tried to do so).

One suggestion: please make Figures 3 and 4 larger.
summary_of_the_review: This paper is overall quite solid: it proposes a new method, it provides empirical validation, and it is well written.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold