P RESERVING I N -C ONTEXT L EARNING ABILITY IN L ARGE
L
ANGUAGE
M ODEL F INE - TUNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Pretrained large language models (LLMs) are strong in-context learners that are able to perform few-shot learning without changing model parameters.
However, as we show, ﬁne-tuning an LLM on any speciﬁc task generally destroys its incontext ability.
We discover an important cause of this loss, format specialization, where the model overﬁts to the format of the ﬁne-tuned task and is unable to output anything beyond this format.
We further show that format specialization happens at the beginning of ﬁne-tuning.
To solve this problem, we propose Prompt Tuning with MOdel Tuning (ProMoT), a simple yet effective two-stage ﬁne-tuning framework that preserves in-context abilities of the pretrained model.
ProMoT ﬁrst trains a soft prompt for the ﬁne-tuning target task, and then ﬁne-tunes the model itself with this soft prompt attached.
ProMoT ofﬂoads task-speciﬁc formats into the soft prompt that can be removed when doing other in-context tasks.
We ﬁne-tune mT5 XXL with ProMoT on natural language inference (NLI) and
English-French translation and evaluate the in-context abilities of the resulting models on 8 different NLP tasks.
ProMoT achieves similar performance on the ﬁne-tuned tasks compared with vanilla ﬁne-tuning, but with much less reduction of in-context learning performances across the board.
More importantly, ProMoT shows remarkable generalization ability on tasks that have different formats, e.g. ﬁne-tuning on a NLI binary classiﬁcation task improves the model’s in-context ability to do summarization (+0.53 Rouge-2 score compared to the pretrained model), making ProMoT a promising method to build general purpose capabilities such as grounding and reasoning into LLMs with small but high quality datasets.
When extended to sequential or multi-task training, ProMoT can achieve even better out-of-domain generalization performance.
I NTRODUCTION
Natural language processing (NLP) has recently been revolutionized by scaling up transformer based large language models (LLMs) together with large scale pretraining (Vaswani et al., 2017; Devlin et al., 2019; Raffel et al., 2020a; Brown et al., 2020;
Rae et al., 2021; Chowdhery et al., 2022; Smith et al., 2022).
In addition to improved downstream performances, these pretrained LLMs can perform a broad array of unforeseen tasks when provided with a few examples in-context.
This in-context few-shot capability allows users to ﬂexibly re-purpose LLMs for speciﬁc tasks with a minimum amount of supervised data, making it extremely convenient for fast prototyping and experimentation, especially in the low data regime.
However, even the largest and most advanced LLMs leave a lot to be desired.
Grounding and eliminating hallucinations (Maynez et al., 2020), reasoning and logical clarity (Creswell & Shanahan, 2022), mathematics (Brown et al., 2020; Chowdhery et al., 2022; Noorbakhsh et al., 2021) are just a few examples where LLMs still lag behind the best human performances, or in some cases, the ﬁne-tuned performances of the same model.
While in many cases, scaling improve these qualities, it is possible that more than a few orders of magnitude in additional scale would be necessary to bridge these gaps, making a pure scaling solution impractical in the near term.
In addition, it is possible that some of these properties may not improve indeﬁnitely with larger scales.
For example, Longpre et al.
(2021) show that larger models have weaker context-based grounding properties as they tend to ignore the context more during reading comprehension question answering (QA).
How do we improve large language models beyond pretraining?
The most common practice is to ﬁne-tune it on a dataset specialized towards a downstream task - on most tasks, in-context learning does not achieve performances equivalent to supervised ﬁne-tuning.
As a result, the pretrainingﬁnetuning paradigm lives on even in the era of LLMs.
However, ﬁne-tuning causes specialization.
As we show in this paper, a ﬁne-tuned model usually forgets its ability to perform unseen tasks with few-shot in-context prompts.
In fact, the in-context performance could drop to none within one thousand steps of ﬁne-tuning.
[Table]
Mercedes’ Lewis Hamilton took the outright championship lead for the ﬁrst time this season with a dominant victory in the Italian Grand Prix.
[Table]
Hamilton won the British Grand Prix.
Traditionally ﬁne-tuned mT5 on RTE
True
ProMoT ﬁne-tuned mT5 on RTE
Lewis Hamilton won the Italian Grand Prix.
Loss of in-context abilities during ﬁne-tuning is problematic when we try to improve general qualities of LLMs such as grounding, reasoning, mathematics, utilizing external resources etc.
For example, grounding is desirable in a large number of tasks, including natural language inference (NLI), summarization, QA, conversation, data-to-text (Ji et al., 2022) and unforeseen tasks that could be performed in-context.
There is no shortage of quality data on a few speciﬁc instantiations of these general capabilities, e.g. high quality NLI datasets that in principle could teach the model the general concept of grounding, if there was no forgetting or specialization.
On the other hand, it is impossible to cover all tasks that use grounding in any ﬁne-tuning set.
Forgetting makes it difﬁcult to utilize these high quality datasets to build general capabilities into the model.
In this work, we discover that the loss of in-context abilities is, to a large extent, caused by format specialization: overﬁtting to the speciﬁc task format during ﬁne-tuning.
For example, if we ﬁnetune the model on NLI binary classiﬁcation, then it learns that the output can only be ”True” or ”False”, making it lose its original ability to ﬂexibly generate different output styles according to the in-context prompts of other tasks.
In addition, we show that the format is usually learned ﬁrst during ﬁne-tuning, before the model fully learns the semantic content of the task.
We propose a simple solution to format specialization: PROmpt Tuning with MOdel Tuning (ProMoT).
ProMoT off-loads format learning to a small amount of task-speciﬁc parameters that are external to the model.
With ProMoT, we preﬁx a soft tunable prompt (Lester et al., 2021) to the input sequence during a 2-stage ﬁne-tuning process: we ﬁrst freeze the model and tune the prompt, then freeze the prompt and tune the model.
Since format information is learned ﬁrst, it mostly enters the soft prompt and is no longer required to be learned by the model.
At inference time, we use the model with the tuned prompt if the inference task has the same format as the ﬁne-tuning target task, otherwise we remove the tuned prompt and use the model directly.
Our experiments show that this simple method can signiﬁcantly alleviate the forgetting on in-context abilities during ﬁne-tuning, and also results in surprising positive transfer across tasks with totally different formats.
Fine-tuning the model only on an NLI binary classiﬁcation dataset, we are able to improve the general grounding properties of a mT5 XXL model, thus improving its in-context performance on summarization.
See Table 1 for a concrete example.
To summarize, our contributions are 4-fold:
• We show empirically that the in-context capabilities are lost during ﬁne-tuning, and that format specialization is one important cause for this loss.
We also discover that format specialization happens at the very beginning of the ﬁne-tuning.
• We propose a novel 2-stage ﬁne-tuning framework: PROmpt Tuning with MOdel Tuning (ProMoT
), where we utilize soft tunable prompts to absorb the task format during ﬁnetuning, thus alleviating in-context capabilities loss.
• We evaluate ProMoT on 10 different NLP tasks including classiﬁcation, summarization, translation and question answering tasks, and compare it with prompt tuning and classical ﬁne-tuning.
• Experimental results show the effectiveness of ProMoT: 1) ProMoT in general improves supervised performance beyond the limit of prompt tuning only.
In certain tasks it matches or slightly outperforms the supervised performance of vanilla ﬁne-tuning; 2) ProMoT signiﬁcantly alleviates the forgetting of in-context capabilities on all tasks; 3) When ﬁnetuned on a single task, ProMoT results in positive task transfer across very dissimilar tasks when they share some semantic aspects; 4) When used in a sequential or parallel multi-task setting, ProMoT result in better models for most unseen in-context learning tasks.
These properties makes ProMoT a promising method to build general purpose capabilities into LLMs with small ﬁne-tuning datasets;
R ELATED W ORK
Large language models are shown to have the abilities to do in-context learning, where they learn a new task during the inference stage by conditioning on a few training examples (Raffel et al., 2020b;
Xue et al., 2020; Radford et al., 2018; Chowdhery et al., 2022).
Chan et al.
; Gao et al.
(2020) studies the effect of pretraining data distribution on in-context learning abilities on image recognition tasks, where the tension between in-context learning tasks and ﬁne-tuning tasks is discussed.
They propose changing the data distribution to ease such tension, which could be difﬁcult for generative
NLP tasks.
ProMoT is an orthogonal method that does not require changes in data distribution.
Related is the notion of catastrophic forgetting, usually analyzed with a sequence of image classiﬁcation tasks.
Ramasesh et al.
(2022) found that as model size increases, the model becomes less prone to catastrophic forgetting.
However these works are mostly restricted to tasks of similar format, e.g. classiﬁcation with the same number of classes.
In this work we are interested in vastly different tasks, e.g. classiﬁcation vs long form generation where the format itself is critical.
Instead of updating the model parameters, prompt-tuning (Lester et al., 2021; Zhang et al., 2021) appends continuous trainable embeddings (soft prompt) before the inputs and optimizes the prompt.
Prompt-tuning underperforms ﬁne-tuning in many cases, as shown in (Lester et al., 2021; Liu et al., 2021) and in our results.
Prompt-tuning does not change the parameters of the pretrained model, thus cannot improve the pretrained model itself using the ﬁne-tuning datasets.
Similar as prompttuning, some other works propose to adapt large language models to a new task by only tuning a small set of additional parameters (Hu et al., 2021;
He et al., 2021).
Another line of work uses ﬁne-tuning to teach the model in-context few-shot or zero-shot abilities, providing a meta learning approach to in-context learning.
Wei et al.
(2021) ﬁne-tunes the pretrained model on large-scale multitask datasets with diverse natural language prompts, improving the zeroand few-shot performance on unseen tasks.
Min et al.
(2021) incorporates the in-context learning objective into ﬁne-tuning on multitask datasets with few-shot prompts.
Such methods require ﬁnetuning on a large collection of different tasks to generalize, with each task being one datapoint of meta-learning.
In contrast, ProMoT is not limited to this regime and shows positive task transfer with a single ﬁne-tuning task.
In addition, such approaches often require human engineered instructions or prompts for each task, a volatile process involving lots of trial and error.
ProMoT does not require extensive prompt engineering as it optimizes the soft prompts with data.
Finally, ProMoT is largely orthogonal to meta-learning style methods.
They can be combined by applying ProMoT to a massively multitask setting where further improved generalization is expected.
One could also apply
ProMoT directly to the meta task of learning to solve in-context tasks.
This, however, is beyond the scope of our current paper and will be left for future exploration.
W HY ARE IN - CONTEXT ABILITIES LOST DURING FINE - TUNING ?
In this section, we show empirically that 1) in-context abilities are lost during ﬁne-tuning; 2) format specialization is an important cause for such loss; 3) format specialization happens at the very beginning of supervised ﬁne-tuning.
3. L OSS OF IN - CONTEXT CAPABILITIES DURING FINE - TUNING
Pretrained LLMs achieve good performance with in-context learning.
However, in many applications one may want to ﬁne-tune LLMs.
In this section, we show that the in-context learning performance usually drop signiﬁcantly after vanilla ﬁne-tuning.
To demonstrate this effect
[Figure]
we ﬁne-tune a pretrained mT5 XXL model(13B parameters) (Xue et al., 2020) on the Recognizing Textual Entailment (RTE) dataset (Wang et al., 2019), which is a binary classiﬁcation dataset predicting whether two given sentences are entailed.
We ﬁne-tune the model with default hyper-parameters for mT5 and task format used in PaLM (Chowdhery et al., 2022) where the output labels are True and False.
[Figure] [Figure]
To probe the model’s in-context capabilities during ﬁne-tuning
[Figure]
we evaluate with two openbook QA tasks, trivia-qa (Joshi et al., 2017) and web questions (Berant et al., 2013) with 1-shot in-context prompts in a closed book setting.
Figure 1 shows that when the accuracy on RTE dataset increases with ﬁne-tuning, the in-context abilities drop drastically.
We show that this phenomenon is generic and not a result of speciﬁc ﬁne-tuning or evaluation tasks with more results in Section 5.2.
F ORMAT SPECIALIZATION AS A CAUSE OF FORGETTING
LLMs such as mT5 XXL have the capacity to handle a large number of tasks.
So why are the incontext few-shot abilities easily lost after a few hundred steps of ﬁne-tuning?
A natural hypothesis is that due to the homogeneity of output formats in ﬁne-tuning datasets, the model quickly learns to follow this output format no matter what the input sequence is.
This leads to loss of in-context learning abilities on other tasks that do not share the format of the ﬁne-tuned task.
To be more speciﬁc, by ”format” we refer to the characteristics of the ﬁne-tuning labels as a subset of all possible output sequences.
For example, the format of RTE is a set of two labels, ”True” or ”False”, among all possible sequences of tokens of various lengths.
More generally, format might include factors such as the language used, typical output lengths and styles, special tokens or punctuation, upper/lower case styles etc.
that are common among most data points of a ﬁne-tuning task.
Since, by deﬁnition, most data points share the same format, the model receives a strong gradient signal that the output should follow this format independent of the input signal, thus its in-context performance on tasks with any other format will drop drastically, even if they share important semantic similarities with the ﬁne-tuned task.
To verify this hypothesis, we ﬁne-tune the mT5 XXL model on RTE task (classiﬁcation) and then evaluate on TriviaQA task (question answering task) with 1-shot prompt.
In evaluation, we count the percentage of outputs which are ’True’ or ’False’ in the test set.
Figure 2 shows that as the ﬁnetuning proceeds, the model outputs more ’True’ or ’False’ even with a 1-shot prompted input from
TriviaQA.
In particular, after 300 ﬁne-tuning steps, 90% of the output becomes ’True’ or ’False’.
The same phenomenon happens on other in-context tasks.
For 1-shot WMT16 En-De translation, after 500 steps, more than 99% of the output becomes ’True’ or ’False’.
This indicates that format specialization is an important reason for loss of in-context capabilities during ﬁne-tuning.
3. F ORMAT LEARNING HAPPENS FIRST DURING STANDARD FINE - TUNING
[Figure]
Next, we show experimental evidence that format learning happens ﬁrst during standard ﬁnetuning.
This is not surprising as the overwhelming majority of ﬁne-tuning data points has very similar formats, causing a gradient signal that dominates over other, more nuanced elements such as the semantic content of the task.
More concretely, for the RTE dataset, the format refers to the fact that output ∈ { True , False } , while the semantic content refers to the correlation between the input sequence and the output label.
We isolate format learning from semantic learning by creating a randomized RTE dataset where the output labels are randomly shufﬂed, thus are no longer correlated with the input sequences.
The gradients of format learning, g format , are then given by the gradients on the randomized RTE dataset.
By comparing with the gradient g on the original RTE we can probe as to when format learning happens during ﬁne-tuning.
As we can see from Figure 3, at the very beginning of ﬁne-tuning (step 0), The full RTE gradient g is highly aligned with the format only gradient g format , signiﬁed by cos( (cid:104) g 0 , g format,0 (cid:105) )
≈ 1 .
Since randomized RTE and original RTE share the format information only and contain totally different semantic content, this alignment implies that the model is mostly learning the format.
After 400 ﬁne-tuning steps, this alignment disappears where the cosine similarity drops to around 0 .
1 Comparison between more steps can be found in
Appendix Figure 8.
P ROPOSED M ETHOD : PRO MPT TUNING WITH MO DEL T UNING (P RO M O T)
The observations from Section 3 inspire us to decompose format learning from ﬁne-tuning, which can alleviate severe forgetting of in-context abilities during ﬁne-tuning.
The key idea is to ofﬂoad format learning in a separate set of parameters external to the model during the earliest training stage, and then train the model which will now focus more on the semantic skills.
We propose a two-stage ﬁne-tune strategy called ProMoT, illustrated in Figure 4.
At the ﬁrst stage, ProMoT uses prompt-tuning to capture the format in a trainable soft prompt while the model itself is frozen.
At the second stage, ProMoT freezes the learned soft prompt and ﬁne-tunes the model itself to focus on semantic skills that might be more transferable.
Stage 1: Prompt-Tuning.
Instead of updating the weight parameters, Prompt-tuning (Lester et al., 2021) prepends a continuous trainable prompt (soft prompt) before the embedded inputs and then updates this soft prompt during training.
The soft prompt for a given ﬁne-tuned task P e ∈ R p
×
e is a small set of free parameters taking the form of a few trainable embeddings, where p is the prompt length and e is the embedding size.
Given an input sequence from the task ( x 1 , ..., x n ) , prompttuning ﬁrst embeds the input sequence with the embedding layer in the pretrained model, and then concatenates the soft prompt before the embedded input.
The soft prompt is then optimized to reduce the loss while the pretrained model is frozen.
As indicated in Section 3.3, ﬁne-tuning ﬁrst learns the format.
We expect that by prompt tuning ﬁrst, the soft prompt will learn the format.
Although we cannot guarantee that the soft prompts only learns the format, the small capacity can prevent the
[Figure]
soft prompt from learning all semantic skills in most realistic NLP tasks, as demonstrated by the performance gap between prompt tuning and standard ﬁne-tuning.
Stage 2: Fine-tuning with trained prompt.
After prompt-tuning, we expect the trained prompt to already store the format information.
We then freeze the soft prompt and ﬁne-tune the pretrained model.
Importantly, as shown in Figure 4, the soft prompt is still prepended before the input during this stage, forcing the model to learn things not captured already by the soft prompt.
Evaluation.
After the two-stage ﬁne-tuning, we obtain a ﬁne-tuned model checkpoint and a trained soft prompt for this speciﬁc ﬁne-tuning target task.
We expect the soft prompt stores most of the format information, and we only use this prompt during inference when the inference task has the same format as the ﬁne-tuned target task.
Otherwise, we remove the learned prompt and simply feed the original input into the ﬁne-tuned model.
ProMoT is a general framework that can be combined with different prompt-tuning and ﬁne-tuning techniques.
For example, we can use few-shot ﬁne-tuning at the second stage where the inputs are appended with few-shot prompts, which can further improve the few-shot performance as proposed by Min et al.
E XPERIMENTS 5. S ETTINGS
Datasets.
We use RTE (Wang et al., 2019) and WMT14 En-Fr as our two ﬁne-tuning target tasks.
RTE is a binary classiﬁcation task and WMT14 En-Fr is a translation task.
They are selected as examples of classiﬁcation and generative tasks.
To evaluate in-context abilities, we use four types of tasks including natural language inference from superGLUE (Wang et al., 2019), translation from WMT, QA including triviaQA (Joshi et al., 2017) and web questions (Berant et al., 2013) that are evaluated in a closed book setting, and summarization with XSum (Narayan et al., 2018) and WikiLingua (Ladhak et al., 2020).
For each task, we use 1-shot and 4-shots in-context prompts where the prompt template is the same with PaLM (Chowdhery et al., 2022).
Input templates and output post-processing can be found in the Appendix.
For classiﬁcation tasks, we report the accuracies.
For QA tasks, we report the exact match ratio.
For translation tasks, we report the BLEU score (Papineni et al., 2002).
And for summarization tasks, we report the Rouge-2 score (Lin, 2004) .
We report the results on development set for classiﬁcation tasks (RTE, CB and WiC) and results on test set for all other tasks.
Models.
We use the mT5 (Xue et al., 2020) checkpoint for T5x XXL model (Raffel et al., 2020b) in all of our experiments, which contains 13B parameters.
mT5 is pretrained on a large-scale multilingual dataset, which makes it a good choice for the translation tasks used in our experiments.
Comparing methods.
In our experiments, we have several different training conﬁgurations and baselines, including
• Traditional ﬁne-tuning: Fine-tune the pretrained model without trainable prompt.
• Prompt-tuning:
Tune the trainable prompt with pretrained model frozen.
The in-context learning performance of prompt-tuning is the same as the pretrained model.
• ProMoT:
Our proposed two-stage ﬁne-tuning strategy.
• Fine-tuning/Prompt-tuning/ProMoT with 1-shot: For each input/output pair during training and evaluation, we sample and prepend a 1-shot example from the training split.
Hyper-parameters.
For all mT5 models, we ﬁne-tune with learning rate 0.001, drop rate 0.1 and label smoothing 0.1.
For all prompt-tuning experiments, we use learning rate 0.2.
For all tasks except summarization tasks, we choose the model input sequence length larger than the input length in datasets.
For summarization, we cut each input to 1024 tokens.
We use Adafactor optimizer and batch size 64 without data-packing across all experiments.
In inference, we use beam search to decode the outputs with width 4.
More experimental settings are provided in the appendix.
S INGLE T ASK F INE - TUNING
[Table]
[Table]
We ﬁne-tune the pretrained model with ProMoT on two target tasks and evaluate on several few-shot evaluation tasks.
At ProMoT stage 1, we run prompt-tuning for 5000 steps and save a checkpoint every 1000 steps, then select the prompt checkpoint with the best performance on target task.
At
ProMoT stage 2, we freeze the trained prompt and ﬁne-tune the model for 1000 steps, checkpointing every 100 steps.
We then pick the model checkpoint with highest performance on the ﬁnetuned task as our ﬁnal checkpoint.
For comparison, we run prompt-tuning and traditional ﬁne-tuning for and 1000 training steps respectively and report the performance of the best checkpoint.
We explore ﬁnetuning with more steps in Appendix A.2.
We list the results for RTE in Table 9 and WMT14 En-Fr translation in Table 3.
We see that after traditional ﬁne-tuning on RTE, the performance on binary classiﬁcation tasks (CB and WiC) with the same format may increase.
However, the performance on other tasks drops to nearly zero.
This further validates that ﬁne-tuned models suffer from format specialization as discussed in Section 3.
With ProMoT, the model achieves similar accuracy on RTE while forgetting much less on all fewshot evaluation tasks compared to standard ﬁne-tuned models.
More importantly, ProMoT models trained with only RTE outperform the pretrained model on summarization tasks like XSum and
WikiLingua.
Since ProMoT is a general framework, we can combine it with natural language fewshot prompts during ﬁne-tuning stage, which is shown in Min et al.
(2021) to improve in-context learning abilities.
More concretely, training input of ProMoT + 1-shot is the concatenation of the soft prompt, the natural language 1-shot prompt, and then the original input sequence.
Table 9 shows that ProMoT + 1-shot further boosts the performance over the original ProMoT on some tasks.
When we apply ProMoT ﬁnetuning on English to French translation, we ﬁnd similar trend (Table 3):
Compared to standard ﬁne-tuning, we observe signiﬁcantly less forgetting on in-context learning tasks across the board.
Compared to the pretrained model, we observe positive transfer on translation tasks for other language pairs as well as natural language inference binary classiﬁcations.
5. S EQUENTIAL T RAINING AND M ULTITASK T RAINING
As we can see from Section 5.2, ProMoT alleviates forgetting of in-context abilities across the board and has positive transfer on some tasks.
It is however not surprising that ProMoT on different tasks have different inﬂuences on in-context evaluation tasks.
For example, ProMoT on RTE causes some forgetting on translation while improving classiﬁcation and summarization.
On the contrary,
ProMoT on En-Fr has some forgetting on summarization while boosting other translation tasks.
This inspires us to apply ProMoT on multiple datasets for a generally better model.
In this section, we explore injecting knowledge into a single model with ProMoT by training on multiple tasks with sequential training and multitask training.
For sequential training, we ﬁrst run
ProMoT on the 1-shot RTE dataset, and then on 1-shot En-Fr translation dataset.
For multitask training, we mix the data from different tasks in one batch and train a soft prompt for each task.
All the training conﬁgurations are the same in Section 5.2.
We illustrate the results in Figure 5 and
Table 7.
In Figure 5, we ﬁrst report the accuracy on two ﬁne-tuning target datasets RTE and WMT14 En-Fr.
For sequential two-stage training, we evaluate the RTE accuracy using the trained prompt for RTE together with the ﬁnal model checkpoint after sequential training.
After the second ProMoT on EnFr translation, the accuracy on RTE does not drop much, revealing the potential of our method in the continual-learning space.
More importantly, after sequential ProMoT, we obtain a model with signiﬁcantly improved in-context performances compared to the pretrained model on both unseen classiﬁcation tasks (CB, WiC) and translation tasks (XSum, WikiLingua).
The exact numbers in
Figure 5 and more results on 4-shots evaluation can be found in Table 6 in the Appendix.
Results for multi-task training and related analysis can be found in Appendix A.2.
5. A BLATION S TUDY
In this section, we conduct several ablation study in Table 4 to show the effectiveness of our method.
First, we jointly ﬁne-tune both the soft prompt and the model at the same time.
This results in severe forgetting of in-context learning abilities similar to that of standard ﬁne-tuning.
We thus show that the beneﬁt of less forgetting and positive transfer comes from the two-stage nature of ProMoT instead of merely having the soft prompt.
During joint training, soft prompt and the main model are trained together for 1000 steps and then we evaluate the in-context 1-shot performance on evaluation tasks.
In addition, we also ﬁne-tune the models while attaching a random soft prompt that is ﬁxed during training.
As we expected, a random prompt contains no format information and does not help the model ofﬂoad format learning, resulting in severe forgetting.
Another natural baseline is to ﬁnetune the model with natural language prompts instead of soft prompts in place - after all, the natural language prompts contain few-shot examples that also capture the task format to some extend.
We compared this approach in a 1-shot scenario and did not observe signiﬁcantly reduced forgetting
[Figure]
[Table]
compared to standard ﬁne-tuning.
Most of in-context tasks still drop to zero performances.
This demonstrates that soft prompts in ProMoT works better than natural language prompts in capturing format information and alleviating the forgetting of in-context learning abilities during ﬁne-tuning.
C ONCLUSIONS
In this paper, we found one important cause of the lose of in-context abilities during LLM ﬁnetuning: format specialization.
Our analysis shows that format specialization happens at the very beginning of ﬁne-tuning.
This motivates us to develop ProMoT, a simple yet effective two-stage ﬁne-tuning framework that utilizes soft trainable prompts to absorb the task format before ﬁnetuning the model.
Our experimental results on a diverse set of NLP tasks show the effectiveness of ProMoT for preserving in-context capabilities during ﬁne-tuning.
ProMoT also shows surprising positive transfer across very different tasks, making it a promising method that could build general purpose capabilities into LLMs with small ﬁnetunig datasets.
We also explored sequential training and multi-task training on two training tasks which can achieve even better generalization.
A A PPENDIX A. E XPERIMENT D ETAILS
Input template used in experiments In Table 5, we list the natural language input template used in our experiments for each task
[Table]
Output post-processing For each task, we ﬁrst extract the text after < extra id 0 > and before < extra id 1 > , then trim the text by locating and remove the text after the second preﬁx token (Q:,
Translate, Article:).
For classiﬁcation tasks including RTE, CB and WiC, we check whether the ﬁrst output token is True or False.
A. A DDITIONAL E XPERIMENT R ESULTS
More results of sequential ProMoT training In Table 6 we show the exact numbers in Figureand additional evaluation results on 4-shots datasets.
[Table]
WiC 49.
triviaQA 17.
web questions 9.
WMT16 ende 3.
WMT16 enro 1.
XSum 6.
WikiLingua 4.
Training more steps: trade-off between ﬁne-tuning target task and in-context learning abilities
In Section 5.2, we report the results of the best checkpoints within 1000 steps of ﬁne-tuning.
With a longer training period, we can see a more clear trade-off between the performance on ﬁne-tuning target task and the performance on in-context learning abilities.
Here we show the long-term tradeoff between ﬁne-tuning target task and in-context evaluation tasks by scattering the performance of different checkpoints within 20000 steps ﬁne-tuning.
In ﬁgure 6, and 7, we plot the trade-off on classiﬁcation and translation tasks, respectively.
As we can see from the ﬁgures, datapoints for ProMoT is higher than traditional ﬁne-tuning on the ﬁgures, which implies that with the same performance on ﬁne-tuning target task, forgetting is alleviated with ProMoT ﬁne-tuning.
[Figure]
[Figure]
Multi-task training
Another commonly-adopted method to train a model on multiple tasks is parallel multi-task training, especially when we known the training tasks beforehand.
During multi-task training, data points from different tasks are sampled according to corresponding ratios to construct the training batches.
Multi-task training has been shown to improve the generalization ability of large language models during ﬁnetuning, often with natural language prompts attached to distinguish the tasks (Sanh et al., 2021;
Wei et al., 2021; Min et al., 2021).
In this section, we apply
ProMoT on RTE and WMT14 En-Fr in a multi-task ﬁnetuning setting.
We show that ProMoT generalize substantially better on a subset of in-context tasks than traditional multi-task ﬁnetuning.
At the ﬁrst stage, we train a different soft prompt for each task.
At the second stage, we prepend the corresponding trained and frozen prompt on each training example and tune the pretrained model on a mixture of data from different tasks.
We use the same training and evaluation tasks and keep most of hyper-parameters the same except doubling training steps in the ﬁrst stage.
We show the results in Table 7, both on ﬁne-tuned tasks and
[Table]
task ProMoT, forgetting on QA and summarization are alleviated compared to sequential ProMoT.
Besides, although traditional multi-task ﬁne-tuning can improve model generalization and thus alleviate the forgetting problem on some tasks, ProMoT multi-task training has much better transferred performance on translation tasks on unseen language pairs.
Additional experiments on T5 and FLAN-T5 To show the performance of our method on
English-based pretrained model, we did two additional experiments on FLAN-T5 and T5 XXL with ﬁne-tuning target task RTE.
[Table] [Table]
WMT16 ende 0. WMT16 enro 0. XSum 1. WikiLingua 1.
[Figure]
a t , i > )
[Figure]
Plotting more steps for Figure 3 To further strengthen our conclusion in Figure 3, here we plot the gradient alignment from step 0 to step 400.
As we can see from the ﬁgure, gradient alignment drops signiﬁcantly after 300 steps which is matched with Figure 2 where the true and false ratio increases before 300 steps and then remains stable.