Evaluating the Generalization Property of Prefix-based Methods for Data-to-text Generation (2023)

Abstract: Fine-tuning is the prevalent paradigm to adapt pre-trained language models to downstream tasks. Lightweight fine-tuning methods, such as prefix-tuning, only tune a small set of parameters which alleviates cost. Such methods were shown to achieve results similar to fine-tuning; however, performance can decrease when the inputs get farther from the training domain. Moreover, latest works questioned the efficiency of recent lightweight fine-tuning techniques depending on the task and the size of the model. In this paper, we propose to evaluate the generalization property of prefix-based methods depending on the size of the pre-trained language model in the multi-domain setting on data-to-text generation. We found that their performance depends heavily on the size of the model.

## introduction fine-tuning is the prevalent approach to adapting pre-trained language models (plms) to various downstream tasks @xcite . however, fine-tuning is both computationally and memory expensive. lightweight fine-tuning methods address this problem by freezing most of the plms parameters, e.g. fine-tuning the top layers, or by training only a smaller set of added parameters. a first method, adapter-tuning @xcite , inserts task-specific layers between the layers of plms. @xcite presented a second method, prefix-tuning, where a prompt is optimized as hidden-states, i.e. key-value for transformer @xcite . their method outperformed adapter-tuning and also achieved comparable performance with fine-tuning on data-totext benchmarks when using gpt-2 medium and large @xcite . @xcite introduce a third lightweight fine-tuning method, prompt-tuning, in which continuous embeddings are optimized as soft prompts. their experiments show that the performance gap between prompt-tuning and fine-tuning reduces with increase of model's size. in this paper, we focus on prefix-based methods. lightweight fine-tuning methods are also used for multi-task learning. hyperformer @xcite builds on adapter-tuning and uses hypernetworks to generate task and layer-specific adapter parameters, conditioned on task and layer embeddings. when published, hyperformer achieved better performance on average on the glue benchmark (wang et al., 2018) compared to finetuning and adapter-tuning, when using t5-small and t5-base @xcite . @xcite propose hyperprompt, a multi-task method based on prefix-tuning and hyperformer's work, which outperforms hyperformer. they showed that only tuning the added parameters of hyperformer and hyperprompt did not lead to similar performance when compared to tuning both added parameters and the plm's parameters on superglue (wang et al., 2019), a more difficult nlu benchmark, when using t5-large. this observation questions the efficiency of lightweight fine-tuning since the performance of those methods seem to vary depending on the task difficulty. clive et al. (2022) concatenate two prefixes : one for the task and one for the domain. we consider here data-to-text generation tasks, consisting in generating fluent descriptions of data available in table or graph format and typically extracted from databases. we consider a challenging multi-domain setting where data is supposed to come from multiple sources, each with its own term and term-relations distributions. this could be considered as a specific multi-task problem where each domain corresponds to a task. the challenge here is to adapt plms in order to handle this multi-domain setting when current practice considers mono-domain settings. we then propose to evaluate the generalization property of prefix-based methods in the multi-domain setting, both in zero-shot and after few-shot fine-tuning on new target domains. with the question of efficiency depending on the model's size in mind, we compare results for t5-small and t5-base transformers. ## conclusion our study of prefix-based methods for transformer shows that their performance depends heavily on the plm's size. scores of best prefix-based models were only comparable to fine-tuning for both trained and new categories when using t5-base. we also observe a difference of behaviour linked to the model's size for few-shot domain adaptation, where prefix-based models are able to reach results of fine-tuning as the number of training samples increases only with t5-base. in the context of saving memory, additional experiments on few-shot domain adaption could be done : we could freeze the hypernetworks in hyperprefix and hyperprompt-share and only train the new global prompts associated with the new domains. this type of few-shot domain adaption would be interesting since we would only need to save an additional matrix of dimension (l × d).