To Adapt or to Fine-tune: A Case Study on Abstractive Summarization (2022)

Abstract: “Recent advances in the field of abstractive summarization leverage pre-trained language models rather than train a model from scratch. However, such models are sluggish to train and accompanied by a massive overhead. Researchers have proposed a few lightweight alternatives such as smaller adapters to mitigate the drawbacks. Nonetheless, it remains uncertain whether using adapters benefits the task of summarization, in terms of improved efficiency without an unpleasant sacrifice in performance. In this work, we carry out multifaceted investigations on fine-tuning and adapters for summarization tasks with varying complexity: language, domain, and task transfer. In our experiments, fine-tuning a pre-trained language model generally attains a better performance than using adapters; the performance gap positively correlates with the amount of training data used. Notably, adapters exceed fine-tuning under extremely low-resource conditions. We further provide insights on multilinguality, model convergence, and robustness, hoping to shed light on the pragmatic choice of fine-tuning or adapters in abstractive summarization.”

## introduction in the current era of research, using large pre-trained language models (plm) and fine-tuning these models on a downstream task yields dominating results in many tasks @xcite @xcite . the scope of our work is on abstractive summarization, which is the task of generating a concise and relevant summary given a long document. recent works have demonstrated the success of fine-tuning plms on summarization @xcite @xcite . nonetheless, such a paradigm becomes increasingly expensive with the ever-growing sizes of plms, since both the training time and space requirement increase along with the number of parameters. the issue becomes more severe when multiple languages or domains are introduced, as separate models need to be trained and saved depending on the setup. @xcite proposed lightweight adapters as an alleviation of the large overhead of finetuning plm on a downstream task. while many researchers have followed and adopted their idea, experiments are rarely done on summarization; from both quantitative and qualitative perspectives, it remains a myth of which direction one should pick in practice. in this work, we perform a thorough exploration of using adapters with a plm on the task of abstractive summarization by examining different scenarios. our experiments are designed along three dimensions: 1) languages involved: monolingual, crosslingual, and multilingual; 2) data availability: high, medium, low, and scarce; 3) knowledge being transferred: languages, domains as well as tasks. through comprehensive experimental results, we demonstrate that with a realistic availability of resources, fine-tuning a plm is superior to using adapters for the purpose of obtaining the best text quality. however, the game changes under low-resource settings: adapters have shown better, if not, on par performances compared to fine-tuning, especially in domain adaption. ## conclusions and future work with large plms coming to light, we investigate fine-tuning and adapter strategies for transfer learning in abstractive summarization. we demonstrated that the performance gap between the two strategies is positively correlated with the availability of training resources, despite the languages being tested. further analysis on domain adaptation and task adaption produces agreeing observations. we conclude that for realistically large summarization datasets, full fine-tuning will guarantee the best output quality. on the other hand, when resources are scarce, the advantages of adapters emerge in the niche market. most summarization datasets are web-crawled or machine-translated, resulting in non-optimal data quality. we plan to perform more qualitative analysis on the model outputs such as linguistic interpretation and human evaluation. in addition, we only experimented with fine-tuning and using adapters on mbart and bart for abstractive summarization, so there is room for research on other large plms, as well as other nlp tasks in the future.