Domain Aligned Prefix Averaging for Domain Generalization in Abstractive Summarization (2023)

Abstract: Domain generalization is hitherto an underexplored area applied in abstractive summarization. Moreover, most existing works on domain generalization have sophisticated training algorithms. In this paper, we propose a lightweight, weight averaging based, Domain Aligned Prefix Averaging approach to domain generalization for abstractive summarization. Given a number of source domains, our method first trains a prefix for each one of them. These source prefixes generate summaries for a small number of target domain documents. The similarity of the generated summaries to their corresponding source documents is used for calculating weights required to average source prefixes. In DAPA, prefix tuning allows for lightweight finetuning, and weight averaging allows for the computationally efficient addition of new source domains. When evaluated on four diverse summarization domains, DAPA shows comparable or better performance against the baselines demonstrating the effectiveness of its prefix averaging scheme.

## introduction abstractive document summarization aims at filtering the most crucial information in a document to present a concise view of it @xcite . this document may take the form of a news article @xcite , a scientific paper @xcite , a dialogue @xcite , or a social media post @xcite . the advent of pretrained models @xcite has significantly improved abstractive summarization on several of the aforementioned domains. however, these approaches require extensive manual labelling of data which limits their use to domains without any labelled data. given that real-world applications of summarization often face the problem of adapting to new domains, it becomes crucial to develop summarization systems that do well in data-scarce settings by leveraging information from the source domains. domain generalization accounts for learning a robust model for unseen domains from a set of source domains. this problem is closely related to transfer learning, multitask learning and domain adaptation all of which involve learning a model from a set of source tasks/domains to perform well on a set of target tasks/domains. however, in the case of domain generalization, labelled data for the target domain is unavailable. previous works on domain generalization mainly focus on learning domain invariant features @xcite @xcite . such methods work well on classification tasks where learning domain invariant features is sufficient to predict target classes. however, they may be insufficient for language generation tasks which have writing style, grammar as their ingredients @xcite . moreover, such methods involve sophisticated algorithms for training and cannot be used for lightweight domain generalization. prefix tuning @xcite is a lightweight approach to adapting pretrained language models to downstream tasks. it augments the transformer self attention via prefix tokens learned through backpropagation on the task data while keeping the pretrained model's parameters frozen. prompt tuning based approaches have been shown to do well on lifelong learning @xcite and zero-shot domain adaptation @xcite , which inspires us to adapt it to domain generalization for abstractive summarization. concurrently, weight averaging has performed well on domain generalization tasks in computer vision @xcite @xcite . to improve functional diversity, these methods average model parameters from different runs and/or checkpoints. @xcite applied weight aver-aging to nlp tasks. they merged models trained on different tasks/domains through fisher weight averaging. their promising results motivate us to apply model merging through weight averaging for domain generalization. keeping in mind the goal of generating a lightweight and parameter-efficient approach, and the benefits brought by weight averaging, we propose a lightweight domain aligned prefix averaging, dapa, approach to domain generalization for abstractive summarization. our algorithm consists of three stages. first, prefixes are trained for each source domain. in the second stage, these source prefixes generate summaries for a small number of unlabelled target domain documents. in the third stage, the target domain prefix is obtained through a weighted average of these source prefixes. a higher document-summary similarity score, calculated from the summaries generated in the second stage, would assign a greater weightage to the corresponding source prefix. through our prefix averaging scheme, we can identify source prefixes essential to ensure good performance on the target domain. our extensive experimentation on four domains demonstrates the benefits bought by dapa. dapa comes with the following advantages: i) it is a lightweight approach to domain generalization since the backbone pretrained model is frozen and only source prefixes are trained. ii) through our novel prefix averaging scheme, dapa is able to generalize well onto target domains. moreover, freezing the backbone model's parameters further preserves generalization. iii) our approach supports the efficient addition of new source domains since it only involves recomputing the prefix averaging weights. to this end, we summarize our contributions as follows: the rest of the paper is structured as follows: we explore related works in section 2. section 3 describes our proposed approach dapa. section 4 and 5 provide results for our domain generation experiments and a set of analysis we conduct on our approach. section 6 provides the conclusion and thoughts for future work. section 7 discusses limitations and section 8 sheds light om ethical risks of our work. ## method we first describe the domain generalization problem in section 3.1. then we move on to describe prefix tuning in section 3.2. finally, in section 3.3, we describe our proposed approach, dapa. ## analysis to study the impact of various factors in dapa, and better understand its efficacy, we conduct a series of analysis. ## conclusion in this paper, we present dapa, a lightweight, domain aligned prefix averaging approach to domain generalization in abstractive summarization. dapa utilizes source prefixes to generate summaries for a small number of target domain documents. the similarity of these summaries to their corresponding documents are used for calculating weights required to average source prefixes. dapa can easily account for the addition of new source domains since only the prefix averaging weights need to be recomputed. on four diverse summarization domains, dapa either performs comparably or outperforms the baselines. we also perform an in-depth analysis of various components of dapa to further strengthen our design choices. in future, we would like to develop an improved similarity function f and analyze the loss landscapes of these models to corroborate our prefix averaging strategy.