Hypernetwork-Assisted Parameter-Efficient Fine-Tuning with Meta-Knowledge Distillation for Domain Knowledge Disentanglement (2024)

Abstract: Domain adaptation from labeled source domains to the target domain is important in practical summarization scenarios. However, the key challenge is domain knowledge disentanglement. In this work, we explore how to disentangle domain-invariant knowledge from source domains while learning specific knowledge of the target domain. Specifically, we propose a hypernetwork-assisted encoder-decoder architecture with parameter-efficient fine-tuning. It leverages a hypernetwork instruction learning module to generate domain-specific parameters from the encoded inputs accompanied by task-related instruction. Further, to better disentangle and transfer knowledge from source domains to the target domain, we introduce a meta-knowledge distillation strategy to build a meta-teacher model that captures domain-invariant knowledge across multiple domains and use it to transfer knowledge to students. Experiments on three dialogue summarization datasets show the effectiveness of the proposed model. Human evaluations also show the superiority of our model with regard to the summary generation quality.

## introduction recently, domain adaptation for text summarization has attracted much research interest @xcite @xcite . most prior work performs pretraining on large-scale out-of-domain datasets and then adapts to the in-domain summary data. for dialogue summarization, a couple of studies have leveraged large-scale summary data that is fairly distinct from the dialogue domain, e.g., the news domain, to facilitate dialogue summarization @xcite in few-shot settings. however, this fails to acknowledge the huge gap between dialogue and general articles, e.g., that dialogue involves a dynamic information exchange flow with multiple interlocutors @xcite . recent work explored prompt-based fine-grained transfer learning between various dialogue domains in zero-shot settings @xcite . however, these studies did not consider how to transfer knowledge from the source domains to the target domain, and @xcite pointed out that directly fine-tuning the prompt initialized with the source prompt on target domain might lead to catastrophic forgetting of source knowledge. considering a typical example in figure 1 , domain adaptation aims to improve the generalizability of the model from the source domains to the target domain, however, the key challenge is the disentanglement of domain knowledge, whereby various domains contain domaininvariant and domain-specific knowledge which are always entangled. for example, we may take academic and product as source domains, and committee as the target domain, where academic consists of academic meetings, committee contains formal discussions on a wide range of issues (e.g., the energy market), and product focuses on product design in an industrial setting. although the content discussed in the three domains is different, the key characteristics of the dialogue are the same (e.g., multiple participants, and a dynamic information exchange flow). this phenomenon suggests that the model needs to learn domain-invariant characteristics (that is, characteristics of the dialogue) in the source domains while focusing on what is being discussed in the specific domain. inspired by the recent success of performing new tasks through the use of instructions alone @xcite , and considering the inherent problems faced by domain adaptation, in this work, we propose a novel hypernetworkassisted encoder-decoder based architecture with parameter-efficient fine-tuning, which leverages a phd d: actually i forgot to say that the multilingual net are trained on features... ## approach overview. to address the domain knowledge disentanglement, we propose a novel hypernetworkassisted encoder-decoder based architecture with parameter-efficient fine-tuning. figure 2 gives an overview of our approach, which leverages a hypernetwork instruction learning (hil) module to generate domain-specific parameters (decoder adapters) for the underlying pre-trained language model (plm), and applies meta-knowledge distillation to disentangle and transfer domain-invariant features for parameter-efficient cross-domain learning. ## case study we conduct a case study with an example from qmsum to illustrate the advantages of our model. furthermore, we explore applying llm to a specific domain of dialogue summarization through in-context learning. in ## conclusion in this work, we propose a novel hypernetworkassisted encoder-decoder architecture with metaknowledge distillation for domain knowledge disentanglement in cross-domain dialogue summarization. it leverages hypernetwork instruction learning to generate preferable domain-specific adapter parameters and disentangles and transfers domaininvariant features to better improve cross-domain transferability with a model-agnostic distillation strategy. our model achieves strong results on diverse datasets with several different settings.