PEK : A Parameter-Efficient Framework for Knowledge-Grounded Dialogue Generation (2024)

Abstract: Pre-trained language models (PLMs) have shown great dialogue generation capability in different scenarios. However, the huge VRAM consumption when fine-tuning them is one of their drawbacks. PEFT approaches can significantly reduce the number of trainable parameters, which enables us to fine-tune larger dialogue generation models. However, the reduction in parameter quantity can diminish a PLM’s expressive capacity and affect the PLM’s learning from certain specific examples like knowledge-related conversations. Previous works have demonstrated that injecting external knowledge into dialogue generation models can improve the model’s performance in knowledge-related conversations. Nonetheless, these methods are designed for the scenario where most parameters of the entire framework are trainable. In this paper, we propose PEK, a parameter-efficient framework for knowledge-enhanced dialogue generation. It enables PLMs to leverage external knowledge documents and knowledge graphs to enhance its generation capabilities with an acceptable number of trainable parameters. Evaluation results on the Wizard of Wikipedia and CMU_DoG datasets show that our approach outperforms baseline methods on multiple evaluation metrics, which validates the effectiveness of our approach.

## introduction the success of pre-trained models like bert @xcite , gpt-2 @xcite , bart @xcite and t5 @xcite , demonstrates that more parameters and larger datasets, can lead to better language representation capabilities and great performances on various nlp tasks. generally, fine-tuning on downstream task datasets helps further improve a pretrained model's performance. however, since the advent of pre-trained language models, one of their * corresponding author. an effective solution is using parameter-efficient methods to reduce the number of trainable parameters. however, reducing the number of trainable parameters may lead to a decrease in model performance. finding ways to maintain model performance is a challenging problem. several approaches have been explored, like @xcite , adapter @xcite @xcite , p-tuning @xcite , prompttuning @xcite , lora @xcite and adalora @xcite . nonetheless, these methods still have some shortcomings, such as a significant drop in model performance on certain tasks, introduction of training or inference latency, or imprecise allocation of computational resources. using peft approaches, we can significantly reduce the number of trainable parameters, which enables us to fine-tune larger dialogue generation models. however, the reduction in parameter quantity can diminish a plm's expressive capacity and affect the plm's learning from certain specific examples like knowledge-related conversations. pretrained models acquire a considerable amount of knowledge during the pre-training process. in the scenario of full fine-tuning, the model can implicitly learn to apply this knowledge in downstream tasks during the fine-tuning process. nevertheless, in the peft scenario, where the number of trainable parameters is significantly reduced, this transfer process may be affected. moreover, the knowledge obtained through pre-training is often not explicitly represented, which brings us challenges to explore better ways to utilize it. some works @xcite @xcite @xcite have attempted to integrate external knowledge such as documents and knowledge graphs into dialogue generation models and achieved remarkable results. the achievements of these methods have demonstrated to us that introducing external knowledge helps to enhance a dialogue generation model's generative capabilities in knowledge-related conversations. nonetheless, these methods are usually designed for the scenario where most parameters of the entire framework are trainable. how to inject external knowledge into plms to enhance its generation capabilities with a small number of trainable parameters in the entire framework remains a subject for further research. to address the issues mentioned above, we present a parameter-efficient structure which can introduce external knowledge graph to plms. our contributions in this paper can be summarized as: (1) we propose pek, a parameter-efficient knowledge-grounded dialogue generation framework, which enables plms to leverage external knowledge sources to generate knowledgeenriched responses and requires only a small number of trainable parameters; (2) we propose t-lora, which is based on svd decomposition and can allocate more computational resources to weight matrices with less redundancy. t-lora does not require a dynamic parameter sensitivity estimation process during fine-tuning, which also means that it does not consume additional computational resources during training; (3) we propose a geometric mean weighted attention mechanism. it can effectively measure the importance of knowledge graph triplets and mitigate the issue of knowledge noise introduced by irrelevant triplets. ## conclusion in this paper, we propose pek, a parameterefficient framework for knowledge-grounded dialogue generation. we make an improvement to the vanilla lora, which can allocate more computational resources to matrices with less information redundancy. moreover, to incorporate knowledge graphs into dialogue generation, we introduce a geometric mean weighted mechanism. the evaluation results on both the wizard of wikipedia dataset and the cmu_dog dataset have shown that our method outperforms all the peft baselines and some of the non-peft methods.