Diffusion Language Model with Query-Document Relevance for Query-Focused Summarization (2023)

Abstract: Query-Focused Summarization (QFS) aims to generate summaries from source documents that can answer specific queries. Although the QFS task has gained increasing attention recently, its development is constrained by the fact that mainstream QFS models are BART variants, which are autoregressive and suffer from long-term dependencies and exposure bias. To address these problems, we adopt a diffusion language model that performs well in non-autoregressive scenarios to effectively resolve issues related to autoregressive methods. However, QFS requires guidance from queries to generate adequate summaries, while diffusion language models have limited sensitivity to queries. In this paper, we propose QFS-DLM, a non-autoregressive diffusion language model that incorporates query-document fragment relevance and query-document global relevance to enhance the adaptability of QFS tasks. Firstly, we extract key fragments from documents based on queries and assign higher weights to them, thereby emphasizing crucial and continuous information within the document. Secondly, we calculate global relevance scores between queries and documents, and then integrate these scores into the model’s loss function, enabling the model to prefer high-quality data and distance itself from low-quality data. Overall, our method achieves state-of-the-art performance on Debatepedia and PubMedQA datasets in ROUGE scores, GPT-4, and human evaluations.

## introduction query-focused summarization (qfs) aims to generate summaries that address specific queries by extracting crucial information from source documents @xcite . inspired by the significant generation capabilities of bart @xcite , researchers manage to adopt it into qfs * *corresponding author. ## methodology in this section, we present our approach to incorporating query-document relevance into the diffusion language model. first, we describe the method of incorporating query-document fragment relevance. then, we introduce the query-document global relevance, which is only used during the training phase, and our model architecture as shown in figure 1 . based on the performance of the diffusion language model genie foot_1 @xcite in the general text summarization, we choose it as our base model. the genie has undergone extensive pretraining and adopts the sequence-to-sequence framework. unlike the traditional autoregressive text generation paradigm that generates one token at a time, genie parallelly outputs embedding sequences at each denoising step, making it a nonautoregressive model. our parameters is presented ## conclusion in this work, we propose qfs-dlm, a diffusion language model that incorporates query-document relevance. firstly, we extract key fragments from documents based on queries and assign higher weights to them, thereby emphasizing crucial and continuous information within the document. then, we calculate global relevance scores between queries and documents, and then integrate them into the model's loss function, enabling the model to prefer high-quality data and distance itself from low-quality data. this approach generates summaries that are more focused on query-related information, improving the performance and adaptability of the diffusion language model for the qfs task. experimental results demonstrate that our model achieves state-of-the-art performance on the debatepedia and pubmedqa datasets.