Instructive Dialogue Summarization with Query Aggregations (2023)

Abstract: Conventional dialogue summarization methods directly generate summaries and do not consider user’s specific interests. This poses challenges in cases where the users are more focused on particular topics or aspects. With the advancement of instruction-finetuned language models, we introduce instruction-tuning to dialogues to expand the capability set of dialogue summarization models. To overcome the scarcity of instructive dialogue summarization data, we propose a three-step approach to synthesize high-quality query-based summarization triples. This process involves summary-anchored query generation, query filtering and query-based summary generation. By training a unified model called InstructDS (Instructive Dialogue Summarization) on three summarization datasets with multi-purpose instructive triples, we expand the capability of dialogue summarization models. We evaluate our method on four datasets, including dialogue summarization and dialogue reading comprehension. Experimental results show that our approach outperforms the state-of-the-art models and even models with larger sizes. Additionally, our model exhibits higher generalizability and faithfulness, as confirmed by human subjective evaluations.

## introduction both verbal and non-verbal conversations play a crucial role in the realm of communication. they serve as channels for humans to exchange information, ideas and emotions @xcite . in an era of information explosion overload, dialogue summarization has become increasingly essential. the process involves extracting the key dialogue information, enabling people to grasp the essence of their daily interactions. conventional dialogue summarization models typically approach the problem as an unconstrained sequence-to-sequence task, treating dialogue-summary pairs as straightforward inputoutput pairs @xcite @xcite . although fine-tuning pre-trained language models such as bart @xcite has shown promising results, these models fail to consider the specific preferences of users who have distinct backgrounds, objectives, intents, and applications for the summaries they require. in order to address this challenge, several methods have been proposed to integrate queries when generating summaries @xcite @xcite @xcite . however, these models primarily concentrate on domains such as news @xcite , wikipedia @xcite , and meetings @xcite . the exploration of query-based summarization for dialogues remains limited. furthermore, @xcite propose controllable generation using personal named entity planning, and @xcite suggest controlling summary conciseness. however, both methods focus on specific aspects of controllability and still lack the flexibility to incorporate user requirements as shown in figure 1 . the primary obstacle in instruction-based dialogue summarization is the scarcity of training data. while existing datasets contain dialogue-summary pairs, creating query-based dialogue summarization datasets with limited human involvement is challenging due to high costs, limited diversity, and potential quality issues. in this work, shed light by self-instruct @xcite , we propose to synthesize query-dialogue-summary (qds) triples by leveraging the conditional question generation and answering ability of general large language models (llms) @xcite . the process involves requesting llms to generate multiple candidate queries based on the reference summary. a filtering mechanism, employing text-based and semantic-based methods, is then applied to ensure the quality of collected queries. finally, the querybased summarization is generated by triggering the question-answering ability of llms. this approach demonstrates a promising solution to generate query-based dialogue summarization triples while reducing human involvement and enhancing data diversity. the instructds framework is shown in figure 2 . through joint training with qds triples, instructds can cater to user preferences by producing querybased summaries. this mixed training paradigm enhances the model's understanding of dialogue, which improves the factual quality of generated summaries. our model exhibits superior domain generalizability by incorporating multiple datasets into a unified framework and the user's conciseness requirement can be fulfilled by our length-aware augmentations. our main contributions are summarized as follows: this method leverages the question generation and answering capabilities of large language models (llms). we validate its effectiveness through evaluations conducted by human annotators. ## conclusion in this work, we present instructds, which is the first instructive dialogue summarization model that excels in both general dialogue summarization and query-based dialogue summarization. instructds can generate high-quality query-based summarization tailored to user requirements. this achievement is made possible through a combination of multi-dataset training, synthesized qds triples and length-awareness augmentations. experimental results demonstrate that instructds establishes new state-of-the-art performance across all four benchmark datasets.