Dialogue Summarization with Mixture of Experts based on Large Language Models (2024)

Abstract: Dialogue summarization is an important task that requires to generate highlights for a conversation from different aspects (e.g., content of various speakers). While several studies successfully employ large language models (LLMs) and achieve satisfying results, they are limited by using one model at a time or treat it as a black box, which makes it hard to discriminatively learn essential content in a dialogue from different aspects, therefore may lead to anticipation bias and potential loss of information in the produced summaries. In this paper, we propose an LLM-based approach with role-oriented routing and fusion generation to utilize mixture of experts (MoE) for dialogue summarization. Specifically, the role-oriented routing is an LLM-based module that selects appropriate experts to process different information; fusion generation is another LLM-based module to locate salient information and produce finalized dialogue summaries. The proposed approach offers an alternative solution to employing multiple LLMs for dialogue summarization by leveraging their capabilities of in-context processing and generation in an effective manner. We run experiments on widely used benchmark datasets for this task, where the results demonstrate the superiority of our approach in producing informative and accurate dialogue summarization.

## introduction dialogue summarization is a crucial task that aims to extract essential information from a dialogue, which attracts much attention from existing studies in recent years @xcite . different from documents that are monographs from one writer, dialogues involve contents from different roles and thus summarizing them needs to consider the interactions among existing studies for dialogue summarization @xcite @xcite @xcite tend to utilize end-to-end approaches to produce dialogue summaries, where advanced text encoders, such as large language models (llms), are used to identify the key content in the dialogue. to further enhance the capability of models to identify essential information in the dialogue, additional information is applied with the interaction among dialogue participants, the structure of the dialogue, and the topics they are discussing, etc. @xcite @xcite @xcite @xcite @xcite . although these studies obtain promising results, they are mainly performed with a single-model or black box design, thus is potentially limited in generating biased output in the dialogue summary where diversified information are not comprehensively processed. therefore, it is inevitable to consider whether there are more reasonable model designs to overcome such limitation. since mixture-of-experts (moe) with llms demonstrates their effectiveness in many tasks @xcite @xcite , it is inspired to leverage such design for dialogue summarization to fully leverage the potential of llms to understand and generate summaries from different aspects. however, applying moe to llms for dialogue summarization is not a trivial task, where careful design of the "mixture" is essential to prevent issues such as information dilution, so that essential contents are processed and preserved by appropriate experts, and the outputs from all experts retains a sufficient informationbearing without being distracted. therefore, in this paper, we propose an moe-based approach for dialogue summarization that alters the inner structure of llms. specifically, we perform an utterance based processing with role-oriented routing, which is a part of llm, to effectively identify the mapping of different experts for a particular utterance with the support of the entire dialogue, so as to avoid "blind men and the elephant" phenomenon foot_0 when comprehending the content and extracting key information from the utterance. then use another parts of llm as experts and use the routing to selected some of them to take the utterance and generate outputs based on essential content in it. the fusion generation uses another llm to combine the output of each expert and chooses the most valuable ones from them to form the final summary with all essential information from various aspects included in the dialogue. we run experiments on four widely used benchmark english and chinese datasets. the results and further analyses demonstrate the superiority of our approach, which outperforms strong baselines and achieves state-of-the-art performance on all datasets, also show the validity of our design for each component. ## the approach the overall architecture of our approach for dialogue summarization is illustrated in figure 1 , where the moe framework consists of three main components, namely, role-oriented routing (ror), expert processing, and fusion generation (fg). specifically, ror is based on the first k layers of an llm that takes the input dialogue @xmath0, u 1 ), • • • (r l , u l )] with l utterances associated with their speakers (i.e., r l denotes the speaker of the l-th utterance u l ), then selects appropriate experts to process each utterance u l and its r l . the selected experts (each of which is the last (k llm -k) layers of the llm, where k llm is the number of layers in the llm) generate key content of u l with the guidance of r l , which is repeated l times to process all utterance-speaker pairs. the fg combines and emphasizes the content procured by the experts to form the final summary y. details regarding ror, moe, and fg in our approach for dialogue summarization are elaborated in the subsequent subsections. ## related work dialogue modeling has attracted attention from many existing studies @xcite @xcite , especially dialogue summarization. a large body of dialogue summarization studies is devoted to leveraging advanced text encoders, such as bart @xcite , to achieve a more nuanced modeling of dialogue content and thus optimize role-specific summarization @xcite @xcite . to enhance the quality and relevance of generated summaries, many studies adopt particular elements in dialogues or extra features, such as important utterances @xcite , dialogue topics @xcite et al. @xcite @xcite , and semantic relations among sentences @xcite @xcite to extend the capability of their summarization models. with the growing recognition of the importance of the structures and interactions in dialogues, summarization is thus performed by incorporating them as core components in several state-of-the-art studies @xcite @xcite @xcite @xcite @xcite @xcite @xcite . although summarization performance is promoted accordingly with such methodology improvements, these studies mainly use a single-model design to capture various types of essential information in the dialogue and generate summaries in an end-toend manner. as a choice of using multiple models, moe offers a solution to separately model different aspects of the input and process them accordingly, achieve remarkable success in handling complex tasks, such as language modeling, natural language inference, question answering, etc., @xcite @xcite @xcite , where they cover various applications, such as textual-only @xcite @xcite @xcite as well as cross-modal scenarios @xcite , and so far few are used for dialogue summarization. therefore, compared with previous studies, this pa-per proposes a way of applying multiple models for dialogue summarization and introduce a novel design that improves moe, where ror is proposed to address the challenge of effectively selecting appropriate experts in the particular dialogue circumstance, and the fg highlights the salient content generated by different experts. ## conclusion in this paper, we propose an moe approach that alter llms for dialogue summarization, where a specific router and fusion generator are designed to facilitate the mixture process of experts. specifically, the routing effectively organizes the matching of different experts to utterances, and the fusion generator further optimizes the information processed by experts and then utilizes appropriate contents from them to provide final summaries with essential information from different aspects. experiment results and analysis on four english and chinese benchmark datasets for dialogue summarization illustrate the effectiveness of our approach, which outperforms strong baselines and existing studies on all datasets, and show that moe successfully distinguishes different contents in each dialogue with processing using appropriate experts.