E xplain M eet S um: A Dataset for Explainable Meeting Summarization Aligned with Human Intent (2023)

Abstract: To enhance the explainability of meeting summarization, we construct a new dataset called “ExplainMeetSum,” an augmented version of QMSum, by newly annotating evidence sentences that faithfully “explain” a summary. Using ExplainMeetSum, we propose a novel multiple extractor guided summarization, namely Multi-DYLE, which extensively generalizes DYLE to enable using a supervised extractor based on human-aligned extractive oracles. We further present an explainability-aware task, named “Explainable Evidence Extraction” (E3), which aims to automatically detect all evidence sentences that support a given summary. Experimental results on the QMSum dataset show that the proposed Multi-DYLE outperforms DYLE with gains of up to 3.13 in the ROUGE-1 score. We further present the initial results on the E3 task, under the settings using separate and joint evaluation metrics.

## introduction meeting summarization typically is a form of long document summarization, because the input is usually given as a long conversational sequence from multi-party dialogues. among various approaches for long document summarization, the extract-thengenerate method is one of the promising methods; it first automatically selects "salient" contents which are relevant to a specific summarization and employs them to guide the generation of a summary @xcite @xcite @xcite @xcite , thereby inducing the manner of dealing with both efficiency (in processing a long input) and effectiveness (in locating accurately informative relevant contents). * these authors contributed equally to this work. aiming to achieve a high degree of explainability in meeting summarization, this paper proposes a new dataset called explainmeetsum, an augmented version of qmsum, by manually and explicitly annotating evidence sentences that faithfully "explain" and "support" each summary sentence. figure 1 illustrates an example of the annotation of evidence sentences. based extensively on explainmeetsum, we propose multi-dyle, a generalized version of dyle that enables multiple extractors, and present a novel explainability-aware benchmark task, called explainable evidence extraction (e3), as follows. ## multi-dyle figure 2 shows the overall architecture of the multi-dyle model. the key novelty of multi-dyle is the employment of m heterogeneous extractors with separate sets of extractive oracles, foot_1 m oracle losses, and a consistency loss under the generalized extractivegenerator framework. this section presents the details of multi-dyle, including a brief description of dyle @xcite . 1) , e (2) , which compute relevance scores for each sentence x @xmath0; ii) for the j-th extractor, we select the top k snippets with the highest relevance scores, denoted as x (j) k (i.e. eq. ( 1 )); iii) the resulting m list of top-k sentences are then merged to obtain x k as a final set, but allowing the "duplicated" snippets (i.e. eq. ( 2 )); iv) the merged set is used to guide the summary generation process at the decoding time steps (i.e., eq. ( 3 )); v) the dynamic weights computed by the generator over all decoding time steps are reflected back as a supervised signal to train each j-th extractor, thus leading to the consistency loss l consist (i.e. eq. ( 4 )); vi) multi-dyle is trained with multi-task learning using the combined losses (i.e., eq. ( 6 ))-generation loss l gen , oracle losses l (j) oracle (i.e., eq. ( 5 )), and consistency loss l consist . ## explainable evidence extraction (e3) in this section, we introduce the details of e3, which identifies all cess and pess for a given summary, and baseline e3 models. ## experiments in our experiment, we first compare the summarization performance of multi-dyle, introduced in section 4, with that of dyle and its simple variants to check whether the use of multiple extractors lead to performance improvement. we further present the performance of our baseline ee described in section 5 under the settings of separate and joint tasks in sections 6.2 and 6.3, respectively. an illustration and examples of joint tasks are describe in appendix c, and the implementation details for the multi-dyle and ee models are presented in appendix d. as aforementioned in section 4, multi-dyle uses sentence-level snippets whereas the original version of dyle uses turn-level snippets. to clarify the different setups for using extractive oracles, with the abuse of notation, x rog o , x ces o , and x p es o refer to the sets of rouge-based, cesbased, and pes-based extractive oracles (in ex-plainmeetsum), respectively. the various types of multi-dyle are defined as follows: some variants of dyle using x rog o are denoted as follows: overall, the results confirm that the use of human annotated evidence sentences improves performance under the same framework, even without changing the model's architecture. ## conclusion in this paper, we presented a novel explainmeet-sum, as an explainability-enhanced qmsum by providing complete manual annotation of two types of evidence sentences, ces and pes, as explanations to faithfully support or explain each sentence in gold summaries. equipped with explainmeet-sum, we proposed multi-dyle as a generalized dyle to enable the addition of an explainable extractor based on ces-based extractive oracles. we further defined a novel task, e3, which aims to extract explainable evidence sentences when a summary sentence is given. the experimental results obtained on qmsum using explainmeetsum showed that the proposed multi-dyle based on an additional extractor towards a human-aligned explanation outperformed dyle and led to improvements in the joint evaluation settings involving e3. in future work, we would like to invent a joint learning framework for meeting summarization and e3, extensively employing human-supervised explainable signals from explainmeetsum, towards better explainable meeting summarization. furthermore, developing a novel joint evaluation metric for meeting summarization and e3 to overcome the limitations of the rouge-driven scores would be worthwhile.