Reconstruct Before Summarize: An Efficient Two-Step Framework for Condensing and Summarizing Meeting Transcripts (2023)

Abstract: Meetings typically involve multiple participants and lengthy conversations, resulting in redundant and trivial content. To overcome these challenges, we propose a two-step framework, Reconstruct before Summarize (RbS), for effective and efficient meeting summarization. RbS first leverages a self-supervised paradigm to annotate essential contents by reconstructing the meeting transcripts. Secondly, we propose a relative positional bucketing (RPB) algorithm to equip (conventional) summarization models to generate the summary. Despite the additional reconstruction process, our proposed RPB significantly compresses the input, leading to faster processing and reduced memory consumption compared to traditional summarization methods. We validate the effectiveness and efficiency of our method through extensive evaluations and analyses. On two meeting summarization datasets, AMI and ICSI, our approach outperforms previous state-of-the-art approaches without relying on large-scale pre-training or expert-grade annotating tools.

## introduction although numerous achievements have been made in the well-structured text abstractive summarization @xcite @xcite , the research on meeting summarization is still stretched in limit. there are some outstanding challenges in this field, including 1) much noise brought from automated speech recognition models; 2) lengthy meeting transcripts consisting of casual conversations, content redundancy, and diverse topics; 3) scattered salient information in such noisy and lengthy context, posing difficulties for models to effectively capture pertinent details. to this end, previous works adapt the language model to long inputs through techniques such as long-sequence processing @xcite @xcite and hierarchical learning @xcite , or tailor the input to an acceptable length through sentence compression @xcite and coarseto-fine generation @xcite . however, these approaches do not specifically target the critical information in meeting transcripts. @xcite utilizes the token-wise loss as a criterion to annotate contents with dialogpt @xcite , suffering from labeling unpredictable contents as critical information. besides, the commonly used pre-processing procedure that extends models' positional embedding by copying, and truncating the lengthy input compromises the positional relationships learned during pre-training, and results in a loss of important information due to the brutal truncation. consequently, a natural question is -how can we precisely capture the salient contents from noisy and lengthy meeting transcripts, and summarize them with conventional language models? our observation is that meetings are characterized by extensive communication and interaction, with specific texts often containing pivotal content that drives these interactions. based on this understanding, we propose a two-step meeting summarization framework, reconstrcut before summarize(rbs), to address the challenge of scattered information in meetings. rbs adopts a reconstructor to reconstruct the responses in the meeting, it also synchronically traces out which texts in the meeting drove the responses and marks them as essential contents. therefore, salient information is captured and annotated as anchor tokens in rbs. to preserve the anchors but compress the lengthy and noisy input, we propose the relative positional bucketing (rpb), a dynamic embedding compression algorithm inspired by relative positional encoding (rpe) @xcite . our rpb-integrated summarizer can preserve the anchors and compress the less important contents according to their relative position to the context: as you can see it's supposed to be original, trendy and user friendly. i'm not really what you guys have already received in your emails. what did you get? response: industrial designer: got the project announcement about what … context: i'm not really what you guys have already received in your emails. what did you get? industrial designer: got the project announcement about what … response: design a remote control … as you can see it's supposed to be original, trendy and user friendly. i'm not really what you guys have already received in your emails. what did you get? industrial designer: got the project announcement about what the project is. designing a remote control … embedding remaining layers d 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 d 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑛𝑛 𝑐𝑐 reconstructor rpb annotate anchors reconstruct retrace anchors. this allows the summarizer to generate a concise and informative summary of the meeting transcripts. although rbs introduces an additional importance assessment step, the introduction of rpb greatly compresses the length of the original text, making rbs even faster and memory-efficient than the traditional one-step approach. the experimental results on ami @xcite and icsi @xcite show that rbs outperforms previous state-of-the-art approaches and surpasses a strong baseline pre-trained with large-scale dialogue corpus and tasks. extensive experiments and analyses are conducted to verify the effectiveness and efficiency of each process of our approach. to sum up, our contributions are as follows: (1) we propose the rbs, an efficient and effective framework for long-text meeting transcripts summarization; (2) without external annotating tools or large-scale pre-training corpus and tasks, our method can efficiently generate meeting minutes with conventional language models (plms); (3) extensive experiments demonstrate the effectiveness of our framework. ## methods the main architecture is shown in figure 1 . our framework comprises two components: the reconstructor and the summarizer. the reconstructor is responsible for reconstructing meeting transcripts and identifying the context that drives the interaction. meanwhile, before generating the summary, the summarizer compresses the lengthy input and preserves critical content. ## analysis in this section, we conduct further analysis to show the effectiveness of the rbs. we aim to investigate the correlation between anchor tokens and salient information in the meeting transcript. through extensive experiments, we will demonstrate the validity of our approach in capturing anchor tokens, and the significance of anchor tokens in conveying important information. furthermore, we will analyze the impact of different methods for aggregating the importance scores. we will also provide justification for our bucketing algorithm, analysis of the computing complexity is also provided to prove the efficiency of the framework. additionally, the potential for reusing the parameters of the reconstructor is explored in appendix a.3. ## related work long-sequence processing techniques such as sliding-window attention (beltagy et al., 2020), sparse sinkhorn attention @xcite , and hierarchical learning @xcite are well-explored. these approaches target specifically lengthy input but ignore capturing salient information. sentence compression @xcite and coarse-to-fine generation @xcite are developed to tailor the input length. however, the error propagation in intermediate steps severely limits their performance. meanwhile, language models have gradually equipped with dialogue acts (goo and chen, 2018), discourse relationship @xcite , coreference resolution @xcite , and topicsegmentation @xcite @xcite . despite the modest advances, these methods require external annotating tools or expertgrade annotators to accomplish the task. @xcite employ the dialoggpt @xcite as an annotator to capture keywords and topics. despite the performance, adopting the tokenwise loss to label keywords needs to be considered more deeply. additionally, cross-domain @xcite and large-scale dialogue-specific pretraining @xcite are utilized. however, large-scale datasets, excessive optimization steps, and sufficient computational resources are necessities and luxuries. ## conclusion in this paper, we proposed rbs, a meeting summarization framework that accurately captures salient contents from noisy and lengthy transcripts. rbs uses a two-step process to evaluate content importance and dynamically compress the text to gen-erate summaries. we introduce rpb, an anchorbased dynamic compression algorithm that condenses the original text, making rbs faster and more memory-efficient than one-step approaches. without resorting to expert-grade annotation tools or large-scale dialogue-related pretraining tasks, experimental results on ami and icsi datasets show that rbs outperforms various previous approaches and reaches state-of-the-art performance.