Improving Long Dialogue Summarization with Semantic Graph Representation (2023)

Abstract: Although Large Language Models (LLMs) are successful in abstractive summarization of short dialogues, summarization of long dialogues remains challenging. To address this challenge, we propose a novel algorithm that processes complete dialogues comprising thousands of tokens into topic-segment-level Abstract Meaning Representation (AMR) graphs, which explicitly capture the dialogue structure, highlight salient semantics, and preserve high-level information. We also develop a new text-graph attention to leverage both graph semantics and a pretrained LLM that exploits the text. Finally, we propose an AMR node selection loss used jointly with conventional cross-entropy loss, to create additional training signals that facilitate graph feature encoding and content selection. Experiments show that our system outperforms the state-of-the-art models on multiple long dialogue summarization datasets, especially in low-resource settings, and generalizes well to out-of-domain data.

## introduction summarization of long dialogues is receiving more attention as virtual meetings become prevalent and advanced speech recognition technologies become accessible to accurately transcribe dialogues. although the state-of-the-art (sota) models for abstractive summarization achieved remarkable performance on non-conversational documents via pretraining on datasets involving wikipedia, books, stories, and news, generating high-quality summaries for long dialogue transcripts remains challenging. in this paper, we present the use of a semantic representation, abstract meaning representation (amr), to address some of the challenges in long dialogue summarization and we demonstrate its use on two different genres of long dialogues: meetings and screenplays of tv episodes. * these authors contributed equally to this work. one major challenge with dialogues is their complex structure in combination with their informal nature. chen and yang (2021a) discussed the impact of repetitions, false-starts, and hesitations on dialogue structure and suggested that key information can be spread across different portions of the dialogue. multiple speakers and diverse spoken language styles also complicate the interactions involving speakers and coreferences. for example, it may be challenging to associate speakers with their opinions and actions as well as with their reactions to other speaker utterances. admittedly, recent encoder-decoder llms like bart @xcite achieve good performance on short dialogues given adequate finetuning. long dialogues, however, have more content as well as speakers and this brings greater complexity to dialogue structure, leading to performance degradation for encoderdecoder llms. a second challenge lies in the transformer architecture of llms, which often draws spurious correlations @xcite and easily overfits small or homogeneous training sets. this limitation is particularly pertinent to long dialogue summarization, as it often involves lowresource domains. meeting summarization datasets are particularly small. in fact, major meeting summary datasets, ami @xcite and icsi @xcite , have sizes of 137 and 59 transcripts respectively. in these very low-resource scenarios, a greater risk of overfitting is present. to address these challenges, we propose a novel abstract meaning representation (amr) for long dialogues, to capture diverse entity interactions and complex structures. amr, as a semantic graph representation, captures the most salient semantic knowledge using concept nodes and preserves interconcept relations with labeled edges. it is believed to convey information largely orthogonal to what conventional models exploit from the text input @xcite . amrs thus provide reliable semantic and structural cues to alleviate overfitting, which we show is particularly helpful for the low-resource setting. our approach generalizes the sentence amr introduced by @xcite and specifically designs an algorithm for building topic-segment amrs for long spans of conversations to preserve global information. our approach to incorporating amrs for summarization also differs from previous research. existing methods of amr-based summarization rely on graph-to-graph operations for content selection and additional text generation modules @xcite , an approach that does not surpass state-of-the-art encoder-decoder llms. instead, our work leverages both graph semantics and a pretrained encoder-decoder model. we develop a novel text-graph attention that exploits structural and global information in amr to improve text encoding. we also propose a node selection loss used jointly with the standard cross-entropy loss for additional training signals. our work is orthogonal to the existing research on dialogue-specific pretraining @xcite , which improves models' familiarity with dialogue styles and has achieved the state-ofthe-art for long-dialogue summarization with its two model variants, dialoglm and dialogled. with our novel architecture, our amr-based system can make full use of an llm's pretrained weights while providing better global information aggregation and fine-grained structural cues, from topic segments to speaker-action associations and inter-utterance coreferences. our code is available at https://github.com/bobby-hua/ summarization-via-semantic-graph . in sum, the contributions of our work are: our dialogue amr is a more comprehensive representation of dialogue structures, which captures not only many useful features from the previous research but also fine-grained semantic structure and inter-utterance coreferences. the increased granularity of our representation allows it to effectively guide the encoding of text tokens via our novel cross-attention mechanism. summarization with amr current research that applies amrs to summarization has primarily relied on graph-to-graph operations to produce summary amr graphs. @xcite first introduces summarization via amrs by merging sentence amrs into document amrs and conducting subgraph selection with integer linear programming. other research adopts similar graphto-graph methods, introducing new heuristics and algorithms for summary amr generation and applying more recent generative models to generate fluent summaries @xcite @xcite . the graph-tograph algorithms in these models are incompatible with the encoder-decoder llms behind current sota summarization models. our work instead leverages both graph semantics and a pretrained encoder-decoder model, via our novel text-graph attention and additional node selection loss. dialogue @xcite is the only work on dialogue amr to the best of our knowledge. its algorithm connects sentence amrs with a dummy node and models coreference and identical concepts with additional edges. they only apply their dialogue amrs to short conversations and for dialogue relation extraction and response generation. our work is different in that we apply topic segmentation to process long dialogues and use node merging to allow a single amr graph to capture a longer text span. node merging, along with our speaker and utterance nodes, also creates a hierarchy of nodes with different centrality to help the graph encoder aggregate information at different levels. root speaker utterance coreference nodes (group 1) coreference nodes (group 2) participant utterance sentence sentence amr graphs topic segment amr graph w/o merging topic segment amr graph connect with utterance node connect other utterance amrs with speaker node connect with root node merge based on coreference finally, our model differs from bai et al. (2021)'s feature fusion and dual-attention model variants in terms of architecture, suggesting a promising way to combine amrs with the llm encoder and decoder. ## model architecture our model consists of an amr graph encoder, a text encoder with text-graph cross-attention, and a decoder. the full architecture is illustrated in figure 2 . ## results we report rouge scores @xcite as automatic evaluation metrics and also report human evaluation results. to test the statistical significance of the improvement brought by our model, we use the almost stochastic order test (aso) @xcite as implemented by @xcite . aso tests the statistical significance by evaluating stochastic dominance. aso is more suitable for deep learning models because it does not require t-test's normal distribution assumption, which likely does not hold for neural networks @xcite . we also report t-test results for the sake of completeness. ## conclusion in this work, we propose a novel amr algorithm to capture long dialogue structures. we develop a text-graph cross-attention and node selection loss to effectively extract structural features and integrate them into an encoder-decoder llm for summarization. empirically, we show that our system outperforms the sota models and achieves particularly promising results in low-resource settings and out-of-domain evaluation.