Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders (2023)

Abstract: Pre-trained sentence representations are crucial for identifying significant sentences in unsupervised document extractive summarization. However, the traditional two-step paradigm of pre-training and sentence-ranking, creates a gap due to differing optimization objectives. To address this issue, we argue that utilizing pre-trained embeddings derived from a process specifically designed to optimize informative and distinctive sentence representations helps rank significant sentences. To do so, we propose a novel graph pre-training auto-encoder to obtain sentence embeddings by explicitly modelling intra-sentential distinctive features and inter-sentential cohesive features through sentence-word bipartite graphs. These fine-tuned sentence embeddings are then utilized in a graph-based ranking algorithm for unsupervised summarization. Our method is a plug-and-play pre-trained model that produces predominant performance for unsupervised summarization frameworks by providing summary-worthy sentence representations. It surpasses heavy BERT- or RoBERTa-based sentence representations in downstream tasks.

## introduction unsupervised document summarization involves generating a shorter version of a document while preserving its essential content @xcite . it typically involves two steps: pretraining to learn sentence representations and sentence ranking using sentence embeddings to select the most relevant sentences within a document. most research focuses on graph-based sentence ranking methods, such as textrank @xcite and lexrank @xcite , to identify the significant sentence by utilizing topological relations. continual improvements have been demonstrated by several attempts @xcite @xcite ; wang * jianxin li is the corresponding author. 𝑺 𝒂 𝒘 𝟎 𝒘 𝟏 𝒘 𝟐 𝒘 𝟑 𝒘 𝟒 𝑺 𝒄 𝑺 𝒃 𝒘 𝟎 𝒘 𝟏 𝒘 𝟐 𝒘 𝟑 𝒘 𝟒 argentine beijing fan messi shocked 𝑺 𝒂 …… during a friendly match between argentina and australia at the worker's stadium in beijing. ## methodology in what follows, we describe our pre-training model bi-gae (as shorthand for bipartite graph pre-training with graph convolutional auto-encoders ) used for unsupervised extractive summarization. we will introduce bipartite graph encoding and the pre-training procedure using our bi-gae. ultimately, we will utilize the pre-trained sentence representations for the downstream unsupervised summarization. ## conclusion in this paper, we introduce a pre-training process that optimizes summary-worthy representations for extractive summarization. our approach employs graph pre-training autoencoders to learn intrasentential and inter-sentential features on sentenceword bipartite graphs, resulting in pre-trained embeddings useful for extractive summarization. our model is easily incorporated into existing unsupervised summarization models and outperforms salient bert-based and roberta-based summarization methods with predominant rouge-1/2/l score gains. future work involves exploring the potential of our pre-trained sentential representations for other unsupervised extractive summarization tasks and text-mining applications.