GRETEL : Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization (2022)

Abstract: Recently, neural topic models (NTMs) have been incorporated into pre-trained language models (PLMs), to capture the global semantic information for text summarization. However, in these methods, there remain limitations in the way they capture and integrate the global semantic information. In this paper, we propose a novel model, the graph contrastive topic enhanced language model (GRETEL), that incorporates the graph contrastive topic model with the pre-trained language model, to fully leverage both the global and local contextual semantics for long document extractive summarization. To better capture and incorporate the global semantic information into PLMs, the graph contrastive topic model integrates the hierarchical transformer encoder and the graph contrastive learning to fuse the semantic information from the global document context and the gold summary. To this end, GRETEL encourages the model to efficiently extract salient sentences that are topically related to the gold summary, rather than redundant sentences that cover sub-optimal topics. Experimental results on both general domain and biomedical datasets demonstrate that our proposed method outperforms SOTA methods.

## introduction due to the well-known limitation of pre-trained language models (plms) @xcite that they fail to capture long-range dependencies @xcite , attempts have been proposed to integrate neural topic models (ntms) @xcite @xcite into plms, which have shown significant improvement in the performance of the text summarization task @xcite @xcite . in addition to the local contextual information captured in plms, ntms can provide an approximation of the global semantics captured from document contents, i.e., latent topics, as well as their posterior topic representations. the global semantics are further used to guide the model, to generate coherent summaries which cover the most relevant topics discussed within the document, via the attention mechanism @xcite @xcite or graph neural networks (gnns) @xcite . however, there exists the semantic gap between latent topics as the approximation global semantics, and the true global semantics due to two major limitations for existing methods. the first limitation concerns the nature of unsupervised topic inference in these methods, where topics and posterior topic distributions are learned from documents in an unsupervised manner, without considering the key semantic information conveyed in the gold summary (mostly the abstract). existing methods using document-word features, without accessing the semantic information of the gold summary, can extract sub-optimal topics with high-frequency words. however, sub-optimal topics with highfrequency words, do not necessarily cover the true global semantics that is condensed in the gold summary. this results in the wrong assignment of the document with sub-optimal topics, and con-sequently the model extract redundant sentences containing high-frequency topic words. another limitation is that existing methods rely on document word features such as bag-of-words (bows) to extract latent topics, and disregard the sequential and syntactic dependency between words. this may lead to sequentially and syntactically correlated words being allocated to different topics. one solution is to provide ntms with contextual representations from plms, which is nevertheless challenging to directly apply in existing methods for text summarization. plms in existing methods are forced to truncate the input to a limited length owing to the complexity of language models. thus the representations from partial content of the document cannot necessarily help ntms to mine informative topics that cover the whole content of the document, especially for long documents. overall, although existing methods encourage the summary with sentences that are topically similar to the document topic distribution, the summary focuses more on sentences with high-frequency words and can have low semantic similarity with the gold summary. to help understand these limitations, we make a detailed analysis based on the benchmark datasets in section 3. in response to the above, we propose a novel graph contrastive topic enhanced language model (gretel), that incorporates the graph contrastive topic model (gctm) empowered by the semantic information of the gold summary and the global document context, with plms for long document extractive summarization. the first distinguishing feature of gretel is the employment of the hierarchical transformer encoder (hte) to fully embed the global context of long documents, to inform topic representations of documents and sentences. the global contextual information captured in hte but missing in the bow feature, allows the model to learn more discriminative document and sentence topic representations, and coherent topics. secondly, it utilizes graph contrastive learning with the supervised information from the gold summary. it pushes close topic representations of documents and sentences that have high semantic similarity with the gold summary and pulls away otherwise. this encourages the model to capture better global semantic information: latent topics that describe the most key information in the original document content. therefore, it allows the model to select key sentences that are topically similar to the gold summary. experimental results demonstrate that our method can effectively distinguish salient sentences from documents with both global and local semantics, leading to superior performance compared to previous methods. ## related work topic enhanced plms for text summarization. many studies have investigated the application of plms for extractive summarization, including bertsum @xcite , discobert @xcite , matchsum @xcite et al. however, these methods fail to capture the global context of long documents due to the limitation of plms. to address it, several studies combined topic modeling with plms to introduce global semantic information. @xcite proposed to extract firstly latent topics independently and then use them to improve the summarization model. other studies @xcite @xcite proposed to use the bow as input features for neural topic modeling and improved the transformer encoder and decoder with the extracted latent topics with an attention mechanism for abstractive summarization @xcite @xcite . @xcite ; @xcite proposed to use the graph neural network to infuse topics into contextual representations from plms, for multidocument abstractive summarization and extractive summarization. fu et al. (2020) considered extract both document and paragraph-level topic distribution, and use them to guide the abstractive summarization. plms for long document summarization. to address the limitation of encoding the full context of long documents using plms, another direction is to design the efficient sparse self-attention or using a sliding window. @xcite proposed a step-wise model with a structured transformer. @xcite proposed a computationally efficient method based on the head-wise positional strides, to identify salient content for long documents. @xcite employed a transformer with multi-granularity sparse attentions. @xcite used a sliding selector network with dynamic memory, in which the sliding window is used to encode input documents, segment by segment. @xcite divided long documents into multiple blocks and encoded them by independent transformer networks. ## dataset-dependent analysis for limitations to better illustrate the limitations of existing methods, we present a dataset-dependent analysis, with the aim to answer two key questions: 1) do the extracted topics from topic models tend to focus on high-frequency words? and 2) due to it, would there be a semantic gap between topics as the approximation of global semantics and true global semantics in the gold summary? we first present the top-10 words of topics learned by the traditional topic model lda on the pubmed @xcite dataset, as shown in t1: type treatment consistent needed lower disorders sensitive patient acid way t2: male group treatment followed cells per side plasma american health t3: al dna clinical risk observed tube lower inflammatory et features t4: type al clinical mice bacteria high vs posterior conditions side t5: differences performed results side number higher size tube et patients t6: dna revealed smoking control mental number change sd light versus top high-frequency words: patients, study, using, cells, group, treatment, et, one, al, data, studies, two, patient, results, cell, time, however, figure, significant, reported, high, disease, analysis, clinical, found, age, years, associated, showed, different, compared, risk, levels ## method to address the aforementioned limitations of existing methods, we propose our method gretel, to better capture and incorporate the global semantics to improve plms, for long document extractive summarization. given u sentences {s 1 , • • • , s u } of a document i from the corpus d, extractive summarization aims to select v informative sentences from u sentences (v ⌧ u) as the summary s for the document i. this task can be formulated as a binary sentence classification problem. we assign label y i,@xmath0,j (j 2 {1, • • • , u}) for the summary, or y i,@xmath1. as shown in figure 1 , different from previous methods, we leverage the contextual representations from plms, and gold summary to guide the topic inference. to this end, we first employ the hierarchical transformer encoder (hte) to fully encode the global context of long documents, and then design the supervised graph contrastive loss, to push close the document topic distribution and topic distributions of salient sentences. this helps our method to capture better global semantics, that effectively distinguish between salient and nonsalient sentences, according to their contextual and semantic connections to the gold summary. ## experimentation details in this section, we present the details of the datasets used, evaluation metrics, and different baselines. ## results and analysis a series of experiments were conducted to demonstrate the efficacy of the proposed method. ## conclusion in this paper, we propose a novel framework gre-tel for extractive summarization of long texts, that furnishes plms with the neural topic inference, to fully incorporate the local and global semantics. experimental results on both general and biomedical datasets show that our model outperforms existing state-of-the-art methods, and global semantics empowered by graph contrastive learning and plms can yield more discriminative sentence representations to select salient sentences, that are topically similar to the gold summary. for future work, we would explore the feasibility of extending this framework to abstractive and multidocument summarization tasks.