Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT (2020)

Abstract: Most current extractive summarization models generate summaries by selecting salient sentences. However, one of the problems with sentence-level extractive summarization is that there exists a gap between the human-written gold summary and the oracle sentence labels. In this paper, we propose to extract fact-level semantic units for better extractive summarization. We also introduce a hierarchical structure, which incorporates the multi-level of granularities of the textual information into the model. In addition, we incorporate our model with BERT using a hierarchical graph mask. This allows us to combine BERT’s ability in natural language understanding and the structural information without increasing the scale of the model. Experiments on the CNN/DaliyMail dataset show that our model achieves state-of-the-art results.

## introduction the aim of text summarization is to generate a compressed shorter highlight of a given document. the generated summaries must conform to natural language constraints and cover the most important information conveyed in the source text. in this paper, we focus on extractive summarization which generates summaries by selecting salient sentences or other semantic units from the source text. as we know, sentences are regarded as the basic textual units making up the final summaries in most extractive summarization models. one of the problems with sentence-level extractive summarization is that there exists a gap between the human-written gold summary and the training objective. since most datasets do not contain sentence-level labels, oracle summaries are normally generated by a greedy algorithm, which maximizes the rouge score between the sentences in the source text and in the gold summary @xcite . as pointed out by @xcite , sentences with high rouge scores may not necessarily lead to an optimal summary. such discrepancy may be due to the overlapping contents and over extraction. therefore, some researches try to perform extraction at a lower level such as words or phrases @xcite @xcite . although these models are able to learn which phrases or words are more important directly from the gold summary, it is hard to maintain semantic integrity when information is scattered. inspired by @xcite , we propose to extract facts, a granularity between phrases and sentences, as the primary textual units to generate summaries. in this work, we first develop a heuristic algorithm to split a sentence into several smaller units, where each unit is considered as a single fact description. then we apply a one-to-one strategy to match each fact in the gold summary to one fact in the source text to obtain the oracle label. in this way, we smooth the gap between the gold summary and oracle labels and further reduce redundancy and over-extraction. meanwhile, fact units can still well keep semantic integrity and thus are able to provide faithful summaries. considering facts are relatively independent semantic units, in order to maintain rich contextual information and to capture potential relationships among facts, we believe the sentence-level information still plays an important supporting role. moreover, based on the assumption that the selected facts should be semantically close to the majority of source documents, we also import the document-level information as part of our model. we will show that such a fact-sentence-document hierarchical text modeling facilitates to capture more structured contextual information for effective fact extraction. recent works @xcite @xcite have demonstrated that it is highly beneficial to apply pretrained language models such as bert to extractive summarization models. following this trend, we adopt a bert-based encoder to generate contextualized representations for further extraction. it is challenging to impose the structure information when fine-tuning the bert model in a downstream task. @xcite separately encode representations for each granularity with bert and then capture the structure information with a graph network stacked upon the encoder. however, this method not only results in a large complex model but also cannot fully utilize the advantage of pretrained language models. based on the idea that bert can be regarded as a full connected graph, we propose to directly impose the structure information on bert with a graphbased mask to jointly learn contextual representations of different text granularities within a single bert. on the one hand, we significantly reduce the time cost of running bert several times plus running a graph neural network to running bert once. on the other hand, we are able to utilize the complex structure of bert to capture the structure information. we conduct experiments on the cnn/dailymail dataset, a well-known extractive summarization benchmark dataset. the results show that using facts as extractive units improves the summarization quality and our model achieves better performance than the state-of-the-art model. the contributions of our work can be summarized as follows. (1) we propose to extract fact-level semantic units for extractive summarization to fill the gap between the gold summary and oracle labels, reducing redundancy and over-extraction. (2) we import a hierarchical structure to remedy the information loss after splitting sentences into facts. (3) we propose a graph-based mask algorithm to impose the structure information on bert directly. to the best of our knowledge, we are the first to combine the pretrained language model and the structure information without increasing the scale of the model, which provides a new idea for fine-tuning pretrained models on downstream tasks. ## extraction of facts and alignment with gold summary we propose to explicitly align a gold summary with its corresponding facts descriptions in the source text. for this purpose, we develop a heuristic algorithm to break up the source text and the summary into smaller granularities as introduced below. when splitting sentences into smaller semantic units, we need to ensure each unit has a proper size while maintaining the integrity of information. an intuitive idea might be to split sentences with commas and other conjunctions. but this straightforward strategy is not capable of handling complicated sentences in documents. therefore, we leverage the dependency parser to handle this issue. to begin with, we adopt a dependency parser to convert a sentence into the labeled tuples in the form of (word1, word2, label), where the label denotes a grammatical relation between a pair of words. after that, the sentence is split using the labels representing punctuation marks, conjunctions and clauses, including punct (punctuation marks), cc (coordination relationship) and mark (finite clause). to acquire a more complete semantic unit, we merge units that are connected by some special labels such as the relative clause modifier (acl:relcl), the adverbial clause modifier (advcl), the appositional modifier (appos) and the clausal complement (ccomp). furthermore, we use the conjunct (conj) to identify whether a conjunction connects two sentences or two phrases. a conjunct is a relation between two elements connected by a coordinating conjunction. when the distance between the two elements is less than a threshold, we regard the two coordinated elements are words or phrases rather than clauses. if so, the two split parts are merged back as one unit. moreover, we predefine a minimum unit length and a maximum clause length. if the length of a unit is smaller than the minimum unit length, this unit will be merged with the preceding unit. if a clause is longer than the maximum clause length, we regard it as an independent semantic unit. take the dependency trees in figure 1 as an example. the sentence is split into five parts based on the abovementioned rules. we merge the first three parts back together since they are connected by the appos label while not exceeding the threshold length for being an independent clause. as for the last two parts, although there is a coordination relationship, it turns out that they are two coordinate entities based on the conj label. hence, we also merge the last two parts as a fact. in the experiments, we adopt the well-known nlp pipeline stanford corenlp @xcite for dependency parsing. once facts are extracted, the next step is to obtain oracle labels for them from the gold summary. we match each fact in the summary with one corresponding fact that has the maximum rouge score from original text. such an alignment allows each part of the summary to be accurately mapped to a semantic unit in the source text.