Extractive Summarization via Fine-grained Semantic Tuple Extraction (2024)

Abstract: Traditional extractive summarization treats the task as sentence-level classification and requires a fixed number of sentences for extraction. However, this rigid constraint on the number of sentences to extract may hinder model generalization due to varied summary lengths across datasets. In this work, we leverage the interrelation between information extraction (IE) and text summarization, and introduce a fine-grained autoregressive method for extractive summarization through semantic tuple extraction. Specifically, we represent each sentence as a set of semantic tuples, where tuples are predicate-argument structures derived from conducting IE. Then we adopt a Transformer-based autoregressive model to extract the tuples corresponding to the target summary given a source document. In inference, a greedy approach is proposed to select source sentences to cover extracted tuples, eliminating the need for a fixed number. Our experiments on CNN/DM and NYT demonstrate the method’s superiority over strong baselines. Through the zero-shot setting for testing the generalization of models to diverse summary lengths across datasets, we further show our method outperforms baselines, including ChatGPT.

## introduction the objective of automatic text summarization is to condense the content of an original document while preserving its essential information. existing summarization techniques can be categorized into two main approaches: extractive and abstractive methods @xcite . abstractive methods aim to generate new sentences, often referred to as paraphrased sentences, to compose a summary @xcite , while extractive techniques generate summaries by selecting and extracting salient sentences directly from the source text @xcite . in this study, we focus on extractive summarization, primarily formulated as sentence-level classification. this task typically involves a greedy method to derive binary labels for sentences in a source document, indicating their inclusion or exclusion in the summary @xcite . nevertheless, previous research @xcite demonstrates the drawbacks of this sentencecentric granularity for extraction as it can introduce redundancy and unnecessary information into the output. besides, during inference, a fixed-length cutoff or threshold is often applied to restrict the sentence length of the output summary. this practice is inherently limited as it fails to accommodate the varying characteristics of different documents, which may necessitate extractive summaries of different lengths. for instance, a long document may need more sentences to comprehensively cover its salient information, whereas a short document may suffice with a more concise representation. additionally, in real-world applications, expecting users to specify the exact number of sentences to be extracted when utilizing a summarization system may not be always feasible or practical. motivated by the shortcomings outlined above, we present a new fine-grained autoregressive approach for extractive summarization via semantic tuples extraction. to this end, we exploit the inherent interdependence between information extraction (ie) and text summarization as both tasks share a common objective: extracting accurate information from unstructured texts in alignment with a user's specific requirements and presenting the extracted information in a concise manner @xcite . while summarization aims to present this information in natural language sentences, ie aims to transform relevant information into structured representations @xcite . to effectuate this integration, we first use an ie tool to convert each sentence into a semantic meaning representation based on predicate-argument structures @xcite , which we call semantic tuples in this work. we identify these semantic tuples corresponding to the target summary as the objective of extraction. leveraging a transformer-based autoregressive model @xcite , we train the model to extract the target semantic tuples from each source document. this can encourage the model to concentrate on salient information at a more granular level compared to conventional approaches that perform extraction at the sentence level. during inference, we introduce a greedy strategy to select source sentences that cover the extracted semantic tuples, avoiding the requirement to specify a fixed number of sentences for extraction. by following standard evaluation protocols, we demonstrate that our proposed method outperforms competitive baselines on cnn/dm and nyt. furthermore, to highlight the advantage of our approach, we examine the impact of fixed sentence extraction requirements on model generalization under a zero-shot setting. this involves assessing the model's performance on a different dataset, where the anticipated summary lengths deviate from those in the training data. in contrast to baselines that consistently output summaries of the same length for different documents, our method excels due to its capacity to dynamically extract sentences to cover the identified semantic tuples. we also compare the proposed approach to using chatgpt @xcite . to do this, we provide chatgpt with a prompt without specifying the number of sentences to extract. the results reveal the low performance of chatgpt in this task -a revelation consistent with recent work @xcite . upon manual examination of the extractive summaries output by chatgpt, we discovered that chatgpt tends to optimize recall by selecting more sentences than expected. while chatgpt has demonstrated commendable capabilities across a diverse spectrum of tasks, our observations suggest that current fine-tuning approaches on smaller models may still present promising avenues for enhancing extractive summarization performance. our contributions can be summarized as follows: ## fine-grained semantic tuples construction in this section, we introduce the process of converting sentences from text into semantic tuples, which in our case are fine-grained semantic representations based on predicate-argument structures @xcite . the overall pipeline is shown in figure 1 . this is different from conventional approaches for extractive summarization, which rely on sentences as the primary granularity. to extract semantic tuples from a given source document, we employed stanford corenlp @xcite to first perform coreference resolution, thereby replacing identified mentions (e.g., pronouns) with their corresponding entity names. subsequently, an ie tool was employed to extract fine-grained semantic information from the sentences: we conducted a comparative analysis of different ie systems, including allennlp openie @xcite , stanford corenlp ope-nie @xcite , knowledge base-based openie @xcite , and amr @xcite . our selection was based on factors such as system accessibility and ie performance on summarization datasets. ultimately, we chose the openie tool provided by allennlp, which enables us to extract a list of propositions from each sentence, effectively yielding semantic tuple candidates. each semantic tuple is composed of a single predicate and a variable number of arguments. to ensure the data's integrity, we excluded any semantic tuples with arguments exceeding 20 tokens. moreover, we associated each predicate with its arguments based on predicted argument roles, adhering to the conventions established by @xcite where 'arg0' denotes the agent, "arg1" refers to the direct object, and "arg2" represents the indirect object. however, upon inspecting the results, we noted that the extracted semantic tuples exhibited certain inaccuracies in the predicted argument roles, potentially leading to semantic ambiguities. considering the high performance of llms in various tasks @xcite , we leveraged an llm to identify the most plausible semantic tuples from all candidates to address this concern. specifically, for each semantic tuple, we generated permutations by exploring all possible argument role assignments, i.e., "arg0" to "arg2", and concatenated each candidate accordingly to form a text representation. for instance, one candidate semantic tuple {became, arg1: evnika saadvakass, arg2: a youtube sensa-tion} would have been transformed into "became evnika saadvakass a youtube sensation". to find the most appropriate semantic tuple, we input all candidate texts into an llm 1 , calculating their perplexity. the candidate with the lowest perplexity was regarded as aligning best with the language model, thus warranting selection as the final semantic tuple. continuing with the previous example, after querying the language model with all different combinations, we obtain {arg0: evnika saadvakass, became, arg1: a youtube sen-sation} as the ultimate result. this pipeline enables us to enhance the accuracy and reliability of the extracted semantic tuples, ultimately contributing to a more robust knowledge representation. ## methodology the overview of the proposed method is shown in figure 2 . given a source document @xmath0, x 2 , • • • , x |x| } consisting of a sequence of sentences x i , we consider each sentence x i to have a semantic meaning representation in the form of predicate-argument structures @xcite , namely semantic tuples. the process of extractive summarization entails the following steps: ## experiments and results we introduced our experimental settings and results in this section, and included the implementation details in appendix sec. a. additionally, we follow previous work in text summarization and related tasks @xcite to mainly report rouge-1 (unigram), rouge-2 (bigram), and rouge-l (longest common subsequence) scores @xcite for evaluation. ## conclusion this study introduces an innovative, fine-grained, and autoregressive technique for extractive summarization via the extraction of semantic tuples. diverging from conventional strategies that focus on sentence-level extraction, our approach operates at a more nuanced and semantically-structured granularity. during the inference process, we use a greedy approach to select sentences to cover the extracted semantic tuples, eliminating the necessity to predefine a fixed number of sentences for extraction. empirical assessments conducted on cnn/dm and nyt establish the superior efficacy of our method compared to competitive baselines. furthermore, our investigation into the generalization capabilities of our approach within zeroshot settings highlights its remarkable adaptability across diverse summary lengths, outperforming baseline models and achieving better generalization. in addition, we explored the suitability of prominent large language models for the task of extractive summarization by evaluating chatgpt's performance in generating extractive summaries. we found chatgpt to underperform relative to baseline models, emphasizing the potential of finetuning-centric methodologies for enhancing summarization performance.