At Which Level Should We Extract? An Empirical Analysis on Extractive Document Summarization (2020)

Abstract: Extractive methods have been proven effective in automatic document summarization. Previous works perform this task by identifying informative contents at sentence level. However, it is unclear whether performing extraction at sentence level is the best solution. In this work, we show that unnecessity and redundancy issues exist when extracting full sentences, and extracting sub-sentential units is a promising alternative. Specifically, we propose extracting sub-sentential units based on the constituency parsing tree. A neural extractive model which leverages the sub-sentential information and extracts them is presented. Extensive experiments and analyses show that extracting sub-sentential units performs competitively comparing to full sentence extraction under the evaluation of both automatic and human evaluations. Hopefully, our work could provide some inspiration of the basic extraction units in extractive summarization for future research.

## introduction automatic text summarization aims to produce a brief piece of text which can preserve the most important information in it. the important contents are identified and then extracted to form the output summary @xcite . in recent decades, extractive methods have proven effective in many systems @xcite @xcite @xcite @xcite . in previous works, extractive summarization systems perform extraction on the sentence level @xcite @xcite . as the extraction unit, a sentence is a grammatical unit of one or more words that express a statement, question, request, etc. there are several advantages of extracting sentences. first, extractive systems are simpler, easier to develop, and faster during run-time in real application scenarios, compared with abstractive systems. moreover, original sentences in the input are naturally fluent and grammatically correct. finally, extracted sentences are factually faithful to the input document, compared with abstractive methods @xcite . despite the success of extractive systems, from previous works, it is still not clear whether extracting at sentence level is the best solution for extractive methods. there are several drawbacks of extracting the full sentences. the most obvious issue is that the extracted sentences may contain unnecessary information. some previous works have also noticed this problem and try to solve it by compressing or rewriting the extracted sentences @xcite @xcite . furthermore, extracted sentences may contain duplicate contents. thus, methods such as maximal marginal relevance (mmr) @xcite and sentence fusion @xcite are proposed to avoid or merge duplicate contents. the redundancy and unnecessity issues might be caused by extracting full sentences since an important sentence may also contain unnecessary information. besides, different importance sentences may have duplicate (un)important words. this inspires us that we can perform extraction at a finer granularity so that the important and unimportant contents could be separated. therefore, we argue that extracting sub-sentential units in a sentence could be a solution. as for the sub-sentential units, we mainly focus on the non-terminal nodes in a constituency parsing tree in this paper. in a given parsing tree of a sentence, the root node represents the entire sentence while the leaf nodes represent each corresponding lexical token. an extractive system could perform extraction on the non-terminal nodes which can express more fine-grained information. to keep the advantages of extractive methods, we choose the nodes which can still express a full statement. specifically, the nodes with the clause tag such as s and sbar are used for creating extraction units. in this paper, we conduct experiments and analyses to answer the following questions: q1 does extracting full sentences introduce unnecessary or duplicate information? ( §3) q2 can extracting sub-sentential units solve these problems of full sentence extraction? how to perform sub-sentential unit extraction? ( §4) q3 can extracting sub-sentential units improve the performance of extractive document summarization systems? ( §5, §6.1, §6.2) q4 does extracting sub-sentential units cause any other issues? ( §6.4) 2 related work extractive document summarization methods have proven effective and been extensively studied for decades. as an effective approach, extractive methods are popular and dominate the summarization research. as the very first work in 1950s, luhn (1958) uses lexical frequency to determine the importance of a sentence. many research works further develop extractive summarization methods. the most important step is to determine the sentence importance. many different methods have been proposed and they can be roughly categorized from different perspectives. from the perspective of having supervision or not, there are two major types: unsupervised methods and supervised methods. one of the difficulties in training an extractive system is the lack of extraction labels. the reason is that most of the reference summary is written by human experts, therefore, it is hard to find the exact appearance in the input document. without natural training labels, unsupervised and supervised methods treat extractive summarization as different problems. graph-based methods @xcite @xcite ) are very useful unsupervised methods. in these methods, the input document is represented as a connected graph. the vertices represent the sentences, and the edges between vertices have attached weights that show the similarity of the two sentences. the score of a sentence is the importance of its corresponding vertex, which can be computed using graph algorithms. supervised methods for extractive summarization create training labels manually. @xcite directly train regression models using rouge scores as the supervision. @xcite @xcite search the oracle extracted sentences as the training labels. @xcite propose treating document summarization as a sequence labeling task. they first encode the sentences in the document and then classify each sentence into two classes, i.e., extraction or not. @xcite propose a system called summarunner with more features, which also treat extractive document summarization as a sequence labeling task. @xcite propose using pointer networks @xcite to repeatedly extract sentences. recently, reinforcement learning (rl) is also introduced in unsupervised extractive summarization @xcite . @xcite treat sentence extraction as a bandit problem so that they can train a rl-based system whose reward is the rouge scores. @xcite propose that using human judgments as reward is better than rouge. however, these methods still need reward as a signal, which differ from the previously introduced fully unsupervised methods. ## q1: review of extracting full sentence performing sentence extraction in summarization systems have proven effective in previous works @xcite @xcite . despite the success of these systems, it is still unclear whether performing content extraction at the sentence level is the best solution. in this section, we will examine the drawbacks of extraction at the sentence level. ## q2: efficacy of extracting sub-sentential units in this section, we propose an alternative to performing extraction on full sentence for extractive document summarization. instead, we perform extraction on sub-sentential units. specifically, the extraction units are based on the clause nodes in the constituency parsing tree of a sentence. figure 2 shows two simplified examples of the constituency tree. the root node in a constituency tree represents the entire sentence, and the leaf node represents its corresponding lexical token. extracting on the root node is essentially extracting the full sentence, while extracting on the leaf node is doing compressing by extracting words @xcite . we perform extraction on the non-terminal nodes which can both express a relatively complete meaning and be human-readable. therefore, the clause nodes, such as s and sbar, become a good choice. in this section, we introduce how to perform extraction on the sub-sentential units, and present a bert-based model for it.