Understanding Points of Correspondence between Sentences for Abstractive Summarization (2020)

Abstract: Fusing sentences containing disparate content is a remarkable human ability that helps create informative and succinct summaries. Such a simple task for humans has remained challenging for modern abstractive summarizers, substantially restricting their applicability in real-world scenarios. In this paper, we present an investigation into fusing sentences drawn from a document by introducing the notion of points of correspondence, which are cohesive devices that tie any two sentences together into a coherent text. The types of points of correspondence are delineated by text cohesion theory, covering pronominal and nominal referencing, repetition and beyond. We create a dataset containing the documents, source and fusion sentences, and human annotations of points of correspondence between sentences. Our dataset bridges the gap between coreference resolution and summarization. It is publicly shared to serve as a basis for future work to measure the success of sentence fusion systems.

## introduction stitching portions of text together into a sentence is a crucial first step in abstractive summarization. it involves choosing which sentences to fuse, what content from each of them to retain and how best to present that information @xcite . a major challenge in fusing sentences is to establish correspondence between sentences. if there exists no correspondence, it would be difficult, if not impossible, to fuse sentences. in ## annotating points of correspondence we cast sentence fusion as a constrained summarization task where portions of text are selected from each source sentence and stitched together to form a fusion sentence; rephrasing and reordering are allowed in this process. we propose guidelines for annotating points of correspondence (poc) between sentences based on halliday and hasan's theory of cohesion @xcite . we consider points of correspondence as cohesive phrases that tie sentences together into a coherent text. guided by text cohesion theory, we categorize poc into five types, including pronominal referencing ("they"), nominal referencing ("johnny kemp"), common-noun referencing ("five women"), repetition, and event trigger words that are related in meaning ("died" and "drowned"). an illustration of poc types is provided in a human annotator is instructed to identify a text span from each of the source sentences and summary sentence, thus establishing a point of correspondence between source sentences, and between source and summary sentences. as our goal is to understand the role of poc in sentence fusion, we do not consider the case if poc is only found in source sentences but not summary sentence, e.g., "kashyap said" and "said goa police superintendent kartik kashyap" in annotations are performed in two stages. stage one removes all spurious pairs that are generated by the heuristic, i.e. a summary sentence that is not a valid fusion of the corresponding two source sentences. human annotators are given a pair of sentences and a summary sentence and are asked whether it represents a valid fusion. the pairs identified as valid fusions by a majority of annotators move on to stage two. stage two identifies the corresponding regions in the sentences. as shown in figure 1 , annotators are given a pair of sentences and their summary and are tasked with highlighting the corresponding regions between each sentence. they must also choose one of the five poc types (repetition, pronominal, nominal, common-noun referencing, and event triggers) for the set of corresponding regions. we use amazon mechanical turk, allowing only workers with 95% approval rate and at least 5,000 accepted tasks. to ensure high quality annotations, we first run a qualification round of 10 tasks. workers performing sufficiently on these tasks were allowed to annotate the whole dataset. for task one, 2,200 instances were evaluated and 621 of them were filtered out. in total, we annotate points of correspondence for 1,599 instances, taken from 1,174 documents. similar to @xcite , we report fleiss' kappa judged on each word (highlighted or not), yielding substantial inter-annotator agreement (κ=0.58) for annotating points of correspondence. we include a reference to the original article that each instance was taken from, thus providing context for each instance. figure 2 shows statistics of poc occurrence frequencies and the distribution of different poc types. a majority of sentence pairs have one or two points of correspondence. only a small percentage (6.5%) do not share a poc. a qualitatively analysis shows that these sentences often have an implicit discourse relationship, e.g., "the two men speak. scott then gets out of the car, again, and runs away." in this example, there is no clear portion of text that is shared between the sentences; rather, the connection lies in the fact that one event happens after the other. most of the poc are a flavor of coreference (pronominal, nominal, or common-noun). few are exact repetition. further, we find that only 38% of points of correspondence in the sentence pair share any words (lemmatized). this makes identifying them automatically challenging, requiring a deeper understanding of what connects the two sentences. ## resolving coreference coreference resolution @xcite is similar to the task of identifying points of correspondence. thus, a natural question we ask is how well state-of-theart coreference resolvers can be adapted to this task. if coreference resolvers can perform reasonably well on poc identification, then these resolvers can be used to extract poc annotations to potentially enhance sentence fusion. if they perform poorly, coreference performance results can indicate areas of improvement for future work on detecting points of correspondence. in this paper, we compare three coreference resolvers on our dataset, provided by open-source libraries: stanford corenlp (manning et al., 2014), spacy @xcite , and allennlp @xcite . we base our evaluation on the standard metric used for coreference resolution, b-cubed algorithm @xcite , with some modifications. each resolver is run on an input pair of sentences to obtain multiple clusters, each representing an entity (e.g., johnny kemp) containing multiple mentions (e.g., johnny kemp; he; the singer) of that entity. more than one cluster can be detected by the coreference resolver, as additional entities may exist in the given sentence pair (e.g., johnny kemp and the police). similarly, in section §2, human annotators identified multiple poc clusters, each representing a point of correspondence containing one mention from each sentence. we evaluate how well the resolver-detected clusters compare to the human-detected clusters (i.e., pocs). if a resolver cluster overlaps both mentions for the gold-standard poc, then this resolver cluster is classified as a hit. any resolver cluster that does not overlap both poc mentions is a miss. using this metric, we can calculate precision, recall, and f1 scores based on correctly/incorrectly identified tokens from the outputs of each resolver. the results are presented in ## sentence fusion truly effective summarization will only be achievable when systems have the ability to fully recognize points of correspondence between sentences. it remains to be seen whether such knowledge can be acquired implicitly by neural abstractive systems through joint content selection and generation. we next conduct an initial study to assess neural abstractive summarizers on their ability to perform sentence fusion to merge two sentences into a summary sentence. the task represents an important, atomic unit of abstractive summarization, because a long summary is still generated one sentence at a time @xcite . we compare two best-performing abstractive summarizers: pointer-generator uses an encoderdecoder architecture with attention and copy mechanism @xcite ; transformer adopts a decoder-only transformer architecture similar to that of @xcite , where a summary is decoded one word at a time conditioned on source sentences and the previously-generated summary words. we use the same number of heads, layers, and units per layer as bert-base @xcite . in both cases, the summarizer was trained on about 100k instances derived from the train split of cnn/dailymail, using the same heuristic as described in ( §2) without poc annotations. the summarizer is then tested on our dataset of 1,599 fusion instances and evaluated using standard metrics @xcite . we also report how often each summarizer actually draws content from both sentences (%fuse), rather than taking content from only one sentence. a generated sentence counts as a fusion if it contains at least two non-stopword tokens from each sentence not already present in the other sentence. additionally, we include a concat-baseline creating a fusion sentence by simply concatenating the two source sentences. the results according to the rouge evaluation @xcite are presented in ## related work uncovering hidden correspondences between sentences is essential for producing proper summary sentences. a number of recent efforts select important words and sentences from a given document, then let the summarizer attend to selected content to generate a summary @xcite @xcite @xcite . these systems are largely agnostic to sentence correspondences, which can have two undesirable consequences. if only a single sentence is selected, it can be impossible for the summarizer to produce a fusion sentence from it. moreover, if non-fusible textual units are selected, the summarizer is forced to fuse them into a summary sentence, yielding output summaries that often fail to keep the original meaning intact. therefore, in this paper we had investigated the correspondences between sentences to gain an understanding of sentence fusion. establishing correspondence between sentences goes beyond finding common words. humans can fuse sentences sharing few or no common words if they can find other types of correspondence. fusing such disparate sentences poses a serious challenge for automated fusion systems @xcite @xcite @xcite @xcite . these systems rely on common words to derive a connected graph from input sentences or subject-verb-object triples @xcite . when there are no common words in sentences, systems tend to break apart. there has been a lack of annotated datasets and guidelines for sentence fusion. few studies have investigated the types of correspondence between sentences such as entity and event coreference. evaluating sentence fusion systems requires not only novel metrics @xcite @xcite but also high-quality ground-truth annotations. it is therefore necessary to conduct a first study to look into cues humans use to establish correspondence between disparate sentences. we envision sentence correspondence to be related to text cohesion and coherence, which help establish correspondences between two pieces of text. halliday and hasan ( 1976 ) describe text cohesion as cohesive devices that tie two textual elements together. they identify five categories of cohesion: [s1] palin actually turned against the bridge project only after it became a national symbol of wasteful spending. discofuse ( geva et al., 2019) [s1] melvyn douglas originally was signed to play sam bailey. ## conclusion in this paper, we describe a first effort at annotating points of correspondence between disparate sentences. we present a benchmark dataset comprised of the documents, source and fusion sentences, and human annotations of points of correspondence between sentences. the dataset fills a notable gap of coreference resolution and summarization research. our findings shed light on the importance of modeling points of correspondence, suggesting important future directions for sentence fusion.