Semantically Driven Sentence Fusion: Modeling and Evaluation (2020)

Abstract: Sentence fusion is the task of joining related sentences into coherent text. Current training and evaluation schemes for this task are based on single reference ground-truths and do not account for valid fusion variants. We show that this hinders models from robustly capturing the semantic relationship between input sentences. To alleviate this, we present an approach in which ground-truth solutions are automatically expanded into multiple references via curated equivalence classes of connective phrases. We apply this method to a large-scale dataset and use the augmented dataset for both model training and evaluation. To improve the learning of semantic representation using multiple references, we enrich the model with auxiliary discourse classification tasks under a multi-tasking framework. Our experiments highlight the improvements of our approach over state-of-the-art models.

## introduction generative nlp tasks, such as machine translation and summarization, often rely on human generated ground truth. datasets for such tasks typically contain only a single reference per example. this may result from the costly effort of human annotations, or from collection methodologies that are restricted to single reference resources (e.g., utilizing existing corpora; @xcite . however, typically there are other possible generation results, such as ground-truth paraphrases, that are also valid. failing to consider multiple references hurts the development of generative models, since such models are considered correct, at both training and evaluation, only if they follow one specific and often arbitrary generation path per example. in this work we address sentence fusion, a challenging task where a model should combine related sentences, which may overlap in content, into a compact coherent text. the output should preserve the information in the input sentences as well as their semantic relationship. it is a crucial component in many nlp applications, including text summarization, question answering and retrieval-based dialogues @xcite @xcite @xcite . our analysis of state-of-the-art fusion models @xcite indicates that they still struggle to correctly detect the semantic relationship between the input sentences, which is reflected in inappropriate discourse marker selection in the generated fusions ( §4). at the same time, discofuse @xcite , the dataset they use, is limited to a single reference per example, ignoring discourse marker synonyms such as 'but' and 'however'. noticing that humans tend to judge these synonyms as equally suitable ( §3), we hypothesize that relying on single references may limit the performance of those models. to overcome this limitation, we explore an approach in which ground-truth solutions are automatically expanded into multiple references. concretely, connective terms in gold fusions are replaced with equivalent terms (e.g., {'however', 'but'} ), where the semantically equivalent sets are derived from the penn discourse treebank 2.0 @xcite . human evaluation of a sample of these generated references indicates the high quality of this process ( §3). we apply our method to automatically augment the discofuse dataset with multiple references, using the new dataset both for evaluation and training. we will make this dataset publicly available. we then adapt a seq2seq fusion model in two ways so that it can exploit the multiple references in the new dataset ( §4). first, each training example with its multiple references is converted into multiple examples, each consisting of the input sentence pair with a different single reference fusion. hence, the model is exposed to a more diverse and balanced set of fusion examples. second, we direct the model to learn a common semantic representation for equivalent surface forms offered by the multiple references. to that end, we enhance the model with two auxiliary tasks: predicting the type of the discourse relation and predicting the connective pertaining to the fused output, as derived from the reference augmentation process. we evaluate our model against state-of-the-art models in two experimental settings ( §5, 6): indomain and cross-domain learning. the crossdomain setting is more challenging but may also be more realistic as labeled data is available only for the source domain but not for the target domain. to evaluate against multiple-reference examples, we measure the similarity of each generated fusion to each of the ground-truth fusions and report the highest score. this offers a more robust analysis, and reveals that the performance of fusion models is higher than previously estimated. in both settings, our model demonstrates substantial performance improvement over the baselines. ## multiple references in sentence fusion in this section we discuss the limitations of using single references for evaluation and training in sentence fusion. we then propose an automatic, precision-oriented method to create valid fusion variants. human-based evaluation confirms the reliability of our method, which generates pseudoreferences that are considered as suitable as the original references. finally, we demonstrate the effectiveness of the new references for fusion evaluation. our observations, which are used here for reference generation and evaluation, will also guide our fusion model design and training ( §4). ## a semantically directed fusion model in the previous section we have established the importance of multiple references for sentence fusion. we next show ( §4.1) that the state-of-the-art fusion model fails to detect the semantic relationship between the input sentences. we aim to solve this problem by expanding the training data to include multiple-references (mrs) per input example, where together these references provide a good coverage of the semantic relationship and are not limited to a single connective phrase. we then present our model ( §4.3), which utilizes auxiliary tasks in order to facilitate the learning of the semantic relationships from the mr training data ( §4.2). ## experimental setup we follow prior work and use the balanced version of discofuse ( §2). the dataset consists of 4.5m examples for wikipedia ('w') and 12m examples for sports ('s'), split to 98% training, 1% test and 1% dev. we evaluate fusion models both in in-domain and cross-domain settings (training in one domain and testing on the other domain). we denote with @xmath0, and similarly use @xmath1. we evaluate the following fusion models: 4 transformer -the transformer-based model by @xcite . lasertagger -the sequence tagging model by @xcite . ## results we report results with mr-exact ( for the sr evaluation, our auxbert is best performing, indicating the value of our multitask framework. on the other hand, training with the augmented dataset has a negative impact. this is we further analyze in the semantic relationship phenomena, on the other hand, require deeper understanding of the relationship between the two input sentences. they tend to be more difficult as they involve the choice of the right connective according to this relation. on these phenomena (top part of finally, to explore the relative importance of the different components of our model, we looked at model performance sliced by the clusters we introduced in §3.2 (see ## ablation analysis in this analysis, we focus on potential alternative evaluation measures. as mentioned in §2, a possible direction for solving issues in evaluation of sentence fusion-stemming from having a single reference-could be to use similarity-based evaluation metrics @xcite @xcite . we notice two limitations in applying such metrics for sentence fusion. first, similarity-based met-fusion bertscore mr-exact (r) ruby is the traditional birthstone for july and is usually more pink than garnet, -however some rhodolite garnets have a similar pinkish hue to most rubies . (g) although ruby is the traditional birthstone for july and is usually more pink , 0.9670 1 than garnet some rhodolite garnets have a similar pinkish hue to most rubies. (b) ruby is the traditional birthstone for july and is usually more pink than garnet , 0.9893 0 thus some rhodolite garnets have a similar pinkish hue to most rubies . (r) the water level in the wells has risen. as a result, work on agricultural lands -is going on. (g) the water level in the wells has risen, hence work on agricultural lands 0.9713 1 is going on. (b) the water level in the wells has risen. however, work on agricultural lands 0.9745 0 is going on. (r) august 28, which is the second day after school starts, is their first away game. ## conclusions we studied the task of sentence fusion and argued that a major limitation of common training and evaluation methods is their reliance on a single reference ground-truth. to address this, we presented a method that automatically expands ground-truth fusions into multiple references via curated equivalence classes of connective terms. we applied our method to a leading resource for the task. we then introduced a model that utilizes multiple references by training on each reference as a separate example while learning a common semantic representation for surface form variances using auxiliary tasks for semantic relationship prediction in a multitasking framework. our model achieves state-of-the-art performance across a variety of evaluation scenarios. our approach for evaluating and training with generated multiple references is complementary to an approach that uses a similarity measure to match between similar texts. in future work, we plan to study the combination of the two approaches. machine translation (volume 2: shared task papers, day 1), pages 521-525, florence, italy. association for computational linguistics. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in international conference on learning representations. wei zhao, maxime peyrard, fei liu, yang gao, christian m. meyer, and steffen eger. 2019. moverscore: text generation evaluating with contextualized embeddings and earth mover distance. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 563-578, hong kong, china. association for computational linguistics. zhi min zhou, man lan, zheng yu niu, yu xu, and jian su. 2010. the effects of discourse connectives prediction on implicit discourse relation recognition. in proceedings of the 11th annual meeting of the special interest group on discourse and dialogue, pages 139-146. association for computational linguistics.