Causal Intervention for Abstractive Related Work Generation (2023)

Abstract: Abstractive related work generation has attracted increasing attention in generating coherent related work that helps readers grasp the current research. However, most existing models ignore the inherent causality during related work generation, leading to spurious correlations which downgrade the models’ generation quality and generalizability. In this study, we argue that causal intervention can address such limitations and improve the quality and coherence of generated related work. To this end, we propose a novel Causal Intervention Module for Related Work Generation (CaM) to effectively capture causalities in the generation process. Specifically, we first model the relations among the sentence order, document (reference) correlations, and transitional content in related work generation using a causal graph. Then, to implement causal interventions and mitigate the negative impact of spurious correlations, we use do-calculus to derive ordinary conditional probabilities and identify causal effects through CaM. Finally, we subtly fuse CaM with Transformer to obtain an end-to-end related work generation framework. Extensive experiments on two real-world datasets show that CaM can effectively promote the model to learn causal relations and thus produce related work of higher quality and coherence.

## introduction a comprehensive related work usually covers abundant reference papers, which costs authors plenty of time in reading and summarization and even forces authors to pursue ever-updating advanced work @xcite . fortunately, the task of related work generation emerged and attracted increasing attention from the community of text summarization and content analysis in recent years @xcite . related work generation can be considered as a variant of the multi-document summarization task @xcite . distinct from multi-document summarization, related work generation entails comparison after the summarization of a set of references and needs to sort out the similarities and differences between these references @xcite . recently, various abstractive text generation methods have been proposed to generate related work based on the abstracts of references. for example, @xcite used the context of citation and the abstract of each cited paper as the input to generate related work. @xcite encoded the citation network and used it as external knowledge to generate related work. @xcite proposed a target-aware related work generator that captures the relations between reference papers and the target paper through a target-centered attention mechanism. equipped with well-designed encoding strategies, external knowledge, or novel training techniques, these studies have made promising progress in generating coherent related work. however, those models are inclined to explore and exploit spurious correlations, such as highfrequency word/phrase patterns, writing habits, or presentation skills, to build superficial shortcuts between reference papers and the related work of the target paper. such spurious correlations may harm the quality of the generated related work, especially when distribution shift exists between the testing set and training set. this is because spurious correlations are different from genuine causal relations. they often do not intrinsically contribute to the related work generation and easily cause the robustness problem and impair the models' generalizability @xcite . figure 1 illustrates the difference between causality and spurious correlation. the phrases "for example" and "later" are often used to bridge two sentences in related work. their usage may be attributed to writers' presentation habits about organizing sentence orders or the reference document relations corresponding to the sentences. ideally, a related work generation model is expected to learn the reference relation and distinguish it from the writing habits. however, previous generation models easily capture the superficial habitual sentence organization (a spurious correlation) instead of learning complex causal reference relations, especially when the habitual patterns frequently occur in the training set. in this case, the transitional phrases generated mainly based on writing habits are likely to be unsuitable and subsequently affect the content generation of related work during testing when the training and testing sets are not distributed uniformly. fortunately, causal intervention can effectively remove spurious correlations and focus on causal relations by intervening in the learning process. it not only observes the impact of the sentence order and document relation on generating transitional content, but also probes the impact of each possible order on the whole generation of related work, thereby removing the spurious correlations @xcite . accordingly, causal intervention serving as an effective solution allows causal relations to exert a greater impact and instruct the model to produce the correct content. accordingly, to address the aforementioned gaps in existing work for related work generation, we propose a novel causal intervention module for related work generation (cam), which effectively removes spurious correlations by performing the causal intervention. specifically, we first model the relations among sentence order, document relation, and transitional content in related work generation and identify the confounder that raises spurious correlations (see figure 2 ). then, we implement causal intervention that consists of three compo-nents: 1) primitive intervention cuts off the connection that induces spurious correlations in the causal graph by leveraging do-calculus and backdoor criterion @xcite , 2) context-aware remapping smoothens the distribution of intervened embeddings and injects contextual information, and 3) optimal intensity learning learns the best intensity of overall intervention by controlling the output from different parts. finally, we strategically fuse cam with transformer @xcite to deliver an end-to-end causal related work generation model. our main contributions are as follows: to the best of our knowledge, this work is the first attempt to introduce causality theory into the related work generation task. • we propose a novel causal intervention module for related work generation (cam) which utilizes causal intervention to mitigate the impact of spurious correlations. cam is subtly fused with transformer to derive an end-to-end causal related work generation model, enabling the propagation of intervened information. • extensive experiments on two real-world benchmark datasets demonstrate that our proposed model can generate related works of high quality and verify the effectiveness and rationality of bringing causality theory into the related work generation task. ## problem formulation given a set of reference papers @xmath0, ..., r |d| }, we assume the ground truth related work @xmath1, w 2 , ..., w m ), where r @xmath2, w i 2 , ..., w i |r i | ) denotes a single cited paper, w i j is the j-th word in r i , and w j is the j-th word in related work y . generally, the related work generation task can be formulated as generating a related work section ŷ = ( ŵ1 , ŵ2 , ..., ŵ m ) based on the reference input d and minimizing the difference between y and ŷ . considering that the abstract section is usually well-drafted to provide a concise paper summarization @xcite , we use the abstract section to represent each reference paper. ## methodology we first analyze the causalities in related work generation, identify the confounder that raises spurious correlations, and use a causal graph to model these relations. then, we implement cam to enhance the quality of related work through causal intervention. finally, we describe how cam, as an intervention module, is integrated with transformer to intervene in the entire generation process. the overall structure of our model is shown in figure 3 . ## conclusions in this paper, we propose a causal intervention module for related work generation (cam) to capture causalities in related work generation. we first model the relations in related work generation using a causal graph. the proposed cam implements causal intervention and enables the model to capture causality. we subtly fuse cam with transformer to obtain an end-to-end model to integrate the intervened information throughout the generation process. extensive experiments show the superiority of cam and demonstrate our method's effectiveness.