Generating Commonsense Counterfactuals for Stable Relation Extraction (2023)

Abstract: Recent studies on counterfactual augmented data have achieved great success in the coarse-grained natural language processing tasks. However, existing methods encounter two major problems when dealing with the fine-grained relation extraction tasks. One is that they struggle to accurately identify causal terms under the invariant entity constraint. The other is that they ignore the commonsense constraint. To solve these problems, we propose a novel framework to generate commonsense counterfactuals for stable relation extraction. Specifically, to identify causal terms accurately, we introduce an intervention-based strategy and leverage a constituency parser for correction. To satisfy the commonsense constraint, we introduce the concept knowledge base WordNet and design a bottom-up relation expansion algorithm on it to uncover commonsense relations between entities. We conduct a series of comprehensive evaluations, including the low-resource, out-of-domain, and adversarial-attack settings. The results demonstrate that our framework significantly enhances the stability of base relation extraction models.

## introduction the relation extraction (re) task aims to extract the semantic relation between entities in the text. re can facilitate a wide range of downstream applications such as knowledge graph construction @xcite and question answering @xcite , having aroused much attention in recent years. typical re methods based on pre-trained language models like bert @xcite , or roberta @xcite . while getting impressive performance, they exhibit a certain degree of instability in re @xcite , even the chatgpt @xcite . instability has always been associated with neural networks. for example, @xcite finds that bert performs poorly in the de-biased set constructed by replacing high-frequency words with low-frequency ones. the instability is primarily caused by spurious correlations between the highfrequency words and labels. specifically, this can be attributed to neural networks tending to learn shortcuts between instances and labels @xcite . such property may hurt the model's stability in some challenging scenarios, e.g., lowresource or cross-domain scenarios. in attempts to mitigate spurious correlations, counterfactual augmented data (cad) is a rising trend. cad can be defined as: flipping the label of an instance with minimal editing @xcite , which explicitly provides localized views of decision boundaries @xcite . current cad methods consist of the following three steps: (1) identifying causal terms, that are causally related to the label; (2) editing the causal term, to flip the label to other prepended ones; and (3) filtering out inconsistent instances, where the predicted label is different from the prepended one. existing methods have made great success in coarse-grained natural language processing tasks like sentiment analysis @xcite @xcite and natural language inference @xcite @xcite . however, the fine-grained re task is under-explored @xcite . due to the intrinsic property of re, i.e. invariant entity constraint and commonsense constraint, existing methods encounter the following two challenges. firstly, existing methods struggle to accurately identify causal terms, due to the constraint of invariant entities @xcite . that is, keeping the entities unaltered. the state-of-the-art methods primarily employ the model's gradient attributions to identify the causal term @xcite , i.e. selecting certain words with the highest gradient. however, entities convey more information @xcite , which implies that the gradient of causal terms can be confounded by the related entities. @xcite propose a syntactic-tailored strategy, i.e. taking the words along the shortest dependency path (sdp) between entities as causal terms. the drawback is its heavy reliance on syntactic parsing quality. secondly, and more importantly, all existing methods ignore the commonsense constraint. they assume that all labels can be treated as prepended ones. this is feasible in coarse-grained tasks. for example, in sentiment analysis, simply replacing the causal terms can flip the sentiment orientation, as shown in fig. 1 (a ). however, for re, the nature of entities imposes commonsense constraints on causal terms, as shown in fig. 1 (b ). in brief, the relation between entities should conform to the cognitive understanding of the real world. although existing methods pre-train a re base model on the existing dataset for consistency filtering @xcite @xcite , there remains an issue of entanglement with the base model. for example, in low-resource scenarios, the base model is severely affected by spurious correlations and propagates the errors backward. to address these two challenges, we present a commonsense counterfactual generation (ccg) framework for stable relation extraction. firstly, to identify the causal terms accurately, we present a novel intervention-based strategy, which can accurately identify the editable words by potential interventions. after obtaining preliminary results, we further utilize the constituency parser for correction. secondly, to satisfy the commonsense constraint, we leverage the concept knowledge base wordnet @xcite and utilize its hierarchical structure knowledge i.e. the common hypernyms, to interconnect entities, thereby expanding their relations. we design a bottom-up relation expansion algorithm for implementation, which can uncover commonsense relations between entities. we conduct comprehensive evaluations on a series of challenging scenarios, including the lowresource, out-of-domain, and adversarial-attack settings. the results demonstrate that our ccg generates more reasonable counterfactuals, which can consistently enhance re models' stability in various scenarios. furthermore, we also conduct a series of in-depth analysis studies. the results confirm the effectiveness of various ccg strategies and demonstrate that our generated counterfactuals are not only semantically readable but, more importantly, consistent with commonsense. ## conclusion in this paper, we solve the problems in existing methods of cad, i.e., struggling to accurately identify causal terms under the invariant entity constraint and ignoring the commonsense constraint. we aim to produce the most human-like counterfactuals, i.e., not only grammatically correct and semantically readable but also consistent with commonsense. to this end, we need to satisfy the variant entity and commonsense constraints. to meet the first one, we present a novel intervention-based strategy, which can accurately identify the editable words by potential interventions. to meet the second one, we exploit wordnet to expand potential relations such that the counterfactual generation is constrained by commonsense. extensive experimental results prove that our framework generates commonsense counterfactuals and outperforms the state-of-the-art baselines. it also consistently enhances the base models' stability across all settings.