Uncovering Main Causalities for Long-tailed Information Extraction (2021)

Abstract: Information Extraction (IE) aims to extract structural information from unstructured texts. In practice, long-tailed distributions caused by the selection bias of a dataset may lead to incorrect correlations, also known as spurious correlations, between entities and labels in the conventional likelihood models. This motivates us to propose counterfactual IE (CFIE), a novel framework that aims to uncover the main causalities behind data in the view of causal inference. Specifically, 1) we first introduce a unified structural causal model (SCM) for various IE tasks, describing the relationships among variables; 2) with our SCM, we then generate counterfactuals based on an explicit language structure to better calculate the direct causal effect during the inference stage; 3) we further propose a novel debiasing approach to yield more robust predictions. Experiments on three IE tasks across five public datasets show the effectiveness of our CFIE model in mitigating the spurious correlation issues.

## introduction the goal of information extraction (ie) is to detect the structured information from unstructured texts. previous deep learning models for ie tasks, such as named entity recognition (ner; @xcite , relation extraction (re; @xcite and event detection (ed; nguyen and grishman 2015), are largely proposed for learning under some reasonably balanced label distributions. however, in practice, these labels usually follow a long-tailed distribution @xcite . figure 1 shows such an unbalanced distribution on the ace2005 @xcite dataset. as a result, performance on the instance-scarce (tail) classes may drop significantly. for example, on an existing model for ner @xcite , the macro f1 score of instance-rich (head) classes head classes tail classes 20% training instances 80% training instances can be 71.6, while the score of tail classes sharply decreases to 41.7. the underlying causes for the above issue are the biased statistical dependencies between entities foot_0 and classes, known as spurious correlations @xcite . for example, an entity gardens appears 13 times in the training set of ontonotes5.0, with the ner tag location loc, and only 2 times as organization org. a classifier trained on this dataset tends to build spurious correlations between gardens and loc, although gardens itself does not indicate a location. most existing works on addressing spurious correlations focus on images, such as re-balanced training @xcite , transfer learning @xcite and decoupling @xcite . however, these approaches may not be suitable for natural language inputs. recent efforts on information extraction @xcite incorporate prior knowledge, which requires data-specific designs. causal inference @xcite is promising in tackling the above spurious correlation issues caused by unbalanced data distribution. along this line, various causal models have been proposed for visual tasks @xcite . despite their success, these methods may be unsatisfactory on textual inputs. unlike images, which can be easily disentangled with detection or segmentation methods for causal manipulation, texts rely more on the context involving complex syntactic and semantic structures. hence it is impractical to apply the methods used in images to disentangle tokens' representations. recent causal models @xcite @xcite ) on text classification eliminate biases by replacing target entities or antonyms. these methods do not consider structural information, which has proven effective for various ie tasks as they are able to capture non-local interactions @xcite . this motivates us to propose a novel framework termed as counterfactual information extraction (cfie). different from previous efforts, our cfie model alleviates the spurious correlations by generating counterfactuals @xcite based on the syntactic structure @xcite . from a causal perspective, counterfactuals state the results of the outcome if certain factors had been different. this concept entails a hypothetical scenario where the values in the causal graph can be altered to study the effect of the factor. intuitively, the factor that yields the most significant changes in model predictions has the greatest impact and is therefore considered as the main effect. other factors with minor changes are categorized as side effects. in the context of ie with language structures, counterfactual analysis answers the question on "which tokens in the text would be the key clues for re, ner or ed that could change the prediction result". with that in mind, our cfie model is proposed to explore the language structure to eliminate the bias caused by the side effects and maintain the main effect for prediction. we show the effectiveness of cfie on three representative ie tasks including ner, re and ed. our code and the supplementary materials are available at https: //github.com/heyyyyyyg/cfie . specifically, our major contributions are: ## model figure 2 demonstrates the proposed cfie method using an example from the ace2005 dataset @xcite on the ed task. as shown in figure 2 (a), two event types "life:die" and "sw:quit" of the trigger killed have 511 and 19 training instances, respectively. such an unbalanced distribution may mislead a model to build spurious correlations between the trigger word killed and the type "life:die". the goal of cfie is to alleviate such incorrect correlations. cfie employs scm @xcite as causal diagram as it clearly describes relationships among variables. we give the formulation of scm as follows. scm: without loss of generality, we express scm as a directed acyclic graph (dag) @xmath0, f, u}, where the set of observables (vertices) are denoted as @xmath1, ..., v n }, the set of functions (directed edges) as @xmath2, ..., f n }, and the set of exogenous variables (e.g., noise) @xcite as @xmath3, ..., u n } for each vertice. here n is the number of nodes in g. in the deterministic case where u is given, the values of all variables in scm are uniquely determined. each observable v i can be derived from: ## related work long-tailed ie: re @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , ner @xcite @xcite @xcite are mainstream ie tasks in nlp. for the long-tailed ie, recent models @xcite leverage external rules or transfer knowledge from data-rich classes to the tail classes. plenty of re-balancing models are also proposed, including re-sampling strategies @xcite ) that aim to alleviate statistical bias from head classes, and re-weighting approaches @xcite which assign balanced weights to the losses of training samples from each class to boost the discriminability via robust classifier decision boundaries. another line is decoupling approaches @xcite that decouple the representation learning and the classifier by direct re-sampling. different from the above works that are based on conventional approaches, we tackle task instances #instances/class in training set predictions of dep-guided lstm predictions of sgg-tde predictions of cfie ner the wangma computer company was named for the wubi font, the coding method of chinese characters invented by the famous computer expert -yongmin wang. s-norp : 568 o : 243 i-org : 42 s-language : 20 …… ed the camp chapman attack was a suicide attack by human khalil abu-mulal al-balawi against the central intelligence agency facility inside forward operating base chapman on december 30, 2009. s-attack : 545 o : 230 s-terroism : 32 e-terrorism : 12 …… s -n o r p s -l a n g u a g e o s -g p e 72.38 15.07 s -n o r p s -l a n g u a g e o s -p e r s o n 31.63 30.31 s -l a n g u a g e s -n o r p b -n o r p o 27.60 35.17 s -a tt a c k o s -t e rr o ri s m e -t e rr o ri s m 42.70 4.07 s -a tt a c k s -t e rr o ri s m e -t e rr o ri s m o 32.82 5.17 e -t e r r o r is m s -a tt a c k s -t e r r o r is m o 20.70 26.07 figure 11: two cases selected from ontonote5.0 and maven for ner and ed tasks respectively, with unbalanced distributions for the targeting entity and event trigger. the two baseline models dep-guided lstm and sgg-tde tend to predict incorrect results caused by the spurious correlations, while our proposed cfie model is able to yield better predictions. causal inference: causal inference @xcite has been applied in many areas, including visual tasks @xcite @xcite @xcite @xcite , model robustness and stable learning @xcite @xcite @xcite , generation @xcite , language understanding @xcite , and recommendation systems @xcite @xcite @xcite @xcite . works most related to ours are @xcite that generates counterfactuals for weakly-supervised ner and text classifications, respectively. our method is remotely related to @xcite proposed for image classifications. the key differences between our methods and previous ones: 1) counterfactuals in our method are generated by a task-specific pruned dependency structure on various ie tasks. while in previous works, counterfactuals are generated by replacing the target entity with another entity or their antonyms @xcite , or simply masking the targeting objects in an image @xcite . these method do not consider the complex language structure that has been proven useful for ie tasks. 2) compared with previous method sgg-tde @xcite , our inference mechanism is more robust for various ie tasks, simultaneously mitigating the spurious correlations and strength-ening salient context. ## concluding remarks this paper presents cfie, a novel framework for tackling the long-tailed ie issues in the view of causal inference. extensive experiments on three popular ie tasks, named entity recognition, event detection, and relation extraction, show the effectiveness of our method. our cfie model provides a new perspective on tackling spurious correlations by exploring language structures based on structured causal models. we believe that our models may also find applications in other nlp tasks that suffer from spurious correlation issues caused by unbalanced data distributions. our future work includes developing more powerful causal models for the long-tailed distribution problems using the task-specific language structures learned from the data. we are also interested in addressing the spurious correlations in various vision and language tasks @xcite @xcite @xcite @xcite .