Counterfactual Generator: A Weakly-Supervised Method for Named Entity Recognition (2020)

Abstract: Past progress on neural models has proven that named entity recognition is no longer a problem if we have enough labeled data. However, collecting enough data and annotating them are labor-intensive, time-consuming, and expensive. In this paper, we decompose the sentence into two parts: entity and context, and rethink the relationship between them and model performance from a causal perspective. Based on this, we propose the Counterfactual Generator, which generates counterfactual examples by the interventions on the existing observational examples to enhance the original dataset. Experiments across three datasets show that our method improves the generalization ability of models under limited observational examples. Besides, we provide a theoretical foundation by using a structural causal model to explore the spurious correlations between input features and output labels. We investigate the causal effects of entity or context on model performance under both conditions: the non-augmented and the augmented. Interestingly, we find that the non-spurious correlations are more located in entity representation rather than context representation. As a result, our method eliminates part of the spurious correlations between context representation and output labels. The code is available at https://github.com/xijiz/cfgen .

## introduction the natural language processing community has witnessed the paradigm shift from small data to big data, such as transformer @xcite and its successors. it is not surprising that machine learning methods can easily surpass human performance if sufficient data is available @xcite . however, data acquisition is a challenging task for some special domains. for example, figure 1 : interventions on observational examples of named entity recognition. more details can be found in figure 3 medical concept normalization, a basic subtask of named entity recognition (ner) in the medical area, has always been troubled by lack of enough electronic health records due to the privacy protection. small data with selection biases @xcite often induce the poor performance of machine learning models on inputs whose distribution is different from that of training data, which yet seems trivial to humans. the same issues are also mentioned in terms like dataset bias, model robustness, and real understanding. in natural language inference, models trained on hypotheses-only (vs hypotheses-premises) can outperform a majority-class baseline @xcite . in reading comprehension, models trained on question-only or passage-only (vs question-passage) still achieve high accuracy @xcite , models predicted on a broken question (vs original question) still make the same correct prediction @xcite . the key challenge behind this phenomenon is caused by spurious correlations of statistical learning. spurious correlations can be vividly explained by an example in computer vision @xcite : if we consider an image dataset of cows and camels in their natural habitat, a classifier trained on this dataset will establish spurious correlations between the output labels (cows, camels) and the landscape of the image (green pastures, deserts). as a result, an image of cows taken on sandy beaches makes the classifier make a wrong prediction. in this background, we could not help thinking that is there any way to eliminate spurious correlations except more data annotated by humans? from a causal perspective, spurious correlations are caused by confounding factors rather than a direct or indirect causal path. if we directly intervene on the precursor variable in spurious correlations to create counterfactual data, we can eliminate the impact of spurious correlations in models to a certain degree @xcite . in this paper, based on the above analysis, we mainly focus on exploring the spurious correlations in ner from a causal perspective. we decompose the sentence into two different parts: entity and context, and rethink the relationship between them and the generalization ability of the ner model. considering the sentence "john lives in new york", we observe that the location entity "new york" and the context "john lives in" are highly correlated but are not causal to each other. in other words, we can intervene on the location entity to set it to another different location entity without destroying the sentence correctness at the grammatical level. therefore, we propose the counterfactual generator, which generates new counterfactual examples by the interventions on the existing observational examples. our method requires neither an additional entity dictionary nor a similar domain dataset. figure 1 demonstrates the intervention process for observational examples. we utilize new counterfactual examples to enhance the existing observational examples. experiments show that our method improves the generalization ability under limited observational examples. before the enhancement, we find that the model performance is mainly driven by entity representation. after the enhancement, the importance of entity representation increases in most cases, and generalization ability improves in all cases. we conclude that the non-spurious correlations between input features and output labels do locate in context representation @xcite , but the previous two phenomena show that they are more located in entity representation. in summary, our work have the following contributions: ## counterfactual generator in this section, we firstly define the ner problem. then, we present our method by introducing a structural causal model to describe the mechanism of the ner model inference. ## experiments in this section, we mainly evaluate our method across three ner datasets, including two medical concept recognition datasets, and a conventional ner dataset. ## discussion in this section, we will firstly review our previous results, and then try to answer some potential questions that others may ask for a deep understanding of our method. secondly, we will provide some real counterfactual examples to vividly illustrate our method. finally, some limitations of our method that we have found so far are presented to guide future research. we hope these limitations can help readers understand our method better. ## related works we introduce the related works from two aspects: data modification and causality recently, there is an increasing number of research works about data modification for providing the interpretability of neural models. for example, feng et al. and gururangan et al. reveal that neural models are overconfident in their predictions by reducing words or sentences; ebrahimi et al. also find that adversarial examples generated by some manipulations at a character-level or a word-level can trick neural classifier. additionally, considerable attention has been paid to utilize data modification for augmenting dataset or providing a supervised signal in the training process. for example, a rule-based data augmentation protocol has been proposed to provide a compositional inductive bias @xcite ; kaushik et al. create new counterfactual sentences by modifying the original sentence for ameliorating the harm of spurious correlations; xiong et al. introduce the typeconstrained entity replacements to provide extra training signal for learning better factual knowledge. interestingly, the above two points about data modification have high connections with causal inference because we can regard data modification as an intervention @xcite from a causal perspective. for instance, ilse et al. provide a theoretical foundation from a causal perspective for explaining current data augmentation in computer vision. kaushik et al. investigate the spurious correlations in two tasks: sentiment analysis and natural language inference. named entity recognition in this paragraph, we mainly focus on named entity recognition with limited supervision. one way to train the ner model with low-resource is dictionary-based distantly supervision @xcite @xcite which builds a dictionary of entities for creating training data without too much effort. few-shot learning is another promising way for training the ner model under limited supervision by transferring prior knowledge of the source domain to a new domain @xcite . there are also some works that focus on redefining ner as a different problem for reducing the need of hand-labeled training data. for example, linking rules @xcite based on votes recognize entities through whether adjacent elements in a sequence belong to the same class; lin et al. propose a new effective proxy of human explanation, "entity triggers", for encouraging label-efficient learning of ner models. ## conclusion in this paper, we propose a weakly-supervised method from a causal perspective and provide the interpretability of our method with the structural causal model. our method improves generalization ability under limited observational examples. our causal experiments suggest the spurious correlations are more located in entity representation rather than context representation. importantly, our method eliminates part of the spurious correlations between input features and output labels.