C o V ariance-based Causal Debiasing for Entity and Relation Extraction (2023)

Abstract: Joint entity and relation extraction tasks aim to recognize named entities and extract relations simultaneously. Suffering from a variety of data biases, such as data selection bias, and distribution bias (out of distribution, long-tail distribution), serious concerns can be witnessed to threaten the model’s transferability, robustness, and generalization. In this work, we address the above problems from a causality perspective. We propose a novel causal framework called c \underline{\textbf{o}} variance and \underline{\textbf{v}} ariance \underline{\textbf{o}} ptimization framework (OVO) to optimize feature representations and conduct general debiasing. In particular, the proposed \underline{\textbf{c}} ovariance \underline{\textbf{op}} timizing (COP) minimizes characterizing features’ covariance for alleviating the selection and distribution bias and enhances feature representation in the feature space. Furthermore, based on the causal backdoor adjustment, we propose \\underline{\textbf{v}} ariance \underline{\textbf{op}} timizing (VOP) separates samples in terms of label information and minimizes the variance of each dimension in the feature vectors of the same class label for mitigating the distribution bias further. By applying it to three strong baselines in two widely used datasets, the results demonstrate the effectiveness and generalization of OVO for joint entity and relation extraction tasks. Furthermore, a fine-grained analysis reveals that OVO possesses the capability to mitigate the impact of long-tail distribution.

## introduction named entity recognition (ner, @xcite ) and relation extraction @xcite ) are both the key fundamental tasks of information extraction and both significant predecessor tasks for building knowledge graphs that are used widely in daily life @xcite @xcite . there are two major methods to extract the final result. furthermore, we show how the recently developed efficient subwindow search(ess) procedure can be integrated into the last stage of our method. the confidence at the upper right is predicted by pl-marker @xcite . the groundtruth relation of the example is part-of, but due to the bias in the train set, the no-relation is inferred. after using our ovo, part-of is predicted correctly. one is to build a separate model (pipeline) to get the result in one step @xcite , and the other is to use multiple models (joint) to extract entities and relations in turn @xcite . suffering from various biases in data, such as data selection bias, out-of-distribution, and longtail distribution @xcite , models may learn spurious correlations and make erroneous predictions. although we would like to be able to obtain a uniform (or unbiased) data set, such a dataset is almost nonexistent in real applications. various biases are always mixed up in real scenarios, for example, selected data in a way that differs from the target population (data selection bias), training and testing data distribution shift (out-of-distribution), the labels with numerous samples may have more chances to inference (long-tail distribution lead to predicting majority bias), and some of the data biases are even unknown. because statistical correlation is the basis of supervised learning algorithms, the learning algorithms may learn many spurious correlations due to these biases. finally, this leads to a greatly re-duced predictive ability of the models. as shown in fig. 1 (left), there are different degrees of the data bias of long-tail distribution in scierc @xcite , which is common for other datasets as well, making models focus more on the labels with numerous samples and performing poorly on the others (fewer chances to the labels with few samples). in recent years, debiasing has become a significant research focus. there are different ways to address the data bias by reweighting samples @xcite @xcite . these methods focus on the reweights in the sample space and ignore spurious correlations due to data selection bias. recently, stable learning @xcite @xcite proposed serial ways to de-bias the spurious correlation to find the causal effect of each input feature on the basis of the correlation. specifically, these methods require extensive calculation resources to identify causality and focus solely on causal effects between features. additionally, they fail to consider the causal effects for each feature dimension within the same class label. in this work, we introduce a method called the covariance and variance optimization framework (ovo), to optimize the feature vectors from two aspects, decorrelating between features and each feature dimension of the same class label. our contributions can be summarized as follows: ## related work stable learning aims to research how to reduce the agnostic data selection bias and model misspecification problem and make the feature more causal by reweighting samples. crlr @xcite proposes an equation to evaluate the degree of confounders called causal regularized logistic regression and gains the optimal weight by minimizing the confounder. srdo @xcite considers the smallest feature of the covariance matrix as the degree of confounder. stablenet @xcite adopts stable learning technology into image classification tasks using deep learning models and the results show the strong domain adaptation of the proposed models. span-level ner and re models are the approaches using the boundaries of spans as a total feature to recognize the entities and the relation in the training phase of the model, we insert cop and vop as extra optimization for feature vectors. for cop, there is an inner loop to learn the optimal w using gradient descent, and loss w is the loss function of w calculated according to eq. 4. for vop, backdoor adjustment is used to reduce the bias from datasets. note that loss w only computes the gradient of w, and only the final optimal weights w created in cop will be applied to the model training process. the loss v op will be used in model training directly. loss orig means the loss employed on the original method; o means zero matrix. variance and covariance are two important metrics in deep learning. in confidence learning (mukherjee and awadallah, 2020), they are negatively correlated with the confidence of models and samples. moreover, in active learning @xcite , they are correlated with generalization error and bias which can be indirectly reduced by minimizing variance or covariance. recently, ideal @xcite uses variance as one of the measures for the inconsistency of unlabeled samples. in this work, we use them to estimate the feature representation and correlation between features respectively. ## method in this section, we first briefly introduce the general inference process of the joint entity and relation extraction methods. then, we will describe covariance optimizing (cop) and variance optimizing (vop). ## conclusion in