Finding Influential Instances for Distantly Supervised Relation Extraction (2022)

Abstract: Distant supervision (DS) is a strong way to expand the datasets for enhancing relation extraction (RE) models but often suffers from high label noise. Current works based on attention, reinforcement learning, or GAN are black-box models so they neither provide meaningful interpretation of sample selection in DS nor stability on different domains. On the contrary, this work proposes a novel model-agnostic instance sampling method for DS by influence function (IF), namely REIF. Our method identifies favorable/unfavorable instances in the bag based on IF, then does dynamic instance sampling. We design a fast influence sampling algorithm that reduces the computational complexity from \mathcal{O}(mn) to \mathcal{O}(1) , with analyzing its robustness on the selected sampling function. Experiments show that by simply sampling the favorable instances during training, REIF is able to win over a series of baselines which have complicated architectures. We also demonstrate that REIF can support interpretable instance selection.

## introduction to expand the training data for relation extraction (re), distant supervision (ds) was proposed by @xcite who assumed that if two entities are related in existing kbs, then all sentences contain both of them express this relation. however, this heuristic inevitably suffers from wrong labels @xcite and undermines model performance. for example, the sentence "bill gates redefined the software industry, ... said rob glaser, a former microsoft executive" does not mention the relation founder but is still treated as a positive training sample in ds. dealing with noisy instances in ds has been a focus in re. there are three main genres in the literature: (1) incorporating an attention module @xcite to allocate confidence level among instances in the same bag; (2) using reinforcement learning @xcite for instance selection; and (3) leveraging adversarial training @xcite to enhance the re model's robustness against noise. however, they are either black-box models thus unable to provide meaningful interpretation of sample selection or sensitive to datasets. more importantly, none of them is theoretically guaranteed to truely reduce the "noise" from the dataset. in this work, we propose to leverage influence function (if) to evaluate instance quality then do instance selection for ds. influence function is a powerful tool drawn from robust statistics @xcite . it is able to approximate the influence of a single data point on the whole model learned on the dataset. creating to this merit, it has been successfully utilized for inspecting outliers @xcite and denoising datasets @xcite based on shallow machine learning models, e.g., logistic regression. although @xcite extends if to interpreting deep networks, it is still elusive if it works for denoising datasets for deep networks. in this work, we develop the relation extraction by influence subsampling (reif) framework, which aims for denoising ds for deep learning re models. the high-level idea of reif is shown by fig. 1 . each instance is assigned a quality measure ϕ, from which its sampling probability is obtained via the sampling function π. accordingly, the better an instance's quality is, the more likely it is picked during training. we will explain the operational meaning of ϕ in section 3.2. in a nutshell, the main contributions of this paper are • we develop a novel if-based denosing framework for ds re, namely reif, for denoising re by sampling favorable training instances. ## related work there are a series of works trying to address the noisy label difficulty in ds by multi-instance learning (mil) @xcite @xcite . mil considers the training labels in bag level instead of instance level. each bag contains at least one instance with the labeled relation while the exact label of each instance is unknown. as mil being proved effective in relation extraction, it was firstly introduced to neural relation extraction by @xcite , where the piece-wise convolutional neural network (pcnn) was developed, and only one instance with the largest predicted probability was selected in each bag. later, attention @xcite @xcite @xcite , reinforcement learning @xcite @xcite , and adversarial training @xcite @xcite have been proposed for further improvement. however, above works usually require intense trials in fine-tuning of the hyper-parameters in practice, or are not interpretable to human-beings. in this work, we propose a model-agnostic and interpretable instance selection method via if, which is easy-to-use for most dl models without many hyperparameters to choose. 1 code is available in the supplementary materials. ## methodology in this section, we elaborate on the major steps of reif associated with the technical details and the theoretic foundation of measuring data quality by influences. also, an analysis supporting our choice of sampling function is given. ## efficient implementation recap eq. ( 12 ), computing φ i requires ϕ i,j in eq. ( 11 ) for @xmath0, . . . , m on all validation samples. as a result, the computation of all {φ i } n @xmath1. moreover, for dnns with massive parameters, computing the layer-wise gradients ∇ θ ℓ(θ) is intractable. these limitations prevent the use of if from dl re models. to address it, we here propose a rather efficient implementation of reif. we demonstrate how to reduce the complexity of calculating influences from o(mn) to o(n), then to o(1). in addition, we show how to compute the influence function by stochastic estimation. ## experiments we concentrate on the following research questions: rq1. how does our reif perform as compared with classical baselines? rq2. how does the sampling ratio r influence the performance of the reif? rq3. does the sigmoid function lead to more robust sampling than the deterministic sampling? rq4. how does the proposed dynamic instance sampling perform compared with the post-hoc sampling using if? ## manual evaluation & case study held-out evaluation usually suffers from false negative examples in freebase @xcite . to further check our method, we perform manual evaluation by choosing the entity pairs which are labeled as "na" but predicted a relation (not "na") with high confidence. the top-k precisions are reported in fig. 6 reports an example of calculating influences that support instance selection. picking a relation children as the example, influences and attention scores @xcite are computed, from which we can identify that the influences quantitatively measure their individual quality. recall in section 3.2 that the smaller influences indicate better data quality. the first and the last instances are clearly right and wrong, respectively, in terms of indicating the relation children between their entities. by contrast, the second one tends to be right ## conclusion & discussion in this work, we proposed an efficient subsampling scheme to find the influential instances for ds, namely reif. our method is model-agnostic, therefore it can be engaged in the majority of re models. reif can be generalized to other tasks which also confront noisy data. for instance, in other weak supervision scenarios such as active learning, our method can be an effective approach obtain s t ← h -1 t ∇ θ l( θt ) by stochastic estimation as done in eq. ( 22 ); 15: end for dimension d @xmath2. during training, we fix the batch size @xmath3, dropout ratio @xmath4.5, and use the adadelta (zeiler, 2012) with parameters ρ = 0.95 and ε = 10 -6 for optimization. since we find the default hyperparameters already lead superior performance of reif, we did not make further tuning.