Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction (2021)

Abstract: Low-resource Relation Extraction (LRE) aims to extract relation facts from limited labeled corpora when human annotation is scarce. Existing works either utilize self-training scheme to generate pseudo labels that will cause the gradual drift problem, or leverage meta-learning scheme which does not solicit feedback explicitly. To alleviate selection bias due to the lack of feedback loops in existing LRE learning paradigms, we developed a Gradient Imitation Reinforcement Learning method to encourage pseudo label data to imitate the gradient descent direction on labeled data and bootstrap its optimization capability through trial and error. We also propose a framework called GradLRE, which handles two major scenarios in low-resource relation extraction. Besides the scenario where unlabeled data is sufficient, GradLRE handles the situation where no unlabeled data is available, by exploiting a contextualized augmentation method to generate data. Experimental results on two public datasets demonstrate the effectiveness of GradLRE on low resource relation extraction when comparing with baselines.

## introduction relation extraction (re) aims to discover the semantic relation that holds between two entities and transforms massive corpus into structured triplets (entity head , relation, entity tail ). for example, from "a letter head was delivered to my office tail ...", we can extract a relation entity-destination between head and tail entities. neural re methods leverage high-quality annotated data or human curated knowledge bases to achieve decent results @xcite . however, these manually labeled data would be laborintensive to obtain. this motivates a low resource relation extraction (lre) task where annotations are scarce. lots of efforts are devoted to improve the model generalization ability beyond learning directly from existing, limited annotations. distant supervision methods leverage facts stored in external knowledge bases (kbs) to obtain annotated triplets as the supervision @xcite . however, these methods should make a strong assumption that two co-occurring entities convey kb relations regardless of specific contexts, which makes model generate relations based on contextless rules and limits the generalization ability. to leverage unlabeled data, @xcite propose to assign pseudo labels on unlabeled data and leverage pseudo labels to iteratively improve the generalization capability of the model. however, during the training process, self-training models suffer from the gradual drift problem @xcite caused by noisy pseudo labels. @xcite alleviate the noise in pseudo labels by adopting a meta-learning scheme during pseudo label generation, then leveraging pseudo label selection and exploitation scheme to obtain high-confidence pseudo labels. however, when limited annotations are directly used during training, the trained models inevitably possesses selection bias towards, if not overfit on, limited labeled data, which impedes lre models from further generalizing beyond the annotations. to improve the generalization ability for lre, we propose to use existing annotations as a guide-line instead of having them directly involved in training, as well as introducing an explicit feedback loop when consuming annotations. more specifically, we first encourage pseudo-labeled data to imitate labeled data on the gradient descent directions during the optimization process. we illustrate this idea in figure 1 . g l represents the average gradient descent direction on labeled data. g u and g u represent the correct and incorrect pseudo labels on unlabeled data, which guides the gradient descent direction in a positive/negative fashion @xcite @xcite . based on how well the pseudo-labeled data mimics the instructive gradient descent direction obtained from limited labeled data, we then design a reward to quantify the behavior and aim to use the reward as an explicit feedback. this learnable setting can be naturally formulated into a reinforcement learning framework, which aims to learn an imitation policy that maximizes the reward through trial and error. when comparing with methods where annotations are directly used in the traditional learning schema, this formulation also allows a feedback mechanism and thus increases generalization ability beyond limited annotations. we name our method as gradient imitation reinforcement learning in this paper. we propose a framework called gradlre, which integrates gradient imitation reinforcement learning and is able to handle two major scenarios in lre: 1) a typical scenario when limited labeled data and large amounts of unlabeled data are available, and an extreme yet practical scenario where 2) even unlabeled data is absent: only limited labeled data is available. gradlre handles the former scenario via pseudo labeling optimized through gradient imitation reinforcement learning and tackles the later scenario by using a contextualized data augmentation module. to summarize, the main contributions of this work are as follows: ## proposed model the proposed framework gradlre consists of three modules: relational label generator (rlg), gradient imitation reinforcement learning (girl) and contextualized data augmentation (cda). as illustrated in figure 2 , two low resource relation extraction scenarios are handled. for the first scenario where limited labeled data and large amounts of unlabeled data are available, the input of rlg is labeled data and unlabeled data. labeled data consists of sentences and relation mentions: [sentence, entity 1 , entity 2 , relation]. for the second scenario where only limited labeled data is available, we adopt cda to generate unlabeled data and utilize these unlabeled data the same way as in the first scenario. in a traditional self-training setting, we fine-tune rlg directly using the labeled data, and let rlg assign pseudo labels on unlabeled data as pseudolabeled data. however, we argue that such learning paradigm suffers from selection bias due to the lack of feedback loops: the bias occurs when a model itself influences the generation of data which is later used for training. in this work, we complete the feedback loop and alleviate such bias by leveraging girl to learn a policy that maximizes the likelihood between the expected gradient optimization direction from pseudo labels, and the average gradient optimization direction on labeled data. ## experiments we conduct extensive experiments on two datasets to prove the effectiveness of our gradient imitation reinforcement learning for low resource relation extraction tasks, and give a detailed analysis of each module to show the advantages of gradlre. ## related work relation extraction aims to predict the binary relation between two entities in a sentence. recent literature leverage deep neural network to encode the features among two entities from sentences, and then classify these features into pre-defined specific relation categories. these methods could gain decent performance when sufficient labeled data is available @xcite @xcite . however, it's labor-intensive to obtain large amounts of manual annotations on corpus. low resource relation extraction methods gained a lot of attention recently @xcite @xcite @xcite , since these methods require fewer labeled data and deep neural networks could expand limited labeled information by exploiting information on unlabeled data to iteratively improve the performance. one major method is the self-training work proposed by @xcite . self-training incrementally assigns pseudo labels to unlabeled data and leverages these pseudo labels to iteratively improve the classification capability of the model. however, these methods always endure gradual drift problem @xcite @xcite @xcite : during the training process, the generated pseudo label data contains noise and could not been corrected through the model itself. using these pseudo label data iteratively cause the model to deviate from the global minima. our work alleviates this problem by encouraging pseudo-labeled data to imitate the gradient optimization direction on the labeled data, and introducing an effective feedback loop to improve generalization ability via reinforcement learning. reinforcement learning is widely used in nature language processing @xcite @xcite @xcite . these methods are all designed with rewards to force the correct actions to be executed during the model training process, so as to improve model performance. @xcite applies policy gradient method to model future reward in a joint entity and relation extraction task. in our work, we define reward as the cosine similarity between gradient vectors calculated from pseudo-labeled data and labeled data. data augmentation methods are leveraged in natural language processing to improve the generalization ability of the model by generating discriminative samples @xcite @xcite . @xcite contextually augment data by replacing the one-hot representation of a word by a distribution provided by bert over the vocabulary. however, they only consider the replacement of a word which limits its capability to expand the sentence semantics @xcite . in our work, we use [mask] to replace a span of words and leverage bert masked language modeling task to fill the [mask]. ## conclusion in this paper, we propose a reinforcement learning framework model gradlre for low resource re. different from conventional self-training models which endure gradual drift when generating pseudo labels, our model encourages pseudo-labeled data to imitate the gradient optimization direction in labeled data to improve the pseudo label quality. we find our learning paradigm gives more instructive, explicit, and generalizable signals than the implicit signals that are obtained by training model directly with labeled data. contextualized data augmentation is proposed to handle the extremely low resource re situation where no unlabeled data is available. experiments on two public datasets show effectiveness of gradlre and augmented data over competitive baselines.