Semi-supervised Relation Extraction via Data Augmentation and Consistency-training (2023)

Abstract: Due to the semantic complexity of the Relation extraction (RE) task, obtaining high-quality human labelled data is an expensive and noisy process. To improve the sample efficiency of the models, semi-supervised learning (SSL) methods aim to leverage unlabelled data in addition to learning from limited labelled data points. Recently, strong data augmentation combined with consistency-based semi-supervised learning methods have advanced the state of the art in several SSL tasks. However, adapting these methods to the RE task has been challenging due to the difficulty of data augmentation for RE. In this work, we leverage the recent advances in controlled text generation to perform high-quality data augmentation for the RE task. We further introduce small but significant changes to model architecture that allows for generation of more training data by interpolating different data points in their latent space. These data augmentations along with consistency training result in very competitive results for semi-supervised relation extraction on four benchmark datasets.

## introduction relation extraction is one of the essential components in constructing structured knowledge bases @xcite , performing interpretable question answering @xcite , improving web search, and many other information extraction pipelines. it aims to discover the semantic relation between a given head entity and tail entity based on the context in the input sentence. for example, given a sentence "the battle led to panic on the frontier, and settlers in the surrounding counties fled.", the goal is to extract the cause-effect relation between the head entity 'battle' and the tail entity 'panic'. the re task requires a high level of language understanding and involves a significant level of semantic complexity @xcite . due to this semantic complexity input sentence: the battle led to panic on the frontier, and settlers in the surrounding counties fled. ## related work semi-supervised learning for nlp semi-supervised learning algorithms can be categorized into two broad classes-1) self-ensembling methods and 2) self-training methods. self-ensembling methods leverage the smoothness and cluster/low-density assumptions of the latent space @xcite . they train the models to make consistent predictions under various kinds of perturbations to either a) the data @xcite , or b) the model parameters themselves @xcite . the former methods are broadly referred to as consistency training methods and have resulted in state-of-the-art performances for several semisupervised nlp tasks. @xcite add adversarial noise to both labelled and unlabelled data and train models to make consistent predictions on the original and the corresponding noisy data-point. many recent methods leverage large pre-trained language models for more advanced data augmentation techniques, like back-translation @xcite , and further improve performance in the low-data regime @xcite . recently, @xcite adapted the mixup algorithm @xcite as another form of data augmentation for textual data and show state-of-the-art performance on text classification and ner tasks. due to the difficulty of data augmentation for relation extraction task (figure 1 ), these methods have not been adapted for semi-supervised relation extraction (ssre) task so far. in this work, we fill that gap and demonstrate the empirical success of consistency training for ssre. self-training methods are the oldest heuristic methods of iteratively expanding the labelled training set by including high-confidence pseudo-labels from the unlabelled data. all of the existing works on ssre fall under this paradigm. these methods famously suffer from the confirmation bias problem where the incorrect predictions of the initially trained model affect the quality of pseudo-labels and eventually cause the label distribution to drift away from the true data distribution, resulting in a semantic drift. @xcite was one of the first works to address this by training two different models and augmenting the labelled set with the 'consensus' set, i.e., the data points which are labelled the same by both models. several works have developed on this idea of improving the pseudo-label quality via various strategies like meta-learning @xcite or reinforcement learning @xcite . these set of methods constitute our baselines. ## proposed approach in our approach we build on the base model architecture described in §3 and introduce additional model design elements that are necessary to obtain an improved performance in semi-supervised relation extraction (ssre) task. we first describe the two data augmentation techniques we perform, and the model architectural changes we introduce that facilitate these augmentations. then, we describe the training procedure we follow to leverage unlabelled data and achieve state-of-the-art performance on three out of four benchmark datasets for ssre. ## experiments we perform experiments on four benchmark datasets for sentence-level re and compare the proposed model, remix, against current state-ofthe-art ssre approaches. we further conduct ablation studies and sensitivity analysis to expose the significance of different design choices of remix. ## conclusion in this paper, we propose a consistency-trainingbased semi-supervised algorithm for relation extraction and empirically show the merit of this class of methods in comparison to the current state-ofthe-art self-training class of methods. in future work, one could bootstrap the self-training methods with consistency training as done in some previous works on vision tasks @xcite . additionally, we show how the entity type information, when available, can result in massive performance boosts in the semi-supervised scenario. this is important because in most practical use cases when entities have already been identified, the entity type information is easy available and could be effectively leveraged in the proposed fashion.