S 2yn RE : Two-stage Self-training with Synthetic data for Low-resource Relation Extraction (2023)

Abstract: Current relation extraction methods suffer from the inadequacy of large-scale annotated data. While distant supervision alleviates the problem of data quantities, there still exists domain disparity in data qualities due to its reliance on domain-restrained knowledge bases. In this work, we propose S2ynRE, a framework of two-stage Self-training with Synthetic data for Relation Extraction.We first leverage the capability of large language models to adapt to the target domain and automatically synthesize large quantities of coherent, realistic training data. We then propose an accompanied two-stage self-training algorithm that iteratively and alternately learns from synthetic and golden data together. We conduct comprehensive experiments and detailed ablations on popular relation extraction datasets to demonstrate the effectiveness of the proposed framework.

## introduction relation extraction systems aim at discovering relational knowledge between entities by reading from unrestricted texts @xcite . although neural methods, especially pre-trained language models, have greatly advanced the state-of-the-art relation extraction capability @xcite , they still require large quantities of training data @xcite . however, high-quality annotations are usually very expensive to obtain, making low-resource relation extraction a very practical challenge in many real-world scenarios. distant supervision @xcite , which automatically annotates relational statements by aligning entities with an existing knowledge bases @xcite , has been widely explored as an effective way to construct large scale relational * joint work of ustc and baidu inc. † corresponding author: zhendong mao dataset. many recent works exploit such data in a pretraining stage to learn relational representations @xcite @xcite . although this line of methods have seen certain improvements, they still inevitably raise the concern that the distantly annotated data can vary considerably from downstream tasks both in target schema and in context distributions, thus may not be able to offer optimal transferability. for instance, due to the reliance on existing knowledge bases, current works mostly resort to wikidata as the source of relational triples and wikipedia @xcite as the corpus for distant supervision. this circumscribes distant data to only factual knowledge between world entities, while downstream tasks may be of other special interests involving various domains, ranging from semantic relation between nominals @xcite to chemicalprotein interactions @xcite . meanwhile, recent advances in large-scale pretrained language models (llm) @xcite @xcite have demonstrated their great potential in generating realistic texts of various domains @xcite . accordingly, several very recent works have explored the possibility to exploit llm as an alternative training data pool @xcite . however, these attempts are confined to nli task, while still not effectively explored in the area of relation extraction. in this paper, we study the construction of synthetic data for relation extraction tasks to simultaneously address both training data scarcity in low resource scenarios and domain disparity in distant supervision. we employ llm to estimate and adapt to the target domain distribution with only a few training instances, and synthesize a large amount of ones accordingly. the procedure is overall very simple but also carefully designed with two critical choices: 1) we linearize relational state-ments into natural language sequences where entity pairs are indicated by special marker tokens; 2) we resort to unconditional generation instead of label-conditioned ones, which relaxes the requirements for strict label-semantic correspondence but increases sample availability and diversity. we experiment with both gpt2 and the recent very large llms like gpt-3.5. for standard size generative lms like gpt2-large, we first finetune it to adapt to the target domain, while for the capable gpt-3.5 model, we directly apply in-context learning @xcite . we empirically found finetuned gpt2-large produces synthetic data of equivalent quality to prompted gpt-3.5. in general, it is observed that with only a few accessible samples, we are able to successfully synthesize a large amount of domain-customized training data with satisfactory quality. to effectively learn from such synthetic data, we novelly advocate a two-stage self-training algorithm. the approach in general follows the selftraining framework @xcite , which is widely employed to exploit unlabeled data. typically, such methods iteratively annotate and learn pseudo labels for unlabeled data to bootstrap the model's performance. distinctively, we make a two-stage adaptation where in each of the iterations, the model is firstly trained on synthetic instances, then on golden ones. such sequential training procedure favors golden data with more importance since they are introduced in the latter stage of the training curriculum. we refer to our method as s 2 ynre, a framework of two-stage self-training with synthetic data for relation extraction. the contributions of this paper is three-fold: relation extraction relation extraction is one of the fundamental tasks in natural language processing @xcite , where lots of research efforts have been made to advance the state-of-theart methods @xcite @xcite , as well as the low-resource scenario @xcite @xcite . one of the most prominent methods is distant supervision @xcite , which automatically constructs annotated relational data by aligning corpus with existing knowledge base. many recent works investigate how to learn effectively with such distant data @xcite @xcite . generally, they propose various pretext tasks that pre-train a model to learn relational representation. we will further explain some of these works for comparison in section 5.3. learning from synthetic data built upon massive corpora, pre-trained language models are promising at producing texts of eligible quality, resulting in a surge of research interests in its usage for data augmentation @xcite . one straightforward way is to introduce mask corruptions in the way language models are pretrained, then collect predictions as augmented data @xcite . later works further developed such technique into conditional augmentation @xcite . nevertheless, these methods are mostly editing existing instances, which limits the diversity and scale of augmented data. with increasingly powerful llms, recent works turn to direct synthesis of new instances @xcite @xcite . different from this work, most of them focus on zero-shot language understanding where no labeled data is available @xcite @xcite . they investigate ways to generate label-conditioned data by prompting llms, but these methods can hardly be applied to low-resource or full data scenarios while still preserving effectiveness. with the existence of labeled data, synthetic data needs to be of higher quality to bring further utility. several works thus propose to finetune the generator @xcite @xcite . there are also explorations for learning from synthetic and golden data together, including threshold-based confidence filtering (anaby-tavor et al., 2020), classical semi-supervised learning @xcite or restricting the usage of synthetic data within a supplemental intermediate task @xcite . ding et al. (2020) similarly formulates ner task data as sequential language. specifically for relational data synthesis, papanikolaou and pierleoni (2020) explore the biomedical domain and chia et al. (2022) focus on zero-shot setting of triplet extraction. by contrast, syn 2 re distinguishes not only in applied scenario and synthesis strategy, but also in the two-stage learning framework, which is specially designed for improved synthetic data adaptation. 3 preliminary this section formulates the task of relation extraction and the baseline models used throughout all experiments. task formulation a typical relation extraction task is defined by a corpus of relational statements and a set of relations, i.e., schema s. assume the training dataset d @xmath0, s i , o i )} n @xmath1, where x i is a sequence of words {w i l } l @xmath2, y @xmath3, s @xmath4: w s end ] and o @xmath5: w o end ] are subject and object entities within the context. the target is to learn a function f θ (x i , s i , o i ) that predicts the correct relation label y i . baseline model as s 2 ynre is a data-centric framework, we keep the model architecture simple but competitive, which is the vanilla finetuning of pre-trained language models. instead of autoregressive lms, we use auto-encoding networks like bert as they usually perform better on language understanding downstream tasks. @xcite comprehensive study of building relation extractors, we inject spe-cial marker tokens to the input word sequence: after the encoding process of transformer, the representation h in corresponding positions will be concatenated for classification: ## discussion distant supervision is the most prevalent solution for low-resource relation extraction, and also the main investigated and compared baseline in this paper. both distant data and the proposed synthetic data can essentially be recognized as ways of data augmentation to produce sufficient number of additional data. the critical difference which determines the effectiveness lies in their consistency with golden training data, i.e., domain affinity. and in this paper, the superiority of synthetic data is both experimentally proved ( ## conclusion in this paper, we present s 2 ynre, a framework of two-stage self-training with synthetic data for relation extraction. we show that synthetic data generated using llms can resolve data scarcity in low-resource scenarios and mitigate domain disparity compared to distant supervision. to enable effective learning from such synthetic data, we then propose a novel two-stage self-training algorithm that continually bootstraps model performance by iteratively and alternately training the synthetic and golden data together. the proposed framework brings substantial improvements and achieves new state-of-the-art for low-resource relation extraction. in the future, we expect new possibilities brought by llms and will further explore accompanied techniques to exploit their potential.