Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study (2022)

Abstract: This paper presents an empirical study to build relation extraction systems in low-resource settings. Based upon recent pre-trained language models, we comprehensively investigate three schemes to evaluate the performance in low-resource settings: (i) different types of prompt-based methods with few-shot labeled data; (ii) diverse balancing methods to address the long-tailed distribution issue; (iii) data augmentation technologies and self-training to generate more labeled in-domain data. We create a benchmark with 8 relation extraction (RE) datasets covering different languages, domains and contexts and perform extensive comparisons over the proposed schemes with combinations. Our experiments illustrate: (i) Though prompt-based tuning is beneficial in low-resource RE, there is still much potential for improvement, especially in extracting relations from cross-sentence contexts with multiple relational triples; (ii) Balancing methods are not always helpful for RE with long-tailed distribution; (iii) Data augmentation complements existing baselines and can bring much performance gain, while self-training may not consistently achieve advancement to low-resource RE. Code and datasets are in https://github.com/zjunlp/LREBench.

## introduction relation extraction (re) aims to extract relational facts from the text and plays an essential role in information extraction @xcite . the success of neural networks for re has been witnessed in recent years; however, open issues remain as they still depend on the number of labeled data in practice. for example, @xcite found that the model performance drops dramatically as the number of instances for one relation decreases, e.g., for long-tail. an extreme scenario is few-shot re, where only a few support examples are given. this motivates a low-resource re (lre) task where annotations are scarce @xcite . many efforts are devoted to improving the generalization ability beyond learning directly from limited labeled data. early, @xcite proposes distant supervision for re, which leverages facts in kg as weak supervision to obtain annotated instances. @xcite ; @xcite ; @xcite try to assign pseudo labels to unlabeled data and leverage both pseudo-labeled data and gold-labeled data to improve the generalization capability of models iteratively. some studies apply meta-learning strategies to endow a new model with the ability to optimize rapidly or leverage transfer learning to alleviate the data-hungry issue @xcite @xcite . other studies @xcite focus on the long-tailed class distribution, especially in tail classes that only allow learning with a few instances. with the prosperity of the pre-trained language models (plms), the pre-train -fine-tune paradigm has become standard for natural language processing (nlp), leading to a tremendous increase in lre performance. more recently, a new methodology named prompt learning has made waves in the community by demonstrating astounding few-shot capabilities on lre @xcite . in this work, we benchmark more realistic scenarios on diverse datasets for low-resource re, in which models have to handle both extreme few-shot instances and long-tailed distribution, and can also make use of data augmentation or unlabeled in-domain data without cross-validation @xcite . these settings are appealing as: (i) such models mirror deployment in applied settings; (ii) few-shot settings are realistic with longtailed distribution; (iii) diverse datasets cover different languages (chinese and english), domains @xcite , and contexts (one or more sentences with single or multiple relational triples). specifically, we focus on improving the generalization ability from three directions shown in figure 1 . instead of using limited few-shot data, we create different types of prompts for re and empirically analyze low-resource performance. we further implement many popular balancing methods for long-tailed distribution, which can mitigate performance decay in instance-scarce (tail) classes. we also leverage more generated training instances by data augmentation and self-training in conjunction with the limited labeled data. our contributions include: (i) we present the first systematic study for low-resource re, an important problem in information extraction, by investigating three distinctive schemes with combinations. (ii) we conduct extensive comparisons with in-depth analysis on 8 re datasets and report empirical results with insightful findings. (iii) we release both the data and the source code of these baselines as an open-sourced testbed for future research purposes. to shed light on future research on low-resource re, our empirical analysis suggests that: (i) previous state-of-the-art methods in the low-resource setting still struggle to obtain better performance than that in the fully-supervised setting (cross-sentence lre is extremely challenging), which indicates that there is still much room for low-resource re. ## methods for low-resource re in this paper, we conduct a comprehensive empirical study with three distinctive schemes against difficulty in low-resource re: plms-based promptbased tuning, balancing long-tailed data and leveraging more instances, as shown in figure 2 . ## benchmark design in this paper, we provide a comprehensive empirical study for low-resource re and design the lrebench (low-resource relation extraction benchmark) to evaluate various methods. in the following section, we will detail the datasets chosen for experiments and the reproducibility of all baselines mentioned above. ## related work general and low-resource re relation extraction is essential in information extraction. learning algorithms for re models involve feature-based methods @xcite , semi-supervised @xcite @xcite , graph-based methods @xcite @xcite and applies plms as the backbone @xcite @xcite @xcite . since labeled instances may be limited in practice, low-resource re has appealed to researchers @xcite . prompting methods for re though fine-tuning plms has waved the nlp community, there is still a big gap between pre-training and fine-tuning objectives, hindering the few-shot performance. hence, prompt-based tuning is proposed in gpt-3 @xcite and drawn much attention. a series of researches have illustrate the decent performance of prompt-based tuning @xcite @xcite , especially in few-shot classification tasks @xcite @xcite . typically, ptr @xcite encodes prior knowledge using logic rules in prompt-based tuning with several sub-prompts for text classification. knowprompt @xcite incorporates knowledge among relation labels into prompt tuning for re with synergistic optimization for better performance. methods for long-tailed distribution data many re-balancing methods are proposed to tackle the long-tailed problem @xcite . data distribution re-balancing meth-ods re-sample the dataset into a more balanced data distribution @xcite . various re-weighing losses @xcite @xcite @xcite assign balanced weights to training samples from each class. for re, @xcite introduces causal inference to mitigate the spurious correlation issues for information extraction. data augmentation for nlp an effective method for nlp in low-resource domains is data augmentation. token-level da approaches include replacing tokens with their synonyms @xcite , deleting tokens @xcite , inserting random tokens @xcite or replacing meaningless tokens with random tokens @xcite . ## conclusion we provide an empirical study on low-resource re. specifically, we analyze the prompt-based tuning for few-shot re, balancing methods for long-tailed re datasets, and use data augmentation or unlabeled in-domain data. we systematically evaluate baselines on 8 benchmark datasets in low-resource settings (e.g., 8-shot, 10%) and provide insightful findings. we hope this study can help inspire future research for low-resource re with more robust models and promote transitioning the technology to real-world industrial scenarios.