H yper N etwork-based Decoupling to Improve Model Generalization for Few-Shot Relation Extraction (2023)

Abstract: Few-shot relation extraction (FSRE) aims to train a model that can deal with new relations using only a few labeled examples. Most existing studies employ Prototypical Networks for FSRE, which usually overfits the relation classes in the training set and cannot generalize well to unseen relations. By investigating the class separation of an FSRE model, we find that model upper layers are prone to learn relation-specific knowledge. Therefore, in this paper, we propose a HyperNetwork-based Decoupling approach to improve the generalization of FSRE models. Specifically, our model consists of an encoder, a network generator (for producing relation classifiers) and the produced-then-finetuned classifiers for every N-way-K-shot episode. Meanwhile, we design a two-step training framework along with a class-agnostic aligner, in which the generated classifiers focus on acquiring relation-specific knowledge and the encoder is encouraged to learn more general relation knowledge. In this way, the roles of upper and lower layers in an FSRE model are explicitly decoupled, thus enhancing its generalizing capability during testing. Experiments on two public datasets demonstrate the effectiveness of our method.

## introduction relation extraction is a fundamental task in information extraction that aims to identify the semantic relations between two entities in sentences @xcite . typically, conventional approaches are highly dependent on a large amount of labeled data and cannot deal with unseen relations well. therefore, recent studies @xcite turn to few-shot relation extraction (fsre) that only requires a handful of labeled instances and enhance the generalizing capability of fsre model to new relations. current fsre studies are usually conducted in an n -way-k-shot setting. in this setting, the model is trained on a series of episodes, each of which has n relation classes. in each episode, every relation class includes k support instances (as support set) and q query instances (as query set). under this setting, most existing studies @xcite @xcite employs prototypical network @xcite for fsre. the prototypical network is designed to learn a suitable prototypical vector for each relation using the support instances from the support set. then, these vectors are used to predict the relation of the query instances. moreover, to further improve the performance of the model, some recent studies @xcite introduce relation description and adopt contrastive learning in prototypical network. although these methods have achieved improvements, their models still tends to overfit relation classes appearing in the training set, and exhibit an unsatisfactory generalizing capability to unseen relations. to further explore what restricts the generalizing capabilities of prototypical network models, inspired by @xcite , we conduct a preliminary study to investigate the internal representations within different model layers. specifically, we measure the layer-wise class separation between representations of instances belong into different relation classes during training. we observe that the representations present a higher class separation in the upper layers of the model than those in the lower layers. this intuitively shows that the upper layers are likely to learn relationspecific knowledge and overfit the relation classes in the training set, while the lower layers basically acquire more general relation knowledge. we speculate that this over-adaptation of upper layers limits the generalizing capability of the model. based on the above finding, in this paper, we propose a hypernetwork-based decoupling approach for fsre. as illustrated in figure 1 , our model consists of three components: an encoder at the bottom, a network generator in the middle (for generating the initialized relation classifiers) and the generated relation classifiers at the top. in each episode, the network generator takes in the encoder output and generates an initialized relation classifier for current episode. subsequently, the generated classifiers are fine-tuned, so that the model can quickly learn relation-specific knowledge and adapt to new relations. meanwhile, the encoder and generator are encouraged to learn more general relation knowledge from the training set. in this way, we explicitly decouple the roles of the lower and upper components in our fsre model. to this end, our model update procedure consists of two steps: the fine-tuning of the classifier and the update of the encoder together with the generator. for the classifier fine-tuning, we first use the generator to produce an initialized classifier for every episode to discriminate the relation classes within them. then, we use corresponding support set to optimize its generated classifier, which endows the classifier with relation-specific knowledge. consequently, our model can effectively adapt to new relations through such produce-then-finetune process. moreover, the update of the encoder and generator resorts to learning more general knowledge not specific to a certain episode, which is crucial for an fsre model not to overfit. to better learn such knowledge, we train the model across different episodes so that it is not biased towards a narrow set of relations. particularly, we treat a collection of m sampled episodes as an updating interval for the encoder and generator. in each updating interval, the encoder and generator are jointly optimized through maximizing the overall performance of all trained classifiers on the query set of these m episodes. during testing, when there is an episode containing unprecedented relation types not existing in the training set, we use the generator to produce a fresh classifier that is then fine-tuned by samples from the support set. finally, only the encoder and the fine-tuned classifier are used to predict the relation of each query instance. by doing so, the model can quickly adapt to new episode. however, as each updating interval contains only a handful of episodes, so that our model is still prone to bias towards relation classes in these episodes. to address this issue, we additionally design a class-agnostic aligner, which undergoes all episodes in the training set throughout the training process. thus, with the help of the aligner, our encoder is able to learn more global general relation knowledge, further alleviating the overfitting to specific relations. experimental results on two public benchmarks show that our model consistently outperforms all competitive baselines. extensive ablation and case studies demonstrate the effectiveness of various components in our model. ## preliminary study to better understand the limited generalizing capabilities of current fsre models, inspired by @xcite , given a bert-based prototypical network model, we first investigate layerwise class separation between representations of instances belonging to different relations. specifically, the class separation indicates the dispersion of representations belonging to the same class relative to the overall dispersion of all embeddings. in each n -way-k-shot training episode, we measure the class separation in every layer of the model following the metrics in @xcite . particularly, we denote the average withinclass cosine distance and the overall average cosine distance in the l-th layer as d l in and d l overall , respectively. they are calculated using the following equations: as explored in @xcite , higher class separation means the model tends to overfit the classes in the training set and is less transferable to unseen classes. hence, we depict the layer-wise change of class separation with respect to the number of training steps in figure 2 . we can observe that sep l in upper layers (e.g., layer-12 and layer-10) quickly increase at early steps and consistently stay much higher than those in lower layers (e.g., layer-1 and layer-2). therefore, figure 2 shows that the upper layers of the prototypical network model are likely to learn relation-specific knowledge and overfit the relation classes during training. ## methodology in this section, we introduce our hypernetworkbased decoupling approach for fsre. specifically, we first provide the problem formulation of fsre and give a detailed description of our model architecture. then, we elaborate the two-step training strategy of our model. moreover, we additionally design a class-agnostic aligner to further enhance the generalizing capability of our model. finally, we describe how our model is used for test in an n -way-k-shot setting. ## related work relation extraction(re) is a critical and fundamental task in natural language processing (nlp), which aims to identify the semantic relations between two entities within a given text @xcite @xcite @xcite . however, conventional approaches are highly dependent on a large amount of labeled data and cannot deal with unseen relation classes well. therefore, recent studies @xcite turn to few-shot relation extraction (fsre) that aims to train a model to classify instances into novel relations with only a handful of training examples. most existing studies @xcite @xcite employ prototype networks for fsre, which aims to learn a suitable prototypical vector for each relation using a handful of annotated instances. @xcite employs an attention mechanism to enhance the robustness of the prototype network to noisy data. @xcite proposes a bayesian meta-learning method with an external global relation graph to model the posterior distribution of relational prototypes. @xcite focuses on enhancing the performance of prototype network on complex relations through an adaptive focal loss and a hybrid network. moreover, some studies @xcite @xcite use supplementary information about entities and relations, such as relation descriptions, to enhance the prototype vectors of relations. despite impressive results achieved, these methods still tends to overfit relation classes appearing in the training set, which limits their generalizing capability to new relations. in this paper, inspired by @xcite , we propose a hypernetwork-based decoupling method along with a two-step training strategy to prevent overfitting of the fsre model to the relations within the training set. on the other hand, some studies @xcite @xcite focus on further training pre-trained language models (plms) using noisy re datasets. soares et al. (2019) 2020 ) proposes an entity-masked contrastive pre-training framework for fsre. @xcite introduces three structure pre-training tasks to pre-train the large language model (glm with 10b parameters), allowing it to better comprehend structured information in text. unlike several other studies, zhang and lu (2022) introduces a more rigorous few-shot evaluation scenario by filtering relations contained in fewre 1.0 from the pre-trained corpus. meanwhile, they propose a label prompt dropout method to prevent the model from overfitting to the relation description. these methods are compatible with our model, which can provide our model with a better pre-trained encoder. ## conclusion in this paper, we propose a hypernetwork-based decoupling approach to improve the generalizing capability of fsre models. specifically, our model consists of an encoder, a network generator (for generating relation classifiers) and the generated classifiers. our generator aims to generate a properly initialized relation classifier for each episode, allowing our model can quickly adapt to new relations. meanwhile, we design a two-step training strategy along with a class-agnostic aligner, in which the generated classifiers focus on acquiring relation-specific knowledge while the encoder is encouraged to learn more general relation knowledge. in this way, the roles of upper and lower layers in an fsre model are explicitly decoupled, thus enhancing its generalizing capability during testing. experiments on two public fsre datasets and extensive ablation studies show that our model consistently outperforms all competitive baselines.