Minimally-Supervised Relation Induction from Pre-trained Language Model (2022)

Abstract: Relation Induction is a very practical task in Natural Language Processing (NLP) area. In practical application scenarios, people want to induce more entity pairs having the same relation from only a few seed entity pairs. Thus, instead of the laborious supervised setting, in this paper, we focus on the minimally-supervised setting where only a couple of seed entity pairs per relation are provided. Although the conventional relation induction methods have made some success, their performance depends heavily on the quality of word embeddings. The great success of Pre-trained Language Models, such as BERT, changes the NLP area a lot, and they are proven to be able to better capture relation knowledge. In this paper, we propose a novel method to induce relation with BERT under the minimally-supervised setting. Specifically, we firstly extract proper templates from the corpus by using the mask-prediction task in BERT to build pseudo-sentences as the context of entity pairs. Then we use BERT attention weights to better represent the pseudo-sentences. In addition, We also use the IntegratedGradient of entity pairs to iteratively select better templates further. Finally, with the high-quality pseudo-sentences, we can train a better classifier for relation induction. Experiments onGoogle Analogy Test Sets (GATS), Bigger Analogy TestSet (BATS) and DiffVec demonstrate that our proposed method achieves state-of-the-art performance.

## introduction relation induction is a task to judge whether two entities have a certain relation based on some given entity pairs of that relation, which was first proposed in @xcite . for instance, given {(germany, berlin), (f rance, p airs), (italy, rome)}, relation induction is to predict whether new entity pairs such as (china, beijing) have the same relation as the * corresponding author given entity pairs. in practical scenarios, only a few seed entity pairs are available. it is challenging to judge the relation of the target entity pairs in this minimal supervision setting. word embedding, such as skip-gram @xcite and glove @xcite , are widely used in many natural language processing (nlp) tasks, and it was reported that word embeddings can capture the relational knowledge @xcite . one intuitional method for relation induction task is using word embeddings to represent relations and induce relations based on vector translation or similarity @xcite @xcite @xcite . however, the performance of these methods heavily depends on the pre-trained word embedding and these methods are rather noisy. according to the assumption that if two entities have a relationship in a known knowledge base, then all sentences that mention these two entities will express that relationship in some way @xcite , many distant-supervised methods of relation extraction, such as pcnn @xcite and pcnn-bagatt @xcite are proposed. inspired by these methods, distant supervision might be another way to induce relation. to induce relation in the distant supervised way, we need a method to select proper sentences from corpus and extract relational knowledge from sentences. luckily, many pre-trained language models (plms), such as bert @xcite , @xcite and xlnet @xcite , have been recently proposed and boost a great performance for many nlp tasks, such as question answering @xcite ), text summarization @xcite and information extraction @xcite . in order to better understand the plms, several works @xcite @xcite have proven that plms can capture syntactic and semantic knowledge. @xcite have explored the possibility of inducing relation from bert in a distant supervised way and got a good result. to take the advantage that bert can capture context knowledge, they select templates from corpus and fill entities in them to let bert predict the relation. existing methods are developed under the assumption of sufficient seed entity pairs. however, in practical scenarios, only a few entity pairs are available for a particular relation. these methods have difficulty in coping with the minimal supervision setting. the main reasons are: (1) due to the lack of labeled entity pairs, the model tends to over-focus on the surface cues of the entity pairs and ignores the contextual semantics. by simply memorizing the seed entity pairs, it is difficult to generalize the model to other entity pairs. (2) the quality of templates is very important for relation induction.when the seed entity pairs of a certain relation are sparse, the number of candidate templates for this relation mined from the corpus will be reduced. therefore, two major challenges should be addressed for the relation induction in the minimallysupervised setting. (1) how to obtain a good generalized relation induction model? (2) how to obtain high-quality templates? so we propose a novel approach called ist for minimally-supervised relation induction with iteratively-selected templates from plm. specifically, for the first challenge, we use surface-agnostic features based on attention maps of bert. many works @xcite @xcite have revealed that the attention heads in bert can capture much knowledge and some attention heads are related to certain relations, and some works use attention weights to predict relations @xcite . for the second challenge, we use integrated gradient (ig) @xcite to score the templates and iteratively select better templates. intuitively, if a sentence can well express the relational knowledge between two entities, then the importance of these two entities must be high in the sentence. on the contrary, if a pair of entities do not play an important role in a sentence, this sentence certainly does not express the relationship between them. so ig might be used to select high-quality sentences to express relations. we summarize our key contributions as follows: ## our approach in this section, we first formulate the minimallysupervised relation induction task and give an overview of our approach. we then describe the details of each module in our approach. ## baselines we compare our approach with three kinds of baselines. the first kind is using the combination of pre-trained word embeddings to present relations. specifically, following vu and shwartz (2018), we use s ⊕ t ⊕ (s ⊙ t) to represent the relation between (s, t) and use a mlp classifier to make predictions. here, the pre-trained word embeddings we used are glove @xcite 3 and skip-gram @xcite 4 . these two baselines are referred to as mlp sg and mlp gl respectively.we also use the trans approach @xcite for relation induction by building subspaces for entities using word embeddings and modeling the relations with relative positions between subspaces. ri-bert induces relational knowledge from bert, and our approach would degenerate to it when not using attention maps as the surfaceagnostic features and not using score ig to refine the templates. we implement the method by ourselves since there is no open source. autoprompt tries to elicit knowledge from plm using automatically-constructed prompts. here, we generate templates with autoprompt for each relation. since there is only one template can be generated for each relation, we use a thresholdbased method to determine whether a new entity pair (x, y) has a relation. when p(x, y) > δ, the prediction would be positive. here, δ = 0.8 is the best threshold in our experiments. bert predict is a simple baseline proposed by ourselves. after k templates are selected with score bert , we directly use bert maskprediction task to judge relation. specifically, for an entity pair (s, t) and a template τ , if bert can predict τ (s, _) or τ (_, t), the score of (s, t) will be increased by 1. the max score is 2k, so if the score of (s, t) ≥ ϵ • 2k, (s, t) is predicted to have the relation r. ϵ is a threshold and set to 0.7 for its best performance. ## conclusion in this paper, we propose a novel minimallysupervised relation induction approach. our proposed approach can iteratively select proper templates using score ig and socre bert , and obtain a good generalized ability with surface-agnoistic features based on attention maps of bert. experiments illustrate that our approach achieves state-ofthe-art performance on three standard benchmarks.