How Knowledge Graph and Attention Help? A Qualitative Analysis into Bag-level Relation Extraction (2021)

Abstract: Knowledge Graph (KG) and attention mechanism have been demonstrated effective in introducing and selecting useful information for weakly supervised methods. However, only qualitative analysis and ablation study are provided as evidence. In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE). We find that (1) higher attention accuracy may lead to worse performance as it may harm the model’s ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns, which is closely related to real-world datasets; (3) KG-enhanced attention indeed improves RE performance, while not through enhanced attention but by incorporating entity prior; and (4) attention mechanism may exacerbate the issue of insufficient training data. Based on these findings, we show that a straightforward variant of RE model can achieve significant improvements (6% AUC on average) on two real-world datasets as compared with three state-of-the-art baselines. Our codes and datasets are available at https://github.com/zig-kwin-hu/how-KG-ATT-help .

## introduction relation extraction (re) is crucial for knowledge graph (kg) construction and population. most recent efforts rely on neural networks to learn efficient features from large-scale annotated data, thus correctly extract the relationship between entities. to save the manual annotation cost and alleviate the issue of data scarcity, distant supervision relation extraction (dsre) @xcite is proposed and becomes increasingly popular as it can automatically generate large-scale labeled data. dsre is based on a simple yet effective principle: if there is a relation between two entities in kg, then all sentences containing mentions of both entities are assumed to express this relation and will form a sentence bag as its annotations. although effective, distant supervision may introduce noise to a sentence bag when the assumption fails -some sentences are not describing the target relation @xcite (a.k.a. noisy annotation). to alleviate the negative impacts of noise, recent studies @xcite @xcite ) leveraged attention to select informative instances from a bag. furthermore, researchers introduced kg embeddings to enhance the attention mechanism @xcite . the basic idea is to utilize entity embeddings as the query to compute attention scores, so that the sentences with high attention weights are more likely to be valid annotations @xcite . previous studies have shown performance gain on dsre with attention module and kg embeddings, however, it's still not clear how these mechanisms work, and, are there any limitations to apply them? in this paper, we aim to provide a thorough and quantitative analysis about the impact of both attention mechanism and kg on dsre. by analyzing several public benchmarks including nyt-fb60k @xcite , we observe lots of disturbing bags -all of the bag's sentences are valid or noisy annotations, which shall lead to the failure of attention. as shown in figure -1, all of annotations in the first disturbing bag are valid, while the learned attentions assign the second annotation with a very low weight, which suggests an inefficient utilization of annotations and exacerbates the data sparsity issue. or, in the second bag, all sentences are noisy, can attention and kg still improve the performance? if so, how do they work and to what extent can they tolerate these disturbing bags? answering these questions is crucial since this type of noise is common in practice. the unveiling of their working mechanism shall shed light on future research direction, not limited to dsre. to achieve this, we propose a paradigm based on newly curated dsre benchmark, bagrel-wiki73k extracted from fewrel @xcite and wikidata 1 , for quantitative analysis of attention and kg. with extensive experiments, we conclude the following innovative and inspiring findings: based on the above observations, we propose a new straightforward yet effective model based on pre-trained bert @xcite for re with concatenated kg embedding, namely bre+ce. instead of in-bag attention, it breaks the bag and ensembles the results of all sentences belonging to the bag. for each sentence, we directly incorporate entity embeddings into bert, rather than to enhance attentions, to improve the robustness of extracting both context and mention features. bre+ce significantly outperforms existing state-of-the-arts on two publicly available datasets, nyt-fb60k @xcite and gids-fb8k @xcite , by 6% auc on average. we summarize our contributions as follows: ## related work to address the issue of insufficient annotations, @xcite proposed distant supervision to generate training data automatically, which also introduces much noise. from then, dsre becomes a standard solution that relies on multi-instance learning from a bag of sentences instead of a single sentence @xcite . attention mechanism @xcite accelerates this trend via strong ability in handling noisy instances within a bag @xcite . aside from intra-bag attention, @xcite also designed inter-bag attention simultaneously handling bags with the same relation. to deal with only-one-instance bags, @xcite utilized a new selective gate (seg) framework to independently assign weights to each sentence. external kg is also incorporated to enhance the attention module @xcite . however, due to the lack of sentencelevel ground truth, it is difficult to quantitatively evaluate the performance of the attention module. previous researchers tend to provide examples as case study. foot_0 therefore, we aim to fill in this research gap by constructing a dataset and providing a framework for thorough analysis. ## preliminary knowledge graph (kg) is a directed graph @xmath0, r, t }, where e denotes the set of entities, r denotes the set of relation types in g, and @xmath1, r, t)} ⊆ e × r × e denotes the set of triples. kg embedding models, e.g., rotate @xcite , can preserve the structure information in the learned vectors e h , e t and e r . we adopt transe @xcite in experiments. bag-level relation extraction (re) takes a bag of sentences @xmath2, s 2 , . . . , s m } as input. each sentence s i in the bag contains the same entity pair (h, t), where h, @xmath3. the goal is to predict a relation @xmath4, t). attention-based bag-level re uses attention to assign a weight to each sentence within a bag. given a bag b from the dataset d, an encoder is first used to encode all sentences from b into vectors {s 1 , s 2 , . . . , s m } separately. then, an attention module computes an attention weight α i for each sentence and outputs the weighted sum of {s i } as s to denote b: we mark this way of computing ω i as ka. w s and b s are learnable parameters. given a bag representation s, the classification layer further predicts a confidence of each relation: from equation 10, we can see that the model's output on the whole bag depends on three aspects: ## benchmark to quantitatively evaluate the effect of attention and kg on bag-level re, we first define two metrics to measure the noise pattern (section 4.1). then, we construct a kg and a bag-level re dataset (section 4.2). finally, we introduce a general evaluation framework to assess attention, kg and the entire re model (section 4.3). ## method to evaluate the effects of attention and kg, we design two straightforward bag-level re models without the attention module, bre and bre+ce. by comparing their performance with bre+att (bre with attention module) and bre+ka (bre with kg-enhanced attention module), we can have a better understanding of the roles of att and knowledge-enhanced att. bre uses bert @xcite as the encoder. specifically, we follow the way described in @xcite : entity mentions in sentences are highlighted with special markers before and after mentions. then the concatenation of head and tail entity representations are used as the representation s . since bre does not have attention mechanism, it breaks the bags and compute loss on each sentence: bre can be viewed as a special case of bre+att. its attention module assigns all sentences in all bags with the same attention weight 1. during inference, given a bag, bre uses the mean of each sentence's prediction as the whole bag's prediction: bre+ce concatenates an additional feature vector r ht with bert output, where r ht is defined based on entity embeddings of h and t. the concatenated vector is used as the representation of the sentence and fed into the classification layer. ## experiment we apply our proposed framework on bagrel-wiki73k and two real-world datasets to explore the following questions: ## conclusion in conclusion, we construct a set of datasets and propose a framework to quantitatively evaluate how attention module and kg work in the bag-level re. based on the findings, we demonstrate the effectiveness of a straightforward solution on this task. experiment results well support our claims that the accuracy of attention mechanism depends on the noise pattern of the training set. in addition, although effectively selecting valid sentences, attention mechanism could harm model's robustness to noisy sentences and aggravate the data sparsity issue. as for kg's effects on attention, we observe that it promotes model's performance by enhancing its robustness with external entity information, instead of improving attention accuracy. in the future, we are interested in developing a more general evaluation framework for other tasks, such as question answering, and improving the attention mechanism to be robust to noise and insufficient data, and an effective approach to incorporate the kg knowledge to guide the model training.