Learning Relatedness between Types with Prototypes for Relation Extraction (2021)

Abstract: Relation schemas are often pre-defined for each relation dataset. Relation types can be related from different datasets and have overlapping semantics. We hypothesize we can combine these datasets according to the semantic relatedness between the relation types to overcome the problem of lack of training data. It is often easy to discover the connection between relation types based on relation names or annotation guides, but hard to measure the exact similarity and take advantage of the connection between the relation types from different datasets. We propose to use prototypical examples to represent each relation type and use these examples to augment related types from a different dataset. We obtain further improvement (ACE05) with this type augmentation over a strong baseline which uses multi-task learning between datasets to obtain better feature representation for relations. We make our implementation publicly available: https://github.com/fufrank5/relatedness

## introduction relation extraction identifies specific semantic relationships between two entities within a single sentence. for example, there is a physical.located relationship between george bush and france in the sentence: george bush traveled to france on thursday for a summit. relation extraction is a crucial task for many applications such as knowledge base population. relation schemas are mostly pre-defined in existing datasets. the definition of the relation type depends on the annotation guide. there is no clear intrinsic ontology for relation types. in practice, relation types can be created based on interests. this leaves datasets with similar, related or overlapping schemas. for example, the annotation guides for automatic content extraction (ace) 03-05 changed from year to year. the later created entities, relations and events (ere) dataset was similar in the schema, but differs in details. because of the difficulty of annotating relations, these datasets are all small individually and hard to be utilized together. it is not an easy task to learn relatedness between relation schemas across different datasets since there is no instance-level labels available for the relatedness. however, we can observe the connections between the relation types from different datasets based on relation names or annotation guides. we propose to simplify the relatedness as binary (related or not) and to use manual review of relation names to decide the relatedness labels. this would give the prior knowledge that one relation type in one dataset may have closer relationships to some types than the others in another dataset. then we design a model to recognize this similarity. we propose to use prototypical examples to represent each relation type. we rank these representations higher for related types, and lower for unrelated types using a pairwise loss function. our base model is a multi-task learning model which focuses on learning a strong encoder using multiple datasets regardless of the relation schemas. we take the step further to explore utilizing the relatedness between the relation types. experiments on ace05 and ere show that it can further boost the performance, especially in the low-resource settings. multi-task learning: training multiple relation datasets at the same time could improve the robustness of the model and reduce annotation cost for relation extraction. @xcite proposed to use a shared encoder to learn more general feature representation. we use a similar multitask learning base model and incorporate the similarity between the relation schemas to further improve the performance. ## relation model with multi-task learning the majority of neural relation models @xcite @xcite encode a sentence using a deep architecture to a vector representation followed by a softmax classifier, while the others @xcite use a function to compute the score between label embedding and sentence representation. inspired by @xcite where the shared encoder helps in the case of the multi-task learning, we choose the latter so that all relation types (including from different datasets) will share the whole model pa-rameters except the label embeddings. suppose we obtain the sentence representation φ(x) with a neural architecture. we define the label embedding as w @xmath0, a d-dimension vector for each type. we compute the l 1 distance between them and learn a scoring function to estimate the scores s θ (x) for every type: we do not use the dot product (dos santos et al., 2015; @xcite as the scoring function because the l 1 distance works slightly better in the multi-task learning experiments. the probability of every class is computed as the softmax output of the scores. similar to @xcite , we jointly train two relation tasks at the same time with cross-entropy losses. ## conclusion we use prototypes of relation types to learn the relatedness between them in a multi-task learning framework. with prior knowledge of relatedness between relation types, the model obtains further improvement in addition to sharing the encoder of the sentence. the prior knowledge is ob-tained through manual review of relation names, which costs significantly less than annotating on the instance-level in text. in this paper, we simplify the relatedness as binary. it would be interesting to further explore the relationships between relation types as a more dynamic metric.