Improving Relation Extraction with Relational Paraphrase Sentences (2020)

Abstract: Supervised models for Relation Extraction (RE) typically require human-annotated training data. Due to the limited size, the human-annotated data is usually incapable of covering diverse relation expressions, which could limit the performance of RE. To increase the coverage of relation expressions, we may enlarge the labeled data by hiring annotators or applying Distant Supervision (DS). However, the human-annotated data is costly and non-scalable while the distantly supervised data contains many noises. In this paper, we propose an alternative approach to improve RE systems via enriching diverse expressions by relational paraphrase sentences. Based on an existing labeled data, we first automatically build a task-specific paraphrase data. Then, we propose a novel model to learn the information of diverse relation expressions. In our model, we try to capture this information on the paraphrases via a joint learning framework. Finally, we conduct experiments on a widely used dataset and the experimental results show that our approach is effective to improve the performance on relation extraction, even compared with a strong baseline.

## introduction relation extraction (re) is an important task in information extraction, which identifies semantic relations between entities in text @xcite @xcite . the task becomes a typical classification problem if an entity pair in a text is given. in recent years, supervised models have achieved great progress on this task with the help of a massive amount of manually annotated high-quality data @xcite @xcite . however, diverse expressions for a same semantic relation are difficult to be fully covered by humanannotated data. for example, sentences (1) "steve jobs co-founded apple computer."; (2) "steve jobs was the co-founder of apple computer."; and (3) "steve jobs started apple computer with wozniak." express the same semantic relation between person "steve jobs" and company "apple computer" in different wording. generally, it is difficult for the supervised model trained on sentence (1) and ( 2 ) to recognize the semantic relation in sentence (3). to solve the above challenge, we can use two possible solutions. the first solution is to hire annotators to label more data. while the human-annotated data is reliable, it is costly and non-scalable, with regard to both time and money. the second one is to adopt the distant supervision (ds) mechanism to automatically build a large-scale labeled data @xcite . however, with the strong assumption that all sentences containing two given entities in a relation triple express the same relation, ds may result in the severe wrong labeling problem. in this paper, we use an alternative solution that uses a paraphrase data which collects sentences conveying the same meaning in different wording. in the literature, there exist many paraphrase datasets, such as simple wikipedia @xcite , twitter url corpus @xcite , and para-nmt @xcite . however, these general paraphrase datasets do triple: <all basotho convention, org:founded_by, tom @xmath0, who set up the [all basotho convention] four months ago ... #2 [tom taba] , who four months ago , formed a [wholly basotho] , ... #3 four months ago , [tom thabane] set up the [all basoto conference] , ... #4 [tom thabane] , who founded the [all basoto congress] four months ago , ... an example from our rep data. #1 is a human-annotated sentence, and #2-4 are paraphrase sentences. blue words with underlines mean different clues for relation "org:founded by" between two entities. not have explicit clues for entities and relations. our preliminary experimental results show that using such paraphrase datasets harms the performance of relation extraction. therefore, it is difficult to learn useful information for relation extraction from the general paraphrase data. in this paper, we propose to automatically build a task-specific paraphrase data which has the explicit clues instead of using general paraphrase datasets for relation extraction. motivated by the recent success of deep neural networks in machine translation @xcite @xcite , we adopt more than one neural machine translation (nmt) systems to generate possible paraphrases via back-translation for each sentence in an existing re data. the back-translation is the procedure in which a system translates a sentence into another language, and then translates back to the original language. however, we can not convey the annotations of entities during back-translation since word alignment information is unavailable. to solve this problem, we design a contextual similarity based method to align entities between the human-annotated sentences and corresponding paraphrase sentences. we combine the human-annotated sentences with these paraphrase sentences as our new training data, named as relational paraphrase (rep) data. then, we propose a joint learning framework to train a relation extractor on the rep data. though back-translation is a convenient way for us to generate paraphrase sentences, there is some noise due to the wrong translation. in order to reduce the effect of the noise in the rep data, we propose a multiinstance learning module to model multiple paraphrase sentences. to build a strong baseline, we choose bert's fine-tuning mechanism to encode sentences and train the relation extractor @xcite . in summary, we make following contributions: to the best of our knowledge, the rep data is the first task-specific paraphrase data for re. experimental results on a widely used re data show that our approach can effectively improve the performance as compared with the strong baseline. ## relational paraphrase data in this section, we describe how to build the relational paraphrase (rep) data, which is a task-specific paraphrase data for re. as shown in figure 1 , we build the rep data by generating paraphrase sentences for human-annotated sentences from an existing re data. in this way, our rep data contains two parts: rep-gold and rep-auto. rep-gold is the original training set of the existing re data and rep-auto is the auto-generated paraphrase data. an example from the rep data is shown in figure 2 . ## our approach in this section, we describe our relation extraction system in detail. to train on both rep-gold and rep-auto, we propose a joint learning framework. during training, each human-annotated sentence in rep-gold is provided with three corresponding paraphrase sentences from rep-auto and we put the four sentences into one input unit. as shown in figure 4 , there are three key components in our system: (1) a sentence encoder, which encodes sentences into distributed representations; (2) a multiinstance learning module, which models three paraphrase sentences from one input unit into a mixed distributed representation; (3) a relation extractor, where the input representations are classified into different relations. ## analysis and discussion here, we study the effectiveness of our system in different situations. thus, we compare the outputs of ours (rep-gold+bag-avg) with baseline on bert base on the test set. the results are shown in set. (1) performance by number of sentences. we first sort the relations by the number of sentences they have in the training set. then, we split the test set into two approximately equal sets according to the sorted relations, a small set and a large set. from the table, we find that the improvement gap is larger on the small set than that on the large set (+1.37 vs +0.47). the reason might be that the relations in the small set have less training sentences that results in lack of diverse expressions and our relational paraphrase sentences can provide more help for them. (2) performance by sentence length. we sort the relations by the average length of sentences they have in the training set. then, we also split the test set into two approximately equal sets according to the sorted relations, a short set and a long set. we find that our system is not sensitive to the sentence length (+0.74 vs +1.08). (3) performance by entity distance. entity distance means the number of words between two entities in a sentence. we sort the relations by the average entity distance of sentences they have in the training set. then, we also split the test set into two approximately equal sets according to the sorted relations, a short set and a long set. we find that our system achieves the more significant improvement (+1.55) on the long set than the short set (+0.39). the reason might be that the nmt systems have more chances to generate different expressions of relations for the sentences with longer entity distance. in addition to the development of models for sentence encoding, studies on relieving the dependence on human-annotated data are also popular. distant supervision is proposed by @xcite to automatically build labeled data for re. although a lot of approaches have been proposed to relieve the wrong labeling problem in distant supervised data @xcite @xcite , there is a gap between models that trained on supervised data and distant supervision data. using some carefully selected human-annotated examples as partial supervision, @xcite combines the reliablity from human-annotated data and the large coverage from distant supervision data. based on the directionality of relations, @xcite proposes a data augmentation method to alleviate the sparse problem. @xcite generates aliases for relation names via phrase-level paraphrases. @xcite proposes to combine the distant supervision data with an existing human-annotated re data. none of the above studies has used the paraphrase sentences. in this paper, we propose to enlarge the coverage of relation expressions by building a relational paraphrase data for an existing re data. ## conclusion in this paper, we show that using the newly built task-specific paraphrase data can have a substantial effect on the performance of relation extraction. in particular, we demonstrate that our proposed system consistently outperforms the strong baseline system using bert. the gains we find come not only from the joint learning framework, but also from the multi-instance learning strategies which model the paraphrase sentences at bag level. our code and data resources are available at https://github.com/jjyunlp/rep-re .