G enerative RE : Incorporating a Novel Copy Mechanism and Pretrained Model for Joint Entity and Relation Extraction (2021)

Abstract: Previous neural Seq2Seq models have shown the effectiveness for jointly extracting relation triplets. However, most of these models suffer from incompletion and disorder problems when they extract multi-token entities from input sentences. To tackle these problems, we propose a generative, multi-task learning framework, named GenerativeRE. We firstly propose a special entity labelling method on both input and output sequences. During the training stage, GenerativeRE fine-tunes the pre-trained generative model and learns the special entity labels simultaneously. During the inference stage, we propose a novel copy mechanism equipped with three mask strategies, to generate the most probable tokens by diminishing the scope of the model decoder. Experimental results show that our model achieves 4.6% and 0.9% F1 score improvements over the current state-of-the-art methods in the NYT24 and NYT29 benchmark datasets respectively.

## introduction the seq2seq based models have attracted much attention in recent years @xcite @xcite to jointly extract entities and relations. these models can transform the joint entity and relation extraction task into a sequence generation task, in which the relation triplets are generated in a sequence manner. early attempts @xcite @xcite @xcite are limited due to the out of vocabulary and overlapping issues @xcite @xcite . to overcome these problems, a copying @xcite or pointing mechanism @xcite has been used. however, two key problems remain: firstly, the model only considers single tokens when copying from input sentences and generates tokens in a token-by-token manner, thus, losing tokens while copying multi-token entities @xcite . this results in incompletion errors(see appendix a.5). secondly, some previous attempts also @xcite suffer from word disorder whilst extracting multi-token relation triplets from a long input sentence as shown in appendix a.5. these issues worsen when more fine-grained tokenization methods are applied, such as workpiece @xcite , which splits the whole sequence into subwords and logically deteriorates these issues @xcite . according to our experimental results on nyt24 and nyt29 , 80.3% examples contain multi-token triplets in nyt24 dataset @xcite , and 80.9% in nyt29 datasets @xcite . thus, although word incompletion and disorder problem are very common in our task, it has not been fully explored. to address the issues aforementioned, we propose a multi-task learning framework, called gen-erativere, which incorporates a novel copy mechanism with a generative pretrained model @xcite for joint entity and relation extraction. specifically, we first design a bio labelling method by calculating the longest common subsequence between input sentence and output triplets sentence, which enables the bio labels to locate the boundaries for the complete multi-token entities. during the training stage, we adopt a generative pretrained model as our backbone model network, and propose a multi-task learning framework to learn the masked tokens and their corresponding bio labels simultaneously. during the inference stage, at each time step, we first predict the bio label of each token. bio labels are aligned with three masking strategies on the probability distribution over the entire vocabulary list, and the model is guided by different mask strategies to extract the correct token in a correct order. experimental results show the effectiveness of our model in alleviating the incompletion issue and disorder issue whilst copying multi-token entities. ## experiments datasets and experiment settings in this work, we use nyt24 @xcite and nyt29 @xcite as our experimental datasets. further information can be found in appendix a.2. we utilize unilm-base-cased foot_0 as our pretrained model. the model structure of unilm follows that of bert-large @xcite . the experimental parameters are aligned with those in baselines, full details are listed in appendix a.3.