B io C opy: A Plug-And-Play Span Copy Mechanism in S eq2 S eq Models (2021)

Abstract: Copy mechanisms explicitly obtain unchanged tokens from the source (input) sequence to generate the target (output) sequence under the neural seq2seq framework. However, most of the existing copy mechanisms only consider single word copying from the source sentences, which results in losing essential tokens while copying long spans. In this work, we propose a plug-and-play architecture, namely BioCopy, to alleviate the problem aforementioned. Specifically, in the training stage, we construct a BIO tag for each token and train the original model with BIO tags jointly. In the inference stage, the model will firstly predict the BIO tag at each time step, then conduct different mask strategies based on the predicted BIO label to diminish the scope of the probability distributions over the vocabulary list. Experimental results on two separate generative tasks show that they all outperform the baseline models by adding our BioCopy to the original model structure.

## introduction recent neural seq2seq systems have been successful in various nlp tasks which utilize an encoder to convert a source sentence into a fixed vector, and a decoder to generate a target sentence by using the semantic information amongst the fixed vectors. however, seq2seq suffers from the outof-vocabulary and rare word problem @xcite that words in source sentence are not able to be obtained to generate the target sentence. due to this problem, @xcite , propose the so-called 'copy mechanism' where it locates certain words in the input sentence and put these words into the target sequence. in their work, every output word can be generated either by predicting from the vocabulary or copying from the source sequence. unfortunately, most of them copy the words from the source sentence to the target sentence in a word-by-word manner. however, in many cases, the copied words are generated consecutively from a span in the source sequence. in this paper, we propose a novel and portable copy mechanism to solve the problem of low accuracy when copying spans in the seq2seq framework. to implement this new copy mechanism, we propose a bio-tagged strategy that annotates the target sequences with bio tags by matching the longest common subsequence (lcs) with the source sequence. therefore, the bio tags are able to exactly locate the start and end position of every single span in target sequences. in the inference stage, we design a span extractor to determine copying spans from the source sequence. specifically, the bio tag is firstly predicted at each time step to indicate the position of the copied span, then we decide a copy algorithm to guide the span extractor by generating the eligible n-gram set. since our method does not change the structure of the model itself, this biocopy can be seen as a plug-and-play component, which is simple and effective enough to transfer and apply to any generative seq2seq framework. the experimental results in generative relation extraction and abstractive summarization tasks indicate the effectiveness of our proposed copy mechanism. ## approach the conventional seq2seq framework generates one token at each time step. therefore, given the predicted token sequence, the mode decoder predicts the current token by computing the probability distribution over the entire vocabulary list: in our case, we add an extra sequence prediction task to the model decoder. specifically, the former decoder just models the distribution on each token, but the current decoder predicts an additional label distribution: y t , z @xmath0, z t |@xmath1, x) = p(y t |@xmath2, x)p(z t |@xmath3, x) bio-tag building as we mentioned above, we utilize the supervised method to train the bio tag z. in order to acquire the labels, we compute the longest common subsequence @xcite between the source sentence and target sentence. the token will be considered as copied from source sentence as long as it appears in the longest common subsequence, and different tags are assigned according to the specific meaning of bio. in summary, in the training phase, besides the original token sequence prediction task, we also introduce one more sequence prediction task, of which the tags are all given. it is easy to implement and does not add any extra computational cost. the loss function l is defined as the following: inference in the inference stage, at each time step, we first predict the bio-tag z t , the result will be one of three circumstances: if z @xmath4, we do not need to change anything. if z @xmath5, we mask all the token probability distributions of which they are not in the source sentence. if z @xmath6, it indicates the token sequence from current token to its nearest token with z @xmath7. therefore, we mask all the tokens in the token probability distribution that cannot constitute the corresponding n-gram in the source sentence. in this way, the model decoder still generates tokens in a step-bystep manner, rather than generating a segment at one time step. according to utilise the mask operation, the tokens where their z @xmath8. a detailed example has been shown in figure 1 for illustration. it should be pointed out that the proposed copy mechanism can not only improve the model performance in terms of long-span extraction, but also ensures the consistency between the generated text and the original text, thereby avoiding professional errors, which is quite necessary for practical use. ## related work generative relation extraction. @xcite proposed copyre, a joint model based on a copy mechanism, which transforms the joint extraction task into a generation task. @xcite propose two different encoders. the word decoder generates multiple triplets in a token-bytoken manner, and each triplets is distinguished by the special token, while the pointer decoder simply utilizes pointernet @xcite to generate start and end indexes for each entity. copy-mtl @xcite introduced a multi-task framework with a sequence labeling layer in the encoder, to alleviate the problem that copyre can only extract the last token in a multi-token entity. abstractive summarization. the previous work mainly leverage an encoder-decoder framework by choosing different model structures. @xcite utilize transformers or graph neural network @xcite for model encoder. despite most of the previous work conducting the seq2seq model, some works @xcite deploy a non-autoregressive model decoder to tackle this task, which also shows great effectiveness. ## conclusion in this paper, we propose biocopy, a plug-and-play copy mechanism to alleviate the long-span copying problem in generative tasks. by adding an extra sequence prediction layer in the training stage, our proposed approach is able to diminish the scope of probability distribution on each token. experiments in generative relation extraction and abstractive summarization verifies the model's effectiveness.