C opy N ext: Explicit Span Copying and Alignment in Sequence to Sequence Models (2020)

Abstract: Copy mechanisms are employed in sequence to sequence (seq2seq) models to generate reproductions of words from the input to the output. These frameworks, operating at the lexical type level, fail to provide an explicit alignment that records where each token was copied from. Further, they require contiguous token sequences from the input (spans) to be copied individually. We present a model with an explicit token-level copy operation and extend it to copying entire spans. Our model provides hard alignments between spans in the input and output, allowing for nontraditional applications of seq2seq, like information extraction. We demonstrate the approach on Nested Named Entity Recognition, achieving near state-of-the-art accuracy with an order of magnitude increase in decoding speed.

## introduction sequence transduction converts a sequence of input tokens to a sequence of output tokens. it is a dominant framework for generation tasks, such as machine translation, dialogue, and summarization. seq2seq can also be used for information extraction (ie), where the target structure is decoded as a linear output based on an encoded (linear) representation of the input. as ie is traditionally considered a structured prediction task, it remains today that ie systems are assumed to produce an annotation on the input text. that is, predicting which specific tokens of an input string led to, e.g., the label of person. this is in contrast to text generation which rarely, if ever, needs hard alignments between the input and the desired output. our work explores a novel extension to seq2seq that provides such alignments. indices corresponding to the tokens in the input sequence. cn refers to the copynext symbol, our proposed method of denoting the operation that copies the next token from the input. in (d), the next token from token 4 would be 5. specifically, we extend pointer (or copy) networks. unlike the algorithmic tasks originally targeted by @xcite , tasks in nlp tend to copy spans from the input rather than discontiguous tokens. this is prevalent for copying named entities in dialogue @xcite , entire sentences in summarization @xcite , or even single words (if subtokenized). the need to efficiently copy spans motivates our introduction of an inductive bias that copies contiguous tokens. like a pointer network, our model copies the first token of a span. however, for subsequent timesteps, our model generates a "copynext" symbol (cn) instead of copying another token from source. copynext represents the operation of copying the word following the last predicted word from the input sequence. figure 1 highlights the difference between output sequences for several transductive models, including our copynext model. we apply our model for the nested named entity recognition (nner) task @xcite . unlike traditional named entity recognition, named entity mentions in nner may be subsequences of other named entity mentions (such as [[last] year] in figure 1 ). we find that both explicit copying and copynext lead to a system faster than prior work and better than a simple seq2seq baseline. it is, however, outperformed by a much slower model that performs an exhaustive search over the space of potential labels, a solution that does not scale to large complex label sets. ## related work pointer networks @xcite @xcite are seq2seq models that employ a soft attention distribution @xcite to produce an output sequence consisting of values from the input sequence. pointer-generator networks @xcite extend the range of output types by combining the distribution from the pointer with a vocabulary distribution from a generator. thus, these models operate on the type level. in contrast, our model operates at the token level. instead of using soft attention distribution of the encoder states, we use hard attention, resulting in a single encoder state, or a single token, to feed to the decoder. this enables explicit copying of span offsets. closest to our work, @xcite and panthaplackel et al. (2020) have tackled span copying by extending pointer-generator networks and predicting both start and end indices of entire spans that need to be copied. using those offsets, they perform a forced decoding of the predicted tokens within the span. these works focus on text generation tasks, like sentence summarization, question generation, and editing. in contrast, we are concerned with information extraction tasks as transduction, where hard alignments to the input sentence are crucial and output sequences must represent a valid linearized structure. specifically, we study nested named entity recognition (nner). prior work uses several approaches to model nner: machine reading comprehension @xcite , transition-based methods @xcite , mention hypergraphs @xcite @xcite , and seq2seq models @xcite . ## model description we formulate the task as transforming the input sentence x to a linearized sequence y which represents the gold structure: labeled spans. specifically, y contains input word indices, copynext symbols, and labels from a label set l. as described earlier, the model (figure 2 ) is reminiscent of pointer networks. we extend its capabilities by introducing the notion of a "copy next" operation where the network predicts to copy the word sequentially after the previous prediction. ## experiments and results our experiments analyze the effects of various choices in different components of the system. we use the nne dataset and splits from @xcite , resulting in 43,457, 1,989, and 3,762 sentences in the training, development, and test splits. experiments for model development and analysis use the development set. linearization strategy previous work @xcite has shown that linearization scheme affects model performance. we experiment with several variants of sorting spans in ascending order based on their start index. we also try sorting based on end index and copying the previous token instead. we find sorting based on end index performs poorly, while sorting by start all perform similarly. our final linearization strategy sorts by start index, then span length (longer spans first). additional ties (in span label) are broken randomly. roberta embedding layer recent work suggests that ner information may be stored in the lower layers of an encoder @xcite . we found using the 15th layer of roberta rather than the final one (24th), is slightly helpful (see appendix a.1). ## conclusion and future work we propose adopting pointer and copy networks with hard attention and extending these models 5 efficiency is measured using wall clock time for the entire test set, performed with intel xeon 2.10ghz cpu and a single geforce gtx 1080 ti gpu. with a copynext operation, enabling sequential copying of spans given just the start index of the span. on a traditionally structured prediction task of nner, we use a sequence transduction model with the copynext operation, leading to a competitive model that provides a 16.7x speedup relative to current state of the art (which performs an exhaustive search), at a cost of 4.7% loss in f1, largely due to lower recall. our model is a step forward in structured prediction as sequence transduction. we have found in initial experiments on event extraction similar relative improvements to that discussed here: future work will investigate applications to richer transductive semantic parsing models @xcite .