S pan BERT : Improving Pre-training by Representing and Predicting Spans (2020)

Abstract: We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERTlarge, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.1

## introduction pre-training methods like bert @xcite have shown strong performance gains using self-supervised training that masks individual words or subword units. however, many nlp tasks involve reasoning about relationships between two or more spans of text. for example, in extractive question answering @xcite , de-termining that the ''denver broncos'' is a type of ''nfl team'' is critical for answering the question ''which nfl team won super bowl 50?'' such spans provide a more challenging target for self supervision tasks, for example, predicting ''denver broncos'' is much harder than predicting only ''denver'' when you know the next word is ''broncos''. in this paper, we introduce a spanlevel pretraining approach that consistently outperforms bert, with the largest gains on span selection tasks such as question answering and coreference resolution. we present spanbert, a pre-training method that is designed to better represent and predict spans of text. our method differs from bert in both the masking scheme and the training objectives. first, we mask random contiguous spans, rather than random individual tokens. second, we introduce a novel span-boundary objective (sbo) so the model learns to predict the entire masked span from the observed tokens at its boundary. span-based masking forces the model to predict entire spans solely using the context in which they appear. furthermore, the sbo encourages the model to store this span-level information at the boundary tokens, which can be easily accessed during the fine-tuning stage. figure 1 illustrates our approach. to implement spanbert, we build on a welltuned replica of bert, which itself substantially outperforms the original bert. while building on our baseline, we find that pre-training on single segments, instead of two half-length segments with the next sentence prediction (nsp) objective, the equation shows the mlm and sbo loss terms for predicting the token, football (in pink), which as marked by the position embedding p 3 , is the third token from x 4 . together, our pre-training process yields models that outperform all bert baselines on a wide variety of tasks, and reach substantially better performance on span selection tasks in particular. specifically, our method reaches 94.6% and 88.7% f1 on squad 1.1 and 2.0 @xcite , respectively-reducing error by as much as 27% compared with our tuned bert replica. we also observe similar gains on five additional extractive question answering benchmarks (newsqa, triviaqa, searchqa, hotpotqa, and natural questions). 2 spanbert also arrives at a new state of the art on the challenging conll-2012 (''ontonotes'') shared task for document-level coreference resolution, where we reach 79.6% f1, exceeding the previous top model by 6.6% absolute. finally, we demonstrate that spanbert also helps on tasks that do not explicitly involve span selection, and show that our approach even improves performance on tacred @xcite and glue @xcite . whereas others show the benefits of adding more data @xcite and increasing model size @xcite , this work demonstrates the importance of designing good pre-training tasks and objectives, which can also have a remarkable impact. notation given a sequence of word or subword tokens @xmath0, x 2 , . . . , x n ), bert trains an encoder that produces a contextualized vector representation for each token: masked language model also known as a cloze test, mlm is the task of predicting missing tokens in a sequence from their placeholders. specifically, a subset of tokens y ⊆ x is sampled and substituted with a different set of tokens. in bert's implementation, y accounts for 15% of the tokens in x; of those, 80% are replaced with [mask], 10% are replaced with a random token (according to the unigram distribution), and 10% are kept unchanged. the task is to predict the original tokens in y from the modified input. bert selects each token in y independently by randomly selecting a subset. in spanbert, we define y by randomly selecting contiguous spans (section 3.1). ## model we present spanbert, a self-supervised pretraining method designed to better represent and predict spans of text. our approach is inspired by bert @xcite , but deviates from its bi-text classification framework in three ways. first, we use a different random process to mask spans of tokens, rather than individual ones. we also introduce a novel auxiliary objective-the sbo-which tries to predict the entire masked span using only the representations of the tokens at the span's boundary. finally, spanbert samples a single contiguous segment of text for each training example (instead of two), and thus does not use bert's next sentence prediction objective, which we omit. ## results we compare spanbert to the baselines per task, and draw conclusions based on the overall trends. ## ablation studies we compare our random span masking scheme with linguistically-informed masking schemes, and find that masking random spans is a competitive and often better approach. we then study the impact of the sbo, and contrast it with bert's nsp objective. 10 ## related work pre-trained contextualized word representations that can be trained from unlabeled text @xcite @xcite have had immense impact on nlp lately, particularly as methods for initializing a large model before fine-tuning it for a specific task @xcite @xcite . beyond differences in model hyperparameters and corpora, these methods mainly differ in their pre-training tasks and loss functions, with a considerable amount of contemporary literature proposing augmentations of bert's mlm objective. while previous and concurrent work has looked at masking @xcite or dropping @xcite multiple words from the input-particularly as pretraining for lan-guage generation tasks-spanbert pretrains span representations @xcite , which are widely used for question answering, coreference resolution, and a variety of other tasks. ernie @xcite shows improvements on chinese nlp tasks using phrase and named entity masking. mass @xcite focuses on language generation tasks, and adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence. we attempt to more explicitly model spans using the sbo objective, and show that (geometrically distributed) random span masking works as well, and sometimes better than, masking linguisticallycoherent spans. we evaluate on english benchmarks for question answering, relation extraction, and coreference resolution in addition to glue. a different ernie @xcite focuses on integrating structured knowledge bases with contextualized representations with an eye on knowledge-driven tasks like entity typing and relation classification. unilm @xcite uses multiple language modeling objectivesunidirectional (both left-to-right and right-to-left), bidirectional, and sequence-to-sequence predictionto aid generation tasks like summarization and question generation. xlm @xcite explores cross-lingual pre-training for multilingual tasks such as translation and cross-lingual classification. kermit @xcite , an insertion based approach, fills in missing tokens (instead of predicting masked ones) during pretraining; they show improvements on machine translation and zero-shot question answering. concurrent with our work, roberta @xcite presents a replication study of bert pre-training that measures the impact of many key hyperparameters and training data size. also concurrent, xlnet @xcite combines an autoregressive loss and the transformer-xl @xcite architecture with a more than an eight-fold increase in data to achieve current stateof-the-art results on multiple benchmarks. xlnet also masks spans (of 1-5 tokens) during pretraining, but predicts them autoregressively. our model focuses on incorporating span-based pretraining, and as a side effect, we present a stronger bert baseline while controlling for the corpus, architecture, and the number of parameters. related to our sbo objective, pair2vec @xcite encodes word-pair relations using a negative sampling-based multivariate objective during pre-training. later, the word-pair representations are injected into the attention-layer of downstream tasks, and thus encode limited downstream context. unlike pair2vec, our sbo objective yields ''pair'' (start and end tokens of spans) representations which more fully encode the context during both pre-training and finetuning, and are thus more appropriately viewed as span representations. @xcite focus on improving language generation speed using a block-wise parallel decoding scheme; they make predictions for multiple time steps in parallel and then back off to the longest prefix validated by a scoring model. also related are sentence representation methods @xcite , which focus on predicting surrounding contexts from sentence embeddings. ## conclusion we presented a new method for span-based pretraining which extends bert by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. together, our pretraining process yields models that outperform all bert baselines on a variety of tasks, and reach substantially better performance on span selection tasks in particular.