S TRUCT BERT: I NCORPORATING L ANGUAGE S TRUCTURES INTO P RETRAINING FOR D EEP L ANGUAGE U NDERSTANDING
A BSTRACT
Recently, the pre-trained language model, BERT (and its robustly optimized version
RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classiﬁcation, natural language inference, semantic textual similarity and question answering.
Inspired by the linearization exploration work of Elman (Elman, 1990), we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training.
Speciﬁcally, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively.
As a result, the new model is adapted to different levels of language understanding required by downstream tasks.
The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models at the time of model submission), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.
I NTRODUCTION
A pre-trained language model (LM) is a key component in many natural language understanding (NLU) tasks such as semantic textual similarity (Cer et al., 2017), question answering (Rajpurkar et al., 2016) and sentiment classiﬁcation (Socher et al., 2013).
In order to obtain reliable language representations, neural language models are designed to deﬁne the joint probability function of sequences of words in text with self-supervised learning.
Different from traditional word-speciﬁc embedding in which each token is assigned a global representation, recent work, such as Cove (McCann et al., 2017), ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2018), derives contextualized word vectors from a language model trained on a large text corpus.
These models have been shown effective for many downstream NLU tasks.
Among the context-sensitive language models, BERT (and its robustly optimized version
RoBERTa (Liu et al., 2019b)) has taken the NLP world by storm.
It is designed to pre-train bidirectional representations by jointly conditioning on both left and right context in all layers and model the representations by predicting masked words only through the contexts.
However, it does not make the most of underlying language structures.
According to Elman (Elman, 1990)’s study, the recurrent neural networks was shown to be sensitive to regularities in word order in simple sentences.
Since language ﬂuency is determined by the ordering of words and sentences, ﬁnding the best permutation of a set of words and sentences is an essential problem in many NLP tasks, such as machine translation and NLU (Hasler et al., 2017).
Recently, word ordering was treated as LM-based linearization solely based on language models (Schmaltz et al., 2016).
Schmaltz showed that recurrent neural network language models (Mikolov et al., 2010) with long short-term memory (Hochreiter & Schmidhuber, 1997) cells work effectively for word ordering even without any explicit syntactic information.
In this paper, we introduce a new type of contextual representation, StructBERT, which incorporates language structures into BERT pre-training by proposing two novel linearization strategies.
Speciﬁcally, in addition to the existing masking strategy, StructBERT extends BERT by leveraging the structural information: word-level ordering and sentence-level ordering.
We augment model pre-training with two new structural objectives on the inner-sentence and inter-sentence structures, respectively.
In this way, the linguistic aspects (Elman, 1990) are explicitly captured during the pre-training procedure.
With structural pre-training, StructBERT encodes dependency between words as well as sentences in the contextualized representation, which provides the model with better generalizability and adaptability.
StructBERT signiﬁcantly advances the state-of-the-art results on a variety of NLU tasks, including the
GLUE benchmark (Wang et al., 2018), the SNLI dataset (Bowman et al., 2015) and the SQuAD v1.
question answering task (Rajpurkar et al., 2016).
All of these experimental results clearly demonstrate
StructBERT’s exceptional effectiveness and generalization capability in language understanding.
We make the following major contributions:
• We propose novel structural pre-training that extends BERT by incorporating the word structural objective and the sentence structural objective to leverage language structures in contextualized representation.
This enables the StructBERT to explicitly model language structures by forcing it to reconstruct the right order of words and sentences for correct prediction.
• StructBERT signiﬁcantly outperforms all published state-of-the-art models on a wide range of NLU tasks at the time of model submission.
This model extends the superiority of BERT, and boosts the performance in many language understanding applications such as semantic textual similarity, sentiment analysis, textual entailment, and question answering.
S TRUCT BERT M ODEL P RE - TRAINING
StructBERT builds upon the BERT architecture, which uses a multi-layer bidirectional Transformer network (Vaswani et al., 2017).
Given a single text sentence or a pair of text sentences, BERT packs them in one token sequence and learns a contextualized vector representation for each token.
Every input token is represented based on the word, the position, and the text segment it belongs to.
Next, the input vectors are fed into a stack of multi-layer bidirectional Transformer blocks, which uses self-attention to compute the text representations by considering the entire input sequence.
The original BERT introduces two unsupervised prediction tasks to pre-train the model: i.e., a masked LM task and a next sentence prediction task.
Different from original BERT, our StructBERT ampliﬁes the ability of the masked LM task by shufﬂing certain number of tokens after word masking and predicting the right order.
Moreover, to better understand the relationship between sentences,
StructBERT randomly swaps the sentence order and predicts the next sentence and the previous sentence as a new sentence prediction task.
In this way, the new model not only explicitly captures the ﬁne-grained word structure in every sentence, but also properly models the inter-sentence structure in a bidirectional manner.
Once the StructBERT language model is pre-trained with these two auxiliary tasks, we can ﬁne-tune it on task-speciﬁc data for a wide range of downstream tasks.
2. I NPUT R EPRESENTATION
Every input x is a sequence of word tokens, which can be either a single sentence or a pair of sentences packed together.
The input representation follows that used in BERT (Devlin et al., 2018).
For each input token t i , its vector representation x i is computed by summing the corresponding token embedding, positional embedding, and segment embedding.
We always add a special classiﬁcation embedding ([CLS]) as the ﬁrst token of every sequence, and a special end-of-sequence ([SEP]) token to the end of each segment.
Texts are tokenized to subword units by WordPiece (Wu et al., 2016) and absolute positional embeddings are learned with supported sequence lengths up to 512 tokens.
In addition, the segment embeddings are used to differentiate a pair of sentences as in BERT.
TransformerEncoder
TokenSequence ! "
[$%&'] ! ) !
+ ! , [$%&'] … ! - . / (cid:1) .
0 (cid:1) .
1 (cid:1) . 2 (cid:1) .
3 (cid:1) .
4 (cid:1) .
5 (cid:1) .
6 (cid:1) 7 /
PositionalEmbedding !!
+ ! ) !
Pre-shuffledTrigram
ActualTokens … (a) Word Structural Objective
Sampler
CLS ! "
SEP !
# SEP TransformerEncoder "$ "$ "$
CLS !
# SEP ! "
SEP TransformerEncoder
Classlabel= Classlabel=
CLS ! "
SEP !
%&'( SEP
TransformerEncoder
Classlabel=
Corpus ) " )
*+,- …
NextSentPrediction
Prev.
SentPrediction
RandomSentPrediction (b) Sentence Structural Objective
We use a multi-layer bidirectional Transformer encoder (Vaswani et al., 2017) to encode contextual information for input representation.
Given the input vectors X = { x i } Ni =1 , an L -layer Transformer is used to encode the input as:
[Equation]
where l ∈ [1 , L ] , H 0 = X and H L
= [ h L 1 , · · · , h LN ] .
We use the hidden vector h Li as the contextualized representation of the input token t i .
2. P RE - TRAINING O BJECTIVES
To make full use of the rich inner-sentence and inter-sentence structures in language, we extend the pre-training objectives of original BERT in two ways: 1 (cid:13) word structural objective (mainly for the single-sentence task), and 2 (cid:13) sentence structural objective (mainly for the sentence-pair task).
We pre-train these two auxiliary objectives together with the original masked LM objective in a uniﬁed model to exploit inherent language structures.
2.3. W ORD S TRUCTURAL O BJECTIVE
Despite its success in various NLU tasks, original BERT is unable to explicitly model the sequential order and high-order dependency of words in natural language.
Given a set of words in random order from a sentence, ideally a good language model should be able to recover this sentence by reconstructing the correct order of these words.
To implement this idea in StructBERT, we supplement
BERT’s training objectives with a new word structural objective which endows the model with the ability to reconstruct the right order of certain number of intentionally shufﬂed word tokens.
This new word objective is jointly trained together with the original masked LM objective from BERT.
Figure 1a illustrates the procedure of jointly training the new word objective and the masked LM objective.
In every input sequence, we ﬁrst mask 15% of all tokens at random, as done in BERT (Devlin et al., 2018).
The corresponding output vectors h Li of the masked tokens computed by the bidirectional Transformer encoder are fed into a softmax classiﬁer to predict the original tokens.
Next, the new word objective comes into play to take word order into consideration.
Given the randomicity of token shufﬂing, the word objective is equivalent to maximizing the likelihood of placing every shufﬂed token in its correct position.
More formally, this objective can be formulated as:
[Equation]
where θ represents the set of trainable parameters of StructBERT, and K indicates the length of every shufﬂed subsequence.
Technically, a larger K would force the model to be able to reconstruct longer sequences while injecting more disturbed input.
On the contrary, when K is smaller, the model gets more undisturbed sequences while less capable of recovering long sequences.
We decide to use trigrams (i.e., K = 3 ) for subsequence shufﬂing to balance language reconstructability and robustness of the model.
Speciﬁcally, as shown in Figure 1a, we randomly choose some percentage of trigrams from unmasked tokens, and shufﬂe the three words (e.g., t 2 , t 3 , and t 4 in the ﬁgure) within each of the trigrams.
The output vectors of the shufﬂed tokens computed by the bidirectional Transformer encoder are then fed into a softmax classiﬁer to predict the original tokens.
The new word objective is jointly learned together with the masked LM objective in a uniﬁed pre-trained model with equal weights.
2.3. S ENTENCE S TRUCTURAL O BJECTIVE
The next sentence prediction task is considered easy for the original BERT model (the prediction accuracy of BERT can easily achieve 97%-98% in this task (Devlin et al., 2018)).
We, therefore, extend the sentence prediction task by predicting both the next sentence and the previous sentence, to make the pre-trained language model aware of the sequential order of the sentences in a bidirectional manner.
As illustrated in Figure 1b, given a pair of sentences ( S 1 , S 2 ) as input, we predict whether S 2 is the next sentence that follows S 1 , or the previous sentence that precedes S 1 , or a random sentence from a different document.
Speciﬁcally, for the sentence S 1 , 13 of the time we choose the text span that follows S 1 as the second sentence S 2 , 13 of the time the previous sentence ahead of S 1 is selected, and 1 3 of the time a sentence randomly sampled from the other documents is used as S 2 .
The two sentences are concatenated together into an input sequence with the separator token
[SEP] in between, as done in BERT.
We pool the model output by taking the hidden state corresponding to the ﬁrst token
[CLS], and feed the encoding vector of [CLS] into a softmax classiﬁer to make a three-class prediction.
2. P RE - TRAINING S ETUP
The training objective function is a linear combination of the word structural objective and the sentence structural objective.
For the masked LM objective, we followed the same masking rate and settings as in BERT (Devlin et al., 2018).
5% of trigrams are selected for random shufﬂing.
We used documents from English Wikipedia (2,500M words) and BookCorpus (Zhu et al., 2015) as pre-training data, following the preprocessing and the WordPiece tokenization from (Devlin et al., 2018).
The maximum length of input sequence was set to 512.
We ran Adam with learning rate of 1e-4, β 1 = 0 .
9 , β 2 = 0 .
999 , L2 weight decay of 0.01, learning rate warm-up over the ﬁrst 10% of the total steps, and linear decay of the learning rate.
We set a dropout probability of 0.1 for every layer.
The gelu activation (Hendrycks & Gimpel, 2016) was used as done in GPT (Radford et al., 2018).
We denote the number of Transformer block layers as L , the size of hidden vectors as H , and the number of self-attention heads as A .
Following the practice of BERT, We primarily report experimental results on the two model sizes:
StructBERT Base : L = 12 , H = 768 , A = 12 , Number of parameters =
110 M
StructBERT Large : L = 24 , H = 1024 , A = 16 , Number of parameters = 340 M
Pre-training of StructBERT was performed on a distributed computing cluster consisting of 64 Telsa
V100 GPU cards.
For the StructBERT Base , we ran the pre-training procedure for 40 epochs, which took about 38 hours, and the training of StructBERT Large took about 7 days to complete.
E XPERIMENTS
In this section, we report results of StructBERT on a variety of downstream tasks including General
Language Understanding Evaluation (GLUE benchmark), Standford Natural Language inference (SNLI corpus) and extractive question answering (SQuAD v1.1).
Following BERT’s practice, during ﬁne-tuning on downstream tasks, we performed a grid search or an exhaustive search (depending on the data size) on the following sets of parameters and chose the model that performed the best on the dev set.
All the other parameters remain the same as those in pre
[Table]
results https gluebenchmark leaderboard submitted model name ALICE)
3. G ENERAL L ANGUAGE U NDERSTANDING 3.1. GLUE BENCHMARK
The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) is a collection of nine NLU tasks, covering textual entailment (RTE (Bentivogli et al., 2009) and
MNLI (Williams et al., 2017)), question-answer entailment (QNLI (Wang et al., 2018)), paraphrase (MRPC (Dolan & Brockett, 2005)), question paraphrase (QQP), textual similarity (STS-B (Cer et al., 2017)), sentiment (SST-2 (Socher et al., 2013)), linguistic acceptability (CoLA), and Winograd
Schema (WNLI (Levesque et al., 2012)).
On the GLUE benchmark, given the similarity of MRPC/RTE/STS-B to MNLI, we ﬁne-tuned
StructBERT on MNLI before training on MRPC/RTE/STS-B data for the respective tasks.
This follows the two-stage transfer learning STILTs introduced in (Phang et al., 2018).
For all the other tasks (i.e., RTE, QNLI, QQP, SST-2, CoLA and MNLI), we ﬁne-tuned StructBERT for each single task only on its in-domain data.
Table 1 presents the results of published models on the GLUE test set obtained from the ofﬁcial benchmark evaluation server.
Our StructBERT Large ensemble suppressed all published models (excluding RoBERTa ensemble and XLNet ensemble) on the average score, and performed the best among these models in six of the nine tasks.
In the most popular MNLI task, our StructBERT Large single model improved the best result by 0.3%/0.5%, since we ﬁne-tuned MNLI only on its in-domain data, this improvement is entirely attributed to our new training objectives.
The most signiﬁcant improvement over BERT was observed on CoLA (4.8%), which may be due to the strong correlation between the word order task and the grammatical error correction task.
In the SST-2 task, our model improved over BERT while performed worse than MT-DNN did, which indicates that sentiment analysis based on single sentences beneﬁts less from the word structural objective and sentence structural objective. [Table] [Table]
With pre-training on large corpus, XLNet ensemble and RoBERTa ensemble outperformed all published models including our StructBERT Large ensemble.
To take advantage of the large data which RoBERTa is trained on, we continued pre-training with our two new objectives from the released RoBERTa model, named StructBERT RoBERTa .
At the time of model submission, our
StructBERT RoBERTa ensemble, which was submitted under a different name ALICE, achieved the best performance among all published models including RoBERTa and XLNet on the leaderboard, creating a new state-of-the-art result of 89.0% on the average GLUE score.
It demonstrates that the proposed objectives are able to improve language models in addition to BERT.
3.1. SNLI
Natural Language Inference (NLI) is one of the important tasks in natural language understanding.
The goal of this task is to test the ability of the model to reason the semantic relationship between two sentences.
In order to perform well on an NLI task, a model needs to capture the semantics of sentences, and thus to infer the relationship between a pair of sentences: entailment, contradiction or neutral.
We evaluated our model on the most widely used NLI dataset: The Stanford Natural Language
Inference (SNLI) Corpus (Bowman et al., 2015), which consists of 549,367/9,842/9,824 premisehypothesis pairs in train/dev/test sets and target labels indicating their relations.
We performed a grid search on the sets of parameters, and chose the model that performed best on the dev set.
Table 2 shows the results on the SNLI dataset of our model with other published models.
StructBERT outperformed all existing systems on SNLI, creating new state-of-the-art results 91.7%, which amounts to 0.4% absolute improvement over the previous state-of-the-art model SJRC and 0.9% absolute improvement over BERT.
Since the network architecture of our model is identical to that of
BERT, this improvement is entirely attributed to the new pre-training objectives, which justiﬁes the effectiveness of the proposed tasks of word prediction and sentence prediction.
3. E XTRACTIVE Q UESTION A NSWERING
SQuAD v1.1 is a popular machine reading comprehension dataset consisting of 100,000+ questions created by crowd workers on 536 Wikipedia articles (Rajpurkar et al., 2016).
The goal of the task is to extract the right answer span from the corresponding paragraph given a question.
We ﬁne-tuned our StructBERT language model on the SQuAD dataset for 3 epochs, and compared the result against the state-of-the-art methods on the ofﬁcial leaderboard 1 , as shown in Table 3.
We can see that even without any additional data augmentation (DA) techniques, the proposed StructBERT model was superior to all published models except XLNet+DA on the dev set.
With data augmentation and large corpus used during pre-training, XLNet+DA outperformed our StructBERT which did not We have submitted the model under the name of ALICE to the SQuAD v1.1 CodaLab for evaluation on the test set. However, due to crash of the Codalab evaluation
[Table]
use data augmentation or large pre-training corpus. It demonstrates the effectiveness of the proposed pre-trained StructBERT in modeling the question-paragraph relationship for extractive question answering.
Incorporating the word and sentence structures signiﬁcantly improves the understanding ability in this ﬁne-grained answer extraction task.
3. E FFECT OF D IFFERENT S TRUCTURAL O BJECTIVES
We have demonstrated the strong empirical results of the proposed model on a variety of downstream tasks.
In the StructBERT pre-training, the two new structural prediction tasks are the most important components.
Therefore, we conducted an ablation study by removing one structural objective from pre-training at a time to examine how the two structural objectives inﬂuence the performance on various downstream tasks.
Results are presented in Table 4.
From the table, we can see that: (1) the two structural objectives were both critical to most of the downstream tasks, except for the word structural objective in the
SNLI task.
Removing any word or sentence objective from pre-training always led to degraded performance in the downstream tasks.
The StructBERT model with structural pre-training consistently outperformed the original BERT model, which shows the effectiveness of the proposed structural objectives.
For the sentence-pair tasks such as MNLI, SNLI, QQP and SQuAD, incorporating the sentence structural objective signiﬁcantly improved the performance.
It demonstrates the effect of inter-sentence structures learned by pre-training in understanding the relationship between sentences for downstream tasks.
For the single-sentence tasks such as CoLA and SST-2, the word structural objective played the most important role.
Especially in the CoLA task, which is related to the grammatical error correction, the improvement was over 5%.
The ability of reconstructing the order of words in pre-training helped the model better judge the acceptability of a single sentence.
We also studied the effect of both structural objectives during self-supervised pre-training.
Figureillustrates the loss and accuracy of word and sentence prediction over the number of pre-training steps for StructBERT Base and BERT Base .
From the two sub-ﬁgures on top, it is observed that compared with BERT, the augmented shufﬂed token prediction in StructBERT’s word structural objective had little effect on the loss and accuracy of masked token prediction.
On the other hand, the integration of the simpler task of shufﬂed token prediction (lower loss and higher accuracy) provides
StructBERT with the capability of word reordering.
In contrast, the new sentence structural objective in StructBERT leads to a more challenging prediction task than that in BERT, as shown in the two ﬁgures at the bottom.
This new pre-training objective enables StructBERT to exploit inter-sentence structures, which beneﬁts sentence-pair downstream tasks.
R ELATED W ORK 4. C ONTEXTUALIZED L ANGUAGE R EPRESENTATION
A word can have different semantics depending on the its context.
Contextualized word representation is considered to be an important part of modern NLP research, with various pre-trained language models (McCann et al., 2017; Peters et al., 2018;
Radford et al., 2018;
Devlin et al., 2018) emerging recently.
ELMo (Peters et al., 2018) learns two unidirectional LMs based on long short-term memory networks (LSTMs).
A forward LM reads the text from left to right, and a backward LM encodes the text from right to left.
Following the similar idea of ELMo, OpenAI GPT (Radford et al., 2018) expands the unsupervised language model to a much larger scale by training on a giant collection of free text corpora.
Different from ELMo, it builds upon a multi-layer Transformer (Vaswani et al., 2017) decoder, and uses a left-to-right Transformer to predict a text sequence word-by-word.
In contrast, BERT (Devlin et al., 2018) (as well as its robustly optimized version RoBERTa (Liu et al., 2019b)) employs a bidirectional Transformer encoder to fuse both the left and the right context, and introduces two novel pre-training tasks for better language understanding.
We base our LM on the architecture of BERT, and further extend it by introducing word and sentence structures into pre-training tasks for deep language understanding.
4. W ORD & S ENTENCE O RDERING
The task of linearization aims to recover the original order of a shufﬂed sentence (Schmaltz et al., 2016).
Part of larger discussion as to whether LSTMs are capturing syntactic phenomena linearization, is standardized in a recent line of research as a method useful for isolating the performance of text-totext generation (Zhang & Clark, 2015) models.
Recently, Transformers have emerged as a powerful architecture for learning the latent structure of language.
For example, Bidirectional Transformers (BERT) has reduced the perplexity for language modeling task.
We revisit Elman’s question by applying BERT to the word-ordering task, without any explicit syntactic approaches, and ﬁnd that pre-trained language models are effective for various downstream tasks with linearization.
Many important downstream tasks such as STS and NLI (Wang et al., 2018) are based on understanding the relationship between two text sentences, which is not directly captured by language modeling.
While BERT (Devlin et al., 2018) pre-trains a binarized next sentence prediction task to understand sentence relationships, we take one step further and treat it as a sentence ordering task.
The goal of sentence ordering is to arrange a set of sentences into a coherent text in a clear and consistent manner, which can be viewed as a ranking problem (Chen et al., 2016).
The task is general and yet challenging, and once is especially important for natural language generation (Reiter & Dale, 1997).
Text should be organized according to the following properties: rhetorical coherence, topical relevancy, chronological sequence, and cause-effect.
In this work, we focus on what is arguably the most basic characteristics of a sequence: their order.
Most of prior work on sentence ordering was part of the study of downstream tasks, such as multi-document summarization (Bollegala et al., 2010).
We revisit this problem in the context of language modeling as a new sentence prediction task.
C ONCLUSION
In this paper, we propose novel structural pre-training which incorporates word and sentence structures into BERT pre-training.
A word structural objective and a sentence structural objective are introduced as two new pre-training tasks for deep understanding of natural language in different granularities.
Experimental results demonstrate that the new StructBERT model can obtain new state-of-the-art