Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models (2020)

Abstract: Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations. We propose Conpono, an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. Given an anchor sentence, our model is trained to predict the text k sentences away using a sampled-softmax objective where the candidates consist of neighboring sentences and sentences randomly sampled from the corpus. On the discourse representation benchmark DiscoEval, our model improves over the previous state-of-the-art by up to 13% and on average 4% absolute across 7 tasks. Our model is the same size as BERT-Base, but outperforms the much larger BERT-Large model and other more recent approaches that incorporate discourse. We also show that Conpono yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse: textual entailment (RTE), common sense reasoning (COPA) and reading comprehension (ReCoRD).

## introduction pretraining large language models has become the primary method for learning representations from unsupervised text corpora. since the initial improvements demonstrated by elmo @xcite and bert @xcite , many alternative pretraining methods have been proposed to best leverage unlabeled data. these methods include bi-directional language modeling @xcite , masked language models @xcite , word order permutation @xcite , more robust training @xcite and more efficient architectures @xcite . however, little focus has been put on learning discourse coherence as part of the pretraining objective. while discourse coherence has been of great interest in recent natural language processing literature @xcite @xcite , its benefits have been questioned for pretrained language models, some even opting to remove any sentence ordering objective @xcite . however, in a recently published benchmark for evaluating discourse representations, @xcite found that the best performing model was surprisingly bert, despite comparing against models specifically designed for discourse, such as dissent @xcite and a new recurrent network trained on a large range of sentence ordering objectives. we show that combining transformer encoders with our intersentence coherence objective, we can further improve discourse-level representations in language models. we present a model that trains a sentence-level encoder to capture discourse relationships between sentences, including ordering, distance and coherence. the encoder is trained by using its output to predict spans of text that are some k sentences away from a context in either direction. the predictions are made discriminatively with a sampled-softmax that contrasts the correct target sentence against negatives, including hard examples sampled from the same paragraph. our objective is inspired by the recently proposed constrastive predictive coding (cpc) @xcite , but, among other differences, is applied on the sentence-level rather than the token-level and is bi-directional. we call this the contrastive position and ordering with negatives objective (conpono) 2 . we evaluate our model on discoeval @xcite , a recently published benchmark for evaluating and probing for various aspects of discourselevel semantics in representations output by discourse models. we observe that the representations learned with conpono outperform bert-large and achieve a new state-of-the-art despite using fewer parameters and training on the same data. furthermore, we show that our new objective improves model performance on other tasks including textual entailment, common-sense reasoning and reading comprehension. we compare conpono against bert-base on rte @xcite , copa @xcite and record @xcite , while controlling for model size, training data and training time. our main contributions are: figure 1 illustrates the conpono model. the intuition is that if the model is able to accurately predict the surrounding target sentences given some anchor text, then the vector representations for these sentences should also be useful for downstream tasks. the input to the model is a paragraph that is split into sentences. a sentence is chosen at random as the anchor, and will be denoted as s i . we encode s i with a transformer encoder to produce a vector c i . the surrounding sentences are denoted as s i+k where @xmath0.. -1, 1 .. k], meaning the maximum distance we use is k. we report results for @xmath1..4]. these sentences, s i+k , are encoded jointly with the anchor sentence. we use just a single encoder g θ so all text is encoded with the same weights. the encoded vectors are named t i+k because these are the target vectors the model tries to identify given the anchor and a target distance k. equation 1 defines t i+k and c i as a function g θ of the input sentences. note that the conpono g θ is different from the encoder in cpc because we input both the anchor and the target into the encoder, rather than separate anchor and target encoders. given the anchor and targets, we define a logbilinear model in equation 2 to score the plausibility of target t i+k being in position k from anchor c i . the full set of parameters for our model is θ for the encoder and a w k for each k. cpc has the same bi-linear form as equation 2 but the architecture for the encoders is different. the loss for each k is given in equation 3 where the score for the correct target is contrasted to scores of random samples s j , sampled from both indocument and random sentences from the corpus, s. to train conpono, we sample negative examples randomly from the corpus and from the same paragraph but different k as hard negatives. note that when |k| is greater than 1, there will be sentences between the anchor sentence and target sentence that will be purposely omitted from the input. the missing context is intended to create a challenging objective where the model may not be able to rely on trivial signals that often appear in contiguous sentences. ## evaluation we evaluate our model on the discoeval benchmark @xcite and on the rte @xcite , copa @xcite and record @xcite datasets. we chose the discoeval benchmark because it is intended to evaluate a model's ability to represent the "role of a sentence in its discourse context". we also report results on rte, copa and record because these tasks have a discourse or sentence ordering aspect to them but are not exclusively designed for discourse evaluation. ## related work some of the largest improvements on benchmarks such as glue @xcite have come from elmo's large scale bi-directional language modeling @xcite ), bert's masked language models @xcite , xlnet's generalized autoregressive pretraining @xcite ), roberta's robust training @xcite ) and albert's parameter reduction techniques @xcite . as discussed in section 2.2, most language model were limited to nsp or bso for inter-sentence representation learning. we showed that by comparing to bert, which uses nsp and bert-base bso which we train with the bso objective that our objective is able to improve the discourse-level representations by training on more fine-grained sentence ordering, non-contiguous @xcite vaughn joined the united states national guard for what had been planned as a one-year assignment , but when [2]world war ii broke out , he was sent abroad until the war ended in 1945 . although it lasted only twenty-three years and enrolled fewer than 1,200 students , black mountain college was one of the most fabled experimental institutions in art education and practice . it launched a remarkable number of the artists who spearheaded the avant-garde in the america of the 1960s . early approaches to sentence representation, such as skip-thought vectors @xcite , mimicked word embedding methods in addition to left-to-right language modeling to use unlabeled data to learn sentence level representations. dis-sent @xcite focused more on collecting data that could be used to train a supervised classification model on pairs of sentences. these and other innovations in sentence representation lead to the creation of more evaluations for discourse and coherence representation @xcite . like other unsupervised representation learning models, conpono is trained to generate a latent variable that encodes inter-sentence relationship and discourse coherence. our objective is inspired by the contrastive predictive coding (cpc) objective @xcite . cpc was originally designed to be a "universal unsupervised learning approach to extract useful representations from high-dimensional data" and was previously implemented on the token-level for text models. we utilize the k-distance predictions of cpc because it naturally captures discourse and sentence ordering properties when applied on the sentencelevel. furthermore, by combining our objective with a transformer encoder, our model is able to benefit from cross-attention between the anchor and the target sentences, which we show outperforms encoding the anchor and target separately, as implemented in cpc. in section 3.3 we show that the cross-attention is an important factor in learning a good representation for downstream tasks and effectively optimizing our inter-sentence objective. ## discussion in this paper we present a novel approach to encoding discourse and fine-grained sentence ordering in text with an inter-sentence objective. we achieve a new state-of-the-art on the discoeval benchmark and outperform bert-large with a model that has the same number of parameters as bert-base. we also observe that, on discoeval, our model benefits the most on ordering tasks rather than discourse relation classification tasks. in future work, we hope to better understand how a discourse model can also learn fine-grained relationship types between sentences from unlabeled data. our ablation analysis shows that the key architectural aspects of our model are cross attention, an auxiliary mlm objective and a window size that is two or greater. future work should explore the extent to which our model could further benefit from initializing with stronger models and what computational challenges may arise.