SLM : Learning a Discourse Language Representation with Sentence Unshuffling (2020)

Abstract: We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation in a fully self-supervised manner. Recent pre-training methods in NLP focus on learning either bottom or top-level language representations: contextualized word representations derived from language model objectives at one extreme and a whole sequence representation learned by order classification of two given textual segments at the other. However, these models are not directly encouraged to capture representations of intermediate-size structures that exist in natural languages such as sentences and the relationships among them. To that end, we propose a new approach to encourage learning of a contextualized sentence-level representation by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. Through experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show that this feature of our model improves the performance of the original BERT by large margins.

## introduction recent representation learning methods in nlp such as bert @xcite have focused on learning two types of representations: the bottomlevel -a contextual representation centered at a single word, trained by recovering randomly masked tokens or predicting previous and next words, and the top-level text, implicitly represented as a single @xcite symbol and trained by predicting a relation between input segments, which usually consist of multiple sentences. however, natural language text has, in contrast, a very dominant hierarchical structure, with words grouped together into intermediate semantic units such as phrases and then sentences to convey the full meaning of a given text. neither the transformer architecture nor the pre-training task leverages this hierarchy, treating the language as a flat sequence of tokens instead. inspired by prior works about sentence-based representations for recurrent networks @xcite @xcite @xcite @xcite , we seek to incorporate a more explicit hierarchy into the transformer by extending it with the capacity to learn contextualized sentence-level representations. equipping the model with such a capacity allows it to learn languages at multiple levels of granularity, ranging from fine-grain connections across words to high-level discourse relations between sentences and paragraphs. predicting an original sequence of sentences requires deep understanding of natural language, including a variety of phenomena such as discourse relations, coreference, temporal dependencies, entailments, narratives, and so on. figure 1 shows a set of statements from a conversation in the coqa dataset @xcite , and clues that can potentially be used to reconstruct their original ordering. to figure out the correct order, in this case c-d-b-a, a model has to understand that d and a are answers to questions c and b respectively and that 'she' in b refers to d, and it may need to know that b is a follow-up question on c. not only are the semantics of separate sentences important but the relationships between them are crucial in solving this task. we therefore seek to encourage the model to capture these vital properties by training it to reconstruct the original sentence ordering. to achieve this, we propose a pre-training objective that extends the word-level language modeling to analogously learn representations for sentences. typical word-level language models are trained by guessing the neighbors of given words or by predicting masked words based on their context. applying the same idea at a higher level, our approach learns representations that support predicting the next sentence representation among shuffled sentences for given previous representations, based on the semantic relations between them. to allow the model to be effectively trained with the new unshuffling objective, we propose a pointerbased neural module that is specialized to predict the sentence order, called the sequence reconstructor (sr). in the sr, each sentence is represented by a sentential token that is inserted in front of it. a pointer network layer stacked on the transformer decoder is trained to point at the next contextualized sentence representation based on the previous sentence representations. we show that our method achieves robust improvements over standard bert's performance on the following downstream nlp tasks: glue for natural language inference @xcite , squad for question answering @xcite , and discoeval @xcite for discourse aware sentence representations. we match the score of text-to-text transfer transformer base (t5 base ) @xcite , a state-of-theart model that uses bert base hyperparameters, while using only half the parameters, shorter training (3/8 tokens overall), and a fraction (1/37) of the data compared to t5. moreover, we investigate the effect of the proposed objective through a qualitative analysis of the neighbor sentences of sentences that have similar sentential representations. we show that the results support our aim of enriching the transformer model with sentencelevel language understanding. the contributions of our work are threefold: 1) we propose a new self-supervised pre-training objective that extends a word-level language modeling strategy to the sentence-level; 2) we propose a pointer-based neural module to train the model for this objective, and demonstrate it leads to significant improvements over diverse nlp tasks; and 3) we provide a qualitative analysis showing that the learned contextualized sentence representations embed more subtle and rich structural, semantic, and relational information. ## proposed method in this section, we describe the details of our proposed methods. a conventional representation model is extended with sentence representations to gather sentence information and a sequence reconstructor (sr) to predict their original ordering. fine-tuning architectures are modified to inject sentence representation into downstream nlp tasks. ## experiments 3.1 benchmark datasets to measure the excellence of our propose slm on various downstream tasks, we conduct experiments with three well-known benchmarks: general language understanding evaluation (glue), the stanford question answering (squad) v1.1 and v2.0, and discourse evaluation (discoeval) @xcite . we exclude the wnli task from glue following bert which always achieves 65.1 accuracy of the majority class. rst-dt task is also excluded from discoeval which requires a tree structure encoding. metrics used are matthew's correlation for cola, f1 for mrpc, spearman's correlation for sts-b, exact match (em) / f1 for squad, and accuracy for rest of tasks. we report development set results for squad to compare with other models, both development and test set scores for glue, and test set scores for discoeval. ## results we compare our slm model with the original bert, xlnet @xcite , electra @xcite , ernie 2.0 @xcite , conpono @xcite , bart @xcite and t5 which are the current state-of-theart for the benchmark datasets we use. we mainly compare with models using bert base hyperparameters. ## related work contextualized representation learning for nlp: self-supervised representation learning became popular after contextualization methods were introduced. elmo @xcite dynamically contextualizes representations of adjacent words using bi-directional recurrent encoders. bert @xcite adopted a deep transformer encoder and has proposed the masked language modeling objective incorporating a bi-directional context. after the success of bert, researchers started exploring various new pre-training objectives such as span boundary representations @xcite , reordering of local permutations @xcite , detecting incorrectly replaced tokens @xcite , combining multiple tasks @xcite , a decoder-based masked word prediction @xcite , and so on. another line of works tried to scale up the model with more parameters, training data, and computation resources. roberta @xcite trains bert large model with 10x more data for 16x more iterations. albert @xcite uses the xxlarge model whose parameters are reduced by weight sharing and factorizations. t5 proposes a multi-task encoder-decoder architecture that uses up-to about 33x weight parameters, 37x data size compared to the original bert large model. sentence representation models: several works tried to learn sentence representations using adja-cent sentences, especially in recurrent networks. skipthoughts @xcite and fastsent @xcite proposed an encoder-decoder architecture that encodes a sentence and generates the next and previous sentences. these works focus on independent representations of single independent sentences and do not consider the dynamic contextualization using neighboring sentences. we believe that, similar to words, sentence representations should also be properly contextualized in order to embed richer meaning including relationships to other sentences. hlstm @xcite considered contextualization of sentence representations by incorporating previous step's sentence representation for word prediction. hibert @xcite proposed a hierarchical transformer encoder trained by recovering masked sentences, focusing in particular on summarization. however, both of these models are trained by performing prediction at the word level, wheres our sentence unshuffling approach demands fine understanding of the relations among the sentences at the more global discourse level. learning from sentence ordering: there are prior works about learning contextualized sentence representations by recovering the original sequence of sentences. @xcite and @xcite proposed rnn-based pointer networks for reconstructing the sequence order. however, they utilized hierarchical models that encode each sentence separately without any access to other sentences. this fact not only restricts long-term contextualization of words and sentences but also limits downstream tasks due to the dedicated architectures. on the other hand, our method encodes multiple sentence representations at the same time with both inner and inter-sentence contextualization. the effect of predicting textual segment order in pretrained language models has been widely investigated as well. bert proposed the next sentence prediction task (nsp) which predicts whether two given text segments are from the same documents or not. the results of spanbert @xcite question the value of nsp, suggesting this might be due to noise from merging 2 unrelated texts from different documents. consequently, albert @xcite and structbert @xcite added sentence ordering objectives by predicting the order of text segments. bart @xcite proposed the sentence permutation task which is similar with ours. it predicts the original sequence of sentences using an auto-regressive decoder, which reconstructs the whole sentences by word prediction. however, their approach does not provide any representation of each sentence. moreover, while slm shows strong task improvements, that is not the case for these models. ## conclusions in this paper, we proposed the sentence-level language modeling objective for contextualized sentence representation learning. our approach extends a word-level language modeling strategy to the sentence-level by reconstructing the original order of shuffled sentences. in addition, we designed a special sequence reconstructor (sr) module to learn to perform the sentence re-ordering. it reconstructs the original order by pointing to the next sentence among encoded sentence representations using a pointer network and a transformer decoder. we evaluated the effect of the proposed idea on three benchmarks, glue, squad, and discoeval, and showed consistent improvements over the previous approaches. we matched performance with the state-of-the-art model with the same model size using much fewer parameters, computation, and data. through a qualitative analysis, we showed that our model can embed not only semantic but also relational features of sentences. we are excited about future work that could extend our motivation and further aim at incorporating stronger hierarchy into the language model architectures and the pre-training tasks.