Cross-Thought for Sentence Encoder Pre-training (2020)

Abstract: In this paper, we propose Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering. Instead of using the original signals of full sentences, we train a Transformer-based sequence encoder over a large set of short sequences, which allows the model to automatically select the most useful information for predicting masked words. Experiments on question answering and textual entailment tasks demonstrate that our pre-trained encoder can outperform state-of-the-art encoders trained with continuous sentence signals as well as traditional masked language modeling baselines. Our proposed approach also achieves new state of the art on HotpotQA (full-wiki setting) by improving intermediate information retrieval performance.

## introduction encoding sentences into embeddings @xcite @xcite is a critical step in many natural language processing (nlp) tasks. the benefit of using sentence embeddings is that the representations of all the encoded sentences can be reused on a chunk level (compared to word-level embeddings), which can significantly accelerate inference speed. for example, when used in question answering (qa), it can significantly shorten inference time with all the embeddings of candidate paragraphs pre-cached into memory and only matched with the question embedding during inference. there have been several models specifically designed to pre-train sentence encoders with largescale unlabeled corpus. for example, skipthought @xcite uses encoded sentence embeddings to generate the next sentence (figure 2(a) ). inverse cloze task @xcite defines some pseudo labels to pre-train a sentence encoder (figure 2(b) ). however, pseudo labels may bear low accuracy, and rich linguistic information that can be well learned in generic language modeling is often lost in these unsupervised methods. in this paper, we propose a novel unsupervised approach that fully exploits the strength of language modeling for sentence encoder pre-training. popular pre-training tasks such as language modeling @xcite , masked language modeling @xcite and sequence generation @xcite are not directly applicable to sentence encoder training, because only the hidden state of the first token (a special token) @xcite can be used as the sentence embedding, but no loss or gradient is specifically designed for the first special token, which renders sentence embeddings learned in such settings contain limited useful information. another limitation in existing masked language modeling methods @xcite ) is that they focus on long sequences (512 words), where masked tokens can be recovered by considering the context within the same sequence. this is useful for word dependency learning within a sequence, but less effective for sentence embedding learning. in this paper, we propose cross-thought, which segments input text into shorter sequences, where masked words in one sequence are less likely to be recovered based on the current sequence itself, but more relying on the embeddings of other surrounding sequences. for example, in figure 1 , the masked words "george washington" and "united states" in the third sequence can only be correctly predicted by considering the context from the first sequence. thus, instead of performing selfattention over all the words in all sentences, our proposed pre-training method enforces the model to learn from mutually-relevant sequences and automatically select the most relevant neighbors for masked words recovery. the proposed cross-thought architecture is illustrated in figure 2(c ). specifically, we preappend each sequence with multiple special tokens, the final hidden states of which are used as the final sentence embedding. then, we train multiple cross-sequence transformers over the hidden states of different special tokens independently, to retrieve relevant sequence embeddings for masked words prediction. after pre-training, the attention weights in the cross-sequence transformers can be directly applied to downstream tasks (e.g., in qa tasks, similarity scores between question and candidate answers can be ranked by their respective sentence embeddings). ## related work sequence encoder many studies have explored different ways to improve sequence embeddings. @xcite proposes deep structured semantic encoders for web search. @xcite uses lstm as the encoder for non-factoid answer selection, and @xcite proposes tree-lstm to compute semantic relatedness between sentences. @xcite also uses tree-based cnn as the encoder for textual entailment tasks. @xcite proposes long short-term memory-networks (lstmn) for inferring the relation between sentences, and @xcite combines lstm and self-attention mechanism to improve sentence embeddings. multi-task learning @xcite has also been applied for training better sentence embeddings. recently, in additional to supervised learning, models pre-trained with unsupervised methods begin to dominate the field. pre-training several methods have been proposed to directly pre-train sentence embedding, such as skip-thought @xcite , fast-sent @xcite , and inverse cloze task @xcite . although these methods can obtain better sentence embeddings in an unsupervised way, they cannot achieve state-of-the-art performance in downstream tasks even with further finetuning. @xcite proposes to pre-train lstm with language modeling (lm) task, and @xcite pre-trains transformer also with lm. instead of sequentially generating words in a single direction, @xcite proposes the masked language modeling task to pre-train bidirectional transformer. most recently, @xcite ; @xcite propose to jointly train sentence-embedding-based information retriever and transformer to re-construct documents. however, their methods are usually difficult to train with reinforcement learning methods involved, and need to periodically re-index the whole corpus such as wikipedia. in this paper, to pre-train sentence encoder, we propose a new model cross-thought to recover the masked information across sequences. we make use of the heuristics that nearby sequences in the document contain the most important information to recover the masked words. therefore, the challenging retrieval part can be replaced by soft-attention mechanism, making our model much easier to train. ## cross-thought in this section, we introduce our proposed pretraining model cross-thought, and describe how to finetune the pre-trained model on downstream tasks. specifically, most parameters in downstream tasks can be initialized by the pre-trained cross-thought, and for certain tasks (e.g., ranking) the attention weights across sequences can be directly used without additional parameters (figure 3 ). ## experiments in this section, we conduct experiments based on our pre-trained models, and provide additional detailed analysis. ## conclusion we propose a novel approach, cross-thought, to pre-train sentence encoder. experiments demonstrate that using cross-thought trained with short sequences can effectively improve sentence embedding. our pre-trained sentence encoder with further finetuning can beat several strong baselines on many nlp tasks.