Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022)

Abstract: We provide the first exploration of sentence embeddings from text-to-text transformers (T5) including the effects of scaling up sentence encoders to 11B parameters. Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks, it is unclear how to produce sentence embeddings from encoder-decoder models. We investigate three methods to construct Sentence-T5 (ST5) models: two utilize only the T5 encoder and one using the full T5 encoder-decoder. We establish a new sentence representation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark. Our encoder-only models outperform the previous best models on both SentEval and SentGLUE transfer tasks, including semantic textual similarity (STS). Scaling up ST5 from millions to billions of parameters shown to consistently improve performance. Finally, our encoder-decoder method achieves a new state-of-the-art on STS when using sentence embeddings.

## introduction sentence embeddings providing compact meaning representations that are broadly useful for a variety of language processing tasks include classification, question-answering, semantic retrieval, bitext mining, and semantic similarity tasks. we explore sentence embeddings from a new family of pre-trained models: text-to-text transfer transformer (t5) @xcite . unlike encoder-only models, which use a transformer encoder to predict random masked tokens, t5 uses an encoder-decoder architecture and a generative span corruption pre-training task. t5 models can be scaled up to hundreds of billions of parameters 1 our models are released at https://tfhub.dev/ google/collections/sentence-t5/1 . transfer sts st5-encdec (11b params) 90.46 84.94 st5-enc (11b params) as shown in figure 2 , we explore three ways of turning a pre-trained t5 encoder-decoder model into a sentence embedding model: (i) using the first token representation of the encoder; (ii) averaging all token representations from the encoder; to our knowledge, we are the first to study using large-scale pre-trained text-to-text models for sentence representation learning and to scale sentence embedding models up to 11 billion parameters. we summarize our contributions as follows: (i) even without fine-tuning, encoder-only st5 models perform well on sentence transfer tasks, outperforming state-of-the-art fine-tuned models such as sim-cse bert and simcse roberta @xcite ; (ii) encoder-decoder sentence embedding models achieve strong performance on sts, establishing a new state-of-the-art on sentence embedding based sts; (iii) contrastive learning is effective for fine-tuning sentence encoders from t5-style pre-trained models, particularly using our proposed two-stage contrastive learning approach; ## related work sentence embedding models have been trained using a variety of methods including: supervised natural language inference pairs (nli) @xcite @xcite ; conversational input-response and question-answer pairs @xcite ; translation pairs @xcite ; paraphrasing pairs @xcite and adjacent sentence pairs @xcite . @xcite achieved the previous state-of-the-art on sts with bert and roberta models by combining contrastive learning that constructs positive and negative sentence pairs using nli data. in parallel, text-to-text transfer transformers (t5) @xcite , as shown in figure 2a , are gaining popularity due to their competitive performance, effective scaling to larger model sizes, and ease of use in solving tasks as simple text-totext mapping problems. however, extracting high quality text embeddings from t5 has not been previously explored. moreover, while recent work has shown that scaling up models improves sentence embedding performance @xcite , the largest model sizes investigated only include 355 million parameters rather than the 11 billion parameters available in the largest t5 model. ## sentence-t5 (st5) we explore producing sentence embeddings from t5 models, ranging in size from 220 million to 11 billion parameters, both as raw sentence embeddings extracted from pretrained t5 models and using fine-tuning to refine the representations. ## experimental goals our experiments aim to answer the following: with these goals, we study transfer and sts performance of t5 sentence embeddings using a variety of model and training configurations, comparing st5 to state-of-the-art methods including sbert/sroberta @xcite and simcse @xcite . ## results ## scaling up t5 we leverage the existing checkpoints from large t5 models to study the effect of scaling sentence encoders. the parameters of the t5 models are listed in ## sentglue evaluation in this section, we introduce a new sentence representation transfer benchmark -sentglue -which extends the sentence evaluation toolkit, senteval, to nine tasks from the glue benchmark including: cola, sst-2, mrpc, sts-b, qqp, mnli-m, mnli-mm, qnli, rte 10 . the glue benchmark has been widely adopted for assessing language understanding models. glue tasks are either single sentence or sentence pair classification (e.g. nli) or similarity (sts) tasks. the best models on the glue leaderboard are fine-tuned cross-attention models like bert or t5. such models change all the parameters in the underlying model during finetuning and for pairwise tasks they allow for early fusion of input features from both sentences being 10 we found the wnli task from the glue benchmark is too challenge for existing sentence embedding models, thus we exclude it in the current version. compared. for sentglue, we introduce the constraint that each input needs to be independently encoded into a fixed embedding space representation that can then be feed to additional layers in order to make a prediction. we believe this best adapts the spirit of the original senteval benchmark for sentence embeddings to the glue tasks. from ## conclusion we obtaining sentence embeddings from t5, investigating three architectures and two-stage contrastive learning for fine-tuning our representations. we compare the difference between encoder-only and encoder-decoder methods and analyze their performance on downstream tasks. through extensive experiments on sts, senteval and glue tasks, we show that encoder-only models have strong transfer performance while encoder-decoder models perform better on sts tasks. we demonstrate the effectiveness of scaling up t5 models, greatly improving sentence embedding quality. these findings suggest that future improvements in the scale and quality of pre-trained t5 models may provide further sentence embeddings improvements.