A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024)

Abstract: Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. They capture the meaning of a sentence, enabling machines to understand and reason over human language. In recent years, significant progress has been made in developing methods for learning sentence representations, including unsupervised, supervised, and transfer learning approaches. However there is no literature review on sentence representations till now. In this paper, we provide an overview of the different methods for sentence representation learning, focusing mostly on deep learning models. We provide a systematic organization of the literature, highlighting the key contributions and challenges in this area. Overall, our review highlights the importance of this area in natural language processing, the progress made in sentence representation learning, and the challenges that remain. We conclude with directions for future research, suggesting potential avenues for improving the quality and efficiency of sentence representations.

## introduction the sentence, together with the word, are the two fundamental grammatical units of human language. representing sentences for machine learning, which involves transforming a sentence into a vector or a fixed-length representation, is an important component of nlp. the quality of these representations affects the performance of downstream nlp tasks like text classification and text similarity @xcite . deep neural networks have played a major role in obtaining sentence representations. while there have been significant advancements in the development of large language models (llms) such as gpt-3 @xcite , bloom @xcite , they learn through effective word representations and modelling of the language at the (next) word level. endowing models with the ability to learn effective representations of higher linguistic units beyond words -such as sentences -is useful. for instance, sentence representations can help in retrieving semantically similar documents prior to generation. langchain foot_0 and various other frameworks like dspy @xcite , have underscored the critical demand for proficient sentence representations. the documents retrieved serve as valuable resources for generating fact-based responses, using custom documents to address user queries, and fulfilling other essential functions. however, current language models exhibit drawbacks in obtaining sentence representations out-ofthe-box. for instance, @xcite showed that out-of-the-box representations from bert @xcite are fraught with problems such as anisotropy-representations occupying a narrow cone, making every representation closer to all others. also, they are impractical for real-life applications: finding the best match for a query takes hours @xcite . to overcome the inadequacy of directly using sentence representations from language models, numerous methods have been developed. several works have proposed to post-process the representations from bert to alleviate the anisotropy @xcite or repurpose representations from different layers of the model @xcite . but there has been a steadily growing body of works that move away from such postprocessing and introduce new methods. perhaps due to the rapid advancements in the field (figure 1 ), there are no literature reviews discussing the diverse range of techniques for learning sentence representations. the present paper offers a review of these techniques, with a specific emphasis on deep learning methods. our review caters to two audiences: (a) researchers from various fields seeking to get insights into recent breakthroughs in sentence representations, and (b) researchers aiming to advance the field of sentence representations. ## supervised sentence representations natural language understanding involves intricate reasoning. one way to learn better sentence representations is by excelling at tasks that demand reasoning. large-scale supervised datasets for natural language understanding have emerged over the years: snli @xcite , mnli @xcite , anli @xcite . to that end, neural network methods utilize supervised datasets to learn sentence representations. ## unsupervised sentence representations unlike supervised methods, unsupervised learning techniques do not rely on explicit positive and negative examples but instead employ alternative techniques to mine them. hence this has garnered significant attention in recent years. additionally, they may also modify the learning objectives. ## trends & analysis limited advantages of supervision: ## challenges practical applications and the rise of tools: sentence representations are commonly employed for sentence retrieval in practical applications, as evidenced by the increasing number of benchmarks @xcite . however, their utility extends beyond retrieval, as demonstrated by recent work @xcite , which leverages sentence representations for identifying documents that share a similar stance on a topic and for isolating documents that diverge from the consensus. the increasing use of sentence representations in practical applications such as retrieval and providing an appropriate context to generative language models to rely on has lead to the rise of tools known as vector databases. these tools enable storing vectors as indices and include algorithms for fast retrieval of similar vectors. popular options such as pinecone foot_1 and milvus foot_2 also offer services for cloud hosting and resilience. these vector databases can be integrated with other frameworks such as langchain, that facilitate the development of llm applications. adapting to different domains: research has shown that sentence representations learned in one domain may not accurately capture the semantic meaning of sentences in another domain @xcite . some solutions have been proposed in the literature, such as generating queries using a pretrained t5 model on a paragraph from the target domain, or using a pretrained cross-encoder to label the query and paragraph, or using a denoising objective @xcite . nonetheless, training models that work well across domains remains challenging. cross-lingual sentence representations: creating sentence representations that can be used across languages, especially those with limited annotated data, poses a significant challenge @xcite . new solutions for cross-lingual retrieval are being developed and deployed for real-world use cases. 4 many scholarly works @xcite @xcite have addressed cross-lingual sentence representation learning in recent times, but they require aligned data between languages, which is hard to obtain. ## conclusions this survey offers an overview of sentence representations, presenting a taxonomy of methods. while major innovations focused on obtaining better quality data for contrastive learning, modern advances in generative technologies can accelerate the automatic generation of supervised data at low cost. although llms play a crucial role in informing the advancement of sentence representations, further enhancements in sentence representation learning are necessary to personalize current llms to achieve tailored results. we highlighted that better multilingual and multi-domain sentence representations are needed, now that llms are being deployed in different domains at a rapid pace. we hope that this survey can accelerate advances in sentence representation learning.