L ong E mbed: Extending Embedding Models for Long Context Retrieval (2024)

Abstract: Embedding models play a pivotal role in modern NLP applications such as document retrieval. However, existing embedding models are limited to encoding short documents of typically 512 tokens, restrained from application scenarios requiring long inputs. This paper explores context window extension of existing embedding models, pushing their input length to a maximum of 32,768. We begin by evaluating the performance of existing embedding models using our newly constructed LongEmbed benchmark, which includes two synthetic and four real-world tasks, featuring documents of varying lengths and dispersed target information. The benchmarking results highlight huge opportunities for enhancement in current models. Via comprehensive experiments, we demonstrate that training-free context window extension strategies can effectively increase the input length of these models by several folds. Moreover, comparison of models using Absolute Position Encoding (APE) and Rotary Position Encoding (RoPE) reveals the superiority of RoPE-based embedding models in context window extension, offering empirical guidance for future models. Our benchmark, code and trained models will be released to advance the research in long context embedding models.

## related work text embedding models. text embeddings encode semantic information of text as lowdimensional vectors, enabling numerous nlp applications. early attempts on embeddings models include latent semantic indexing @xcite and weighted average of word embeddings @xcite . modern embedding models @xcite @xcite exploit supervision from labeled query-document pairs, adopting a multi-stage training paradigm that pre-trained on large-scale raw text pairs using contrastive loss, then fine-tuned on small scale but high-quality datasets. existing efforts in developing long-context embedding models typically involve first obtaining a long-context backbone model, either by pretraining with long inputs from scratch @xcite @xcite or using existing ones @xcite , followed by training the backbone model to produce embeddings. instead, this paper endows ex-isting embedding models with the ability to handle long context through context window extension. context window extension for llms. due to the high cost of pre-training an llm from scratch, there have been many efforts towards extending the context window of existing llms in a plug-andplay manner. we categorize these efforts as follows: 1) divide-and-conquer, which involves segmenting long inputs into short chunks, processing each chunk with the model, and aggregating the results, as demonstrated by pcw @xcite ; 2) position reorganization, which reorganizes position ids to accommodate longer inputs, as exemplified by selfextend @xcite , dca @xcite . 3) position interpolation, which introduces new position embeddings by interpolating existing ones, includes pi @xcite , ntk @xcite , yarn @xcite , and resonance rope @xcite . our paper thoroughly investigates these three lines of methods on embedding models. we also acknowledge other efforts in context extension, such as token compression @xcite @xcite and memory-based transformers @xcite . however, the former is not applicable for bidirectional attention, and the latter requires complex mechanisms for accessing encoded content, hence we do not experiment with these two categories. in addition to their plug-and-play usability, further fine-tuning on top of these methods with long training samples has been proven to yield better performance @xcite @xcite . addressing the overhead of training on long inputs and the scarcity of extremely long training data, a line of research investigates simulating long inputs within short context, including randomized positions @xcite , positional skip-wise (pose) training @xcite , and skipalign @xcite . this paper also leverage these efforts to synthesize long training samples from the original training data, facilitating further fine-tuning on top of plug-and-play methods. ## the longembed benchmark in this section, we first identify two limitations of existing retrieval benchmarks for evaluating longcontext capabilities ( § 3.1). then, we introduce the retrieval tasks adopted in our longembed, including both synthetic ones ( § 3.2) and real ones ( § 3.3). ## conclusion this paper explores context window extension of existing embedding models. through extensive experiments on our longembed benchmark, we show that training-free context window extension strategies can effectively increase the input length of these models by several folds. further, our analysis reveals the superiority of rope-based embedding models over ape-based ones in context window extension. hence, we advocate for the use of rope for future embedding models.