Language Models are Universal Embedders (2025)

Abstract: In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters. As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario. In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option. To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark. Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks, even extending to languages and tasks for which no finetuning/pretraining data is available. We also present detailed analyses and additional evaluations. We hope that this work could encourage the development of more robust open-source universal embedders.

## introduction embeddings, which transform discrete text or code sequences into continuous vectors, are widely used in many fields @xcite . they have recently gained broader attention by manipulating knowledge and memories for large language models (llms) and llm-based agents @xcite @xcite . in such scenarios, their usages are inevitably coupled with different languages and tasks. this brings a demand for robust and universal embedders, where one single model can be applied across diverse tasks and languages, encompassing both natural and programming languages. the common approach to building effective embedders is finetuning pretrained language models a possible solution is to use multilingual large language models (mllm), such as bloom @xcite series. these models adopt a decoderonly architecture and are pretrained on meticulously curated, large-scale, multilingual corpora, roots @xcite , by the next token prediction objective. they are not only skilled in english but also excel in other languages, including natural ones such as chinese and programming languages like python, showing their wide-ranging language abilities. therefore, one major question arises: is it feasible to derive universal embedders from mllms? to study this inquiry, we examine two scenarios: (1) task versatility: we explore strategies of data compositions that enable the model to adapt effectively to a variety of embedding tasks. (2) multilinguality: we investigate the process of obtaining em-beddings across multiple languages using limited data, especially considering that some of them are hard to acquire suitable training data. by synthesizing insights from above cases, we evaluate whether mllms can be trained to generate high-quality embeddings across both languages and tasks. in practice, we construct embedders by conventional methods (detailed in §2.1) based on bloom @xcite models. 1 for task versatility, in line with prior works @xcite , we categorize all embedding tasks into symmetric and asymmetric types and combine datasets from both sides for training ( §2.3). regarding multilinguality, we employ parameter-efficient fine-tuning to maximally preserve the modeling abilities of various languages ( §2.2). for evaluation, we select 5 languages (4 natural, 1 programming) and compile a universal embedding benchmark ( §3.1). all models are trained with monolingual data and evaluated on the benchmark (as shown in figure 1 ), which helps us to analyze the performance of different languages, e.g., densely, lessly or not pretrained ones, more effectively. through extensive experiments, we find that: additionally, we provide various detailed analyses ( §3.3, §3.4, §4), e.g., scaling the model size, and the model performance in additional benchmarks such as mteb @xcite and codesearchnet @xcite , to better understand the model behaviors. we hope that our findings could foster the development and research of more powerful universal embedders. ## method figure 2 shows our method and evaluation. for clarity, the details of embedding model are not presented. next, we describe this model design. ## main experiments to assess the viability of converting mllms into universal embedding models, we conduct two parts of experiment. the first part aims to evaluate the potential of the lms and validate employed strategies on the compiled benchmark ( §3.1). we expand to broader open evaluations in the second part ( §4). ## extended evaluations the second part experiment consists of evaluations on more tasks and domains ( §4.1), as well as diverse languages of multilingual ( §4.2) and crosslingual ( §4.3) tests. we evaluate bloom models (1b1, 3b, 7b1) finetuned on english data. ## related work text and sentence embeddings are useful for many downstream tasks and applications @xcite . early studies start from similar ideas of word vectors @xcite @xcite , also shift to neural networks @xcite then pre-trained transformers @xcite @xcite . the subsequent work mainly focus on using contrastive loss to supervise or improve representation learning @xcite @xcite @xcite , translation augmentation @xcite , large-scale pre-training @xcite @xcite , and prompt @xcite . as most of them are under specific tasks, @xcite compile mteb with diverse tasks, domains, and languages for evaluations. recently, embeddings have gained attention and a batch of large-scale pretrained models have emerged, such as e5 @xcite , bge @xcite , gte @xcite , uae @xcite . most of them are targeted to and evaluated on english, while we explore the languages beyond english. pre-trained transformer encoders, i.e., bert @xcite , or that of t5 @xcite are currently the mainstream for embedding models, which are computation-effective than encoder-decoders @xcite . gpt-style decoder-only models @xcite are promising alternatives, since they have theoretically stronger representations @xcite . pioneering gpt-based studies show impressive performance on both text and code @xcite , especially for semantic search @xcite . we continue this line, exploring the unified embeddings across multiple natural and programming languages. a concurrent work @xcite fine-tune mistrial-7b @xcite with data from diverse source and carefully crafted instructions, showing state-ofthe-art performance on english mteb. taking into account a more general scenario with various languages, we do not use complex prompts, but only a set of special symbols for asymmetric inputs. multi-and cross-lingual text embeddings follow the developments of english ones, from crosslingual word embeddings @xcite to rnns @xcite and transformers @xcite @xcite . to learn models without enough supervisions, translation information @xcite @xcite and multilingual pretrained encoders @xcite are explorated to improve embeddings @xcite . however, such bertlike multilingual encoders do not support code, which is currently one of the crucial requirements. therefore, we shift our focus to pre-training decoder models that can simultaneously support natural languages and programming languages, aiming to evaluate and analyze the potential of constructing universal embeddings from them. ## conclusion we propose the development of unified embeddings models (universal embedders) for various tasks across multiple natural and programming languages based on multilingual decoder-only models. to evaluate the potential, we present straightforward strategies to construct embedding models from them, and design a universal embedding benchmark for evaluation and analysis. through extensive experiments, we demonstrated the ver-satility of embedders constructed from mllms, showing their capabilities cross languages and tasks. the models can generate reasonably good embeddings for languages that have not been finetuned or pre-trained, and the quality can be significantly improved with the corresponding fine-tuning data. these characteristics strongly indicate the great potential of mlm for building universal embedders. additionally, we provide various analyses and extended evaluations to reveal the interesting properties of the model. we hope that our work could inspire more open-source high-quality universal embedders.