Evaluating Embedding API s for Information Retrieval (2023)

Abstract: The ever-increasing size of language models curtails their widespread access to the community, thereby galvanizing many companies and startups into offering access to large language models through APIs. One particular API, suitable for dense retrieval, is the semantic embedding API that builds vector representations of a given text. With a growing number of APIs at our disposal, in this paper, our goal is to analyze semantic embedding APIs in realistic retrieval scenarios in order to assist practitioners and researchers in finding suitable services according to their needs. Specifically, we wish to investigate the capabilities of existing APIs on domain generalization and multilingual retrieval. For this purpose, we evaluate the embedding APIs on two standard benchmarks, BEIR, and MIRACL. We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective on English, in contrast to the standard practice, i.e., employing them as first-stage retrievers. For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best albeit at a higher cost. We hope our work lays the groundwork for thoroughly evaluating APIs that are critical in search and more broadly, in information retrieval.

## introduction language models (lms), pre-trained on a massive amount of text, power dense retrieval models in ad hoc retrieval @xcite . dense retrievers @xcite @xcite @xcite ; inter alia) essentially measure relevance via similarity between the representations of documents and queries. as lms are rapidly scaling up to gigantic models @xcite @xcite @xcite , their use as the backbone of dense retrieval models has become limited primarily because large language models (llms) are computationally expensive and deploying them on commodity hardware is cumbersome and often impractical. to alleviate this problem, many companies, e.g., openai, and cohere, set out to offer access to their proprietary llms through a family of apis. for dense retrieval, semantic embedding apis are designed to provide llm representations for queries as well as documents. these apis are especially appealing in the ir ecosystem because they afford practitioners and researchers the benefit of scale and allow for wider outreach of llms in ir. however, although nowadays, the surge of companies offering such apis with various model sizes has given us more options, a lack of thorough analysis of these apis has made it more difficult to determine one's best option for a particular use-case. besides, llm-based apis are often expensive and experimenting with all of them to determine the most suitable is prohibitively costly. in this paper, we analyze embedding apis for various realistic scenarios in ad hoc retrieval. to this end, we select three embedding apis available on the market, i.e., openai, cohere, and aleph-alpha, and assess their usability and effectiveness on two crucial directions that stand at the core of most ir applications. first, we study domain generalization where retrieval is conducted over collections drawn from a broad range of domains. understanding for which domains embedding apis work well or poorly elucidates their limitations while setting the stage for their wide adoption in their successful domains. we leverage the widely adopted beir benchmark @xcite for this purpose. on beir, we use the apis as re-rankers on top of bm25 retrieved documents because the large size of document collections in beir makes full-ranking (i.e., first-stage retrieval) via the apis impractical. our results show that embedding apis are reasonably effective re-rankers in most domains, suggesting that re-ranking is not only budget-friendly, but also is effective. however, we find that on datasets collected via lexical matching, they struggle. in particular, bm25 outperforms the full-fledged embedding apis on bioasq (bio-medical retrieval) and signal1m (tweet retrieval). we also explore the capabilities of embedding apis in multilingual retrieval where they are tested in several non-english languages, ranging from low-resource to high-resource languages. more precisely, we use miracl @xcite , a large-scale multilingual retrieval benchmark that spans 18 diverse languages. the manageable size of the corpora allow us to evaluate the apis as full-rankers as well as re-rankers. we find that the winning recipe for non-english retrieval is not re-ranking, unlike retrieval on english documents. instead, building hybrid models with bm25 yields the best results. our experiments also indicate that the apis are powerful for low-resource languages, whereas on high-resource languages, open-source models work better. overall, our findings offer insights on using embedding apis in real-world scenarios through two crucial aspects of ir systems. in summary, our key contributions are: we hope our work lays the groundwork for thoroughly evaluating apis that are critical in search and more broadly, for information access. embedding apis largely follow a similar strategy to generate sentence representations @xcite . dense retrieval. while the paradigm has been around for a long time @xcite , the emergence of pre-trained lms brought dense retrieval @xcite to the mainstream in ir. recent dense retrieval models adopt a bi-encoder architecture and generally use contrastive learning to distinguish relevant documents from non-relevant ones @xcite , similar to sentence embedding models. lms are shown to be an effective source to extract representations @xcite @xcite @xcite . this essentially means that with lms as the backbones and analogous objectives, dense retrievers and sentence embedding models have become indistinguishable in practice. ## apis semantic embedding apis are generally based on the so-called bi-encoder architecture, where queries and documents are fed to a fine-tuned lm in parallel @xcite @xcite . the key ingredient of bi-encoders is contrastive learning, whose objective is to enable models to distinguish relevant documents from non-relevant ones. in our experiments, we adopt the following semantic embedding apis, presented alphabetically: foot_0 aleph-alpha: this company has trained a family of multilingual lms, named luminous, 2 with three flavours in size, base (13b), extended (30b), and supreme (70b). the luminous models support five high-resource languages: english, french, german, italian, and spanish. however, no information is available about the data on which these lms are trained. we used luminous base that projects text into 5120-dimension embedding vectors. cohere: this company offers lms for producing semantic representations in two sizes: small (410m) and large (6b), generating 1024dimension and 4096-dimension embedding vectors, respectively. models are accompanied by model cards @xcite . 3 cohere also provides a multilingual model, multilingual-22-12, foot_3 that is trained on a large multilingual collection comprising 100+ languages. the data consists of 1.4 billion question/answer pairs mined from the web. the multilingual model maps text into 768dimension embedding vectors. openai: the company behind the gpt models @xcite @xcite also offers an embedding service. we use the recommended second-generation model, text-embedding-ada-002 @xcite that embeds text into a vector of 1536 dimensions. the model, initialized from a pretrained gpt model, is fine-tuned on naturally occurring paired data with no explicit labels, mainly scraped from the web, using contrastive learning with in-batch negatives. all the apis described above use transformerbased language models @xcite , but differ from each other in various ways: these distinctions may potentially lead to substantial differences in the overall effectiveness of the embedding apis. nevertheless, due to the nondisclosure of several details by the api providers, it remains challenging to identify the specific factors that contribute to the strengths and weaknesses of embedding models. yet, as the number of such apis continues to grow, we believe that high-level comparisons on standard benchmarks can provide valuable insights into how well these models operate under various practical scenarios. for practitioners building systems on top of these apis, this comparative analysis is useful as they are primarily interested in the end-to-end effectiveness and performance of these apis and are not typically concerned with their minutiae. ## experiments in this section, our main goal is to evaluate embedding apis in two real-world scenarios that often arise in ir applications: domain generalization and multilingual retrieval. ## conclusion the incredible capabilities of transformer-based language models at scale have attracted a handful of companies to offer access to their proprietary llms via apis. in this paper, we aim to qualitatively and quantitatively examine semantic embedding apis that can be used for information retrieval. our primary focus is to assess existing apis for domain generalization and multilingual retrieval. our findings suggest that re-ranking bm25 results is a suitable and cost-effective option for english; on the beir benchmark, openai ada2 performs the best on average. in multilingual settings, while re-ranking remains a viable technique, a hybrid approach produces the most favorable results. we hope that our insights aid practitioners and researchers in selecting appropriate apis based on their needs in this rapidly growing market.