Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation (2025)

Abstract: Context-aware embedding methods boost retrieval accuracy by conditioning on corpus statistics (e.g., term co-occurrence and topical patterns) extracted from neighboring documents. However, this context-aware approach requires access to the target corpus or requires domain-specific finetuning, posing practical barriers in privacy-sensitive or resource-constrained settings. We present ZEST, a zero-shot contextual adaptation framework that replaces real corpus access with a one-time offline synthesis of a compact proxy. Given only a handful of exemplar documents representative of the general target domain, we use a multi-step hierarchical procedure to generate a synthetic context corpus of several hundred documents that aims to emulate key domain-specific distributions. At inference, the frozen context-aware encoder uses this proxy corpus – without any finetuning or target corpus access – to produce domain-adapted embeddings. Across the MTEB benchmark, ZEST’s zero-shot synthetic context adaptation using only five example documents performs within 0.5% of models leveraging full target corpus access – demonstrating remarkable efficacy without any retraining. ZEST thus provides a practical method for deploying high-performance, adaptable embeddings in constrained environments.

## introduction effective neural information retrieval relies heavily on high-quality dense vector representations for documents and queries @xcite . however, standard embedding approaches typically generate these representations without dynamically incorporating information from the specific target corpus being searched @xcite . this lack of dynamic context sensitivity limits their adaptability and retrieval performance, especially when deployed in domains that differ from their pretrain-ing data @xcite , a limitation partially addressed by traditional methods leveraging corpus statistics @xcite . however, such lexical methods based on surface counts have largely been superseded, as their bagof-words nature fundamentally fails to capture semantic meaning. to provide neural embeddings with sensitivity to the target corpus, context-aware architectures have recently emerged @xcite . such models utilize multi-stage processing, where the final embedding is conditioned on representations derived from neighboring documents within the target corpus. although these methods significantly enhance retrieval by tailoring embeddings to domain-specific characteristics, their requirement to access the target corpus during inference is a critical limitation. practical constraints related to data privacy or corpus scale often make such corpus access infeasible; for instance, sensitive medical documents may not be exposed during inference. this critical limitation motivates the search for alternative ways to provide essential domain signals to context-aware models. while large language models (llms) possess good generative capabilities for retrieval data @xcite , their effective use in this zero-access environment is nontrivial. the core challenge is not merely generating domain-relevant text, but ensuring that any llmderived output possesses an appropriate degree of representational fidelity. in this work, to overcome this critical barrier, we introduce zero-shot embeddings via synthetic context (zest), enabling effective contextual adaptation without accessing the target corpus. zest, depicted in figure 1 , operates in two phases. first, during offline synthesis, a llm is employed to hierarchically generate a compact set of domain anchors from a minimal set of example documents randomly chosen from a domain-relevant source. a frozen, pretrained context-aware model (m 1 , m 2 ) embeds new documents or queries (d/q) by conditioning on d synth (with its document representations pre-processed via m 1 ) instead of the inaccessible target corpus d. this provides domain-adapted embeddings without requiring model retraining or direct corpus access. ## background this section first revisits dense neural retrieval, then describes the class of context-aware embedding architectures most relevant to our work, and finally formalizes the zero-shot deployment setting that zest is designed to address. dense retrieval. given a query q and a corpus d, dense vector retrieval methods learn two neural encoders ϕ : @xmath0: @xmath1, q) = ϕ(d) • ψ(q). training typically relies on a contrastive loss that favours high similarity for relevant pairs (q, d + ) and low similarity for negatives (q, d -): context-aware embedding architectures. recent retrievers mitigate the limitation of contextagnostic embeddings by incorporating information from the target corpus at inference time. these approaches vary in their mechanism. one prominent strategy, which our work adapts for zero-shot settings, involves a multi-stage architecture exemplified by contextual document embeddings (cde) @xcite . here, a firststage encoder m 1 processes a sample of j context documents {d 1 , . . . , d j } ⊂ d from the target corpus. a second-stage encoder m 2 then conditions the final embedding of a target text (document d or query q) on these context representations alongside the text's own token sequence. formally, the first-stage encoder m 1 maps an input document to a vector in r h . the second-stage encoder is m 2 : r j×h × r t ×@xmath2, so that, given a document d whose tokens are embedded as e(d) = e(w 1 ), . . . , e(w t ) , the contextual document representation is a single dense vector: this differs from unsupervised adaptation methods such as gpl @xcite and boot&switch @xcite , which require corpus access or additional tuning. it also contrasts with late-interaction systems such as col-bert @xcite , which, while domain-robust, incur higher online computational costs due to their token-level interaction mechanisms at query time, unlike the single-vector representations produced by cde. by explicitly exposing corpus statistics (such as term frequency or topical patterns via neighbor documents) to the embedding generation process, cde-style models improve robustness across domains. however, their reliance on accessing the target corpus d at inference time remains a significant practical hurdle. our work focuses on overcoming the corpus access requirement. problem setting: zero-shot contextual adaptation. we consider deploying a frozen contextaware model (m 1 , m 2 ) in a new domain where the full corpus d is inaccessible. instead, in lieu of supplying documents from the target corpus, the practitioner can supply only a small exemplar set d @xmath3, . . . , d k ex } that typifies the general domain. the challenge is to generate domainsensitive embeddings ϕ(d; •) and ψ(q; •) without accessing d and without any parameter updates. the remainder of this paper presents zest, a solution to this problem. ## method: zero-shot embeddings via synthetic context zest enables corpus-aware adaptation without direct corpus access by substituting the real neighbor set required by context-aware models with a compact, llm-generated proxy corpus d synth . this involves a one-time offline synthesis phase and an online inference phase using the fixed proxy. ## experimental setup this section details the experimental protocol designed to evaluate the effectiveness of zest in realistic zero-shot domain adaptation scenarios using established retrieval benchmarks. ## results and discussion this section presents the empirical evaluation of zest, demonstrating its ability to achieve effective contextual adaptation in zero-shot scenarios. we analyze its performance against established baselines, investigate the impact of key hyperparameters through ablation studies, and discuss the implications of our findings. ## conclusion we introduced zest, a novel method that enables context-aware document retrieval adaptation without requiring access to the target corpus. by employing llms guided by few-shot examples to synthesize a representative context corpus offline, zest allows a pretrained contextual embedding model to adapt effectively during online inference. our findings show zest substantially improves zero-shot retrieval over context-agnostic methods, nearing the performance of models with full corpus access. zest addresses a critical practical limitation of context-aware models, offering a viable path toward more adaptable and effective document embeddings in scenarios constrained by corpus access, privacy, or scale. this data-centric adaptation strategy opens new possibilities for deploying sophisticated retrieval models in challenging real-world environments.