Preserving Multilingual Quality While Tuning Query Encoder on E nglish Only (2025)

Abstract: A query encoder of a dual passage retrieval system can be tuned for specific types of queries or domains, while the precomputed and stored documents representations are kept intact. Switching from one query encoder to another when needed is easily feasible, unlike overhauling the embeddings of a whole knowledge base. In this work we raise a question: Can the generic, original qualities of the encoder be preserved or at least left not too degraded when it is tuned on a narrow domain? We conducted experiments on a high quality multilingual embedding model: Tuning it on a single English-only dataset, we observe that the tuning not only preserves the multilingual qualities, but even improves them. The embedding qualities on distinctly different data are also improved or at least preserved. Drawing on our observations, we suggest a more general hypothesis: Tuning with intentionally low learning rate can preserve or improve a system’s properties acquired in training, but not specifically targeted by tuning. We call this adiabatic tuning and provide tentative explanations.

## introduction advances in neural nlp methods have resulted in high quality dense vector text representations @xcite @xcite . such representations are often used at the initial stages of an information retrieval system, selecting the most relevant documents, ranked relative to the query @xcite @xcite . a dual encoder is successfully used to train the representations @xcite @xcite @xcite . a dual encoder dense passage retrieval system is efficient for two main reasons: (1) it allows using the simple inner product of query and document representations, and (2) it allows modifying the query representation for a task or domain, while keeping the stored and precomputed (query-invariant) document representations intact. if the representation was pretrained in a multilingual setting, tuning on english-only samples may be expected to degrade the multilingual qualities and there may not be enough cross-lingual samples for tuning on a specific domain or types of queries. a multilingual query generator may be employed to overcome a shortage of cross-lingual data @xcite , but, in this work, we follow an arguably simpler strategy. in order to understand the effect of english-only tuning on multilingual qualities of a representation, and to assess a possible degradation, we consider a simple setup: a state of the art multilingual embedding model is taken as the starting point, and fine-tuned by english only samples as the query representation part of a dual encoder. we assume that our observations of the degradation or preservation of the multilingual qualities may be generalized to other pretrained system qualities that are not directly targeted in tuning. in order to obtain preliminary confirmation of this hypothesis, we also observe the effect of tuning on the embedding quality for queries and text chunks of very different styles, the likes of which could be present in the training of the original encoder, but certainly not targeted in tuning. our contribution: 1. we show that fine-tuning a query encoder on an english-only dataset may not only preserve multilingual qualities of query-document embeddings matching, but even improve them. 2. we hypothesize that a tuning regime with intentionally low learning rate (far below of what is necessary to avoid overfitting) preserves or improves the properties acquired in the training, but not targeted by tuning. we call this adiabatic tuning and suggest support-ing observations and conjectural explanations. 3. we add a dataset with graded difficulty, based on arxiv titles and abstracts. although high-resource languages can be used for cross-lingual transfer @xcite , our setting does not have such a goal: the tuning is set to improve the query part of a dual encoder on a certain dataset, with no driving mechanism for preserving or improving the other qualities of the system. our starting point is one of the best (for its lean size) multilingual embedding models which differs from starting with a multilingual language model and then aligning the generated embeddings for different languages @xcite . ## conclusion we considered tuning the query part of a dual encoder starting from a high quality multilingual embedding model, and using english-only samples in the tuning. we found that multilingual qualities are quite stable in many scenarios of the tuning, and can be not only preserved but improved. we explain this by speculating that most of the transformer, except the embedding block, depends weakly on multiple languages. we think of this as a particular case of a general pattern: tuning a certain model quality, if done carefully enough (adiabatic tuning), can also retain or even improve the related (but not targeted by tuning) qualities. this allows a resource-light adjustment of multilingual embeddings for a specific query type or domain, even a narrow domain (appendix k).