Examining Multilingual Embedding Models Cross-Lingually Through LLM -Generated Adversarial Examples (2025)

Abstract: The evaluation of cross-lingual semantic search models is often limited to existing datasets from tasks such as information retrieval and semantic textual similarity. We introduce Cross-Lingual Semantic Discrimination (CLSD), a lightweight evaluation task that requires only parallel sentences and a Large Language Model (LLM) to generate adversarial distractors. CLSD measures an embedding model’s ability to rank the true parallel sentence above semantically misleading but lexically similar alternatives. As a case study, we construct CLSD datasets for German–French in the news domain. Our experiments show that models fine-tuned for retrieval tasks benefit from pivoting through English, whereas bitext mining models perform best in direct cross-lingual settings. A fine-grained similarity analysis further reveals that embedding models differ in their sensitivity to linguistic perturbations.

## introduction when selecting a model for semantic search in a specific domain and language pair, standard embedding benchmarks such as (multilingual) mteb @xcite provide only partial coverage. we argue that evaluation should be carried out directly on the target language pair and domain of interest. however, the lack of suitable datasets makes it difficult to reach well-founded decisions about applications, such as choosing an embedding model or deciding whether to use english as a pivot language. we propose a simple and efficient method for dataset generation and evaluation that simulates real-world challenges of cross-lingual semantic retrieval in large text collections. our approach uses llms to create adversarial examples from parallel sentences, designed to test the cross-lingual embedding capabilities of multilingual models. concretely speaking, four distractor sentences are generated for each parallel target sentence. we require these sentences to be very similar to the original target sentence in terms of syntactic structure and surface word forms, but to be semantically dissimilar. our proposed task, cross-lingual semantic discrimination (clsd), involves identifying the correct target sentence from among four distractor sentences in the target language, given an original sentence in the source language. to better understand how these distractor modifications influence semantic similarity both crosslingually and monolingually, we perform a finegrained analysis and correlate the semantic changes with linguistic properties at the level of part-ofspeech tags. ## conclusion this paper advances cross-lingual semantic search by identifying gaps in current evaluation practices and introducing adversarial examples for testing multilingual embeddings. in a german-french case study, we contribute four new adversarial cross-lingual evaluation datasets in the news domain. our comparative experiments highlight that direct cross-lingual and pivot-through-english evaluation each have advantages depending on the embedding model. fine-grained analyses further demonstrate how generated distractors reveal differences in the cross-lingual capabilities of multilingual models and show how small perturbations can affect their semantic representations. future work should extend this approach to additional language pairs and explore more refined adversarial generation methods to improve the evaluation of semantic search models.