From Benchmark to Better Embeddings: Leveraging Synonym Substitution to Enhance Multimodal Models in U krainian (2025)

Abstract: We study the robustness of text–image retrieval for Ukrainian under synonym-substitution attacks (SSA). On Multi30K with OpenCLIP, we evaluate two SSA methods: dictionary-based and LLM-based, and find Ukrainian degrades far more than English (e.g., GPT-4o SSA drops HIT@1 from 32.1 \to 10.9 vs. 41.6 \to 30.4). We introduce a Hybrid method that filters dictionary candidates with an LLM to preserve sense and grammar, yielding higher-quality perturbations (Ukrainian HIT@1 16.8 vs. 7.6/10.9). To mitigate this problem, we propose synonym-augmented fine-tuning, injecting one-word substitutions into training; it boosts robustness (Hybrid 28.1, GPT-4o 25.1) without harming original performance. This is the first systematic SSA evaluation for Ukrainian multimodal retrieval and a practical recipe for improving models in low-resource, morphologically rich languages. We release code, prompts, and trained checkpoints at https://github.com/YuriiLaba/UA-B2BE.

## introduction embeddings provide vector-space representations of discrete inputs: words, sentences, images, and audio, making semantic similarity and downstream learning feasible @xcite . over the past decade, unimodal text embeddings have evolved into multimodal embeddings that jointly encode text and images into a shared space, enabling retrieval, grounding, and cross-modal reasoning @xcite . while this shift has powered significant advances in multimodal understanding, it also raises questions about the stability of these representations under minor, human-preserving edits to the input @xcite . a key vulnerability is robustness to local lexical variation @xcite . substitutions at the character, word, or sentence level * these authors contributed equally to this work can perturb an embedding in ways that mislead a model even when the change preserves meaning for humans. in this work, we focus on word-level synonym substitution: a setting that is intuitively harmless for people yet often misleads models, especially when tokenization, morphology, or pretraining distributions underrepresent particular lexical forms. most robustness studies of synonym substitution focus on english @xcite @xcite . for ukrainian, a morphologically rich, lower-resource language, both empirical measures and standardized evaluation tools are scarce, as it ranks in the lower tiers of data availability in cc100 @xcite and falls outside the high-resource classes in @xcite . this gap limits ability to diagnose failures and improve multilingual and multimodal systems. at the same time, many state-of-the-art (sota) multimodal encoders that produce embeddings are trained from scratch on web-scale english-centric corpora. multilingual coverage is frequently bootstrapped via machine translation @xcite . this introduces wellknown issues: translation artifacts, domain and register mismatches, and lexical biases that distort the target-language signal. we therefore frame the central problem: there is no reliable measure of synonym-robustness for ukrainian multimodal embeddings, and the ecosystem lacks the tooling to build one. concretely, ukrainian has limited high-quality synonym resources (e.g., wordnet-style inventories with usable synsets), and there are no ready-to-use synonym-substitution methods for robustness testing (candidate generation, form inflection, grammatical filtering). this paper makes four contributions: (i) we construct a naive dictionary-based synonym substitution method for ukrainian and quantify how it affects multimodal retrieval performance. (ii) we develop an llm-based substitution method that generates context-and grammar-aware ukrainian synonyms and evaluate its impact. (iii) we introduce a hybrid method that combines dictionary coverage with llm selection and inflection, yielding higher-quality substitutions and more informative robustness assessments. (iv) we propose and evaluate a fine-tuning approach, using synonymaugmented data, to improve text-to-image retrieval under the synonym-substitution attack (ssa). finally, we replicate our evaluation and fine-tuning approach on czech to assess cross-lingual generality and observe consistent gains, indicating transfer beyond a single language. a second line of work replaces clip's original english text encoder with a multilingual language model and trains on multilingual image-text data @xcite . this approach bootstraps cross-lingual lexical coverage. these models are often "ready-to-use" for many languages out of the box. however, representation quality for a given target language depends on its diversity and quality in the training mixture. a third approach uses teacher-student transfer to adapt clip to new languages @xcite . in such approaches, english clip acts as the teacher while a multilingual or target-language encoder learns to replicate its image-text alignment. these methods leverage parallel or translated data and are more computeefficient than full training, but translation artifacts and lexical biases can distort the embedding space. another approach to generate multimodal embeddings is vision-language models @xcite @xcite . however, many of them are closed and do not expose embedding apis, and they are expensive. the ukrainian visual-word sense disambiguation benchmark @xcite offers a way to measure multimodal model performance in ukrainian, showing m-clip foot_0 @xcite as the best baseline. however, openclip multilingual variants were not evaluated. to fill this gap, we evaluated openclip models that support ukrainian on the same benchmark and compared it against m-clip. we observed that openclip model foot_1 achieved stronger performance metrics on the ukrainian visual-wsd benchmark ( synonym-based adversarial attacks are wellstudied in high-resource languages such as english, where various approaches exist to generate semantically and syntactically valid substitutions @xcite . those methods, however, depend on resources that are either missing or not practically usable in ukrainian, e.g., wordnet or counter-fitting synonym embeddings @xcite . even the recently released wikipediamapped ukrainian wordnet offers only one lemma per synset and thus no usable synonym sets @xcite . recently @xcite adapt english ssa to ukrainian via multilingual models, but the approach targets only text and is engineering-heavy. also, prior work shows that many automatic substitution methods for english still yield grammatically or semantically invalid replacements @xcite . llm-based synonym substitution @xcite offers a way to avoid external language resources, but, to the best of our knowledge, it has not been evaluated for low-resource languages. given these limitations, instead of directly adapting existing synonym substitution techniques from english, we propose a novel approach leveraging llms in combination with an unstructured ukrainian synonym dictionary to generate contextually and grammatically appropriate synonym replacements for ukrainian. ## conclusion this work presents the first systematic evaluation of text-to-image retrieval under ssa for ukrainian. in the absence of ready-to-use ssa frameworks for this low-resource language, we establish baseline methods using a dictionary-based and an llmbased (gpt-4o) methods, analyzing their respective strengths and weaknesses. building on these insights, we propose a novel hybrid substitution method that better balances lexical coverage with grammatical and semantic correctness. we further demonstrate that fine-tuning on the ukrainian multi30k dataset improves retrieval across all ssa methods. extending this idea, we introduce synonym-augmented fine-tuning, in which additional training pairs are created by substituting nouns with llm-generated synonyms. this approach improves text-to-image retrieval under ssa while maintaining performance on the unperturbed data. finally, by replicating our experiments on czech, another morphologically rich slavic language, we show that both the hybrid substitution method and synonym-augmented fine-tuning generalize beyond ukrainian. together, these results highlight a practical path toward more robust multimodal embeddings in low-resource languages.