Precision vs. Perturbation: Robustness Analysis of Synonym Attacks in U krainian NLP (2025)

Abstract: Synonym-based adversarial tests reveal fragile word patterns that accuracy metrics overlook, while virtually no such diagnostics exist for Ukrainian, a morphologically rich and low‐resource language. We present the first systematic robustness evaluation under synonym substitution in Ukrainian. Adapting TextFooler and BERT‐Attack to Ukrainian, we (i) adjust a 15000‐entry synonym dictionary to match proper word forms; (ii) integrate similarity filters; (iii) adapt masked‐LM search so it generates only valid inflected words. Across three text classification datasets (reviews, news headlines, social‐media manipulation) and three transformer models (Ukr‐RoBERTa, XLM‐RoBERTa, SBERT), single‐word swaps reduce accuracy by up to 12.6, while multi‐step attacks degrade performance by as much as 40.27 with around 112 model queries. A few‐shot transfer test shows GPT‐4o, a state‐of‐the‐art multilingual LLM, still suffers 6.9–15.0 drops on the same adversarial samples. Our results underscore the need for sense‐aware, morphology‐constrained synonym resources and provide a reproducible benchmark for future robustness research in Ukrainian NLP.

## introduction natural language processing (nlp) has undergone a rapid transformation over the past decade. early systems were built on rule-based heuristics and simple statistical models, where model behavior was largely transparent and evaluation relied on basic accuracy or coverage metrics. as these models lacked generalization power, error patterns were relatively easy to identify and correct. with the introduction of neural networks-particularly transformer architectures @xcite trained on large-scale text corpora-modern nlp systems have achieved remarkable performance across diverse tasks such as machine translation, sentiment analysis, and question answering. these models can capture complex semantic and syntactic patterns, often surpassing human-level benchmarks. however, the internal behavior of these models is difficult to interpret, and traditional metrics such as accuracy or f1 score often overestimate performance and fail to reveal model vulnerabilities, motivating the need for more comprehensive evaluation methods @xcite . this has motivated the development of stress tests and behavioral diagnostics to probe how models behave under controlled perturbations. one such technique is the synonym substitution attack, which evaluates whether a model's prediction is sensitive to small, meaning-preserving changes in the input text. these attacks are appealing because they preserve grammaticality and semantics from a human perspective while often revealing inconsistent model behavior. while synonym substitution attacks have been widely studied in english, their applicability to low-resource and morphologically rich languages remains underexplored. ukrainian, for example, poses additional challenges: it exhibits complex inflectional morphology, has limited lexical resources, and lacks large-scale evaluation benchmarks for adversarial robustness. as a result, it is unclear how well multilingual or ukrainianspecific language models perform under such perturbations. to our knowledge, this work presents the first systematic evaluation of synonym substitution attacks in ukrainian. we explore whether current models-both monolingual and multilingual-are robust to these types of perturbations, and we assess whether the vulnerability persists in modern llms. contributions. our main contributions are: resources. we release the complete codebaseincluding dataset loaders, fine-tuning scripts, and attack pipelines-in a single public repository so that other researchers can benchmark the robustness of their own ukrainian or multilingual models with minimal effort: https://github.com/ mudryi/ukr-synonym-robustness . ## related work adversarial attacks were first explored in computer vision, where imperceptible pixel-level perturbations can drastically alter model outputs @xcite . in contrast, nlp inputs are discrete, so crafting adversarial examples requires careful preservation of grammar and semantics. adversarial attacks in nlp are commonly categorized into character-level, word-level, sentencelevel, and syntactic-level perturbations, depending on the granularity and linguistic structure affected. character-level attacks such as hotflip use white-box gradients to identify single-character edits that maximally increase loss, demonstrating that even single letter change can mislead a classifier @xcite . in the black-box setting, deepwordbug applies simple heuristics-swaps, deletions, or insertions-to high-saliency tokens, achieving substantial accuracy drops with minimal edit distance @xcite . word-level synonym substitution attacks replace important words with context-preserving alternatives. early genetic algorithm approaches by @xcite and the pwws method by @xcite ranked words by importance before substituting them using wordnet. textbugger @xcite combined both character-level and word-level perturbations and introduced semantically similar replacements using embedding-based nearest neighbors. textfooler later showed that a small number of carefully chosen synonym swaps can reduce bert's accuracy by over 50% while keeping the text fluent @xcite . building on this, bert-attack leverages masked language model infilling to generate higher-quality substitutes with fewer queries, further exposing model brittleness @xcite . going beyond individual words, sentence-level testing frameworks probe models' sensitivity to diverse linguistic phenomena. @xcite introduced syntactically controlled paraphrase networks (scpn) to generate paraphrases under alternate parse templates, revealing that models often fail on syntactic variations despite preserved meaning. @xcite proposed checklist, a behavioral testing framework that uses task-agnostic "capabilities" and targeted test suites, such as minimum functionality tests, invariance tests, and directional expectation tests, to uncover fine-grained weaknesses in nlp models beyond traditional accuracy metrics. in a complementary line of research, several studies have shown that models often rely on unintended lexical artifacts present in the training data itself, e.g. annotation artifacts in nli @xcite ), heuristic "competency problems" @xcite , and surface-cue reliance in reading-comprehension benchmarks @xcite . such lexical shortcuts further motivate synonym-substitution tests, because they imply that changing a single word can flip a prediction even outside an explicit adversarial setting. while most of these methods target english, recent work extends robustness evaluation to lowresource and morphologically rich languages. @xcite adapted masked-lm synonym attacks to arabic, finding that bert-based classifiers can be even more vulnerable than traditional models. for chinese, @xcite argot framework uses homophone and look-alike character substitutions to generate readable, highsuccess-rate adversarial examples. recent work such as promptrobust @xcite demonstrates that even advanced llms are sensitive to minor textual perturbations-including synonyms, typos, and rephrasings-across a range of tasks. however, such evaluations remain largely limited to high-resource languages like english. robustness diagnostics for morphologically rich, low-resource languages such as ukrainian are still lacking, motivating the need for adapted adversarial benchmarks. in this work, we focus exclusively on synonym substitution attacks. we select this approach because it generates fluent, semantically-preserving perturbations that are both realistic and challenging for models to detect, offering a clear benchmark for word-level robustness while maintaining the original intent of the text. ## synonym substitution attack formulation adversarial attacks in nlp aim to slightly perturb a valid input x @xmath0, w 2 , . . . , w n ], where each w i denotes a word token, to generate an adversarial counterpart x adv such that: to maintain plausibility, the adversarial perturbation is constrained by a predefined budget, typically limiting the number of modified tokens: in the case of synonym substitution attacks (ssa), the perturbation involves replacing one or more words with contextually appropriate synonyms. an adversarial example takes the form x @xmath1, w ′ 2 , . . . , w n ], where w ′ 2 is a synonym of w 2 selected to preserve fluency and meaning. the candidate set for substitution is typically constrained by part-of-speech tags, semantic similarity, or language model likelihood. synonym substitution attacks (ssa) challenge models by preserving surface structure and meaning while revealing overreliance on specific tokens. unlike character-level or noise-based attacks, synonym perturbations generate more realistic inputs, making them ideal for evaluating semantic robustness. we define ssa as a sequence of word replacements aimed at flipping the model's prediction while not changing the overall meaning of the sequence and maintaining its grammatical and semantic validity. this involves identifying important words, selecting appropriate synonyms, and substituting them sequentially until misclassification or a stopping condition occurs. sections 6.1-6.2 detail our adaptations of textfooler and bert-attack for ukrainian. ## experimental setup to measure how robust modern models are to ukranian synonyms substitutions, we need to select a dataset where we want to measure robustness and models themselves. ## one-word replacement baseline before implementing full synonym substitution attacks, we first evaluate a simple baseline to assess the robustness of each model to single-word replacements. specifically, for each dataset, we identify the 1000 most frequent words in the training corpus and extract candidate synonyms from the publicly available ukrainian synonym dictionary synonimy.info (synonimy.info, 2025), which provides non-commercial use. although the dictionary offers broad coverage, it contains some outdated or overly specific entries and includes occasional mismatches that do not reflect true synonymy in modern usage (e.g., песпосiпака). to improve substitution quality, we apply a multi-step filtering process: we discard badly formatted or duplicated items, remove words that differ from the original only in grammatical form, and eliminate antonyms by cross-referencing with an antonym dictionary (antonimy.info, 2025). to further expand synonym coverage for high-impact words (identified using a leave-one-out strategy; see section 6.1). to increase coverage we manually added several examples from the official online version of the словник синонiмiв української мови (наукова думка, 1999). finally, a portion of the resulting synonym sets was manually reviewed to confirm whether generated replacements preserved both grammatical compatibility and original meaning. to ensure grammatical correctness, each synonym is morphologically transformed to match the original word form using pymorphy2. for each word in the top 1000 most frequent words, we generated all valid one-word replacements by substituting it with each of its synonyms (if present in a given sentence). each original test example, therefore, produces multiple perturbed variants, each containing exactly one synonym substitution. we apply this procedure only to test-set samples that were correctly classified by the model, ensuring that we are measuring actual robustness rather than model errors. the total number of generated examples is calculated as the number of test samples times the number of overlapping top-1000 words times the number of valid synonyms per word. once all replacements are generated, we group them by their original (unperturbed) sample and select the one that causes the most harmful prediction change -defined as the replacement that leads to the largest drop in the target model's predicted probability for the original class (i.e., the highest reduction in confidence or a misclassification). this one-to-one mapping allows us to evaluate worst-case single-word synonym substitution per sentence. examples of both successful and unsuccessful substitutions-along with model predictions-are provided in appendix b.1. we report the model's test-set accuracy after applying the most harmful replacement to each example. ## synonym substitution attacks to perform more advanced synonym substitution attacks that support multiple word replacements, we adapt two widely used adversarial frameworks: textfooler @xcite and bert-attack @xcite . while both methods have demonstrated strong performance in english, they cannot be applied directly to ukrainian due to limited language resources and morphological complexity. each method requires adaptation with respect to the availability of synonym sources and ukrainian linguistic characteristics. in particular, we integrate a dictionary-based synonym set into the textfooler pipeline, combined with morphological transformations to ensure grammatical correctness. the bert-attack method, in contrast, relies on a masked language model for synonym generation, which we adjust to work effectively with ukrainian inputs. these two frameworks were selected due to their complementary strengths. textfooler offers simplicity and transparency: it requires only a synonym list, allows precise control over pos and similarity constraints, and provides interpretability by clearly identifying which words trigger model changes. bert-attack, on the other hand, leverages a language model to propose replacements that better fit the surrounding context. it requires no explicit synonym dictionary and tends to generate more fluent, human-like paraphrases using the model's own learned vocabulary and semantics. for each synonym substitution attack, we report the original and adversarial accuracy, the accuracy drop, the average word change rate, and the average number of model queries per sample. change rate denotes the percentage of words modified in the input, while queries indicates the number of model forward passes required to construct the adversarial example. all results are in tables 5 and 6. ## llm evaluation on attacked samples given the growing use of large language models (llms) in real-world nlp applications, we examine whether state-of-the-art llms remain vulnerable to synonym substitution attacks. while prior work has shown that such perturbations can mislead traditional models, it remains unclear whether modern instruction-tuned llms-especially those with advanced contextual reasoning-exhibit similar vulnerabilities, particularly in low-resource languages like ukrainian. we sample 200 adversarial examples for each combination of three datasets, three target models, and two attack strategies, yielding 3,600 examples in total (1,800 per attack type). all samples are selected from inputs that successfully fooled the original finetuned classifiers (xlm-roberta, ukr-roberta, and sbert), and are reused to test the robustness of gpt-4o -a strong, closed-source llm with competitive multilingual capabilities, including ukrainian. we construct dataset-specific, few-shot prompts for gpt-4o (complete templates are listed in appendix c). although these prompts were not tuned to counter our attacks, a substantial portion of examples that fooled the finetuned classifiers likewise fooled gpt-4o. this finding indicates that even high-capacity, instruction-tuned llms remain vulnerable to meaning-preserving perturbations in morphologically rich, low-resource languages. ## conclusion and future work we presented the first systematic evaluation of synonym substitution attacks for the ukrainian language, demonstrating that both simple oneword replacements and advanced frameworks like textfooler and bert-attack can significantly degrade model performance-causing accuracy drops of up to -40.2% with around 113 queries per sample. few-shot evaluations with gpt-4o show that even large instruction-tuned llms remain susceptible, with accuracy declines ranging from 6.9% to 15.0%. error analysis shows that some successful attacks work by changing the meaning or producing grammatically invalid substitutions rather than truly revealing model vulnerabilities. this highlights the limitations of current synonymsubstitution strategies. to address these, future work should explore hybrid adversarial pipelines that combine synonym dictionaries with masked language model proposals, integrate word sense finally, establishing standardized ukrainian adversarial benchmarks-evaluating not just prediction accuracy, but also fluency and semantic fidelity-will be key to enabling robust and reproducible evaluation of model resilience in lowresource settings.