Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models (2024)

Abstract: The automatic evaluation of natural language generation (NLG) systems presents a long-lasting challenge. Recent studies have highlighted various neural metrics that align well with human evaluations. Yet, the robustness of these evaluators against adversarial perturbations remains largely under-explored due to the unique challenges in obtaining adversarial data for different NLG evaluation tasks. To address the problem, we introduce AdvEval, a novel black-box adversarial framework against NLG evaluators. AdvEval is specially tailored to generate data that yield strong disagreements between human and victim evaluators. Specifically, inspired by the recent success of large language models (LLMs) in text generation and evaluation, we adopt strong LLMs as both the data generator and gold evaluator. Adversarial data are automatically optimized with feedback from the gold and victim evaluator. We conduct experiments on 12 victim evaluators and 11 NLG datasets, spanning tasks including dialogue, summarization, and question evaluation. The results show that AdvEval can lead to significant performance degradation of various victim metrics, thereby validating its efficacy.

## introduction recent advancements in generative language models, especially advanced large language models (llms), have led to their widespread application in creating human-like content across various domains. concurrently, numerous automatic text evaluation metrics have been developed, aiming to identify unhelpful responses and understand the model's capacity. these metrics have exhibited good correlations with human judgment across diverse natural language generation (nlg) tasks, e.g., dialogue generation @xcite , summarization @xcite , question generation @xcite . despite their strong performance, the robustness of these metrics against adversarial attacks or misleading inputs remains critically under-explored. therefore, in this paper, we aim to automatically identify "achilles' heel", critical weaknesses, within well-established automatic evaluation metrics. adversarial attack techniques are widely adopted to assess the robustness of neural models. yet, their application in nlp has predominantly focused on general classification tasks, e.g., sentiment classification @xcite . these attack techniques all build upon the label-preserving assumption that the gold label of input text will remain unchanged after adding bounded perturbation on the input text @xcite . this assumption, however, does not hold on the nlg evaluation tasks, where even slight modifications can drastically alter, often negatively, a text's perceived quality, which directly correlates with the regression label in the evaluation tasks. for example, changing a single word "movie" to "dish" in the dialogue response "that's my favorite movie" will perfectly preserve the positive sentiment classification label but significantly alter the response relevance evaluation label, given that the dialogue context is about a movie. consequently, these traditional attacking techniques can easily obtain low-quality adversarial texts with high metric scores while struggling to produce high-quality adversarial texts that score low. existing studies on nlg evaluators' robustness mainly rely on manual effort or heuristic perturbation rules to ensure the perturbed text accords with specified labels. these studies also fail to obtain high-quality adversarial texts with low metric scores and cannot scale up to generate large-scale diverse adversarial data @xcite . to overcome these limitations, we propose ad-veval, designed to obtain data that lead to two different types of disagreement between humans and victim evaluators: 1) text valued by humans but underrated by victim evaluators; 2) text underrated by humans but valued by victim evaluators. inspired by recent works on using llms to iteratively improve their response based on current feedback @xcite @xcite , we adapt llms as the data generator to iteratively optimize the benign input text with the adversarial objective. as mentioned above, label preservation posts a unique challenge in attacking nlg tasks. recognizing the recent trend of leveraging large language models (llms) as a substitute for human annotators to generate and annotate high-quality data @xcite @xcite , we also utilize llms as the evaluator to give feedback during the optimization process, thereby imposing an implicit quality constraint on the generated text. we evaluate adveval on 11 datasets covering three distinct nlg tasks against 12 automatic evaluation metrics that vary in their reliance on references, models, and llms. experimental results demonstrate adveval's ability to generate highquality adversarial data effectively. through manual inspection, we also confirm the label preservation property of adveval. further experiments on adversarial learning, ablation studies, and case studies convincingly validate the effectiveness and generalization ability of our proposed adveval framework.