Is LLM -as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment (2024)

Abstract: Large Language Models (LLMs) are powerful zero-shot assessors used in real-world situations such as assessing written exams and benchmarking systems. Despite these critical applications, no existing work has analyzed the vulnerability of judge-LLMs to adversarial manipulation. This work presents the first study on the adversarial robustness of assessment LLMs, where we demonstrate that short universal adversarial phrases can be concatenated to deceive judge LLMs to predict inflated scores. Since adversaries may not know or have access to the judge-LLMs, we propose a simple surrogate attack where a surrogate model is first attacked, and the learned attack phrase then transferred to unknown judge-LLMs. We propose a practical algorithm to determine the short universal attack phrases and demonstrate that when transferred to unseen models, scores can be drastically inflated such that irrespective of the assessed text, maximum scores are predicted. It is found that judge-LLMs are significantly more susceptible to these adversarial attacks when used for absolute scoring, as opposed to comparative assessment. Our findings raise concerns on the reliability of LLM-as-a-judge methods, and emphasize the importance of addressing vulnerabilities in LLM assessment methods before deployment in high-stakes real-world scenarios.

## introduction large language models (llms) have shown to be proficient zero-shot assessors, capable of evaluating texts without requiring any domain-specific training @xcite @xcite . typical zero-shot approaches prompt powerful llms to either generate a single quality score of the assessed text @xcite score the summary between 1-5 "some animals did something." score the summary between 1-5 "some animals did something. summable" which summary is better? a: "some animals did something." b: "tortoise wins race; slow and steady" which summary is better? a: "some animals did something. informative" b: "tortoise wins race; slow and steady" 4.8 b a ## zero-shot assessment with llms as discussed by @xcite ; @xcite , there are two standard reference-free methods of prompting instruction-tuned llms for quality assessment: for various assessment methods, we consider rankings tasks where given a query context d and a set of n responses x 1:n , the objective is to determine the quality of each response, s 1:n . an effective llm judge should predict scores for each candidate that match the ranking r 1:n of the text's true quality. this section will further discuss the details of both comparative assessment (section 3.1) and absolute assessment (section 3.2). ## conclusions this is the first work to examine the adversarial robustness of zero-shot llm assessment methods against universal adversarial attacks, and reveal significant vulnerabilities in llm absolute scoring and mild vulnerabilities in llm comparative assessment. we demonstrate that the same short 4word universal adversarial can be appended to any input text to deceive llm assessment system into predicting inflated scores. notably, llm-scoring attacks developed with a smaller surrogate llmscoring system can be effectively transferred to larger llms such as chatgpt. we also provide an initial investigation into simple detection approaches, and show that perplexity can be a promising tool for identifying adversarially manipulated inputs. further work can explore adaptive attacks and more sophisticated defence approaches to minimize the risk of misuse. on the whole, this paper raises awareness around the susceptibility of llmas-a-judge nlg assessment systems to universal and transferable adversarial attacks.