Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks (2024)

Abstract: Large Language Models (LLMs) are increasingly becoming the preferred foundation platforms for many Natural Language Processing tasks such as Machine Translation, owing to their quality often comparable to or better than task-specific models, and the simplicity of specifying the task through natural language instructions or in-context examples.Their generality, however, opens them up to subversion by end users who may embed into their requests instructions that cause the model to behave in unauthorized and possibly unsafe ways.In this work we study these Prompt Injection Attacks (PIAs) on multiple families of LLMs on a Machine Translation task, focusing on the effects of model size on the attack success rates.We introduce a new benchmark data set and we discover that on multiple language pairs and injected prompts written in English, larger models under certain conditions may become more susceptible to successful attacks, an instance of the Inverse Scaling phenomenon (McKenzie et al., 2023).To our knowledge, this is the first work to study non-trivial LLM scaling behaviour in a multi-lingual setting.

## introduction general purpose pretrained large language models have become the dominant paradigm in nlp, due to their ability to quickly adapt to almost any task with in-context few-shot learning @xcite @xcite or instruction following @xcite . in most settings, the performance of llms predictably increases with their size according to empirical scaling laws @xcite @xcite , however recent works have discovered scenarios where not only llms misbehave, but they even become worse with increasing size, a phenomenon known as inverse scaling, or exhibit non-monotonic performance w.r.t. size, e.g. u-shaped scaling or inverse u-shaped scaling @xcite @xcite , with many more such scenarios being discovered during the inverse scaling prize @xcite . one such class of scenarios is prompt injection attacks (pias), where the end-user embeds instructions in their requests that contradict the default system prompt or fine-tuning and thus manipulate the llm to behave in ways not intended by the system developer, such as performing a task different than the intended one, revealing secret information included in the system prompt, subvert content moderation, and so on. in the inverse scaling prize, pias were evaluated on simple tasks such as word capitalization and repetition, showing strong asymptotic inverse scaling, meaning that the larger the llms are, the more susceptible they become to these attacks. in this work, we evaluate the scaling behavior of prompt injection attacks on prompt-based machine translation. prompt-based machine translation (pmt) consists of using a general-purpose llm to do machine translation by asking it to translate a text, optionally prepending a small number (1-5) of parallel examples in the prompt @xcite . this approach is competitive with task-specific neural machine translation systems on high and medium resource language pairs @xcite . in order to evaluate pmt under pias, we create a parallel test set of questions, which we consider as our clean (non-adversarial) examples for pmt, then we transform them into adversarial examples by adding a prefix to the source side that asks the system to ignore its instructions and just answer the question. we evaluate our clean and adversarial examples on multiple llm families, both in the zero-shot and few-shot setting, identifying scaling trends. we release our data and the code needed to reproduce our experiments foot_0 . ## experiments due to limitations of the models and our own budget and time constraints, we do not evaluate all translation directions and prompting strategies on all model families. we perform the following experiments ( the experiments are divided into two parts: we first report our results of the clean examples in section 3.1, then report the results of adversarial examples in section 3.2. we only report the accuracy in this section, the bleu scores of each experiment can be found in appendix c. in section 3.3, we display the average performance of x-to-english language pairs and englishto-x language pairs. computational resources for the gpt and in-structgpt models, we spent about 200 us dollars on the openai api. the experiments with t5 and flan-t5 models except the largest variants were done on the hpe sgi 8600 system with nvidia gv100 gpu. the experiments on the llama2, llama2-chat and the largest variants of t5 and flan-t5 were performed on a cluster of nvidia a100 40gb/80gb gpus (note that a single node with 4 a100 40gb gpus is sufficient to run all experiments). ## conclusion in this paper, we investigated the scaling behaviour of llms in the task of machine translation of fac-tual questions, both on clear examples and on adversarial examples constructed according to a simple prompt injection attack where we tell the model to answer the questions instead of translating them. we found inverse scaling under certain model series and zero-shot scenarios. in addition to the effect from the model size, we also found that performance severely deteriorates when the prompt is written in english, indicating inverse scaling in the dimension of the amount of training data. to our knowledge, this is the first work to investigate non-monotonic scaling and prompt injection attacks in a multi-lingual setting.