LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History (2024)

Abstract: With the recent emergence of powerful instruction-tuned large language models (LLMs), various helpful conversational Artificial Intelligence (AI) systems have been deployed across many applications. When prompted by users, these AI systems successfully perform a wide range of tasks as part of a conversation. To provide some sort of memory and context, such approaches typically condition their output on the entire conversational history. Although this sensitivity to the conversational history can often lead to improved performance on subsequent tasks, we find that performance can in fact also be negatively impacted, if there is a _task-switch_. To the best of our knowledge, our work makes the first attempt to formalize the study of such vulnerabilities and interference of tasks in conversational LLMs caused by task-switches in the conversational history. Our experiments across 5 datasets with 15 task switches using popular LLMs reveal that many of the task-switches can lead to significant performance degradation.

## introduction recent advancements in natural language processing (nlp) @xcite , have led to their widespread deployment of large language models (llms) across various applications @xcite @xcite . one of the popular nlp tasks includes conversational systems where llms are capable of engaging in dialogues that mimic human interactions @xcite . a typical interaction involves a series of conversation turns starting with the user and the llm responds to the user. this interaction is however focused on a specific topic or a task @xcite . 1 code available on github. give sentiment of this review. "the food was terrible." the sentiment is negative give sentiment of this review. "the brunch menu is amazing..." the sentiment is positive solve the problem. "john has five delicious apples. he eats two. how many are left?" ## related work large language models (llms) are becoming a crucial building block of conversation-based virtual assistants @xcite @xcite . leveraging in-context or few-shot examples, llms have demonstrated remarkable capabilities for downstream tasks @xcite . in contrast to the resource-intensive fine-tuning process @xcite , in-context learning eliminates the need for parameter updates, while achieving state-of-the-art performance @xcite @xcite @xcite @xcite . however, despite its advantages, incontext learning tends to suffer from sensitivity to prompts, input distribution, and formats, which can potentially impact the model's performance @xcite @xcite @xcite . @xcite observe that the incontext examples implicitly bias the model. in our work, we aim to study the bias that may arise due to chat history (in-context examples) when a user switches the task. furthermore, recent works @xcite have looked at the vulnerability of llm to prompt injections and adversarial attacks. unlike prompt injection, where a malicious prompt may be added to the conversation of llm, our setting, is concerned with non-malicious task-switches. while a few recent works have investigated the reliance on shortcuts in conversation history @xcite @xcite , our work aims to evaluate the influence of the conversation on a new task. our work is also differentiated from the study topic change in task-oriented dialogue systems @xcite @xcite as we consider a stronger shift of taskswitch from open dialogue llms. ## conversational task-switch this work introduces and formalizes task-switch in a conversation for llms. a conversation between a user and the llm consists of multiple conversation turns. now consider (u k , r k ) as the k-th turn of the conversation where u k corresponds to the k-th user prompt and the model's corresponding response r k . each user prompt u k can be viewed as an instance of a specific task request, e.g. sentiment classification or mathematical reasoning. a conversation history of l turns can be defined as @xmath0, r k )} l @xmath1. subsequently, the next response, r l+1 for model θ is given as: in this work, we consider conversations with a single task-switch, where all user requests in the conversation history h belong to the same task, and the final user request u l+1 is a different task. we refer to the task associated with h as the conversation history task (ch task) t h where @xmath2. when the tasks t h and t t are sufficiently different (as per human understanding of language and tasks), the conversation history h ideally must not impact the response, r l+1 . for a model robust to such task-switches, t @xmath3, its response r l+1 is conditionally independent of the conversation history, however, in practice, models can be sensitive to the conversation history, h, which can harm the quality of the response r l+1 after a task-switch, t @xmath4. we define τ (•), as a reference-free metric for the task-switch sensitivity of a model θ, to measure the extent of this vulnerability. theoretical and empirical implications of other definitions for taskswitch sensitivity are considered in appendix e. the task-switch sensitivity metric can be interpreted as: to simulate a setting where the model has perfect performance on the ch-task, t h we adopt teacherforcing, s.t. @xmath5, rk )} l @xmath6, where r is the reference ground-truth response. ## conclusions and future work this work formalizes and investigates the sensitivity of large language models (llms) to taskswitch scenarios within conversational contexts. we introduce a task-sensitivity as a reference-free metric that can explain a model's behavior to taskswitches along with the performance change. by experimenting with various task-switch settings, we observe that even advanced models like gpt-4 can exhibit vulnerabilities to task-switches. our work additionally lays the foundation for future work on 'side-channel' vulnerabilities of llms to undesired information leakage/bias from the conversation history. further work will focus on developing adaptive context management strategies within llms to mitigate the risk of task-switch sensitivity.