PPTC - R benchmark: Towards Evaluating the Robustness of Large Language Models for P ower P oint Task Completion (2024)

Abstract: The growing dependence on Large Language Models (LLMs) for finishing user instructions necessitates a comprehensive understanding of their robustness to complex task completion in real-world situations. To address this critical need, we propose the PowerPoint Task Completion-Robustness (PPTC-R) benchmark to measure LLMs’ robustness to the user PPT task instruction and software version (Powerpoint). Specifically, we construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. To assess the robustness of Language Models to software versions, we vary the number of provided APIs to simulate both the newest version and earlier version settings. Subsequently, we test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates these robustness settings, aiming to evaluate how deviations impact LLMs’ API calls for task completion. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark, particularly in the version update and the multilingual settings. However, we find that all LLMs lose their robustness when confronted with multiple challenges (e.g., multi-turn) simultaneously, leading to significant performance drops. We further analyze the robustness behavior and error reasons of LLMs in our benchmark, which provide valuable insights for researchers to understand the LLM’s robustness in task completion and develop more robust LLMs and agents.

## introduction large language models (llms) like gpt-4 (ope-nai, 2023) exhibit strong performance on a variety of basic natural language tasks and human examinations @xcite @xcite @xcite . this has spurred hope for their potential to assist humans in completing tasks in complex * equal contribution environments, such as purchasing items in web-shop @xcite , creating and editing pow-erpoint slides in pptc @xcite , and navigating computers in miniwob++ @xcite . recent works @xcite @xcite , such as promptbench @xcite , have studied the robustness of llms to task prompts for basic natural language tasks. however, there remains a lack of benchmarks for evaluating llm robustness in complex task completion, a critical factor for their performance in real-world scenarios. to address this gap, we introduce powerpoint task completion-robustness (pptc-r), a benchmark designed to measure and analyze the robustness of llms to user instructions and software versions in the context of pow-erpoint task completion. our benchmark has two distinct features: (1) previous robustness evaluations have focused on traditional natural language tasks, where the model's output is typically limited to generating text strings or options. in contrast, our benchmark evaluates how adversarial perturbations affect llms' api calls for complex power-point task completion. (2) while prior studies have mainly constructed benchmarks by attacking the task prompt or input text, we consider how shifts in software versions impact llm performance, offering a new perspective on robustness evaluation. to measure the robustness of llms to user instructions, we construct adversarial user instructions through various perturbations. these include: 1) (language level) translating the original english instructions into 14 non-english languages (figure 1 ), 2) (sentence level) adding gpt-4generated chitchat sentences to the original instructions as noise, and 3) (semantic level) prompting gpt-4 to rephrase the original instructions in four different ways while preserving the same meaning (figure 2 ). these perturbations mimic typical variations encountered by users and developers in daily powerpoint use with an ai agent. our llmbased sentence and semantic perturbation methods efficiently generate large quantities of high-quality, novel adversarial data. additionally, we assess the impact of software versions on ppt task completion by varying the number of available apis. we simulate version updates by: (1) introducing many new apis to the existing list, which may affect the llm's api selection, and (2) removing several apis from the existing list, simulating scenarios where the current software version lacks capabilities to fully address user instructions, requiring the llm to find alternative solutions. these perturbations (three for user instructions and two for apis) form our pptc-r benchmarks, encompassing a total of five settings. we evaluate 3 closed-source llms (e.g. gpt-4 and chatgpt) and 4 representative open-source llms (e.g. llama-2 and wizardlm) using our benchmarks. gpt-4 demonstrates the highest performance and robustness across all five settings (sec.4.3). for instance, gpt-4 maintains high turnbased performance even with the introduction of 97 new apis, whereas other llms experience greater performance drops (e.g. chatgpt) or sustain low performance levels (e.g. llama-2). notably, we observe a unique robustness degradation in all llms: their robustness significantly decreases when task difficulty increases or when transitioning to more complex environments. we identify three primary error causes for llms: distraction by chitchat (see the bottom of figure 2 ), invoking unavailable apis, and misinterpreting instructions with new expressions (sec.5.1). we also explore llm behavior with varying numbers of new apis (sec5.3). in summary, our paper's contributions are: ## pptc-r benchmark in this section, we introduce our powerpoint task completion-robustness (pptc-r) benchmark, detailing its dataset components, design principles, and the collection and validation process. ## conclusion deploying llms and llm-based agents to complete users' task instructions is becoming increasingly important. however, there is still a lack of evaluation and analysis of llms' robustness in complex task completions. we introduce the pow-erpoint task completion-robustness benchmark to assess llm robustness in handling adversarial instructions and adapting to different software versions in complex ppt task completion. our benchmark results for seven llms show that gpt-4 is the strongest, though all llms' robustness degrades as task difficulty increases. we further conduct a detailed analysis of error reasons and robustness behaviors to gain a deeper understanding.