PPTC Benchmark: Evaluating Large Language Models for P ower P oint Task Completion (2024)

Abstract: Recent evaluations of Large Language Models (LLMs) have centered around testing their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs. However, the evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal instructions in a complex multi-modal environment has not been investigated. To address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark to assess LLMs’ ability to create and edit PPT files based on user instructions. It contains 279 multi-turn sessions covering diverse topics and hundreds of instructions involving multi-modal operations. We also propose the PPTX-Match Evaluation System that evaluates if LLMs finish the instruction based on the prediction file rather than the label API sequence, thus it supports various LLM-generated API sequences. We measure 3 closed LLMs and 6 open-source LLMs. The results show that GPT-4 outperforms other LLMs with 75.1% accuracy in single-turn dialogue testing but faces challenges in completing entire sessions, achieving just 6% session accuracy. We find three main error causes in our benchmark: error accumulation in the multi-turn session, long PPT template processing, and multi-modality perception. These pose great challenges for future LLM and agent systems .

## introduction recent evaluation works for large language models (e.g. chatgpt and gpt-4 (openai, 2023)) focus on their zero-shot/few-shot abilities on basic natural language tasks @xcite @xcite and their tool-use ability to generate apis for solving user instructions, such as basic apis like a calculator in tool transformer @xcite , rapidapis in toolllm @xcite , and hugggingface apis in gorilla @xcite . however, these tool-use works emphasize the translation of natural language instructions into apis and ignore the challenge of using apis in the observation of complex multi-modal environments to finish user instructions. also, their evaluation approach focuses on comparing the generated apis with the label api sequence, assuming there's only one unique solution. this approach becomes impracticable in situations with multiple/unlimited correct solutions. to address these challenges, we introduce power-point task completion (pptc), a benchmark that measures llms' performance in creating and editing ppt file tasks based on user instructions. we choose powerpoint as it includes various elements like textbox, table, and image and supports a wider range of apis than word and excel. our benchmark has three distinctive features from other task completion benchmarks: (1) multiturn dialogue with varying difficulty. our pptc benchmark simulates the multi-turn dialogue session between the user and the llm and contains 279 multi-turn sessions. each multi-turn session in our benchmark includes 2 to 17 turns. each turn consists of a user instruction that describes the user's needs, a feasible solution that provides the correct solution, and the resulting label output file. some turns can be easily addressed using a single api, while over half of the instructions require multiple apis for completion. we provide the llm with a reference api file that contains all feasible apis for selection. (2) multi-modality. finishing the instruction of our benchmark requires understanding the multi-modal ppt file content and using multi-modal api operations (e.g., pptc has 268 image operation-related instructions). (3) evaluation based on the final status: we propose the pptx-match evaluation system to evaluate the llm's outcome. to identify if the llm completes the instruction, it checks the ppt file produced by executing the llm-generated apis rather than the llm-generated apis, thus all api sequences that lead to the correct final status are acceptable. to finish the instruction, we use the current instruction, past turns' instructions (dialogue history), the ppt file content (specific environment information), and the reference api file as the input to prompt the llm to generate an api sequence as the solution (see figure 1 (a) ). then we use the api executor to execute the api sequence and return the user the resulting ppt file (see figure 1 (b) ). we name the resulting ppt file as the prediction file. in the evaluation step (see figure 1 (c )), the pptx-match evaluation system first uses the python-pptx library to extract all attributes from the prediction ppt file and the label output file. then it uses the position relation checker to check if objects' positions conform to the label relation and the attribute content checker to check if the attribute's content is matched with the corresponding label attribute's content. the llm correctly completes the current turn's instruction if all attributes of the file pass these tests. we measure the performance of three closedsource llms (gpt-4, chatgpt, and davince-003) and six open-source llms (e.g., llama-2) in our benchmark. we further test planning (e.g., cot @xcite ) and content selection algorithms' performance based on gpt-4. evaluation metrics include turn-based accuracy as the ratio of correctly completed turns to the total number of turns and session-based accuracy as the ratio of correctly completed sessions to the overall session count. experiment results show that gpt-4 is the strongest llm among all llms but still encounters challenges when completing entire multi-turn sessions. for example, although gpt-4 achieves 75.1% turn-based accuracy in the creating new ppt file task, it only achieves 22.7% session-based accuracy as errors made in previous turns. gpt-4 and other llms also struggle to process long ppt templates (complex file environment). for example, gpt-4 only achieves 38.1% turn-based accuracy in the editing task. we further find that gpt-4 struggles to finish instructions involving non-text modality operations, especially for position-related operations, such as 'put object a on the top of the slide'. it only achieves 24% accuracy in these instructions. in summary, this paper has the following contributions: ## pptc benchmark in this section, we introduce our power-point task completion (pptc) benchmark, including the overview of our benchmark, its collection and validation process, and the pptx-match evaluation system for evaluation. we further analyze the statistics information of our benchmark. ## algorithms in this section, we introduce the algorithms we considered to enhance the llm's performance in our benchmark. these algorithms can be categorized into two approaches: planning algorithms that help the llm in decomposing the user instruction and solving it step by step and selection algorithms that assist the llm in choosing important environmental information or apis. ## analysis in this section, we analyze the reasons for gpt-4's errors. we further analyze the influence of model size and dialogue history. ## conclusion we introduce the powerpoint task completion benchmark to measure llms' ability to complete multi-turn user instructions within the context of the multi-modal powerpoint software. we further propose the pptx-evaluation system to access and compare the performance of 10 llms and 2 lvms. we further analyze the behavior of llms and find three main error factors that limit their performance. our benchmark and findings can help the research community design better ai assistants.