I nstruct E xcel: A Benchmark for Natural Language Instruction in Excel (2023)

Abstract: With the evolution of Large Language Models (LLMs) we can solve increasingly more complex NLP tasks across various domains, including spreadsheets. This work investigates whether LLMs can generate code (Excel OfficeScripts, a TypeScript API for executing many tasks in Excel) that solves Excel specific tasks provided via natural language user instructions. To do so we introduce a new large-scale benchmark, InstructExcel, created by leveraging the ‘Automate’ feature in Excel to automatically generate OfficeScripts from users’ actions. Our benchmark includes over 10k samples covering 170+ Excel operations across 2,000 publicly available Excel spreadsheets. Experiments across various zero-shot and few-shot settings show that InstructExcel is a hard benchmark for state of the art models like GPT-4. We observe that (1) using GPT-4 over GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can help improve performance on this benchmark.

## introduction spreadsheet software, especially microsoft excel, is used every day by a diverse set of users (from complete novices to highly sophisticated power users) to solve a wide range of important tasks. to help users accomplish their tasks, excel has introduced an array of features, including analyze data 2 where users can specify their task in natural language (nl) for a limited scope of tasks. as even seemingly simple manipulations can often require knowledge of complex concepts like loops, conditionals, and regular expressions @xcite ˚work done during internship at microsoft : currently at google deepmind 1 instructexcel, along with the code used in our experiments, is available at https://github.com/ microsoft/instructexcel . meanwhile, the advent of large language models, together with the instruction learning paradigm @xcite @xcite , has paved the way for non-experts to accomplish many tasks just by providing instructions in natural language. can we leverage this novel paradigm to let excel users automatically execute complex tasks from nl instructions? excel officescripts foot_0 provide a versatile type-script api for most standard functions in excel, giving us a simple intermediate representation for executing nl instructions in excel. our task is to convert a user's description of a simple action or set of actions in excel (such as editing formatting) into an executable officescript accomplishing the action(s) on the given spreadsheet. figure 1 shows an example input and output, with additional examples in we introduce instructexcel, a benchmark to investigate the instruction paradigm for generating officescripts from nl instructions. we extract publicly available excel sheets and ask crowdworkers to think of an action they want to perform on the sheet and write a summary of it in nl. subsequently, we ask crowdworkers to execute that action in the sheet and use the automate feature to record the action and generate the underlying code. automatic code generation improves output quality and eliminates potential bugs prevalent in humanwritten code. instructexcel contains over 10k samples covering 170+ officescripts operations across 2,000 publicly available excel sheets. we evaluate openai's gpt-3.5 turbo and gpt-4 models @xcite as well as a t5 model @xcite , on instructexcel and find it to be a challenging benchmark even using state of the art tools, indicating significant room for future work on this task. we also find that gpt-4 improves over gpt-3.5, and dynamic prompting and more in-context examples improve performance, but additional in-context instructions do not help. in addition to evaluating capabilities of stateof-the-art models on the full benchmark, we also illustrate the broader utility of instructexcel through a case study on conditional formatting rules. instructexcel contains 660 examples that represent conditional formatting tasks, which are studied in isolation @xcite . viewed through the lens of this particular task, instruc-texcel provides a data source that can be further processed to enable experiments with taskspecific ground truth. many other sub-tasks can be extracted from instructexcel in the same manner, such as charting/plotting @xcite or formula creation @xcite . we believe instructexcel will encourage development of user-friendly methods to automate various excel tasks and help non-expert excel users in their daily workflow. ## instructexcel in this section, we describe the instructexcel benchmark. we start with the schema used to represent the task, followed by our dataset construction procedure, and an overview of the dataset statistics. ## experiment we conduct experiments with supervised, zero-shot and few-shot learning to assess the performance of popular llms on instructexcel. we use gpt-3.5 turbo and gpt-4 in our experiments (ope-nai, 2023), specifically the "gpt-3.5-turbo-16k" and "gpt-4-32k" models. 5 we also experiment with finetuning the t5 large lm adapt model @xcite . 6 in all experiments, a single example consists of a triplet with 1) a natural language query, 2) the data from the spreadsheet converted into a string using a pandas excelfile object, and 3) the ground truth or generated officescripts code. we elaborate below on our full experiment setup. ## results finetuning improves over in-context learning on automated metrics model performance after finetuning is notably higher than model performance in all other settings, when comparing on automated metrics. however, gpt-4+dp outperforms the finetuned t5 model when comparing manual annotation score. the relative performance gain of finetuning over in-context learning is therefore unclear, though both appear effective. ## case study: formatting rules in excel the instructexcel benchmark contains a wide range of excel tasks, including charting/plotting @xcite , formatting @xcite , and formulas @xcite . we can derive specific benchmarks for these individual downstream tasks from instructexcel. in this case study, we present an analysis of conditional formatting rules as a downstream task. to focus on the task of learning conditional formatting rules, we extract all examples from the instructexcel dataset that involve any formatting operator. these extracted samples are then manually validated to ensure their relevance to formatting tasks. we obtain a total of 660 valid cf tasks after filtering. to facilitate benchmarking, we transform the extracted conditional formatting tasks into an intermediate language (il) representation specifically designed for formatting operators @xcite . figure 6 showcases an example of this il. to evaluate the performance of various approaches on this task, we employed the original natural language (nl) descriptions associated with the tasks to generate simplified formatting rules. as these rules were already in a parsed format, we report the following evaluation metrics: (1) exact match, (2) sketch match over the ast of the parsed rules, and (3) execution match -verifying the equivalence of the resulting formatting by executing the generated rules on data and comparing the outcomes @xcite . ## conclusion we introduce instructexcel, a benchmark to investigate the capability of models in generating excel officescript code from nl instructions. in-structexcel consists of over 10,000 samples covering 170+ excel operations across 2,000 pub-