Free your mouse! Command Large Language Models to Generate Code to Format Word Documents (2024)

Abstract: Recently, LLMs have significantly improved code generation, making it increasingly accessible to users. As a result, LLM-powered code generation applications have sprung up, vastly boosting user productivity. This paper mainly explores how to improve the efficiency and experience of users in formatting the document. Specifically, we propose an automatic document formatting method, Text-to-Format, which is driven by various prompting strategies. Text-to-Format takes the user’s formatting instructions and then generates code that can be run in Microsoft Word to format the content in a document. Further, to evaluate automatic document formatting approaches and advance the document formatting task, we built an evaluation specification including a high-quality dataset DocFormEval data, a code runtime environment, and evaluation metrics. Extensive experimental results on data reveal that the prompting strategy’s effect positively correlates with how much knowledge it introduces related to document formatting task. We believe the constructed DocFormEval data and the exploration about Text-to-Format can help developers build more intelligent tools for automatic document formatting, especially in offline scenarios, where the data privacy is the top priority.

## introduction code generation @xcite @xcite aims to transform task requirements expressed in natural language into executable code. it improves the productivity of both non-expert users and developers. for example, text-to-sql @xcite enhances development efficiency for programmers while lowering the threshold for end-users to access the knowledge stored in the database. recently, the development of large language models (llms) @xcite @xcite has led to the innovation of the code generation domain. as a result, llm-powered code generation applications have sprung up, e.g., @xcite and opendevin @xcite , vastly boosting user productivity and experience. what other scenarios can benefit from code generation deserves to be explored. microsoft word, a globally popular word processing software, has a set of rich and powerful tools for document creation and editing. as depicted in figure 1 (a), it is utilized by individuals from various sectors, such as business and education, to compose project proposals, draft business contracts, write essays, make test papers, etc. after completing content creation, it is necessary to set the document format, which varies from scenario to scenario. we refer to the document formatting task as docform. to achieve this, microsoft word enables users to adjust document format using the mouse to select and click, including changing fonts, sizes, colors, spacing, etc. however, when formatting documents, especially complex ones, users often encounter numerous repetitive formatting operations, as well as the necessary format validation work, which can take up the user's creative time. to minimize formatting effort, previous work @xcite provides limited auto-formatting functions for specific requirements through preset templates. on the one hand, the increasing number of formatting requirements means that experts have to draft more templates, which is time-consuming and costly manually. on the other hand, the formatting standards vary according to the users' needs, making it almost impossible to address all possible variations. to free users from tedious and repetitive formatting work and allow them to focus on more challenging content creation, developing a more intelligent automated document formatting tool is imperative and valuable. inspired by the recent success of code generation, in this paper, we propose text-to-format, an automatic document formatting method powered by llms. specifically, as illustrated in figure 1 (b ), text-to-format takes in natural language formatting instructions i provided by the user and generates a formatting code c for the specified text t . upon executing the code c, the format of the specified text can be automatically adjusted as the user desires. it is noticed that existing code generation evaluation datasets, e.g., humaneval @xcite and ds-1000 @xcite , lack test instructions for the document formatting task. therefore, to better validate the feasibility of text-to-format, we first construct an evaluation dataset, docformeval, for the document formatting task. we make three efforts to improve the quality of docformeval and guarantee it is close to practical document formatting scenarios. first, we delve deep into microsoft word to gather 125 atomic formatting operations on font and paragraph levels and heuristically synthesize the initial input-output data d s . then, inspired by recent advancements in data generation @xcite @xcite , we utilize the powerful gpt-4 turbo to diversify the text formatting instructions in d s to align with human language representations. lastly, we manually review the rewritten data and discard low-quality examples. as llms-driven methods have dominated various code generation tasks, in this paper, we focus on exploring the impact of different prompt strategies on text-to-format performance. specif-ically, we first compare the performance of two methods, prompt learning @xcite @xcite , and retrievalaugmented generation (rag) @xcite @xcite , on docformeval in the zero-shot setting. furthermore, the impact of the number of demonstrations used in the prompting on the results is explored in the few-shot setting. finally, we explore the impact of the selfrefinement mechanism @xcite on automatic document formatting. our contributions are three-fold as follows: ## related work in-context learning for code generation recently, large language models (llms) @xcite @xcite have showcased remarkable knowledge transfer and logical reasoning abilities in few-shot or zero-shot scenarios @xcite . these capabilities attract many works that focus on exploring how to provide well-designed in-context examples for the current prompt, guiding llms to generate compliant code more effectively. studies @xcite attempt to add examples of input-output directly into the context. on the other hand, studies @xcite @xcite are inspired by the specialty of code generation itself, adding tutorials on how to self-debug code to in-context examples, thereby prompting llms to generate more robust code. retrieval-augmented code generation llms typically have difficulty covering all types of code due to their inherently outdated knowledge. ad-ditionally, the high frequency of code updates and iterations makes it costly to continually finetune llms based on new information. therefore, some research efforts are leveraging the concept of retrieval-augmented generation (rag) @xcite @xcite , where external knowledge relevant to the current prompt is provided to the llms as additional auxiliary cues. for instance, doc prompting @xcite utilizes a natural language to code generation method by retrieving code documentation. in addition, apiretriever @xcite introduces a framework designed to adapt llms to private libraries, leveraging an apicoder to generate code using these api docs. meanwhile, re-pocoder @xcite employs the iterative generate-retrieval procedure to do repository-level code completion. inspired by those works, this paper introduces rag to improve the text-to-format. ## conclusion this paper highlights the document formatting task (docform) in microsoft word, a valuable application scenario for code generation. for docform, we first propose text-to-format, an automatic document formatting algorithm driven by various prompting strategies. text-to-format takes a textual formatting instruction and then generates