Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization (2024)

Abstract: How-to procedures, such as how to plant a garden, are now used by millions of users, but sometimes need customizing to meet a user’s specific needs, e.g., planting a garden without pesticides. Our goal is to measure and improve an LLM’s ability to perform such customization. Our approach is to test several simple multi-LLM-agent architectures for customization, as well as an end-to-end LLM, using a new evaluation set, called CustomPlans, of over 200 WikiHow procedures each with a customization need. We find that a simple architecture with two LLM agents used sequentially performs best, one that edits a generic how-to procedure and one that verifies its executability, significantly outperforming (10.5% absolute) an end-to-end prompted LLM. This suggests that LLMs can be configured reasonably effectively for procedure customization. This also suggests that multi-agent editing architectures may be worth exploring further for other customization applications (e.g. coding, creative writing) in the future.

## introduction ai is headed towards a future where humanmachine interactions are seamlessly integrated to enrich our daily routines, offering personalized and tailored experiences @xcite . for instance, a software engineer's daily routine would involve a co-pilot that customizes the same underlying logic differently for two engineers (even though we observe the structure might broadly remain the same). another application is smart assistant and planners that can customize how-to procedures, a popular query making up a large fraction of search engine queries @xcite , based on a user's specifications (though the sequence of steps broadly stay the same). for example, a user looking for "how to plant a garden", may have space restrictions in their apartment, or not want to use pesticides. despite the need for customization, it is challenging to author new customized how-to procedures for every users' nuanced needs. automatically customizing procedures requires the interpretation of the nuanced user needs expressed in natural language @xcite . these customization requests or hints can take up various forms e.g., constraints ("i live in an apartment"), personal preferences ("i prefer organic farming"), or execution method ("plant a hydroponic garden"). these implicitly encode multiple requirements and their interpretation is subjective -for example, living in an apartment can entail a lack of space, limited space, convincing roommates, and more. contemporary approaches to customization focus on constraints in specific domains @xcite . llms could be considered a strong baseline for faithfully customizing procedures to different users' needs @xcite , and we did find that the generated customized procedures are fluent and coherent. however, in our experiments §2.2, we found that ~60% of the procedures generated by contained errors (missing steps, extra steps, wrong steps, underspecified steps), making the output inadequately customized or inexecutable, as shown in figure 1 . we observe that even though uncustomized and customized procedures share some inherent structure, end-to-end systems disregard that and produce entirely new structures which introduces unwanted changes. ## models in this section, we describe our multi-agent approach for customizing procedures for users' needs using semi-symbolic edits. we disentangle the task of generating customized procedures into two aspects, customizability and executability. ## experiments to tailor procedures according to customization hints, one needs to make implicit inferences out of the hints, identify the steps that require changes, and finally consistently apply changes to the different steps. through our experiments, we aim to understand how well models can modify generic procedures to incorporate aspects captured in customization hints. ## analysis we analyze different aspects of the sequential setting and its generated plans. ordering of agents matter. in the sequential setting, we flip the order of the modify agent and verify agent , as illustrated in figure 8 . we find that this approach only generates usable customized procedures 34.47% of the time. while unintuitive, this setting addresses the lack of detail in some wikihow procedures while also correctly making direct changes for customization. ## related work over the past few years, as information about users has been readily available, user-facing technology has become increasingly customized. customization has been widely studied in the context of such technology, like search engines @xcite and game content @xcite . there is some work on customized procedural content generation @xcite . however, they focus on video game content rather than everyday procedures related to pragmatic goals. previous research has shown the value of customization in various settings. @xcite present a method to customize collaborative plans for robot-assisted dressing. building customized care plans is crucial for patients with severe illnesses @xcite . procedural text understanding addresses the task of tracking entity states throughout the text @xcite . generating such texts @xcite involves different types of understanding such as temporal reasoning and entity linking. @xcite customized models are designed to capture language patters specific to individual users. @xcite examined methods for creating customized lms using interpolating, fine-tuning and priming. similarly, @xcite present another approach for fine-tuning and interpolation to customize lms. llms have also been used to study constrained planning @xcite and new interfaces for personalization @xcite . role-based prompting is a recent trend. pseudo dialog prompting @xcite is a method to build prompts for llms so that chatbots mimic fictional characters, while @xcite release a dataset of character annotations to induce personas in llm output. however, it is still unclear how to encapsulate more fine-grained aspects of customization. ## conclusion this paper studies the capabilities of current llms to customize open-domain procedures. first , we show that current llms cannot effectively customize open-domain procedures. next, we propose several llm-based agent architectures, and find that sequentially using semi-symbolic edits from the modify agent and the verify agent provides an improvement of 10.5% in generating fully correct procedures over naive prompting. even though we show that using edits helps executability, we find that generated procedures are often not sufficiently customized, and there is clear room for improvement. finally, we discuss the generalizability of our framework for other diverse applications such as coding and creative writing that require customization.