Interpreting User Requests in the Context of Natural Language Standing Instructions (2024)

Abstract: Users of natural language interfaces, frequently powered by Large Language Models (LLMs), must often repeat their full set of preferences each time they make a similar request. We describe an approach to LLM-based dialogue modeling in which persistent user constraints and preferences – collectively termed standing instructions – are provided as additional context for such interfaces. For example, when a user states “I’m hungry”, a previously expressed preference for Persian food can be automatically added to the LLM prompt, influencing the search for relevant restaurants.We develop NLSI, a language-to-program dataset consisting of over 2.4K English dialogues spanning 17 domains, in which each dialogue is paired with a user profile (a set of user-specific standing instructions) and corresponding structured representations (a sequence of API calls). A key challenge in NLSI is to identify which subset of the standing instructions is applicable to a given dialogue. NLSI contains diverse phenomena, from simple preferences to interdependent instructions such as triggering a hotel search whenever the user is booking tickets to an event. We conduct experiments on NLSI using prompting with large language models and various retrieval approaches, achieving a maximum of 46% exact match on API prediction. Our results demonstrate the challenges in identifying the relevant standing instructions and their interpretation into API calls

## introduction large language models (llms) such as gpt-3 @xcite , @xcite , and llama 2 @xcite are increasingly being used with tools and apis @xcite to provide additional functionality to users. for example, chatgpt allows several external plugins such as opentable for searching restaurants or expedia for booking travel. 2 these applications must learn to identify which service the user is seeking while respecting preferences across diverse domains that are unique to each user. understanding such preferences can aid in personalising the user experience by providing tailored responses, increased accuracy in recommendations and saving user time. however, in most cases, users must verbalise their preferences in detail during the interaction, including for repeated requests. past work has explored learning preferences from user-system interactions over time @xcite . these preferences can be hard to learn while also requiring significant amounts of training data. further, these learnt preferences are implicit and usually cannot be interpreted or edited by the user. we propose incorporating personalised standing instructions explicitly as additional context while interpreting a user's requests. standing instructions are user-provided natural language statements to change or prescribe system behaviour under certain circumstances. for example, in figure 1 , the user wishes to look for some nearby restaurants. in the absence of standing instructions, the user might have to interact for multiple turns with the system to arrive at their preferred restaurant cuisine and location. by looking up the relevant standing instructions for restaurants, the system can directly search for persian restaurants in san leandro, saving the user's time as well as providing customised/localised recommendations. explicit natural language instructions are also both controllable and interpretable. a user can inspect and edit their standing instructions, especially for preferences that change over time. further, the generated outputs can be directly linked to the relevant standing instructions, improving the user's trust in the system @xcite . our work is related to @xcite , which conditions a dialogue model's response on a set of developer guidelines. their work focuses on controlling response generation in open-domain dialogue systems with a focus on reducing toxicity and enhancing safety. more recently, commercial llm providers have introduced system prompts foot_1 /custom instructions foot_2 /preamble foot_3 which have an option to include guidelines at the beginning of every conversation to improve response generation. however, not much is known about how it operates, and no evaluations of its usage have been documented, especially in the task-oriented setting. this work makes the following contributions: (i) we systematically study the incorporation of standing instructions in a task-oriented setup. we develop and introduce nlsi (natural language standing instructions), an english-language dataset in which every example consists of a conversation between the user and a dialogue agent, accompanied by a collection of standing instructions (user profile) and a sequence of api calls reflecting user intents. (ii) we investigate six reasoning types for using standing instructions that range from a single instruction for a specific attribute to more complex situations such as the user proposing multiple preferences for same aspect, etc. these reasoning types introduce challenges pertaining to subset selection of relevant standing instructions and then inferring the structured api calls and their arguments. these include instructions that specify a single preference to more complex ones that involve multi-hop, cross-domain, and conflict reasoning. (iii) we use this dataset to benchmark a variety of methods involving the selection and interpretation of user utterances in the presence of standing instructions. we observe that our llm-based methods are far from perfect, raising new challenges in retrieval, reasoning, and semantic parsing. ## task overview we are interested in translating a user utterance into a sequence of api calls in the context of userspecific standing instructions (figure 1 ). consider a conversational context x, which consists of dialogue history between the user and the system (if any) and the user's current utterance. we assume a user profile u consisting of a sequence of natural language instructions u 1 , u 2 , ...u m . in this setting, instruction following consists of a selection task (which obtains a set of standing instructions z from the user profile u that are relevant to x) followed by an interpretation task (which predicts api calls y based on the conversational context and the relevant subset of standing instructions z). we assume access to a schema s that lists the valid api method names and their keyword arguments (slots). formally, an agent of this kind is described by a generative model: user: my schedule is free today and i plan to go to an event in seattle, wa. i want to look for events in that area. ## methods given the recent success of using llms to generate outputs in structured prediction tasks @xcite @xcite , we use an llm-based method to interpret a user utterance into a structured api call. we use incontext learning @xcite by providing k demonstrations, where k is tuned on the dev set. these demonstrations are obtained by retrieving examples from the training set that are most similar to the current dialogue of the test example using the bm25 similarity measure @xcite as in @xcite ; @xcite . the examples are arranged in a best-first order. we describe the different paradigms (figure 2 ) used for the interpretation task by selecting the instructions implicitly (direct interpretation), jointly (select-and-interpret) or explicitly (select-then-interpret). ## experiments we benchmark the dataset on the above methods to explain the various challenges on the benchmark. we used gpt-3.5 (text-davinci-003), gpt-4 as the base llms from gpt family. we use llama 2 (7b) for the selection task and codellama 2 (7b) for the interpretation task from the llama 2 family @xcite . ## conclusion we proposed the use of standing instructions -a set of natural language statements that contain the user's preferences to aid the interpretation of the user's requests. to facilitate this, we created nlsi, a language-to-program dataset based on sgd. this enabled us to explore two tasks: standing instruction selection and interpretation task of generating api calls which are conditioned on the selected instructions and conversational context. we experimented with several methods for the selection and interpretation tasks. our results show that while llms are capable of incorporating standing instructions as an additional context to an extent, their usage of standing instructions is far from perfect. the models struggled to select the instructions in the user profile that were relevant to the given dialogue, which in turn affected the interpretation task. moreover, as reasoning types become more intricate and involve complex reasoning or interactions among the respective standing instructions, the interpretation of these instructions becomes increasingly challenging for these methods. this calls for the development of new approaches in incorporating standing instructions, reasoning-based retrieval, and memoryaugmented representations.