LLM -based Frameworks for API Argument Filling in Task-Oriented Conversational Systems (2024)

Abstract: Task-orientated conversational agents interact with users and assist them via leveraging external APIs. A typical task-oriented conversational system can be broken down into three phases: external API selection, argument filling, and response generation. The focus of our work is the task of argument filling, which is in charge of accurately providing arguments required by the selected API. Upon comprehending the dialogue history and the pre-defined API schema, the argument filling task is expected to provide the external API with the necessary information to generate a desirable agent action. In this paper, we study the application of Large Language Models (LLMs) for the problem of API argument filling task. Our initial investigation reveals that LLMs require an additional grounding process to successfully perform argument filling, inspiring us to design training and prompting frameworks to ground their responses. Our experimental results demonstrate that when paired with proposed techniques, the argument filling performance of LLMs noticeably improves, paving a new way toward building an automated argument filling framework.

## introduction task-oriented conversational systems, illustrated in figure 1 , largely consist of three processes: external api selection, argument filling, and response generation @xcite . the api selection phase selects which one from the predefined pool of apis must be called to complete the user request. once the appropriate external api to carry out the user request has been selected, the argument filling phase must reliably identify and provide correct arguments to the api by faithfully following the api schema and dialogue history. an api schema, an example of which is also demonstrated in figure 1 , is typically assumed to be given as a part of the api and includes required arguments and their types. therefore, the api schema and dialogue history provide sufficient information for the conversational agent to identify which arguments are necessary to complete the api call. lastly, the response generation phase, as the name suggests, returns an appropriate response to the user based on the api output. the user dissatisfaction in argument filling mainly stems from the conversational agent being incapable of adhering to the api schema and dialogue history. the erroneous arguments that digress away from the api schema are considered "syntax errors", and hallucinated responses that deviate from the user utterances are considered "hallucinations." in figure 2 , we provide examples of each error type that occurs when performing argument filling for the "hair appointment" api. large language models (llms) trained with instructions have recently been garnering much attention as a promising model for enabling human-like and safe user-agent interactions in open-domain conversations @xcite . the aim of this paper is to explore whether the strength of llms can be harnessed specifically for the purpose of argument filling in taskoriented conversational systems. to construct an llm-backed framework for argument filling, their outputs must strictly follow and stay faithful to the pre-defined api schema and user utterances, a process commonly known as "grounding." our initial zero-shot performance evaluation of llms of various sizes reveals that llm-generated responses suffer severely from both syntax errors and hallucinations, necessitating the development of additional techniques to appropriately ground their responses for the task of our interest. we investigate two separate and unique avenues to tackle the problem of grounding for open-and hello, i would like to get a haircut this friday at 3pm. please book an appointment at the nearest salon. 1) api selection • party plan • hair appointment • ride book • door unlock • …. ## conclusion this paper explored and uncovered the powerfulness of leveraging llms to automate the argument filling process, a core component in task-oriented conversational systems. the strong experimental results indicate that the proposed methods, used in conjunction with open-or closed-source llms, are effective for restricting the llm's generative behavior, specifically for argument filling.