Large Language Models as Zero-shot Dialogue State Tracker through Function Calling (2024)

Abstract: Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. However, their effectiveness in task-oriented dialogues (TOD), which requires not only response generation but also effective dialogue state tracking (DST) within specific tasks and domains, remains less satisfying. In this work, we propose a novel approach FnCTOD for solving DST with LLMs through function calling. This method improves zero-shot DST, allowing adaptation to diverse domains without extensive data collection or model tuning. Our experimental results demonstrate that our approach achieves exceptional performance with both modestly sized open-source and also proprietary LLMs: with in-context prompting it enables various 7B or 13B parameter models to surpass the previous state-of-the-art (SOTA) achieved by ChatGPT, and improves ChatGPT’s performance beating the SOTA by 5.6% average joint goal accuracy (JGA). Individual model results for GPT-3.5 and GPT-4 are boosted by 4.8% and 14%, respectively. We also show that by fine-tuning on a small collection of diverse task-oriented dialogues, we can equip modestly sized models, specifically a 13B parameter LLaMA2-Chat model, with function-calling capabilities and DST performance comparable to ChatGPT while maintaining their chat capabilities. We have made the code publicly available at https://github.com/facebookresearch/FnCTOD.

## introduction recent years have seen the rapid development of large language models (llms) that have demonstrated exceptional natural language understanding and generation capabilities. the integration of llms into industry applications, particularly as 1 https://github.com/facebookresearch/fnctod †work undertaken while interning at meta. *correspondence authors: zekunli@ucsb.cs.edu, and pacrook@meta.com. conversational assistants, is a notable trend. finetuned with conversations between users and assistants, these models are further aligned with human preferences to enhance their ability to deliver fluent, helpful, and polite responses to user inquiries. notable examples include proprietary systems such as chatgpt foot_0 and claude foot_1 , as well as open-source models such as llama2-chat @xcite , vicuna @xcite , baichuan @xcite . the primary focus of these chat-tuned llms has typically been on responding in general contexts. however, for another important type of conversation, task-oriented dialogues (tod), the model is required to extract the intentions of users at each turn of the conversation, represented as slot-value pairs of per-domain predefined schemas; a process known as dialogue state tracking (dst). the challenge lies in the model's ability to accurately summarize user needs over multiple turns of conver- ## approach our method redefines dst as function calling, treating each domain as a distinct function, and the slot values within the domain as its arguments. as shown in figure 2 , this paradigm is represented in chat-tuned models by embedding function specifications within system prompts, as shown in figure 3 . the model is tasked with generating function calls followed by a response, as shown in figure 4 . we provide more details of our approach below. ## conclusion we introduce a new approach to tackle the challenging task of zero-shot dst with llms, enabling them to handle both general conversations and task-oriented dialogues in diverse domains without the need for additional data collection. our experimental results on multiwoz demonstrate that our approach not only delivers exceptional performance in advanced chatgpt models (setting a new benchmark) but also across a range of moderately sized open-source llms. furthermore, we demonstrate that we can fine-tune the open-source model llama-2-13b-chat using only 7,200 training samples from 36 diverse domains, resulting in fnctod-llama2-13b, which achieves function calling, zero-shot dst performance comparable to chatgpt. 7 limitations in this work, we propose a novel approach to solve zero-shot dst with llms. our approach achieves outstanding performance with various llms, both modestly-sized open-source and advanced proprietary llms, setting the new state-of-the-art. however, it is important to recognize that the current accuracy may still not be good enough for the practical deployment of such zero-shot systems. we anticipate that with further advancements in the nlu and nlg capabilities of base llms, our approach could achieve even greater performance levels. in addition, our approach can handle both the dst and response generation task in tod. we evaluate dst with the well-established metric jga with results suggesting the strong zero-shot dst performance of our approach. for the response evaluation, due to the current lack of a more realistic evaluation setting for response generation in tod, we evaluated delexicalized responses as this is widely used in prior work. this setting and associated metrics have some known shortfalls in terms of being able to game-the-metrics with nonnatural responses as well as presenting a data mismatch with how llms are trained. in the era of llms, we advocate for the development of more realistic evaluation approaches for full-natural-languageresponse generation in tod. additionally, while this work concentrates dst and response generation, the two critical tasks in tod, our approach can also be extended to include other complex tasks and handle various scenarios in dialogues as in the langchain framework. we plan to explore these extensions in future research. for instance, a delexicalized response might be: "it is a restaurant that serves [value_food] food near the @xcite . the address is @xcite and the phone number is @xcite ." this strategy facilitates easier automatic determination of whether the dialogue answers specific requests by the conversation's end. if both, all entities of the type expected are mentioned, and provision of all required information about those entities is also detected, then the dialogue is considered successful @xcite . this results in a binary "success" metric per dialogue. regardless of the user's input, the evaluation is on whether the entities and information about those entities provided to the user by the end of the conversation satisfies their goal. for example, given a user goal at the beginning of a dialogue to obtain the phone number of a restaurant serving french cuisine, the dialogue is marked as a success if by the end the system has mentioned at least one restaurant and provided a restaurant phone number. while jga indicates whether the correct search for a restaurant serving french food was attempted. training data for constructing our fine-tuning dataset, we selected five high-quality, multi-turn tod corpora, excluding multiwoz, as detailed in hyperparameters we fine-tuned the llama-2-13b-chat checkpoint from hugginface. foot_3 . we utilize low rank approximation (lora) @xcite and limited our fine-tuning to the parameters in the q_proj and v_proj modules. further details about the fine-tuning hyperparameters can be found in