Evaluating Modular Dialogue System for Form Filling Using Large Language Models (2024)

Abstract: This paper introduces a novel approach to form-filling and dialogue system evaluation by leveraging Large Language Models (LLMs). The proposed method establishes a setup wherein multiple modules collaborate on addressing the form-filling task. The dialogue system is constructed on top of LLMs, focusing on defining specific roles for individual modules. We show that using multiple independent sub-modules working cooperatively on this task can improve performance and handle the typical constraints of using LLMs, such as context limitations. The study involves testing the modular setup on four selected forms of varying topics and lengths, employing commercial and open-access LLMs. The experimental results demonstrate that the modular setup consistently outperforms the baseline, showcasing the effectiveness of this approach. Furthermore, our findings reveal that open-access models perform comparably to commercial models for the specified task.

## introduction dialogue evaluation stands as a critical discipline within natural language processing (nlp), gaining heightened significance with the emergence of large language models (llm). the introduction of recent commercial and open-access models has transformed the landscape by enabling building dialogue applications where such models are finetuned to follow instructions @xcite . as these models showcase impressive capabilities in generating coherent and contextually relevant responses to the given prompts, evaluating their performance on multi-turn interactions (dialogue settings) requires a deeper look that goes beyond conventional metrics @xcite . recent advancements in the field suggested incorporating llms into self-contained modules that have certain task objectives (instructed by prompting (brown et al., 2020)) and placed in a simu-lated environment for benchmarking or frameworks for developing agents with llms @xcite form filling is one of the traditional tasks for conversational interfaces, and a whole markup scheme (voicexml) has been designed around it @xcite . in this paper, we want to explore how the capabilities of modern llms as "linguistically programmed linguistic processors" can address this task. we propose the idea of building a modular dialogue system for the special purpose of form-filling. typical form filling involves a user answering questions sequentially until all required fields are completed. our proposed idea is to decompose a larger task of form filling into smaller sub-tasks and assign a specific module to them. each module is realized through prompting a general purpose llm and is responsible for solving only the assigned sub-task. we do not program each module in a traditional programming language but instead use the "programming through prompting" approach based on llms. such a division of the overall task into multiple sub-tasks includes benefits such as handling the context window size limit since providing the complete form and all interactions in a single prompt text might lead to reaching the token limits of current llms @xcite ). another dimension is to rank various llms for their instruction-following abilities, such as form filling and its sub-tasks. the overview of how a modular dialogue system works for the form-filling task is illustrated in figure 1 . one module (chunk generator) is responsible for going through all fields and finding commonalities among them to create chunks. such commonalities can refer to questions that can be asked together, e.g., place of birth and date of birth, which can be asked in a single question, leading to shorter dialogues. such a way of filling out the form reduces the number of required turns for the 1. when is your birthday? 2. do you have siblings? 3. where were you born? ## modular dialogue setup our proposed methodology is to build a dialogue system capable of leading another participant to fill out any form. the dialogue system is built on top of multiple sub-modules acting on their restricted task descriptions. a module, in this context, is built by prompting an llm with the defined task description and is expected to follow the instructions in the task description. we divide modules into two distinct groups: interactive & task-performing modules. next, we explain each in detail. ## experimental setup in this section, we provide details about forms that are used for the experiments, compared baseline dialogue system and llms used as the basis for realizing the modular dialogue setup. ## results baseline setup: the results for the evaluated models are available in ## conclusion in this paper, we presented a study of evaluating dialogue systems built on top of llms with the task of form filling. the dialogue system is built by defining a prompt text that includes instructions on handling the given input. specifically, we focus on building such a dialogue system on llms by defining multiple sub-modules that are assigned a specific role to solve. our modular setup has been tested on the selected four forms from various topics and lengths by employing commercial and open-access llms. our experimental results suggest that the modular setup outperforms the baseline. another important finding is that open-access models are on par with the commercial model for the selected task. in future work, we plan to extend the modular dialogue system into multiple similar tasks, compare more commercial and open-access models, and perform the study that involves users filling out the forms to compare against the proposed method. hugo touvron, louis martin, kevin stone, peter albert, amjad almahairi, and et al. 2023. llama 2: open foundation and fine-tuned chat models. corr, abs/2307.09288. yue wu, xuan tang, tom m. mitchell, and yuanzhi li. 2023. smartplay : a benchmark for llms as intelligent agents. corr, abs/2310.01557. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in proceedings of the 56th annual meeting of the association for computational linguistics, acl 2018, melbourne, australia, july 15-20, 2018, volume 1: long papers, pages 2204-2213. association for computational linguistics. xuhui zhou, hao zhu, leena mathur, ruohong zhang, haofei yu, zhengyang qi, louis-philippe morency, yonatan bisk, daniel fried, graham neubig, and maarten sap. 2023. sotopia: interactive evaluation for social intelligence in language agents.