DIALIGHT : Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models (2024)

Abstract: We present DIALIGHT, a toolkit for developing and evaluating multilingual Task-Oriented Dialogue (ToD) systems which facilitates systematic evaluations and comparisons between ToD systems using fine-tuning of Pretrained Language Models (PLMs) and those utilising the zero-shot and in-context learning capabilities of Large Language Models (LLMs). In addition to automatic evaluation, this toolkit features (i) a secure, user-friendly web interface for fine-grained human evaluation at both local utterance level and global dialogue level, and (ii) a microservice-based backend, improving efficiency and scalability. Our evaluations reveal that while PLM fine-tuning leads to higher accuracy and coherence, LLM-based systems excel in producing diverse and likeable responses. However, we also identify significant challenges of LLMs in adherence to task-specific instructions and generating outputs in multiple languages, highlighting areas for future research. We hope this open-sourced toolkit will serve as a valuable resource for researchers aiming to develop and properly evaluate multilingual ToD systems and will lower, currently still high, entry barriers in the field.

## introduction task-oriented dialogue (tod) systems are designed to model interactions between human users and system agents, focusing on accomplishing specific, predefined tasks such as assisting with hotel or restaurant bookings, or providing domainspecific faq information @xcite @xcite . these systems serve not only as access points to cutting-edge ai applications but also as drivers of technological expansion. the prevailing approach in tod system development has predominantly involved fine-tuning pretrained language models (plms), like t5 @xcite and bart @xcite , on task-specific dialogue datasets @xcite . however, recent research trends indicate a paradigm shift from finetuning to an increased reliance on large language models' (llms) inherent capacity for in-context learning and generalisation for natural language understanding and generation. in our work, we categorise systems as fine-tuned-based using plms (hereafter ft-based) or in-context-learning-based using llms (icl-based), noting that llms can be fine-tuned and smaller plms can use icl. 1 several pilot works @xcite @xcite have explored the integration of llms into tod systems. these studies indicate that ft-based approaches outperform icl-based approaches, as evidenced by superior automatic evaluation scores. this applies even with smaller plms and when the number of training examples is limited. on the other hand, instruction-based training of llms demonstrates its potential in aligning model outputs more closely with human preferences @xcite . in tod, icl-based systems @xcite have been shown to generate responses that exceed previous models in critical human evaluation dimensions such as informativeness, helpfulness, and perceived humanness. despite these early results, a systematic, comparative human evaluation of these two approaches in tod systems remains a gap in current research. 2 ## related work dialight represents a novel addition to the landscape of tod system toolkits, complementing existing frameworks such as pydial @xcite , convlab-3 @xcite , and their antecedents @xcite , as shown in another range of publicly available implementations exists for both traditional ft-based and iclbased e2e tod systems, typically accompanying research publications as supplementary code. for traditional ft-based approaches, works include those by @xcite ; @xcite ; @xcite ; @xcite ; eric et al. (2017); lin et al. (2020); peng et al. (2021) ; @xcite . the icl-based category is exemplified by the work of @xcite . 3 however, these systems lack a unified setup for comparative experimentation. our toolkit fills this gap, enabling fair and comprehensive comparisons between the aforementioned two types of systems. additionally, many of these systems, primarily designed to enhance benchmark results, exhibit key limitations, such as: 1. absence of implementation for lexicalisation in utterances, 2. lack of a dialogue system agent for real user interaction, 3. existence of english-centric heuristics in evaluation, foot_3 4. inadequate support for human evaluation. dialight specifically addresses these limitations. while existing tools for human evaluation, such as dialcrowd @xcite , offer simple approaches for conducting human evaluation experiments in tod systems, they are not without limitations. dialcrowd is not opensourced, available solely via its designated website, which imposes significant constraints. these include limited customisation flexibility, increased maintenance complexity, 5 and challenges in aligning with data protection regulations such as gdpr, thus affecting its broader applicability in research. in contrast, our human evaluation tool is opensourced, enabling 'one-click' deployment on local or cloud servers. ## system architecture this section delineates the architecture and implementation of the proposed e2e dialogue system within dialight. as illustrated in figure 2 in the appendix, despite the term 'end-to-end', stateof-the-art e2e tod systems typically employ a pipelined approach in the background, incorporating three key components: 1) a dialogue state tracking (dst) model, 2) a database interface, and 3) a response generation (rg) model. in the following, we describe the system pipeline and provide an indepth examination of each constituent component. ## automatic evaluation as part of dialight, we have implemented a range of automatic evaluation metrics: 1. for dst evaluation, metrics include joint goal accuracy (jga), slot f1, slot recall, and slot precision. 2. for response generation evaluation, we utilize the bleu score @xcite , rouge-l @xcite , and meteor (banerjee and @xcite . 3. for evaluating the overall systems, we report inform rate, success rate, and bleu. additionally, we provide an interface to facilitate easy extension for future additional metrics. ## human evaluation in dialight, we offer an open-sourced human evaluation tool specifically designed for tod systems. this tool is comprehensive, providing all essential functionalities to conducting human evaluation experiments in a production setting. in the following, we detail the features supported by our web interface, provide an overview of the highlevel design of our backend servers, and present a case study of human evaluation experiments. ## conclusion and outlook we introduced dialight, a comprehensive toolkit for advancing multilingual tod systems supported by language models of different families, offering the essential infrastructure for system development and evaluation. dialight supports the development of and includes baseline systems for finetuning and in-context learning paradigms, enabling comparative experiments within a unified setup for automatic and human evaluation. we have publicly released all our source code to facilitate future research and invite the research community to adapt and contribute to this toolkit. utilising this toolkit, we executed a performance analysis across several languages and models for the two modeling paradigms. the results indicate that despite their potential, llms, even the most powerful ones, are far from 'solving' the tod task, especially in a multilingual context. instead, our results open up new avenues for future research and exploration. in appendix b, we provide some examples of future work aimed at addressing the bottlenecks identified in the baseline systems.