An Improved, Strong Baseline for Pre-Trained Large Language Models as Task-Oriented Dialogue Systems (2025)

Abstract: Large Language Models (LLMs) have recently been studied within the context of Task-Oriented Dialogues (TOD). However, previous research is inconclusive on their effectiveness, with some studies claiming that LLMs are unable to perform the TOD task and others making sophisticated additions to their setup and coming to opposite conclusions. In this work, we take a detailed look at previous results that state LLMs perform insufficiently as a TOD system. As a result, we propose an updated, stronger baseline for multiple out-of-the-box LLM performances as TOD systems. We introduce a Self-Checking mechanism as a simple, yet effective, component to drastically improve their performance. Our results show that newer, pre-trained LLMs can, in fact, perform as TOD systems out-of-the-box, challenging the previous understanding. We show that LLMs can even perform competitively to fine-tuned models in certain metrics. Based on this, we propose directions for future research. Our code is published on Github.

## introduction tod systems are a special case of chatbots, where the system helps the user achieve a certain task. over the last years, they have undergone multiple evolutionary steps from rule-based multi-model approaches to end-to-end deep-learning based models. llms have shown remarkable out-of-the-box performance on many tasks, adapting to a task merely by prompting. naturally, they have also been investigated as tod systems, due to their strong natural language generation (nlg) and natural language understanding (nlu) abilities. we provide an updated, strong baseline for the general performance of pre-trained llms, i.e., without sophisticated mechanisms, on the multi-woz 2.2 @xcite ) benchmark dataset, even achieving comparable results to the fine-tuned sota in some cases. based on our empirical evaluation, we propose future directions for this field of research. to the best of our knowledge, we are the first to give an extensive overview across multiple llms on their baseline, zero-shot performance on the dst and response generation (rg) task. pre-trained, instruction-following large language models exhibit strong nlu and nlg capabilities, making them an obvious choice to utilize in the sense of a tod system. @xcite test multiple models on two tod benchmark datasets and come to the conclusion that they perform insufficiently. @xcite propose instruc-tods, a framework for using llms as zero-shot, end-to-end tod systems. later, @xcite frame dst as a function call and show improvement due to this. however, they also perform extensive post-processing to fix model output. similar to our baseline, @xcite build a dst system based on llm-inference. however, their inference requires similar, annotated examples from the dataset. @xcite also focus on dst and train one model to generate context-aware slot names and a second model for the dst. importantly, @xcite evaluate chatgpt in a comparable setting to ours. additionally, llms have been investigated for the sourcing of synthetic tods @xcite . ## method to assess the capacities of pre-trained llms to act as tod systems, we investigate multiple models from different model families in a unified setting. we build a simple pipeline based on related work @xcite ) that performs the full tod task in three sequential inference steps. first, we detect the domain of the user utterance. based on this, we select the state tracking prompt that includes the possible slots. then, the llm will track the state by updating the slot values. with this state information, we can query the database. lastly, we generate a response to the user based on the dialogue history, state prediction, and database result. the full pipeline is visualized in fig. 2 since the evaluation is based on string equality, the model needs to follow the instructions strictly so that the output can be parsed correctly. we thus introduce self-check after each model inference and evaluate its effect on the llm performance. we focus solely on the out-of-the-box performance of the llms and thus perform no finetuning. moreover, we conduct all experiments with two examples given in the prompt to allow in-context learning (zero-shot in the sense of not providing the ground-truth domain) and without the oracle belief state to employ a realistic scenario. we evaluate the dst and rg capabilities of each model using the standard tod metrics bleu, joint goal accuracy (jga), slot-f1 and success. first, we reproduce results by @xcite with tk-instruct-11b @xcite , opt-iml-30b @xcite and @xcite . second, we add new baselines for various models from the llama @xcite , qwen @xcite and @xcite families, to which we will refer as newer generation. while all of these models have seen wide-spread use, they differ in some characteristics besides the number of parameters. for example, llama 3.1 has been pre-trained on roughly 15 trillion tokens (met) and qwen-2.5 on 18 trillion tokens with a focus on knowledge, coding and mathematics @xcite . mixtral-8x7b-instruct uses a sparse mixture of experts (smoe) mechanism to speed up the inference @xcite . notably, we do not optimize the prompts for each llm but instead evaluate all models in a unified setup. ## results and discussion our empirical results suggest that newer generation llms can rival fine-tuned models with their out-of-the-box performance on the tod task. we found the models from the llama and mistral families to exhibit clearly improved performance due to the self-checking. for the qwen models, the improvement for state-tracking was negligible and only relevant for the bleu score. in contrast, the performance of prior-generation llms was negatively impacted in nearly all cases. qualitative analysis suggests that this can be attributed to these models having weaker instruction-following ability, as their output of the self-check is mostly unrelated to the original task. the findings of this study also indicate that the performance of a llm is not solely determined by its size. for instance, llama-3.1-8b-instruct outperformed the previous-generation models, even though it has fewer parameters than each of them. similarly, qwen-2.5-14b-instruct demonstrates better performance than llama-3.1-70b-instruct and llama-3.3-70b-instruct in three out of four metrics. nevertheless, the largest model, llama-3.1-405b-instruct, had the best overall performance when using self-checking. without it, it completely failed the task. this is due to the model returning mostly empty responses for the dst prompt, which was ill-suited for this specific model. since we did not optimize the prompts for each model, we saw the self-checking help in allowing models to perform well, even if the prompt is not optimized for it. it is striking that all llms exhibit very low bleu scores. this, however, is not due to the models generating lowquality responses. instead, the problem lies within the metric, which is reference-based and thus rewards imitating the linguistic style of the training dataset. previous studies have consistently shown human judges to highly rate llm-generated responses @xcite . dst task, especially when using fine-tuning. however, while impossible to ascertain, the probability of the llms having seen the multiwoz test data during their pretraining is very high, due to it being a common benchmark. fine-tuning the llm on the training data appears to make the knowledge of the test data emerge more clearly, thus making the model exhibit strong performance on the test data. this could be attributed to the weights being adapted to the multiwoz task, thereby allowing more direct access to the latent test data knowledge by generating multiwoz-specific text patterns more efficiently, consequently improving the performance. despite these promising results, using llms in this fashion has obvious drawbacks. their size requires large gpus, yet even then, without a large cluster and sophisticated performance optimizations the inference time can quickly become too long for real-world usage. for example, with the largest model, the full inference for one dialogue took up to 15 minutes on our hardware 2 , bringing the total inference time for the test dataset to 2 dgx a100 with eight 80gb gpus. ## future directions based on our empirical results, we propose to aim future research in new directions. naturally, improving upon state-of-the-art results remains a desirable objective. however, the opportunity for realworld usage needs to be taken into account, which can mean focusing on improving the performance of smaller, specialized models. furthermore, due to their extensive pre-training, llms could be a valuable asset for improving the out-of-distribution generalization of tod systems. lastly, hybrid approaches could be promising. specifically, using llms to create performant, efficient, fine-tuned tod models by data augmentation and / or knowledge distialltion. ## conclusion in conclusion, we provide a new, strong baseline for llm performance on multiwoz, the most prevalent tod dataset. we follow a basic pipeline that requires no sophisticated mechanisms and evaluate multiple models from different families in a unified setting. additionally, we propose self-checking as a simple, prior-free method to improve performance in many cases. we show that the open-weight model llama-3.1-405b-instruct comes close to and even surpasses some more sophisticated methods, suggesting the general ability to perform dst. however, due to its significantly smaller size but comparable performance, even without self-checking, qwen-2.5.-14b-instruct is a reasonable candidate for future research. our results indicate that the newer generation of llms clearly outperform previous models. this is true regardless of model size. we attribute this to crucially enhanced instruction-following. lastly, given the performance of the llms, we propose future research directions.