Dynamic Task-Oriented Dialogue: A Comparative Study of Llama-2 and Bert in Slot Value Generation (2024)

Abstract: Recent advancements in instruction-based language models have demonstrated exceptional performance across various natural language processing tasks. We present a comprehensive analysis of the performance of two open-source language models, BERT and Llama-2, in the context of dynamic task-oriented dialogues. Focusing on the Restaurant domain and utilizing the MultiWOZ 2.4 dataset, our investigation centers on the models’ ability to generate predictions for masked slot values within text. The dynamic aspect is introduced through simulated domain changes, mirroring real-world scenarios where new slot values are incrementally added to a domain over time.This study contributes to the understanding of instruction-based models’ effectiveness in dynamic natural language understanding tasks when compared to traditional language models and emphasizes the significance of open-source, reproducible models in advancing research within the academic community.

## introduction in recent years, the landscape of natural language processing (nlp) has witnessed a shift towards leveraging instruction-based models, marking a departure from traditional approaches. these instruction-based models have demonstrated exceptional performance across a diverse range of complex tasks that were traditionally deemed challenging for automated solutions. unlike closedsource linguistic models, typified by industry leaders such as openai, which have dominated the market, we observe a growing interest in open-source alternatives. the inherent transparency and reproducibility of open-source models provide a conducive platform for academic research, fostering valuable experiments in diverse domains. numerous studies have already assessed the effectiveness of open-source instruction-based models across various natural language processing (nlp) tasks. several notable examples include research efforts focused on fine-tuning llama-2 for diverse applications, such as question answering and text summarization in the medical domain @xcite . additionally, investigations have been conducted using opt to generate synthetic dialogues in social contexts @xcite . other studies have compared the performance of open-source models like alpaca-lora with proprietary alternatives, specifically in the realm of dialogue state tracking @xcite . furthermore, there have been assessments of llama's performance in responding to user instructions within real-world scenarios @xcite . one critical challenge in real-world applications is the dynamic nature of domains, where constant changes necessitate adaptations in dialogue systems. previous studies @xcite have shown how this domain shifts significantly deteriorate the performance of models trained on outdated data. with the emergence of instruction-based models, we aim to explore their efficacy in addressing this challenge compared to traditional models. in this study, we focus on the task of dynamically substituting slot values for masked entities in task-oriented dialogues. this becomes particularly crucial in scenarios where domains evolve, prompting changes in slot values (e.g., a restaurant transitioning from offering "indian" to "italian" cuisine). we conduct a comparative analysis involving llama-2 @xcite , a state-of-the-art open-source instruction-based model, and bert @xcite , a traditional open-source language model. figure 1 provides an illustrative example, depicting an original dialogue with masked slot values (a), a dialogue with values generated by llama to replace the masks (b), and a dialogue with values generated by bert for the same task (c). the primary contributions of this paper can be summarized in three key points: (i) providing comparative analysis of two popular open-source language models, llama-2 and bert, in the specific task of generating substitutes for masked slot values; (ii) assessing model performance in dynamic contexts, where slot values undergo changes, with insights into how well llama-2 and bert deal with real-world scenarios marked by evolving information; (iii) systematically investigating the impact of fine-tuning on model behavior, drawing attention to distinct strategies applied to llama-2 and bert, and offering valuable observations on model adaptability under varying conditions during inference. the paper is structured as follows: section 2 offers background insights into relevant topics discussed in the paper; section 3 outlines the methodology employed for our task, emphasizing the nuances of the dynamic slot value generation; section 4 details the experimental settings, specifically the introduction of domain changes; section 5 describes the evaluation metrics utilized for a comprehensive assessment; section 6 presents the experimental results; finally, section 7 provides a comprehensive discussion of the findings and their implications for instruction-based and traditional language models in dynamic contexts. ## methodology in this section, we outline the methodology employed to evaluate the performance of bert and llama-2 for the task of substituting slot values in a dialogue in the context of dynamic domain changes. the primary task involves masking specific slot values in the utterances of a dialogue, both in user and system turns and assessing how well language models can generate appropriate substitutions for these masks. ## discussion the results of our experiments reveal several interesting patterns and insights into the performance of instruction-based language models such llama-2, in comparison to traditional language models like bert. the most apparent trend is observed in llama-2's performance, starting with a higher accuracy in the no-change scenario and gradually test set bert llama-2 0 sol 1 sol 2 sol 3 sol 0 sol 1 sol 2 sol 3 sol the comparison becomes more nuanced when domain changes are introduced. in most metrics, llama-2 demonstrates significantly better performance than bert across various scenarios, suggesting that even without specific fine-tuning, instruction-based models have better performance than traditional language models finetuned for the specific domain setting. in scenarios such as "dialogue incoherence", this outcome is expected, given that bert solely replaces individual masks and lacks awareness of the dialogue's evolution, so that it becomes impossible for it to preserve coherence throughout the dialogue. however, there are instances where bert outperforms llama-2, particularly for the metrics "wrong slots "and "kb quantifiers misalignment". in the case of "wrong slots", the constraint imposed on llama-2 by instructions to extract values from the kb may lead to more instances of values being assigned to the wrong slot. for "kb quantifiers misalignment", the observed difference could be attributed to quantity values where dialogues necessitate indicating zero instances, possibly due to errors in the preceding part of the dia-logue. bert is more inclined to generate a value of zero in such cases (as in the example in figure 1 ), whereas llama tends to avoid failure examples and always say that there is at least one restaurant available. the manual evaluation provides valuable insights that complement the quantitative results obtained from the automatic assessments. notably, neither of the two models successfully assigns correct slot values to dialogues expecting 0 solutions in the kb. in these cases, models are expected to generate out-of-kb values; while they occasionally do, resulting in seemingly coherent dialogues, errors often manifest in subsequent turns. it is essential to acknowledge that generating out-of-kb values is not always indicative of an error. this observation extends to the "wrong slot" measure, where values substituted for one slot type may correspond to another slot type yet remain acceptable within the utterance. similarly, the metric for "dialogue incoherence" occasionally misclassifies cases as incorrect during automated assessment, which are instead considered correct in manual evaluation. for instance, instances where the model generates "north" after previously stating "northwest" could be technically correct, as "northwest" inherently implies "north." a similar situation arises with terms like "affordable," "economic," "moderate," and "reasonable," which may be considered synonymous but are treated as distinct values in the automated measure. additionally, a noteworthy observation is the significant difficulty observed in bert's ability to generate restaurant names, which instead tends to substitute values like "it" or "that" pronouns. overall, our study sheds light on both strengths and weaknesses of instruction-based language models like llama-2 as compared to traditional models like our study has a number of limitations that should be taken into consideration when interpreting the results. firstly, we utilized the smallest variant of llama-2, with 7 billion parameters. it is plausible that larger versions of llama-2 could yield improved performance. secondly, our experiments were conducted exclusively on the multiwoz 2.4 dataset, focusing specifically on the restaurant domain and only considering informable slots. consequently, the generalizability of our findings to other task-oriented dialogue scenarios, domains, or datasets may be limited. furthermore, in the case of llama, we did not extensively explore the impact of using different prompts. the potential influence of varied prompts on performance remains an area that requires further investigation, and it is plausible that alternative prompts could lead to more favorable outcomes. finally, the comparison between instructionbased models and traditional language models in this study was restricted to two specific models -bert and llama-2 -each configured with specific parameter settings. therefore, caution is advised when attempting to generalize these findings to a broader range of models and contexts. ## conclusion and future work this study delved into the performance analysis of two language models, namely bert and llama-2, focusing on their ability to generate substitutions for masked slot values in task-oriented dialogues. the experimentation was grounded in the dynamic context of domain changes, simulating scenarios where new slot values are introduced. we used the multiwoz 2.4 dataset, specifically concentrating on the restaurant domain and informable slots. our methodology involved finetuning llama-2 only on the zero-changes scenario, while bert was finetuned for each change scenario. llama-2 demonstrated superior performance in the no-change scenario, emphasizing the efficacy of instruction-based models with consistent domain information, but faced a significant decline with increasing domain changes, ultimately falling below bert for some of the considered quality features. we highlighted strengths and weaknesses of both approaches in dynamic task-oriented dialogue scenarios. as emphasized in section 8, there is potential for improvement by incorporating diverse models and datasets (such as sgd @xcite ). to address this, future research will delve into larger versions of llama-2, explore alternative models, and incorporate varied datasets. additionally, the investigation will consider the use of diverse prompts, including slot descriptions as seen in @xcite , to enhance our comprehension of instruction-based models within dynamic contexts.