Lexical Entrainment for Conversational Systems (2023)

Abstract: Conversational agents have become ubiquitous in assisting with daily tasks, and are expected to possess human-like features. One such feature is lexical entrainment (LE), a phenomenon in which speakers in human-human conversations tend to naturally and subconsciously align their lexical choices with those of their interlocutors, leading to more successful and engaging conversations. As an example, if a digital assistant replies “Your appointment for Jinling Noodle Pub is at 7 pm” to the question “When is my reservation for Jinling Noodle Bar today?”, it may feel as though the assistant is trying to correct the speaker, whereas a response of “Your reservation for Jinling Noodle Baris at 7 pm” would likely be perceived as more positive. This highlights the importance of LE in establishing a shared terminology for maximum clarity and reducing ambiguity in conversations. However, we demonstrate in this work that current response generation models do not adequately address this crucial human-like phenomenon. To address this, we propose a new dataset, named MultiWOZ-ENTR, and a measure for LE for conversational systems. Additionally, we suggest a way to explicitly integrate LE into conversational systems with two new tasks, a LE extraction task and a LE generation task. We also present two baseline approaches for the LE extraction task, which aim to detect LE expressions from dialogue contexts

## introduction there is potential for enormous variability in people's lexical choices in dialogues, whether they are communicating with human or machine partners @xcite . one question that has attracted some attention from the linguistics community has been: how do people deal with such high lexical variability when they talk to each other? in 1 the dataset is available at https://github.com/ zhengxiangshi/lexicalentrainment . the referential communication studies @xcite , the likelihood that people in one conversation would choose the same terms to refer to the same common object as people in another conversation was only 10%. despite the high variability across conversations, it is relatively low within a conversation @xcite . when people engage in a conversation, they naturally adapt their way of speaking to align with their conversational partner. for instance, they tend to refer to something based on how their conversational partners refer to it, using the same terms when discussing the same object repeatedly and negotiating a common description, particularly for items that may be unfamiliar to them @xcite . this linguistic phenomenon is known as lexical entrainment (le) @xcite . le plays a key role in the success and naturalness of interactions in conversations. @xcite demonstrated that lexical repetition is a reliable predictor of task success given the first five minutes in task-oriented dialogues. the degree of entrainment with respect to the most frequent words is also a distinguishing factor between dialogues rated as most natural and those rated as less natural, and is strongly correlated with task success and engaged and coordinated turn-taking behaviour @xcite . additionally, le is associated with a broad range of positive social behaviours and outcomes @xcite , such as building effective dialogues @xcite and engaging students in tutorial dialogues @xcite . in conversational systems, an explicit mechanism of lexical choice (e.g., by le) is needed to ensure meaningful results @xcite , thus making conversations more engaging and successful. despite its importance, le has not been systematically studied for conversational systems, and this interesting fact of language-based communication has been overlooked. there is a scarcity of datasets specifically designed to study le, and it is not explicitly modelled in any state-of-the-art conversational systems. the only existing dataset for lexical entrainment, presented by @xcite , consists of 1 859 pairs of user utterances, system responses, and dialogue acts. however, it is limited to the context of a single preceding user utterance and constrained to the public transportation domain. most importantly, this dataset lacks explicit annotations of le, making it difficult to be used to explicitly model le in conversational systems. to this end, we propose a new dataset named multiwoz-entr with annotated le information, comprehensive dialogue context, and detailed statistical analysis. the dataset is built on the top of the multiwoz @xcite , a task-oriented dialogue dataset with 7 distinct domains. mul-tiwoz is selected as a starting point for two reasons: (1) it is a task-oriented dataset, which tends to contain more le expressions than other kinds of dialogue datasets @xcite , such as open-domain datasets (e.g., @xcite ; and (2) it is collected based on humanto-human conversations. additionally, we present a workflow of a conversational system integrated with le modules, as illustrated in figure 1 . a standard pipeline architecture for conversational systems generally decomposes into three modules, a natural language understanding (nlu) module, a dialogue management (dm) module, and a natural language generation (nlg) module. each of these modules consists of several sub-modules that can be modelled independently. the integration of le could be accomplished through two additional submodules (tasks), a le extraction sub-module and a le generator sub-module. the objective of the le extraction sub-module is to identify lexical candidates and extract le expressions given the dialogue context. the objective of the generator module is to make use of these extracted lexical candidates to generate more human-like responses. additionally, we provide two baseline models for the le extraction sub-module. in summary, the main contributions of this paper are as follows: ## limitations of current approaches current conversational systems do not sufficiently address the phenomenon of le, leading to inconsistencies between generated responses and actual human responses. this can result in sub-optimal generated responses, which are not captured by current evaluation metrics, such as inform rate, suc-cess rate, and bleu score. to demonstrate the discrepancies between generated responses and ground-truth responses, we evaluate four state-ofthe-art response generation models @xcite in both end-to-end fashions, that is, using only the dialogue context as input to generate responses, and policy optimization fashion, that is using the ground-truth dialogue states to generate responses. for end-to-end models, mintl @xcite and augpt @xcite are evaluated. for policy optimization models, hdsa @xcite and marco @xcite are evaluated. in the following, we present two examples of generated responses that demonstrate the consequences of neglecting le. in the first case study (see the above-mentioned issues stem from neglecting the phenomenon of le, which occurs naturally and unconsciously in human-to-human conversations. we consolidate our observation with a quantifiable analysis in §6.1. model utterance (case study 1) user yes, please book it for 2 people on saturday for 4 nights. will you give me the confirmation number when that is complete? agent sure, booking was successful. your confirmation number is 7mg032v9. augpt booking was successful. reference number is: 00000016. mintl i have booked you at the express by holiday inn cambridge for 4 nights starting saturday. your reference number is d77jcgj2. is there anything else i can help? hdsa booking was successful. reference number is 7mg032v9. is there anything else i can help you with? marco i have booked you at the [hotel_name]. your reference number is 7mg032v9. is there anything else i can help you with? model utterance (case study 2) user i need a cab too please, arriving by my booked time , car type and contact number would be helpful too please. agent you need a cab to get you to the restaurant before the booked time and where do you need cab to pick you up at? augpt i have booked a blue skoda for you. the contact number is 85534112262. mintl i have booked you a bmw. the contact number is 07338019809. where will you be departing from? hdsa where will you be departing from and where are you going? marco i would be happy to book a taxi for you . where are you departing from and what is your destination? ## multiwoz-entr to facilitate le research for conversational systems, we introduce a new dataset named multiwoz-entr based on the multiwoz 2.1 @xcite . ## le extraction as depicted in figure 1 , to incorporate the le phenomenon into conversational systems, it is necessary first to identify lexical choices that can potentially be used as le expressions through an extraction process. the resulting candidates can then be fed into le generators to produce appropriate responses. in this section, we introduce a novel task of extracting le expressions from the preceding dialogue context and present two baseline models: an end-to-end approach and a pipeline approach. ## experiments and results in this section, we first implement quantitative analysis using our new proposed le measure. then we evaluate two baseline models on the multiwozentr for the le extraction task. additionally, we study the effect of different domains on the model performance. we also compute the entr agent against the number of turns in dialogues. as shown in figure 2 , the degree of le tends higher as the number of turns increases, in line with expectations that more le happens when people speak more. the curves of response generation models are mainly under their counterparts for humans, indicating discrepancies between generated responses and human-level responses. ## related work conversational systems. conversational ai has been a long-standing area of exploration in the natural language processing (nlp) research community @xcite @xcite @xcite @xcite @xcite . there are broadly two categories of conversational systems, open-domain @xcite and task-oriented @xcite @xcite . open-domain conversational systems mainly focus on open-ended conversations such as chit-chat, while task-oriented conversational systems target accomplishing a goal described by users in natural language such as hotel booking, movie recommendation, collaborative building @xcite @xcite @xcite @xcite . a number of corpora with various domains and annotations have been collected for task-oriented conversational systems. some datasets @xcite contain conversations with a single domain, while others @xcite @xcite contain multi-domains conversations. multiwoz dataset @xcite spans 7 distinct domains with over 10 000 dialogues. lexical entrainment (le) for conversational systems. some previous works @xcite @xcite @xcite attempted to integrate le in the conversational system. de @xcite presented an entrainment model for aligning the language use of a virtual agent to the level of politeness and formality displayed by the user's utterances. @xcite proposed an embodied conversational agent with the ability to choose when to share appreciation with a human partner in a museum setting. @xcite used rule-based and statistical models for the integration of le of an information-providing task in the public transport domain for spoken dialogue systems. @xcite implemented a context-aware generator to model to entrain the user's way of speaking by learning implicitly from data. @xcite designed a rule-based natural language generator to produce utterances entrained to a range of utterance features used in prior utterances by a human user, including referring expressions, syntactic template selection, and tense/modal choices. ## conclusion in this paper, we highlight the importance of incorporating le into conversational systems and propose a new dataset, named multiwoz-entr, designed to facilitate the study of le. we conduct a comprehensive statistical analysis of the multiwoz-entr dataset, providing valuable insights into its properties. our investigation of current state-of-the-art response generation models highlights the issue of neglecting le, and our proposed measure serves as evidence for the lack of le in these models. to address this deficiency, we introduce two new tasks, namely, le extraction and generation, and propose two baseline models for the le extraction task under various settings. our work lays the foundation for further research on the le generation task.