Aligning to Adults Is Easy, Aligning to Children Is Hard: A Study of Linguistic Alignment in Dialogue Systems (2024)

Abstract: During conversations, people align to one another over time, by using similar words, concepts, and syntax. This helps form a shared understanding of the conversational content and is associated with increased engagement and satisfaction. It also affects conversation outcomes: e.g., when talking to language learners, an above normal level of linguistic alignment of parents or language teachers is correlated with faster language acquisition. These benefits make human-like alignment an important property of dialogue systems, which has often been overlooked by the NLP community. In order to fill this gap, we ask: (RQ1) Due to the importance for engagement and satisfaction, to what degree do state-of-the-art dialogue systems align to adult users? (RQ2) With a potential application to child language acquisition in mind, do systems, similar to parents, show high levels of alignment during conversations with children? Our experiments show that ChatGPT aligns to adults at roughly human levels, while Llama2 shows elevated alignment. However, when responding to a child, both systems’ alignment is below human levels.

## introduction conversation allows people to share information by creating a collective representation of the conversational context, achieved in part by linguistic alignment @xcite : when two people are conversing, the content of their speech as well as how it is phrased prime the other person to respond in a certain way. this reduces the chance of misunderstandings, since the used words and phrasing already have an established shared meaning, and thus, makes communication more efficient and enjoyable @xcite . linguistic alignment is critical in a variety of conversations, even those between a human and a virtual agent: prioritizing alignment in responses ## related work exploring linguistic alignment linguistic alignment is a mechanism by which humans mimic their partners in conversation -from phonology, to syntax and semantics @xcite . this kind of repetition lightens the cognitive load of language production, as certain structures are already primed from previous usage. alignment at multiple levels such as lexical and syntactic results in alignment of situation models, as language production, comprehension, and interactive production are all interwoven @xcite . alignment contributes to the success of a variety of human interaction. from the workplace -employees who show elevated levels of alignment over time are more likely to remain in the company @xcite -to the language classroom or nursery @xcite . in some cases, the alignment of task-specific vocabularies strongly correlates with conversation outcomes @xcite . alignment as a feature of communication is also critical in humancomputer interaction. lexical alignment affects human understanding of a conversational agent during live conversation @xcite . it also contributes towards decreasing user frustration and perceived task load when interacting with a dialogue system @xcite . and efl classroom (amin, 2023). context a any jury's not going to disregard the evidence, you know. b uh, that's true. b i, i, i think our judicial system is attorney welfare myself. a that may very well be. response b i, i hold it in the utmost contempt. gpt it's definitely a possibility that needs to be looked into. llama2 yeah, it's like, you know, they're just trying analysis of dialogue systems while common to use the automatic scoring methods of word overlap with a ground truth (such as bleu @xcite or rouge @xcite ) or words embeddings to evaluate dialogue systems, these metrics do not correspond highly with human judgement @xcite . other automatic metrics, such as context coherence -how well the response matches the context of the conversation @xcite can result in improved systems. outside of automatic metrics, human evaluation is critical and can look at dimensions such as informativeness, grammatically, coherence as well as how human-like or engaging the system is @xcite . the downside of human evaluation, however, is time and cost. ## conclusion and future work dialogue systems show great potential to assist humans across a variety of tasks. the success of these interactions, like human-human interaction, correlates with linguistic alignment. thus, we explore how state-of-the-art dialogue systems align to both adults and children. we find that, when responding to adult speakers, chatgpt shows approximately human-level alignments and provides constructive responses. llama2, however, overly mimics the conversation. this could be positive when talking with children or language learners as it results in heightened alignment. however, both models align below human levels. we conclude that sota dialogue systems have room for improvement in regards to reaching ideal levels of alignment under various circumstances. in the future, we plan to investigate alignment to adult learners or non-typical speakers, in addition to exploring techniques to create dialogue systems with a closer-to-human level of alignment. we will also explore how well dialogue systems match the user in multi-turn conversational structures, and related outcomes @xcite .