Dialogue Acts as a Lens on Human– LLM Interaction: Analyzing Conversational Norms in Model-Generated Responses (2025)

Abstract: Large language models (LLMs) have revolutionized natural language generation across various applications. Although LLMs are highly capable in many domains, they sometimes produce responses that lack coherence or fail to align with conversational norms such as turn-taking, or providing relevant acknowledgments. Conversational LLMs are widely used, but evaluation often misses pragmatic aspects of dialogue. In this paper, we evaluate how LLM-generated dialogue compares to human conversation through the lens of dialogue acts, the functional building blocks of interaction. Using the Switchboard Dialogue Act (SwDA) corpus, we prompt two widely used open-source models, Llama 2 and Mistral, to generate responses under varying context lengths. We then automatically annotate the dialogue acts of both model and human responses with a BERT classifier and compare their distributions. Our experimental findings reveal that the distribution of dialogue acts generated by these models differs significantly from the distribution of dialogue acts in human conversation, indicating an area for improvement. Perplexity analysis further highlights that certain dialogue acts like Acknowledge (Backchannel) are harder for models to predict. While preliminary, this study demonstrates the value of dialogue act analysis as a diagnostic tool for human-LLM interaction, highlighting both current limitations and directions for improvement.

## introduction large language model-based dialogue systems are hugely successful in open domain language generation tasks such as question-answering @xcite . although these systems generally produce high-quality fluent dialogues and are able to hold conversations, their utterances sometimes fail to capture the nuances and emotions that are common in human-human interactions. effective conversation depends on subtle patterns of dialogue acts -utterances that serve functions such as asking questions, signaling agreement, or providing acknowledgment. we aim to investigate the extent of interpersonal synergy exhibited by llms in comparison to human interactions. synergy in terms of interactive and cooperative conversations refers to the way one agent responds to the other based on coordination and engagement between agents @xcite . dialogue acts are labels assigned to utterances that classify the intent of the speaker. in our study, we prompt the llama 2 model developed by meta @xcite and the mistral-7b model developed by mistral ai @xcite , with context from the switchboard dialogue acts (swda) foot_0 corpus of human telephonic conversations @xcite to generate the next utterance. we then conduct dialogue act classification with a bert-base model @xcite and compare the dialogue acts of the llm-generated responses to those of the gold-standard responses. our study illustrates both the promise and the limits of current llms as conversational partners, and proposes dialogue act analysis as a human-centered diagnostic tool that complements surface-level metrics. dialogue acts, a basic unit of conversation and indication of quality and engagement @xcite , allow us to measure the quality and type of utterances generated by models. by analyzing the specific types of dialogue act the llama 2 and mistral models struggle to generate, we gain a better understanding of the current limitations of llms. dialogue acts have been especially studied in work with regard to classroom dialogue @xcite , thus models that better follow human conversation styles or use quality dialogue could mimic a teacher's discourse and guide each student individually based on their utterances, leading to more personalized feedback for each student @xcite . @xcite and @xcite suggest that mistral outperforms llama 2 in several aspects, including reduced hallucinations and enhanced engagement in back-and-forth conversations. although mistral tends to generate more engaging and informative dialogues, typical human conversations often diverge from this pattern. human interactions frequently involve various cues and conventions, such as acknowledging others' opinions and providing affirmations. our experimental results demonstrate that the llama 2 model adapts its responses to more accurately reflect the nature of the ongoing conversation, aligning itself with the conversational style of the interlocutor. in this work, we ask whether dialogue act analysis can serve as a diagnostic lens for evaluating conversational coherence in large language models. specifically, we investigate whether systematic differences in dialogue act distributions between human and model-generated utterances can reveal where llms diverge from human conversational norms. beyond nlp evaluation, this question connects to cognitive science perspectives on pragmatic competence, highlighting whether llms reproduce or miss key interactional strategies, and where dialogue act mismatches may help explain why chatbot interactions sometimes feel less natural. ## background and related work recent work by @xcite highlights that large language models often fail to establish common ground in conversation, using significantly fewer grounding acts such as clarifications, acknowledgments, and follow-up questions compared to humans. their study introduces a taxonomy of grounding behaviors and demonstrates that instruction-tuned models systematically underuse these acts, particularly in high-stakes domains like emotional support and teaching. while their analysis focuses on how llms manage grounding, our work addresses a complementary question: how well llms reproduce the broader functional structure of human conversation as captured by dialogue acts. by analyzing utterance-level dialogue act distributions, we contribute an orthogonal yet critical view of conversational alignment, revealing that models overproduce questions and opinionated statements, but underproduce backchannels and agreement. together, these findings indicate that llms diverge from human norms not only in their ability to construct shared understanding but also in their broader interactional strategies. ## conclusions and future work we analyze dialogues generated by llama 2 and mistral, using various levels of prompting and incontext learning, comparing them to the original human-human interactions from swda, utilizing dialogue acts to gauge similarity. in our research, we initially computed the percentages of the top four categories of dialogue act tags in both the original and corresponding llm-predicted utterances. our findings suggest that dialogue acts are not only a descriptive tool but also a potential predictor of when conversational systems fail to align with human norms. this analysis shows that in contrast to humans, who commonly use both opinionated and non-opinionated statements, language models exhibit a preference for generating opinion statements, potentially to add perceived value to the conversation. additionally, these models tend to ask more questions, aiming to contribute more actively to the dialogue. upon conducting further investigation using a confusion matrix, we discovered significant variations in the dialogue acts between the generated and original utterances, which were apparent in the differing proportions, as discussed previously. llama 2's higher perplexity suggests that it might be more sensitive to context shifts and nuanced dialogue structures, resulting in more accurate classifications but higher uncertainty. additionally, we found that context length significantly impacts response quality. future research could expand this work in several directions. one promising avenue is to evaluate newer models, to assess whether advances in training and alignment reduce the dialogue act discrepancies we observed. another is to extend dialogue act analysis beyond distributional comparisons, incorporating metrics for conversational flow, user engagement, and appropriateness in interactive settings. finally, integrating human-in-the-loop evaluations, where human participants interact with models and provide feedback on coherence and naturalness, would help connect dialogue act diagnostics more directly to real-world conversational quality. taken together, these directions highlight the potential of dialogue act analysis to bridge natural language processing and human-computer interaction, supporting the development of conversational systems that are not only fluent but also socially and pragmatically aligned with conversational norms.