Evaluating Spoken Language Features in Conversational Models: The Case of Disfluencies and Feedbacks (2025)

Abstract: Understanding how language is processed and represented cognitively increasingly involves the use of specialized language models. Yet, because most models are predominantly trained on written text, they struggle to reflect the characteristics of language as it naturally unfolds in spoken interaction. This gap limits their capabilities in capturing features typical of spontaneous speech, such as repetitions, feedback cues, and hesitations. In this work, we introduce linguistically motivated evaluation metrics designed to target these specific spoken-language phenomena. We apply them to analyse outputs from language models fine-tuned on spoken English and French, comparing their behaviour statistically with human dialogue corpora. Our findings highlight the value of these metrics for assessing the degree to which model-generated utterances resemble authentic human conversation.

## introduction the cognitive bases of human language processing are increasingly being explored using large language models across various domains, such as linguistics @xcite and cognitive neuroscience @xcite . we argue that investigating how humans are capable of producing and understanding language in natural settings, such as conversational interactions, first requires the development of language models specifically adapted to spoken conversation. this presents a unique challenge, distinct from the requirements of classical dialogue systems, as such studies aim not only to replicate human spoken language but also to fully capture all of its dimensions. studying the cognitive bases of human language processing can require the estimation of word probabilities for different linguistic phenomena in a conversational context, for instance,to assess the processing difficulty of an event and predict the associated neuro-physiological signals (brain activity, gaze, movements, etc.) @xcite @xcite . language models offer a tool for such estimation, as they are designed to predict word probability distributions. however, in order to use such models in these studies, they must first be adapted to conversational settings, which raises the question of the methodology that can be used to develop such models. given the complexity of the task, this paper focuses on specific phenomena of spoken language derived from conversation transcriptions without incorporating acoustic or prosodic features. while this is a notable limitation, it represents an essential first step in a largely underexplored area. the evaluation of large language models typically assesses response quality and relevance within the context. our objective, however, is different: we seek to determine whether the generated conversations from these models replicate natural spoken language, including its specific phenomena such as pauses, hesitations, and repetitions. to this end, we employ a set of evaluation metrics that provide an analytical perspective. the choice of these metrics is based on different works in descriptive linguistics focusing on spoken language @xcite @xcite , which showed that the occurrence of such phenomena during a spontaneous interaction is not random, but can follow certain statistical patterns. to summarize, this paper introduces a set of linguistic metrics designed specifically to assess the naturalness of the generated conversation. these metrics aim to improve the evaluation of model quality from the perspective of studying the cognitive bases of conversation. to do this, we compare a pre-trained and a fine-tuned language model based on their performance in generating human-like spoken conversational sequences. ## related works the evaluation of large language models is a difficult task. previous approaches to language model evaluation mostly relied on lexical-level metrics that compare generated sequences to ground-truth sequences (bleu @xcite , rouge @xcite , meteor @xcite ) and semantic-level metrics like bertscore @xcite and @xcite , which leverage embeddings from pre-trained language models to assess the quality of generated sequences. given the one-to-many nature of human language and the impressive evolution of large language models, more recent works have shifted toward using more sophisticated approaches to assess models' performance across different domains, including logical and mathematical reasoning (math @xcite ), question answering @xcite and evaluation in multi-turn settings (multichallenge @xcite ). human evaluation has also been widely used to assess the naturalness, coherence, and factuality of llm outputs (van der @xcite . more recently, foundational models like gpt-4 have been leveraged to evaluate other models in the "llm-as-judge" paradigm to overcome the time and cost limitations of human evaluation @xcite . while most of the works focused on in-task evaluation of llms, other works have proposed evaluating the linguistic features of these models in offtask settings. in @xcite , the authors compared the linguistic diversity of llms to that of humans, while in (toro, 2023), the phonological biases of llms were studied, showing that these models tend to favor consonants over vowels when identifying words. in another study @xcite , a semantic and morphosyntactic evaluation revealed that llms still exhibit noticeable differences compared to human-generated text. in our work, we propose an off-task linguistic evaluation of models trained on spoken data by assessing their ability to generate specific phenomena of spoken language. ## method we propose a set of linguistic metrics more specifically adapted to the evaluation of linguistic features of human spoken language in sequences generated by a language model. among the different phenomena of spoken language, we will pay attention into two particular: disfluencies and feedbacks. disfluencies refer to any phenomenon that disrupts the smooth, ideal word-to-word flow of speech, such as repetitions, hesitations, and restarts @xcite . in our study, we will focus on two common disfluencies: repetitions and filled pauses. repetitions refer to repeated words and phrases that humans produce while talking, while filled pauses (fp) refer to vocalized hesitation pauses that occur in speech which are transcribed with words like 'euh' in @xcite . drawing inspiration from various studies on linguistics research, which have shown that disfluencies tend to occur at specific grammatical locations or after certain categories of words @xcite (see figure s1 in appendix f, indicating that word repetition is more common in certain categories than others, such as pronouns), we propose two novel metrics for evaluating disfluencies generation by language models based on frequency and on word categories and through comparison to a human reference corpora. first, we propose to evaluate the frequency of occurrences of these phenomena to assess potential over-generation by the model as they are very frequent in human spoken language @xcite . we define the frequency of filled pauses (freq-fp) as the ratio of words that represent a filled pause (words like 'euh' , 'um' and 'uh') in the sequences generated by a model. for repetition (freq-rep), we took inspiration from @xcite and calculated the ratio of repeated words in a sequence using the following score: where ngram-rep(seq, k) is a function that returns the identical, contiguous repeated n-grams of size k in a sequence seq, and n (set to 4 for the study) defines the maximum size of n-grams considered in the detection of the repetition in the sequence. the final freq-rep was obtained by averaging the rep-token score for all the sequences generated by the models. second, to evaluate the pattern of occurrence of such phenomena, we propose two novel metrics which compare the distribution of the categories of the words (pos tags foot_0 ) preceding filled pauses (kl-fp) or the categories of repeated words (kl-rep) in the model-generated sequences to the distribution found in a reference corpus of human spoken language. our metrics can be formalized as follows: where p fp and p rep are the distributions of the word categories of repeated words or words preceding a filled pause in the sequences generated by a model, while q fp and q rep are the distributions from a human spoken language reference corpus which is hypothesized to be large enough and diverse to reflect the statistical pattern of the production of disfluencies by a human. the different distributions are compared with kl divergence @xcite normalized with a non-linear transformation to obtain a score between 0 and 1 (a higher score is better). additionally, we will consider another phenomenon of human spoken language: feedbacks, which refer to the expressions a person produces to convey understanding and interest in what their interlocutor is saying (e.g., 'ok', 'yeah') @xcite . we follow the classification of feedbacks in @xcite and consider only generic feedbacks, as they are easier to classify given that they consist of a finite set of words (see appendix c for an example of words that can be found in a generic feedback). for feedbacks, we consider only their frequency defined as the ratio of turns generated by the model that can be classified as feedback. ## experiment to demonstrate our evaluation approaches, we finetuned two pre-trained language models : gpt-fr @xcite and gpt-2 (simoulin and crabbé, 2021), on french and english datasets of spoken conversation and compared the models before and after fine-tuning. as our metrics are designed to capture whether sequences follow patterns of spoken language, we expect the scores to be higher after fine-tuning. ## results the results of the evaluation using the proposed metrics are illustrated in tables 1 and 2 . we additionally report the divergence of word category distributions in the generated sequences, both before and after fine-tuning, compared to the reference and training corpora (kl-token). overall, we observe different trends across the two languages and metrics. for repetitions, the freq-rep scores remain relatively similar before and after fine-tuning (less than 3 repeated words per 100 words) and are com- ## discussion the primary contribution of this work is a set of linguistically inspired metrics to evaluate the extent to which language models generate specific phenomena of spoken language. our experimental results overall, the results show that fine-tuning on spoken data improves alignment with human-like generation of repetitions and filled pauses, as reflected in higher kl-rep and kl-fp scores. however, fine-tuning can also lead to over-generation, particularly of feedback. in the french model, nearly half of the generated turns were identified as feedback, likely due to its high frequency in the training data. this suggests that while fine-tuning helps align models with spoken language, regularization strategies may be necessary to avoid the over-generation of such frequent phenomena. another key observation is the difference in kl-token scores between the two languages. in french, the initial kl-token score before fine-tuning is high, reflecting a mismatch between the word category distributions in the generated sequences and those in spoken corpora-likely due to deeper structural differences between spoken and written french. fine-tuning reduces this divergence, though it remains more pronounced than in english, where spoken and written language appear to be more similar. ## limitations our study presents promising results for evaluating how natural, from a linguistic perspective, conversations generated by language models are. however, our evaluation remains relatively basic and additional analysis is needed for a more comprehensive assessment of the fine-tuned models. future work could explore other phenomena of spoken language, such as turn-taking in the generated conversations or incorporate more advanced syntactic analyses, such as syntactic trees, in comparisons with the reference corpora. additionally, we acknowledge that the models used in this study are quite outdated compared to recent models such as llama foot_1 , which may have resulted in the generation of contextually irrelevant sequences or caused the problem of text degeneration observed in the sequences generated by the fine-tuned models. we therefore plan to investigate in future work how fine-tuning larger models on broader datasets may improve the results. furthermore, while the experiment conducted in this study showed that these metrics can capture hidden aspects of the models such as the generation of degenerate sequences rather than human-like repetitions, further study is needed to assess the robustness of these metrics like exploring how these metrics correlate with human judgment and other evaluation approaches. andreas stolcke, klaus ries, noah coccaro, elizabeth shriberg, rebecca bates, daniel jurafsky, paul taylor, rachel martin, carol van ess-dykema, and marie meteer. 2000. dialogue act modeling for automatic tagging and recognition of conversational speech. computational linguistics, 26(3):339-374. juan manuel toro. 2023. emergence of a phonological bias in chatgpt. arxiv preprint arxiv:2305.15929. chris van der lee, albert gatt, emiel van miltenburg, and emiel krahmer. 2021. human evaluation of automatically generated text: current trends and best practice guidelines. computer speech & language, 67:101151. weizhe yuan, graham neubig, and pengfei liu. 2021. bartscore: evaluating generated text as text generation. advances in neural information processing systems, 34:27263-27277. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in international conference on learning representations.