Une Approche Linguistique pour l’Évaluation des Caractéristiques du Langage Parlé dans les Modèles Conversationnels (2025)

Abstract: L’étude du traitement du langage et de ses bases cognitives chez l’humain repose de plus en plus sur des modèles de langue adaptés. Cependant, la majorité des modèles existants sont principalement entraînés sur des données écrites, ce qui limite leur pertinence pour l’étude du langage tel qu’il se manifeste dans des contextes naturels, comme lors de conversations spontanées. En effet, ces modèles ne sont pas entraînés pour traiter avec précision les caractéristiques spécifiques du langage parlé, telles que les disfluences et les hésitations. Dans cet article, nous proposons un ensemble de métriques inspirées par la recherche linguistique afin d’évaluer certains phénomènes du langage parlé (feedback, répétition et hésitation) dans des énoncés générés par différents modèles de langue, à travers une comparaison statistique avec des corpus de conversations humaines. Nos résultats, obtenus sur de petits modèles de langue fine-tunés sur des données de conversations parlées en français et en anglais, démontrent le potentiel de ces métriques pour évaluer la similarité des séquences générées avec celles produites par des locuteurs humains.

## introduction the cognitive bases of human language processing are increasingly being explored using large language models across various domains, such as linguistics @xcite and cognitive neuroscience @xcite . we argue that investigating how humans are capable of producing and understanding language in natural settings, such as conversational interactions, first requires the development of language models specifically adapted to spoken conversation. this presents a unique challenge, distinct from the requirements of classical conversational agents or dialogue systems, as such studies aim not only to replicate human spoken language but also to fully capture all of its dimensions. studying the cognitive bases of human language processing can require the estimation of word probabilities for different linguistic features in a conversational context, for instance,to assess the processing difficulty of a linguistic event and predict the associated neuro-physiological signals (brain activity, gaze, movements, etc.) @xcite @xcite . language models offer a tool for such estimation, as they are designed to predict word probability distributions. however, in order to use such models in these studies, they must first be adapted to spoken conversational settings. this raises the question of which methodology can be used to develop such models, as spoken language differs from the written language on which most recent language models are trained @xcite . given the complexity of the task, this paper focuses on specific phenomena of spoken language derived from conversation transcriptions without incorporating acoustic or prosodic features. while this is a notable limitation, it represents an essential first step in a largely underexplored area. the evaluation of large language models typically assesses response quality and relevance within the context. our objective, however, is different : we seek to determine whether the generated conversations from these models replicate natural spoken language, including its specific phenomena such as pauses, hesitations, and repetitions. to this end, we employ a set of evaluation metrics that provide an analytical perspective. the choice of these metrics is based on different works in descriptive linguistics focusing on spoken language @xcite @xcite , which showed that the occurrence of such phenomena during a spontaneous interaction is not random, but can follow certain statistical patterns. to summarize, this paper introduces a set of linguistic metrics designed specifically to assess the naturalness of the generated conversation. these metrics aim to improve the evaluation of model quality from the perspective of studying the cognitive bases of conversation. to do this, we compare a pre-trained and a fine-tuned language model based on their performance in generating human-like spoken conversational sequences. ## related works the evaluation of large language models is a difficult task. previous approaches to language model evaluation mostly relied on lexical-level metrics that compare generated sequences to ground-truth sequences (bleu @xcite , rouge @xcite , meteor @xcite ) and semantic-level metrics like bertscore @xcite and bartscore @xcite , which leverage embeddings from pre-trained language models to assess the quality of generated sequences. given the one-to-many nature of human language and the impressive evolution of large language models @xcite , more recent works have shifted toward using more sophisticated approaches to assess models' performance across different domains, including logical and mathematical reasoning (math (hendrycks et al., 2021)), question answering @xcite , coding (arcade @xcite ), and evaluation in multi-turn settings (multichallenge @xcite ). human evaluation has also been widely used to assess the naturalness, coherence, and factuality of llm outputs (van der @xcite . more recently, foundational models like gpt-4 have been leveraged to evaluate other models in the "llm-as-judge" paradigm to overcome the time and cost limitations of human evaluation @xcite . while most of the works focused on in-task evaluation of llms, other works have proposed evaluating the linguistic features of these models in off-task settings. in @xcite , the authors compared the linguistic diversity of llms to that of humans, while in @xcite , the phonological biases of llms were studied, showing that these models tend to favor consonants over vowels when identifying words. in another study @xcite , a semantic and morphosyntactic evaluation revealed that llms still exhibit noticeable differences compared to human-generated text. in the context of spoken dialogue, several studies have explored the potential of using pretrained language models for the development of spoken dialogue systems. early work focused on leveraging pretrained language models to improve language understanding tasks @xcite . more recent works, such as @xcite , examined the robustness of large language models on spoken language, finding that these models are not sufficiently robust to spoken noise. in @xcite , louradour et al. proposed a set of models for spoken language by continuing the training of large language models on a large dataset of spoken conversation transcripts. while our paper does not aim to build such models, we propose an off-task linguistic evaluation of models trained on spoken data by assessing their ability to generate specific phenomena characteristic of spoken language. ## method we propose a set of linguistic metrics more specifically adapted to the evaluation of linguistic features of human spoken language in sequences generated by a language model. among the different phenomena of spoken language, we will pay attention into two particular : disfluencies and feedbacks. disfluencies refer to any phenomenon that disrupts the smooth, ideal word-to-word flow of speech, such as repetitions, hesitations, and restarts @xcite . in our study, we will focus on two common disfluencies : repetitions and filled pauses. repetitions refer to repeated words and phrases that humans produce while talking, while filled pauses (fp) refer to vocalized hesitation pauses that occur in speech which are transcribed with words like @xcite . drawing inspiration from various studies on linguistics research, which have shown that disfluencies tend to occur at specific grammatical locations or after certain categories of words @xcite (see figure s1 in appendix f, indicating that word repetition is more common in certain categories than others, such as pronouns), we propose two novel metrics for evaluating difluencies generation by language models based on frequency and on word categories and through comparison to a human reference corpora. first, we propose to evaluate the frequency of occurrences of these phenomena to assess potential over-generation by the model as they are very frequent in human spoken language @xcite . we define the frequency of filled pauses (freq-fp) as the ratio of words that represent a filled pause (words like 'euh', 'um' and 'uh') in the sequences generated by a model. for repetition (freq-rep), we took inspiration from @xcite and calculated the ratio of repeated words in a sequence using the following score : where ngram-rep(seq, k) is a function that returns the identical, contiguous repeated n-grams of size k in a sequence seq, and n (set to 4 for the study) defines the maximum size of n-grams considered in the detection of the repetition in the sequence. the final freq-rep was obtained by averaging the rep-token score for all the sequences generated by the models. second, to evaluate the pattern of occurrence of such phenomena, we propose two novel metrics which compare the distribution of the categories of the words ( pos tags foot_0 ) preceding filled pauses (kl-fp) or the categories of repeated words (kl-rep) in the model-generated sequences to the distribution found in a reference corpus of human spoken language corpus. our metrics can be formalized as follows : where p fp and p rep are the distributions of the word categories of repeated words or words preceding a filled pause in the sequences generated by a model, while q fp and q rep are the distributions from a human spoken language reference corpus. the different distributions are compared with kl divergence @xcite normalized with a non-linear transformation to obtain a score between 0 and 1 (higher score is better). additionally, we will consider another phenomenon of human spoken language : feedbacks, which refer to the expressions a person produces to convey understanding and interest in what their interlocutor is saying (e.g., 'ok', 'yeah') @xcite . we follow the classification of feedbacks in @xcite and consider only generic feedbacks, as they are easier to classify given that they consist of a finite set of words (see appendix c for an example of words that can be found in a generic feedback). for the evaluation of feedbacks, we consider only their frequency defined as the ratio of turns generated by the model that can be classified as feedback. ## experiment to demonstrate our evaluation approaches, we fine-tuned two different pre-trained language models, gpt-fr @xcite and gpt-2 @xcite , on french and english datasets of spoken conversation and compared the models before and after finetuning. as our metrics are designed to capture whether sequences follow patterns of spoken language, we expect the scores to be higher after finetuning. details on the data as well as the fine-tuned models can be found in appendix a and b. to generate sequences for the evaluation, we used a test subset of the finetuning corpora and prompted the models to complete a multi-turn conversation by providing the first seven turns in a conversation as context. see appendix e for examples of generated turns. for reference corpora, we used the eslo and cid corpus @xcite for french and the switchboard dialog act corpus @xcite for english. ## results the results of the evaluation using the proposed metrics are reported in tables 1 and 2 . we additionally report the divergence of word category distributions in the generated sequences, both before and after finetuning, compared to the reference and training corpora (kl-token). overall, we observe different trends across the two languages and metrics. for repetitions, the freq-rep scores remain relatively similar before and after finetuning (less than 3 repeated words per 100 words) and are comparable to the scores from the reference and training corpora. the exception is the english model before finetuning, which produces sequences with a high frequency of repetition. however, the kl-rep scores improve after finetuning, with the most substantial improvement observed in the french model. the improvement after finetuning suggests that the models begin to generate repetitions in patterns more aligned with how humans produce repetition in speech. a similar trend is observed for filled pauses. while the overall frequency freq-fp remains close to that of the reference corpora both before and after finetuning, the kl-fp scores improve significantly after finetuning, especially for french, suggesting a better alignment with the distributional patterns of filled pause production in spoken language. regarding kl-token, the distribution of word categories in the sequences generated by the french model differs considerably from the word categories in the reference spoken corpora and remains very relatively divergent even after finetuning. in contrast, the english model shows a closer alignment with the spoken language distributions both before and after finetuning. for feedbacks, the results show a high frequency of feedback turns in both languages. interestingly, the different corpora display significant variation in feedback frequency, particularly in french, likely due to differences in the nature of the tasks in each corpus. ## discussion the primary contribution of this work is a set of linguistically inspired metrics to evaluate the extent to which language models generate specific phenomena of spoken language. our experimental results provide insights into the gap between spoken and written language, and how this gap manifests in models trained on the two different forms of language. the frequency-based metrics show that the models can generate various spoken language phenomena, such as repetitions and filled pauses, at rates similar to those found in spoken language corpora. however, when evaluated using divergence-based metrics (kl-rep and kl-fp), a more nuanced picture emerges : while the frequencies may align, the placement and distributional patterns of these phenomena often do not. for example, although the freq-rep scores suggest similar rates of repetition before and after finetuning, the low kl-rep scores before finetuning indicate that these repetitions are unnatural and likely the result of text degeneration @xcite , where the model loops or repeats phrases unnaturally. this issue is particularly visible in the english model, where frequent repetitions before finetuning deviate significantly from natural human repetition patterns. such observations underscore the utility of divergence-based metrics in revealing qualitative improvements not captured by raw frequency scores. overall, the results show that finetuning on spoken data improves alignment with human-like generation of repetitions and filled pauses, as reflected in higher kl-rep and kl-fp scores. however, finetuning can also lead to over-generation, particularly of feedback. in the french model, nearly half of the generated turns were identified as feedbacks, likely due to its high frequency in the training data. this suggests that while finetuning helps align models with spoken language, regularization strategies may be necessary to avoid the over-generation of such frequent phenomena. another key observation is the difference in kl-token scores between the two languages. in french, the initial kl-token score before finetuning is high, reflecting a mismatch between the word category distributions in the generated sequences and those in spoken corpora,likely due to deeper structural differences between spoken and written french. finetuning reduces this divergence, though it remains more pronounced than in english, where spoken and written language appear to be more similar. ## limitations our study presents promising results for evaluating how natural, from a linguistic perspective, conversations generated by language models are. however, our evaluation remains relatively basic, and additional analysis is needed for a more comprehensive assessment of the fine-tuned models. future work could explore other phenomena of spoken language, such as turn-taking in the generated conversations, or incorporate more advanced syntactic analyses, such as syntactic trees, in comparisons with the reference corpora. additionally, we acknowledge that the models used in this study are quite outdated compared to recent models like llama 3 , which may have resulted in the generation of contextually irrelevant sequences or caused the problem of text degeneration. we therefore plan to investigate in future work how finetuning larger models on larger datasets may improve the results. yoon j., hwang s., kim k.-e., ran h. & bang j.-u. (2023). adapting text-based dialogue state tracker for spoken dialogues. in 24th sigdial workshop on dstc11 : the eleventh dialog system technology challenge, p. 81-88 : association for computational linguistics. yuan w., neubig g. & liu p. (2021). bartscore : evaluating generated text as text generation. advances in neural information processing systems, 34, 27263-27277. zhang t., kishore v., wu f., weinberger k. q. & artzi y. (2020). bertscore : evaluating text generation with bert. in international conference on learning representations.