Zero-Shot Evaluation of Conversational Language Competence in Data-Efficient LLM s Across E nglish, M andarin, and F rench (2025)

Abstract: Large Language Models (LLMs) have achieved oustanding performance across various natural language processing tasks, including those from Discourse and Dialogue traditions. However, these achievements are typically obtained thanks to pretraining on huge datasets. In contrast, humans learn to speak and communicate through dialogue and spontaneous speech with only a fraction of the language exposure. This disparity has spurred interest in evaluating whether smaller, more carefully selected and curated pretraining datasets can support robust performance on specific tasks. Drawing inspiration from the BabyLM initiative, we construct small (10M-token) pretraining datasets from different sources, including conversational transcripts and Wikipedia-style text. To assess the impact of these datasets, we develop evaluation benchmarks focusing on discourse and interactional markers, extracted from high-quality spoken corpora in English, French, and Mandarin. Employing a zero-shot classification framework inspired by the BLiMP benchmark, we design tasks wherein the model must determine, between a genuine utterance extracted from a corpus and its minimally altered counterpart, which one is the authentic instance. Our findings reveal that the nature of pretraining data significantly influences model performance on discourse-related tasks. Models pretrained on conversational data exhibit a clear advantage in handling discourse and interactional markers compared to those trained on written or encyclopedic text. Furthermore, the models, trained on small amount spontaneous speech transcripts, perform comparably to standard LLMs.

## introduction the performance levels achieved by current large language models (llms) are impressive across all natural language processing tasks, including those from the discourse and dialogue traditions, which have traditionally been considered more challenging. however, these results are typically obtained by pretraining models on enormous amounts of data. in contrast, humans learn to speak and communicate through dialogue and spontaneous speech with only a fraction of the language exposure required by artificial models. more broadly, there is growing interest in evaluating whether much smaller and carefully selected pretraining datasets can support performance on specific tasks. initiatives like babylm @xcite ) explore this question by providing english pretraining datasets of 10m and 100m tokens, as well as evaluation benchmarks, including blimp @xcite , which aims to evaluate the linguistic competence of llms. such pretraining datasets include a majority of spontaneous speech transcripts originating from child-directed speech (macwhinney, 2014) and everyday conversations. the evaluation metrics employed, while a good starting point, appear however biased in two ways: (i) they tend to favor canonical written forms ; and (ii) prioritize syntactic, semantic, and commonsense pragmatic competence. language and communicative competence include many other dimensions, in particular if one focuses on spontaneous speech in a conversational context. indeed, most benchmarks concerned with language are centered on high-level tasks and relatively standard morphological and syntactic tasks revolving around the idea of grammaticality. grammaticality is an ill-defined concept when used to analyze spontaneous speech productions. in this context, discourse and interactional marking, as well as linguistic performance-management like disfluencies, are crucial. moreover, most of these benchmarks are for english, whereas english constitutes the exception rather than the norm, at least with regard to data availability. due to data scarcity, it is still impossible to gather a 100m-token dataset based solely on real spoken conversational data, but the 10mtokens dataset is accessible for a few languages like english, french, mandarin, and a few others. our stance focussing on 'conversational speech' is adopted because it is the genre through which humans acquire their basic language skills, and it is also of special significance with regard to language emergence @xcite . moreover, it is quite distant from the usual written or web content on which lms are trained, increasing the risk of biases against this crucial genre in the resulting models. our focus in this paper is to assess the adequacy of llms with respect to conversational and spontaneous speech phenomena. we construct "small" (10m-tokens) pretraining datasets from different sources (conversational vs. wikipedia-style text). we also create evaluation benchmarks extracted from high-quality spoken corpora in english, french, and mandarin. more specifically, we focus on discourse and interactional markers @xcite and fillers @xcite . following the approach of blimp @xcite , we design zero-shot classification tasks based on these phenomena. our results show that pretraining data type has a clear impact on the model's ability to handle these features: conversational data offers an advantage over written encyclopedic text. moreover, standard llms do not seem to have a strong advantage compared to our small models trained on spontaneous speech transcripts. the paper is organized as follows. we present the related work (section 2) and justify our proposal in more detail in section 3. we then detail our methods and describe the creation of the pretraining datasets and benchmarks (section 4). the next section presents the experiments, their hypotheses, and the results (section 5). we conclude with a discussion of the limitations of this work and future directions. ## related work since the emergence of large language models, there has been interest from the computational linguistics community in understanding why they are so successful. @xcite explore the conditions (e.g., the amount of training data) under which roberta develops and leverages linguistic features, such as part of speech (pos) and morphology, as opposed to relying on simpler surface-level features like position-based or lengthbased cues. more recently, several studies have probed llms to better characterize their performance across various domains, particularly with regard to linguistic competence versus commonsense reasoning. these studies have also examined the relationship between model performance and the amount of training data required for different tasks. in particular, @xcite used training sets of varying sizes (1m, 10m, 100m, and 1b tokens) to show that syntactic and semantic competence becomes robust in the 10m-100m range, whereas larger datasets are needed to achieve strong results in pragmatic and commonsense reasoning tasks. more broadly, there have been proposals for evaluating the performance of llms on diverse linguistic tasks. @xcite leveraged a substantial body of generative syntax-semantics literature to develop benchmarks based on acceptability judgments, drawing either from the linguistic literature, as in the cola benchmark, or by exploiting more sources and data augmentation methods, as in blimp @xcite . in addition to these binary decision tasks, @xcite combined three other types of evaluation metrics: classifier probing (following @xcite ), which includes tasks from pos tagging to coreference resolution; information-theoretic probing based on the minimum description length (mdl) principle; and fine-tuning on higher-level tasks such as those in the superglue benchmark. blimp @xcite has inspired a series of language-specific benchmarks, such as climp for mandarin chinese @xcite , as well as benchmarks for other languages like japanese @xcite , dutch @xcite , russian @xcite , german @xcite , italian @xcite , and now a multilingual blimp @xcite . these are important additions to the evaluation landscape. while these benchmarks represent important extensions to the general evaluation framework, they all rely on syntax-semantics structures mostly derived from introspection and textbook data. turning now to conversational training data, even before the llm era, pannitto et al. (2020) trained a model on child-directed speech. in terms of specialized language models, @xcite developed lms based on child-directed speech transcripts and evaluated them on wordsense disambiguation tasks. they concluded that word acquisition trajectories could be better captured by multimodal models that incorporate acoustic features, among other aspects. the discourse and dialogue phenomena we investigate to increase awareness and methods for making llms more conversation-aware foot_0 are wellstudied phenomena. discourse markers @xcite have been extensively studied in terms of their definition, analysis, and use in discourse parsing @xcite . there are different sorts of discourse markers, ranging from discoursesemantic connectives (but, because, ...) to attitudinal or interpersonal markers (ah, oh, ...), adverbs (frankly, actually, ...), or conversational feedback markers (yeah, ok, ...). recently, sadlier-brown et al. (2024) compared human and llm performance in predicting the presence of actually. disfluencies (shriberg, 1994) have long been known as a hallmark of spontaneous speech. there is a vast literature on their detection, processing, and, more recently, their generation by llms to produce fluent yet natural-sounding speech @xcite . these two dimensions (discourse markers and disfluencies) benefit from being studied jointly, as they are known to interact deeply @xcite . ## a proposal for a new source of metrics most initiatives for linguistic llm evaluation are grounded in text-based and/or handcrafted paradigms, potentially coupled with behavioral and/or physiological lab measures. in contrast, we propose using actual spontaneous conversational transcripts to build complementary benchmarks that test real-world language use in spontaneous speech. these metrics will remain fundamentally linguistic in nature rather than focusing on end-toend evaluation. language is acquired, especially in its early stages, within spontaneous, conversational environments. while conversational language shares grammatical structures with other genres, its unique characteristics suggest that simply listing syntactic "errors" or semantic incongruities does not fully capture linguistic competence. furthermore, in a conversational context, what may be considered a production error from a normative grammatical perspective is often perfectly acceptable and successfully achieves its communicative purpose. therefore, we aim to develop a complementary approach that provides a broader set of metrics for evaluating language models from both cognitive and communicative perspectives. specifically, we propose using spontaneous speech corpora, as they offer insights into human language processing through various observable production phenomena. our approach is a form of classifier probing @xcite @xcite , but rather than focusing on meta-linguistic tasks (e.g., predicting syntactic categories), we aim to predict phenomena that serve as partial indicators of language processing. @xcite explored the prediction of speech-production variables, such as speech reduction and prosodic prominence; however, this approach requires fine-tuning a pretrained model, which obscures the overall interpretation of the results. we propose here a preliminary set of simple tasks exploiting lexical markers related to spontaneous speech, discourse, and interactional competence. we approach the question by designing tasks involving several kinds of discourse markers and fillers (see ## data and method we built the reference datasets from existing highquality linguistic corpora and created the "tasks" based on these reference datasets; and, how we selected and processed the base pretraining data (from a wider range of sources) to create our llms. crucially, there is no overlap between the pretraining data and the reference sources. the reference data for the benchmarks come from linguistic conversational corpora, as listed in the main reason for choosing these corpora is the high quality of their transcriptions, in which disfluencies are accurately rendered. ## discussion and conclusion from a machine learning perspective, it might seem trivial that models trained on data similar to test sets perform better than models trained on other types of data. first of all, it is worth emphasizing that the pretraining datasets and benchmarks in our experiments are completely independent, as they do not come from the same raw corpora. also, the pretraining datasets and the corpora used to build benchmarks have been curated by different teams and transcribed with different conventions. nevertheless, we cannot deny that the conversational datasets are, in all aspects (sentence length distribution, lexical frequencies, etc.), more similar to the benchmarks than the wikipedia datasets are. as trivial as it may seem, this is in fact one of our main points: to produce models more closely related to spontaneous speech, one should use datasets made of spontaneous speech (and not generic textual/web content). through this set of experiments, we aim to demonstrate the value of the proposed approach and to generalize it to other conversational, and spontaneous speech phenomena. from a broader perspective, we hope to show that benchmarks from the rapidly expanding "blimp-family", which require significant amounts of expert and naive human input to build, can be complemented with benchmarks derived from numerous existing highquality linguistic corpora, without additional human effort.