G m SLM : Generative Marmoset Spoken Language Modeling (2025)

Abstract: Marmoset monkeys exhibit complex vocal communication, challenging the view that nonhuman primates’ vocal communication is entirely innate, and show similar features of human speech, such as vocal labeling of others and turn-taking. Studying their vocal communication offers a unique opportunity to link it with brain activity—especially given the difficulty of accessing the human brain in speech and language research. Since Marmosets communicate primarily through vocalizations, applying standard LLM approaches is not straightforward. We introduce Generative Marmoset Spoken Language Modeling (GmSLM), an optimized spoken language model pipeline for Marmoset vocal communication. We designed a novel zero-shot evaluation metrics using unsupervised in-the-wild data, alongside weakly labeled conversational data, to assess GmSLM and demonstrate its advantage over a basic human-speech-based baseline. GmSLM generated vocalizations closely matched real resynthesized samples acoustically and performed well on downstream tasks. Despite being fully unsupervised, GmSLM effectively distinguish real from artificial conversations and may support further investigations of the neural basis of vocal communication and provides a practical framework linking vocalization and brain activity. We believe GmSLM stands to benefit future work in neuroscience, bioacoustics, and evolutionary biology. Samples are provided under: https://pages.cs.huji.ac.il/adiyoss-lab/GmSLM/.

## introduction marmoset monkeys are small primates with surprisingly complex vocal communication. although it was once assumed that nonhuman primates' vocal communication is entirely innate and inflexible, recent studies show that marmosets can learn new vocalizations and even label their conspecifics (oren * equal-contribution last authors figure 1 : high level description of gmslm. @xcite . these findings open the door for advanced computational analyses powered by large language models-to systematically investigate and decode marmoset vocalization patterns, as a novel approach to uncover the cognitive and evolutionary mechanisms behind vocal communication. however, as marmosets use vocal communication, with no well-defined orthography, it is unclear how recent advancements in llms, typically operate over discrete tokens, could be effectively utilized. in recent years, the field of generative spoken language modeling (gslm), often known to as textless nlp, has gained traction @xcite , demonstrating remarkable performance across various speech and audio tasks, encompassing both generative @xcite and discriminative approaches @xcite . the objective of gslm is to develop a spoken language model that operates without text. the typical gslm framework begins with an unsupervised process for acoustic units discovery, producing a discrete representation of the audio signal. then, a language model is applied to these units to estimate the likelihood of the audio signal. finally, a generative network converts the units back into a time-domain audio signal. since the gslm pipeline enables efficient encoding, sequential modeling, and decoding of raw audio signals without relying on textual supervision, it could be ideally suited for modeling marmosets vocalizations. in this work, we introduce gmslm (stands for generative marmoset spoken language modeling), a tailored adaptation of the gslm pipeline for marmosets-vocalizations, where each modeling component is optimized specifically for marmosetvocalization. to evaluate gmslm, we developed a set of zero-shot evaluation metrics to measure the quality of its sequential modeling and conducted additional evaluations using weakly labeled conversational marmoset data. a visual description can be seen on figure 1 . we compare the proposed approach to the naive human-speech based baseline, and demonstrate gmslm significantly enhances system performance. finally, we assess the quality of the sampled and generated marmosetsvocalizations and empirically show these are comparable to their resynthesized counterparts. leveraging manually annotated data (at the speaker level), we empirically demonstrate that, despite being entirely unsupervised, the proposed method assigns higher likelihood to authentic marmosets conversations compared to unnatural ones. we evaluate gmslm on downstream supervised tasks, where it performs strongly, confirming its utility beyond the training objective. ablation studies support the need for our model architecture given the data's complexity. we also use gmslm to explore marmosets' vocal communication and interpret learned representations. overall, our work establishes a foundation for future research in animal language and related downstream applications. more broadly, it opens a new direction for unsupervised spoken language modeling in nonhuman species. we hope it inspires further research in this emerging field. ## approach recall that the proposed approach, (gmslm) builds on the gslm pipeline, which was originally designed for human speech. in this section, we outline the modifications and adaptations made to better suit marmoset vocalizations. we begin by detailing the data collection and preprocessing phase, followed by an explanation of the adjusted system components. finally, we describe the system evaluation process. ## related work textless nlp was introduced by @xcite , showing how raw speech can be used to build gslm systems. @xcite extended this with multi-stream speech language models (slms) that combine pseudo-text with prosodic features, opening new directions in spoken language modeling. later works improved performance by initializing slms from text lms @xcite , framing tasks like emotion @xcite and speaking style conversion @xcite as discrete translations, and jointly modeling dialogue @xcite . @xcite proposed a cascade of lms for semantic and acoustic tokens, enabling high-quality, speaker-consistent speech generation @xcite . semantic units from these models correlate with phonemes @xcite , while others adapted this approach for speech translation @xcite . additional advances include augmenting text lms with speech for qa @xcite , state-space slms for long-context modeling @xcite , and preference-tuned slms using llm feedback @xcite . ## conclusion we introduce generative marmoset spoken language modeling (gmslm), a novel pipeline for modeling, generating, and evaluating marmoset vocalizations without labeled supervision. our approach bridges nonhuman primates vocal communication and modern generative language models. through zero-shot evaluations, gmslm effectively captures structural properties of marmoset vocalizations, distinguishing authentic from artificial utterances. it outperforms human-speech-only baselines and achieves high performance on downstream tasks. layer selection and model complexity analyses show that ssl-based representations provide richer features, with the first layer performing best-likely reflecting an optimal balance between acoustic and semantic information in marmoset vocalizations. contextual analysis indicates key communicative information spans ≈six seconds. our work introduces a new direction for unsupervised spoken language modeling beyond human speech. we position gmslm as a foundational tool for studying vocal communication in species with limited annotations, fostering interdisciplinary research at the intersection of computational linguistics, bioacoustics, and neuroscience.