Audio-Aware Large Language Models as Judges for Speaking Styles (2025)

Abstract: Audio-aware large language models (ALLMs) can understand the textual and non-textual information in the audio input. In this paper, we explore using ALLMs as an automatic judge to assess the speaking styles of speeches. We use ALLM judges to evaluate the speeches generated by SLMs on two tasks: voice style instruction following and role-playing. The speaking style we consider includes emotion, volume, speaking pace, word emphasis, pitch control, and non-verbal elements. We use four spoken language models (SLMs) to complete the two tasks and use humans and ALLMs to judge the SLMs’ responses. We compare two ALLM judges, GPT-4o-audio and Gemini-2.5-pro, with human evaluation results and show that the agreement between Gemini and human judges is comparable to the agreement between human evaluators. These promising results show that ALLMs can be used as a judge to evaluate SLMs. Our results also reveal that current SLMs, even GPT-4o-audio, still have room for improvement in controlling the speaking style and generating natural dialogues.

## introduction ever since the introduction of gpt-4o voice mode (openai, 2024), the speech community has been moving forward rapidly to propose a model or pipeline that can understand input speech and generate fluent output speech like gpt-4o @xcite . the research on this topic can be largely divided into two types: (1) audio-aware large language models (allms) that can take texts and audios as input and generate texts @xcite @xcite , and (2) spoken language models (slms) that can take audio and texts as input and generate speech output @xcite * work done during an internship at microsoft genai. dcml0714@gmail.com † correspondence: xiaofei.wang@microsoft.com @xcite @xcite . only recently, some open-source slms have shown performance close to gpt-4o @xcite . when building better slms, a critical question is how to evaluate the speech they generate. the textual contents in speech can be easily evaluated by first transcribing the speech into text and evaluating the text using evaluation metrics for text. for example, using accuracy for question answering (qa) @xcite or using llm-as-ajudge @xcite for reference-free evaluation. evaluating paralinguistic aspects of the speech, including emotion, prosody, and emphasis, is much more challenging and typically relies on human evaluation @xcite , which is costly and has been shown to have high variance @xcite . in this paper, we explore whether allms can be an automatic judge of the speaking styles. to this end, we construct two tasks that test slm's ability to generate speeches with appropriate speaking styles and evaluate the generated speeches with allm judges. in voice style instruction following (voice style if), we instruct slms to say a given sentence with fine-grained speaking style instructions. in role-playing, we prompt an slm with some role-playing contexts and make it engage in a multi-turn role-playing dialogue. we evaluate the speech generated by the four slms, gpt-4oaudio (4o-audio) (openai, 2024), gpt-4o-miniaudio (4o-mini-audio), step-audio @xcite , and qwen-2.5-omni @xcite , using two allm judges: gpt-4o-audio (openai, 2024) and gemini-2.5-pro @xcite . we compare the evaluation results given by allm judges with human evaluation results and show that the geminihuman judge agreement can be comparable to the human-human judges' agreement. we summarize our contributions as follows: ## related works using large language models (llms) as a judge to evaluate the quality of texts, i.e., llm-as-ajudge @xcite , is a mainstream way to evaluate texts in nlp @xcite . however, there has been no success in applying slms or allms to evaluate speeches beyond their textual contents. @xcite try to use qwen-2-audio and 4o-audio as a judge, but find that the results do not align with human evaluation. a related work is @xcite , which finetunes an allm on mean opinion score (mos) datasets; however, the fine-tuned model can only be used for mos prediction. we show that allms can be used for more general evaluation beyond mos prediction, including the style adherence to the instructions and realism of speaking styles. our focus on evaluating paralinguistic aspects using allm is very different from @xcite , which uses allms to evaluate the signal aspect (mos or noise) of the audio; our contribution is orthogonal to the prior work. ## task introduction: styleset our goal is to understand whether allms can evaluate the speaking styles of the speech from slms. therefore, we construct two tasks that require slms to generate speech with proper styles and evaluate the speech using allm judges. we call the two tasks the styleset. precisely, we use voice style instruction following to evaluate whether slms can control the speaking styles when explicitly asked to do so, and we use role-playing to see if slms can understand the role-playing contexts and show appropriate speaking styles. examples of styleset are in appendix a.3 and appendix fig. 1 . ## conclusion and future work this paper attempts to use allm to judge the speaking styles generated by slms. we use two tasks, voice style if and role-playing, to generate speeches from slms that have diverse speaking styles and evaluate those speeches with allms. by comparing the evaluation results from human and allm judges, we find that allms can be used as automatic judges on these two tasks and achieve agreement with human judges comparable to the agreement within human judges. while our paper demonstrates the potential of using allms as an automatic judge for speaking styles, we only use closed-source allms as a judge. this is mainly because we found that open-source allms is still incapable of being an automatic judge on the tasks we study. consequently, collecting the training data and fine-tuning an open-source allm judge will be important in future work. a natural next step following our work is to fine-tune slms with rewards or feedback provided by allm judges. although reinforcement learning (rl) has been applied to fine-tune slms in recent studies, existing approaches typically derive rewards or preferences solely from the textual transcriptions of speech outputs @xcite , overlooking the paralinguistic dimensions of spoken responses. we advocate for fine-tuning slms with allm-derived rewards that reflect paralinguistic cues, enabling future models to generate speech that is not only accurate in its semantics but also natural and stylistically expressive.