P od A gent: A Comprehensive Framework for Podcast Generation (2025)

Abstract: Existing automatic audio generation methods struggle to generate podcast-like audio programs effectively. The key challenges lie in in-depth content generation, appropriate and expressive voice production. This paper proposed PodAgent, a comprehensive framework for creating audio programs. PodAgent 1) generates informative topic-discussion content by designing a Host-Guest-Writer multi-agent collaboration system, 2) builds a voice pool for suitable voice-role matching and 3) utilizes LLM-enhanced speech synthesis method to generate expressive conversational speech. Given the absence of standardized evaluation criteria for podcast-like audio generation, we developed comprehensive assessment guidelines to effectively evaluate the model’s performance. Experimental results demonstrate PodAgent’s effectiveness, significantly surpassing direct GPT-4 generation in topic-discussion dialogue content, achieving an 87.4% voice-matching accuracy, and producing more expressive speech through LLM-guided synthesis. Demo page: https://podcast-agent.github.io/demo/. Source code: https://github.com/yujxx/PodAgent.

## introduction audio programs are an important channel for information acquisition. compared to video or text media, audio can free your eyes and hands, allowing you to access information conveniently in a variety of scenarios. podcasts are digital audio programs that are available for streaming or downloading over the internet. as shown in figure 1 (above), the content of a podcast typically consists of viewpoints shared by various individuals. to accommodate diverse interests, podcasts often explore a wide range of topics, including economics, culture, psychology, and more. however, many content creators still face the complex process of transforming creative ideas into a final product. additionally, providing strong, well-founded viewpoints and producing high-quality podcast-like audio on any given topic remains a significant challenge. recent advancements in generative models have made it possible to automatically create highquality content. large language models (llms) @xcite @xcite @xcite have achieved breakthrough capabilities in generating coherent and contextually appropriate text. in addition, foundation models for other modalities, such as vision @xcite @xcite and audio @xcite @xcite , have made remarkable strides in the creation of multimodal content. while existing models can generate podcastlike content, they have not yet achieved the creation of complete, professionally structured podcast episodes. for instance, audio-enhanced multimodal llms @xcite @xcite primarily focus on enabling multimodal interactions, but these interactions are typically constrained by short context windows and limited reasoning capabilities. text-to-audio (tta) models @xcite @xcite ) can generate various audio types, like speech, sound effects, and music. however, since these models prioritize general audio synthesis, they are inherently limited in producing coherent and intelligent spoken content. although zero-shot text-to-speech (tts) models @xcite @xcite can generate high-quality speech for any speaker, they rely on the provided text and lack the ability to generate long-form informative content. a straightforward approach is to combine the strengths of these models-using llms to generate rich text, tts models to convert it into spoken content, and tta models to add appropriate sound effects and background music to produce complete, informative, and professionally structured podcast episodes (figure 1 below). this approach naturally aligns with the emerging paradigm of ai agents @xcite . empowered by llms, various ai agents @xcite @xcite are created to coordinate multiple ai tools to accomplish complex tasks through perception, decision-making, and action execution. a notable implementation of this approach is wavjourney @xcite , which leverages llms to generate an audio script that connects various models for audio generation. while wavjourney represents a significant step forward with its extensive audio generation workflow, its current implementation still faces challenges in producing complete and intellectually rich content (an example demonstrated in through observation and analysis of existing automated audio program creation systems, we identify four critical challenges as follows: content depth and insight generation. for a given topic, how to automatically generate rich and insightful viewpoints and provide meaningful analysis? natural dialogue generation. how to create engaging conversational content that flows naturally between speakers, maintaining coherence while avoiding repetition? appropriate voice representation. how to match suitable voice characteristics with different content and roles, ensuring consistency and authenticity in the audio presentation? speech quality and expressiveness. how to generate robust long-form speech with appropriate prosody and emotion that matches the content's intent and maintains listener engagement? in this work, we present podagent, a fully automated and comprehensive framework for creating content-rich and professionally structured audio programs. to tackle the aforementioned challenges, we: ## podagent this section introduces podagent, a fully automated comprehensive framework for creating informative and professionally structured audio programs. the focus will be on the key contributions of this work: 1) host-guest-writer system for generating conversation scripts 2) voice-role matching for selecting suitable voices, and 3) speech synthesis enhanced by llm-predicted instruction. ## experimental setups our experiments will center around the key contributions of this work: topic-based discussion content generation, voice-role matching, and expressive speech synthesis. we follow wavjourney's setup for generating music and sound effects but use a more recent open-source framework, cosyvoice2, for speech generation. ## experimental analysis throughout our experiments, we set the guest number to 2, except for those conducted in the guest number analysis. to ensure experimental consis-tency and reliable performance, we exclusively employed gpt-4 for all llm-dependent tasks. ## conclusion in this study, we proposed podagent, a comprehensive framework for creating audio programs that addresses the shortcomings of previous automated podcast-like generation methods. the key components of podagent include: 1) a host-guest-writer system generating comprehensive, multi-perspective conversation scripts, 2) a preset diverse voice pool for suitable voice-role assignment, and 3) llm-guided speech generation for enhanced expressiveness. given the absence of established benchmarks in podcast generation, we designed a thorough experimental setup encompassing both qualitative and llm-based evaluation of the conversation scripts, as well as voicerelated metrics. our extensive experimental results demonstrate podagent's capability to produce highquality, complete, and realistic audio programs.