ESP net- S peech LM : An Open Speech Language Model Toolkit (2025)

Abstract: We present ESPnet-SpeechLM, an open toolkit designed to democratize the development of speech language models (SpeechLMs) and voice-driven agentic applications. The toolkit standardizes speech processing tasks by framing them as universal sequential modeling problems, encompassing a cohesive workflow of data preprocessing, pre-training, inference, and task evaluation. With ESPnet-SpeechLM, users can easily define task templates and configure key settings, enabling seamless and streamlined SpeechLM development. The toolkit ensures flexibility, efficiency, and scalability by offering highly configurable modules for every stage of the workflow. To illustrate its capabilities, we provide multiple use cases demonstrating how competitive SpeechLMs can be constructed with ESPnet-SpeechLM, including a 1.7B-parameter model pre-trained on both text and speech tasks, across diverse benchmarks. The toolkit and its recipes are fully transparent and reproducible at: https://github.com/espnet/espnet/tree/speechlm.

## introduction the advent of large language models (llms) has significantly advanced machine intelligence, particularly in the text domain @xcite . as research expands beyond text, llms are increasingly applied to multimodal scenarios @xcite @xcite , such as speech @xcite and vision @xcite , with the aim of achieving higher-level intelligence and enhancing human-computer interactions. within this context, speech language models (speechlms) have emerged as a powerful paradigm addressing challenges unique to speech processing. speechlms have demonstrated remarkable progress across a variety of speech tasks, including zero-shot generalization @xcite , tasktemplate: { name: asr conditions: @xcite ], targets: [(text, bpe)], } preprocessing trainconfig: { optimizer: ..., model_arch: ..., } training inferenceconfig: { infermethod: ..., inferparam: ..., } inference evalconfig: { evalmetric1: ..., evalmetric2: ..., } ## related work the espnet-speechlm toolkit builds upon prior works in two main directions: text llm ecosystem: some popular development tools in text llm ecosystems can be generalized to any large-scale sequential modeling task, which means they are also suitable for speechlm training. examples of this include deepspeed @xcite and flashattention @xcite . to preserve text capability, it is common to initialize speechlms from pre-trained text llms, which can rely on open-source platforms like hugging-face transformers foot_0 . these tools are integrated into espnet-speechlm. we also noticed that current text llm training frameworks @xcite provide limited support for speech features. our toolkit is presented as a supplement in this direction. open-sourced speechlms and speech toolkits: current research on speechlms and their transparency has been significantly advanced by prior open-source speechlm research works @xcite @xcite @xcite . speechlm research also greatly benefits from general speech processing and sequence-to-sequence modeling toolkits @xcite @xcite @xcite , as they provide a wide range of components applicable to speechlm development. espnet-speechlm is presented as a combination of cutting-edge speechlm research and well-established speech processing techniques within the open-sourced community. more specifically, it is built upon the existing es-pnet @xcite codebase to better exploit prior community efforts and compare with existing non-speechlm works. we summarize espnet-speechlm and related codebases in tab.1. ## espnet-speechlm toolkit this section outlines the hierarchical design of the espnet-speechlm toolkit. we first introduce the fundamental concepts of speechlms in §3.1 followed by a detailed description of the espnet-speechlm workflow in §3.2. lastly, we highlight key features of our toolkit in §3.3. ## user cases this section provides several user cases to demonstrate the performance of speechlms built from espnet-speechlm. we first build single-task speechlm-style asr and tts models in §4.2. as a highlight of this demo, we present a 1.7b pretrained speechlm that covers 4 tasks similar to @xcite : asr, tts, text auto-regressive prediction (textlm), and speech auto-regressive prediction (audiolm) ( §4.3). these models are released through espnet huggingface hub foot_4 . ## future works we will continue the development of the espnet-speechlm toolkit, such as supporting more tokenization methods, more task templates, more modeling options, and llm inference engines @xcite . we are also interested in applying this toolkit to our speechlm research. for pre-training, we are interested in larger-scale models and models that can capture rich paralinguistic information in speech. for post-training, we are interested in achieving conversational interactions, speechbased instruction following ability, and even agentalike behaviors. our plan also includes real-time and duplex design for speechlm. ## conclusion this demo presents espnet-speechlm, a toolkit that covers the whole workflow of speech language model development, with comprehensive support in multiple design choices. we also provide user cases for both single-task and multi-task training, showing competitive performance with other models in the market. the toolkit promises to keep full transparency in data, code, recipes, and pre-trained models.