VERSA : A Versatile Evaluation Toolkit for Speech, Audio, and Music (2025)

Abstract: In this work, we introduce VERSA, a unified and standardized evaluation toolkit designed for various speech, audio, and music signals. The toolkit features a Pythonic interface with flexible configuration and dependency control, making it user-friendly and efficient. With full installation, VERSA offers 65 metrics with 729 metric variations based on different configurations. These metrics encompass evaluations utilizing diverse external resources, including matching and non-matching reference audio, text transcriptions, and text captions. As a lightweight yet comprehensive toolkit, VERSA is versatile to support the evaluation of a wide range of downstream scenarios. To demonstrate its capabilities, this work highlights example use cases for VERSA, including audio coding, speech synthesis, speech enhancement, singing synthesis, and music generation. The toolkit is available at https://github.com/shinjiwlab/versa.

## introduction with the rapid advancements in artificial intelligence-generated content (aigc), deep generative models have demonstrated remarkable capabilities in producing high-quality outputs across various domains, including image, video, and sound generation. as generative models become increasingly sophisticated, the need for comprehensive aigc evaluation has grown, aimed at identifying the strengths and weaknesses of the generated outputs. as an essential part of the language processing community, diverse generative models for speech, music, and general audio have shown significant potential in applications such as conversational interfaces @xcite , entertainment @xcite @xcite , and task management @xcite . due to the perceptual nature of sound-based applications, human subjective assessment is widely regarded as the gold standard for evaluating sound generative models. the most fundamental and widely used metric for these models is the mean opinion score (mos) @xcite . the initial purpose of mos was to measure the naturalness of generated audio, but it has since evolved into various specific forms, such as evaluating speaker similarity @xcite , comparative performance against baseline systems @xcite , emotional similarity @xcite , and alignment with prompts @xcite . despite its importance, subjective evaluations relying on human input are challenging to conduct due to their laborintensive nature. furthermore, achieving statistically significant results often requires a substantial number of samples @xcite . additionally, range-equalizing bias is frequently observed in mos evaluations due to the psychological grounding of human subjective assessments @xcite . such biases introduce considerable challenges in achieving comparable results across different evaluation datasets and participants, thereby complicating the process of benchmarking generative models effectively. last but not least, certain evaluation variants may require individuals with expert knowledge to conduct the assessment, particularly in the music domain @xcite . an alternative approach is to develop automatic metrics that align with human preferences. the design of such metrics can vary based on the use of external references and the specific application scenarios, which further complicates their selection. the basic setup for evaluation involves using only the candidate audio being evaluated @xcite @xcite , which has been actively discussed in recent voicemos challenges @xcite @xcite , 2024b). in addition, a variety of external resources can be optionally utilized for evaluation, as illustrated in fig. 1 . these resources may include matching reference audio @xcite , non-matching reference audio @xcite , textual transcription @xcite , textual captions @xcite , or visual cues @xcite . 1 the focus of automatic metrics can vary significantly depending on the application scenario. for instance, in voice conversion, discriminative speaker embeddings can be employed to measure speaker similarity between the generated speech and speech from the same speakers @xcite . in cases involving domain-specific content, such as singing, pre-trained models in singing voices may better align with assessments of singing naturalness @xcite . for a finegrained, sample-level generation, signal-to-noise ratio-related metrics are more suitable, particularly for tasks such as speech enhancement and separation @xcite . due to the creative nature of music, distributional metrics are often better suited for evaluating music generation @xcite . given the diversity of audio signals, effectively evaluating them requires a broad understanding of general audio events and their contextual relevance @xcite . considering the diversity of metrics and the evolution of generative models for speech, audio, and music, the need for standardized evaluation metrics has become increasingly evident. without a unified framework, the diversity in evaluation methodologies often leads to inconsistent results, making it challenging to benchmark models or assess advancements effectively. standardization ensures that all systems are evaluated under comparable conditions, fostering fairness, reproducibility, and meaningful insights across studies. moreover, centralizing metrics within a single toolkit not only reduces redundancy and inefficiencies but also encourages collaboration by providing researchers with a shared foundation for assessing performance. this need for consistency and centralization underpins the development of versa, a toolkit designed to address these challenges. extended from its prior version in @xcite , this work introduces a versatile evaluation toolkit for speech, audio, and music: versa. built on a pythonic interface, versa integrates 65 metrics and more than 729 variants, offering a wide array of automatic evaluation tools tailored for speech, audio, and music. by providing diverse metrics, versa aims to serve as a one-stop solution for multi-domain, multi-level, and multi-focus sound evaluation across various downstream applications. with examples showcasing the use of versa in espnet-codec and espnet-speechlm, we anticipate that the versa toolkit will become a key component in advancing sound generation benchmarks, addressing challenges in speech generation, and supporting multi-modal generative frameworks. the codebase is publicly available at https://github.com/wavlab-speech/versa . foot_1 ## versa this section details the design of versa, focusing on its general framework, supported metrics, and potential benefits for the community. ## comparison to other toolkits as discussed in sec. 1, the challenges associated with subjective evaluation have propelled the community toward exploring objective evaluation metrics. the growing demand for sound evaluation toolkits has led to numerous efforts in domainspecific evaluation for sound generation. in the speech domain, prior to the deep learning era, the international telecommunication union telecommunication standardization sector (itu-t) played a pivotal role in designing evaluation metrics for speech processing tasks such as speech coding and speech enhancement. more recently, text-to-speech (tts) toolkits have begun incorporating speech quality assessment features, exemplified by espnet-tts @xcite and amphion @xcite . additionally, the speech human evaluation estimation toolkit (sheet) framework provides an allin-one recipe-style toolkit for data preparation, speech quality prediction model training, and evaluation @xcite . in speech enhancement, foundational signal-level metrics were first consolidated in @xcite , followed by extensions such as pysepm, which supports 10 met-rics @xcite . more recent advancements include the addition of 14 speech enhancement metrics in clearervoice (clearervoice-studio, 2023). in the audio domain, audioldm-eval focuses on evaluating audio language models with nine types of metrics @xcite . stability ai has introduced three audio metrics (stability-ai, 2023), while sony csl has open-sourced four additional types of audio metrics @xcite . in the music domain, mir_eval is a pioneering toolkit that aggregates metrics for music information retrieval tasks @xcite . more recently, microsoft released fadtk, which emphasizes a comprehensive fad-embedding space for generative music evaluation @xcite . a summary of the related toolkits is presented in ## demonstration we demonstrate several use cases of versa in diverse scenarios, including speech coding, speech synthesis, speech enhancement, singing synthesis, and music generation. speech coding remains one of the most widely utilized applications within the speech-processing community. in this demonstration, we leverage versa to evaluate nine publicly available codecs. more details and corresponding results are discussed in ## conclusion in this work, we introduced versa, a comprehensive and versatile evaluation toolkit for assessing speech, audio, and music signals. with its flexible pythonic interface and extensive suite of metrics, versa empowers researchers and developers to conduct rigorous and reproducible evaluations across diverse generative tasks. through its integration of more than 65 metrics and 729 variants, versa provides unparalleled support for evaluating speech synthesis, audio coding, music generation, and more. the toolkit not only simplifies the evaluation process but also addresses challenges such as bias in subjective evaluations and the need for domain-specific expertise.