MAD Speech: Measures of Acoustic Diversity of Speech (2025)

Abstract: Generative spoken language models produce speech in a wide range of voices, prosody, and recording conditions, seemingly approaching the diversity of natural speech. However, the extent to which generated speech is acoustically diverse remains unclear due to a lack of appropriate metrics. We address this gap by developing lightweight metrics of acoustic diversity, which we collectively refer to as MAD Speech. We focus on measuring five facets of acoustic diversity: voice, gender, emotion, accent, and background noise. We construct the metrics as a composition of specialized, per-facet embedding models and an aggregation function that measures diversity within the embedding space. Next, we build a series of datasets with a priori known diversity preferences for each facet. Using these datasets, we demonstrate that our proposed metrics achieve a stronger agreement with the ground-truth diversity than baselines. Finally, we showcase the applicability of our proposed metrics across several real-life evaluation scenarios. MAD Speech is made publicly available.

## introduction recent progress in generative spoken language modeling @xcite @xcite @xcite @xcite @xcite brought models that can generate speech in a wide range of voices, prosody and recording conditions as found in large-scale datasets of natural speech @xcite @xcite . these generative models are typically evaluated in terms of faithfulness to their inputs, i.e. speaker identity and transcript @xcite @xcite . thus it remains unknown how diverse their outputs are, since those metrics neither capture variability nor take into account factors of the perceived speech diversity (i.e., recording conditions, prosody, styles, and accents). being able to measure speech diversity would also help detect mode collapse when finetuning models @xcite , selecting training & inference hyperparameters, and optimizing for human feedback @xcite . moreover, such measurements are vital when building synthetic data or mixing existing natural datasets. in this work, we focus on measuring acoustic diversity of a set of speech utterances, i.e. measuring the diversity in the distribution of voices, genders, accents, emotions, and background noise. throughout the paper, we use the term "facets" to describe these different aspects of acoustic diversity. we propose a set of metrics, which we collectively refer to as mad speech (measures of acoustic diversity of speech). we build these metrics by combining a pretrained speech representation and an aggregation function that is used to calculate the diversity of a sample in the representation space. we experiment with a set of off-theshelf continuous speech representations such as hu-bert @xcite , wav2vec-bert @xcite , and soundstream @xcite as well as with speechsim, a lightweight representation model related to trill @xcite and cola @xcite . we explore the average pairwise cosine dissimilarity and vendi score @xcite as aggregation functions. further, we learn per-facet projection models with the goal of highlighting contributions of each facet "independently" from others. such a setup is beneficial since it (a) provides a better insight into the relative contributions of different aspects of diversity, and (b) allows to avoid fusing of the dif-ferent facets in the same metric with uncontrolled relative importance. to evaluate the proposed metrics, we build a collection of datasets with known ground-truth acoustic diversity levels. we achieve that by subsampling standard speech datasets, while controlling for some measurable parameter that reflects the diversity of the samples along a particular facet (e.g., number of unique voices). repeating that for all facets, we build a suite of datasets that can be used to measure to what extent a particular metric is capable of measuring acoustic diversity. finally, we run a set of studies to measure the impact of various improvements proposed in the literature on the acoustic diversity: fast non-autoregressive decoding of soundstorm replacing the auto-regressive semantic-to-acoustic model of audiolm @xcite and best-of-k decoding of spear-tts to increase quality of generated audio @xcite . we also investigate how changes in the temperature sampling affect the acoustic diversity and compare several off-the-shelf tts systems. these studies highlight that acoustic diversity changes non-trivially across the considered scenarios and it must be taken into account when selecting a model and a sampling setup. diversity metrics for generative audio already the first paper proposing a generative spoken language model (gslm) recognized the importance of measuring speech diversity @xcite . however, since gslm is not able to generate acoustically diverse content (e.g., it supports only a predefined set of speakers), the focus of @xcite was on measuring the diversity on the transcript level. in a follow-up paper, @xcite proposed a modification of gslm that has a more expressive prosody yet their focus was on improving and measuring withinutterance variability of prosody. to the best of our knowledge, our work is the first to study acrossutterance acoustic diversity in speech generation. ## learning representations we take a two-steps approach for measuring diversity of a set of speech utterances. first, we encode raw waveform audio in a compact representation and, afterwards, we estimate the diversity of a set of utterances in this embedding space. hence, it is vital that the information about the relevant properties of speech is preserved in the embedding space, while the irrelevant details are omitted. we calculate embeddings in a hierarchical way: at first, we map audio to generic embeddings and then apply lightweight specialized projection models that are tailored to capture a specific facet. as a result, our approach allows us to get reliable per-facet diversity metrics. more details can be found in section 6.1. speechsim we aim to learn speech representations where acoustically similar audio segments are close to each other in the embedding space. to do so, we train a self-supervised contrastive model with positive pairs coming from non-overlapping fixed-length chunks of the same speech utterance. the main idea being that such speech segments are likely to be acoustically similar. we refer to this model as speechsim. speechsim provides a speech representation that are sensitive to changes in acoustic diversity, however, it has all types of acoustic properties mixed up together. indeed, our experiments in section 7 show that the vanilla speechsim embeddings are strongly biased towards speaker voice. we therefore learn projection models (heads) on top of speechsim embeddings to extract different types of acoustic information. ## from speech representations to diversity metrics once a set of speech utterances is encoded with one of the representations (section 3), we can measure the diversity within the set in a variety of ways. we focus on two measures: mean pairwise dissimilarity and vendi score @xcite . mean pairwise dissimilarity denoting the sample size as n and embeddings as e, we calculate: we subtract the mean cosine similarity from 1, as it is more convenient to measure diversity (dissimilarity) rather than similarity. this way, an agreement with the ground-truth diversity would result in a positive correlation (see section 5). vendi score @xcite proposed to measure diversity of a sample as the exponential of the shannon entropy of the eigenvalues of the pairwise similarity matrix, normalized by the number of vectors. in practice, we built it by computing pairwise cosine similarity and dividing by the number of examples. formally, denoting the eigenvalues of the normalized similarity matrix as {λ k } n @xmath0, vendi score is calculated as: eq. 2 is well-defined, as the normalized similarity matrix is positive semi-definite and 0 ≤ λ @xmath1. ## experimental setup details about the training data and the baseline representations we use can be found in appendix a and appendix b respectively. ## evaluating mad speech in this section, we use the collection of per-facet diversity benchmarks (section 5) to assess which speech representations are more suitable for estimating diversity of a set of speech utterances when only one facet is changed (section 7.1). next, we run a series of experiments with two facets changed in the opposite directions (section 7.2). ## comparing generative models in this section, we show that despite that the acoustic diversity was mostly ignored in the literature when proposing new models and approaches, it does change. we start by assessing whether the introduction of soundstorm @xcite to the audiolm framework @xcite lead to any hidden penalties in acoustic diversity (section 8.1) before comparing acoustic diversity of off-the-shelf tts models (section 8.2). additional studies on the effect of the best-k sampling technique used by @xcite on acoustic diversity and how temperature changes acoustic diversity of the synthesized speech can be found in appendix c.1 and appendix c.2. in this section, we use the same terminology as @xcite where semantic tokens refer to tokens learned with a self-supervised masked language modeling objective @xcite while acoustic tokens refer to tokens learned with a full reconstruction objective from a residual vector quantizer @xcite . ## conclusions & future work we introduced mad speech, a set of metrics for evaluating acoustic diversity in speech. mad speech metrics are calculated in two steps: (1) a set of utterances is mapped to an embedding space, (2) acoustic diversity is estimated by comparing the resulting embeddings with an aggregation function (cosine dissimilarity or vendi score). calculating embeddings is done by firstly applying a generic representation model with a subsequent metric-specific projection. we focus on five facets of acoustic diversity: voices, gender, emotion, accents, and background noise. in order to validate the proposed metrics, we built a collection of datasets with a controlled level of acoustic diversity. having these sets allows us to evaluate a metric by calculating its spearman rank correlation with the ground-truth acoustic diversity. our empirical study demonstrated that our proposed metrics have a highest agreement with the ground-truth diversity levels when compared to baseline approaches and that they are insensitive to variations of other facets of diversity. finally, we highlighted a necessity of controlling acoustic diversity when developing new models and approaches. to this end, we took some recent modelling improvements and demonstrated that they, in fact, affect acoustic diversity of speech.