E mph A ssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models (2024)

Abstract: We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis. We apply this to two tasks: speech resynthesis and speech-to-speech translation. In both cases, the benchmark evaluates the ability of the model to encode emphasis in the speech input and accurately reproduce it in the output, potentially across a change of speaker and language. As part of the evaluation pipeline, we introduce EmphaClass, a new model that classifies emphasis at the frame or word level.

## introduction in recent years, significant advancements have been made in the development of self-supervised learning (ssl) models for speech, extending beyond the traditional text-only methods prevalent in the field @xcite . such speechbased models find successful application across various domains from generative language modelling @xcite @xcite to speech-to-speech translation (s2st) @xcite @xcite @xcite . unlike text-only models, they exploit additional cues present in the speech signal which are absent in textual input. one crucial speech-only cue is prosody. also termed the "music of speech" @xcite , prosody is marked by the perceived loudness, rhythm, and pitch of speech. prosody not only adds naturalness to an utterance but also has the capacity to modify the meaning of the conveyed message, both at a global level, such as in the expression of different emotions, and at a local level, by influencing the interpretation of individual phrases or words @xcite . for * currently at apple instance, slower speech may suggest hesitation, while altering something like pause placement can actually change the segmentation into words or syntactic constituents, with downstream consequences for the meaning. hence, accurately capturing these prosodic elements is essential in ssl speech models for any application @xcite . to address this, @xcite proposed explicitly adding prosodically-relevant information such as fundamental frequency and duration to the speech representations models learn, while others aimed at explicitly modelling emotions in such representations @xcite . although some progress has been made, robust evaluation metrics for prosody remain scarce, and human evaluation, while insightful, is subjective -which can limit reproducibility; as well as being expensive and time intensive -which can hinder its utility in large-scale applications. objective evaluations of prosody fall into two main categories: one focuses on utterance-level features like emotion and speech rate to assess global prosody, and the other examines local prosody, which is concerned with prosodic effects at the level of a word or a phrase, such as breaks, turn ends and emphasis. in addition, one may address prosody for two classes of models: generative decoder-only models (the speech equivalent of gpt @xcite (e.g. gslm, @xcite @xcite ), and speech-tospeech (encoder-decoder) approaches, which take speech as input and produce output in a different voice (speech resynthesis) or a different language (s2st). in this paper, we address the second class of models. in the context of speech-to-speech (s2s) models, evaluating global prosody can be relatively straightforward, as the features are not directly related to the lexical content. the assessment of local prosody, however, presents more of a challenge, as it necessitates mapping at the lexical level. this can be relatively feasible in the context of speech resynthesis, where the model directly reconstructs the input signal and, therefore, preserves lexical content (e.g., by correlating prosodic attributes such as duration and fundamental frequency (f0) between input and output utterances; @xcite . however, this becomes more complicated when evaluating s2st models, as one needs to ensure the correct prosodic feature is applied to the correct word(s) @xcite (alignment problem). although scarce, there have been recent efforts made to establish benchmarks in the prosodic evaluation of speech models allowing models comparison, including evaluation corpora and pipelines, both at the global prosodic level (pragmatic information : @xcite ) and at the local prosodic level (prosodic pauses: de seyssel et al. ( 2023) ). yet, there is a need for more benchmarks to cover other aspects of prosody, and all types of speech models. in this work, we introduce the emphassess benchmark, which is focused on local prosody for speech-to-speech models and includes: (i) a new, automatic pipeline for emphasis evaluation that is modular, handles multiple languages and kinds of outputs (including paraphrases and translations, (ii) a novel dataset, the emphassess test set, for evaluating model emphasis preservation in english and spanish according to our pipeline, and (iii) emphaclass, an emphasis classifier that we finetuned with english data over an existing multilingual ssl model to support our pipeline. in fact, the issue of accurate emphasis transfer in s2st models has attracted some research attention over the years. studies by @xcite @xcite approach this topic using cascaded models (with separate automatic speech recognition, machine translation, and text-to-speech models). a more recent approach by @xcite integrates the two first components into a single encoder module capable of multilingual embeddings. similar to other prosodic features, emphasis in s2s models is primarily evaluated through human evaluation @xcite , although @xcite proposed leveraging an emphasis classification algorithm to calculate f1 scores by matching emphasised words in the input and output utterances. yet, this method is limited to a single language pair and cannot handle variations in translation outputs, only recognising one "gold" translation per dataset utterance. consequently, this metric is ill-suited for comprehensive automatic benchmarking across various models. word-level emphasis classification. as suggested by @xcite , a robust wordlevel emphasis classification system is critical in automatic evaluation of emphasis transfer in s2st models. existing algorithms, predominantly designed for text-to-speech applications, often rely on traditionally engineered features (e.g. mfccs or fbanks), sometimes augmented with other prosodic-related information (e.g. f0, duration) @xcite @xcite . some also incorporate lexical information from textual transcripts @xcite . however, these models frequently suffer from limited generalisability across different datasets, voice types, and languages. there is a compelling argument for using the speech waveform directly as input to enhance generalisability. to our knowledge, the only study to have adopted this approach is that of @xcite , which employed a crnn framework for classifying emphasis in children's speech; their work, however, was limited to a single language (and is not open-sourced). we propose that leveraging pretrained models trained on multilingual datasets could result in significant advancements in this field. where should the emphasis be? which word(s) from the transcription (if any) are emphasized? is the emphasis at (and only at) the correct location ? what is the transcription from the generated utterance? where are the transcription words' boundaries? a. output generation b. input output emphasis comparison precision : 1.0 recall : 1.0 f1 : 1.0 ## introducing emphassess in this study, we introduce emphassess, a versatile automatic benchmark for evaluating emphasis preservation in s2s models, including s2st ones. essentially, this benchmark comprises a carefully curated dataset of english utterances with emphasised words, accompanied by an automatic evaluation pipeline, and results on some of the most recent s2s ssl models. our evaluation framework, inspired by the methodology of @xcite , assesses emphasis alignment between the source and the model's output utterances. our benchmark's novelty lies in its capacity to handle various output types, including paraphrases and translations. guided by the data we have for setting optimal baselines, the emphassess benchmark is specifically designed for english-to-english and englishto-spanish s2s models. however, our work goes further, laying the groundwork for extending this benchmark to other language pairs. moreover, the evaluation pipeline itself is already capable of being applied to a broad spectrum of language pairs. also, while we focus here on unsupervised speech language models, emphassess is versatile enough to be applied to any s2s framework. the emphassess evaluation pipeline's modular structure is a key feature, with each module designed to function independently and allow for straightforward modifications. we leverage a suite of distinct open-source models, each finetuned for particular tasks. the pipeline can therefore be upgraded to incorporate improvements in each module seamlessly. although such enhancements may necessitate a re-evaluation of the models within our benchmark, this inherent adaptability is a considerable benefit, ensuring emphasses can remain current with the latest research for years to come. finally, we introduce and open-source, as part of this automatic evaluation pipeline, a novel emphasis classifier at the word level: emphaclass. this classifier is finetuned over an existing multilingual ssl model with the hope of enhancing its robustness across multiple languages and variability. the evaluation code, emphasis classifier and dataset introduced in this paper are available in our related repository foot_0 . ## the emphassess dataset the emphassess dataset comprises synthetically generated speech utterances, each containing at least one emphasised word. accompanying these utterances are metadata detailing the transcription, the positional index of the emphasised word(s), and information about the synthetic voice employed for synthesis. in total, the dataset boasts 3652 speech samples derived from 913 unique transcripts (with each transcript being rendered in 4 distinct voices). the dataset generation started with a selection of transcripts from a list of handwritten transcripts with emphasis annotations foot_1 previously created for company-internal text-to-speech purposes. transcripts containing characters beyond letters or specific punctuation marks foot_2 or those featuring proper nouns (identified using the nltk toolkit; bird 2006) were excluded, to ensure the translations are as straightforward as possible. moreover, we ensured a minimum of two distinct versions with different emphases for string identical sentences (those with matching word tokens but possibly differing emphasis position indices). this approach was adopted to mitigate any bias should a model exhibit a preference for emphasising a particular word over others. finally, we filtered out transcripts that could face alignment challenges with emphasised words during translation. we set up an algorithm to assess the difficulty of aligning emphasised words in an english sentence with their counterparts in multiple target languages, using the simalign word-alignment tool @xcite . simply put, if an emphasised word in the source matched consistently to a corresponding word across a list of other languages (german, french, spanish, and chinese), the sentence was labelled "easy"; otherwise, it was deemed "difficult." only "easy" transcripts were retained for our dataset. we were left with 913 distinct transcriptions (with varying emphases) derived from a pool of 299 unique transcriptions. we ensured that the distribution of transcripts was well balanced, in terms of where the emphasis was located. next, we employed an internal text-to-speech (tts) tool with a 16 khz sample rate to synthesise all 913 transcripts, each in the four distinct opensource english expresso voices (nguyen et al., 2023a), namely ex01, ex02, ex03 and ex04, resulting in a comprehensive set of 3,652 speech samples. finally, we compiled a dataset that is available as part of the benchmark. this dataset comprised four columns: an id column that denotes the unique identifier for each speech segment, a src_sentence column that contains the corresponding tokenised text transcript presented in list 5 the emphaassess evaluation pipeline the evaluation pipeline, as illustrated in figure 1 , is divided into two main stages. the first one (left panel) corresponds to the generation of utterances from the evaluated s2s model. that is, for each utterance from the emphassess dataset, we need to generate the corresponding utterance output from the evaluated model. hence, this inference stage is dependent on the model tested, and we will not expand on it here. in the second stage (right panel), we perform the automatic evaluation by comparing the input and output utterances. the objective is twofold: firstly, to ascertain whether the emphasis is retained in the generated utterance, and secondly, to determine whether the emphasis is correctly positioned on the corresponding word. at this stage, available resources include the input (original) utterance, the corresponding output utterance, and the tokenised transcript of the input with the location of the emphasised word(s) identified. a schematic overview of the evaluation pipeline is shown in the right panel of figure 1 . initially, we obtain a transcription of the generated utterance (1) and the timealigned word boundaries (2). this information can be used in addition to the raw waveform to detect emphasis at the word level in the output utterance using a classifier (3). at this stage, we must determine which word(s) in the generated utterance should be emphasised to obtain evaluation scores (4). we use word-to-word alignment at the text level to address this, a technique borrowed from the machine translation field. finally, we can use this information to compute precision, recall and f1 score (5). we will now detail our methodology for each of these steps. ## results we benchmarked a series of models on the em-phassess evaluation, both within language (english to english) and using translations (english to spanish). ## conclusion we have introduced an evaluation framework for emphasis in speech-to-speech (s2s) models. this framework comprises an english dataset, an automated evaluation pipeline, and a results benchmark focusing on english-to-english and englishto-spanish models. crucially, our framework offers a generalisable approach applicable to other language pairs, the only major requirement being the acquisition of a relevant dataset to establish a reliable gold standard. additionally, we have open-sourced an emphasis-classification model that has been finetuned on english data. the model builds on a multilingual ssl architecture and has shown impressive accuracy in classifying emphasised speech in english on our dataset, along with reasonable performance in other languages (for further details, refer to the appendix). the model's robustness in english makes it a plausible starting point for finetuning classifiers in other languages, potentially minimising the volume of data needed for training. interestingly, the fact that the successful results were achieved without retraining the encoder, suggests that the inherent features in the original xls-r model were adequate for emphasis classification. there is an existing agenda for future research centring around the evaluation of prosody within ssl models. firstly, on the subject of emphasis, we aim to scrutinise its functional role more closely-specifically, its ability to convey importance. we intend to investigate whether such a function is intrinsically represented within these models. beyond emphasis, other aspects of prosody, such as turn-taking and speech grouping, merit attention. we are interested in determining whether these elements, too, are encoded within ssl models. improved benchmarks and evaluations for these prosodic features could pave the way for the development of more expressive and nuanced models. to conclude, the emphassess benchmark sets a new standard for the evaluation of prosodic features in s2s models, offering both methodological contributions and actionable insights that could pave the way for more natural and effective machinegenerated speech across various applications.