Aligning Speech Segments Beyond Pure Semantics (2024)

Abstract: Multilingual parallel data for speech-to-speech translation is scarce and expensive to create from scratch. This is all the more true for expressive speech translation, which aims at preserving not only the semantics, but also the overall prosody (e.g. style, emotion, rate-of-speech). Existing corpora contain speech utterances with the same meaning, yet the overall prosody is typically different, as human annotators are not tasked with reproducing these aspects, or crowed-sourced efforts do not specifically target this kind of alignment in priority. In this paper, we propose a novel alignment algorithm, which automatically forms pairs of speech segments aligned not only in meaning, but also in expressivity. In order to validate our approach, we train an expressive multilingual speech-to-speech translation system on the automatically aligned data. Our experiments show that in comparison to semantic-only approaches, expressively aligned data yields large improvements in source expressivity preservation (e.g. 43% uplift in speech rate preservation on average), while still maintaining content translation quality. In some scenarios, results also indicate that this alignment algorithm can outperform standard, semantic-focused approaches even on content translation quality.

## introduction in traditional machine translation (mt), the underlying goal is to preserve the meaning of the source. however, more recently there have been efforts to develop expressive speech-to-speech translation systems (s2st) @xcite , where the aim is to maintain not only the meaning, but also the expressivity of the source speech (e.g. tone, emotion, style, etc). while different parts of translation models can be pretrained in an unsupervised manner, large amounts of highquality end-to-end data remains crucial to achieve the best performance. on one hand, human-curated parallel data for the text domain (bitexts), are freely available for several languages, for instance the well known europarl @xcite or un corpora @xcite . on the other hand, human created aligned speech-to-speech parallel data is a scarce resource. to complement existing speech parallel data, automatic alignment algorithms have evolved as an important technique to provide additional data for a large number of languages and domains @xcite . however, these speech-to-speech alignment algorithms search only for speech segments with the same semantics, totally disregarding expressive properties of the source and target speech. in this paper, we extend similarity-based speech-to-speech alignment with an expressive criterion. our experiments show that the use of expressively aligned data substantially boosts the preservation of expressivity in a multilingual expressive s2st system. in addition, we observe that in some cases the expressively aligned data also improves content translation quality on some test sets. the main contributions of this work are: ## methodology following duquenne et al. (2023a), we first preprocess the data by segmenting the speech signal into plausible segments using a voice activity detector model, followed by language identification of each segment, and then subsequently encode each speech segment into a multilingual embedding space introduced by @xcite . once all segments are encoded, we then perform a k-nearest-neighbor search in the embedding space using the faiss library @xcite which allows for efficient search at scale. we then calculate a margin-based score over each candidate neighbor which has been shown to yield alignments which have a similar meaning @xcite . in this work, we use the ratio margin r, defined as: r(x, y) = cos(x, y) since the multilingual speech encoders are trained with a teacher-student method using a textbased encoder as the teacher, this forces the speech representation to focus on semantics, ignoring other elements of speech such as the prosody @xcite . therefore, the k-nearest-neighbors retrieved in the embedding space above are likely based on semantic characteristics only. however, it is possible that from the neighbors retrieved, that the prosodic characteristics of some may better preserve the source than others. in order to capture such signals, in comparison to previous approaches such as @xcite which choose a neighbor based on semantics only, we instead choose a neighbor which maximises a blend of both semantics: r, with another term related to the prosodic similarity of the speech segments: p . we define our expressive scoring function e as follows: instead of modifying the nearest neighbor search, a more straight-forward strategy could involve filtering existing speech alignments such as speech-matrix @xcite with a prosodic scoring function. however, the volume of the resulting dataset would likely be drastically reduced as no explicit prosody-preservation goal was enforced to begin with during the nearest neighbor search (i.e. an expressively and semantically aligned neighbor would be chosen by chance). ## experiment in order to validate the effectiveness of expressive alignments, we perform a controlled experiment where we align a set of raw monolingual speech data using both a semantic-only scoring function, and our expressive scoring function. we then subsequently train two expressive s2st systems on the resulting alignments from both approaches, and evaluate the results. ## related work research on aligning texts was initially based on document meta-information @xcite , crosslingual document retrieval @xcite or machine translation and information retrieval @xcite . however, many current alignment techniques are based on a similarity measure in a multilingual embedding space @xcite . one such technique for aligning text was introduced by @xcite , which uses a margin-based measure of similarity @xcite in order to determine the candidacy of potential alignments. this technique was then extended to the speech modality @xcite , where it was used to create high-quality speech-to-speech aligned data such as speechmatrix @xcite which covers seventeen languages, and seamlessalign @xcite which provides a total of 585k hours in 95 languages. both corpora consider only semantics during the alignment procedure. ## conclusion we introduce the first speech-to-speech alignment method which can align speech not only in terms of semantics, but also the expressivity. we validate our method by performing large-scale speech-tospeech alignment, and train an expressive s2st model on the resulting data. our results show that expressive alignments can further boost the capability of expressive models, where such speechto-speech data is extremely scarce, and in some instances even improve content translation quality.