Prosody in Cascade and Direct Speech-to-Text Translation: a case study on K orean Wh-Phrases (2024)

Abstract: Speech-to-Text Translation (S2TT) has typically been addressed with cascade systems, where speech recognition systems generate a transcription that is subsequently passed to a translation model. While there has been a growing interest in developing direct speech translation systems to avoid propagating errors and losing non-verbal content, prior work in direct S2TT has struggled to conclusively establish the advantages of integrating the acoustic signal directly into the translation process. This work proposes using contrastive evaluation to quantitatively measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Specifically, we evaluated Korean-English translation systems on a test set containing wh-phrases, for which prosodic features are necessary to produce translations with the correct intent, whether it’s a statement, a yes/no question, a wh-question, and more. Our results clearly demonstrate the value of direct translation systems over cascade translation models, with a notable 12.9% improvement in overall accuracy in ambiguous cases, along with up to a 15.6% increase in F1 scores for one of the major intent categories. To the best of our knowledge, this work stands as the first to provide quantitative evidence that direct S2TT models can effectively leverage prosody. The code for our evaluation is openly accessible and freely available for review and utilisation.

## introduction speech-to-text translation (s2tt) is the task of automatically generating a text translation in a target language given an input speech signal. traditionally, s2tt has been achieved by concatenating two systems: one in charge of generating an intermediate transcription of the source speech signal and one of translating the intermediate text into a target language. although such a pipeline, known as "cascade" architecture, remains the dominant technology in speech-to-text translation, it has some shortcomings. firstly, it is affected by error propagation for which errors in the transcription phase are carried over and amplified in the translation phase. secondly, some information is lost as non-verbal content (e.g. prosody) is discarded from the text. as a potential solution to these issues, "direct" systems that can perform translation directly from speech signals without needing intermediate transcriptions have emerged in the last few years. @xcite claim direct systems have an advantage over the cascade architecture by modelling prosody during the translation process. however, there is no conclusive evidence to support this claim as both types of systems have similar overall performances, and current datasets do not regularly include instances where speech signals are necessary to disambiguate the meaning of an utterance, making quantitative analysis on the effect of prosody in s2tt particularly challenging @xcite . the aim of this paper is to investigate the potential of direct s2tt to effectively leverage nonlexical information, particularly prosody, and quantify their impact. since identifying ambiguous utterances that rely on prosody for disambiguation is nontrivial, especially in english where sentence structure typically carries more weight than prosodic cues, we focus on korean wh-phrases where the presence of a prosodic boundary distinguishes wh-interrogatives from wh-indefinites (e.g., ᄋ ᅥᄃ ᅵᄀ ᅡ ᆻᄋ ᅥᄋ ᅭ (eodi gasseoyo) → where did you go?/did you go somewhere?), as well as other interpretations. in this paper, we (i) introduce a new contrastive evaluation framework for korean-english s2tt systems, designed for ranking translations of ambiguous utterances containing wh-particles; (ii) quantitatively demonstrate the capacity of direct s2tt systems to effectively model prosodic cues from the input, yielding an overall improvement over cascade models of 12.9% in accuracy for ambiguous utterances, and up to a 15.6% increase in f1 scores within one of the major intent types; (iii) highlight the limitations of punctuations in disambiguating certain intent types despite being strong signals in distinguishing questions from statements. ## korean prosody and wh-particles prosody refers to the acoustic features that are exhibited across multiple phonetic segments, also known as suprasegmental features @xcite . these suprasegmental features can take shape in a multitude of ways. for example, by stressing a single word in a phrase (phrasal stress), by adding pauses or modifying the length of syllables (boundary cues) or by varying the tonal and stress patterns in the utterance (metre) @xcite . in an intonational language like korean, the intended meaning of an utterance is often conveyed via intonation and rhythm instead of lexical pitch accents or tones @xcite . while prosodic structures in korean utterances are still debated, there are at least two levels of prosody above the word: the accentual phrase (ap) and the intonation phrase (ip). the ap is the basic unit for prosodic analysis marked by a tonal pattern thlh which consists of variations of the pitch between low (l) and high (h), with t being either l or h depending on the phrase's initial segment, while the ip consists of one or more aps and a boundary tone on the right edge of the phrase. korean wh-particles are an example of a linguistic phenomenon where the tonal patterns and ip boundary tones are necessary to disambiguate the meaning of the utterance, as otherwise they can be interpreted as both interrogative particles or indefinite pronouns (e.g. ᄂ ᅮᄀ ᅮ" (nugu) → "who" / "somebody"). figure 1 shows the pitch contours for the recordings of the utterance "ᄂ ᅮᄀ ᅡᄀ ᅡᄋ ᅵ ᆸᄒ ᅢ ᆻᄃ ᅢᄋ ᅭ" (nuga gaiphessdaeyo). by varying the boundary tone h+l%, h+lh%, and l+h%, the utterance can be interpreted as a statement, yes/no question or wh-question respectively. ## contrastive evaluation contrastive evaluation is an automatic accuracybased evaluation technique that measures the capability of a system to distinguish correct from incorrect outputs. this is achieved by asking a generative model θ to score and rank a set of predefined outputs, each containing a correct and a contrastive utterance (e.g., "the cat sleeps" vs. "the cat sleep" @xcite ). following previous work @xcite , we define the score of an utterance as the sum of the target token log probabilities normalised by the length of the full target sequence y : in this work, we perform contrastive evaluation of cascade and direct s2tt systems on korean whphrases. since multiple prosodic realisations can occur per utterance (as in figure 1 ), in contrast to previous work where only one contrastive utterance per example was available, we consider a model having correctly identified the intended translation only if its score is higher compared to the score of all the possible incorrect translations. in addition to the general accuracy of the model in identifying the correct translation, we report contrastive precision, recall and f1 scores of the systems on the various wh-phrases' intent types. ## experimental setting in our experiment, we adopted the prosem corpus @xcite recorded by two korean native speakers of a different gender. all the utterances make use of one of the six korean wh-particles and are further classified into seven intent categories: statements, yes/no questions, wh-questions, rhetorical questions, commands, requests, and rhetorical commands, with the first three categorised as major intent types. for our experiments, we utilise state-of-the-art pretrained models. specifically, we use open ai's whisper models @xcite for both the s2tt direct systems and the asr components in the cascade systems, reporting results obtained from all the provided multilingual models. as for the mt component in the cascade systems, we make use of the korean-english baseline model provided for the tatoeba challenge (tiedemann, 2020), trained on approximately 34.5m opus mt parallel data @xcite . ## conclusion the objective of this paper was to test whether direct s2tt systems could take advantage of the prosodic information contained in the speech signal. to achieve this, we conducted quantitative analyses focused on korean wh-particles which can represent either wh-interrogatives or wh-indefinites encompassing a range of intents in accordance with the input acoustic features. our contrastive evaluation results provide compelling evidence that the direct s2tt systems outperform the cascade systems in overall accuracy and f1 score across all the major intent types on ambiguous utterances. cascade systems perform better than random primarily thanks to the inclusion of punctuation in the transcriptions. however, it's essential to note that while punctuation marks play a valuable role in aiding disambiguation, they are not sufficient to resolve all types of intents, emphasizing the importance of considering prosody in s2tt systems.