Translating Dislocations or Parentheticals : Investigating the Role of Prosodic Boundaries for Spoken Language Translation of F rench into E nglish (2023)

Abstract: This paper examines some of the effects of prosodic boundaries on ASR outputs and Spoken Language Translations into English for two competing French structures (“c’est” dislocation vs. “c’est” parentheticals). One native speaker of French read 104 test sentences that were then submitted to two systems. We compared the outputs of two toolkits, SYSTRAN Pure Neural Server (SPNS9) (Crego et al., 2016) and Whisper. For SPNS9, we compared the translation of the text file used for the reading with the translation of the transcription generated through Vocapia ASR. We also tested the transcription engine for speech recognition uploading an MP3 file and used the same procedure for AI Whisper’s Web-scale Supervised Pretraining for Speech Recognition system (Radford et al., 2022). We reported WER for the transcription tasks and the BLEU scores for the different models. We evidenced the variability of the punctuation in the ASR outputs and discussed it in relation to the duration of the utterance. We discussed the effects of the prosodic boundaries. We described the status of the boundary in the speech-to-text systems, discussing the consequence for the neural machine translation of the rendering of the prosodic boundary by a comma, a full stop, or any other punctuation symbol. We used the reference transcript of the reading phase to compute the edit distance between the reference transcript and the ASR output. We also used textometric analyses with iTrameur (Fleury and Zimina, 2014) for insights into the errors that can be attributed to ASR or to Neural Machine translation.

## introduction in french @xcite , as in different languages @xcite , parentheticals have specific prosodic patterns. several papers have shown the crucial role of prosodic boundaries for dislocations in french @xcite @xcite . we aimed to investigate the effect of prosodic boundaries and analyse whether the second prosodic boundary of the parenthetical was accurately translated and distinguished from the final rise of the left periphery dislocations. our small-scale analysis compares a successive pipeline including vocapia and systran automated speech translation generated by systran pure neural server (spns9) foot_0 and a multitask multilingual pipeline using whisper, an automatic speech recognition (asr) system trained on audio data for transcription and translation. speech technologies such as automatic speech recognition were already coupled with automatic translation within phrase-based statistical machine translation (pbmt) @xcite . as part of a more general project on error evaluation of asr systems, era project (advanced errors analysis for speech recognition), asr systems have been analysed in @xcite . in a more pragmatically oriented paper, eight asr platforms were assessed for accuracy and time-saving purposes on five documents from different fields of research in humanities @xcite . preliminary investigations of chatgpt-3 for translation suggest a better performance @xcite and @xcite ; other audio large language models have been built for speech recognition such as lxsr-53 large model @xcite ). the whisper paper describes its performance in relation to other systems such as mslam @xcite ), a multilingual speech and language model that learns cross-lingual cross-modal representations of speech, trained on librispeech and other resources like a thousand hours of speech from babel. the rest of the paper is organised as follows: section 2 details our experiments; section 3 presents the results; section 4 discusses them and outlines further research. ## experiments and results our analysis compares two translation pipelines for spoken data: asr (vocapia) then translation (spns9 mt engine) and audio llm translation output from the speech signal with whisper. in the whisper output, the numbers of segments produced by the different models do not match for translation and transcription tasks, so that we can reasonably assume that the translation is not based on the transcription. we both resort to quantitative and to qualitative analyses. we report wer and bleu scores for the different systems, analyse vocabulary growth curves for the data sets produced by different models and then discuss some characteristic phenomena, such as the use of punctuation marks ("." and ",") more qualitatively. ## discussion this experiment is really intended as a pilot study, we did not control for the effect of speech rate nor did we rely on inter-annotator agreement for the reference transcription textgrid of the time stamps represented figure 5 . ## conclusion in this paper, we compared the performance of different whisper models for the translation task from french into english. we compared these multilingual models trained on multimodal data with systran pure neural server translations, generated from the transcribed text and from the vocapia asr output, and analysed the different translation outputs. whisper large models and vocapia fared better, but for whisper some translations generated by smaller size models were more accurate for some sentences, including a better containment of the gender bias effect. for translations, the main finding is that the medium model does much better on the transcription task than for the translation task, probably because in our data the translation segment often corresponds to two utterances on the sound files. for the transcription task, the key finding is the apparent anarchic distribution of time stamps across models for the same speech signal. more research is needed to better understand the time interval (mis)management of the whisper transcriptions and translations encoded in the srt file outputs. should whisper be used for the translations of subtitles, one may wonder about the absence of pauses in the time stamps. more research is needed to evaluate the potential architectural effect of the training of whisper on 30s windows of speech. outputs for all test models. jean-baptiste yunès implemented the jupyterhub, supervised some of the experiments and conducted impact measurements. all authors contributed to the analysis of the outputs.