Producing S tandard G erman Subtitles for S wiss G erman TV Content (2022)

Abstract: In this study we compare two approaches (neural machine translation and edit-based) and the use of synthetic data for the task of translating normalised Swiss German ASR output into correct written Standard German for subtitles, with a special focus on syntactic differences. Results suggest that NMT is better suited to this task and that relatively simple rule-based generation of training data could be a valuable approach for cases where little training data is available and transformations are simple.

## introduction in switzerland, two thirds of the population speak swiss german, which is primarily a spoken language with many regional dialects. swiss german has no standardised written form @xcite , thus written communication relies on standard german. swiss german is widely used on swiss tv, for example in news reports, interviews or talk shows. in order to make these contents accessible to people who cannot understand spoken swiss german, either due to hearing impairments, or because they only understand standard german, these tv programs need to be subtitled in standard german. for daily tv content, where large amounts of subtitles need to be produced within a short time frame and in a cost-effective manner, being able to automate the subtitling process would be advantageous. the passage project, which is the context of the present study, focuses on this task. subtitling can be automated by combining automatic speech recognition (asr) with intralingual machine translation to improve the output to achieve compliance with subtitling standards @xcite . in the passage project a first asr step is used to produce a normalised transcription of spoken swiss german, keeping the original syntax and expressions but only using standard german words. in a second step, a neural machine translation (nmt) and an edit-based approach are explored to transform this normalised transcription into correct written standard german (see figure 1 for an example). to achieve this, multiple issues must be dealt with: asr errors, incorrect detection of sentence boundaries, features related to spontaneous spoken language, such as dysfluencies or informal language, and finally the syntactic divergences between swiss german and standard german @xcite . in the present study, we focus on the second step, and more specifically on the systems' ability to transform swiss german syntactic phenomena into their standard german counterparts. we compare different approaches and investigate whether additional synthetic training data targeting these phenomena can improve the models. to evaluate the systems' performance on this task, we perform human evaluations of several test suites. the paper is structured as follows, section 2 introduces the syntactic phenomena we have focused on, section 3 presents the data and architectures used, followed by section 4 which describes the evaluation approach. results are given in section 5. section 6 presents our conclusions and directions for future work. ## syntactic divergences between swiss german and standard german the syntactic differences between swiss german and standard german can be classified into two main types: features related to the mainly spoken usage of swiss german on one hand and dialectspecific features on the other @xcite . the latter are language phenomena involving among others the positioning of verbal forms, the construction of clauses or the use of cases and pronouns. these phenomena also differ from region to region @xcite , thus the tv content, which includes transcripts of speakers from all regions of german speaking switzerland, covers a large number of variations. for this study, we have focused on a subset of phenomena that occur in our corpora and that require different transformations: in this section we describe the initial data that were provided to build the systems, the aligned and synthetic corpora that were derived from these data, and the different architectures that we have used. ## evaluation methodology the objective of our systems is to convert as many swiss german syntactic phenomena as possible into standard german, while not introducing any additional errors into the asr output. to assess the systems' performance, we have therefore performed two human evaluations, as described in the following sections. ## conclusion in this study we have compared two architectures and the use of synthetic data for the task of translating normalised swiss german asr output into correct written standard german, with a special focus on syntactic differences. in terms of syntactic transformations, the nmt systems outperform the edit-based systems. we observe large differences between the studied phenomena, some being transformed more successfully than others. for nmt, the addition of targeted synthetic training data improves the results, producing a larger number of transformed phenomena while also having a slight positive impact on precision. these results suggest that the relatively simple rule-based generation of training data could be a valuable approach for cases where little training data is available and transformations are simple (e.g. inversion, insertion or replacement). while results are promising, this study presents several limitations. we have only studied a subset of the syntactic phenomena that distinguish swiss german from standard german. additionally, due to the constraints of human evaluation, only a limited set of data could be included. in terms of synthetic training data, we have only aimed to reproduce the syntactic phenomena, but not the orality markers which are very frequent in the asr output the systems need to deal with. finally, the evaluation in this study was focused on system performance in terms of performed edits. an ongoing evaluation with different target groups will show whether these syntactic changes have an impact on understandability, accessibility and general satisfaction. future work includes extending to other phenomena and specialising with different settings.