Improving S tandard G erman Captioning of Spoken S wiss G erman: Evaluating Multilingual Pre-trained Models (2023)

Abstract: Multilingual pre-trained language models are often the best alternative in low-resource settings. In the context of a cascade architecture for automatic Standard German captioning of spoken Swiss German, we evaluate different models on the task of transforming normalised Swiss German ASR output into Standard German. Instead of training a large model from scratch, we fine-tuned publicly available pre-trained models, which reduces the cost of training high-quality neural machine translation models. Results show that pre-trained multilingual models achieve the highest scores, and that a higher number of languages included in pre-training improves the performance. We also observed that the type of source and target included in fine-tuning data impacts the results.

## introduction in switzerland, over 60% of the population speaks swiss german, which is a collection of spoken dialects with many regional variations. swiss german is widely used in daily life and in the media, both on the radio and on swiss tv. as these dialects lack a standardised written form, standard german is often used for written communication, captions and subtitles. standard german is also used to make swiss german content accessible to people who cannot understand the dialects. the passage project @xcite , which is the product of a collaboration between srf and recapp it, aims at making swiss tv shows more accessible by automatically generating standard german captions for spoken swiss german using a cascade approach. figure 1 illustrates the two main steps. in a first step, our project partner's asr transcribes spoken swiss german into normalised swiss german, maintaining the original syntax and expressions of swiss german, but using german words @xcite . a second step, using machine translation (mt) approaches, aims at transforming these normalised transcriptions into fully correct standard german. our contribution to this pipeline focuses on the mt step. in this context, mt could be used to different ends @xcite . in our case, the objective is a minimal trans-formation to produce a correct standard german transcription. this mainly involves resolving divergences between swiss and standard german by performing syntactical and lexical transformations, correcting speech recognition issues and removing spoken language phenomena such as dysfluencies. it would also be possible to condense and further transform content to achieve compliance with subtitling or captioning standards, which is not the aim of our task, but could be added as a subsequent step in the cascade approach. since our input is not an actual language, but rather an artificial intermediate state between swiss german and standard german, the task is comparable to translation from low-resourced languages. we therefore propose to use multilingual pre-trained models which are often the best alternative in low-resource settings @xcite . in the absence of models for normalised swiss german, we propose to use models trained on high-resource languages including german, that could generalise to normalised swiss german @xcite . some researchers have used multilingual pre-trained language models to generalise unseen languages -i.e. languages that are not covered in the pre-trained model @xcite @xcite . for example, @xcite fine-tuned multilingual pre-trained models on 15 unseen languages to perform downstream tasks -pos (part-of-speech tagging), ner (named entity recognition) and dep (dependency parsing). their study shows that using multilingual pre-trained models increases the performance on these tasks for languages that have the same writing systems as the pre-trained languages. however, most of the researchers have applied these models to natural language understanding tasks from an unseen source language @xcite , but not many to machine translation. multilingual models have already been applied to swiss german, for example @xcite fine-tuned a multilingual pre-trained model to transcribe swiss german and generate standard german using an end-to-end approach. the model outperformed the transformer baseline by at least 8 bleu points. ## methodology in the following sections, we describe in more detail the data (section 2.1), systems (section 2.2) and evaluation methodology (section 2.3). ## results in this section, we present the results by hypothesis. ## conclusion in this study we have applied different machine translation approaches to the task of improving asr output for automatic standard german captioning of swiss german tv content in a cascade architecture. overall, mt is able to improve the output, both according to automatic metrics and user perspective. our first hypothesis was confirmed by the better performance of all pre-trained models for this task. among the tested approaches, bart-based approach achieved the highest scores, possibly due to its ability to handle noisy text. we also observed that a higher number of pre-trained languages improves performance on the task, meaning that more languages are useful to enable generalisation to an unseen language. however, we did not elucidate the influence of individual languages included in pre-training. we also assessed the impact of fine-tuning data. in-domain data improved performance on the task, but there is no difference in performance when using as source side of the aligned data automatic or human transcriptions. fine-tuning data for this task does not have to be clean human normalised transcriptions, which are more expensive to produce. result shows that asr output, despite being noisier, does not lead to significantly worse results. we also saw that target-side has a significant impact on performance. for the task as studied here, the subtitles, which contain many changes related to subtitling standards (shortening, simplification) are not ideal as training data. the different systems described in this paper can be tested online at https:// passage-imi.unige.ch/demo/ and the test data is available at https://doi.org/ 10/gr72xj.