Modular Adaptation of Multilingual Encoders to Written S wiss G erman Dialect (2024)

Abstract: Creating neural text encoders for written Swiss German is challenging due to a dearth of training data combined with dialectal variation. In this paper, we build on several existing multilingual encoders and adapt them to Swiss German using continued pre-training. Evaluation on three diverse downstream tasks shows that simply adding a Swiss German adapter to a modular encoder achieves 97.5% of fully monolithic adaptation performance. We further find that for the task of retrieving Swiss German sentences given Standard German queries, adapting a character-level model is more effective than the other adaptation strategies. We release our code and the models trained for our experiments.

## introduction when applying natural language processing (nlp) techniques to languages with dialectal variation, two typical challenges are a lack of public training data as well as varying spelling conventions. in the case of swiss german, which is spoken by around 5 million people and is often used for informal written communication in switzerland, these factors make it more challenging to train a bertlike text encoder for written text. in this paper, we adapt pre-trained multilingual encoders to swiss german using continued pretraining on a modest amount of swiss german training data. we evaluate the approaches on part-of-speech (pos) tagging with zero-shot crosslingual transfer from standard german @xcite , as well as dialect identification @xcite and cross-lingual sentence retrieval based on a parallel standard german-swiss german test set @xcite . we find that depending on the multilingual encoder, continued pre-training leads to an average improvement of 10%-45% in average accuracy across the three downstream tasks. we then focus on comparing monolithic adaptation, where all the parameters of the encoder are updated during continued pre-training, to modular adaptation with language-specific modular components (language adapters; @xcite . even though modular adaptation only updates a fraction of the parameters, it is competitive to monolithic adaptation. given these findings, we propose to extend the swissbert model @xcite , which was trained on standard german and other languages, with a swiss german adapter ( we further hypothesize that the architecture of canine @xcite , a tokenization-free model that operates on characters, might be better suited to the highly variable spelling of swiss german. indeed, a canine model adapted to swiss german excels on the retrieval tasks, while pos tagging works better with subwords. finally, we aim to combine the best of both worlds by integrating character-level down-and upsampling modules into a subword-based model and training a character-level adapter for swiss german. however, this jointly modular and tokenization-free strategy underperforms the individual approaches. we hope that our findings can inform the development of modular approaches for other languages with dialectal variation. ## adaptation scenario our goal is to train an encoder model for swiss german (language code gsw) with limited training data. since standard german (language code de) is a closely related language, we focus on transfer learning from standard german to swiss german. we rely on pre-trained multilingual models that have already been trained on standard german, and adapt them to swiss german using continued pre-training. swiss german adaptation data for training on swiss german, we use the swisscrawl corpus (linder et al., 2020), which contains 11m tokens of swiss german text extracted from the web. the text in swisscrawl exhibits some normalizations that eventual input text will not have, e.g., isolation of individual sentences, normalization of punctuation and emoji removal. to diversify the training data, we extend the pre-training dataset with a custom collection of 382k swiss german tweets. in total, we use 18m tokens for pre-training on swiss german. both datasets were automatically mined and may contain some text in other languages. standard german data to promote transfer from standard german to swiss german later on, we include an equal part of standard german data in the continued pre-training data. we use a sample of news articles retrieved from the swissdox@liri database, comparable to the data the swissbert model has been trained on @xcite . ## monolithic approaches we evaluate a subword-based model and a character-based model, with and without continued pre-training on swiss german. we call these models monolithic (non-modular), because the entire model is updated during continued pre-training. ## experimental setup continued pre-training we combine swiss german and standard german training data with a 1:1 ratio. the resulting bilingual dataset contains 37m tokens in total, and we set aside 5% for validation (table a6 ). we set the learning rate to 1e-4 and select the best checkpoint based on the validation loss out of 10 epochs; otherwise we use the default settings of hugging face transformer's mlm example script. we train the models on a nvidia v100 gpu with 32gb of memory and adjust the batch size dynamically to fit the available memory. with the subword-based models, we set the sequence length to 512. with the canine-style models, we use the default downsampling rate of @xmath0. fine-tuning for the downstream tasks that involve fine-tuning (pos and gdi), we fine-tune the model with a learning rate of 2e-5 and a batch size of 16. we train for 10 epochs and select the best checkpoint based on the validation accuracy. we report average and standard deviation across 5 finetuning runs with different random seeds. ## results the adapted canine shows state-of-the-art performance on the retrieval tasks. a simple chrf baseline @xcite achieves only 90.9% and 93.0% accuracy on the two retrieval tasks, and both the original and the adapted canine clearly surpass this baseline. however, the canine model has low accuracy on pos tagging, reflecting previous findings for named entity recognition @xcite . future work could explore alternative strategies for token-level classification tasks. while the monolithic xlm-r model performs best overall, we consider adding a subword-based swiss german adapter to swissbert a competitive alternative, with the number of trainable parameters reduced by 95% (see table a1 for a comparison of the model sizes). finally, a character-level adapter, where character up-and downsampling modules are added to the model specifically for swiss german, performs better than random but clearly worse than the standard approaches. this indicates that while the transformer layers of a subword-based model bear some similarity to the downsampled positions in the canine architecture, continued pre-training cannot completely bridge the gap between the two architectures. future work could pre-train a modular character-level model from scratch to further improve adaptability to new languages and dialects, while taking into account more recent findings regarding the optimal design of character-level modules for text encoding @xcite . ## conclusion we compared strategies for adapting multilingual encoders to swiss german. we found that the monolithic approach of continued pre-training xlm-r is a strong baseline. adding a swiss german adapter to swissbert, a model with a modular architecture, is a viable alternative. finally, adapting canine on swiss german works well for cross-lingual retrieval. the four swiss german encoder models we trained for our experiments will be made available to the research community.