LSDC - A comprehensive dataset for L ow S axon Dialect Classification (2020)

Abstract: We present a new comprehensive dataset for the unstandardised West-Germanic language Low Saxon covering the last two centuries, the majority of modern dialects and various genres, which will be made openly available in connection with the final version of this paper. Since so far no such comprehensive dataset of contemporary Low Saxon exists, this provides a great contribution to NLP research on this language. We also test the use of this dataset for dialect classification by training a few baseline models comparing statistical and neural approaches. The performance of these models shows that in spite of an imbalance in the amount of data per dialect, enough features can be learned for a relatively high classification accuracy.

## introduction compared with the dominant languages of larger countries, minority languages tend to be underrepresented in terms of access to nlp tools. availability of such tools however is of vital importance, since a lack of these indirectly forces groups already under pressure of language shift to resort to tools in the dominant language, with the consequence of a further decrease in the proportion of domains where the language can be used in daily life @xcite . this is especially true for unstandardised languages like low saxon, where the lack of a written norm poses challenges for the development of modern nlp applications, which typically rely on large amounts of, ideally, orthographically uniform data. while a reference corpus exists for middle low saxon @xcite , the few datasets of modern low saxon available so far tend to either be very restricted content-wise (e.g. the dsa data @xcite )) or only represent a fraction of the language area without indication of the dialect (e.g. the opus data @xcite ). in addition, both the dsa data and most of the opus data consist of content translated into low saxon instead of original texts, which will affect the naturalness of the language. the aim of this dataset for low saxon is thus to provide open and for the most part original data in low saxon covering nearly the whole language area in order to foster research and facilitate the development of nlp tools. the composition of this dataset and testing the suitability of language recognition tools is a first step in our larger research project on processing low saxon data and modelling the historical development of the language-internal variation. successful dialect recognition could thus be a useful step in a preprocessing pipeline where it would then be followed by normalisation to one of the writing systems in use, before applying tools developed for standardised languages to low saxon text. in this paper, we will first give an overview of the societal and historical background as well as characteristic features of low saxon dialects, followed by a description of the dataset. 1 we will conclude with the presentation of a few baseline models for dialect identification trained on this data and an analysis of their results. ## background low saxon is an unstandardised west-germanic language with most of its around 5 million speakers today living in the north of germany and the north-eastern parts of the netherlands @xcite . starting from the 20th century, the usage of low saxon and its intergenerational transmission have been in decline but in the late 20th and in the 21st century, minor revitalisation and popularisation efforts have emerged, e.g. by introducing low saxon as a school subject and encouraging young musicians to produce songs in the language. furthermore, especially since the advent of social media, more people have started to use low saxon as a written language for communication in daily life, as described e.g. by @xcite in his phd thesis on low saxon speaking communities on facebook. even though low saxon has some official status in several federal states of germany, e.g. in schleswig-holstein (landesregierung schleswig-holstein, 1992), and is protected under the european charter for regional and minority languages in both germany and the netherlands (council of europe, 2020), there is no official standard variety in use. this is why characteristics of local dialects tend to be rather well reflected in modern written low saxon, not only in terms of lexicon and syntax, but also in the writing system employed. these local writing traditions are based to different degrees and in different ways on the majority language orthography, i.e. german or dutch. some of their characteristics will be explained in more detail below. ## the lsdc dataset we have gathered a comprehensive data set of low saxon dialects covering nearly the whole language area. the collection includes historical texts from the 19th and 20th century as well as contemporary low saxon and therefore spans a period of around 200 years with most dialects being presented in at least two centuries. the total size is 105 876 sentences with an average sentence length of 20.16 words and a total word count of 2 134 753. this lsdc dataset presents a unique resource for modern low saxon which will be made openly available together with the final version of the paper. in the following subsections, we will provide background on the original text sources and the characteristics of the data. ## dialect identification so far, low saxon is still an underresearched language within the field of nlp, which probably at least partly is due to its lack of a standardised form. we are not aware of any comparable previous work on dialect identification for this language, but birkenes (2018) conducted n-gram-based measurements of the distance between dialects of german low saxon for dialect classification and identification of dialect areas using the wenker atlas data @xcite . for testing the usability of the dataset, we chose the fasttext and langid toolkits to train baseline models. both are supervised classifiers where the classes are predefined, but otherwise they differ in both the way the language data is represented and their general architecture. ## conclusions, discussions and future work the lsdc dataset presented in this paper is a comprehensive dataset for contemporary low saxon. with nearly all modern dialect groups being included and two centuries as well as various genres being covered, this dataset provides a unique new resource for nlp research on and application development for this minority language. we have tested the resource on a dialect identification task, demonstrating its use for further collections and classifications of dialectal data. even though the amount of data per dialect is not balanced in lsdc, we have seen that sampling makes it possible to achieve a comparable precision for both high-and low-resource dialects with a neural classification model and that the performance of that model even approximately resembles native speaker intuition. one central aspect affecting dialect identification are the various ways of writing low saxon. as no normalisation of spelling was included, we need to acknowledge that we cannot fully discern in how far the models classify the dialects based on the local writing system or based on actual dialect features such as lexicon, inflectional suffixes and syntax. as a consequence, it would be meaningful to conduct additional dialect identification experiments with orthographically normalised low saxon text data. in addition, one could test the effect of grouping the dialects according to similarity in writing systems and training separate models for instance for dutch low saxon, northern german low saxon and southern german low saxon. the next steps in our research project on the historical development of the language internal variation in low saxon will require us to expand the corpus. first of all, more data in the low-resource dialects will be needed and the time coverage for all dialects needs to be improved, so that roughly the same amount of data is available for each dialect and century. ideally, each dialect should additionally be represented by a variety of genres for each century and have the genre included as part of the annotation. this will allow us to investigate to what extent characteristics of a specific domain are misinterpreted as dialect features. lastly, we will need to increase the time depth in order to close the gap between the middle low saxon reference corpus and our modern low saxon dataset, so that the development of middle low saxon into contemporary low saxon can be traced.