A Survey of Corpora for G ermanic Low-Resource Languages and Dialects (2023)

Abstract: Despite much progress in recent years, the vast majority of work in natural language processing (NLP) is on standard languages with many speakers. In this work, we instead focus on low-resource languages and in particular non-standardized low-resource languages. Even within branches of major language families, often considered well-researched, little is known about the extent and type of available resources and what the major NLP challenges are for these language varieties. The first step to address this situation is a systematic survey of available corpora (most importantly, annotated corpora, which are particularly valuable for NLP research). Focusing on Germanic low-resource language varieties, we provide such a survey in this paper. Except for geolocation (origin of speaker or document), we find that manually annotated linguistic resources are sparse and, if they exist, mostly cover morphosyntax. Despite this lack of resources, we observe that interest in this area is increasing: there is active development and a growing research community. To facilitate research, we make our overview of over 80 corpora publicly available.

## introduction the majority of current nlp today focuses on standard languages. much work has been put forward in broadening the scope of nlp @xcite , with long-term efforts pushing boundaries for language inclusion, for example in resource creation (e.g., universal dependencies @xcite ) and cross-lingual transfer research (de vries et al., 1 we share a companion website of this overview at github.com/mainlp/germanic-lrl-corpora. 2022). however, even within major branches of language families or even single countries, plenty of language varieties are under-researched. current technology lacks methods to handle scarce data and the rich variability that comes with low-resource and non-standard languages. nevertheless, interest in these under-resourced language varieties is growing. it is a topic of interest not only for (quantitative) dialectologists @xcite , but also nlp researchers, as evidenced by specialized workshops like vardial foot_0 , special interest groups for endangered foot_1 and under-resourced languages, 4 and recent research on local languages spoken in italy @xcite , indonesia @xcite and australia @xcite , to name but a few. in this paper, we provide an overview of the current state of nlp corpora for germanic lowresource languages (lrls) and dialects, with a particular focus on non-standard variants and four dimensions: annotation type, curation profile, resource size, and (written) data representation. we find that the amount and type of data varies by language, with manual annotations other than for morphosyntactic properties or the speaker's dialect or origin being especially rare. with this survey, we aim to support development of language technologies and quantitative dialectological analyses of germanic low-resource languages and dialects, by making our results publicly available. finally, based on the experiences we made while compiling this survey, we share recommendations for researchers releasing or using such datasets. ## language varieties our survey contains corpora for more than two dozen germanic low-resource varieties, selected based on dataset availability (appendix a contains the full list). we focus on specialized corpora showcasing regional variation, but not necessarily global variation. this overview does not include any corpora for germanic-based creoles like naija, as those are included in the recent survey by @xcite . figure 1 shows where most of the doculects included in this survey are spoken. ## methodology similarly to @xcite , we search for corpora on several online repositories for language resources: the clarin virtual language observatory @xcite , the lre map @xcite , the european language grid @xcite olac @xcite , ortolang @xcite , and the hamburg centre for language corpora. 6 we also search for corpora on zenodo and on google dataset search, and look for mentions of corpora in articles hosted by the acl anthology and on arxiv. 7 we search for mentions of the word "dialect" and the names of various germanic lowresource languages. we use the following inclusion criteria: we base this survey only on the versions of corpora that are easily accessible to the research community; e.g., if a corpus contains audio material, but only the written material is available for download (and thus as a data source for quantitative research), the corpus is treated as a text corpus. corpus langs annotation size rep. tapaco (subset) (scherrer, 2020) nds, gos paraphrases 1107 sents (nds), zenodo.org/record/3707949 122 sents (gos) wenkersätze (wenker, 1889-1923; schmidt et al., 2020-) deu* translations 2210 samples×40 sents / github.com/engsterhold/wenker-storage (across dialects, deu) sb-ch (subset) (grubenmann et al., 2018) gsw sentiment 2.8k sents github.com/spinningbytes/sb-ch swissdial (dogan-schönberger et al., 2021) gsw , wae topic, translations 2.5-4.6 hrs×8 lects | projects.mtc.ethz.ch/swiss-voice-data-collection (across dialects and into deu) xsid/sid4lr (subset) gsw, bar slot and intent detection, 800 sents (van der goot et al., 2021; aepli et al., 2023) (south tyrol) translations (14 langs) bitbucket.org/robvanderg/sid4lr ## corpora most of the language varieties we survey have no or only a very recent written tradition. this unique challenge is reflected in the different written formats used to represent the data (if the corpora contain any written material at all) and concerns both the transcription of audio data @xcite as well as the elicitation of written data @xcite . we opted to discern between audio data and the following written variants: standard orthographies (of the doculects themselves where existing (e.g., west frisian orthography), or of a closely related higherresource language otherwise |), ad-hoc pronunciation spelling (by speakers of the doculect) , and phonetic or phonemic transcriptions (by linguists) . appendix b provides examples. the following corpora are sorted by annotation and curation type. for an overview sorted by language, see appendix a. some of the corpora share the same data sources. appendix c lists the cases where we are aware of such overlaps. ## conclusion we have presented an analysis of over 80 corpora containing data in germanic low-resource languages, with a focus on non-standardized or only recently standardized varieties. we additionally share the corpus overview on a public companion website (github.com/mainlp/ germanic-lrl-corpora) that can easily be updated as more language resources are released.