Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora (2025)

Abstract: Dialects exhibit a substantial degree of variation due to the lack of a standard orthography. At the same time, the ability of Large Language Models (LLMs) to process dialects remains largely understudied. To address this gap, we use Bavarian as a case study and investigate the lexical dialect understanding capability of LLMs by examining how well they recognize and translate dialectal terms across different parts-of-speech. To this end, we introduce DiaLemma, a novel annotation framework for creating dialect variation dictionaries from monolingual data only, and use it to compile a ground truth dataset consisting of 100K human-annotated German-Bavarian word pairs. We evaluate how well nine state-of-the-art LLMs can judge Bavarian terms as dialect translations, inflected variants, or unrelated forms of a given German lemma. Our results show that LLMs perform best on nouns and lexically similar word pairs, and struggle most in distinguishing between direct translations and inflected variants. Interestingly, providing additional context in the form of example usages improves the translation performance, but reduces their ability to recognize dialect variants. This study highlights the limitations of LLMs in dealing with orthographic dialect variation and emphasizes the need for future work on adapting LLMs to dialects.

## introduction although most languages have standardized orthographies, this assumption does not hold for many languages and language varieties (millour and fort, 2020). one prominent example are nonstandard dialects, which have become increasingly popular in natural language processing (nlp) research @xcite . dialects exhibit a large degree of spelling variation in written data, owing both to individual 1. judging translation pairs ## method in this section, we first introduce dialemma, our annotation framework for obtaining dialect variation dictionaries (section 3.1). we then introduce our dataset (section 3.2), which we obtained by applying dialemma to the bavarian wikipedia. based on this, we introduce two word-level dialect tasks (section 3.3). ## results & discussion we report the task performance for each llm and part of speech (pos), and across all parts of speeches. we additionally report the percentage of outputs where llms fail to follow prompt instructions (instruction following error rate; if error). in the appendix, we show a list of all pos categories and number of instances ( ## conclusion we introduce dialemma, a novel dialect annotation framework and build a dataset consisting of german lemmas and bavarian variants. we conduct a systematic analysis of the ability of llms on two tasks: judging whether superficially similar dialect-standard word pairs are translations of each other, and translating dialect variants into their standard-language counterparts. our results show that their performance varies by model size and the part-of-speech of the input word. in future work, we plan to 1) fine-tune llms for identifying translation pairs and as translators and 2) extend our evaluation protocol to extrinsic evaluation on downstream tasks. we will release our code and dataset for future uptake (cc by-sa 4.0).