DIALECTBENCH : An NLP Benchmark for Dialects, Varieties, and Closely-Related Languages (2024)

Abstract: Language technologies should be judged on their usefulness in real-world use cases. An often overlooked aspect in natural language processing (NLP) research and evaluation is language variation in the form of non-standard dialects or language varieties (hereafter, varieties). Most NLP benchmarks are limited to standard language varieties. To fill this gap, we propose DIALECTBENCH, the first-ever large-scale benchmark for NLP on varieties, which aggregates an extensive set of task-varied varieties datasets (10 text-level tasks covering 281 varieties). This allows for a comprehensive evaluation of NLP system performance on different varieties. We provide substantial proof of performance disparities between standard and non-standard language varieties, and we also identify language clusters with larger performance divergence across tasks.We believe DIALECTBENCH provides a comprehensive view of the current state of NLP for varieties and one step towards advancing it further.

## introduction benchmarking is important for tracking the progress the field of natural language processing (nlp) has made in various tasks. in the past few years, large-scale multilingual benchmarks like xtreme @xcite , xtreme-r @xcite , and xglue @xcite have played a pivotal role in evaluating the multilingual capabilities of nlp models. these efforts have sought to make model evaluation more accessible to researchers and representative of a variety of languages @xcite . however, most benchmarks have focused on the standard varieties of languages, largely neglecting non-standard dialects and language varieties @xcite . we refer to non-standard dialects and language varieties simply as varieties, and sometimes include low-resource related languages, writing system variants, and other kinds of variation. varieties contain subtle but notable variations in vocabulary, pronunciation, orthography and grammar, reflecting regional, social, and cultural differences @xcite . the non-standard nature of these language varieties oftentimes contributes to the scarcity of substantial datasets that accurately capture these variations @xcite . as a result, they have often been absent from widely adopted benchmarks, even from admirable efforts like xtreme-up @xcite , @xcite and @xcite , which focus on underresourced, code-switched, and creole languages, respectively. it is currently challenging to accurately test the robustness of multilingual models on a large suite of varieties without establishing an nlp evaluation framework covering multiple language clusters (each of which contains standard languages alongside its closely related varieties). to this end, we create dialectbench, a largescale benchmark covering 40 language clusters with 281 varieties, spanning 10 nlp tasks. we observe that the performance disparity between different varieties of the same language cluster becomes ## dialectbench dialectbench is a benchmark created to unify dialectal datasets across different tasks to foster research on language varieties and non-standard dialects. below we describe the design choices we undertook to achieve this goal. this includes our language variety and task selection procedures, data collation methods, and evaluation principles. ## experiments here, we report the entire process involved in creating and evaluating baselines for all of the tasks and varieties in dialectbench. additionally, we define a dialectal gap metric to analyze performance disparities within and across clusters. ## results we, first of all, discuss results by highlighting the highest possible score per variety, aka the maximum obtainable evaluation scores regardless of evaluation method or training data. next, we extend our discussion further by reporting the existing dialectal disparity across clusters and varieties. ## discussion high resource vs. low resource varieties the highest-performing varieties are mostly standard high-resource languages and a few high-resource dialects (norwegian dialects) whereas, the majority of the lowest-performing language variants are lowresourced varieties. this clear distinction of language varieties points towards the large existence of in-cluster dialectal gaps. furthermore, this finding correlates with language script differences. we observe that 77.2% of top-10 varieties in terms of maximum obtainable score are written with latin script. another finding is the performance instability of low-resource varieties across tasks. for instance, old guarani performs better in dep parsing whereas, mbyá guarani (paraguay) surpasses it in pos tagging even though the dataset remains the same (i.e. ud). for more detailed comparisons, in appendix ## conclusion we propose dialectbench, the first-ever inclusive dialectal nlp benchmark reporting performance evaluation and disparity across standard and non-standard varieties. this is one step towards the effort of bringing more and more language under the paradigm of nlp technology. we would like to further improve the benchmark, constructing highquality comparable data and expanding the task coverage to speech-based nlp technologies.