R o D ia: A New Dataset for R omanian Dialect Identification from Speech (2024)

Abstract: We introduce RoDia, the first dataset for Romanian dialect identification from speech. The RoDia dataset includes a varied compilation of speech samples from five distinct regions of Romania, covering both urban and rural environments, totaling 2 hours of manually annotated speech data. Along with our dataset, we introduce a set of competitive models to be used as baselines for future research. The top scoring model achieves a macro F1 score of 59.83% and a micro F1 score of 62.08%, indicating that the task is challenging. We thus believe that RoDia is a valuable resource that will stimulate research aiming to address the challenges of Romanian dialect identification. We release our dataset at https://github.com/codrut2/RoDia.

## introduction spoken dialect identification emerged as a challenging task aiming to achieve a fine-grained distinction between varieties of a certain language, having similar implications to spoken language identification @xcite @xcite . despite being a more delicate task, spoken dialect identification received comparatively lower attention, most of it being devoted to dialect identification for widely spoken languages, such as english @xcite , chinese @xcite , and arabic @xcite @xcite . spoken dialect identification for low-resource languages, such as swiss german @xcite and finnish @xcite , has remained relatively underexplored @xcite @xcite . different from prior studies, we focus on spoken language identification in romanian, a low-resource language characterized by its intricate dialectal variations within the country of romania. romanian, a ro-mance language with latin roots, boasts a rich linguistic landscape shaped by historical, geographical, and sociocultural factors @xcite . however, despite its linguistic complexity, romanian remains a low-resource language, with limited studies dedicated to understanding its regional linguistic diversity. this scarcity of resources is not unique to romanian, numerous other languages around the world having similar challenges due to their lower visibility on the global linguistic stage @xcite @xcite . notably, the vardial workshop is one of the main drivers for growing the interest around language variety and dialect identification, through the organization of multiple shared tasks each year @xcite @xcite . due to the success of deep learning frameworks in speech processing @xcite , researchers started to employ such methods in the area of low-resource languages @xcite . this has led to a growing need for resources on lowresource languages. considering dialect identification datasets across different languages, we can distinguish between two types of resources: textbased datasets @xcite @xcite @xcite and speech-based datasets @xcite @xcite @xcite . while various languages have benefited from text-based resources that leverage written materials capturing linguistic variations, the auditory dimension of dialects adds an intricate layer of complexity. text data, although valuable, might not fully encapsulate the nuanced phonetic and prosodic characteristics that are pivotal in dialect differentiation. in contrast to text datasets, audio datasets @xcite @xcite ., muntenesc muntenesc moldovenesc moldovenesc ardelenesc ardelenesc b bă ăn năț ățean ean oltenesc oltenesc 2020; dogan-schönberger et al., 2021; plüss et al., 2023) offer a more holistic representation, capturing not only the lexical disparities, but also the subtle intonations and accents inherent in speech. to the best of our knowledge, rodia is the first dataset to tackle spoken dialect identification in the romanian landscape in accordance with historical, geographical, and sociocultural factors, encouraging the research in this low-resource language. although there are two text datasets addressing romanian dialect identification, moroco (butnaru and ionescu, 2019) and moroco-tweets @xcite , these cover only two dialects: romanian (equivalent to the muntenesc dialect) and moldavian (moldovenesc). in contrast, our dataset is focused on speech and covers five romanian dialects, as shown in figure 1 . we underline that the extra dialects, namely ardelenesc, bȃnȃţean, and oltenesc, are very well characterized by phonetic differences captured only in speech. this explains why moroco @xcite and moroco-tweets @xcite only contain text samples from the other two dialects. the number of audio datasets available in romanian is rather low @xcite , confirming that romanian is indeed a low-resource language. we underline that existing datasets comprising romanian speech samples are mainly focused on automatic speech recognition, ignoring the diversity of dialects within the region. in contrast, our dataset is specifically designed to represent five distinct dialects spoken in romania: muntenesc (accepted as the official language of romania), ardelenesc, moldovenesc, oltenesc, and bȃnȃţean foot_0 . we divide the dataset into 2,164 samples for training and 604 samples for testing, such that there is no overlap between speakers in training and test. without separating the speakers between training and test, a model that overfits to certain speaker-specific features that are not related to dialect (e.g. pitch, loudness, rate, etc.) will reach high scores on the test set. however, these scores are unlikely to represent the actual performance of the model in a real-world scenario, where the audio samples come from unknown speakers. we thus consider that a more realistic evaluation is to separate the speakers. in the proposed setting, models that learn patterns related to speakers will not be able to capitalize on features unrelated to dialect identification. there are five local annotators (one per region), who annotated all samples. the inter-rater quadratic weighted kappa score is 0.83, indicating that the collected labels exhibit a substantial agreement among human evaluators. the average accuracy of our raters is 86%, indicating that the task is fairly easy for humans. note that all raters are native romanian speakers who speak the literary language, as well as at least one of the five dialects. the collected samples comprise interviews and read speech found on youtube. the read speech is actually gathered from videos where various speakers read from different books (without any influence or preparation from our side). the percentage of read speech is 21%. aside from dialect labels, our annotators also labeled each audio sample with the gender and age of each speaker. more precisely, the age and gender of each speaker is estimated by two annotators who had to analyze both video and audio modalities. the age annotation consists in classifying each speaker into a 10-year age group, after watching the video available for the respective speaker. in summary, our audio samples come with dialect, age and gender labels, enabling the study of additional tasks such as gender prediction or age estimation from speech. dataset statistics. for a more comprehensive view, we present both demographic information and audio quality statistics for our new dataset. in to further assess the behavior of the best baseline model, namely wav2vec 2.0, we consider its confusion matrix illustrated in figure 3 . the confusion matrix reveals some interesting patterns. while the model tends to mislabel samples from the oltenesc dialect, we observe that most of these mistakes are caused by a high confusion with the bȃnȃţean dialect. since banat and oltenia are neighboring regions, there are several similarities between these two dialects. for instance, both dialects are characterized by the frequent use of the perfect simple tense, which is hardly encountered in the other romanian dialects. another noticeable problem is with the moldovenesc dialect, which is often wrongly identified as bȃnȃţean and muntenesc. the confusion between the moldovenesc and bȃnȃţean dialects is caused by the fact these two dialects kept the form of words such as câne (dog), pâne (bread) and mâne (tomorrow), from the old romanian. in the literary language, as well as the other dialects, these words are pronounced with an 'i' before the consonant 'n', as follows: câine, pâine and mâine. the confusion with the muntenesc dialect can be attributed to the fact that some residents of the southern part of moldavia lost some of the dialectal features, e.g. they prefer to use the word pantofi to refer to shoes, and the word papuci to refer to slippers, just as the residents of muntenia. in contrast, the residents of the northern side of moldavia regularly use papuci to refer to shoes, and şlapi to refer to slippers. in summary, the confusion matrix shows that romanian dialect identification is not an easy task, requiring researchers to address specific issues in order to come up with more accurate models in the future. the confusion matrix illustrated in figure 3 also shows that the training data distribution does not affect wav2vec 2.0. for example, the moldovenesc dialect is the second-least popular dialect in our dataset, but wav2vec places many of the test samples into the moldovenesc class. overall, the confusion matrix of wav2vec reflects the test data distribution, although the model was trained on a slightly different class distribution. this confirms that the imbalance is not high enough to bias models. speech recognition results. we manually transcribed our data samples to test the performance dialect wer ardelenesc 31.5% bȃnȃţean 30.2% moldovenesc 32.8% muntenesc 24.1% oltenesc 29. 7% ## conclusion in this paper, we introduced rodia, the first dataset for romanian dialect identification from speech. our dataset contains 2,768 speech samples representing five romanian dialects. the audio samples were manually annotated with dialect, age and gender labels, enabling the study of spoken dialect identification in a realistic scenario, where the speakers in the training and test splits are disjoint. we conducted experiments with four state-of-theart speech processing models, establishing a range of baseline performance levels for future research.