CUFE @ V ar D ial 2025 N or SID : Multilingual BERT for N orwegian Dialect Identification and Intent Detection (2025)

Abstract: Dialect identification is crucial in enhancing various tasks, including sentiment analysis, as a speaker’s geographical origin can significantly affect their perspective on a topic, also, intent detection has gained significant traction in natural language processing due to its applications in various domains, including virtual assistants, customer service automation, and information retrieval systems. This work describes a system developed for VarDial 2025: Norwegian slot and intent detection and dialect identification shared task (Scherrer et al., 2025), a challenge designed to address the dialect recognition and intent detection problems for a low-resource language like Norwegian. More specifically, this work investigates the performance of different BERT models in solving this problem. Finally, the output of the multilingual version of the BERT model was submitted to this shared task, the developed system achieved a weighted F1 score of 79.64 for dialect identification and an accuracy of 94.38 for intent detection.

## introduction norwegian dialects represent a rich tapestry of linguistic diversity that reflects the historical, geographical, and social nuances of norway. the country is home to a multitude of dialects, often categorized into four primary groups: northern norwegian (nordnorsk), central norwegian (trøndersk), western norwegian (vestlandsk), and eastern norwegian (østnorsk). these dialects are not merely regional variations; they embody unique grammatical structures, vocabulary choices, and phonetic characteristics that can vary significantly even within short distances. for instance, a speaker from bergen may find it challenging to understand someone from oslo due to the distinct phonetic and syntactic features of their respective dialects. this variation poses considerable challenges for lin-guistic studies and applications in natural language processing (nlp). the ability to identify and differentiate between these dialects is crucial for various applications, including speech recognition systems, language learning tools, and sociolinguistic research. a study found that norwegians are generally less adept at identifying their own dialects compared to speakers of other languages, such as dutch @xcite . this suggests that while norwegians are exposed to multiple dialects throughout their lives, the cognitive mechanisms underlying dialect identification may not be as finely tuned as previously thought. furthermore, the role of intonation in identifying these dialects has been highlighted as particularly significant. in recent years, advancements in machine learning and nlp have opened new avenues for addressing these challenges. among these advancements, bert (bidirectional encoder representations from transformers) has emerged as a powerful tool for various language understanding tasks. bert utilizes a transformer architecture that allows it to capture contextual relationships between words in a sentence effectively @xcite . this capability is particularly valuable for dialect identification, where subtle differences in word usage or syntax can indicate distinct regional affiliations. in this paper, the fine-tuning of multiple bertbased models for identifying norwegian dialects and detecting the intent of norwegain text was explored. different pre-trained models, including a norwegian-specific bert variant and multilingual bert models were investigated to measure their efficacy in dialect identification and intent detection tasks. by leveraging transfer learning and fine-tuning strategies, the model's understanding of norwegian dialects and text intent was improved, even in the context of a relatively under-resourced language, like norwegian. the rest of the paper is organized as follows: section 2 provides an overview of related research. section 3 describes the dataset used for training and validation. section 4 outlines the system, and section 5 concludes the paper. norwegian bert: with the growing interest in nlp applications for low-resource languages, the development of norwegian-specific transformer models has been pivotal. the notram project has created a norwegian transformer model that outperforms multilingual bert (mbert) on various classification tasks, including intent detection @xcite . this model demonstrates that fine-tuning language models specifically on norwegian text can yield significant improvements in performance compared to generic models. norwegian bert application: bert has revolutionized nlp by enabling models to understand context in a bidirectional manner, making it particularly effective for tasks involving nuanced language understanding. @xcite demonstrated the potential of a norwegian bert model for morphosyntactic analysis, highlighting its capacity to handle the complexities of dialectal variations. the model's architecture allows it to capture contextual relationships between words, which can be used for distinguishing between different dialects that may share similar vocabulary but differ significantly in usage. the nomusic corpus: the introduction of the nomusic corpus @xcite represents a significant advancement in resources available for norwegian dialect identification and intent detection. this corpus consists of translations from the xsid dataset (van der goot et al., 2021) into standard norwegian bokmål and eight dialects from three major norwegian dialect areas. this corpus represents the first evaluation dataset focusing on non-standard norwegian varieties, allowing researchers to analyze linguistic variations across different dialects systematically. bert for dialect identification: mawdoo3 ai has developed a multi-dialect arabic bert model specifically for country-level dialect identification. this model was trained on a dataset comprising 21,000 labeled tweets from all 21 arab countries and achieved a micro-averaged f1-score of 26.78% in the nadi shared task @xcite . the success of this model highlights the efficacy of fine-tuning pre-trained transformer models for specific dialectal tasks. bert for intent detection: bert has been effectively utilized for citation intent classification within academic texts. a study analyzed various bert models fine-tuned on labeled datasets to classify citation intents and sentiments, revealing that bert's contextual capabilities enhance its performance in understanding nuanced academic language @xcite . while this application is not directly focused on conversational intent detection, it underscores bert's versatility across different domains and its potential for enhancing understanding in specialized contexts like academic discourse. ## dataset a subset of the norwegian multi-dialectal slot and intent detection corpus (nomusic) @xcite was used for training and testing this system. the nomusic corpus was created by translating the xsid dataset, an evaluation dataset for spoken language understanding (slot and intent detection) @xcite to eight norwegian dialects and norwegian bokmål. for dialect identification, the development dataset consists of 3300 sentences, and the testing dataset consists of 5500 sentences, representing the four main dialects in norway: north norwegian, trøndersk, west norwegian and bokmål. the data distribution among the four dialects is summarized in ## methodology for the dialect identification, three versions of bert were considered: "nbailab/nb-bert-base" @xcite , "ltgoslo/norbert" @xcite , and "bert-base-multilingualcased" @xcite . each bert classifier was fine-tuned on 80% of the development data for 10 epochs with a learning rate of 2e -5 and a model accuracy nbailab/nb-bert-base 74.61 ltgoslo/norbert 73.33 bert-base-multilingual-cased 79.91 due to the difference in the performance between the "bert-base-multilingual-cased" model and the remaining bert models, the multilingual bert model used for final submission. for the dialect identification task, the shared task organizers provided the baseline model from the itdi shared task @xcite , which employs a support vector machine (svm) classifier with tf-idf-weighted features of character 1-to-4-grams. the baseline achieved a weighted f1score of 77.42, whereas the developed multilingual bert model surpassed it with a weighted f1-score of 79.64. similarly, for the intent dectection, three versions of bert were considered: "nbailab/nbbert-base" @xcite , "ltgoslo/norbert" @xcite , and "bertbase-multilingual-cased" @xcite . each bert classifier was fine-tuned on 80% of the development data for 10 epochs with a learning rate of 2e -5 and a batch size of 32, then, the accuracy of each of those models was calculated based on the remaining 20% of the development set, the performance of those models is summarized in due to the difference in the performance between the "bert-base-multilingual-cased" model and the remaining bert models, the multilingual bert model was used for the final submission. the developed bert model significantly outperforms the baseline bert model provided by the shared task organizers. the baseline utilizes an mbert encoder with two separate decoder heads: one for slot detection and another for intent classification. while the baseline achieved an accuracy of 84.15% on the intent detection task, the developed multilingual bert model achieved an accuracy of 94.38%. ## conclusion and future work in this paper, we have explored the efficacy of multiple bert models in the tasks of norwegian dialect identification and intent detection. the multilingual version of bert produced the best results on the development data. finally, the output of the multilingual version of the bert model was submitted to this shared task, and it achieved a weighted-f1 score of 79.64 for dialect identification and an accuracy of 94.38 for intent detection on the test datasets. future work will focus on improving the bert model by leveraging norwegian-translated dataset to address the challenges posed by missing intents in the development dataset. missing intents can impede the model's ability to learn comprehensive patterns for intent recognition, leading to suboptimal performance. by augmenting the development dataset with the norwegian-translated dataset, we can introduce linguistic diversity and contextual richness, compensating for the gaps in the development dataset.