Findings of the V ar D ial Evaluation Campaign 2025: The N or SID Shared Task on N orwegian Slot, Intent and Dialect Identification (2025)

Abstract: The VarDial Evaluation Campaign 2025 was organized as part of the twelfth workshop on Natural Language Processing for Similar Languages, Varieties and Dialects (VarDial), colocated with COLING 2025. It consisted of one shared task with three subtasks: intent detection, slot filling and dialect identification for Norwegian dialects. This report presents the results of this shared task. Four participating teams have submitted systems with very high performance (> 97% accuracy) for intent detection, whereas slot detection and dialect identification showed to be much more challenging, with respectively span-F1 scores up to 89%, and weighted dialect F1 scores of 84%.

## introduction the workshop series on nlp for similar languages, varieties and dialects (vardial), now at its twelfth edition, has traditionally hosted an evaluation campaign with shared tasks on various topics such as language and dialect identification, commonsense reasoning, question answering, and cross-lingual tagging and parsing. the shared tasks have featured many languages and dialects from different families and data from various sources, genres, and domains @xcite @xcite @xcite @xcite @xcite @xcite . the vardial evaluation campaign 2025 consisted of the norsid shared task, which focused on slot filling, intent detection and dialect identification for norwegian dialectal data. as digital assistants are becoming more widespread, it is important that they can support a wide variety of language varieties. where other work has focused on supporting a wider range of languages (e.g. @xcite , we instead focus on dialects, which has shown to be challenging for slot and intent detection systems @xcite @xcite . the norsid shared task included three subtasks: slot filling, intent detection, and dialect classification. each participating team was allowed to send in three submissions per subtask. it was not mandatory for the participants to provide systems for all tasks; they had the option to only take part in a specific subtask. a large number of previous vardial shared tasks focused on language identification, either for national varieties of pluricentric languages, or for dialects and closely related languages. the former includes tasks of discriminating between british and american english (dsl, @xcite @xcite @xcite , or between french spoken in belgium, canada, france and switzerland (fdi, @xcite , to name but a few. the latter includes the identification of various swiss german dialects @xcite , or of the different regional languages spoken in italy (itdi, @xcite . the di-alect identification subtask of this year's norsid task falls in the latter category. an overview of the history of language identification and its challenges can be found in @xcite . the two other subtasks of norsid focus on intent classification and slot filling for task-oriented dialog systems, a task also sometimes referred to as spoken language understanding. three recent surveys provide excellent introductions to the topic: louvan and magnini (2020) and @xcite focus mainly on methods, whereas @xcite survey the available datasets. aspects of dialectal variation and cross-lingual transfer between closely related varieties have been discussed in the sid4lr shared task at vardial 2023 @xcite , which focused on south tyrolian and swiss german dialects as well as neapolitan, a language closely related to italian. ## data the data used in the norsid shared task is taken from the nomusic corpus, which is the norwegian extension of the xsid dataset. we present these resources below. the data in xsid is partitioned into 43,605 sentences for training, 300 for development and 500 for testing. the native english data is translated into the other languages, automatically in the case of the training set, and by humans in the case of the development and test sets. nomusic since xsid currently does not cover norwegian, the nomusic corpus project @xcite was started to fill this gap. it complements xsid with several norwegian versions, taking into account the prevalence of dialects (and dialect writing) in norway. nomusic contains translations of the english xsid development and test sets both into standard norwegian bokmål and into the dialects of ten native speakers of norwegian who regularly write in these dialects. figure 1 shows the origins of the dialect speakers. 2 translators write in northern dialects (n, blue on the map), 3 translators write in central norwegian dialects (t for trøndersk, green) and 5 translators write in western dialects (v for vestnorsk, orange). none of the translators write in an eastern dialect (red on the map), but it is common in this area to write in standard bokmål. therefore, the bokmål translation can be viewed to some extent as representative of the writing traditions in eastern norway. the norsid training data as there was no training data for any norwegian varieties, we followed the procedure from van der @xcite to generate training data from the original xsid english training data using machine translation and annotation transfer. the machine translation model was trained on the norwegian opensubtitles data foot_0 , as it was the largest open parallel data based on transcribed speech. we used the fairseq toolkit v0.9.0 with default hyperparameters, matching the original xsid setup, and relied on the attention weights for transferring the slot labels, which were afterwards automatically corrected to valid bio sequences (i.e. first i becomes a b, and if there is a label mismatch in the span, the b-label is used). it should be noted that the automatic mapping of the slot labels led to some incorrect labeling in the target language. we also noted that the machine translation quality was relatively poor overall with a bleu score of 18.46 (sacrebleu on wordsegmented texts). the machine-translated training set is only available in norwegian bokmål, not in any of the dialects covered by nomusic (nor in the other written norwegian norm, nynorsk). the norsid development and test sets for the purpose of the shared task, we concatenated and shuffled all eleven versions of the nomusic data, keeping intact the division into development and test sets. furthermore, we annotated each prompt with the dialect label (n, t, v, or b for bokmål). an example is shown in figure 2 . in the development set, we also provide a unique sentence identifier (33/8 in the example) that determines the content (all sentences with number 33 have the same meaning) and the translator (all sentences with /8 were produced by translator a8). the blind test set provided to the participants consisted of the # text line and the first two columns of the tokenized format. team slots intents dialect reference hitz ✓ ✓ ✓ bengoetxea et al. (2025) mainlp ✓ ✓ blaschke et al. (2025) ltg ✓ ✓ midtgaard et al. (2025) cufe ✓ ✓ ibrahim (2025) evaluation we used the standard evaluation metrics for the three tasks, namely the span f1 score for slots, accuracy for intents, and weighted f1 score for dialect classification. the english source data in xsid is characterized by a considerable number of duplicates, and the number of duplicates further increased whenever several dialect translators produced the same translation (see ## participants and approaches four teams participated in the shared task (see baseline: for the slot and intent detection subtasks, the baseline we provided is the same as in the original xsid paper, trained on the english data, with an updated version of machamp foot_1 (van der @xcite . the model uses an mbert encoder and a separate decoder head for each task, one for slot detection (with a crf layer) and one for intent classification. for dialect identification, we used the same baseline model as in the itdi shared task @xcite : a support vector machine (svm) classifier with tf-idf-weighted features of character 1-to-4-grams. the model was trained on the development set using the scikit-learn toolkit @xcite . hitz: team hitz @xcite was the only one to address all three subtasks. for slot and intent detection, they compared various combinations of the xsid training data and found that english data alone performed best overall, followed by all germanic languages except norwegian (i.e., english, german, dutch and danish). they also confirmed that multi-task modelling outperformed a single-task setup. for dialect identification, team hitz collected four additional datasets of non-standard norwegian and silver-labeled them using geolocation metadata and linguistic features. on the modelling side, they experimented with both encoder models (fine-tuning) and decoder models (few-shot prompting and supervised fine-tuning). in the end, one of the simplest setups consisting of the norbert3 encoder model fine-tuned on the provided development set (i.e., without the additionally collected data) yielded the best results. mainlp: team mainlp @xcite tried to improve performance for slot and intent detection with a variety of methods: varying the training data, injecting character-level noise, training on auxiliary tasks, and combining layers of models fine-tuned on different datasets. they found that injecting character-level noise is an efficient method for improving performance, training on auxiliary tasks did not lead to substantial improvements, and replacing layers of a model fine-tuned on english sid data with layers from a model fine-tuned on the provided development set could lead to substantial performance improvements. ltg: team ltg @xcite investigated potential improvements of the automatically translated training data. they improve the alignment of the slot labels with simalign @xcite and some heuristics, which leads to substantial performance improvements. they also use an llm foot_2 for translating the training data to cufe: team cufe (ibrahim, 2025) fine-tuned three bert models (mbert, nb-bert and nor-bert) for the intent detection and dialect identification tasks. they only used the provided development set for fine-tuning and found that the multilingual mbert model outperformed the norwegianspecific models. ## results we evaluated the submitted systems according to accuracy for intents, according to the span f1 score for slots (where both span and label must match exactly), and according to weighted f1 score for dialect identification. 5 for intent classification, the baseline was also outperformed by a large margin by all participants. the range of scores show that this task is close to being solved, even without any annotated training data in the target language (cf. @xcite . the detailed results in the northern varieties are slightly more challenging than the other dialects, but for all variants there are several systems which @xmath0. it is also noteworthy that the additional labeled norwegian massive dataset provided by team ltg @xcite did not yield any improvements for intent detection (and only marginal ones for slot filling). for dialect identification, all participating systems outperform the baseline. generally, the systems struggle most with identifying bokmål and nordnorsk, the two varieties with least data (1 and 2 translators, respectively). in the light of these re- ## analysis returning to the slot filling subtask, figure 3 shows multiple metrics for the best submission of each team over the whole test set (all dialects). precision is higher compared to recall for most participants, except ltg. we also report unlabeled f1, where we only check if the label boundaries match and ignore the label, and loose f1 which allows for partial matches. the unlabeled f1 is substantially higher for all teams, showing that finding the right label is still an unsolved issue. the loose f1 is always lower than the unlabeled f1, but still substantially higher than the strict span f1, showing that also finding the exact boundaries of a span is still challenging. furthermore, we looked into the most commonly confused intent pairs. all teams have the same top-2 confusion pairs, namely: searchscreeningevent-searchcreativework and cancel_reminder-cancel_alarm (gold-predicted). upon inspection, almost all mistakes in these categories are on the same instances. for example, the translations of the sentence "i want to see outcast", e.g. "eg vil se outcast." and "ae vil se outcast" are predicted as searchcreativework by all teams, but the more precise label searchscreeningevent was annotated. we also found two erroneous annotations for the cancel_reminder gold label, which clearly described alarms. other common mistakes included the prediction of set_alarm where cancel_alarm was annotated, and the prediction of playmusic where the true intent was search-screeningevent (likely triggered by to the word 'play'). the confusion matrices for dialect classification (figure 4 ) show one clear tendency, namely that the northern (n) and central (t) dialects are rarely confused with bokmål (b), whereas confusions between the western (v) dialects and bokmål is much more common. in fact, the highest numbers of sentence-level overlap with bokmål are observed with some of the western dialect writers. the models also struggle delimiting the three dialect areas (n, t, v), with significant confusion between the non-adjacent areas n and v. in comparison with the baseline, the submitted systems improve mainly by better distinguishing between t and v. ## conclusion this paper presented an overview of the norsid shared task organized as part of the vardial evaluation campaign 2025. the analysis of the results presented above suggests that intent detection is largely a solved task, where most of the remaining errors can be attributed to ambiguous labels. on the other hand, the other two subtasks still show room for improvement. the submitted slot filling models struggle with finding the correct slot boundaries and assigning the correct slot labels. since most submitted models were trained without significant amounts of norwegian training data, the training signal may not have been strong enough to address the first issue. it is also expected that some inconsistencies have remained in the nomusic dataset as a result of the translation and annotation. regarding dialect classification, the most standardized variant (bokmål) obtains the poorest scores, most likely due to the low amount of training data provided. more generally, it remains to be investigated to what extent the four major dialect areas (based on traditional dialectological research) represent the most useful partition of our data; in particular, the five translators of the western dialect area cover a relatively wide area where significant internal variation is expected. finally, it would be interesting to see what levels of dialect identification performance could be achieved by humans. both the slot filling and dialect identification subtasks proved rather challenging, which opens up opportunities for future evaluation campaigns.