Multi-label S candinavian Language Identification ( SLIDE ) (2025)

Abstract: Identifying closely related languages at sentence level is difficult, in particular because it is often impossible to assign a sentence to a single language. In this paper, we focus on multi-label sentence-level Scandinavian language identification (LID) for Danish, Norwegian Bokmål, Norwegian Nynorsk, and Swedish. We present the Scandinavian Language Identification and Evaluation, SLIDE, a manually curated multi-label evaluation dataset and a suite of LID models with varying speed–accuracy tradeoffs. We demonstrate that the ability to identify multiple languages simultaneously is necessary for any accurate LID method, and present a novel approach to training such multi-label LID models.

## introduction correctly identifying the language of a short piece of text might seem like a simple (and possibly already solved) task. while differentiating between two distant languages might be straightforward, we show that, when focusing on a group of closely related languages, this task becomes substantially more challenging. this is especially true when we consider the fact that language identification (lid) tools have to be fast and efficient, as they are often used for preprocessing large quantities of texts. in this paper, we focus on the four closely related scandinavian languages: danish, norwegian figure 1 : scandinavian similarity accurate language identification has to necessarily be multilabel when discriminating between closely related languages. bokmål, norwegian nynorsk, and swedish. in order to accurately differentiate within this group, we move away from the standard single-label (multiclass) language identification and instead treat this problem as multi-label classification task, allowing for the identification of multiple languages simultaneously as illustrated in figure 1 . sentences valid in multiple scandinavian languages are fairly common-they account for about 5% of our evaluation dataset and 16% of the sentences shorter than 6 words. if not accounted for, these examples can skew evaluation of existing systems. the three main contributions of slide (scandinavian language identification and evaluation), are as follows: language identification the task of identifying the language of a text is an "old" nlp task dating back to the 1960s. simple but relatively powerful tools have been available since the 1990s @xcite . in recent years, the main focus of nlp research has shifted towards large language models, and especially towards extending their coverage to an increasing number of languages. as training data for underrepresented languages is mostly found in web crawls, reliable lid systems covering a large number of languages are more important than ever. while the earliest lid systems were restricted to a dozen languages, recent systems cover hundreds @xcite @xcite and even thousands @xcite of languages. in terms of methods, simple linear classifiers with character-level and word-level features have often outperformed more sophisticated neural models @xcite . most currently available large-coverage lid models are based on the fasttext architecture @xcite , a multinomial logistic regression classifier with character n-gram embeddings as input features. these include fasttext-176 @xcite , nllb-218 @xcite , openlid @xcite and glotlid @xcite . different approaches are used by heli-ots @xcite , which bases its decisions on a combination of character n-gram and word unigram language models, and gpt2-lang-ident foot_0 , which is a fine-tuned decoderonly model @xcite . in practice, lid is most often applied to individual sentences, even though the tools can work with longer or shorter segments of text. ## data one of the main contributions of this paper is the release of manually and automatically annotated multi-label datasets. in section 3.1, we introduce the sources from which we compile our datasets. we then present our manually annotated multilabel evaluation dataset (section 3.2). next, we describe a way to obtain multi-label annotations automatically for the larger training set in section 3.3. lastly, we outline different approaches to data augmentation in section 3.4. ## slide evaluation we introduce two evaluation metrics in our comparison: loose and exact-match accuracy. loose accuracy this evaluation metric is designed for models that output only one language label per input, which is common for off-the-shelf classifiers like fasttext and nllb. according to this metric, a prediction is considered correct if the single predicted label is among the gold labels. this metric is unreliable for multi-label models, since a model that always predicts all four languages would get 100%. exact-match accuracy this evaluation metric is more strict and requires an exact match between the predicted and gold labels sets, making it more appropriate for models capable of predicting multiple labels. per-language scores additionally, we report the f 1 -score for each individual language to measure the quality of classifications for each of the four languages separately. here, a true positive prediction happens if and only if the respective language is present both in the set of predicted labels and in the set of gold labels. ## slide training methodology in this section, we present our approach to training the slide models. we explore two main direc-tions: transformer-based models that achieve high accuracy but require more computational resources, and a fast model based on static word embeddings that trades accuracy for faster inference times. ## experiments we evaluate our slide models against several established lid baselines, comparing both prediction accuracy and speed. our evaluation focuses on two key aspects: performance on our manually annotated multi-label test set, and generalization to out-of-domain data. we first describe the baseline models used for comparison, then present our main results and the results of our out-of-domain experiments. ## discussion performance of baseline models the baseline models exhibit varying levels of performance, see performance of slide models our three bert-based lid models slide-xs, slide-small and slide-base perform the best on our test set, with the base version reaching an exact-match accuracy of 96.4%, while the small and xs both reach 95.7%. this comes at the cost of significantly longer runtimes compared to the static embedding models. these models are suitable when high accuracy is of most importance. however, it is worth noting that we measured inference speed solely on a cpu, one sentence at a time, to ensure a fair comparison with the faster baseline models intended for cpu usage. using a gpu with larger batch sizes would result in significantly faster runtimes for the transformer models. while our slide-fast model reaches the same exact-match accuracy as glotlid, 93.4%, it performs better on nynorsk, bokmål and danish, with nynorsk performance increasing by 3%. overall, performance on danish is consistently the lowest-the best model reaches 92% f 1 . our models have been trained on more bokmål than danish data, and we observe a slight tendency to predict only bokmål instead of both bokmål and danish for multilingual samples. we do, however, notice the same trend with lower danish performance across all evaluated models, see as seen in error analysis common error sources are proper names (half of 'other' instances misclassified as scandinavian contains proper names (e.g. 'kruvi: karl marx'), instances in english (30% of 'other' instances misclassified as scandinavian), and loanwords ('-ta avisa kommersant.', 'server med pas-tasalat med bakte grønsaker og tsatsiki til', 'men anne linnet -oh la la.') bokmål and nynorsk are confused most often. if a sentence valid both in bokmål and nynorsk contains irregular bokmål spelling like 'høg' instead of 'høy', and 'tjuvfiske' instead of 'tyvfiske', it is likely to be misclassified as nynorsk only. some errors imply that particular tokens influence the prediction more than a sentence representation as a whole: 'høyre' is a valid word both in nynorsk ('hear') and bokmål ('right'), but the sentence 'i alle år har vi fått høyre at med dagens forbruk er det olje nok for mange tiår.', which is nynorsk because of 'høyre' used as a verb, is misclassified as bokmål, while a both bokmål and nynorsk sentence 'i den nye designen er høgre og venstre spalte på framsida til nettavisa fjerna.' is misclassified as only nynorsk because of the spelling. additionally, some 'other' instances containing subwords matching those in scandinavian are misclassified, although the whole sentence semantics does not make any sense: 'va shiaulteyr er ny skeabey harrish boayrd.' (manx). ## conclusion we release a novel multi-label lid dataset for danish, norwegian bokmål, norwegian nynorsk and swedish with manually annotated validation and test splits. using machine translation for creating a silver multi-label training dataset from a singlelabel one has proved to be efficient. although fine-tuning models for a specific data source may be helpful to obtain high performance on a selected test set, such models (especially the fasttext-based ones) may be not robust towards the test dataset change. also, excessive training data preprocessing may lead to performance degradation on data from unknown domains compared with training without any preprocessing.