Discriminating Between Similar N ordic Languages (2021)

Abstract: Automatic language identification is a challenging problem. Discriminating between closely related languages is especially difficult. This paper presents a machine learning approach for automatic language identification for the Nordic languages, which often suffer miscategorisation by existing state-of-the-art tools. Concretely we will focus on discrimination between six Nordic languages: Danish, Swedish, Norwegian (Nynorsk), Norwegian (Bokmål), Faroese and Icelandic.

## introduction automatic language identification is a core problem in nlp but remains a difficult task @xcite , especially across domains @xcite . discriminating between closely related languages is often a particularly difficult subtask of this problem @xcite . language technology for scandinavian languages is in a nascent phase (e.g. @xcite ). one problem is acquiring enough text with which to train e.g. large language models. good quality language id is critical to this data sourcing, though leading models often confuse similar nordic languages. this paper presents data and baselines for automatic language identification between six closelyrelated nordic languages: danish, swedish, norwegian (nynorsk), norwegian (bokmål), faroese and icelandic. further, we investigate feature extraction methods for nordic language identification and evaluates the performance of a selection of baseline models. finally, we test the models on a data set from a different domain in order to investigate how well the models generalize in distinguishing sim-ilar nordic languages when classifying sentences across domains. ## related work the problem of discriminating between similar languages has been investigated in recent work @xcite which discuss the results from two editions of the "discriminating between similar languages (dsl) shared task". over the two editions of the dsl shared task different teams competed to develop the best machine learning algorithms to discriminate between the languages in a corpus consisting of 20k sentences in each of the languages: bosnian, croatian, serbian, indonesian, malaysian, czech, slovak, brazil portuguese, european portuguese, argentine spanish, peninsular spanish, bulgarian and macedonian. similar work has included @xcite , who include nordic languages in a larger exercise in reproducing a commercial language id system; and @xcite , who attempt native language extraction, a task complex in the nordic context which is rich in cognates and shared etymologies. however, no prior work has focused specifically on the group of nordic languages, leaving users of those languages without high quality automatically-extracted single language corpora @xcite . this is particularly disadvantageous for some nordic language pairs, such as danish/norwegian and faroese/icelandic, where general-purpose many-language systems fall down @xcite . thus, we focus specifically on data for this language and baseline methods. ## the nordic dsl data set this section describes the construction of the nordic dsl (distinguishing similar lanugages) data set. data was scraped from wikipedia. we downloaded summaries for randomly chosen wikipedia articles in each of the languages, saved as raw text to six .txt files of about 10mb each. while bornholmsk would be a welcome addition @xcite , exhibiting some similarity to faroese and danish, there is not yet enough digital text. after the initial cleaning (described in the next section) the data set contained just over 50k sentences in each of the language categories. from this, two data sets with exactly 10k and 50k sentences respectively were drawn from the raw data set. in this way the data sets are stratified, containing the same number of sentences for each language. we split these data sets, reserving 80% for the training set and 20% for the test set. ## baselines 4.1 langid.py we compare with an off-the-shelf language identification system, langid.py @xcite . langid.py comes with a pretrained model which covers 97 languages. the data for langid.py comes from five different domains: government documents, software documentation, newswire, online encyclopedia and an internet crawl. features are selected for cross-domain stability using the ld heuristic @xcite . we evaluated how well langid.py performed on the nordic dsl data set. it is a peculiar feature of the norwegian language that there exist two different written languages but three different language codes. since langid.py also returned the language id "no" (norwegian) on some of the data points we restrict langid.py to only be able to return either "nn" (nynorsk) or "nb" (bokmål) as predictions. figure 1 shows the confusion matrix for the langid.py classifier which achieved an accuracy of 78.3% on the data set. the largest confusions were between danish and bokmål, and between faroese and icelandic. langid.py was able to correctly classify most of the danish instances; however, approximately a quarter of the instance in bokmål were incorrectly classified as danish and just under an eighth was misclassified as nynorsk. furthermore, langid.py correctly classified most of the icelandic data points; however, over half of the data points in faroese were incorrectly classified as icelandic. ## conclusion this paper presented dataset, baseline approaches, and analyses on automatically distinguishing similar nordic languages. we visualized embeddings produced by character level bi-grams, cbow and skipgram. we argue that, of these, fasttext's skipgram embeddings capture most information for discriminating between languages. data and code are available at https://github. com/renhaa/nordicdsl . as baselines, we compared four different classical models: knn, logistic regression, naive bayes and a linear svm with two neural network architectures: multilayer perceptron and a convolutional neural network. the two best performing models, fasttext supervised and cnn, saw reduced performance when going off-domain. using character n-grams as features instead of words increased the performance for the fasttext supervised classifier. training on multiple domains resulted in an expected performance increase.