BENCH ić-lang: A Benchmark for Discriminating between B osnian, C roatian, M ontenegrin and S erbian (2023)

Abstract: Automatic discrimination between Bosnian, Croatian, Montenegrin and Serbian is a hard task due to the mutual intelligibility of these South-Slavic languages. In this paper, we introduce the BENCHić-lang benchmark for discriminating between these four languages. The benchmark consists of two datasets from different domains - a Twitter and a news dataset - selected with the aim of fostering cross-dataset evaluation of different modelling approaches. We experiment with the baseline SVM models, based on character n-grams, which perform nicely in-dataset, but do not generalize well in cross-dataset experiments. Thus, we introduce another approach, exploiting only web-crawled data and the weak supervision signal coming from the respective country/language top-level domains. The resulting simple Naive Bayes model, based on less than a thousand word features extracted from web data, outperforms the baseline models in the cross-dataset scenario and achieves good levels of generalization across datasets.

## introduction the status of "separate language" for bosnian, croatian, montenegrin and serbian is frequently discussed and is in academic circles mostly understood as related to the construction of identity @xcite and diverging and converging tendencies throughout history @xcite . while each is an official language in the respective country, with a separate top-level internet domain @xcite , their mutual intelligibility cannot be disputed. regardless of the mutual intelligibility, differences do exist @xcite . in this paper, we introduce a discrimination benchmark based on two datasets: a newspaper-based one, covering three out of four languages, and a twitter-based one, covering all four languages. the publication of this benchmark coincides with the 10th anniversary of the vardial workshop, in which this language group has been involved from the beginning. the main contributions of this paper are the following. we introduce two datasets, based on previously collected data, that we now encode with maximal structure and publish in an academic data repository following the fair principles @xcite . we introduce a benchmark based on the two datasets, and present baselines for the benchmark. given the low performance of these competitive baselines on the benchmark, we introduce a new web-dataset-based method that shows to carry specificities of each language across the two datasets much better than any model directly trained on one of the two datasets. we hope that the availability of this benchmark, as well as the introduced strong competitors, will motivate further research in discriminating between similar languages. ## benchmark datasets the benchmark consists of two rather different datasets, whose selection was made with the aim of fostering cross-dataset evaluation of different modelling approaches. the first dataset is the parallel newspaper dataset from the "south-eastern times" (setimes) website covering news in languages of south-eastern europe, including bosnian, croatian and serbian. the dataset has been part of the var-dial shared task since 2014 @xcite as part of the dslcc collection @xcite , and was present in the following iteration of the shared task as well @xcite . within vardial, it was available in the form of 22 000 instances per language, each no longer than 100 tokens. we have now published all available content from the setimes website in the form of 9 258 whole documents @xcite . 1 the documents are separated into a train, devel-opment and test subset in an 8:1:1 ratio. while dividing the documents, we made sure that, given that the dataset consists of the same content in the three languages, there is no leakage of parallel data across these three subsets, especially given the mutual intelligibility of the languages covered. we assume that, given the parallel nature of this dataset, it could be very useful in learning the specifics in which the three close languages differ. the median length of instances (documents) is 627 words, while the arithmetic mean length is 849 words. the second dataset is based on tweets, harvested with the tweetcat @xcite tool. this dataset was used as the out-of-domain testing data in the third iteration of the vardial shared task @xcite , but in significantly smaller volume than what we included in this benchmark. we share tweets of 614 users, 394 of which are labeled as tweeting in serbian, 89 in croatian, 75 in bosnian, and 56 in montenegrin. each user is represented with at least 200 tweets, merged in our experiments into one single text per each user. single tweets were not filtered by the language they are written in, which allows for other languages besides the four languages of interest to occur in the dataset, such as infrequent tweets in english. with this decision we wanted to keep the dataset as natural and realistic as possible. the users were split into the train, development and test subset in a 3:1:1 ratio, so that the development and test splits would be large enough. the median length of instances (all tweets of one user) is 5,438 words, while the arithmetic mean length is 7,257 words. the dataset is published as a json file, each primary entry representing one user, the label denoting which language the user is tweeting in, and a list of the users' tweets @xcite ). 2 the benchmark allows for training on any of the two training datasets, as well as using external data, provided that it does not overlap with texts in the test split. hyperparameters or model decisions can be chosen with the help of development data. the two official metrics of the benchmark are micro f1 and macro f1, both considered equally important. the researchers are welcome to add to this benchmark the results achieved on any combination of training and testing datasets (in-domain or out-domain). however, the primary goal of this benchmark is to present results obtained in the cross-dataset scenario, that is, testing the model on test data from a dataset on which the model was not trained on, to prove the general applicability of the resulting model on the task of discriminating between the languages in question. the results of various models can be submitted via the github repository foot_2 through a pull request. ## experiments we experiment with two approaches: the baseline approach -a linear svm model with character n-gram representation, described in section 3.1, and our new approach, presented in section 3.2: a naive bayes model using a text representation based on feature extraction from national web corpora. the classifier selection in each of the approaches is based on best results on the development data, and each of the two classifiers were considered in each of the approaches. ## conclusion in this paper, we introduce the benchić-lang benchmark for discriminating between four very similar languages: bosnian, croatian, montenegrin and serbian. the benchmark consists of two rather different datasets, providing a good test bed for beyond-model generalizability. we introduce two methods for discriminating between the languages. the first, a baseline, is a linear svm model using character n-gram features, showing to perform well in-dataset, but not having generalization power to perform well in the crossdataset setup. for that reason, we introduce another approach, exploiting only web-crawled data and the weak supervision signal coming from the country/language respective top-level domains. we perform heavy feature selection of less than 1000 word features on one subset of the web data, and train a naive bayes model on the remainder of the web data. we show that this model performs much better than the character n-gram models in the cross-dataset setting. what is more, it even outperforms the in-dataset results of the svm model on one of the twitter test sets. while we obtain very stable results on bosnian, croatian and serbian, we must put forward that neither the in-dataset svm twitter, nor the nb web model perform satisfactory on discriminating montenegrin from the three other languages, which is a task to be tackled in future work. besides improving the identification of montenegrin, there are many other directions we hope the community will investigate. one direction is exploiting linguistic features known to vary between the four languages @xcite and base the classification decision on these features. another is to investigate transformer models, finetuning them either on the training data, or on the weak-supervision web data. we have performed an initial experiment on the latter, fine-tuning the bertić model @xcite for one epoch on the 400 000 web documents. during this first epoch we consistently obtained low results, with no tendency of improvement. additional experimentation, potentially with lower learning rates or more complex loss functions, could be performed here. finally, additional datasets should be added to the benchmark, especially such datasets that cover all of the four languages of interest.