bg GLUE : A B ulgarian General Language Understanding Evaluation Benchmark (2023)

Abstract: We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural language inference, fact-checking, named entity recognition, sentiment analysis, question answering, etc.) and machine learning tasks (sequence labeling, document-level classification, and regression). We run the first systematic evaluation of pre-trained language models for Bulgarian, comparing and contrasting results across the nine tasks in the benchmark. The evaluation results show strong performance on sequence labeling tasks, but there is a lot of room for improvement for tasks that require more complex reasoning. We make bgGLUE publicly available together with the fine-tuning and the evaluation code, as well as a public leaderboard at https://bgglue.github.io , and we hope that it will enable further advancements in developing NLU models for Bulgarian.

## introduction natural language understanding (nlu) benchmarks, such as glue @xcite and superglue @xcite , were designed for a rigorous evaluation of language models on a diverse set of natural language understanding (nlu) tasks. the wide adoption of such benchmarks has driven the rapid development of models that perform well on the tasks that are part of these benchmarks, but also beyond @xcite . however, until recently, the focus of such benchmarks has been on english, with little interest in other languages @xcite . to address this, recent work has designed benchmarks to test models on nlu tasks on non-english languages @xcite @xcite or on multiple languages @xcite . here, we aim to improve the diversity of the languages represented in nlu benchmarks by proposing bgglue, a benchmark for bulgarian that consists of nine nlu tasks, including token classification, regression, and classification. thus far, only individual datasets in bulgarian have been used for model development and evaluation. small subsets of up to three downstream tasks in bulgarian have also been included in some multilingual benchmarks @xcite . additionally, there are existing benchmarks focusing on other balto-slavic languages, such as the russian superglue @xcite and the slovene superglue @xcite . however, there are no comprehensive benchmarks for representatives of the eastern south-slavic language subgroup, and for bulgarian in particular. we aim to address these limitations with bgglue. ## data preparation here, we describe the pre-processing steps we took to prepare the datasets before including them in the bgglue benchmark. our main goal was to ensure that the setup evaluated the language understanding abilities of the models in a principled way and in a diverse set of domains. since all of the datasets were publicly available, we preserved the original setup as much as possible. nevertheless, we found that some datasets contained duplicate examples across their train/dev/test splits, or that all of the splits came from the same domain, which may overestimate the model's performance. hereby, we removed data leaks and proposed new topicbased or temporal-based (i.e., timestamp-based) data splits where needed. we deduplicated the examples based on a complete word overlap in two pairs of normalized texts, i.e., lowercased, and excluding all stop words. for the bsnlp task, we combined the data from three consecutive editions of the ner shared task. we selected all bulgarian examples, which encompassed six different topics. we used the latest two topics for testing, and split the rest randomly at a 4:1 ratio for training and validation. the credible news dataset contained data from six sources and various news topics. we split the dataset both by topic and by source. in particular, we included all news articles from the same topic in the same split. for training and validation, we used documents from the largest sources from the two categories. moreover, we extracted the same topics from the two and grouped them together within the splits, keeping the class ratio at 1:10. in the test dataset, we used data points from all six data sources. 2 we note that our test set contains all data points from the two new sources, which are more recent, making them even more challenging. finally, we cleaned the texts from duplicates and removed all keywords indicating the source: the names of the authors, the media source, the urls, etc. more details about the collection process of the news articles are given in appendix c. we prepared an entirely new split for cinexio, as there was no standard one. first, we removed all duplicate comments and we sorted the remaining ones by publication time. next, we split the comments using half for training and the rest equally for validation and testing. finally, we removed from the training set the comments for the same movie that appeared in the other two splits, which changed the distribution by slightly increasing the proportion of the training set. for exams, we kept the original validation and test splits @xcite . we added all additional questions from @xcite to the training set, i.e., the category history online quizzes. the fake news dataset was released as part of a shared task and it was already split into training and testing sets. after manual inspection, we found that some of the articles had only their titles reworded and had the same content. therefore, we removed duplicates based only on exact matches in the article's body. in addition, we designated 10% of the training examples for validation. pan-x contained short sentences from wikipedia automatically annotated based on the available wiki entities. the dataset consists of 20k examples for training and 10k for validation and testing. we kept the proposed splits, but we checked for duplicates by converting each sentence to lowercase and removing the punctuation. this resulted in the removal of 10k sentences overall. our analysis did not find any issues with ct21.t1, u.dep, and xnli, and thus we kept the splits as provided originally. ## experiments in this section, we first describe the baseline systems we experiment with and then we present the evaluation results. ## discussion software tools as part of building the benchmark, we developed a set of software tools that facilitate the training and the evaluation of new models. the toolkit is implemented in pytorch @xcite , using the transformers library @xcite . moreover, we integrate and release publicly all datasets (in accordance with their licenses; see the next paragraph) from bgglue in the huggingface datasets repository @xcite . 5 dataset licenses we keep the licenses as provided by their authors for all datasets included in the bgglue benchmark. ## conclusion and future work we presented bgglue -the first holistic benchmark for evaluating nlu systems in bulgarian. it includes nine challenging tasks that cover token classification, regression/ranking, and text classification. we fine-tuned and evaluated six different pre-trained state-of-the-art language models. our extensive evaluation showed that bgglue contains challenging tasks that are far from being solved. finally, we open-sourced the cleaned versions of the datasets, including the new, more challenging splits, and the source code for training and evaluation, and we released 36 fine-tuned models (one for every task and model combination). all the released artifacts are also integrated into the hugginface hub. we believe that bgglue is a rich and challenging testbed that will cultivate prospective work on bulgarian language understanding. in future work, we plan to add more tasks for bulgarian, e.g., toxicity detection @xcite . we also want to use monolingual bulgarian datasets for pretraining, beyond wikipedia and commoncrawl, e.g., @xcite @xcite @xcite , using which will require a thorough assessment in order to prevent introducing unwanted biases and hazardous behavior in the models trained on them @xcite . finally, we plan to try recent multilingual models such as mdebertav3 @xcite , mt0 and bloomz @xcite .