sk LEP : A S lovak General Language Understanding Benchmark (2025)

Abstract: In this work, we introduce skLEP, the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding (NLU) models. We have compiled skLEP to encompass nine diverse tasks that span token-level, sentence-pair, and document-level challenges, thereby offering a thorough assessment of model capabilities. To create this benchmark, we curated new, original datasets tailored for Slovak and meticulously translated established English NLU resources. Within this paper, we also present the first systematic and extensive evaluation of a wide array of Slovak-specific, multilingual, and English pre-trained language models using the skLEP tasks. Finally, we also release the complete benchmark data, an open-source toolkit facilitating both fine-tuning and evaluation of models, and a public leaderboard at https://github.com/slovak-nlp/sklep in the hopes of fostering reproducibility and drive future research in Slovak NLU.

## introduction the field of natural language processing (nlp) has shifted towards large pre-trained models capable of handling a wide range of tasks. this trend has underscored the need for a standardized evaluation suite to test models across diverse tasks. benchmarks such as glue @xcite and su-perglue @xcite have become widely adopted, driving the development of models that excel on these tests and in general natural language understanding @xcite . however, these benchmarks primarily focus on english. in response, recent work has developed benchmarks for individual non-english languages @xcite @xcite and for multilingual evaluation @xcite @xcite . yet, this development remains skewed toward high-resource languages, often overlooking low-to mid-resource ones. in this work, we introduce sklep, a gluestyle benchmark for slovak, a mid-resource language @xcite with 10 million native speakers, comprising nine tasks. although similar benchmarks exist for other slavic languages such as bulgarian @xcite , polish @xcite , russian @xcite , and slovene @xcite , no equivalent has been established for slovak. this is especially problematic given the emergence of several slovak-specific large language models @xcite @xcite , making it difficult to benchmark their performance against well-established multilingual alternatives. creating a glue-style benchmark for slovak posed non-trivial challenges. while high-quality datasets exist for various tasks, they do not cover the full range expected in such a benchmark. thus, we introduced new datasets and translated established english resources to compile the complete set of tasks in sklep. using the tasks in sklep, we conduct what is, to the best of our knowledge, the first systematic evaluation of existing language models on slovak by finetuning them on our benchmark tasks and comparing their performance across the entire suite. our contributions can be summarized as follows: ## dataset construction and preprocessing our objective in constructing the benchmark was to develop a principled tool for evaluating the language understanding capabilities of the tested models. since most tasks were introduced in previous work, we maintained the original setup wherever possible to enable direct comparisons. some tasks (hs and qa) did not include a validation (development) split. we addressed this by sampling from their training sets to create a validation set matching the size of the test set. we also observed that the xnli and sts datasets contained duplicates, which were removed during preprocessing. in the sts dataset, in particular, we eliminated sentence pairs with identical textual representations that were assigned a score other than 5-likely an artifact of the translation pipeline described below. 1 the sklep code, data and models are available at https: //github.com/slovak-nlp/sklep 3 tasks the sklep benchmark comprises three task types: token classification, sentence-pair tasks, and document classification. below, we describe the datasets, their tasks, creation processes, and any major modifications made for inclusion in sklep. ## machine translation quality assessment due to the limited availability of high-quality slovak datasets for general language understanding, we translated established english tasks into slovak to construct a robust benchmark. to ensure methodological rigor, we conducted a slovak machine translation experiment using 90 sentences, equally drawn from the original nli, sts, and rte datasets. the experiment employed five translation systems: google translate, deepl, gpt-4o (prompted with "translate the given text to slovak"), madlad-400-3b @xcite , and nllb-3.3b @xcite . the resulting translations were evaluated by four native annotators (also co-authors of this work) following the methodology outlined in @xcite . translation quality was assessed using two approaches. first, annotators ranked the translations from each model/service in descending order; in the event of a tie, they skipped to the next available rank. second, annotators graded each translation on two dimensions-fluency (i.e., the extent to which the translation is coherent in the target language) and adequacy (i.e., the extent to which the translation conveys the original meaning)-using a 4-point likert scale. prior to annotation, annotators were instructed to review and adhere to custom translation quality evaluation guidelines adopted from @xcite ## experiments we use the proposed benchmark to evaluate various pre-trained language models (plms) based on the encoder-only transformer architecture. our selection includes well-established multilingual models, slovak-specific models, and monolingual english variants, as the latter have been shown to transfer well to non-english languages @xcite . this evaluation aims to assess the impact of different model variants, sizes, and pre-training regimes on overall performance. we provide a reference implementation for each of the considered tasks using the huggingface transformers framework @xcite and conduct an extensive hyperparameter grid search across all tasks, models and various hyperparameter settings by finetuning the models on the specific task's training data and evaluating it on its development split. more details can be found in appendix a. ## discussion software tools we developed the sklep benchmark as an open-source toolkit that can help researchers and practitioners choose models for specific tasks and supports the development and evaluation of new models. built on the established huggingface transformers framework @xcite , all datasets are prepared for integration with the huggingface datasets repository @xcite . we will release all software and data upon acceptance. dataset licenses to ensure continuity, the original dataset licenses remain unchanged and are summarized in leaderboard all evaluation results are compiled into the sklep leaderboard, offering a standardized way to assess the current state-of-the-art for various slovak nlu tasks. the leaderboard links model metadata (e.g. size as shown in finally, since leaderboards tend to saturate over time, we plan to open our platform to researchers to incorporate their datasets, design new benchmarks @xcite , and develop additional slovak resources using a human-andmodel-in-the-loop approach @xcite . we intend to unveil the leaderboard and all associated data upon acceptance. ## related work the release of the glue @xcite and super-glue @xcite benchmarks has made it much easier to evaluate pretrained language models on english nlu tasks. in the case of glue, comprised of nine tasks in total, these were dominated by four natural language inference (nli) tasks. on the other hand, half of the more sophisticated superglue's tasks are focused on question answering (qa). the english-specific glue and superglue have also inspired the creation of similar benchmarks for various other languages: clue @xcite in chinese , klej @xcite and lepiszcze @xcite in polish, club in catalan (rodriguez-penagos et al., 2021), indonlu for indonesian @xcite , liro in romanian @xcite , parsinlu in persian @xcite , slovene superglue in slovenian (žagar and robnik-šikonja, 2022), norbench in norwegian @xcite , klue in korean @xcite , flue in french @xcite , jglue for japanese @xcite , orca in arabic @xcite , basqueglue in basque (urbizu et al., 2022), russiansuperglue in russian (shavrina et al., 2020), dumb in dutch (de vries et al., 2023), supergleber in german (pfister and hotho, 2024), uinauil (basile et al., 2023) and invalsi (puccetti et al., 2025) in italian, portulan extraglue in portuguese @xcite and superlim in swedish @xcite . they generally assess core language understanding through tasks such as natural language inference, question answering, sentiment analysis, and named entity recognition (ner), along with syntactic and semantic evaluations (e.g., pos tagging, dependency parsing, and semantic similarity) and various language-specific diagnostic or classification challenges. to evaluate models across multiple languages, various multilingual benchmarks have also been proposed. the most prominent ones include xglue @xcite , xtreme @xcite and xtreme-r @xcite covering 19, 40 and 50 languages, respectively. unfortunately, none of these benchmarks cover the slovak language. moreover, they only provide english training data, making them more suitable for evaluating language transfer across languages rather than language-specific performance. the work conceptually and in spirit closest to ours would be the introduction of the slovakbert model @xcite . in it, the authors evaluate it against a set of other models on partof-speech tagging, semantic textual similarity, sentiment analysis and document classification. of these, the part-of-speech dataset is the same one sklep uses, the semantic textual similarity task has been automatically translated using an machine translation model which has since been surpassed by the ones we evaluate in our experiments, and the document classification task can be considered solved, as slovakbert reports macro-f1 score of 99 out of 100. in sklep we build upon this work and extend it into a fully featured slovak general language understanding benchmark.