Superlim: A S wedish Language Understanding Evaluation Benchmark (2023)

Abstract: We present Superlim, a multi-task NLP benchmark and analysis platform for evaluating Swedish language models, a counterpart to the English-language (Super)GLUE suite. We describe the dataset, the tasks, the leaderboard and report the baseline results yielded by a reference implementation. The tested models do not approach ceiling performance on any of the tasks, which suggests that Superlim is truly difficult, a desirable quality for a benchmark. We address methodological challenges, such as mitigating the Anglocentric bias when creating datasets for a less-resourced language; choosing the most appropriate measures; documenting the datasets and making the leaderboard convenient and transparent. We also highlight other potential usages of the dataset, such as, for instance, the evaluation of cross-lingual transfer learning.

## introduction in recent years, the move towards transformerbased large pretrained language models has brought the need for comprehensive and standardized multi-task benchmarking in nlp into focus. since these models, through their generic pretraining setup, can be fine-tuned on many different tasks, a test suite which covers a range of different nlp domains is required to measure overall performance. for this reason, several different benchmarks have been proposed. for english, the most notable suites arguably are the (super)glue benchmark @xcite for natural language understanding and gem @xcite for natural language generation. in this paper, we present superlim, a multi-task benchmark for swedish natural language under-standing. it contains 14 different datasets which cover 15 tasks ranging from word understanding tasks, such as word analogy and word similarity, to text classification tasks, such as natural language inference and sentence acceptability. the release of this benchmark comes at a time when multiple pre-trained language models have been created for swedish, and larger models are under construction (ranging from @xcite . in this active research environment, a multi-task benchmark is crucial to gauge state-ofthe-art performance and guide future development. for swedish this is especially important, since limitations (financial, computational and related to data availability) are more considerable than for english. creating a benchmark for swedish comes with its own unique set of practical challenges and methodological considerations. while there exist high-quality swedish datasets that can be reused, not all of them have been created with the task of evaluating large language models in mind. here, the relatively close linguistic relationship to english comes with possibilities of translating (or adapting) existing datasets. such approaches do however potentially introduce linguistic and cultural anglocentric bias into swedish language benchmarking. therefore, superlim combines datasets of three kinds: a) preexisting ones, reformatted and revised, b) translated or adapted from english, and c) created from scratch specifically for superlim. to facilitate progress tracking and sharing of results, models and datasets, we create an online leaderboard for superlim, which will serve as a platform for submitting and publishing results, and release reference implementations with baseline results for a collection of existing swedish language models. the rest of the paper looks as follows: in section 2 we briefly cover related work. section 3 describes the tasks included in superlim and the corresponding datasets, and section 4 their standardized documentation. section 5 motivates superlim's main evaluation measure: krippendorff's α. the online design of the leaderboard is the subject of section 6. sections 7 and 8 describe the reference implementations and the baseline results. section 9 provides details on where to find superlim and section 10 concludes the paper. ## related work the already mentioned (super)glue has inspired the creation of similar benchmarks for other languages: clue @xcite and climp @xcite for chinese, russian super-glue @xcite , klej @xcite and lepiszcze @xcite for polish, parsinlu for persian @xcite , indonlu for indonesian @xcite and norbench @xcite for norwegian. several suggestions to move to other evaluation approaches have been voiced @xcite @xcite , but the glue-like benchmarks still remain the default approach. superlim is not the only benchmark for swedish. very recently, scandeval has been released @xcite . this multilingual suite contains four tasks for swedish. some of the datasets used for the evaluation contain synthetic data (for instance, incorrect sentences created by scrambling word order in the correct sentences), which is not necessarily problematic, but presumably less valuable than natural data. another benchmark for swedish, overlim @xcite , is entirely machine translated from english. ## tasks and datasets we have three types of tasks: text level (section 3.1), word level (section 3.2) and diagnostic (section 3.3). describing the creation of all datasets in detail is beyond the scope of this paper. instead we focus on briefly describing the datasets, their tasks and any substantial changes that have been made to the original sources for the purposes of superlim. for more detailed informa-tion about the datasets and their creation, we refer to the documentation that accompanies superlim, itself described in section 4. most datasets are divided into predefined training, development and test splits. all test sets are of gold-standard quality: they have been manually annotated from scratch or manually corrected after automatic processing. some of the training and development sets may contain non-corrected data, for instance automatically translated data. our rationale here is that a model can be trained on anything, as long as it yields high, robust, unbiased performance. evaluation, however, must always occur on gold data to serve its purpose. an important caveat, however, is that those test sets that were automatically translated and then manually corrected may suffer from so-called post-editese @xcite . in all cases, the editing was carefully done by native speakers with background in nlp and/or linguistics, who were aware of this potential problem. we hope that these factors have at least mitigated it. the word-level datasets were mostly created with non-contextualized, type level approaches in mind (for instance, static word vectors like those obtained from word2vec). these therefore do not contain large training splits, but small training sets that can be used to calibrate otherwise unsupervised methods, or for few-shot training generative models. the diagnostic datasets only have test splits. here, the training and development splits of swenli are to be used. ## documentation the importance of documenting datasets cannot be overestimated. still this step is often overlooked in the nlp field. since some parts of superlim are revised versions of pre-existing datasets, it is particularly important to be explicit about what changes were made. we therefore devised a documentation sheet: a template inspired by @xcite "datasheets" and pushkarna et al.'s (2022) "data cards". our template, however, is more compact than the former and is specifically adapted to the superlim context. documentation sheets consist of six sections: each superlim task is accompanied by a documentation sheet. the compactness of the template is intentional: it improves the chances that documentation will actually be written, maintained and read. an example can be found in appendix a. ## evaluation measures a benchmark containing a mix of task types will almost by necessity involve a wide range of evaluation methods and measures. the appropriate choices depend on the tasks themselves and which insights we would like to get into our models. good research practice tells us to evaluate models in those ways that teach us most about them. however, in a benchmark "competition" setting, were we would like to summarize a model's performance with one final score, this leads to a problem. in glue, this is dealt with by providing an average over very different beasts like accuracy, matthews correlation coefficient, f-score, and spearman's correlation coefficient (that have different ranges and different intepretations). in the context of the superlim benchmark, we explore an alternative: the use (where possible) of a single family of measures, krippendorff's α (krippendorff, 2018). krippendorff's α is a measure of inter-annotator agreement, for instance used in work on linguistic annotation @xcite . we use α as an evaluation measure by treating the system to be evaluated as one annotator, and the gold-standard creators as another. a perfect system always agrees with the gold-standard creators (α = 1), whereas a poor system shows no systematic agreement with them (α ≈ 0). even lower scores (-1 ≤ α < 0) are signs of systematic mistakes. krippendorff's α is parameterized by a distance metric @xcite which allows us to use it on different types of task. for binary and ternary labelling tasks, we use nominal-α. in the binary case, nominal-α linearly maps to macro-averaged f-score. for scoring tasks we use interval-α. since it is a measure of agreement, interval-α is stricter than correlation measures like pearson's r, which do not require a system to get the sizes of predicted scores correct, only the shape of their distribution. for a benchmark, we find this strictness a desirable property. the selection tasks do not allow for a direct application of α. however, we can calculate a derived measure, a pseudo-α, by pretending, after the fact, that the selection tasks are binary decision tasks and by calculating a nominal-α for this recasted task. we present a more elaborate investigation of α as an evaluation measure in a separate paper @xcite . we currently do not have an α-based measure for the summarization task swedn, for which the benchmark uses rouge-1. swewinogender is evaluated using nominal-α, but in addition also using gender parity, which measures how consistently the model answers across datapoints that only differ in the choice of pronoun. ## leaderboard the benchmarking results are collected at the superlim leaderboard, where both development and test set results are made available. in addition to the evaluation outcomes on the different tasks, the leaderboard also lists information about a participating model's type and size (in number of parameters), and the size of the dataset used to pre-train the model. the superlim project strives for transparency and openness, not just with regards to our datasets but also to the contents of the leaderboard. as part of this, we link participating models' predictions on development and test sets as well as scripts and configuration details used to produce those results. this facilitates replication of results and confirmation of their validity. there is an additional point with this level of transparency: the gold labels of the superlim test sets are not hidden. this makes evaluation easier, but it also makes cheating (intentional or accidental, see discussion in the limitations section) very easy. we hope that asking participants to be open about their methods can act as a countermeasure to this potential problem. the design of the leaderboard is also guided by the idea that there is no single best model (ethayarajh and jurafsky, 2020). the choice of a model for some application sometimes need only depend on the performance on certain subsets of our evaluation suite and can even be restricted to only included models up to a certain size. we therefore allow users of the leaderboard to filter and sort out all the tasks and models that are irrelevant for their specific needs: users can include and exclude tasks, filter out model families, sort by size, overall performance, and task-specific performance. consider the following use case: a developer wants to build a proofreading application using a transformer model, and needs their application to run as fast as possible on limited hardware. the most relevant task in superlim for them is dalaj-ged, and the developer may wish to restrict themselves to models with less than 150m parameters. in the leaderboard, they can now easily select all base models, exclude all word-level tasks to declutter the table, and sort according to performance on dalaj-ged. that would bring on top the kblab/bert-base-swedish-cased-new model, that otherwise performs worse than average. the developer further checks if the performance on the validation set matches the performance on model type size language kb/bert-base-swedish-cased bert base sv kblab/bert-base-swedish-cased-new bert base sv kblab/megatron-bert-base-swedish-cased-600k bert base sv kblab/megatron-bert-large-swedish-cased-165k bert large sv ai-nordics/bert-large-swedish-cased bert large sv xlm-roberta-base roberta base multi xlm-roberta-large roberta large multi ai-sweden/gpt-sw3-. . . gpt 126m-40b sv, no, da, is, en the test set, before finally choosing a small set of suitable models to evaluate on their new application's data. a screenshot of the leaderboard taken from this scenario can be found in figure 1 in appendix b. ## reference implementation to create baselines for superlim, we built a number of reference implementations using the swedish language models listed in we created reference implementations for all text-level tasks except for swedn. static word vectors foot_2 were used as an approach to supersim relatedness and supersim similarity by scaling the word pairs' cosine similarities to 0-10. swesat was addressed by picking the choice with the highest cosine similarity. for swedish analogy we first applied vector algebra to calculate a vector for the fourth (missing) item, and then used cosine similarity to pick the closest vocabulary item. the two word-level selection tasks were also solved using the gpt-sw3-... models @xcite . swedish analogy by inspecting logprobabilities of a swedish translation of the text a synonym is being sought for the search term. the term x serves as a synonym for y, where x is a choice item and y the question's target expression, and choosing the most probable item. swesat employed a few-shot learning strategy to prompt the gpt model to complete the analogy. ## results and discussion the results for the word-level tasks are in particularly low scores are achieved for the inference task swenli and the pronoun resolution task swewinograd. the bad performance for the former is connected to the low scores on both diagnostic tasks, which use the same training data and are of the same task type. overall performance on swediagnostics maxes out at α=.415, a breakdown of swediagnostics results can be found in appendix c. there is no column for the swewinogender diagnostic task in the models perform best on the paraphrase task sweparaphrase. ## distribution all datasets constituting superlim are available under creative commons licenses (cc by 4.0, cc by-sa 4.0, respectively), and can be downloaded together with their documentation sheets from språkbanken text's website 4 or accessed through huggingface. 5 the superlim leaderboard can be found at kblab's website, 6 and includes model predictions as well as detailed information about the participating models. ## conclusion we have presented superlim, a swedish benchmark and analysis platform for natural language understanding in the style of glue and superglue. superlim contains 13 regular tasks and 2 diagnostic tasks from a broad range of task types, including word-level tasks that are not commonly seen in glue-style benchmarks. all tasks come with goldstandard test data, with training and development data -albeit not always of gold standard qualityand with detailed, standardized documentation. we have chosen to make superlim's leaderboard flexible and transparent, which we hope may help promote the dissemination of knowledge about the capacities of natural understanding models for swedish and knowledge about their construction. finally, by having comparable tasks to the de facto standard superglue, superlim not only contributes to the development of natural language understanding for swedish, but also to research into transfer learning and multilingual models, and thus to a much wider part of the field of natural language processing.