KLEJ : Comprehensive Benchmark for P olish Language Understanding (2020)

Abstract: In recent years, a series of Transformer-based models unlocked major improvements in general natural language understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which allow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of languages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing datasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and promote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and applications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an extensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based models.

## introduction the field of natural language understanding (nlu) experienced a major shift towards knowledge reusability and transfer learning, a phenomenon well established in the field of computer vision. such a shift was enabled by recent introduction of robust, general-purpose models suitable for fine-tuning, like elmo @xcite , ulmfit @xcite and bert @xcite . these models significantly improved the state-ofthe-art on numerous language understanding tasks. since then, the progress accelerated significantly and the new transformer-based @xcite models are being published every month to claim the latest state-of-the-art performance. such a pace of development would not be possible without standardized and publicly available nlu evaluation benchmarks. among the most popular ones is the recently introduced glue @xcite ) consisting of a collection of tasks such as question answering, sentiment analysis, and textual entailment with texts coming from a diverse set of domains. some tasks come with numerous training examples, while others have limited training data. on top of that, for some tasks, the training set represents a different domain than the test set. this promotes models that learn general language representations and are effective at transferring knowledge across various tasks and domains. the glue benchmark is constructed based on existing datasets, and its main contribution is the careful choice of tasks together with an online evaluation platform and a leaderboard. unfortunately, most of the progress in nlu is happening for english and chinese. other languages lack both pretrained models and evaluation benchmarks. in this paper, we introduce the comprehensive multi-task benchmark for the polish language understanding -klej (eng. glue, also abbreviation for kompleksowa lista ewaluacji językowych, eng. comprehensive list of language evaluations). klej consists of nine tasks and, similarly to glue, is constructed mostly out of existing datasets. the tasks were carefully selected to cover a wide range of genres and different aspects of language understanding. following glue, to simplify a model evaluation procedure, we adjusted the tasks to fit into a unified scoring scheme (either text classification or regression). alongside the benchmark, we introduce herbert, a transformerbased model trained on several polish text corpora. we compare herbert with a set of both standard and recently introduced nlu baselines. to summarize, our contributions are: ## related work the evaluation of nlu models was always an integral part of their development. even though there are many established tasks on which to evaluate newly proposed models, there is no strict standard specifying which one to choose. the difficulty of a fair comparison between models eventually led to the introduction of multi-task benchmarks that unify the evaluation. one such benchmark is senteval @xcite . it consists of seventeen established tasks used to evaluate the quality of sentence embeddings. additionally, ten probing tasks are provided to detect what linguistic properties are retained in sentence embeddings. in all tasks, models take either a single sentence embedding or a pair of sentence embeddings as the input and solve a classification (or a regression) problem. the authors released a toolkit foot_1 for model evaluation. however, they do not provide a public leaderboard to compare the results of different models. another benchmark for evaluating models is de-canlp @xcite , which consists of ten pre-existing tasks. in contrast to senteval, choice of tasks is much more diverse, ranging from machine translation, semantic parsing to summarization. all tasks have been automatically converted to a question answering format. finally, the glue benchmark @xcite proposes a set of nine tasks. all of them are constructed from existing, well-established datasets. authors selected tasks that are more diverse and more difficult than senteval. otherwise, the design of the benchmark is similar to senteval. the aforementioned benchmarks are limited to the english language. noteworthy attempts at providing multi-language benchmarks include xnli dataset @xcite , with the mnli @xcite dataset translated by professional translators into 14 languages. a similar effort is xquad @xcite which is a translation of the squad dataset @xcite into 10 languages. none of these efforts includes polish. other resources to evaluate the polish language understanding models are scarce. recently, @xcite prepared their version of the senteval probing tasks for the polish language. however, it is more suited for analyzing the sentence embeddings linguistic properties than assessing their quality. the poleval @xcite @xcite foot_2 platform organizes an annual competition in natural language processing for the polish language. during the first three editions, it assembled 11 diverse tasks and attracted over 40 teams. it could serve as the natural benchmark for the polish language understanding, but it lacks the common interface for all tasks, making it difficult and time-consuming to use. we include one of the poleval tasks into the klej benchmark. recently this paper. it contains two sentiment analysis tasks, topic classification and a polish translation of the sick dataset @xcite . similarly to their work, we use the same sentiment analysis dataset, but transform it into a more difficult task. we also use the analogous dataset to sick but created from scratch for the polish language. finally, we considered the topic classification task to be too easy to include into the proposed benchmark. overall, klej benchmark consists of nine tasks. they are more diverse, cover a wider range of objectives and evaluate not only single sentences but also whole paragraphs. ## baselines in this section, we present an overview of several baseline models, which we evaluated using the klej benchmark. we divide these models into three main groups: (1) the lstm-based (hochreiter and schmidhuber, 1997) models using pretrained word embeddings, (2) models based on transformer architecture and (3) bert model trained on polish corpora. we also include the simple baseline by sampling targets from a training set. ## evaluation we first compare models based on their average performance. even though it is not the definite metric to compare models, especially as not all tasks are equally difficult, it gives a general notion of the model performance across all tasks. in comparison to baselines based on the lstm architecture, transformer-based models clearly show superior performance. the only exception is elmo, which achieves competitive results on many tasks. on two of them, the fine-tuned elmo model achieves the best score. in general, the evaluation shows major shortcomings of multilingual pretrained bert models for polish, and possibly other low resource languages. overall, every lstm-based baseline is still on average worse than any of the tested transformer-based models. the klej benchmark was designed to require additional knowledge to promote general language understanding models. as expected, we observe significant increases of models quality when using pretrained word embeddings. the vanilla lstm model achieves the average score of 63.0 while supplying it with the fasttext embeddings boosts performance to 67.7. usage of more recent, contextualized embeddings (elmo) increases the score to 76.6. focusing on fewer languages seems to result in a better model. the slavic-bert has higher scores than multi-bert on seven out of nine tasks. however, without a detailed ablation study, it is difficult to infer the main reason resulting in better performance. it can also be attributed to a better tokenizer, additional russian news corpus or longer training (the slavic-bert was initialized with multi-bert weights). the training corpus seems to play an important role in the performance of a downstream task. both herbert and elmo models were trained mainly on web crawled texts and they excel at tasks from an online domain (cbd, polemo-in, polemo-out, and ar). on the other hand, the other transformerbased models are superior on the czy wiesz? task. it can be related to the fact that it is a wikipedia-based question-answering task and the aforementioned models were trained mainly on wikipedia corpus. interestingly, the slavic-bert, which was additionally trained on russian news corpus, has a lower score on the czy wiesz? task than multi-bert and xlm-17. herbert achieves highly competitive results compared to the other transformer-based models. it has the best performance on average and achieves state-of-the-art results on three tasks, polemo-in, polemo-out and cbd. moreover, herbert has the smallest performance gap between polemo-in and polemo-out, which suggests better generalization across domains. compared to the other transformer-based models it performs poorly on czy wiesz? and psc tasks. the klej benchmark proved to be challenging and diverse. there is no clear winner among evaluated models; different models perform better at different tasks. it suggests that the klej benchmark is far from being solved, and it can be used to evaluate and compare future models. ## conclusion we introduce the klej benchmark, a comprehensive set of evaluation tasks for the polish language understanding. its goal is to drive the development of better nlu models, so careful selection of tasks was crucial. we mainly focused on a variety of text genres, objectives, text lengths, and difficulties, which allows us to assess the models across different axes. as a result, klej benchmark proves to be both challenging and diverse, as there is no single model that outperforms others on all tasks. we find it equally important to provide a common evaluation interface for all the tasks. for that purpose, many existing resources had to be adapted, either automatically (nkjp-ner, psc) or manually (czy wiesz?), to make it easier to use. it's worth mentioning that the main weakness of creating such benchmarks is focusing only on the model performance and not the model efficiency, e.g. in terms of training data, speed or a number of parameters. it seems reasonable to derive additional benchmarks by requiring a given level of efficiency from participating models. we leave it as future work. we also present herbert, a transformer-based model trained specifically for polish and compare it with other lstm-and transformer-based models. we find that it is the best on average and achieves highest scores on three tasks. we plan to continue the work on herbert and use the klej benchmark to guide its development.