Mukayese: T urkish NLP Strikes Back (2022)

Abstract: Having sufficient resources for language X lifts it from the under-resourced languages class, but not necessarily from the under-researched class. In this paper, we address the problem of the absence of organized benchmarks in the Turkish language. We demonstrate that languages such as Turkish are left behind the state-of-the-art in NLP applications. As a solution, we present Mukayese, a set of NLP benchmarks for the Turkish language that contains several NLP tasks. We work on one or more datasets for each benchmark and present two or more baselines. Moreover, we present four new benchmarking datasets in Turkish for language modeling, sentence segmentation, and spell checking. All datasets and baselines are available under: https://github.com/alisafaya/mukayese

## introduction although some human languages, such as turkish, are not classified as under-resourced languages, only a few research communities are working on them @xcite . as a result, they are left behind in developing state-of-the-art systems due to the lack of organized benchmarks and baselines. in this study, we aim to address this gap for the turkish language with mukayese (turkish word for "comparison/benchmarking"), an extensive set of datasets and benchmarks for several turkish nlp tasks. we survey several tasks in turkish nlp and observe an absence of organized benchmarks and research. we demonstrate how the lack of benchmarks affects under-studied languages such as turkish and how it can keep the state of research behind the state-of-the-art of nlp. we accomplish this by presenting state-of-the-art baselines that outperform previous work significantly. we believe that mukayese will set a basis for boosting corresponding author: asafaya19@ku.edu.tr nlp research for turkish. therefore, we encourage research communities from other under-studied languages to follow a similar path. in our work on mukayese, we study seven nlp tasks in the turkish language. we evaluate available datasets in turkish for these tasks and describe the process of creating four new datasets for tasks that do not have accessible datasets. furthermore, in addition to evaluating existing methods, we provide at least two baseline models/methods per task. more details are enlisted in our overall contribution to turkish nlp can be summarized as the following: (a) set of seven organized benchmarks for nlp. (b) four new datasets in turkish for language modeling, sentence segmentation, as well as spellchecking and correction. moreover, mukayese is a part of the turkish data depository (tdd) project foot_0 . the main goal of tdd is collecting and organizing turkish natural language processing nlp resources and providing a research basis for turkish nlp. the rest of the paper is organized as follows: we review similar efforts in section 2. then, we advert to benchmarks and nlp in section 3. next, we give a background on the turkish language resources in section 4. we explain the approach we follow for each task in section 5, and we provide dataset details, evaluation results, and explain the baselines for each task in section 6. ## related work in this section, we discuss efforts similar to ours. we give an overview of efforts on building multilingual benchmarks, and we mention some of the monolingual benchmarks as well. task datasets metrics baselines language modeling -trnews-64 -trwiki-67 -bits-per-char -perplexity -adapt. trans. -sha-rnn machine translation -wmt-16 -must-c -bleu -convs2s -transformer -mbart50 named-entity recognition -wikiann -milliyet-ner -conll f1 -bilstm-crf -bert -bert-crf sentence segmentation -trseg-41 -segment f1-score -spacy -punkt -ersatz spellchecking & correction -trspell-10 -f1-score -accuracy -zemberek -hunspell summarization -mlsum -rouge-l -meteor -transformer -mbart50 -mt5 text classification -offenseval -news-cat -f1-score -bilstm -cnn text -bert there exist various endeavors at building multilingual benchmarks. one example for this is xtreme @xcite , a multilingual benchmark containing 40 different languages and nine different tasks. these tasks include classification, named entity recognition (ner), and question answering (qa). however, most of these datasets are created by translating existing english datasets manually or automatically. therefore, they have limitations and cannot be utilized to build a research basis in a specific language. there are several benchmarks for nlp tasks for both low-resource and high-resource languages when it comes to monolingual benchmarks. @xcite proposes a benchmark for two lowresourced african languages on neural machine translation (nmt), namely somali and swahili. similarly, there are efforts to build benchmarks for high-resource but under-studied languages such as alue benchmark for arabic @xcite , and klej benchmark for polish @xcite . both benchmarks focus on natural language understanding (nlu). most of these benchmarks have public leaderboards to disseminate studies in nlp for their languages. while most of previous benchmarks focus on one task such as nlu or nmt, mukayese covers a comprehensive set of nlp tasks with seven different benchmarks on a variety of tasks. the reasoning behind this is to catalyze the research of turkish nlp, and encourage research in all nlp applications. ## benchmarks and nlp following the research on nlp over the years, we can observe how datasets and benchmarks are fundamental. in this section, we discuss the importance of benchmarks for the progress of nlp. benchmarks are very essential for measuring the progress of nlp. for instance, the squad dataset @xcite is used to examine the progress of english question answering, and glue @xcite , superglue @xcite provide benchmarks for english language understanding. such progress has been enabled by the existence of benchmarks, which allowed for fair and meaningful comparison, and showed if there is room for improvement. in addition, organized benchmarks and datasets enable the research community to make progress with minimal amount of domain knowledge. this is especially important when it comes to languages with fewer speakers, and research communities are more likely to contribute when such organized tasks are presented @xcite . thus, this is essential if we want to include other communities in the development of under-resourced and under-studied languages. however, there are several things to keep in mind when dealing with benchmarks and leaderboards. such leaderboards should be created transparently, and the results need to be evaluated with all factors taken into account. some of these factors are model size, energy efficiency, and generalization (linzen, 2020). otherwise, we can run into the risk of these leaderboards resulting in inefficient and non-robust models. ethayarajh and jurafsky (2020) describe a few limitations of current leaderboards and suggest practices to mitigate these limitations. we take these practices into account and present the benchmarks of mukayese. we provide more details about our methodology in section 5. ## background on turkish the turkish language has distinctive characteristics compared to well-studied languages in the literature, such as english, spanish, and german. due to its agglutinative morphological nature, turkish nouns can produce more than 100 inflected forms, while verbs can produce even more (oflazer and saraçlar, 2018). therefore benchmarks designed for english are not necessarily applicable for turkish. figure 1 : an example of a word constituting multiple inflectional groups @xcite . unlike many other languages, a single word can constitute multiple different inflectional groups. an example is displayed in figure 1 . we provide more details on the features of the turkish language in appendix a. there are several attempts at constructing comprehensive sets of resources and evaluation for turkish. @xcite introduced a morphological parser, and a morphological disambiguator accompanied by a web corpus. more recently, @xcite proposed an online turkish nlp pipeline, which includes normalization, tokenization, morphological analysis, ner, and syntactic parsing. however, among previously proposed methods and datasets, none are presented in a comparative way. this study aims to make a comprehensive inventory of different tools, corpora, and evaluation measures for the turkish language. such inventory may be used for researchers and practitioners who are looking for tools and datasets for turkish nlp. ## methodology in mukayese, we focus on under-researched tasks of nlp in the turkish language. after defining the task and assessing its importance, we construct the following three key elements for each benchmark: datasets are the first element to consider when it comes to a benchmark. we define the minimum requirements of a benchmark dataset as follows: (i) accessible with reasonable size. (ii) satisfactory quality. (iii) a publicly shareable, compliant applicable regulations (gdpr licensing). we chose the dataset sizes in a task-specific manner, unless used in a few-shot setting, benchmarks with small datasets will lack generalizability, and models trained on them might suffer from overfitting. on the other hand, training models on enormous datasets might be costly and inefficient @xcite . another feature to assess is the quality of the dataset. a manually annotated dataset with a low interannotator agreement (iaa) rate is not suitable for benchmarking. moreover, to build a generalizable benchmark, we need to consider using a dataset representing the general domain. for instance, sentence segmentation methods of editorial texts do not work on user-generated content such as social media posts, as we show in subsection 6.4. metrics are the second element of benchmarks. we need to decide on one or more evaluation metrics to evaluate and compare methodologies. in order to do so, we have to answer the following questions: (a) does this metric measure what our task aims to do? (b) how well does it correlate with human judgment? (c) are there any issues/bugs to consider in these metrics? (for example, using accuracy to measure performance on an unbalanced set does not give a representative idea of model performance). baselines are the final element of benchmarking. in order to characterize the performance characteristics of different methodologies, it is bet-ter to diversify our baselines as much as possible. for instance, we can compare pretrained vs. non-pretrained approaches, rule-based systems vs. trained systems, or unsupervised vs. supervised models. ## tasks we provide benchmarks in the form of datasets, metrics, baselines triplets for each of the following nlp tasks: ## conclusion we believe that while some languages such as turkish do not fall under the definition of underresourced languages, they attract relatively little research interest as a result of the lack of organized benchmarks and baselines. to address this problem, we presented mukayese, a comprehensive set of benchmarks along with corresponding baselines for seven different tasks: language modeling, machine translation, named entity recognition, sentence segmentation, spell checking and correction, summarization, and text classification, as well as four new benchmarking datasets in turkish for language modeling, sentence segmentation, and spell checking and correction. for future work, the same methodology can be followed to include more tasks such as dependency parsing, morphological analysis, coreference resolution. we hope that mukayese encourages more researchers to get involved in the development of turkish nlp, and it sets an example and leads to an increase in efforts on under-researched languages.