TURNA : A T urkish Encoder-Decoder Language Model for Enhanced Understanding and Generation (2024)

Abstract: The recent advances in natural language processing have predominantly favored well-resourced English-centric models, resulting in a significant gap with low-resource languages. In this work, we introduce TURNA, a language model developed for the low-resource language Turkish and is capable of both natural language understanding and generation tasks.TURNA is pretrained with an encoder-decoder architecture based on the unified framework UL2 with a diverse corpus that we specifically curated for this purpose. We evaluated TURNA with three generation tasks and five understanding tasks for Turkish. The results show that TURNA outperforms several multilingual models in both understanding and generation tasks and competes with monolingual Turkish models in understanding tasks.

## introduction recent advances in natural language processing (nlp) have predominantly resulted in englishcentric models @xcite @xcite @xcite , which have benefited from the vast amount of training data gathered from an abundance of english resources present on the web. the use of these models fuels further research yielding state-of-the-art results across various tasks @xcite . on the other hand, low-resource languages suffer from lack of data and limited computational resources, leading to a significant gap between models trained on well-resourced versus low-resource languages. several multilingual models have been proposed to bridge this gap @xcite @xcite . while such models address some tasks, they often fall short in those requiring deep understanding of language-specific nuances, such as dependency parsing and named entity recognition @xcite @xcite . thus, multilingual models lag behind monolingual models of the same scale @xcite . recently, pretrained language models built upon transformers @xcite have dominated nlp. these models vary in terms of their architectures and objectives. the architectures are commonly classified as encoder-only, decoderonly, or encoder-decoder models. encoder-only models are typically trained with denoising objectives and focus on natural language understanding (nlu) tasks @xcite . decoder-only models are designed for natural language generation (nlg) tasks with causal language modeling @xcite @xcite . finally, encoderdecoder models deal with nlp tasks that require both nlu and nlg @xcite . towards this end, the text-to-text transformer (t5) @xcite ) employs an encoder-decoder architecture that is pretrained with a denoising objective known as span corruption. the unifying language learning (ul2) framework @xcite proposes the mixture-of-denoisers (mod) pretraining objective which combines several denoising objectives. by coupling the mod objective with an encoder-decoder architecture, state-of-the-art results are achieved in a range of nlp tasks. for the turkish language, low-resource in pretrained language models, encoder-only models exist @xcite , however, there is a need for large-scale pretrained models that can perform both nlu and nlg. this work aims to develop such a model for turkish that performs well across a variety of tasks of both types. towards this end, we first compile a diverse turkish corpus for pretraining purposes that includes web data, scientific articles, graduate theses, books, creative writing, and parliamentary speech transcriptions. subsequently, we pretrain turna on this corpus with an encoder-decoder architecture based on the ul2 framework @xcite . we report performance results for various generation and understanding tasks for turkish. our contributions are summarized as follows: ## related work multilingual language models address multiple languages including those considered low-resource languages. turkish, considered as a low-resource language in this respect, is moderately represented in multilingual models such as mbert @xcite , xlm-r @xcite , mbart @xcite , mt5 @xcite , xglm @xcite , mgpt @xcite , and mdeberta @xcite . however, these models are not up to par in language-specific tasks when compared with monolingual models 1 at https://huggingface.co (kept anonymous during the review process) 2 https://anonymous.4open.science/r/ turkish-academic-text-harvest 3 https://anonymous.4open.science/r/turna-6753 4 https://anonymous.4open.science/r/ turkish-lm-tuner-37af a series of bert models for turkish known as berturk have been trained @xcite including several variations of bert @xcite , distilbert @xcite , convberturk @xcite , and elec-tra @xcite . most of these models were trained on a 35gb corpus consisting of 4.4b tokens drawn from the turkish oscar corpus @xcite , a wikipedia dump foot_0 , and various opus corpora @xcite . some models, like convberturk and electra, were also trained on the turkish portion of the mc4 @xcite corpus -a certain cleaned version of the public web crawl data of common crawl. these models have been evaluated on various downstream tasks (such as part-of-speech tagging, named entity recognition, and question answering) where they generally outperform their multilingual counterparts mbert @xcite and xlm-r @xcite . these models are encoder-only, meaning mostly suitable for nlu tasks. presently, there is a clear need for turkish models that also excel in nlg tasks, which require a decoder component. this work focuses on an encoder-decoder model to address both types of tasks. encoder-only models are typically trained with span corruption with various lengths and frequencies @xcite . conversely, decoder-only models typically employ causal language modeling @xcite @xcite and are designed for generation tasks. a popular model built on the transformer architecture, the text-to-text transformer (t5) @xcite , proposed a unified framework that treats all nlp tasks as conversions from some text to another. it employs an encoder-decoder architecture that is pretrained with a denoising objective. this model has demonstrated success over numerous tasks and is reported to scale well. unilm @xcite is also an encoderdecoder model, but pretrained using unidirectional, bidirectional, and sequence-to-sequence language modeling. this can be seen as a combination of causal and denoising objectives. the unifying language learning framework (ul2) that is based on a pretraining objective called mixture-of-denoisers (mod) was proposed, which combines span corruption objectives with varying mixture parameters @xcite . this study found that among the decoder-only and encoder-decoder models, both of which are trained using the mod objectives, the encoder-decoder models performed better. by using the mod objective and moderately scaling up the model, they achieved state-of-the-art performance on a diverse set of nlu and nlg tasks. ## data we compiled a diverse turkish monolingual dataset to pretrain our model. our dataset comprises of a web corpus, scientific corpora consisting of turkish articles and graduate theses, turkish books, a corpus of creative writing assignments from bilkent university, and transcriptions of parliamentary debates. the details of each corpus are explained in the following subsections, and the training corpora statistics are summarized in during data splitting, we ensured that the validation set of each dataset contained a minimum of 100k tokens. the train-validation splits are reported under each subsection. ## conclusion in this study, we introduced turna, a new turkish language model that adopts an encoder-decoder architecture following the ul2 framework. this model was pretrained on a broad corpus covering web data, scientific articles, theses, books, creative writing, and parliament corpora. our comprehensive evaluations across three generation and five understanding tasks on 13 different datasets showed that turna outperforms existing multilingual models, mt5 and mbart, and performs better than or on par with the turkish encoder-only model berturk. to encourage further research and facilitate benchmarking in turkish nlp, these mod-els and the entire source code for data collection, filtering, model training, and fine-tuning are made publicly accessible.