A Tokenization System for the K urdish Language (2020)

Abstract: Tokenization is one of the essential and fundamental tasks in natural language processing. Despite the recent advances in applying unsupervised statistical methods for this task, every language with its writing system and orthography represents specific challenges that should be addressed individually. In this paper, as a preliminary study of its kind, we propose an approach for the tokenization of the Sorani and Kurmanji dialects of Kurdish using a lexicon and a morphological analyzer. We demonstrate how the morphological complexity of the language along with the lack of a unified orthography can be efficiently addressed in tokenization. We also develop an annotated dataset for which our approach outperforms the performance of unsupervised methods.

## introduction a text, as the input of text processing applications, is composed of a string of characters and is interpreted based on the way it is segmented. words and sentences are two segments in a text which carry meaning at different levels. although the boundaries of words and sentences are specified to some extent in some scripts, e.g. by using whitespaces and punctuation marks, finding such boundaries is a non-trivial task @xcite . for instance, in scripts where spaces are not widely used or the scriptio continua style is used, such as japanese, chinese and classical latin, or languages where words are concatenated to create compound forms as in german, word boundary may not be explicitly specified. in natural language processing (nlp), tokenization generally refers to the task of finding segment boundaries in a text. more specifically, retrieving the boundary of words and sentences are respectively known as word tokenization and sentence tokenization. given a string of characters, a tokenization system, also known as lexical analyzer or tokenizer, splits the input into tokens, i.e. words or sentences @xcite . tokenization is one of the most important and fundamental language processing tasks with many applications, such as part-of-speech tagging and machine translation @xcite . given the recent advances in nlp and artificial intelligence, tokenization is considered a solved problem and has been efficiently addressed for many languages @xcite . although methodologies and approaches in tokenization of one language might be applicable to and beneficial for another language, linguistic and orthographic issues can make tokenization a language-specific problem @xcite . for instance, although whitespaces are generally used in arabic-based scripts, such as urdu, persian and arabic, the joining and non-joining characteristics of graphemes create further complexity in tokenizing compound words, i.e. words consisted of more than one word, and various morphemes, such as affixes and clitics @xcite . in this paper, we carry out a preliminary study on the task of tokenization for the kurdish language with a particular focus on two of the kurdish dialects, i.e. kurmanji and sorani, for which latin and arabicbased scripts are respectively used. we show how the kurdish scripts and their lack of standardized orthographies create variations in writing words, especially compound forms. to address this task, we develop a tokenization system using a basic morphological analyzer and a lexicon and demonstrate that it outperforms regular expression based and unsupervised neural methods. ## related work the notion of word is one of the most basic concepts in various fields and therefore can be defined in different ways. generally speaking, a word refers to a building block of a sentence. however, from a morphological point of view, a word, which is also known as word-token, is defined based on its form and meaning. if a word carries a concrete meaning, it is defined as a word-form such as drives, driving, drove, while a word with an abstract meaning is known as a lexeme or lexical item, e.g. drive @xcite . lexemes are also distinguished by their function as headwords in dictionaries. it is worth mentioning that in addition to lexemes, lemmas are used to refer to the canonical forms of the lexemes. for instance, although ‫خواردن‬ (xwardin) eat and ‫خواردنەوە‬ (xwardinewe) drink are two distinct lexemes in kurdish, they both have one lemma and that is xwardin. the task of retrieving word lemmata is called lemmatization and is of importance in nlp. analogous to the distinction between word-forms and lexemes in morphology, corpus linguistics distinguishes a word as token and type based on their distinctness in a text. while a token can frequently occur, a type is considered the unique form of the token which can also be used as a dictionary entry @xcite . additionally, @xcite describe tokens based on the lexicographic information, the context such as sub-languages and terminologies and, the applications. the application-based definition suggests that word boundary depends on the underlying application for which the tokenization task is required. for instance, the performance of tokenization methods have been studied in various tasks, such as statistical and neural machine translation @xcite ), text classification @xcite and named-entity recognition @xcite . being a basic task in information retrieval, text processing and nlp, the task of tokenization has been widely previously studied for many languages. a wide range of techniques are used for the task, particularly rule-based @xcite @xcite , statistical @xcite and more recently, neural networks @xcite . the latter are particularly beneficial to alleviate open vocabulary problems independent of the language. moreover, given the importance of tokenization in downstream applications, tokenization tools are usually also provided within nlp frameworks such as moses @xcite and opennmt @xcite for machine translation. as the earliest work that addresses tokenization for kurdish language, @xcite discusses some of the issues in word boundary in the arabic-based scripts. although kurdish tokenization has been partially addressed in the context of other tasks, such as text classification @xcite , machine translation @xcite and syntactic analysis @xcite , no previous study is found to explicitly focus on kurdish tokenization. for kurdish as a less-resourced language, we believe that the current study will pave the way for further developments in kurdish language processing. ## an overview of the kurdish language kurdish is a less-resourced indo-european language spoken by 20-30 million speakers in the kurdish regions of iran, iraq, turkey and syria @xcite . there are various points of view regarding the classification of the dialects of kurdish @xcite . however, northern kurdish, also known as kurmanji, central kurdish, also known as sorani and, southern kurdish are less controversially accepted as the kurdish dialects @xcite . historically, many alphabets have been used for writing kurdish among which the latin-based and arabic-based scripts are still widely in use @xcite . although the standardization of the language, in the written and spoken forms, have been a matter of discussion in academia and also among kurdish people, there is no consensus regarding what is meant by a standard writing system or orthography for kurdish @xcite . due to the lack of standardization, different scripts may be used for writing in various dialects. regarding the popularity of the scripts, the arabic-based alphabet is widely used for sorani and southern kurdish while the latin-based is used for kurmanji. due to southern kurdish being an under-documented dialect @xcite , we only focus on the sorani and kurmanji dialects in this study. similar to the arabic-based scripts of persian and urdu, the arabic-based script of kurdish has a zero width non joiner (zwnj, u+200c) character which enables joining characters be written in their nonjoining grapheme. for instance, the character ‫ڵ‬ in ‫هەڵـگرتن‬ (hełgirtin) 'to lift' appears as ‫گرتن‬ ‫هەڵ‬ when followed by a zwnj. moreover, to further add to the length of the joining between graphemes, a dualjoining grapheme known as tatweel or kashida (u+0640) is used. this grapheme does not represent any phoneme but only elongates characters for justification and alignment of the text. ## word boundary in kurdish in both the latin-based and arabic-based scripts of kurdish, whitespaces are used for delimiting word boundaries. in addition, the zwnj in the arabic-based script is also commonly used for separating words, particularly verbs that are consisted of more than one word. having said that, none of these delimiters are deterministic for word boundary in kurdish @xcite due to the issues addressed in this section. ## approach as a preliminary study, we focus on the application of a lexicon of lemmata and morphological analysis for tokenization of kurdish texts. moreover, we follow the common practices in tokenization, such as detecting digits, dates, urls and punctuation marks as distinct tokens. this sub-task is called "normalization prior to tokenization" @xcite . given the complexity of detecting word boundaries in kurdish, particularly in the arabic-based script of the sorani dialect, we carry out the task of tokenization based on the syntactic property of the words. in other words, if a sequence of characters, whether delimited by spaces or not, can have a syntactic role, we consider them as a distinct token. therefore, tokens in kurdish can be words such as bira 'brother', compound words such as germ kirin 'to heat (something) up', clitics such as ‫تان‬ =tan (2.pl pronominal endoclitic) and affixes such as ‫ەکان‬ -ekan (definite plural marker). the following shows an example of how the input sentences in sorani and kurmanji are tokenized in our system: ## conclusion and future work in this paper, we presented a tokenization system for the sorani and kurmanji dialects of kurdish. having a complex morphology and various compound form constructions, kurdish represents non-trivial challenges to the tokenization task. using a lexicon and a morphological analyzer, our system outperforms unsupervised neural methods and can also be used to detect compound forms efficiently. one limitation of the current study is the tokenization of compound verbs. in addition to tense, aspect, person and mood, verbs are inflected according to the patient, i.e. object of transitive verbs, and can be accompanied by other affixes such as ‫ەوە‬ (-ewe) to indicate repetition and ‫یش/ش‬ (=îş/=ş) to indicate emphasis. some clitics appear within the root of the verb, therefore called endoclitic, and create more complex forms. if in the tokenization task, such parts of the verbs are split into tokens, the compound verb is also split into its composing parts instead of being tokenized as one. given the relatedness of lemmatization to the current task, we believe that extending the current study can be beneficial to create a lemmatization system for kurdish as well. moreover, enriching the lexicons, particularly by including further compound form constructions, should also be considered in the future by integrating further collaboratively-curated open resources. a @xmath0:‬ ‫دەگەڕێتەوە‬ ‫ئابووریانەدا‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانى‬ ‫دواکەوتنى‬ ‫بێنن.‬ ‫بەکارى‬ ‫بەرهەمهێنەکان‬ ‫تا‬ ‫هاوردە‬ ‫تازەى‬ ‫تەکنیکیی‬ ‫هۆیەکانى‬ </@xmath1:</@xmath2.</@xmath3, behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin. </@xmath4,</@xmath5.</@xmath6.2: an example of the annotated tokens of two different sentences in sorani (left) and kurmanji (right) in the text corpus format reference . ‫بێنن‬ ‫ی‬ ‫کار‬ ‫بە‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫ی‬ ‫تازە‬ ‫ی‬ ‫تەکنیکی‬ ‫ی‬ ‫یەکان‬ ‫هۆ‬ ‫ی‬ ‫نەبوون‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫دا‬ ‫انە‬ ‫ئابووری‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫ی‬ ‫ەکان‬ ‫شێواز‬ ‫ی‬ ‫دواکەوتن‬ our system . ‫ن‬ ‫بێن‬ ‫بەکاری‬ ‫بەرهەمهێنەکان‬ ‫تا‬ ‫هاوردە‬ ‫ی‬ ‫تازە‬ ‫تەکنیکیی‬ ‫ی‬ ‫یەکان‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫یانەدا‬ ‫ئابوور‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫ی‬ ‫ەکان‬ ‫شێواز‬ ‫ی‬ ‫دواکەوتن‬ bpe . ‫بێنن‬ ‫بەکاری‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫یی‬ ‫تەکنیک‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫انەدا‬ ‫ئابووری‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ unigram . ‫بێنن‬ ‫بەکاری‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫ی‬ ‫تەکنیکی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫انەدا‬ ‫ئابووری‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ word ‫بێنن.‬ ‫بەکاری‬ ‫بەرهەمهێنەکان‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫تەکنیکیی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ ‫بۆ:‬ ‫دەگەڕێتەوە‬ ‫یانەدا‬ ‫ئابوور‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ wordpiece . ‫بێنن‬ ‫بەکاری‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫ی‬ ‫تەکنیکی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ wordpunct . ‫بێنن‬ ‫بەکاری‬ ‫بەرهەمهێنەکان‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫تەکنیکیی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫یانەدا‬ ‫ئابوور‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ reference di evê peywendî ya telefonî de , behsa peywendî yên navber a baxdad ê û parîs di hemû biwaran de hatîye kirin . our system di ev ê peywend îya telefonî de , behs a peywendî yên navbera baxdadê û parîs di hemû biwaran de hatîye kirin . bpe di evê peywendîya telefonî de , behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin . unigram di evê peywendîya telefonî de , behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin . word di evê peywendîya telefonî de, behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin. wordpiece di evê peywendîya telefonî de , behsa peywendîyên navbera û di hemû biwaran de hatîye kirin . wordpunct di evê peywendîya telefonî de , behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin . (a) sorani reference . ‫بێنن‬ ‫ی‬ ‫کار‬ ‫بە‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫ی‬ ‫تازە‬ ‫ی‬ ‫تەکنیکی‬ ‫ی‬ ‫یەکان‬ ‫هۆ‬ ‫ی‬ ‫نەبوون‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫دا‬ ‫انە‬ ‫ئابووری‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫ی‬ ‫ەکان‬ ‫شێواز‬ ‫ی‬ ‫دواکەوتن‬ our system . ‫ن‬ ‫بێن‬ ‫بەکاری‬ ‫بەرهەمهێنەکان‬ ‫تا‬ ‫هاوردە‬ ‫ی‬ ‫تازە‬ ‫تەکنیکیی‬ ‫ی‬ ‫یەکان‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫یانەدا‬ ‫ئابوور‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫ی‬ ‫ەکان‬ ‫شێواز‬ ‫ی‬ ‫دواکەوتن‬ bpe . ‫بێنن‬ ‫بەکاری‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫یی‬ ‫تەکنیک‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫انەدا‬ ‫ئابووری‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ unigram . ‫بێنن‬ ‫بەکاری‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫ی‬ ‫تەکنیکی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫انەدا‬ ‫ئابووری‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ word ‫بێنن.‬ ‫بەکاری‬ ‫بەرهەمهێنەکان‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫تەکنیکیی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ ‫بۆ:‬ ‫دەگەڕێتەوە‬ ‫یانەدا‬ ‫ئابوور‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ wordpiece . ‫بێنن‬ ‫بەکاری‬ ‫ەکان‬ ‫بەرهەمهێن‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫ی‬ ‫تەکنیکی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ wordpunct . ‫بێنن‬ ‫بەکاری‬ ‫بەرهەمهێنەکان‬ ‫تا‬ ‫هاوردە‬ ‫تازەی‬ ‫تەکنیکیی‬ ‫یەکانی‬ ‫هۆ‬ ‫نەبوونی‬ : ‫بۆ‬ ‫دەگەڕێتەوە‬ ‫یانەدا‬ ‫ئابوور‬ ‫لەم‬ ‫بەرهەمهێنان‬ ‫شێوازەکانی‬ ‫دواکەوتنی‬ reference di evê peywendî ya telefonî de , behsa peywendî yên navber a baxdad ê û parîs di hemû biwaran de hatîye kirin . our system di ev ê peywend îya telefonî de , behs a peywendî yên navbera baxdadê û parîs di hemû biwaran de hatîye kirin . bpe di evê peywendîya telefonî de , behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin . unigram di evê peywendîya telefonî de , behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin . word di evê peywendîya telefonî de, behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin. wordpiece di evê peywendîya telefonî de , behsa peywendîyên navbera û di hemû biwaran de hatîye kirin . wordpunct di evê peywendîya telefonî de , behsa peywendîyên navbera baxdadê û parîs di hemû biwaran de hatîye kirin .