KLPT – K urdish Language Processing Toolkit (2020)

Abstract: Despite the recent advances in applying language-independent approaches to various natural language processing tasks thanks to artificial intelligence, some language-specific tools are still essential to process a language in a viable manner. Kurdish language is a less-resourced language with a remarkable diversity in dialects and scripts and lacks basic language processing tools. To address this issue, we introduce a language processing toolkit to handle such a diversity in an efficient way. Our toolkit is composed of fundamental components such as text preprocessing, stemming, tokenization, lemmatization and transliteration and is able to get further extended by future developers. The project is publicly available.

## introduction language technology is an increasingly important field in our information era which is dependent on our knowledge of the human language and computational methods to process it. unlike the latter which undergoes constant progress with new methods and more efficient techniques being invented, the processability of human languages does not evolve with the same pace. this is particularly the case of languages with scarce resources and limited grammars, also known as lessresourced languages. various natural language processing (nlp) tasks are of pipeline architecture; that is, to address a specific task, a few other language processing tasks may be initially required @xcite . with the current advances in the open-source movements, more researchers and industrial developers are encouraged to share their knowledge in an open-source manner, accessible under certain conditions @xcite . therefore, the development of underlying tasks in nlp for a specific language will potentially pave the way for further contributions to the field, by either improving the current tools or further progress in new tasks. for instance, tokenization as a fundamental task is widely required in many other applications such as part-of-speech tagging, machine translation and syntactic analysis. once addressed, future researchers can build upon it for more advanced tasks or eventually improve it. despite a plethora of performant tools and specific frameworks for nlp, such as nltk @xcite , stanza @xcite , teanga @xcite and spacy foot_1 , the progress with respect to less-resourced languages is often hindered by not only the lack of basic tools and resources but also the accessibility of the previous studies under an open-source licence. this is particularly the case of kurdish, a less-resourced indo-european language that is the focus of the current paper. as an example, although the task of spell-checking and stemming for kurdish have been addressed by many previous studies, @xcite @xcite @xcite to mention but a few, none of them provides an implementation of their tool under any licence. on the other hand, some previous studies use specific frameworks that are hardly integrable and inter-operable. for instance, @xcite and @xcite describe their efforts in developing a large-scale morphological lexicon and a part-of-speech tagger for kurdish within the alexina framework under the lgpl-lr licence. despite the valuable impact of this study in the field, for example in @xcite and @xcite , the tool does not as such, projects such as @xcite and ( ahmadi and hassani, 2020a ) tackle the very same topic from scratch. language-specific toolkits have been previously designed for various languages, such as icenlp for icelandic @xcite , vncorenlp for vietnamese @xcite , fudannlp for chinese @xcite , psi-toolkit for polish @xcite and parsipardaz for persian @xcite . in the same vein, in order to facilitate the basic language processing tasks for kurdish in an organized and methodical way and aware of the increasing importance of open-source and inter-operable tools for building more efficient systems and get further advanced in the field, we present klpt-the kurdish language processing toolkit. this toolkit is developed in python and is composed of core modules and is extendable by future developers. ## kurdish language kurdish belongs to the northwestern branch of the iranian languages within the indo-european language family which is spoken by 20-30 million speakers in the kurdish regions of turkey, iraq, iran and syria and also, among the kurdish diaspora around the world @xcite . the division of kurdish into northern kurdish (or kurmanji), central kurdish (or sorani), southern kurdish and laki, respectively with kmr, ckb, sdh and lki iso 639-3 language codes, has been widely studied previously @xcite . based on the structural differences between these, some scholars believe that they are distinct languages and therefore, refer to them as kurdish languages @xcite . on the other hand, it is also commonly believed by both scholars and kurdish people that those are in fact different dialects of the kurdish language @xcite . in this study, we remain with this theory and refer to them as kurdish dialects. it is worth mentioning that despite the linguistic similarities of zazaki, also known as dimlî, and gorani languages and the popular belief that they are dialects of kurdish, studies show that they belong to the zaza-gorani language family which is independent from the kurdish language @xcite @xcite . kurdish has been historically written in various scripts, namely cyrillic, armenian, latin and arabic among which the latter two are still widely in use. efforts in standardization of the kurdish alphabets and orthographies have not succeeded to be globally followed by all kurdish speakers in all regions @xcite @xcite . as such, the kurmanji dialect is mostly written in the latin-based script while the sorani, southern kurdish and laki are mostly written in the arabic-based script. that, not only scatters readers and speakers to communicate together, but also creates further challenges in processing the language (esmaili, 2012; ahmadi, 2019). kurdish language is a highly inflectional language, particularly due to a high number of affixes and clitics @xcite . regarding nouns, although sorani does not have gender or grammatical cases, it has a full article marking system for definite, indefinite and demonstrative in singular and plural forms @xcite . on the other hand, kurmanji has a fewer number of article markers for feminine and masculine genders @xcite . with respect to the verbs, kurdish has a few number of around 300 single-word verbs @xcite , e.g. kirdin/kirin "to do", which are inflected based on person (1,2,3, sg, pl), tense (past, present, future), aspect (indefinite, perfect, progressive, imperfective) and mood (indicative, subjunctive, conditional). unlike kurmanji, sorani kurdish does not have future tense and uses adverbs for this purpose. however, kurdish extensively takes use of compound constructions for creating new verb forms, particularly with (noun + verb), (adjective + verb) and (preposition + verb) forms @xcite . for instance, siław 'hi (n)', pîroz 'holy' (adj) and heł (verbal particle denoting 'up') with the single-word verb kirdin can respectively form compound verbs siław kirdin "to greet", pîroz kirdin "to congratulate" and heł kirdin "to turn on". the stringing characteristic of the arabicbased script of kurdish further adds to this morphological complexity in such a way that several word forms may be concatenated together (ahmadi, 2020b). regarding syntax, kurdish has a subject-object-verb word order and is a null-subject (or pro-drop) language. the presence of grammatical markers for nominative and oblique cases varies within dialects and subdialects. for instance, in the sorani subdialects of sulaymaniyah and erbil, respectively categorized as southern sorani and northern sorani by @xcite , the oblique case is marked differently. another particularity of the kurdish language is its morphosyntactic alignment in the past tense of transitive verbs. in such tenses, an ergative-absolutive alignment occurs where the subject of intransitive verbs behaves like the patient of the transitive verb in the past @xcite . unlike kurmanji which uses oblique cases for this purpose, sorani only uses different pronominal markers to specify ergativity, therefore it is called split-ergative @xcite . except the past tenses, a nominative-accusative alignment is observed in other tenses. not being equally documented and used, kurdish dialects have different levels of linguistic resourcefulness. in comparison to sorani and kurmanji which are widely used by the media and press, southern kurdish and laki are underdocumented and lack basic language resources such as electronic dictionaries and corpora @xcite @xcite . ## current state of kurdish language processing the earliest works in the field of kurdish language processing date back to 2009. our literature review indicates that some of these contributions fail to provide open-source solutions. despite financial and scientific constraints in kurdish language processing, the kurdish language processing project (klpp) @xcite in 2012 foot_2 and kurdish basic language resource kit (kurdish blark) @xcite in 2014 foot_3 have succeeded to promote an open-source vision based on research volunteering within the kurdish scientific communities. however, the outcomes of these projects are mostly released in an unorganized manner for individual tasks. in order to understand the current state of the kurdish language in the realm of nlp and com2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 putational linguistics, we reviewed the scientific publications that directly address an issue in those fields. a total number of 53 publications are collected from the widely-used academic databases and search engines such as google scholar foot_4 , and then classified based on their discussed sub-fields which are illustrated in figure 1 . the kurdish dialects are not evenly discussed in the previous studies, with sorani making up a predominant proportion of almost 90%. although a smaller proportion represents the kurmanji dialect, no publication is found with respect to processing of the southern kurdish or laki dialects. regarding the research focus of the previous works, a range of nlp sub-fields has been addressed, particularly in text mining, morphological and syntactic analysis and, creation of lexical resources. we exceptionally included optical character recognition as it is of importance for converting printed material to electronic forms @xcite . the full list of the surveyed papers can be found in appendix a.2. more importantly, we analyze previous publications from the following two perspectives: figure 2 provides the number of previous publications in the kurdish language processing field per year, and specifies their open-source status and their applicability. although most of these publications are applicable to other dialects, only 18 out of 53 of them provide their resources or tools under an open-source license. among the opensource ones, 11 are outcomes of volunteering projects, klpp and kurdish-blark. given the small number of non-scientific contributions, we did not include them in this survey. a few notable examples of such contributions are kurdînûs foot_8 , vejin dictionaries foot_9 and vejinbooks foot_10 which mostly focus on sorani kurdish and script conversion tasks. ## klpt architecture klpt is implemented in python and is composed of four core modules with specific tasks. although we were inspired by the functionality of relevant nlp toolkits, particularly nltk and spacy, no external library is used in this toolkit. regarding the toolkit design, we followed the rules of scientific software development suggested by @xcite along with common practices in python programming language. figure 3 provides the structure of the toolkit. in order to facilitate the integration of variations specific to dialects and scripts and more importantly, to avoid hard-coding, required files are provided in the data folder. for instance, the data required for the preprocess module is imported from preprocess.json. in addition, third-party programs can be provided in bin. test and docs respectively contain test cases and project documentation. regarding the latter, we use sphinx documentation generator 12 . it is worth noting that each module within the klpt package has been previously studied and evaluated separately. our goal is to introduce the functionality of the modules within the toolkit in this section. ## usages in this section, we provide basic usages of the application programming interface (api) of the klpt package. the package is available on the python package index (pypi) foot_12 in python 3.5 and later and, can be installed as follows: