Building a Corpus for the Z aza–Gorani Language Family (2020)

Abstract: Thanks to the growth of local communities and various news websites along with the increasing accessibility of the Web, some of the endangered and less-resourced languages have a chance to revive in the information era. Therefore, the Web is considered a huge resource that can be used to extract language corpora which enable researchers to carry out various studies in linguistics and language technology. The Zaza–Gorani language family is a linguistic subgroup of the Northwestern Iranian languages for which there is no significant corpus available. Motivated to create one, in this paper we present our endeavour to collect a corpus in Zazaki and Gorani languages containing over 1.6M and 194k word tokens, respectively. This corpus is publicly available.

## introduction a language corpus refers to a collection of data in a specific language or languages which can be utilized as a sample of the language for linguistic purposes. with a significant number of tokens and sentences, a corpus contains various word forms and therefore, is beneficial in the linguistic analysis of a language, for instance in morphology and syntax. moreover, the recent advances in applying statistical and neural methods in natural language processing (nlp) have proved the importance of language resources, including large corpora, in improving various tasks, particularly using language models. however, language resources are not evenly available for all languages; given the number of the human languages around the globe, most of the languages are still considered less-resourced, i.e. languages for which there are only general grammar and few electronic texts available. zaza-gorani languages, a subgroup of the northwestern iranian languages, are not only less-resourced but are also deemed as endangered languages @xcite @xcite . zazaki and gorani are two of the main and most known languages belonging to this family. zazaki, also known as dimlî, is spoken by an estimated number of 2 million speakers in various regions in turkey @xcite . on the other hand, gorani foot_1 , also written as gurani, is the language of ∼300,000 speakers in the parts of the iranian of iraqi kurdistan @xcite . historically, gorani was the high literary language within the sorani kurdish speaking regions in such a way that it played a great role in the formation of modern sorani kurdish and literature @xcite . in this study, we present a corpus for zazaki and gorani. shabaki, as the last language in this language family could not be included due to it being extremely under-documented and least known @xcite . the corpus is built on the news articles from various sources in several topics such as science, politics, culture and art, and contains 1,633,770 tokens in zazaki and 194,563 tokens in gorani. we believe that this resource can pave the way for further developments in the processing of zaza-gorani languages in various nlp tasks such as automatic language and dialect identification @xcite and spelling and grammatical error correction. given the similarities between these languages and kurdish, this corpus can also be beneficial to take use of available resources and tools of kurdish, such as named-entity recognition @xcite . figure 1 : the place of the kurdish and zaza-gorani languages in the western iranian language family. dead and alive languages are respectively specified in red and green and, iso 639-3 language codes are provided in parentheses 2 kurdish vs. zaza-gorani the question of dialects and languages in the kurdish inhabitant regions has been a matter of discussion both in academia and among people. although some have suggested that northern and central kurdish, also widely known by their endonyms, respectively, kurmanji 3 and sorani, are two distinct languages given the structural differences between them, they are more uncontroversially accepted as two dialects of kurdish @xcite . despite the common belief that zazaki and gorani are two dialects of kurdish @xcite , studies indicate a consensus among linguists that those two are two distinct languages on their own @xcite . that said, there is generally a close feeling among all the three ethnic groups, kurds, goranis and zazas, with respect to the kurdish identity and culture with many centuries living together @xcite . however, the formation of new identities among gorani and zazas to distinguish themselves from kurds has been also studied more recently @xcite @xcite . kurdish, zazaki and gorani languages are all in the northwestern branch of the iranian languages within the indo-european language family. regarding the kurdish dialects, kurmanji is spoken in all the regions of kurdistan in iraq, iran, syria and turkey, with a predominant population in the two latter regions. sorani and southern kurdish are both spoken by the kurdish populations of iraq and iran. while the majority of the southern kurdish speakers are located in the southern parts of the iranian kurdistan, particularly in kermanshah and ilam provinces, sorani is the most widely spoken dialect in both kurdish regions of iran and iraq. on the other hand, zazaki is spoken only in the kurdish region of turkey, mostly in tunceli, bingöl, urfa, elazıg and north of diyarbakır in various dialects of dimli, kirdki, kirmanjki and kirmanji @xcite . finally, gorani is mostly spoken in the iranian kurdistan and smaller parts of the iraqi kurdistan in various dialects, including bajelani, sarli, gawrajuyi and hawrami (also known as awromani or awramani), among which the latter is the most popular and known dialect @xcite . these languages and dialects have linguistically influenced each other in various ways, including phonetics, vocabulary and morphology. more specifically, the mutual influence is observed between kurmanji kurdish and zazaki @xcite , and also, between sorani kurdish and gorani @xcite . in order to better compare these languages, we discuss some of the major common features of kurdish, zazaki and gorani from a comparative perspective. additionally, we present the alphabets used for writing. we regret that southern kurdish, laki and shabaki could not be included due to scarcity of resources and grammar books @xcite . 3 badini is also used to refer to the kurmanji spoken in the iraqi kurdistan ## methodology similar to the methodology proposed by @xcite to create the first standard test collection for the kurdish language, we used the material published on news websites in zazaki and gorani languages to build the first corpus for those two languages. in comparison to the sorani and kurmanji dialects of kurdish for which many websites are available, there are a very limited number of websites for zaza-gorani languages. among the available websites, we selected zazaki.net foot_2 for zazaki and firat news agency foot_3 for both languages. our selection criteria were the number of the available articles, availability of metadata in pages' source and the diversity of the covered topics. regarding the topics, the first website focuses on cultural issues and is composed of analytical articles in humanities and provides interviews in such fields. on the other hand, the latter addresses a wider range of news in topics such as women, politics, world, kurdistan, science, culture and art. after crawling the websites, we extract the content of the html pages and further clean them by removing non-relevant information such as urls, hashtags, contact details and cited sentences in languages other than our target ones, e.g. koranic verses in arabic. in most cases, the page's address schema enabled us to identify the language. however, in none of the websites specific tags were found to explicitly identify the language or the dialect in which the article is written. as such, we use a simple classifier to exclude english, turkish and kurdish articles from the zazaki and gorani ones. for this purpose, we manually selecte a list of the most frequent and unique words in each language as features. for instance, ziwan/zan/zon 'language', kerd 'did' and zî 'too, also' are unique to zazaki while ziman, kir and jî are respectively used in kurmanji kurdish. in the case of sorani kurdish and gorani, in addition to the frequent words, we use unique characters as features as well. this step is followed by a manual verification of the documents. figure 3: the 20 most frequent words in the zaza-gorani and kurdish corpora in order to keep the document-level information, we collected the cleaned documents in two directories based on the language. we provide further metadata such as topic, original source of the article and date of publication in a separate json file that can be associated to each document. regarding the date of publication, the zazaki articles dated between 2009 and 2020 while the gorani ones are more recent (2018-2020). ## results we carry out an intrinsic evaluation based on the statistics and the most frequent words in our corpus with comparison to @xcite . this corpus contains 18m tokens in sorani and 4m tokens in kurmanji kurdish. ## conclusion and future work in this paper, we presented our efforts in creating a language corpus for two endangered languages of the zaza-gorani language family. zazaki, gorani and shabaki are the three languages belonging to this language family and are popularly believed to belong to kurdish. we briefly discuss how these languages are different from kurdish, sorani and kurmanji dialects, in terms of phonetics, morphology and syntax. we also report our efforts in collecting documents in various topics from news websites and create the first corpus for zazaki and gorani. we believe that this corpus can pave the way for further developments in linguistics and computer science, particularly in information retrieval and nlp where language modeling is beneficial to various applications such as grammatical and spell checking. as a future work, we suggest a better documentation of the kurdish and zaza-gorani languages, particularly shabaki, southern kurdish and laki, by promoting the usage of those languages within local communities, websites and social media platforms. in the same vein, we invite researchers and native speakers to pay further attention to these languages, both in linguistics and nlp, by providing more analytical grammars, particularly in gorani and southern kurdish, and developing basic language processing tools, such as tokenization, stemming and lemmatization, and resources, such as wordnet @xcite and parallel corpora.