Approaches to Corpus Creation for Low-Resource Language Technology: the Case of S outhern K urdish and L aki (2023)

Abstract: One of the major challenges that under-represented and endangered language communities face in language technology is the lack or paucity of language data. This is also the case of the Southern varieties of the Kurdish and Laki languages for which very limited resources are available with insubstantial progress in tools. To tackle this, we provide a few approaches that rely on the content of local news websites, a local radio station that broadcasts content in Southern Kurdish and fieldwork for Laki. In this paper, we describe some of the challenges of such under-represented languages, particularly in writing and standardization, and also, in retrieving sources of data and retro-digitizing handwritten content to create a corpus for Southern Kurdish and Laki. In addition, we study the task of language identification in light of the other variants of Kurdish and Zaza-Gorani languages.

## introduction language and linguistic data play a critical role in documenting and preserving endangered and under-represented languages. indispensable to computational methods in language technology, data also enables the development of tools and applications, such as speech recognition and machine translation, that can support the revitalization and promote the usage of such languages. as such, speakers of endangered and under-represented languages ultimately have the opportunity to share their language and cultural heritage with future generations. despite the fascinating advances in natural language processing (nlp) in recent years, particularly in working with very limited data in low-resource setups @xcite , collecting data for endangered and less-resourced languages remains a challenging task. in this paper, we focus on southern kurdish (sdh in iso 639-3) which is one of the main varieties of kurdish spoken by an estimated 3.7 million speakers in the provinces of kirmaşan, also spelled as kermanshah, and ilam in iran, and across the adjoining border regions of iraq @xcite . we also shed light on laki (lki) spoken by a few hundred thousand speakers in the same regions @xcite . kurdish in general and southern kurdish and laki in particular, have faced various discriminatory language policies that have led to pernicious sociolinguistic effects on language attitudes and heritage language maintenance such as the lack of children's proficiency in southern kurdish and limited usage of the language in writing @xcite . as such, southern kurdish speakers have been facing centuries-long pressures of persian as the only official language of iran and the administratively dominant one, which led to various phenomena of language shift and change @xcite . although other varieties of kurdish have not been immune from such policies, their larger population and strong kurdish national and political identity be-ing supported for a long time have been beneficial to promote the language, especially for the adjacent central kurdish speakers, also known as sorani (ckb), in iran and iraq @xcite . nevertheless, studies show that there is a positive attitude towards the kurdish language and identity in communities of southern kurdish speakers as well @xcite @xcite . in stark contrast to other varieties of kurdish, particularly northern kurdish (kmr), also known as kurmanji, and central kurdish, southern kurdish varieties and laki have not received much attention in linguistics or computational linguistics. moreover, for southern kurdish and laki, there are relatively much fewer digital resources available, not to say near zero, and both face practical challenges in writing. on the other hand, studying these varieties in loco, i.e. linguistic fieldwork, is not always a viable solution given the geopolitical conditions of the region and limitations in cost and time. contributions this paper sheds light on creating a corpus for southern kurdish and laki. we discuss possible approaches that can be taken to tackle corpus creation for under-represented and endangered languages by relying on local content creation media and also, fieldwork. our corpus contains over 2 million tokens in southern kurdish and laki. in addition to an intrinsic evaluation of the corpus, we also analyze the corpus in a qualitative way and extend our analysis to the task of language identification. ## related work although a less-resourced language, kurdish has increasingly received attention in the past few years in language technology with tools such as the kurdish language processing toolkit (ahmadi, 2020b), services such as google translate foot_1 and models and benchmarks in nlp such as the flores-101 @xcite and nllb @xcite . however, these solely include northern and central kurdish but neither southern varieties nor laki. similarly, wikipedia as an important resource to document languages is only available for northern and central kurdish while southern kurdish and laki along with other adjacent under-represented languages gorani and luri are not supported yet. @xcite study the available lexicographical resources for kurdish varieties and, as illustrated in figure 2 , find that among the 71 dictionaries and terminological resources available for kurdish, laki and zaza-gorani languages in electronic and printed forms, only 13.6% have content for a southern kurdish variety or laki. previously, some linguistic aspects of southern kurdish have been studied such as phonology @xcite , typology (dabir-moghaddam, 2012), morphology @xcite and dialectology @xcite . @xcite studies laki and describes its complex relationship with southern kurdish and also documents a laki variety by collecting a lexicon and a corpus through fieldworking. considering resources for language technology, azin and ahmadi (2021) create an electronic dictionary in ontolex-lemon containing 14,326 entries of varieties of southern kurdish in addition to laki and luri languages. in this resource, entries are represented in both scripts commonly used for kurdish, even though the latin orthography is not much used for southern kurdish, in addition to translations in persian and central kurdish. to the best of our knowledge, this dictionary is the only electronic resource for southern kurdish of considerable size. similarly, @xcite collect audio samples for kurdish spoken dialect recognition using radio and television contents among which 11 hours are collected for southern kurdish. in addition to language documentation, corpora are crucial resources in many other applications such as language learning and teaching @xcite , machine translation and syntactic parsing. to the best of our knowledge, no corpus of considerable size has been created for southern kurdish and laki that is written in any of the two conventionalized kurdish scripts. ## methodology to remedy the lack of data for southern kurdish and laki, we follow three approaches that are described in this section. ## results and analysis in this section, we carry out an intrinsic evaluation of our corpus alongside presenting a qualitative analysis. we also extend our analysis to the task of language identification. ## conclusion data in general, and corpora in particular, provide a foundation for the preservation and promotion of endangered and under-represented languages in language technology. in this paper, we discuss three approaches for data collection and corpus creation of low-resourced and under-represented languages, namely southern kurdish varieties spoken in kermanshah province (iran) and feyli varieties spoken in iraq. while the kermanshahi dataset has been collected from a local radio station and by crawling a news website, we collected data for laki by fieldwork, which despite considerable challenges, seems to be the only solu-tion for a language with near zero online presence. our approaches can be adopted by other underrepresented languages with limited data and without the possibility of fieldwork. we finally provide a brief analysis from quantitative and qualitative perspectives along with the evaluation of language identification for kurdish and zaza-gorani languages with different scripts. our model can be beneficial to detect texts and collecting more data. regarding future work, a data-driven approach can be explored to shed light on the various linguistic differences among the selected languages and varieties. moreover, as our target languages have been under the threat of linguistic assimilation @xcite , particularly through lexical borrowing from persian, arabic and turkish, a new problem transpires which is to determine lexical borrowing. we believe that lexical borrowing detection @xcite ) can be further studied in the future thanks to our data. the collected corpora can pave the way for further developments in language technology and cross-lingual studies. finally, annotating these corpora for other tasks, particularly part-of-speech tagging and named entity recognition can be addressed in the future.