Morphologically annotated corpora of Pomak (2022)

Abstract: The project XXXX is developing a platform to enable researchers of living languages to easily create and make available state-of-the-art spoken and textual annotated resources. As a case study we use Greek and Pomak, the latter being an endangered oral Slavic language of the Balkans (including Thrace/Greece). The linguistic documentation of Pomak is an ongoing work by an interdisciplinary team in close cooperation with the Pomak community of Greece. We describe our experience in the development of a Latin-based orthography and morphologically annotated text corpora of Pomak with state-of-the-art NLP technology. These resources will be made openly available on the XXXX site and the gold annotated corpora of Pomak will be made available on the Universal Dependencies treebank repository.

## introduction in philotis foot_0 we aim at supporting the researchers of living languages to develop annotated (linked) spoken and textual resources without external technical aid: ideally, speakers of the documented language and eager linguists alone would suffice. to this end, we take advantage of open-source nlp tools, semantic web technologies, annotation tools and universally adopted annotation and codification schemes. pomak is our case study of endangered oral language. we make available existing and new textual and oral material of pomak and develop annotated spoken and textual corpora. here, we provide some information about pomaks and their language and, briefly present our experience from the development of a latin-based orthography and a morphologically annotated corpus of pomak. several researchers have highlighted the interdisciplinary nature of language documentation work (to mention but a few: woodbury 2003; mcdonnell 2018; rice 2018; bird 2020) because different linguistic specialisations are required and linguistic activity can hardly be considered independent of its social and situational settings. furthermore, the technical problems of resource development should not be underestimated. @xcite explained that ideally, language technology should support multimodal data and multilayered annotations that would be linked to each other so that they could be studied simultaneously. we would add that technical solutions have to be flexible, among other things because different languages may pose different documentation problems, in particular if the linguistic communities want to exploit legacy material. state-of-the-art tools and methods greatly facilitate traditionally hard tasks such as morphological annotation of corpora @xcite and speech to text transcription @xcite . we have taken advantage of this technology and received excellent results but the overall experience was not devoid of problems. we proceed by introducing pomak as an endangered oral language; next we discuss our experience with the development and morphological annotation of the corpora of pomak. ## about pomak pomak (endonym: pomácky, pomácko, pomácku or other dialectal variants) is a non-standardised east south slavic language variety. pomak is spoken in bulgaria and greece (mainly the rhodope mountain area), in the european part of turkey and, in the places of pomak diaspora @xcite . pomak is included in the map of the european languages equality network 2 . as is the case with all east south slavic varieties, several of the linguistic features that appear in the pomak dialectic continuum are due to mutual interaction and convergence with non -slavic languages of the balkan sprachbund (papadimitriou 2013: 23), mostly latin (solta 1980) and greek @xcite . in comparison to all east south slavic languages, pomak seems to exhibit a more profound phonological, morphological, morphosyntactic and lexical influence by medieval and modern greek @xcite and, due to the predominantly muslim religion of its speakers, a more profound lexical and phonotactical influence by ottoman and modern turkish. there is no widely accepted orthography of pomak. the language is not taught in any of the countries where pomaks reside. ## compiling textual corpora of pomak an oral/endangered language may have some textual and audio legacy @xcite . factors of language vitality and endangerment scores for pomak mature open-source nlp technology that would take full advantage of archived textual material is not available yet @xcite . undoubtedly, a detailed tei-conformant encoding of this material is the optimum approach but, at the moment, we have given priority to (spoken) material collection. we are in the process of defining dublin core and tei-conformant metadata to declare the origins of the material in the corpus and to develop links of medium granularity between the resources. in addition to the above, an oral/endangered language may have resources that can be used to effectively improve the quality of its morphosyntactic annotation. in our work with pomak we had the benefit of the electronic lexicon rodopsky foot_2 , which contains approximately 61.500 lemmas that correspond to about 3.5 x 10 6 unique forms (i.e., combinations of a lexical token and a pos symbol) annotated for lemma, pos and morphological features (figure 1 ). we exploited rodopsky to obtain mature morphological annotation of the corpora so we needed morphological annotation and evaluation facilities separate from the syntactic ones. it goes without saying that, since pomak has been sparsely documented by individuals who employed largely incompatible orthographies and in order to take advantage of rodopsky, which also employs its own orthography, text homogenisation work was deemed necessary. the first step to this direction was the @xcite alphabet (k&k alphabet from now on, illustrated in ## the orthography of pomak a key issue in developing the corpora of pomak was the orthography. no alphabet of pomak proposed so far, let alone orthography, has enjoyed any acceptability. a good alphabet would, at minimum, help maximise the possible impact of the developed resources on the sustainability of the documented language. in the case of pomak, we have adopted the k&k alphabet ( cahill and karan (2008) discuss good practices for developing orthographies for oral languages. armostis et al. (2014 their overall recommendation is that native speakers should participate in the definition of the orthography and have the final word in several decisions. furthermore, they identify the, probably conflicting, good practices that are briefly introduced and discussed immediately below: phonetic transparency. the phonemic analysis of a language is indispensable for orthography design but detailed work with a multitude of languages suggests that the lexical level of phonology is also important. this is because while different phonological processes may result in given surface forms, native speakers may be aware of some phonological processes but not of others, so they may only be aware of the lexical form. after all, the script is meant to be used first and most by native speakers rather than by linguists. we exemplify the application of these ideas with the following pomak words that form minimal pairs on the basis of sound to phoneme correspondence: paláta 'floor' vs. palta 'doused', cíkom 'squeak' vs. číkom 'cut; break', samár 'saddle' vs. šamár 'slap', som 'i am' vs. søm 'i sow', grom 'thunder' vs. grøm ' i heat', pat 'under' vs. paet 'read (past passive participle)', lóka 'valley' vs. lka 'light (adj., acc.masc.sing.), sénem 'i shadow' vs. šénem 'i amuse myself', vris 'fountain; tap' vs. vriš 'you boil; you are full of'. systematic orthographies with reliable soundsymbol representation and consistent spelling enjoy enhanced acceptability, learnability, and usability by native speakers. spelling should not be affected by pronunciation changes due to context. for instance, b [b], d [d], g [g] are devoiced in wordfinal position or before a voiceless consonant. we chose not to orthographically show this devoicing for the sake of consistency across declension (in the case of nominal forms) and conjugation (in the case of verbal forms). this is why we spell hlaeb 'bread (nom|sg)' even though this form is pronounced @xcite given the final position of the originally voiced consonant; in this way spelling is consistent with all other forms, e.g. hlbu 'of/to (the) bread', chlba 'bread (acc|sg), hlbove 'breads (nom|pl)' etc. easily discriminable symbols: similar symbols or crowd adjoining letters, mirror-image symbols, overuse of a letter as part of various digraphs (e.g., bh, dh, ...), superimposing more than one diacritic are not recommended. for example, graphs denoting palato-alveolar sibilants are consistently spelled by adding a háček above their non-palato-alveolar counterparts (as in most other latin-written slavic languages), while graphs denoting palatalised sonorants are consistently indicated by means of a cedilla (or comma depending on the keyboard) below their non-palatalised counterparts as in latvian; this system was preferred to croatian lj and nj or slovak l' and ň since the former requires two graphs and the latter is not consistent. examples: cístem 'i clean' vs. čeréša 'cherry', slónce 'sun' vs. šténe 'puppy, cub', zólezo 'iron' vs. žalvá 'turtle; tortoise', kópele 'lad' vs. kókal , e 'bones (pl)', pésne 'song' vs. kámen , e 'stones (pl)'. ## the gold morphologically annotated corpus we have already said that in our work with pomak we had the benefit of the electronic lexicon rodopsky (fig. 1 ), which contains approximately 3.5 x 10 6 unique forms annotated, among other things, for lemma, pos and morphological features. in order to take advantage of this rich source of linguistic knowledge of pomak, some adaptation work was required: apart from transcribing it to the k&k orthography, the morphological annotation had to be mapped on the universal dependencies frame-work (ud) foot_3 and the conllu format had to be adopted. ud was chosen as a morphosyntactic annotation framework because of its large inventory of annotation features and because it is recognised by several open-source, state-of-the-art nlp tools that we planned to use for the morphosyntactic annotation of the corpus. the mapping on ud revealed problems of which the most important were: the mapping of the morphological annotation of pomak in rodopsky on the ud framework was carried out by native speakers and linguists and the results will be uploaded on the ud language specifications area. furthermore, it revealed interesting parallel phenomena of greek and pomak, in particular in the verb and the degree modification systems that deserve a closer study. once rodopsky was transcribed into a ud and conllu compatible form and was manually corrected, it was mapped on the corpora (both rodopsky and the corpora had been transcribed into the k&k orthography). this initiated an about 30-days long cycle of manual corrections, this time of 6350 sentences and 86700 words selected from the pomak corpus to form the gold tagged corpus that would be used for training and evaluating the nlp tools. this part of the annotation was performed by a native speaker and a linguist fluent in pomak but not in ud, so the manual annotation time reported includes their training in the framework (interannotation agreement kappa scores on 476 sentences: pos tags 0.90, features 0.87, lemmas 0.93). the corpus will be uploaded to the ud language repository. alternatively, we could have proceeded with the morphological annotation of gradually bigger corpora @xcite . however, the selected procedure had clear merits: on the downside of the procedure are: ## morphological annotation of the corpus of pomak the gold morphologically annotated corpus was used to train and evaluate nlp tools that would, in turn, be used to assign morphological annotations to the entire pomak corpus and to future material from the spoken corpora. we conducted a series of experiments with four tools in an effort to identify the one that would yield the best morphological annotation results for pomak. the situation with state-of-the-art open-source nlp tools reminded of the description by @xcite : ". . . although basic principles are quite straightforward to master, the details of use of particular tools and interaction between tools in different setups are highly specific and can often be a source of frustration. thus, not only an effort is required from the ld practitioners to invest in learning, but considerable effort is also required from the developers to invest in harmonisation of tools and making workflows more straightforward and robust." our experience confirms that even people with a training in programming must spend considerable time on state-of-the-art nlp tools. we ran four open-source tools, all implemented in python. all tools provided a command line interface, but: a. instructions often were problematic: (a) outdated compilation instructions (b) instructions for training a model of a new language from scratch: (i) some tools provided insufficient documentation of the addition of languages new to the ud framework, and (ii) the alignment of the processes included in the pipeline was a hard task with some tools with incomplete instructions (c) outdated readme instructions required missing files; we had to correct the code. b. both the separation of morphological from syntactic annotation and the independent evaluation of the two annotation levels were hard. we used rodopsky for the morphological annotation of the pomak corpus and we wanted to evaluate morphological annotation only, however some tools did not allow for this. also, all tools assigned both morphological and syntactic annotation which may not be always desirable because when a new language is documented, the various levels of analysis (morphology, syntax, semantics etc) have not reached the same stage of maturity. morphology is the basic annotation level and it is reasonable to address it first. we think that the unified annotation should be an option and not the rule. we had to rewrite the code of some nlp tools and comment out the parts handling dependency relations in order to obtain evaluation results for the morphological annotation. this said, we would like to note that, probably, the assignment of false dependency relations might eventually be of no or little harm. we plan to compare the unified and the two-stage annotation strategy with future experiments on the corpora of pomak. there is a keen interest in incorporating contextual word embeddings as a functionality @xcite but at the moment, pretrained transformer models are available with few tools only. amongst the ones we tested, spacy v3.2.2 allows for transformer based autoregressive models, while udify supports only bert like models. one might note that pretrained multi-language models can be used by just one openly available nlp library. however, languages with no annotated corpora, such as pomak, must have access to pretrained multi-language models in order to be assigned a reasonable (first) morphosyntactic annotation @xcite . we investigated the performance of the tools spacy v3.2.2 foot_4 @xcite , stanza foot_5 (qi et al., 2020), udify foot_6 @xcite and udpipe 8 (straka et al., 2016) on the gold morphologically annotated corpus of pomak that was further split into training, development and test set (80:10:10). ( ## conclusion we have described the procedure of developing state-of-the-art textual resources for pomak, an endangered, oral european language of the slavic family. a group of linguists, computational linguists and engineers took full advantage of the pomak legacy and cooperated closely with the native speaker community. in this way and in a short period of time (about 8 months), we produced reasonably sized morphologically annotated corpora of good quality and identified the open source nlp tools for the morphological annotation of pomak. we have also reported on our experience with using open nlp tools. we have observed that skilled programmers may still be needed in order to use these tools. furthermore, powerful tools have not been fully exploited yet. in the overall, however, the huge progress in openly available state-of-theart nlp technology has boosted the development of resources for endangered oral languages. bradley mcdonnell. 2018. reflections on linguistic analysis in documentary linguistics. in andrea l. berez-kroeker & gary holton bradley mcdonnell, editor, reflections on language documentation. 20 years after himmelmann 1998. language documentation conservation special publication, volume 20, pages 191-200. minh van nguyen, viet dac lai, amir pouran ben veyseh, and thien huu nguyen. 2021. trankit: a light-weight transformer-based toolkit for multilingual natural language processing. in eacl (system demonstrations), pages 80-90. panayotis g. papadimitriou. 2013. dialects of the pomaks of the greek rhodope. regional analytical slavic and muslim speakers in southeastern europe. thessaloniki: balkan peninsula research institute. [in greek]. peng qi, yuhao zhang, yuhui zhang, jason bolton, and christopher d. manning. 2020. stanza: a python natural language processing toolkit for many human languages. in proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations, pages 101-108, online. association for computational linguistics. sally rice. 2018. reflections on documentary corpora. in andrea l. berez-kroeker & gary holton bradley mcdonnell, editor, reflections on language documentation. 20 years after himmelmann 1998. language documentation conservation special publication, 15, pages 157-172. university of hawai'i press. georg renatus solta. 1980. einführung in die balkanlinguistik mit besonderer berücksichtigung des substrats und des balkanlateinischen. darmstadt: wissenschaftliche buchgesellschaft. milan straka, jan hajič, and jana straková. 2016. ud-pipe: trainable pipeline for processing conll-u files performing tokenization, morphological analysis, pos tagging and parsing. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 4290-4297, portorož, slovenia. european language resources association (elra). anthony c woodbury. 2003. defining documentary linguistics. in peter k. austin, editor, language documentation and description, volume 1, pages 35-51. london: soas. a appendices pronunciation character example word [a], [5] a, a astávem 'leave' [e5], [ae] ae, ae laek 'drug' [b] b, b bába 'grandmother' [t" s] c, c cístem 'clean' [ù] č,č čeréša 'cherry' [d] d, d dórvo 'wood' [e] e, e predávom 'sell' [f] f, f fátom 'catch' [g] [g j ] g, g górlo 'throat' [d" z] ǵ, ǵ ǵvaezda 'star' [d" z] ǧ, ǧ ǧumajá 'mosque' [x] h, h hránem 'feed' [i] i, i visok 'tall' [j] j, j játo 'food' [k], [k j ] k, k kukóška 'hen' [ë], [l j ] l, l lažýca 'spoon' [l] l , , l , kókal , e 'bones' [m] m, m magáre 'donkey' [n], [n j ] n,n nus 'nose' [ñ] n , , n , spañé 'sleep' [o], [u], [a], [5] o, o pot 'road' [ø] ø, ø spøm 'to sleep' [p] p, p pétal 'horse shoe' [r] r, r rábata 'work' [s] s, s sórce 'heart' [s] š, š šápka 'cap' [t] t, t tumafíl 'car' [u] u, u ušá 'ear' [y], [ j u] ü, ü türén 'train' [v] v,v vorh 'top' [1] y, y kysmét 'fortune' [z] z, z zimá 'winter' [z] ž, ž žalvá 'turtle'