Towards a morphological transducer and orthography converter for W estern T lacolula V alley Z apotec (2021)

Abstract: This paper presents work towards a morphological transducer and orthography converter for Dizhsa, or San Lucas Quiaviní Zapotec, an endangered Western Tlacolula Valley Zapotec language. The implementation of various aspects of the language’s morphology is presented, as well as the transducer’s ability to perform analysis in two orthographies and convert between them. Potential uses of the transducer for language maintenance and issues of licensing are also discussed. Evaluation of the transducer shows that it is fairly robust although incomplete, and evaluation of orthographic conversion shows that this method is strongly affected by the coverage of the transducer.

## introduction in this paper, we present work towards a morphological transducer and orthography converter for dizhsa, also known in the academic literature as san lucas quiaviní zapotec (slqz), an endangered language variety of western tlacolula valley zapotec @xcite . 1 to our knowledge, this is the first computational implementation of the morphology of a zapotec language. (throughout the paper we use the term "language variety" in place of "dialect" because of the pejorative force of the word dialecto in spanish.) a morphological transducer, implemented as a finite-state transducer (fst), is a tool that performs morphological analysis (converts between a word form and a morphological analysis) and morphological generation (the reverse). for example, a form like gunydirëng 'they won't do' can be quickly converted to an analysis like @xmath0.apertium.org. the three authors recognise that we live and work on the homeland of the lenape, and pay respect and honor to the caretakers of this land, from time immemorial until now, and into the future. +ëng<@xmath1, followed by a 3rd person proximal plural personal pronominal enclitic); similarly, the analysis can be quickly converted to the form. not all speakers of slqz write their language, though more and more are doing so @xcite . there are published proposals for two orthographies, which we refer to as the phonemic orthography @xcite and the simple orthography @xcite ). an orthography converter between these two orthographies based on the morphological transducer has been developed as part of this work. both tools have the potential to support language maintenance efforts. a morphological transducer can be used in various types of computer-assisted language learning software, such as for learning vocabulary @xcite and complex inflectional systems @xcite . fsts are also used in electronic corpora @xcite , paradigm generators, foot_1 text-reading tools, foot_2 and form-lookup dictionaries @xcite . fsts may be trivially converted to spell checkers @xcite and can also be used in other types of text-proofing and language-learning tools (e.g., @xcite ; they can further serve as core elements of machine translation systems @xcite . morphological transducers are being developed for languages globally @xcite , including for entire language families, such as turkic @xcite . some of these fsts are developed for languages with large corpora, such as the national languages of western europe @xcite . one advantage of fsts is that they can be created for a language without a large quantity of existing text. for example, a morphological trans-ducer has been developed for zacatlán-ahuacatlán-tepetzintla nahuatl @xcite , a threatened language of central mexico with a relatively small corpus of texts. the fact that a morphological transducer can be developed with small corpora creates an entry point especially for threatened languages into the potential benefits of the types of language technology described above. this paper is structured as follows. section 2 situates slqz and overviews its socio-political context and basic morphological properties. section 3 describes the morphological transducer and demonstrates several of the challenges which were overcome in its implementation. section 4 presents a basic evaluation, including naïve coverage and accuracy of orthographic conversion. section 5 overviews some issues related to licensing of the tools and section 6 concludes. ## san lucas quiaviní zapotec san lucas quiaviní zapotec is spoken by 98% of the population in san lucas quiaviní, oaxaca, mexico (diegpo, 2015) and by diaspora communities elsewhere in mexico and the united states, especially the greater los angeles area @xcite , with approximately 3500 total speakers. while children are still acquiring the variety as their first language, it should be considered endangered as the community is shifting to spanish in more and more contexts @xcite . western tlacolula valley zapotec encompasses a number of related varieties, with varying degrees of mutual intelligibility. in the present work we focus on the variety of san lucas quiaviní (slqz), but we also evaluate the transducer on the variety of san juan guelavía (sjgz), also classified as western tlacolula valley zapotec. the two pueblos are separated by no more than 10km, but the two varieties of zapotec differ in many relevant aspects of their grammar, including tone and phonation contrasts, verbal morphophonology, and pronominal systems. understanding the morphotactics of slqz is essential to developing a morphological transducer. a verb form in slqz includes at minimum an aspect marker followed by a verb stem, with very few exceptions. additionally, a negative-marking enclitic, various other adverbial enclitics @xcite , and pronominal enclitcs may follow. nouns generally may be marked as possessed using a prefixwith some suppletive forms and a class of "essen-tially possessed" nouns which are always interpreted as possessed. possessors follow possessed nouns, either as independent noun phrases or as pronominal enclitics. pronominal enclitics also appear after predicate adjectives. the morphophonology of verb forms in slqz is complex. aspectual prefixes often have multiple realisations. additionally, there is a large number of verbs whose stems alternate irregularly or are synchronically suppletive depending on aspect, subject, and any following enclitics. some aspect markers have irregular realisations in these forms. there may also be changes in phonation type before certain enclitics. san lucas quiaviní zapotec has a very complicated system of tone and phonation with over 23 potential contrasts in a stressed syllable (although see chávez peón, 2010 for a different count). representing all of these contrasts results in an orthography that is complicated. members of the speech community have directly and indirectly expressed preference for a practical orthography that underrepresents these contrasts. hence a phonemic orthography (described first in @xcite is used in dictionaries and linguistic work, and a simplified orthography (described in @xcite which collapses many phonemic distinctions, is preferred by speakers of the language. being able to convert the simplified practical orthography to the phonemic orthography would allow linguists and speech scientists to recover the phonemic contrasts from text written in the practical orthography. ## implementation the transducer was implemented manually using the two-level approach @xcite and is designed for use with @xcite , an open-source toolkit for finite-state morphology. in the two-level approach to morphology, the lexicon and morphotactics of a language are implemented in one finite-state transducer (fst), the morphophonology is implemented in another, and they two are intersected into a single fst with an analysis side and a form side. for the dizsha transducer described here, both the morphotactics and morphophonology compile from hand-written patterns, lexicons, and rules. the lexd compiler (swanson and howell, 2021) was used to implement the morphotactics, and twol (part of hfst) was used to implement the morphophonology. the grammatical patterns of slqz were implemented in these formalisms by the first author in part while receiving classroom instruction in the language from the second author and based largely on patterns observed in the first volume of a dizhsa textbook @xcite . later the transducer was expanded and revised in consultation with additional volumes of the textbook and other sources cited here, and under the guidance of the second and third authors, the former of whom is a native speaker and teacher of dizhsa, and the latter of whom is a linguist with expertise in the language. in section 3.1, the size and shape of the transducer's lexicon is presented. section 3.2 discusses some design decisions and how some spelling variants were handled. we explain how some aspects of the language's morphotactics (section 3.3) and morphophonology (section 3.4) were implemented. section 3.5 presents how orthography conversion was implemented. ## evaluation the transducer was evaluated over available texts (4.1) for naïve coverage (4.2) and accuracy of orthographic conversion (4.3). ## licensing we have chosen to license this work under the gnu affero general public license (agpl) because we want it to be available for others to use and build on. this work is also part of a long-term commitment to collaboration with zapotec communities and community members. the agpl license allows for uses of our work that would be inconsistent with our commitment to the community. reciprocity is a defining zapotec cultural value and practice. zapotec speakers have shared their knowledge and language in the creation of these resources. others are allowed to use the tools and in doing so enter into a reciprocal commitment with the zapotec community that we define in what we call the guelaguetza clause, shown below: while licensed under a free/open-source license that permits commercial uses, it is expected that anything created using this resource be made available to the community of san lucas quiaviní free of charge. this is consistent with the community's practice of guelaguetza, a complex system of reciprocity and exchange of goods and labor. this context reminds us that the more broadly available licenses could use refinements in particular cultural contexts, particularly indigenous contexts, and that the field should be open to discussions of how culturally specific practices may interact with open source licensing. ## conclusion this paper has overviewed the development of a morphological transducer and orthography converter for san lucas quiaviní zapotec. an evaluation of the analyser over available texts demonstrates that despite being incomplete, it is fairly robust. future work to improve the transducer will focus on expanding the lexicon, adding missing morphological patterns, refining the morphophonological patterns, and finding better ways to deal with the nuances of slqz verb morphology. text in another variety of western tlacolula valley zapotec was evaluated using the morphological transducer, and the results suggest that a separate transducer might be needed. an evaluation of the orthography converter shows that this method of orthography conversion has potential, but is affected heavily by the coverage of the transducer. it is our hope that this resource will be useful to the slqz community. in particular, we are excited about the many roles it could play in language maintenance efforts. this work also impacts conversations on language technology for under-resourced languages and open licensing in indigenous contexts.