A finite-state morphological analyser for P araguayan G uaraní (2021)

Abstract: This article describes the development of morphological analyser for Paraguayan Guaraní, agglutinative indigenous language spoken by nearly 6 million people in South America. The implementation of our analyser uses HFST (Helsiki Finite State Technology) and two-level transducer that covers morphotactics and phonological processes occurring in Guaraní. We assess the efficacy of the approach on publicly available Wikipedia and Bible corpora and the naive coverage of analyser reaches 86% on Wikipedia and 91% on Bible corpora.

## introduction morphological modelling, under which we subsume both morphological analysis and morphological generation is one of the core tasks in the field of natural language processing. it is used in a wide variety of areas, including but not limited to: orthographic correction @xcite , electronic dictionaries @xcite , morphological segmentation for machine translation @xcite , as an additional knowledge source for parsing languages with nontrivial morphology @xcite , and in computerassisted language-learning applications @xcite . in this article we describe a morphological analyser for paraguayan guaraní (in guaraní: avanẽ'e, iso-639: gn, grn), one of the official languages of paraguay. although guaraní is an official language and spoken by over six million people throughout the south american continent @xcite , it does not benefit from a wide range of freely-available data and tools for building natural language processing systems. if we use wikipedia as a proxy for viability of crowdsourcing linguistic data, as in @xcite , we see that although guaraní has a large speaker population, the potential for crowdsourcing and big freely-available data is limited. 1 the absence of large amounts of textual data means that data-driven approaches are hard to apply. in addition, supervised approaches, including neural networks which have become increasingly popular, require large amounts of annotated data to be trained. this in turn requires large numbers of trained annotators to annotate it. given that neither of these are available, we apply tried-and-tested technique relying on formal linguistic description by means of finite-state transducers. finite-state techniques have been widely applied to morphological modelling of many languages and are state of the art for many languages, especially those with non-trivial morphology such as languages described as agglutinative @xcite or polysynthetic @xcite . the remainder of the article is laid out as follows: in section 2 we give an overview of guaraní, paying special attention to aspects of morphology and morphosyntax. section 3 reviews the prior work, section 4 describes the implementation of the analyser, including information about the linguistic data and tools used. we evaluate our analyser in section 5, giving both a qualitative, quantitative and comparative evaluation. and finally in section 6 we give some final remarks and comment on potential future work for guaraní. ## language guaraní (native name: avanẽ'e) is one of the most spoken indigenous languages of south america that belongs to tupi-guaraní stock. it is divided into dialects or even languages such as paraguayan 1 we note that the guaraní vikipetã, https://gn. wikipedia.org/ currently has a total of 3,767 articles (as of the 15th july 2020), while the english wikipedia, http: //en.wikipedia.org/ has 6,122,333 as of the same date. if we compare with a language with a similar number of speakers and official status, for example catalan, we see that the catalan viquipèdia has vastly more articles 652,079. guaraní, bolivian guaraní and some other dialects spoken in brazil (ava, kaiowá, nhandeva, mbyá etc.). according to ethnologue foot_0 population that speaks all the varieties of guaraní is 6.162.840 people. the majority of guaraní speaking population is located in paraguay where guaraní is considered the official language and consists of 5.850.000 monolinguals and bilinguals. see figure 1 for guaraní speaking area. guaraní is an agglutinative concatenative language. it's morphology has both derivational and inflectional traits: it uses suffixes, preffixes and circumfixes for word production. roots (or stems) affect the phonology of affixes concatenated to the stem and vice-versa, mostly in cases of nasal harmonization or incorporation foot_1 . the majority of the words in guaraní are oxytone with some exceptions when accentuation rules apply @xcite . only recently paraguayan institution academía de la lengua guaraní approved current orthographic standard for written guaraní @xcite . thus in literature published before 2018 writing standards vary significantly. for example, postposition 'hagũa' or 'haguã' can be written with g̃or ã where nasalization is marked graphically by tilde. according to phonological rules, nasalization propagates over the entire syllable if there are any nasal phonemes in it (krivoshein de canese, 1983), therefore, both spellings are acceptable. in addition, tilde indicates the stress for nasal vowels and special nasality marking in haguã may be considered excessive. in wikipedia corpus some nasalized phonemes are also marked with diaeresis '¨' (ï, ÿ, ä, etc.). our transducer handles all the spelling varieties and treats them as orthographic errors. despite guaraní being one of the most spoken low-resource languages of south america grammars thoroughly describing the language are not abundant. throughout this paper we mostly consult with (krivoshein de canese, 1983), (estigarribia, 2017) and @xcite , although there are earlier reliable grammars available @xcite . ## prior work most of the existing computational resources for guaraní so far are online dictionaries or translators supported by the community. they are based on aligned publicly avaliable corporas such as wikipedia or guaraní-spanish bible. for example, iguarani.com and glosbe.com are mostly supported by non-professionals i.e. native speakers or other enthusiasts. glosbe even has its api (application programming interface). but as textual sources for guaraní are scarce these translators are not always reliable and lacking words. at indiana university michael gasser @xcite developed mainumby translation system created mostly for paraguayan translators with implementation of finite state morphological analyzer paramorfo embedded into translator. this analyser is very close to what we have done although is focused mostly on the form generation rather than morphological analysis. the analyser discussed in this paper and paramorfo were built independently and we will evaluate two transducers for comparison. ## development transducer-based morphology modelling is essentially the mapping between elementary morphological units (morphemes) to morphological (part of speech) tags or whole lexemes. this mapping reflects the combinatorics and morphological constraints of natural language i.e. which morphemes can combine into a lexeme and which morphemes are incompatible. fst-based approaches use continuation lexicons term to denote the mapping as we will reference them throughout the paper. the implementation of continuation lexicons in our analyser is entirely built on dictionaries publicly available on the web. one of them is l3 project guaraní dictionary foot_2 from the hltdi-l3 github repository. our two-level transducer uses two formalisms: both of the formalism use specific syntax following hfst platform conventions. our analyser is a part of apertium foot_3 open-source platform and can be used freely and enhanced by any member of open-source community. in the paper it is referred as apertium analyser. lexicon nouns achegety:achegety n ; ! "abecedario" aguara:aguara n ; ! "zorro" aguyjevete:aguyjevete n ; ! "gratitud" ahoja:ahoja n ; ! "manta" ahoiha:ahoiha n ; ! "carpa" figure 2 : lexicon for noun stems from lexc file. the first element before colon is an underlying form, the second element stands for surface form of the nouns adding further lexicons to the surface stem (n-lexicon). after exclamation mark follows the comment with translation to the word. ## evaluation to evaluate the analyser we estimate naive coverage metric and compare it to paramorfo system. naive coverage foot_4 is the ratio of tokens that receive at least one morphological analysis to the total number of tokens in the corpus. we estimate performance of our transducer on two publically available corpora: the bible and the guaraní wikipedia. an example of a fully analysed guaraní sentence is presented in for fair comparison we ran wikipedia texts and the bible through apertium analyser and dropped all the tokens that did not belong to open-class category because paramorfo does not recognize closedclass words (adverbs, conjunctions, numbers, etc.) and punctuation marks. paramorfo segments tokens differently than our analyser so at the end of processing we received different quantity of entry tokens for each analyser. according to we also evaluate conventional quality metrics for the analyser such as precision, recall and f-measure. to estimate precision, recall and f-measure we have annotated 8308 tokens from different sources where each tokens has a corresponding valid analysis in the context. note that this estimate is only the approximation of the scores because in order to have true scores each form should be annotated with all valid