Using LARA to create image-based and phonetically annotated multimodal texts for endangered languages (2022)

Abstract: We describe recent extensions to the open source Learning And Reading Assistant (LARA) supporting image-based and phonetically annotated texts. We motivate the utility of these extensions both in general and specifically in relation to endangered and archaic languages, and illustrate with examples from the revived Australian language Barngarla, Icelandic Sign Language, Irish Gaelic, Old Norse manuscripts and Egyptian hieroglyphics.

## introduction when people are reading documents written in a language less than completely familiar to them, it can often be useful to present the text in multimedia form. this can give the reader access to annotations -typically audio recordings and translations -with a single click, conferring immediate and obvious advantages compared with reading a printed text and looking words up. many such frameworks now exist; prominent examples in- * * authors in alphabetical order. the version of lara from last year's paper represented the document as a text string and al-lowed annotations to be attached to units at two levels, words and segments (typically a segment is a sentence). experience since then has revealed two important ways in which the above needs to be further generalised. first, thinking of a written document abstractly as a text string obscures the important fact that it is also a visual object. for many texts (picture-books, posters, handwritten manuscripts), the visual dimension is as significant as the words. second, it is often necessary to go below the word level and think about the relationship between sounds and letters or other primitive written signs. if the student is uncertain about the writing system, the sound system, or the relationship between them, annotations at the character level can be helpful. these observations are particularly relevant to endangered languages, and indeed it is largely because of our close interaction with the endangered language community that we have become so aware of them. we will have more to say about this later, when we discuss specific languages. in the rest of this paper, we will describe recent work where we have extended lara to allow image-based and phonetic annotations to be added to texts, and we again illustrate with concrete case studies. section 2 presents the new functionality, after which sections 3 to 5 present examples of how it has been used for barngarla, icelandic sign language and irish gaelic. section 6 briefly describes how the same features are also useful for annotating historical texts available in manuscript or related form. the final section concludes and suggests further directions. ## supporting image-based and phonetic annotations in this section, we briefly present the overall architecture of lara and then describe the new functionality which forms the subject of this paper. full details are available in the online documentation @xcite . ## barngarla barngarla is an australian aboriginal language belonging to the thura-yura language group, a subgroup of the large pama-nyungan language family. typically for a pama-nyungan language, barngarla has a phonemic inventory featuring three vowels ([a], [i], @xcite ) and retroflex consonants, an ergative grammar with many cases, and a complex pronominal system. during the twentieth century, barngarla was intentionally eradicated under australian 'stolen generation' policies, the last original native speaker dying in 1960. language reclamation efforts were launched in 2011 (zuckermann, 2020). since then, a series of language reclamation workshops have been held in which about 120 barn-garla people have participated. the primary resource used has been a dictionary, including a brief grammar, written by the german lutheran missionary clamor wilhelm schürmann (schürmann, 1844; @xcite . a number of educational texts have now been constructed using schürmann material as the base; as described in last year's paper, several of them have been converted into lara form. this has highlighted two issues, both of which materially contributed to motivating the new functionality we describe here. first, the original texts are always created as a collaboration between ethnic barngarla people and non-barngarla expert linguists: usually, design aspects are the responsibility of the barngarla members of the team. when converting the texts into lara form, it is thus important to maintain a format that is as close as possible to the original text layout. second, even though revised barngarla is written phonetically, the orthography is not transparent to people whose linguistic heritage is primarily anglophone. a particularly important example is retroflex consonants, which are written using an 'r' before the corresponding nonretroflex version: thus the voiced retroflex plosive [ã] (similar to the final sound of swedish nord, "north") is written 'rd' as for example in barngarla yarda, "country". it is however all too easy for the anglophone reader to interpret this as representing a lengthened preceding vowel followed by [d], as for example in the usual australian pronunciation of "card" or "herd". another important problem is ambiguous phonetic segmentation. barngarla orthography contains both the unigraph 'w', representing the velar approximant [w] and the digraph 'aw', representing the diphthong @xcite . when bargarla digraph 'aw' is followed by letters representing a vowel, as for example in the common words bawoo ("hello"), gawoo ("water"), the anglophone reader most naturally segments the words as b|a|w|oo, g|a|w|oo; in fact, they should be b|aw|oo, g|aw|oo. these issues came to a head during the creation of the latest barngarla text, mangiri yarda (zuckermann and richards, 2021). the main barngarla contributor, emma richards, invested a substantial amount of effort in the design of the book, and it was clear that the approach used for previous barngarla lara texts, trying to reproduce the layout using html formatting, would not yield a good result. the issues with pronunciation also became apparent when recording the audio. the new functionality developed here however made it possible to address both the layout and phonetic issues in a logical way. the draft book is available online here. it is organised as a lara picture-book exactly reproducing the text layout, in which all the barngarla words are annotated with audio information, coupled with a parallel track organised as a "phonetic" lara book, where the reader can spell through each word a letter-group at a time and listen to the associated phonetic value. by the time of the conference, we expect that the book will have been tested with enough barngarla readers to be able to present initial feedback. ## icelandic sign language icelandic sign language (íslenskt táknmál; ítm) is a natural language and the first language of about 250-300 people in iceland, almost exclusively deaf people and their children. a peculiarity of ítm, compared to other sign languages, is that hereditary deafness hardly exists in iceland. this means that deaf children are much less likely to have deaf parents than in other countries, rendering more difficult the intergenerational transmission of the language and contributing to its endangered status. zuckerman et al., 2021 gave further background and outlined some initial experiments in which lara was used to create annotated texts for deaf readers, with audio replaced by signed video. here, we describe two sample image-based texts of this kind. both are direct multimodal transpositions of existing paper texts designed for the ítm community, whose general purpose is to introduce ítm signs, and in particular the handshape inventory, to beginner signers. the signed video content has been taken from youtube videos linked from icelandic signwiki ( https://is.signw iki.org/). ## irish irish, from the celtic branch of the indo-european family, is the first official and national language of the republic of ireland and is now a full work-ing language of the eu. english is the second official language in ireland. despite the official status of irish, however, an erosion of first-language speaker communities is clear and according to the unesco atlas of the world's languages in danger, the language is considered "definitely endangered" @xcite . irish is spoken as a community language in pockets in the rural west of ireland called 'gaeltacht' areas. speakers in urban areas tend to be mostly in individual homes and irish is relatively rarely overheard on the street. irish is a compulsory subject until school leaving age. there are approximately 700,000 learners of irish in the education system in the republic of ireland @xcite . there are also large numbers in the education system in northern ireland and many learning irish abroad, although these numbers are more difficult to quantify. irish shares distinctive features with other celtic languages such as a verb-subject-object (vso) word-order and rich morphology @xcite . as in other celtic languages, initial consonants undergo mutations in specific grammatical contexts, e.g., the lenition of stops to fricatives/approximants; of voiceless stops to voiced stops; of voiced stops to voiced nasals. verbs are inflected for tense, number and person, while nouns are inflected for number and case. prepositions can inflect for person and number. nouns are either masculine or feminine in grammatical gender. @xcite , where there is a fundamental contrast between velarised [c è ] and palatalised [c j ] phonemes irish has three main dialects and a number of sub dialects. these dialects differ at many levels, including their structural features, vocabulary and particularly in their pronunciation. a written standard "an caighdeán oifigiúil" was first introduced in 1958 and the most recent update to this was published online in 2017. however, as with many minority languages, there is no single spoken standard and all dialect variants hold equally. the fact that the writing system does not match in a simple way to any one of the spoken dialects presents challenges to learners. a major feature of the irish sound system is the contrast between palatalised and velarised pairs of consonants as illustrated in figure 5 . the contrast of palatalised and velarised segments not only differentiates words, e.g., /ï j o:ð è / leon 'lion' vs. /ï è o:ð è / lón 'lunch', but serves for grammatical differentiation of the same lexical item, as in /ï è o:ð è / (nominative) vs. /ï è o:ð j / (genitive). latin script is used for the language's writing system, with an alphabet which is superficially similar to english, excluding j, k, q, v, w, x, y, z, (except in loan words). however, the consonants are not marked for the fundamental contrast of palatalisation and velarisation of irish; rather, the palatalisation-velarisation difference is shown by the adjacent vowel letter used @xcite . all this makes it very complex for learners to acquire the link between the orthography and the sounds of the language. there is also a contrast between long and short vowels, which differentiates words, e.g. /m j in j / min '(oat)meal' and /m j i:n j / mín 'smooth'. long vowels are orthographically marked with an acute accent, as in: á é í ó ú. ## manuscripts and other archaic texts although this is not the focus of the current paper, we note in passing that the functionality described here also appears to be relevant to archaic texts in manuscript or inscription form, where the visual appearance of the document is of critical importance. we illustrate with two initial examples. the first is a lara version of an extract from the old norse poem völuspá @xcite , with each verse presented both in facsimile manuscript and plain text form. the second is an inscription in ancient egyptian hieroglyphics taken from @xcite , presented in parallel 'word' and 'sign' views. the two examples are posted here and here. ## summary and further directions we have described extensions recently added to the lara platform to support image-based and phonetically annotated texts, and illustrated with examples from barngarla, icelandic sign language, irish, french, old norse and egyptian hieroglyphics. the work was motivated by the demands of these languages, particularly the first three. we are currently liaising with members of other endangered language communities, a leading example being the austronesian language iaai. the implementation of the new functionalities is still at an early stage, and our current priority is to improve their integration into lara and make them easier to use. in particular, we have started development of an intuitive tool which will enable simple creation of "lara albums", lara picture-book/phonetic documents consisting of images paired with short captions. the intention is to lower the bar to entry for people wishing to create lara texts, so that it can become a routine part of language teaching; this is particularly interesting in the context of the an scéalaí irish platform (cf. §5). we hope to be able to report on this work later in 2022.