Codex to corpus: Exploring annotation and processing for an open and extensible machine-readable edition of the Florentine Codex (2023)

Abstract: This paper describes an ongoing effort to create, from the original hand-written text, a machine-readable, linguistically-annotated, and easily-searchable corpus of the Nahuatl portion of the Florentine Codex, a 16th century Mesoamerican manuscript written in Nahuatl and Spanish. The Codex consists of 12 books and over 300,000 tokens. We describe the process of annotating 3 of these books, the steps of text preprocessing undertaken, our approach to efficient manual processing and annotation, and some of the challenges faced along the way. We also report on a set of experiments evaluating our ability to automate the text processing tasks to aid in the remaining annotation effort, and find the results promising despite the relatively low volume of training data. Finally, we briefly present a real use case from the humanities that would benefit from the searchable, linguistically annotated corpus we describe.

## introduction the nahuatl language, an agglutinating and polysynthetic member of the uto-aztecan family spoken throughout mexico by about 1.5 million people today, has a rich literary tradition @xcite . with a strong preconquest oral tradition and a hieroglyphic writing system, nahuatl speakers quickly adopted the latin alphabet for writing their language after its introduction almost immediately after the spanish invasion. as a result, the volume of the colonialera nahuatl literary canon is unrivalled in latin america @xcite . these texts are invaluable resources to scholars interested in the history, culture, and language of colonial and pre-invasion nahua communities. perhaps the most notable nahuatl text of the early colonial period, the historia general de las cosas de nueva españa "general history of the things of new spain" (florentine codex, fc) is an encyclopaedic work in nahuatl and spanish compiled by indigenous scholars from the colegio de santa cruz de tlatelolco and franciscan friar bernardino de sahagún. the fc is undoubtedly one of the most valuable manuscripts of the early modern period. however, it was forgotten for centuries until angelo maria bandini described it in 1793. he named it "codice fiorentino" after the biblioteca medicea laurenziana in florence, where it is still kept. but only at the beginning of the 20 th century did francisco del paso y troncoso bring it to a wider audience @xcite . charles dibble and arthur anderson published a translation of the books into english throughout the second half of the 20 th century. the original manuscript became available in the world digital library only ten years ago, thanks to the library of congress. the impetus for the present project was the need of the third author, a humanities scholar, to search the text of the fc for specific linguistic constructions and terminology. this proposition is complicated by a number of factors: first, there are few fully digitised versions of the fc, and those that do exist are under copyright, constraining the ability of a scholar to reproduce, annotate, and/or re-release any part of the text that results from a given research endeavour. second, the fc, having multiple authors and being written in the early years of nahuatl alphabetic writing, contains numerous orthographic inconsistencies throughout the 12 books, with many words written in multiple distinct ways and decisions about word tokenisation not being standardised. furthermore, due to constraints on column width in the original manuscript, words are frequently split by line breaks with no indication of whether the following line continues the word from the end of the previous one. keyword searching this text is a seemingly-futile process involving determining all possible spellings for a given word and all possible tokenisations of a single syntactic fo. 1 jc macujlli amuxtli, vncan mo teneoa: intetzaujtl catca, inqujte tzammatia, in iehoantin mexica. jc ce capitulo, vncan mitoa, inte tzaujtl inqujtetzammatia: iniquac ioaltica, aca quicaquja: in chocaia: itla tequanj, inanoce, iuhquj aca ilamachocaia: auh iequene, intlejn qujtoaia, tonalpouhque: in mjtova ia tlamatinjme catca. i nievecauh, inoc tlateo toco: injuh quitoa, ce qujntin veuetque, ve ue tlaca: inoc titixca vitoque, inoc titomattoque, inoc to tlalpan tiqujtoaia: inaia aque tech nelotoque: inaiaque totlah onoque, yzqujtlamantli, y, in quj tetzamma tia, inquitetzaujttaia, in tetzaujtl ipan quimatia innetetzaujloia, in tlatetza ujaia. jnic centlamantli, iehoatl, iniquac aca quicaquia: in tequanj choca, te quanj ipan choca: injuhquj tecciztli qujpitza: tepetl qujnanqujlia: teh cuiooa, tlaoalanja: iuhqujn ilama pul choca. ynjn, qujtovaia ie iaomj qujz: anoço ie mjqujz, tlalmiqujz: anoço , ie itla com monamjctiz, ic word into multiple orthographic words. finally, nahuatl is a morphologically complex language with large amounts of inflection and derivation, making querying the surface/inflected form, instead of e.g., a lemma, particularly difficult. the present project attempts to address these issues by creating an open-source, retokenised, and normalised corpus of the fc with queryable linguistic annotations following the universal dependencies framework @xcite . in the following sections, we describe the corpus, each component involved in its creation, and an investigation into automating the processing. we conclude by outlining a road map for the project's completion and a vision of future applications. ## related work the fc has been the subject of a great deal of research in the humanities by scholars interested in the cultural beliefs and practices of the nahua people during the early colonial period @xcite @xcite @xcite . it has also served as a foundational component for work studying so-called "classical nahuatl," or nahuatl spoken during the period @xcite @xcite . both @xcite and @xcite leverage corpus-based approaches using a multitude of historical nahuatl documents, but it is unclear how much linguistic information was available in the corpus, and to our knowledge, this corpus has not been released to the public. @xcite released axolotl, a large, spanish-nahuatl parallel corpus with a focus on machine translation. it includes nahuatl from multiple variants and time periods, including the early colonial period, but does not include text from the fc. furthermore, the text in axolotl is unprocessed and unannotated. other corpora that include nahuatl texts include the johns hopkins university bible corpus @xcite , a parallel multilingual corpus that includes numerous contemporary nahuatl variants. this corpus has been used to produce morphosyntactically-annotated resources for a large number of languages @xcite . the first open morphosyntactically-annotated corpus of nahuatl was recently released by @xcite and includes 10,000 tokens of the western sierra puebla variety. following this work, we also select ud as our annotation schema. marc eisinger was the first to publish a computerised version of the fc, which is not freely available @xcite . the universidad autónoma de méxico (unam) hosts a website, temoa, containing a large volume of digitised colonial-era nahuatl texts, with minimal processing (at the very least, tokenisation problems in the fc appear to be corrected (universidad nacional autónoma de méxico, 2023). however, the copyright and rights to use for annotation and re-release are retained by unam, 1 making it not possible to create derivative works, such as the annotated corpus described in this paper. furthermore, the original text (before fixing tokenisation) is not available. related to the computational processing of colonial mexican texts, the "digging into colonial mexico" project @xcite involves the creation of a number of processed and machine-readable resources based on colonial mexican documents, mostly written in colonial-era mexican spanish. as for colonial texts written in mexican languages, the ticha project @xcite , a collaboration between members of zapotec-speaking communities and academics from universities in the united states of america, offers an "online digital text explorer" for colonial zapotec texts and includes morphological analyses and translations. ## corpus our corpus comes from a typed transcription upholding the original layout, published in the openaccess repository zenodo 2 to allow the semantic and computational study of the text from the primary source (de sahagún, 2022). in figure 1 we present a folio from the manuscript where the text in spanish (left) and nahuatl (right) is seen in two columns, and an example of the transcription output in our corpus. ## processing a major theme of the processing of the fc is the use of initial detailed hand-annotation in order to bootstrap automated approaches for the remaining text. crucially, the resulting corpus should be usable for academic research and, as such, must maintain the utmost quality. in this context, then, we consider automation a strategy to assist in human annotation, but still require manual auditing of the entirety of the annotated corpus. ## representations in this section we discuss a number of features of nahuatl that require special attention in the universal dependencies framework. ## automated processing we experiment with the existing processed fc data to see to what extent we might be able to automate the retokenisation and normalisation steps. following previous work showing that historical text normalisation can be modelled effectively as a character-based machine translation problem @xcite , we train an encoder-decoder seq2seq model with attention on character sequences for both tasks. while a natural inclination would be to train both retokenisation and spelling normalisation jointly, we are interested in storing each intermediate step for potential future research, and so train a separate model for each task. for the orthography normalisation model, we treat each word as a training instance, and map the unnormalised word (e.g. qujchioa) to its corresponding normalised form (e.g. quichihua). for the retokenisation model, training on each word would not work since the phenomenon we are modelling spans word boundaries. instead, we split the text on unambiguous punctuation ('.,:;?!'), creating numerous subsequences from each sentence. since the objective is to evaluate how well we could automate the text processing for future books, we used two of the three already-complete books (books 1 and 8) for training, and held out book 5 for evaluation. the models used a bidirectional lstm encoder, and training was done using opennmt @xcite . we trained both for 100 epochs. results of the experiments are listed in ## use cases in this section, we provide descriptions of a few research questions that could be informed by our corpus. the use cases are based on information that is available in the corpus and is not found in other editions of the manuscript. the first use case concerns the status of the tlamatinimeh 'sages, wise men' (lit. those who know things). it is widely claimed that there is no philosophy outside western philosophy @xcite , but this claim has been contested by scholars, starting from ángel maría garibay and his student miguel león-portilla who identify the tlamatinimeh with philosophers and argue that the precontact mexicans had long philosophical traditions @xcite . analysing individual words has since this work been the basis of understanding nahua thought. however, to date this process is difficult and error-prone as it involves carefully reading through unannotated concordances of surface forms and it is easy to miss examples that appear in forms that are unknown or unfamiliar to the researcher. our corpus will be able to help by allowing scholars to extract examples that are morphologically and syntactically related. for example, by allowing queries based on lemmas (encompassing for example tlamatini 'sage', tlamatinimeh 'sages', etc.) it will also allow for searching for specific syntactic constructions, such as those where a tlamatini is the subject of a speech verb. the second use case concerns concepts of time. for instance, consider @xcite 's statement about the mexica conceiving time and space as a single unit. he argues that the mexica did not separate time and space but had a perception of a "time-place" . this argument is based on the word cahuitl, which means 'time', and the intransitive verb cahui, which means "to stay or end". however, maffie argues that cahuitl is also related to the transitive verb cahua, which means "to leave or abandon", among other senses. the morphological annotations (including lemmas) in our corpus will allow for searching on lemma (to be able to distinguish forms of cahui from forms of cahua). and the syntactic annotations will allow for the extraction of time and place obliques that are dependents of those two verbs. ## concluding remarks we have outlined the strategies and approaches involved in creating a free and open, linguisticallyannotated corpus of the fc. having nearly completed retokenisation, orthographic normalisation, lemmatisation, part-of-speech tagging, and morphological analysis for 3 of the 12 books, we have established the key linguistic information to include in the corpus, and have engineered the foundations of the annotation process. results of our preliminary experiments into automatic annotation suggest that some tasks, like orthographic normalisation, can largely be automated with the existing data, whereas others, e.g., retokenisation, likely still require more labelled data and/or a more powerful architecture.