Sequence Models for Document Structure Identification in an Undeciphered Script (2022)

Abstract: This work describes the first thorough analysis of “header” signs in proto-Elamite, an undeciphered script from 3100-2900 BCE. Headers are a category of signs which have been provisionally identified through painstaking manual analysis of this script by domain experts. We use unsupervised neural and statistical sequence modeling techniques to provide new and independent evidence for the existence of headers, without supervision from domain experts. Having affirmed the existence of headers as a legitimate structural feature, we next arrive at a richer understanding of their possible meaning and purpose by (i) examining which features predict their presence; (ii) identifying correlations between these features and other document properties; and (iii) examining cases where these features predict the presence of a header in texts where domain experts do not expect one (or vice versa). We provide more concrete processes for labeling headers in this corpus and a clearer justification for existing intuitions about document structure in proto-Elamite.

## introduction proto-elamite (pe) is a largely undeciphered script of the early bronze age, inscribed on clay tablets unearthed in iran. pe shares certain features, most notably its number systems, with another ancient script called proto-cuneiform: these similarities have allowed for a partial decipherment which informs current understandings of the texts as administrative accounts recording amounts of various goods and personnel. figure 1 shows a typical text with annotations explaining how it is divided into columns and entries. @xcite gives a thorough survey of pe from an archaeological perspective; @xcite introduce the corpus to technical audiences and describe initial computer-assisted exploratory analyses. specialists have hypothesized that pe texts frequently begin with a "header", that is, a sign (or string of signs) which "qualifies all transactions recorded in a text" by specifying an institution or owner in charge of the associated account (damerow and @xcite . this understanding of headers depends in part on the claim that they correspond to visually demarcated "colophons" in proto-cuneiform accounts @xcite , which are however also largely undeciphered and therefore not certain to consistently convey ownership information. some (but not all) of the signs that occur at the beginning texts have been tentatively labeled as headers by domain specialists. this labeling is recorded using comments in the transliteration of the texts; no explicit list of header signs has been published. the clearest example of this putative category is the ubiquitous sign m157 , which occurs at the start of fully one-fifth of all pe accounts. most header signs, including m157, may also appear elsewhere in a text with uncertain function. in light of modern scholarship's very partial understanding of the pe corpus, there does not seem to be proof beyond reasonable doubt that headers record ownership, much less that all headers do so. moreover, headers have thus far been identified through manual analysis which has not been fully documented in any publication, and some of the experts who originally identified this category are no longer alive. thus the criteria for identifying headers are opaque and the question of their existence is a matter of qualitative judgement. in this work, we combine computer-aided analysis with domain expertise to undertake the first focused study of headers in pe. we use statistical and neural sequence models to show that headers are a genuine structural phenomenon of pe. we independently replicate manual annotations from past work with high accuracy, and our models also identify and allow us to correct a number of annotation mistakes. based on our results, we argue against the conventional understanding that headers nearly always span only one sign. in conjunction with this, we show that signs in the first and second positions of a text predict distinct information, suggesting they have distinct functions. ## data the vast majority of pe texts have been made publicly available online by the cuneiform digital library initiative (cdli 1 ) in the form of (i) highquality digital images, (ii) hand-drawn line-art, and (iii) ascii transliterations representing experts' current understanding of each text. the transliterations employ a working signlist comprising numbered signs from m001 through m521 (m acknowledging the original standardisation of the signlist by scholar p. meriggi). tilde annotations (as in m001∼a) represent possible variants of a sign; experts remain agnostic as to whether these variants are distinct characters. texts are identified by "p" numbers assigned by the cdli (figure 1 caption). @xcite publish a cleaned snapshot of the transliterated corpus together with a library of tools for interacting with the texts: this snapshot contains roughly 26k readable tokens across 1467 transliterated texts. the data for the present work is a version of this snapshot where transliteration 1 cdli.ucla.edu errors (e.g. transliterations that do not accurately reflect a tablet's photograph) were fixed by domain experts. foot_0 ## methodology the cdli transliterations include rich annotations, including which signs (if any) are understood to comprise a text's header. we propose to train two unsupervised sequence models on the pe corpus and assess whether these models suggest any internal structure at the beginning of these texts. we also aim to evaluate whether and to what extent the features learned by these models can be used to recover the expert annotations, to establish whether these models are identifying the same structure posited by experts. we aim to arrive at a richer understanding of the meaning and purpose of headers by (i) examining which features are useful for predicting this category; (ii) finding correlations between these features and other document properties; and (iii) identifying why the models disagree with (or fail to recover) the human labeling if such disagreements occur. we hope to provide new and quantifiable evidence that headers are a real structural phenomenon in pe, and to concretely justify why any given may have or not have a header. overall, we seek to assess and understand the human labeling rather than to indiscriminately replicate it. ## analysis our results are fully consistent with the prevailing assumption that the beginnings of certain pe tablets exhibit some degree of internal structure. this is suggested by the existence of an hmm state which is strongly localized to the beginning of tablets, but which does not occur at the beginning of every text as a generic "start" state. further evidence is seen in the behaviour of the transformer, where in certain texts the model pays more attention than usual to early tokens. on their own, these features merely confirm that some internal structure is present, but do not tell us what that structure may represent. in this section we interpret our models' predictions in order to understand what factors may have motivated the original human annotations, and what features may be exploited to understand headers' meanings. ## related work hmms have a storied pedigree in the field of decipherment, being first used (under codename ptah) by members of the nsa to analyze the voynich ms @xcite . as this work was originally classified, most hmm-based approaches to decipherment instead trace back to @xcite who demonstrate the effectiveness of hmms on a range of unsupervised decipherment tasks, and whose framework is adopted or used as a baseline in a significant volume of later work @xcite @xcite @xcite . we are not aware of any work which has employed transformers or other neural architectures as feature extractors for a comparable, unsupervised analysis of undeciphered text. ## conclusion this work offers the first and most exhaustive assessment of proto-elamite headers in order to inform the ongoing decipherment of this ancient script. we have demonstrated that two distinct unsupervised sequence modeling techniques exhibit unique behaviours at the beginning of some proto-elamite texts. these behaviours are consistent with, and offer independent evidence in support of, the prevailing hypothesis that these documents begin with a header. the features recovered by these models predict with up to 95% accuracy whether experts understand a text to contain a header. this inspires confidence that the expert labels have been applied according to a consistent logic and following salient structural features of the text. our error analysis has also allowed us to identify and emend 25 mistakes in the expert annotations, expanding the total number of headers in the corpus by nearly 4% and reducing the amount of noise in a low-resource dataset where small errors may have an outsize effect. we have demonstrated that there are measurable differences between the contextual embeddings learned for signs labeled as headers versus those in other contexts, reaffirming that these signs are somehow functionally distinct from the rest of the script. using self-attention scores from a transformer language model, we have demonstrated that the second sign of a text predicts the presence of a header more accurately than the first sign; we have also shown that state sequences from an hmm suggest that many more multi-sign headers exist than were previously assumed. on the basis of these results we have argued against the conventional understanding that header information is localized to a single sign, and suggest that headers may commonly span two or even three signs in some texts. finally, we identify correlations between sign usage at the beginning of a text and other features such as genre, seal impressions, and the presence of a subscript. these correlations suggest that the first sign of a text captures more extra-textual information than later signs, and that if multi-sign headers exist, their two (or more) constituent signs likely convey distinct kinds of information.