Classification of Paleographic Artifacts at Scale: Mitigating Confounds and Distribution Shift in Cuneiform Tablet Dating (2024)

Abstract: Cuneiform is the oldest writing system used for more than 3,000 years in ancient Mesopotamia. Cuneiform is written on clay tablets, which are hard to date because they often lack explicit references to time periods and their paleographic traits are not always reliable as a dating criterion. In this paper, we systematically analyse cuneiform dating problems using machine learning. We build baseline models for both visual and textual features and identify two major issues: confounds and distribution shift. We apply adversarial regularization and deep domain adaptation to mitigate these issues. On tablets from the same museum collections represented in the training set, we achieve accuracies as high as 84.42%. However, when test tablets are taken from held-out collections, models generalize more poorly. This is only partially mitigated by robust learning techniques, highlighting important challenges for future work.

## introduction computational paleography @xcite is a growing interdisciplinary field that uses computational algorithms to decipher and analyse ancient writing systems. we investigate using machine learning to automate large-scale dating of cuneiform 1 , the oldest writing system from around 3,500 bce. similar to general chronicle attribution tasks in paleography, cuneiform dating involves classifying cuneiform tablets into specific time periods rather than precise years. for example, figure 1 shows a tablet comes from ur iii. different from other historical languages, such as ancient greek @xcite or ancient arabic @xcite , cuneiform tablets are more challenging to convert into a machine readable format because the writing system continually evolved over the 3,000 years it was in use. tablets from different collection (museum or private collector) usually in different time period distribution and there is confound (undeier features to machine learning models) from different cameras. the transliteration is usually exhibit bias towards specific time periods. for many writing systems, historians and paleographers have been able to identify distinguishing features in textual content and writing style that allow for inferences about date of origin for individual artifacts. for some writing systems, these processes have even been automated with machine learning to some extent. for example, @xcite showed encouraging results using neural networks trained on ancient greek text to restore and date digitized ancient greek artifacts. can we train similar textual models for cuneiform dating using accompanying manual transcriptions or transliterations? we conduct experiments with a series of light-weight recurrent models that show this is indeed possible. however, relying on manual transcriptions for the purpose of dating is somewhat circular: for cuneiform, transcription and transliteration is as time-intensive as manually dating tablets. further, transliterations themselves might exhibit bias-for example, an expert's approach to transliterating a tablet may al-ready be influenced by preconceived notions about its time period-allowing models to overfit to the tendencies of individual transliterators. thus, we also study whether visual representations of cuneiform tablets can be used effectively for automatic dating. visual representations skirt the issues of manually-intensive transcription and confounds due to transliteration style. further, visual representations may even allow models to automatically extract information about the visual style of writing, which paleographers have found useful for manual dating. in past work, @xcite has shown that relatively accurate dating of cuneiform tablets using 3d scans is possible. however, currently it is not feasible to produce 3d scans of over 100,000 remaining tablets, which are dispersed among museums and private collections around the world. therefore, instead we explore the use 2d photographs from cdli @xcite to address the dating problem-a task that as far as we are aware has not been previously studied. our experiments using convolutional neural models trained on 2d images demonstrate a new problem however: the different imaging setups used by different collections presents a confound that leads to poor generalization (shown in figure 3 ). we find that the gap between performance on tablets from collections that were attested in training data versus those that were not is extremely large. thus, we also evaluate to what extent robust learning methods that attempt to address out-of-distribution (ood) generalization can mitigate this issue. we find that while these methods do help, they do not increase generalization to the point where accurate dating of tablets from unseen collections can be performed reliably. thus, our empirical study highlights this important challenge as an area for future research. we summarize our primary contributions below: in the following sections, we first formulate the problem and then describe the data collection splits we created to address our core research questions. ## problem formulation technically, the dating task can be formulated as either a classification or a regression problem. however, after careful examination, we concluded that treating inferred dates as continuous variables (using regression) does not make sense in this domain because the annotation standard used for manual dating (the source of supervision for learning and evaluation) includes date categories with overlapping time intervals (see figure 5 ). instead, we represent each time period as a categorical class id and treat dating as a multi-class classification problem. next, we layout the core research questions we attempt to answer in this empirical study. to address each, we will carefully design data splits that contain three separate test sets, each measuring a specific aspect of ood generalization, along with a train and validation set. rq1: what models, configurations, and features-either visual or textual-are most effective for automatically dating cuneiform tablets? rq2: how much of a problem do ood effects pose for generalization in this domain? for example, do models overfit to specific features present in individual museum collections? how well do models generalize to tablets from previously unseen museum collections? rq3: how well do existing robust learning techniques address the issue of distribution shift and ood generalization in the context of cuneiform tablet dating? in later sections, we will specify the datasets we use, which specific input representations we compare, and which modeling approaches generally speaking, distribution shift occurs whenever the underlying distribution that generated the training data diverges from the distribution that will generate future test instances. distribution shift poses a substantial challenge for learning systems: patterns that hold true on the training data may not generalize to the test set, leading to poor generalization performance. in the domain of cuneiform data the are two important types of distribution shift. first, cuneiform datasets tend to exhibit substantial label shift due to how tablets are distributed across museum collections. we depict the distribution of tablet dates in museum collections in the ed i-ii ed iiia ed iiib early old babylonian ebla lagash ii middle assyrian middle babylonian neo-assyrian neo-babylonian old akkadian old assyrian old babylonian ur iii time period distribution shift from training to testing set split training ood testing cuneiml dataset (chen et al., 2023) (which we use in experiments) in second, the input representations from individual museum collections may have properties that make the collection itself identifiable. for instance, as shown in figures 1 and 3 , the scanning methodologies used by separate museums leave artifacts like different amounts of color saturation and blurring. similarly, it is possible that different transliteration styles may also be identifiable. because individual collections are biased towards specific date ranges, the confounds mentioned above may cause covariate shift-a type of distribution shift where the distribution on the input variables and the relationship between input and output vary between train and test. for example, a model may learn to identify the collection based on properties of the scanning hardware in order to determine date. this may work on training data, but will not generalize ## data we obtain 38,937 tablet images with transliterations from cdli (cdli contributors, 2024), using prepossessing from cuneiml @xcite . an example is shown in figure 6 . besides transliteration and 2d images, we use several other attributes from the metadata entries, including provenience, collection, and genre, which we use in later experiments for both simple baselines and as additional supervision to mitigate distribution shift. ## methods we describe the baseline models we used in experiments and also several training strategies, adversarial regularization and , to mitigate the distribution shift issues. ## conclusion in this paper, we explore end-to-end cuneiform dating at scale using machine learning. we have identified three major challenges-label imbalance, distribution shift, and circular reasoning-that are prevalent in cuneiform dating. these issues and solutions explored in our paper are broadly applicable to the classification of other ancient artifacts as well. we hope our initial analysis will inspire the community to further adopt machine learning for addressing problems in ancient language processing. samira zare and hien van nguyen. 2022. removal of confounders via invariant risk minimization for medical diagnosis. in medical image computing and computer assisted intervention -miccai 2022, pages 578-587, cham. springer nature switzerland.