Learning the Character Inventories of Undeciphered Scripts Using Unsupervised Deep Clustering (2023)

Abstract: A crucial step in deciphering a text is to identify what set of characters were used to write it. This requires grouping character tokens according to visual and contextual features, which can be challenging for human analysts when the number of tokens or underlying types is large. Prior work has shown that this process can be automated by clustering dense representations of character images, in a task which we call “script clustering”. In this work, we present novel architectures which exploit varying degrees of contextual and visual information to learn representations for use in script clustering. We evaluate on a range of modern and ancient scripts, and find that our models produce representations which are more effective for script recovery than the current state-of-the-art, despite using just ~2% as many parameters. Our analysis fruitfully applies these models to assess hypotheses about the character inventory of the partially-deciphered proto-Elamite script.

## introduction one of the first tasks in decipherment is to solve an instance of the token-to-type problem by recognizing which tokens represent the same underlying character, and thereby to construct a list of every character used in the script (cf. @xcite . accurate character inventories are important for decipherment, as they indicate patterns of frequency and adjacency which can reveal information about the underlying message. however, it can be challenging for human annotators to determine which characters are truly distinct: tokens with different appearances can represent the same underlying character (such as english and ), while visually-similar tokens can represent distinct characters (such as and or ß and b). this work introduces novel, vae-based techniques for learning the character inventory of an unknown script by clustering images of character tokens. we show, through a range of experiments on deciphered and undeciphered scripts from modern and ancient corpora, how the complexity and number of characters in a script impact our models' ability to learn the underlying character inventory. on the ancient cypro-greek syllabary, our models outperform the recent sign2vec architecture @xcite despite using just ∼2% as many parameters. we also apply these models to the undeciphered proto-elamite script (pe; dahl 2019), and find that they independently replicate expert intuitions about the underlying character inventory and suggest new relationships between signs which have not yet been noted in prior work. our work follows the same overall approach, and investigates how changes to the encoder architecture, data quality, and training process can affect the final clustering. ## data we aim to apply these models to the study of the undeciphered proto-elamite script, which is attested across ca. 1581 clay tablets recovered from the ancient city of susa and other parts of the iranian plateau. these texts have been transliterated by domain experts using a work-in-progress list of about 1500 distinct signs; foot_1 the most complete and up-todate transliterations are hosted by the cuneiform digital library initiative foot_2 and described in a recent survey by j. @xcite . each sign has an accompanying digital image, also produced by dahl, depicting its "archetypal" form. these images smooth over many of the irregularities of the original shapes drawn on clay, while still preserving slight visual differences between tokens which may actually represent the same underlying character (such as and ). they therefore represent an intermediate level of detail that is cleaner than segmented images of the original texts, yet still faithful to the original hand. we convert the entire transliterated proto-elamite corpus foot_3 into a set of image sequences by replacing each transliterated sign name with the corresponding sign image (figure 5 ). we evaluate our models on their ability to recover three scripts whose character inventories are already known (english, japanese, and cypro-greek). we construct an english dataset by extracting the first 33k alphanumeric tokens (approximately the same number of tokens as are attested in proto-elamite) from the wikitext-2 corpus @xcite . we convert this text into image sequences by replacing each character token with a handwritten image of that character. we use images from de campos et al. 2009, who provide 55 handwritten instances of all 62 upper-and lowercase english letters and digits: one of these 55 images is chosen at random each time a character occurs. the resulting sequences (figure 5 ) imitate the level of detail in our proto-elamite data, in that each letter is attested by multiple distinct images, and the same image can be used for multiple tokens. we construct a japanese dataset according to the same procedure, using the first 33k tokens from the japanese tatoeba corpus @xcite . as we do not have handwritten character images for japanese, we instead extract the glyphs from two japanese fonts (yuji boku and zen old mincho). the japanese writing system uses three scripts: kanji which are highly logographic, and two syllabaries called hiragana and katakana. similarly, proto-elamite has been speculated to contain a set of possibly-syllabic signs, together with a large number of logograms @xcite . syllabic signs convey phonetic information that can provide crucial insights for decipherment, and are therefore a major focus of decipherment efforts on this script. in our japanese experiments (see section 4), we therefore train on the entire script, but only evaluate the model's ability to recover the two syllabaries. we use the same cypro-greek data as @xcite . unlike the other datasets, this uses manually-segmented images from hand-drawn copies of artifacts bearing the cypro-greek script. there is therefore a greater degree of variation between the character shapes in this data, and no two tokens ever have identical images. this means that our three datasets fall along a cline from fully naturalistic, handwritten sequences (cypro-greek), to synthetic sequences derived from handwritten images (english), to synthetic sequences derived from digital fonts (japanese). we trim extraneous whitespace from all character images and resize them to 64 × 64 pixels with a single grayscale color channel before training. 5 4 experiments we train each of the models from section 2.2 on the four corpora detailed above (see appendix b for hyperparameters and additional training details). after training, we encode each image using the trained encoder and cluster the resulting encodings using (i) agglomerative clustering with varying numbers of clusters (english, japanese, proto-elamite) or (ii) dbscan @xcite with varying ε (cypro-greek). we use dbscan for cypro-greek to enable a fair comparison against the results in @xcite ; however, we find that dbscan is generally not effective when clustering the other scripts. when clustering with dbscan, we follow @xcite in using a minimum cluster size of 2, to imitate a decipherment setting where the true number and frequency of characters is unknown; for the other scripts we vary the number of clusters for the same reason. 6 for english, japanese, and cypro-greek, we report homogeneity, completeness, and vmeasure @xcite relative to the gold labels. homogeneity ranges from 0 to 1, where 1 means that each cluster contains instances from just one of the underlying characters, and smaller values imply that some clusters combine instances of two or more distinct characters. similarly, a completeness of 1 means that each of the underlying characters is represented by a single cluster, while smaller values mean that some characters have been divided across multiple clusters. intuitively, low homogeneity scores mean that a clustering merges together characters which are actually distinct, while low completeness means that it splits some characters into subgroups that are not underlyingly distinct. v-measure is the harmonic mean of homogeneity and completeness. dbscan can label samples as outliers (and thus, not part of any cluster): we only evaluate on tokens which it assigns to a cluster. in our japanese experiments, we only evaluate on hiragana and katakana, in imitation of the proto-elamite setting where we eventually aim to understand the divisions of a putative syllabary comprising only a subset of the overall script. in our cypro-greek experiments, we compare against the sign2vec and deepcluster-v2 results reported in @xcite . for the other scripts, we compare against agglomerative clusterings over the input images. in proto-elamite, where the ground truth is not known, we perform a qualitative evaluation in collaboration with domain experts. we look for sets of tokens which are assigned to the same cluster by our vae+neighbor, vae+lstm, or vae+transformer model, but belong to different clusters in the vanilla vae model. the vanilla vae differs from the other models in that it lacks contextual information; therefore, any groupings which are absent from this model's output likely reflect primarily contextual similarities. contextual resemblances are harder for human annotators to notice than visual resemblances, and so we expect these groupings to reflect similarities which may have been overlooked in prior work. we collaborate with domain experts to assess how these token groupings relate to their intuitions about this script. ## related work scribal hand identification @xcite is a related task which seeks to cluster instances of characters from a known script according to the hand which wrote them. @xcite describe a system for segmenting, transliterating, and deciphering images of historical manuscripts. in the transliteration step, their model implicitly learns an underlying script by clustering character representations obtained from a siamese neural network trained to discriminate between characters from known scripts. this network learns character representations without access to context, similarly to our vanilla vae and the deepcluster-v2 baseline in @xcite . in a setting where the underlying script is already known, dencker et al. (2020) and gordin et al. (2020) also describe systems for automated transliteration from images of cuneiform text. @xcite study the character inventory of proto-elamite using a model similar to our vae+lstm. their model is not variational; it uses softmax decoding over a fixed vocabulary initialized to match the working signlist, which biases it towards recovering the same divisions speculated by experts. our models use a deconvolutional decoder, which sidesteps this bias by allowing an open vocabulary. their evaluation does not test on any known scripts, which makes it challenging to determine the accuracy of the clusters their model produces. ## conclusion we have described four models which add varying degrees of contextual information to a vae, and have shown how these can be used to cluster token images to recover a script's character inventory. on the ancient cypro-greek script, our best models meet or outperform the state-of-the-art sign2vec baseline using just ∼2% as many parameters, which supports our claim that written text lacks the visual complexity to warrant models of the depth used in other image processing applications. our english and cypro-greek experiments also demonstrate that contextual models are more effective for script recovery than contextless models. on synthetic japanese data, which contains many distinct graphemes but little variation between allographs, our models achieve extremely high v-measure (>0.91), suggesting that they handle large character inventories more easily than they handle allography. we apply our models to study the undeciphered proto-elamite script, and show that they capture existing intuitions about this script as well as suggest new parallels between signs which have never been noted in prior work. our best insights for proto-elamite come from the lstm and transformer models, while for cypro-greek our vae+neighbor model is the only one which produces a clustering with precisely the same number of clusters as there are signs in the underlying script. this indicates that it is useful to consider models with varying access to contextual information according to the number of long-distance contextual dependencies the input script is expected to exhibit.