Analyzing Finetuned Vision Models for M ixtec Codex Interpretation (2024)

Abstract: Throughout history, pictorial record-keeping has been used to document events, stories, and concepts. A popular example of this is the Tzolk’in Maya Calendar. The pre-Columbian Mixtec society also recorded many works through graphical media called codices that depict both stories and real events. Mixtec codices are unique because the depicted scenes are highly structured within and across documents. As a first effort toward translation, we created two binary classification tasks over Mixtec codices, namely, gender and pose. The composition of figures within a codex is essential for understanding the codex’s narrative. We labeled a dataset with around 1300 figures drawn from three codices of varying qualities. We finetuned the Visual Geometry Group 16 (VGG-16) and Vision Transformer 16 (ViT-16) models, measured their performance, and compared learned features with expert opinions found in literature. The results show that when finetuned, both VGG and ViT perform well, with the transformer-based architecture (ViT) outperforming the CNN-based architecture (VGG) at higher learning rates. We are releasing this work to allow collaboration with the Mixtec community and domain scientists.

## introduction vast amounts of historical and cultural documents are encoded in pictographic systems @xcite . representations such as egyptian hieroglyphics use pictorial representations corresponding to words and sub-word components to express concepts. other pictorial systems that display scenes that evoke a known narrative have also been used throughout the world. rules govern the depiction of years, dates, names, class, ceremonies, and gender @xcite . the implicit grammatical rules can contribute to a deterministic interpretation of these ancient narratives. mixtec codices are highly structured and have fairly rigid conventions for the representation of people @xcite , such as loincloths on men and skirts on women. consequently, the depiction of persons in these codices follows consistent patterns. unfortunately, due to the ravages of time and conflict, only a few of these codices are presently available. computational analyses of the codices and their underlying structures may help researchers better understand the remaining works. in this paper, we explore how models such as vgg-16 @xcite and vit-16 @xcite perform when used to classify these lowresource patterns and understand the features they find important in this task. ## mixtec codices the researchers labeled data from three popular sources: the codices vindobonensis mexicanus i @xcite ; unbekannt, 1449), selden @xcite , and zouche-nuttall @xcite . codex vindobonensis mexicanus i describes both the mythological and historical founding of the first mixtec kingdoms, codex selden follows the founding of the kingdom of jaltepec and its ruler, lady 6 monkey, and codex zouche-nuttall primarily illustrates the life and conquests of lord 8 deer jaguar claw, but also details the histories of his ancestors. other mixtec codices are extant, but their condition is degraded and not amenable to our current machine-learning pipeline. each codex is made of deerskin folios, and each folio comprises two pages. the codex vindobonensis mexicanus i contains 65 pages, selden 20 pages, and the zouche-nuttall facsimile edition 40 pages. we chose to use the zouche-nuttall facsimile edition over the complete 84-page edition because of its restored quality and high-quality scans available. ## experiment we describe the preprocessing, finetuning, and execution steps of this pipeline. we explore the hyperparameter space for each model first to find the optimal configuration to use during execution. ## model evaluation vit performs consistently higher than vgg for these different learning rates, however, both returned strong results for each metric. the testing results for both vit and vgg were high with a small standard deviation, around 98% and 1% standard deviation for both (see ## discussion the purpose of the experiments is to explore two research questions, namely: can cnn and transformer-based models be finetuned to classify figures from a mixtec codices dataset? and does the model identify the same features experts do? to answer the first question, we analyze and compare the performances of both the pretrained vit and vgg models. both models achieve great results across training, validation, and testing phases when using an appropriate learning rate. smaller learning rates require more epochs to converge, as the steps are smaller, but are less likely to miss a minimum loss. on the other hand, larger learning rates require fewer epochs, but may not converge. as we can see in figure 3 , vit converges for almost all learning rates, and so could be used in environments where compute resources are lacking. features and literature. we assigned reference images for each class (man and woman, and standing/not standing) to understand which features each model learned, as well as to compare these learned features to those highlighted by experts. during training, we generated visualizations of activation and attention per pixel to view how the models learned important features over time. in the left image in figure 1 , the vit model assigned higher attention to areas corresponding to loincloths on man. on the right, vit shows increased attention to the poncho area on a woman. we confirm that these are both features noted by domain experts @xcite . to verify that the model is indeed identifying the same features noted in literature, we masked attributes on the reference images. these features were earlier noted as discriminatory for human evaluators labeling gender: loincloths and anklets for man, and braided hair and dresses for woman. we extended our reference image set by adding three variations to each image: either blocked hair, blocked skirt, or both for woman. this process was replicated for the two features indicative of man. we then tested the finetuned models on the unblocked and blocked reference images and generated class activation and attention maps. vit correctly predicted 100% of the unblocked reference images, 79% of the singly blocked images, and 63% of the double blocked images. figure 1 shows the activation maps of the doubly blocked images. the model fails to find defined areas of attention. this again verifies that the model is learning features defined in literature. ## summary in this paper, we presented a low-resource dataset of figures from three mixtec codices: zouche-nuttall, selden, and vindobonensis mexicanus i. we extracted the figures using segment anything model and labeled them according to gender and pose, two critical features used to understand mixtec codices. using this novel dataset, we finetuned the last few layers of cnn and transformer-based foundational models, vgg-16 and vit-16 respectively, to classify figures as either man or woman and standing or not standing. we found that both models have high accuracy with this task, but that vit-16 may be more reliable for varying learning rates. we confirmed that the models are learning the features said to be relevant by experts using class activation maps and targeted blocking of said features. given that these models can reliably classify figures from a low-resource dataset, this research opens the door for further processing and analysis of mixtec codices. the codices themselves are highly structured and carry a narrative woven through each scene. finetuned state-ofthe-art models could be combined to classify segmented figures within a scene, as well as classify the relationship between figures. these relationships would then be used to extract the narrative from a codex, as defined by domain experts.