L ogogram NLP : Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP (2024)

Abstract: Standard natural language processing (NLP) pipelines operate on symbolic representations of language, which typically consist of sequences of discrete tokens. However, creating an analogous representation for ancient logographic writing systems is an extremely labor-intensive process that requires expert knowledge. At present, a large portion of logographic data persists in a purely visual form due to the absence of transcription—this issue poses a bottleneck for researchers seeking to apply NLP toolkits to study ancient logographic languages: most of the relevant data are images of writing. This paper investigates whether direct processing of visual representations of language offers a potential solution. We introduce LogogramNLP, the first benchmark enabling NLP analysis of ancient logographic languages, featuring both transcribed and visual datasetsfor four writing systems along with annotations for tasks like classification, translation, and parsing. Our experiments compare systems thatemploy recent visual and text encoding strategies as backbones. The results demonstrate that visual representations outperform textual representations for some investigated tasks, suggesting that visual processing pipelines may unlock a large amount of cultural heritage data of logographic languages for NLP-based analyses. Data and code are available at https: //logogramNLP.github.io/.

## introduction the application of computational techniques to the study of ancient language artifacts has yielded exciting results that would have been difficult to uncover with manual analysis alone @xcite . unsurprisingly, one of the biggest challenges in this domain is data scarcity, which, in turn, means that transferring from pre-trained systems on well-resourced languages is paramount. however, it is more challenging to adopt similar techniques for ancient logographic writing systems, translation ? ? ? ? ? ? ? ocr unicode latin photo handcopy visual textual input output ? 弃 慮 ? ? ? parse tree dating … m conventional visual (novel) figure 1 : illustration of the processing flow of old chinese (in bamboo script), an ancient logographic language, best viewed in color. m denotes the pretrained model used in the pipeline. vision-based models directly process visual representations (violet; dashed lines). conventional nlp pipelines (blue; solid lines) first convert visual representations into symbolic texteither automatically, which is quite noisy, or manually, which is labor-intensive. however, as shown, some ancient logographic writing systems have symbol inventories that have not yet been fully mapped into unicode. even when unicode codepoints exist, they are often mutually exclusive with the symbol inventories of high-resource languages, reducing the effectiveness of transferring from pre-trained models. finally, latinization (a potential solution for finding common ground with pre-training languages) loses information from the original input, is not fully standardized, and is difficult to automate. in which individual symbols represent entire semantic units like morphemes or words. the challenges associated with nlp for ancient logographic languages mainly come from two aspects. first, for many ancient languages, most available data sources are in visual forms, consisting of untranscribed photographs or hand-drawn copies (i.e., lineart). adopting the conventional nlp pipeline, which requires converting visual representations into symbolic text, is therefore not straightforward: automatic transcriptions are often noisy due to data scarcity, while manual transcriptions are labor-intensive and require domain expertise. some logographic writing systems, such as old chinese, even include symbol inventories that remain not fully mapped to unicode (depicted in figure 1 ). second, even when perfect unicode transcriptions are available, their symbol inventories are often mutually exclusive with those of high-resource languages, which can substantially reduce the effectiveness of transfer from pre-trained multilingual encoders, such as mbert @xcite . one processing step that might be used to mitigate this issue is latinization of the unicode transcripts @xcite . however, it is challenging to latinize logographic languages due to uncertain pronunciations @xcite and the resulting inconsistent latinization schemes across artifacts from the same language and writing system. such a process is laborioushumanists may devote months or even years to determine the correct transliteration. in contrast, once a correct transliteration is determined, translation into another language may only take minutes. fortunately, advances in visual encoding strategies for nlp tasks offer an alternative solution. recent studies have investigated nlp systems that model text in the pixel space @xcite @xcite , thereby opening new possibilities for the direct use of visual representations of ancient logographic writing systems. these approaches, to date, have primarily been applied to digitally rendered texts. they have not yet been extensively evaluated on handwritten texts, such as lineart, i.e., neatly handcopied versions of texts by scholars. in this paper, we attempt to answer the following questions: (1) can we effectively apply nlp toolkits, such as classifiers, machine translation systems, and syntactic parsers, to visual representations of logographic writing systems? (2) does this strategy allow for better transfer from pre-trained models and lead to better performance? additionally, as shown in figure 1 , many logographic languages have multiple partially processed representations, including artifact photographs, hand-copied lineart, unicode, latin transliteration, and normalizationwe also aim to empirically investigate the extent to which various representations at each stage, including textual and visual modalities, facilitate effective fine-tuning of downstream nlp systems. we have curated logogramnlp, a benchmark consisting of four representative ancient logographic writing systems (linear a, egyptian hiero-glyphic, cuneiform, and bamboo script), along with annotations for fine-tuning and evaluating downstream nlp systems on three tasks, including three attribute classification tasks, machine translation, and dependency parsing. we conduct experiments on these languages and tasks with a suite of popular textual and visual encoding strategies. surprisingly, visual representations perform better than conventional text representations for some tasks (including machine translation), likely due to visual encoding allowing for better transfer from cross-lingual pre-training. these results highlight the potential of visual representation processing, a novel approach to ancient language processing, which can be directly applied to a larger portion of existing data. ## dataset: languages, tasks and challenges our benchmark consists of four representative ancient languages-linear a, egyptian hieroglyphic, cuneiform, and bamboo script ( §2.1). foot_0 each language is associated with a unique writing system and unique challenges. we refer the readers to appendix a for data collection and cleaning details. our benchmark covers three tasks: machine translation, dependency parsing, and attribute classification ( §2.2). ## methods in this section, we will describe feature encoding methods ( §3.1) for both visual and textual inputs, as well as task-specific layers ( §3.2) for each task we consider. ## experiments and analysis we describe our general model fine-tuning approach in §4.1 and analyze model performance on the aforementioned tasks in the succeeding subsections. ## ablation study on ocr and image quality as mentioned earlier, the majority of data from ancient times remain in the form of photographs. we first closely examine two different visual input representations for the zho-en translation task, handcopied figure and photograph ( §5.1). next, we examine ocr performance on ancient logographic languages to gain better understanding of this bottleneck for current nlp pipelines ( §5.2). ## related work because ancient languages are often low-resource, they present challenges that are closely related to other domains of nlp, such as low-resource machine learning and multi-lingual transfer learning. recent work has explored the application of nlp techniques to ancient languages from the following perspectives: multilingual transfer learning and disjoint character sets. @xcite studied hard-toprocess living languages such as uyghur, and reported that a non-contextual baseline outperforms all pre-trained lm-based methods. ancient languages also face the same problem, with even less data available. a major challenge that is mostly specific to ancient logographic languages, however, is the almost non-existent overlap of their symbol inventories with those of high-resource languages. visual representation of languages. recently, several works have studied language processing based on images of text. @xcite pretrained a masked language model on digitally rendered text and achieved comparable performance with text-based pre-training strategies on downstream tasks. @xcite found that a multi-lingual translation system with pixel inputs was able to outperform its textual counterpart. machine learning for ancient languages. @xcite surveyed the status of pipelines for ancient language processing. notably, the study concludes that applying machine learning methods to ancient languages is bottlenecked by the cost of digitization and transcription. according to the missing scripts project, foot_10 only 73 of 136 dead writing systems are encoded in unicode. ancient languages, such as ancient greek or latin (bamman and burns, 2020), benefit greatly from multilingual pre-training techniques, such as mbert, xlm-r @xcite , and bloom @xcite . the applicability of these techniques is limited when it comes to languages that were historically written in obsolete or extinct writing systems-for instance, languages like sumerian and elamite were recorded in cuneiform script and ancient chinese was inscribed on oracle bone or bamboo. however, observations by existing work support the potential utility of visual processing pipelines for ancient languages. logographic writing systems. logography typically denotes a writing system in which each glyph represents a semantic value rather than a phonetic one, however, all the languages studied in our paper have at least some phonetic component based on the rebus principle. this paper emphasizes ancient logographies that (i) possess extensive glyph inventories; (ii) feature glyphs with multiple transliterations or functional uses; and (iii) are low-resource and/or remain in photo format @xcite @xcite . existing research on logographic languages has predominantly focused on those that are well-resourced and still in use, such as modern chinese @xcite , or on data that has already been carefully transcribed into latin or annotated with extra semantic information @xcite @xcite . our paper aims to address the gap in resources (new data) and methodologies (visual-only approaches) for encoding and analyzing ancient logographic languages, which are crucial for a more comprehensive understanding of historical linguistic landscapes. ## conclusion by comparing the results on four representative languages on three downstream tasks, we demonstrated the challenges faced in applying natural language processing techniques to ancient logographic writing systems. our experiments demonstrate, however, that encoding more readily available visual representations of language artifacts can be a successful strategy for downstream processing tasks.