Neural Machine Translation of Artwork Titles Using Iconclass Codes (2020)

Abstract: We investigate the use of Iconclass in the context of neural machine translation for NL<->EN artwork titles. Iconclass is a widely used iconographic classification system used in the cultural heritage domain to describe and retrieve subjects represented in the visual arts. The resource contains keywords and definitions to encode the presence of objects, people, events and ideas depicted in artworks, such as paintings. We propose a simple concatenation approach that improves the quality of automatically generated title translations for artworks, by leveraging textual information extracted from Iconclass. Our results demonstrate that a neural machine translation system is able to exploit this metadata to boost the translation performance of artwork titles. This technology enables interesting applications of machine learning in resource-scarce domains in the cultural sector.

## introduction in the age of mass-digitization, cultural heritage institutions put significant effort in making their (meta) data available to developers and researchers. artificial intelligence, and machine learning in particular, increasingly plays an important role in this process @xcite . recent case studies have demonstrated successful applications of machine learning methods to cultural heritage collections. most of this work relies on advances in computational methods and utilizes a modelling framework known as deep neural networks @xcite . however, such algorithms are dataintensive and require large annotated datasets, which recently have become available in some fields @xcite @xcite . these datasets contain millions of training items, which allowed researchers to achieve impressive results in many tasks. however, the construction of such materials in the domain of cultural heritage material is an even more expensive process, as it requires the intervention of highly-trained subject experts. hence, many institutions can only offer smaller datasets, that contain just a fraction of the number of training examples that are needed to train a deep learning algorithms. transfer learning is a common solution to overcome such a lack of training data @xcite . in neural machine translation (nmt), networks are nowadays commonly pre-trained on large generic datasets of parallel sentences, before they get fine-tuned on a more specific "downstream" corpus. such networks, however, are conventionally only exposed to the actual sentence pairs in the target domain and are ignorant of additional knowledge that might be available such as, for example, iconographic metadata about objects and their relations. in the case of artworks, computational methods that can exploit such additional knowledge are highly appealing. this work aims to apply nmt in the context of cultural heritage metadata using iconclass @xcite as a source of external knowledge. iconclass (see section 3.1.1) contains keywords and definitions of subjects represented in artworks. we propose a simple approach to integrate this external knowledge in an nmt architecture for artwork titles to improve the translation performance. the structure of this paper is as follows. we first present the related work in section 2. then, we describe the datasets and present the applied methods in more detail in section 3. next, we present the results of our case study and discuss them in section 4. finally, we summarize our main contributions and findings with proposals for future work in section 5. ## related work modern nmt systems nowadays often work at the level of an individual sentence pair and aim to translate a source sentence into a target sentence, without making use of additional information other than the source sentence itself. the idea to concatenate a source sentence with additional information, however, is not new. this preprocessing step is appealing due to its simplicity and model-agnostic applicability. previous approaches in this respect are generally divided into 2 categories (see examples in inspired by this previous work, we investigate the use of iconclass in the context of artwork title translations. we use the definitions and keywords associated with iconclass codes to extend and augment the artwork titles. our main contribution is that we demonstrate that iconclass definitions, when provided within a data augmentation strategy, improve translation performance. extended context source cc sieh cc , cc bob cc ! -wo sind sie? target -where are they? data augmentation source how long does a cold last? || combien de temps dure le vol? ## methods in this section, we describe the datasets and the methods that we utilized in our research. we justify our choice for the particular nmt model used and provide details on the experimental settings and evaluation measures. ## results and discussion we present our quantitative results in section 4.1. we divide our experimental results into two different sections. first, we assess the use of the extended context in section 4.1.1. second, we compare the various data augmentation strategies to the baseline in section 4.1.2. in section 4.2 we manually inspect a selection of outputs for the best performing model. ## conclusion and future work in this paper, we utilized the iconclass framework as a source of additional information for nmt of artwork titles. we extracted english and dutch keywords and english definitions from the iconclass codes. this information was concatenated to the source sentences. experiments show that augmenting the source sentences with the iconclass definitions for the objects under scrutiny improves the overall translation quality by a considerable margin. on the basis of a manual inspection of the output, we argue that the nmt model is able to successfully capitalize on the additional information extracted from the iconclass definitions. there are various reasons for this. firstly, the model is able to recognize any named entities in the concatenated part (that are lacking in the actual source title) and it can correctly inject them in the translation. secondly, the data augmentation approach improves the lexical aspects of the translation, providing useful semantic cues in the case of limited context. thirdly, the model is able to detect correct translations in the concatenated part and integrate them appropriately in the translation. however, the augmentation of the source sentences with the keywords and any type of extended context that we applied do not show any promising results. in future work, we would like to extend the current