Zero-Shot Information Extraction to Enhance a Knowledge Graph Describing Silk Textiles (2021)

Abstract: The knowledge of the European silk textile production is a typical case for which the information collected is heterogeneous, spread across many museums and sparse since rarely complete. Knowledge Graphs for this cultural heritage domain, when being developed with appropriate ontologies and vocabularies, enable to integrate and reconcile this diverse information. However, many of these original museum records still have some metadata gaps. In this paper, we present a zero-shot learning approach that leverages the ConceptNet common sense knowledge graph to predict categorical metadata informing about the silk objects production. We compared the performance of our approach with traditional supervised deep learning-based methods that do require training data. We demonstrate promising and competitive performance for similar datasets and circumstances and the ability to predict sometimes more fine-grained information. Our results can be reproduced using the code and datasets published at https://github.com/silknow/ZSL-KG-silk .

## introduction the mentioning of european silk textiles often evokes images of clothes and furniture of the old aristocracies and the lavish lifestyles of kings and queens. nowadays, the knowledge about the occidental way of producing these expensive items is, however, more and more endangered. many museums and collections around the globe fortunately still have silk objects, or at least public records with metadata and images illustrating them. such specific museum data, from many different sources about cultural heritage objects that are partly centuries old, have naturally some gaps: sometimes, the production year or place is unknown, but the material and technique used is described; sometimes, a rich textual description is provided with many little details about the object production and what it depicts, but categorical values informing about the exact material or technique used is not provided (figure 1 ). the recent progress in natural language processing and more specifically in information extraction can help to address these problems. in particular, it is common to train text classification models from metadata records in order to predict missing categorical values in other records. however, such models do require a significant amount of annotated data for training, which is expensive to get when the domain is very specific. in this paper, we first briefly introduce the development of a multilingual knowledge graph (kg) about silk textiles production. this knowledge graph, developed with the help of historians and museum experts, uses the cidoc-crm ontology and a domain expert designed thesaurus defining the silk textile concepts. next, we propose a zero-shot learning (zsl) approach based on the conceptnet common-sense knowledge graph @xcite , to predict the missing categorical metadata while avoiding to rely on training data. we compare our approach with supervised approaches and we show competitive results and demonstrate the ability to predict more fine-grained concepts despite the specificity of this domain. the remainder of this paper is structured as follows. we present some relevant related work in section 2. we summarise the development of the knowledge graph and we describe the dataset being used in our experiments in section 3. we detail our zero shot learning approach as well as two supervised learning baselines in section 4. we analyze the classification results in section 5. finally, we provide conclusions and outline some future work in section 6. ## related work integrating cultural heritage data into a knowledge graph can be difficult and time consuming, as data has to be usually collected from many different sources that generally do not use standard formats. for museum data, there are models and ontologies that make such a process less challenging. we consider the cidoc conceptual reference model (crm) to be one of them. cidoc-crm is an event-centric ontology through which everything can be represented as an event. a man-made object has, for example, been produced at some point in time. certain materials might have been used during this production that has taken place at a specific location in a specific century. cidoc-crm offers not only classes and properties for such an eventbased representation, but is also easily expandable with classes from other ontologies. hence, cidoc-crm comes also with useful extensions such as crmsci (scientific observation model) and cr-mdig (model for provenance metadata). cidoc-crm is an official iso standard since 2006 and this status has been renewed in 2014. there are more and more successful examples of kg development in the cultural heritage realm. for example, @xcite is a knowledge graph about the italian cultural heritage that at least indirectly reuses some cidoc-crm classes and properties. the dutch rijksmuseum collection is available as linked open data @xcite while doremus is the largest knowledge graph available about about classical musical works and interpretations @xcite . information extraction from texts generally rely on supervised machine learning and deep learning classification techniques that require labelled training data. while general purpose transformer-based language models are often used, fine-tuning them often require numerous examples @xcite . one possible solution to address this lack of labelled data is to use active learning @xcite in conjunction with pre-trained models such as bert @xcite that comes with possible bias problems @xcite . in cultural heritage domains, several works are trying to compensate for the lack of labelled data. one method is to leverage human annotations through crowdsourcing together with extracted visual and textual features and automatic annotation through transfer learning @xcite . in general convolutional neural networks (cnn) are often used for image, audio and video data @xcite , whereas recurrent neural networks (rnn) is also used for textual cultural heritage data @xcite . regular neural network @xcite for textual data are also used together with word embeddings. more recently, zero-shot learning approaches have attracted a lot of attention for their ability to offer text classification without relying on training data. it is a form of automatic classification for which textual documents unseen by the model can be analyzed and classified. several frameworks have been proposed over the years, based on bert @xcite , @xcite or other large pre-trained models @xcite . our approach relies on zeste foot_0 (zero shot topic extraction) @xcite ) that provides a framework for extracting topics from textual documents using the conceptnet commonsense knowledge graph. in addition, the framework provides explainability of its classification results using the conceptnet kg neighborhood. ## evaluation in this section, we compare the results of our zero-shot learning (zsl) approach with the supervised methods described in the section 4.1. we present the results alongside each of the three properties of interest: material, technique and depiction. it is worth to note that the precision, recall, f1 scores are obtained on 20% of the dataset following a 5-fold cross validation while the figures reported for the zsl method concerns the entire language specific datasets described in section 3. the results for the prediction of technique concepts are presented in figure 3 provides the confusion matrix of the predictions made for the technique property with the zsl method on the english subset. we can see that the true label is usually predicted the most per class, especially in the case of "embroidery" and "velvet". we observe that "brocaded" is often confused with "embroidery" or ""velvet" while "damask" and "tabby" are rarely predicted. in the case of "damask", this just reflects a low amount of samples, whereas in the case of "tabby", the predictions almost do not work at all (f1-score of 2.9% ). figure 4 shows an example where "embroidery" is correctly predicted, while figure 5 depicts a counter-example (the correct technique should have been "embroidery"). the graph highlights the most relevant words used for the predictions. the first example is based on the text: "spot samplers feature motifs that are scattered in a seemingly random fashion over the surface of the foundation fabric, usually linen. these samplers are rarely signed or dated, and often include motifs that are only partially worked, leading to the conclusion that this type of sampler was made as a personal stitch reference for its maker, and not for display, as band samplers were signed by student embroiderers. the sampler features flowers, obelisks on pedestals, and an "s" motif, in addition to geometric designs that are of the type that would have been used to decorate small purses, cushions, and other accessories.", taken from a record from the met museum. 4 the second text is "an example of the kind of work [catherine de medici] appreciated is the museum's panel of yellow satin embroidered with silk threads. one of a set of three (the others are in the musée historique des tissus, lyon), it hung as a valence around the top of a four-poster bed. various print sources were culled for the airy de ## conclusion and future work in this paper, we hypothesize that a common sense knowledge graph such as conceptnet can feed a zero shot learning method for enriching a domain specific knowledge graph such as one describing the silk textile production. through extensive experiments, we have demonstrated promising results for such an approach in its ability to sometimes reliably predict fine-grained concepts without requiring any training data as supervised classification techniques do. nevertheless, we observe several limitations: the concepts that should be predicted must exist in conceptnet with an appropriate topic neighborhood. our results can be reproduced using the code and datasets published at https: //github.com/silknow/zsl-kg-silk . even if supervised methods for metadata predictions perform generally better, zsl remains an interesting method to get accurate predictions even in specific domains that often suffer from data sparsity. we observe that it is also possible to bootstrap the predictions in using first the zsl method and then applying supervised classification models to further increase performance. we aim to experiment with such hybrid approaches in the future.