Did You Enjoy the Last Supper? An Experimental Study on Cross-Domain NER Models for the Art Domain (2021)

Abstract: Named entity recognition (NER) is an important task that constitutes the basis for multiple downstream natural language processing tasks. Traditional machine learning approaches for NER rely on annotated corpora. However, these are only largely available for standard domains, e.g., news articles. Domain-specific NER often lacks annotated training data and therefore two options are of interest: expensive manual annotations or transfer learning. In this paper, we study a selection of cross-domain NER models and evaluate them for use in the art domain, particularly for recognizing artwork titles in digitized art-historic documents. For the evaluation of the models, we employ a variety of source domain datasets and analyze how each source domain dataset impacts the performance of the different models for our target domain. Additionally, we analyze the impact of the source domain’s entity types, looking for a better understanding of how the transfer learning models adapt different source entity types into our target entity types.

## introduction cultural heritage archives contain vast amounts of unstructured data where valuable knowledge resides. this data can be analyzed and valuable information can be extracted using natural language processing (nlp) tools. nowadays, most of the nlp tasks are performed using deep learning models which rely on large amounts of training data. one of the core nlp tasks is named entity recognition (ner) which consists of finding mentions of named entities from a usually pre-defined set of entity types. machine learning models learn entity and context patterns from labeled corpora allowing them to discover new entity mentions from unseen text. in a scenario where there is a previously annotated large corpus, these models achieve good performance and can find new named entities from the pre-defined set of entity types. in the past, such datasets have been built for domains such as news wire @xcite and biomedical texts @xcite containing annotations for entity types, such as person, location, organization, or protein, and gene expression. large labeled corpora are expensive and timeconsuming to obtain. for less popular domains, large annotated corpora typically don't exist. there is especially a lack of annotated data for domainspecific entity types. one specific domain that requires non-standard entity types to be extracted is the cultural heritage domain. in this paper, we focus on digitized art-historic archives, in particular on the entity type artwork. for this particular entity type, there are no extensive datasets. the entity type artwork is different from the standard ones (person, location, organization, date) and poses some interesting challenges @xcite . not only is the entity type different, but also the structure and noise of digitized art-historic texts are different from news wire or biomedical text collections. one particular challenge is the ambiguity inherent to the definition of such titles due to the fact that sometimes these titles describe a scene or contain other named entities. for instance, the painting titled 'girl before a mirror' by pablo picasso, depicts a girl before a mirror. only the context of the mention identifies this phrase as a painting title. moreover, a big percentage of art-historic archives need to be digitized first using optical character recognition (ocr) software. this routinely introduces errors such as mis-identified characters, the addition of noise, and the loss of formatting structure @xcite @xcite . further, the quality of ocred texts strongly depends on the print quality of the original documents @xcite @xcite . given the aforementioned challenges, different alternatives for solving the task could be explored. one would be manually annotating a large corpus with artwork title information. but besides being a time-consuming and expensive task, it would not scale to further cultural heritage entities such as galleries, art styles, or art movements. another would be focusing on gazetteers and rule-based approaches. but, listing all possible artwork titles would not only be cumbersome, but would also not solve the ambiguity problem for phrases such as 'girl before a mirror'. the most promising approach is to make use of existing, previously annotated corpora from other domains and transfer the learned patterns to the new domain. in combination with deep learning models, this domain adaptation via transfer learning or multi-task learning has shown good results for popular domains @xcite . different models have been proposed in the past under the concept of cross-domain ner to solve the problem. these models learn to identify named entities within a target domain based on patterns learned from a large, labeled dataset from a source domain. the goal of this paper is to evaluate the performance of some of the best of those models for the artwork recognition task. the paper is structured as follows: in section 2 we describe the existing crossdomain ner models. in section 3 we describe the existing ner datasets available for different domains and the construction of a target dataset used for the training and evaluation of artwork recognition. in section 4 we describe the evaluation setup and in section 5 the results are outlined and finally, in section 6, the conclusion and future work are proposed. ## related work in this section, we discuss previously proposed cross-domain ner models and focus on the domain adaptations that those models propose. cross-domain ner models could be divided into two main categories. the models in which the source and target domain share the entity types but have differences in terms of the vocabulary, and the models which consider the disparity between the entity types in the source and the target domain. for instance, one traditional task for the first group would be the transfer of persons, locations and organization from the news domain into the social media domain. in this case, the persons mentioned in the news domain might be different from the persons mentioned in social media. moreover, the language used in social media is different from the language used in news articles. artifacts, such as emojis, hashtags, or '@' as well as the structure of sentences differ between domains. however, the entity types remain constant in the domain adaptation task. @xcite propose a model for low-resource target domains combining multitask learning (mtl) and a mixture of entity experts (moee), aiming to improve generalization and reduce the over-fitting effect when a model learns entities from a source domain. @xcite propose a general neural transfer framework called dual adversarial transfer network (dat-net), wang et al. (2020) extend the popular bi-lstm-crf architecture for multi-domain ner, dividing the domain-specific and independent components of the network, to achieve adaptation over multiple genres. other models deal with different entity types in the target domain compared with the source domain. this is the case for domain-specific entity types where extensively annotated corpora are missing. artwork mentions, for instance, are not annotated in traditional ner datasets, therefore we focus our study on this kind of domain adaptation. within these models, @xcite proposed to transfer the weights of a bi-lstm-crf model with both word and character embeddings. the weights were trained on the source domain and then fine-tuned on the smaller target dataset. they experimented with transferring different parts of the network to the target domain and concluded that transferring the weights from the lower layers of the network, particularly the character bi-lstm layer, improved the performance of the ner model on the target domain compared to a model trained only with the target dataset (no transfer). a similar model proposed by @xcite , called cdma-ner augmented the idea of using a pretrained model by including adaptation layers on top of it to perform the domain adaptation without the need of retraining the source model. the adaptation between domains is based on the bottom layer of the bi-lstm model, particularly on the adaptation of word embeddings. a different kind of domain adaptation is used by models which simultaneously train the source and the target domain in a multi-task learning approach @xcite , @xcite , @xcite . in the multi-task model proposed by jia and zhang (2020) (multi-cell compositional lstm for ner domain adaptation) which is based on an lstm network, each entity type has an independent cell state. additionally a compositional cell combines all the entity type cells into the final output which is then passed to the domain-specific conditional random field (crf). the domain adaptation is performed on the entity type level, and the model leverages the context embeddings provided by bert @xcite . in their experiments, they transferred information from the news domain to the biomedical and the social media domain. another recently proposed model called cross-ner @xcite introduces domain-adaptive pre-training (dapt) as a technique to continue the pre-training of language models such as bert with domain-specific raw texts by masking spans of tokens instead of random tokens for training. they experimented with different masking strategies as well as different corpora selection criteria and concluded that the best performance is obtained when dapt is performed in a set of sentences containing general and task specific entities. the entities used for corpora selection are chosen from predefined resources, such as gazetteers or knowledge graphs. besides the pre-trained language model, which is trained in the target domain, the model uses a linear layer on top. they experimented with training the whole model on only the target domain, jointly training the source and target domain, and pre-traning in the source domain followed by finetuning on the target domain. their results show that pre-training followed by fine-tuning yields better results. in their paper, they also introduce a new dataset with a diverse set of annotated texts from different domains with domain-specific entity types. in their experiments these domains are treated as targets. we use their dataset for our experiments but instead of treating them as target domains, we consider them as source domains. the details of the datasets we use are described in section 3. ## datasets as mentioned in section 2, cross-domain ner relies on the knowledge of a source domain. this knowledge can be in the form of an annotated corpus with domain-specific entity types or a pretrained model specialized in recognizing them. we use a diverse set of source datasets for this purpose. regarding the target domain, we created a dataset with sentences containing art-related entity mentions. in the following subsections, we describe the datasets considered in our experiments as source datasets, as well as the target domain dataset which will serve as a training, validation, and test dataset. ## experimental setup our evaluation aims to shed light on the power of different cross-domain ner models to adapt to the art domain and recognize artwork mentions. for our experiments, we focus on the models cdma-ner proposed by @xcite , muli-cell lstm proposed by @xcite and crossner proposed by @xcite . to compare the performance, we train each of these models using datasets from a set of source domains and a single target domain training dataset. we measure the f1 score for the task of recognizing artwork mentions in the target test set. for the experiments with cdma-ner, glove @xcite word embeddings are used, and for both, multi-cell lstm and cross-ner, which are designed to use pre-trained language models, we use bert base model (cased) as well as an adaptation to the art domain following the domain-adaptive pre-training used in cross-ner. ## results another observation is that dapt is in general not improving the language model for the cross-ner model, which performs better with the original bert model. one possible reason is the digitization noise introduced into the raw text used to perform dapt. for multi-cell lstm, the average improvement is very small. the cdma-ner model in general performs worse than the other models and the baseline, and the reason could be the lack of contextualized representation of words in the glove embeddings. generally, the crossner model performs better than the other two models and its performance is similar to the baseline, although the model is relatively simple in comparison to multi-cell. this suggests that the traditional lstm-crf combination might not be suitable for transfer learning to complex entities such as artworks. the combination of lstm and crf is positive for ner as shown by the performance of the baseline model, but as the architecture becomes more complex, the performance is compromised. another reason why multi-baseline: flairnlp bert .589 cross-domain ner models source domain news ai lit mus pol sci avg cdma-ner .460 .368 .344 .394 .409 .413 .386 multi-cell lstm bert .255 .509 .385 .467 .438 .459 .451 multi-cell lstm dap t .343 .487 .436 .464 .471 .413 .454 crossner bert .537 .519 .578 .535 .521 .512 .533 crossner dap t .594 .488 .507 .477 .482 .528 .496 size-balanced experiments cdma-ner .332 .339 .365 .336 .360 .318 .344 multi-cell lstm bert .495 .455 .460 .446 .484 .441 .457 multi-cell lstm dap t .434 .454 .489 .458 .415 .463 .456 crossner bert .522 .535 .518 .543 .517 .586 .540 crossner dap t .475 .503 .516 .560 .528 .518 .525 the results in bold font correspond to values higher than the baseline cell lstm models might be performing worse than crossner is the fact that there is no overlap between source and target entity types, therefore the weights within the lstm cells are not being strongly shared among domains. the results of training the models with the unbalanced datasets reveal that the size of the source dataset does not guarantee a good target performance. the adapted news dataset is 13 times bigger than the music and literature datasets, but the performance is comparable when training the crossner bert model. one reason for this behavior is the more general definition for entity types in conll03, different from the more specialized entity types in the music and literature datasets. one of the aspects which differentiate the various domains is the set of entity types that are relevant for the domain and are present in the different datasets. to study the impact on the performance of artwork recognition we remove individual entity types from the full music dataset. for each of the 13 entity types in this dataset, we generate an alternative version of the dataset in which the entity type is not considered in the annotations. this means that the tokens which were previously labeled as part of those named entities will remain in the dataset but without the annotation. each altered dataset is used to train the 5 studied models. in figure 1 , the models' performance after altering the dataset is displayed as relative performance change with respect to the original experiment with the complete dataset. this way, we intend to analyze how each model depends on the source entity types to be able to transfer that knowledge to the recognition of artwork mentions. from the figure it is clear that the multi-cell lstm model suffers a greater decrease in performance when the musical artists and bands are not present in the source dataset. this is an indicator of the manner in which this model learns the connections between the source and target entity types through the entity-typed lstm cells. it is interesting, however, that in some cases the performance improves when removing entity types. this suggests that the model is sensitive to the similarity between the source and target entity types. thus, depending on the type of entities we would like to recognize in the target domain, we should select the source dataset. best results are achieved with the most similar entity types in the source domains. to phrase it in terms of the artwork recognition task, it would make sense to first analyze which domains contain titles of human-created creative works and then use those entity types exclusively. figure 2 depicts results of a similar experiment. in this case only one of the entity types is present in the dataset. comparing both figures, it is clear that source datasets with just one entity type perform worse than source datasets with more variety in entity types. it is, however, counter-intuitive that the entity types which help the most in the transfer setting towards recognizing artworks are not song or album, which are the entity types in the music domain that resemble closest to the notion of artwork. additional details of the results can be found in https://github.com/hpi-informationsystems/cross-domain-ner ## conclusions in this paper, we studied the task of complex ner, specifically recognizing artworks in art-historic texts. we discuss the reasons why this is a hard task and why it is promising to leverage annotations from other domains to compensate for the lack of annotated resources for the art domain. we explained the concept of cross-domain ner using transfer learning which has been investigated in the past to achieve the aforementioned domain adaptation and presented related work connected to this concept. based on the problem setup and a collection of annotated datasets, we performed a set of experiments to understand the performance of domain-adapted ner to recognize artworks. in the experiments we analyzed both, the models and the datasets, in order to isolate and understand independently different aspects of the presented approaches. from the experimental evaluation of cross-domain ner approaches for the recognition of artworks we conclude that, although domain adaptation is a promising approach to achieve this goal, a simpler alternative, namely a lstm-crf model with bert base (cased), perform as well as the best cross-domain ner. as future work, we would like to investigate the explainability and interpretability of cross-domain ner models to understand better their limitations and propose new models that not only take into account the differences in terms of entity types and language between domains, but also semantic relations between the domains and the named entities. additionally, it would be of interest to investigate the domain adaptation of other tasks like information extraction and knowledge graph embedding models, which could be jointly trained with ner.