Zero-Shot Cross-Lingual Transfer is a Hard Baseline to Beat in G erman Fine-Grained Entity Typing (2021)

Abstract: The training of NLP models often requires large amounts of labelled training data, which makes it difficult to expand existing models to new languages. While zero-shot cross-lingual transfer relies on multilingual word embeddings to apply a model trained on one language to another, Yarowski and Ngai (2001) propose the method of annotation projection to generate training data without manual annotation. This method was successfully used for the tasks of named entity recognition and coarse-grained entity typing, but we show that it is outperformed by zero-shot cross-lingual transfer when applied to the similar task of fine-grained entity typing. In our study of fine-grained entity typing with the FIGER type ontology for German, we show that annotation projection amplifies the English model’s tendency to underpredict level 2 labels and is beaten by zero-shot cross-lingual transfer on three novel test sets.

## introduction the task of fine-grained entity typing (fet) is to assign a semantic label to a span in a text. the task is distinct from coarse-grained entity typing as done by named entity recognition systems because these systems are restricted to a small set of labels like 'person', 'organization' and 'location' which are not helpful for tasks that require more precise information about the entities. for example, fet assigns the label '/location/city' to the named entity 'berlin' in the sentence 'from 1997 to 2000, it had a permanent exhibition in berlin. @xcite fine-grained entity typing uses a high number of types in a multilevel hierarchy, which can be seen in the level 2 label '/location/city' (see figure 1 ). in this work we use the figer type hierarchy which consists of two levels with 112 types in total (37 level 1, 75 level 2). figer types are derived from the knowledge graph freebase @xcite . they are both interpretable by humans and useful in nlp applications such as relation extraction @xcite . there are systems for named entity recognition and coarse-grained entity typing in languages other than english (e.g. stanza @xcite ), but systems for fet with figer types are only available in english, due to the lack of figer annotated data in other languages. because manual annotation is time consuming and expensive, various methods have been proposed to expand nlp models to other languages without additional manual annotation. the method of annotation projection @xcite uses parallel text to automatically create annotated corpora. annotations from the resource-rich language are transferred to the resource-poor language using word alignment between translated sentences. annotation projection has been used successfully for the task of coarse-grained named entity typing in conjunction with named entity recognition @xcite @xcite . we follow these examples by using a parallel english-german corpus, automatic named entity recognition and a state of the art english fet model @xcite to assign figer type labels on the english side for transfer. we then project the labels onto the german half of the corpus. the output of this process is a german corpus annotated with figer types, which we use to train a german fet model. another approach to the same problem is zeroshot cross-lingual transfer, in which a model built on multilingual word embeddings and trained on high-resource language data is applied to test data in a different language. because the english fet model used in this work @xcite relies on contextualised multilingual word embeddings (xlm-roberta) @xcite it is possible to train it on english data and to test it on german. we compare the two approaches and show that the annotation projection approach amplifies the model's tendency to underpredict level 2 types, which lowers model performance. we also introduce three new test sets for german fet foot_0 on which zero-shot cross-lingual transfer performs better than models trained with german or a mix of german and english data. ## related work to the best of our knowledge there is no work that compares annotation projection directly against zero-shot cross-lingual transfer. while annotation projection has been used in a variety of tasks, there has not been a study of a case where this approach fails. authors admit that the quality of the annotating system plays role (e.g. ehrmann et al. (2011); ni et al. (2017) ), but they don't specify model properties that are necessary for the approach to work, instead focusing on ways to mitigate noise. @xcite ; @xcite and @xcite show the strengths of zeroshot cross-lingual transfer on a variety of different nlp tasks, but they do not address fine-grained entity typing. @xcite conclude that zeroshot performance can be improved by choosing a small amount of high quality training data from the target language. we test their approach for the fet scenario, but arrive at unclear results. ## method in this work we use the hierarchical typing model of @xcite trained on english gold data for the zero-shot approach and also to annotate the english side of the parallel text for annotation projection. we train the model with english silver data to show the amount of noise added by automatic annotation and finally we train it with german data which was produced by annotation projection. in the hierarchical typing model the entity and its context are encoded using multilingual xlm-roberta @xcite . for each type in the figer ontology the model learns a type embedding. it passes the concatenated entity and context vector through a 2-layer feed-forward network that maps into the same space as the type embedding. the score is the inner product between the transformed entity and context vector and the type embedding. for further model details refer to @xcite . ## experimental setup training data to contrast the zero-shot crosslingual transfer approach with models trained on automatically annotated and projected data we use three sources of training data. we use the 2m sentences english figer corpus as described by @xcite as a source of english human annotated data, which we will refer to as en gold. the data set consists of english wikipedia articles and we use it to train the zero-shot gold model. second, we use english machine annotated data @xcite . annotating english data using a model is the first step of the annotation projection. we use this data to train the zero-shot automatic model to examine the amount of noise added by automatic annotation. we generate en automatic from english sentences from the wiki-matrix corpus @xcite foot_1 , using the hierarchical typing model trained on 2 m sentences en gold. lastly, we use german annotation projected data (de projected) that was generated by projecting the labels from the en automatic onto german. for the details of our annotation projection pipeline please refer to appendix a. we use this data to train the annotation projected model. we portion each training corpus into slices of 100, 200, 300 and 400 k sentences to compare the influence of data size. for de projected only 300 k sentences are available, because only part of the parallel sentences in the wikimatrix corpus are of high enough quality for annotation projection. for details of the selection process refer to appendix a. an important point for our experiments is the label distribution in the training corpora (see test sets we compare performance using the following test corpora: 1) a german machine translation of the test split of the english figer corpus @xcite using deepl, which was manually corrected to eliminate translation and labelling errors (de-figer); 2) 500 manually annotated german sentences from the wikimatrix corpus (de-wiki), which we consider to be more challenging than de-figer, because it contains a wider range of type labels; 3) a small challenge set of 135 sentences taken from de-wiki, in which we replaced entities with close string matches to english (e.g. 'präsident nixon') with specifically german entities of the same type (e.g. 'bundeskanzler kohl'), which we call de-germent; and 4) for experiments where we mix german and english data, we also compare against test split of the english figer corpus @xcite (en-figer). data set statistics can be seen in ## results monolingual training figure 2 while for level 1 type labels annotation projected gets close to the performance of zero-shot gold on both test sets, on level 2 type labels the system falls behind zero-shot gold, with a wider gap on de-wiki. the comparison between zero-shot automatic and annotation projected is less clear. on the de-wiki test set annotation projected consistently outperforms zero-shot automatic, while on de-figer both systems perform very similarly. the high performance of zero-shot gold and the noisier zero-shot automatic might be due to the quality of english and german embeddings in xlm-roberta, as both are high resource languages from the same language family. this confirms @xcite who show that this method works especially well for close high resource language pairs and low level semantic tasks. the noise introduced by annotation projection affects level 2 label performance the most (see appendix b and multilingual training the underlying xlm-roberta embeddings allow to train a model with both german and english data. for this we combine slices from de projected with en automatic, because these data sets have the same distribution of labels. the low performance in the data mixing scenario compared to zero-shot gold can be explained with the distribution of labels in the silver corpora. due to the noise added when labels are projected from english to german, the mixed model tested in german profits from the addition of higher quality english data, but not vice versa. ## discussion and conclusions our results show that zero-shot cross-lingual transfer building upon xlm-roberta is a strong baseline for the task of fet and the language pair of english and german. it outperforms annotation projection on three new test sets. we also show that in our specific scenario annotation projection using the hierarchical typing model amplifies the models tendency to underpredict level 2 types. one way to mitigate these shortcomings would be to sample level 1 and level 2 labels in a training corpus so that they have the same distribution as in the gold data, although this would not control for data quality. another way could be to machine translate the manually annotated english corpus into german and then use annotation projection, as suggested by @xcite . this way the label distribution of the human annotated data could be preserved as well. lastly, improving the few-shot approach and designing more challenging test sets are other avenues to explore. of the 2nd meeting of the north american chapter of the association for computational linguistics, pages xxx-xxx. acl. mengjie zhao, yi zhu, ehsan shareghi, roi reichart, anna korhonen, and hinrich schütze. 2020. a closer look at few-shot crosslingual transfer: variance, benchmarks and baselines. arxiv preprint arxiv:2012.15682.