Cross-Lingual Contrastive Learning for Fine-Grained Entity Typing for Low-Resource Languages (2022)

Abstract: Fine-grained entity typing (FGET) aims to classify named entity mentions into fine-grained entity types, which is meaningful for entity-related NLP tasks. For FGET, a key challenge is the low-resource problem — the complex entity type hierarchy makes it difficult to manually label data. Especially for those languages other than English, human-labeled data is extremely scarce. In this paper, we propose a cross-lingual contrastive learning framework to learn FGET models for low-resource languages. Specifically, we use multi-lingual pre-trained language models (PLMs) as the backbone to transfer the typing knowledge from high-resource languages (such as English) to low-resource languages (such as Chinese). Furthermore, we introduce entity-pair-oriented heuristic rules as well as machine translation to obtain cross-lingual distantly-supervised data, and apply cross-lingual contrastive learning on the distantly-supervised data to enhance the backbone PLMs. Experimental results show that by applying our framework, we can easily learn effective FGET models for low-resource languages, even without any language-specific human-labeled data. Our code is also available at https://github.com/thunlp/CrossET .

## introduction recently, various efforts have been devoted to exploring fine-grained entity typing (fget) @xcite , aiming to identify concrete fine-grained entity types for named entity mentions in sentences (figure 1 ). since the type information of named entity mentions is useful for understanding textual semantics, fget is widely applied to enhance entity-related tasks, such * corresponding authors. as coreference resolution @xcite , entity linking @xcite , relation extraction @xcite ) and event extraction @xcite . despite the success of fget, the low-resource problem is always a challenge of fget, since the complex type hierarchy makes it difficult to manually label data. to alleviate the low-resource problem, besides utilizing auto-labeled data @xcite @xcite , manually building fget datasets is the most effective approach @xcite @xcite @xcite . however, existing fget datasets are mainly in english. for datasets in specific languages other than english, such as chinese @xcite , japanese @xcite , @xcite , their scale and quality are not comparable to those english datasets. in this paper, we introduce a cross-lingual framework to learn fget models for low-resource languages, via utilizing the data in high-resource languages (e.g. utilizing english datasets). transferring the typing knowledge from highresource languages to low-resource languages is not easy. as different languages have quite different patterns, it is challenging to understand the semantics of both high-resource and low-resource languages at the same time. with only a few examples of low-resource languages and no parallel data, it is also hard to bridge different languages for knowledge transfer. to handle these issues: (1) we use multi-lingual pre-trained language models (plms) as backbone. multi-lingual plms such as m-bert @xcite are pre-trained on large-scale multi-lingual corpora, taking it as the backbone can well encode data in different languages into the same semantic space @xcite . (2) we apply heuristic rules and crosslingual contrastive learning to bridge multiple languages. we design several entity-pair-oriented heuristic rules to obtain distant supervision, which can automatically annotate entity types by utilizing latent relations between entity pairs. machine translation is used on the auto-labeled data to establish a connection between high-resource and lowresource languages. finally, we apply contrastive learning to learn similarities between cross-lingual auto-labeled types, instead of using pseudo-labels to learn a classifier, which can enhance the type recognition ability and reduce the side effect of auto-labeled data. for convenience, we name our cross-lingual contrastive learning framework "cross-c" in the following sections. we conduct experiments on two popular fget datasets: open-entity @xcite and few-nerd @xcite , and translate their test sets into non-english versions to evaluate the effectiveness of cross-c for lowresource languages. quantitative experimental results show that applying cross-c can easily train effective fget models for low-resource languages, even without any language-specific human-labeled data. besides quantitative experiments, we also provide some visualization of feature spaces and conduct case studies for qualitative analysis to show how cross-c works. ## method in this section, we will introduce our cross-lingual framework to learn fget models for low-resource languages. we will first give some essential notations and definitions, and then elaborate on the details of our framework. ## experiment in this section, we evaluate the effectiveness of our framework cross-c on two typical entityrelated datasets: open-entity and few-nerd. for each dataset, we conduct experiments in both lowresource (few-shot or zero-shot) and full-set settings. in addition to quantitative experiments, to further show how our method works, we also provide some visualization of feature spaces for qualitative analysis. ## related work as one of the most important tasks in the field of information extraction, fget has been studied for a long time. @xcite ; @xcite first propose to classify named entity mentions into various fine-grained entity types, instead of just a few coarse-grained types @xcite . since fine-grained types bring informative semantics for language understanding, these types are widely used to enhance entity-related nlp tasks, such as coreference resolution @xcite , entity linking @xcite , relation extraction @xcite ) and event extraction @xcite . some recent efforts further incorporate entity types to learn distantly-supervised fget methods. since entity types have complex hierarchies, manually annotating fget data is not easy, and thus the lowresource problem is one of the key challenges of fget. to alleviate this issue, distantly-supervised methods have been widely explored for fget. one typical distantly-supervised approach is using knowledge bases to automatically annotate entities mentioned in the text. @xcite ; @xcite collect anchors in the wikipedia pages that correspond to entities in knowledge bases, and then label these anchors with entity types in knowledge bases. this approach is then followed by a series of works @xcite @xcite @xcite to obtain pseudo labels. other approaches use various noun phrases in sentences as type pseudo labels @xcite , which can make full use of the recently proposed plms for data augmentation. human-labeled fget datasets. in addition to the distantly-supervised methods, the construction of fget datasets is also advancing. conll @xcite and ontonotes @xcite are the earliest datasets, although they just cover several coarse-grained types. then, @xcite weld (2012 however, all these efforts only focus on english. there are also some efforts to build datasets in other languages, such as chinese @xcite , japanese @xcite @xcite , but the scale and quality of these non-english datasets are still not comparable with english datasets, i.e., non-english human-labeled data are still scarce. cross-lingual and contrastive learning for fget. although cross-lingual learning has been widely explored in entity linking @xcite @xcite and named entity recognition @xcite @xcite , cross-lingual entity typing has not yet been explored much @xcite . for contrastive learning @xcite , some preliminary works have explored it for extracting relations between entities @xcite and achieved promising results. @xcite further use contrastive learning to analyze the impact of entity information on relation extraction. similar to cross-lingual learning, the exploration of contrastive learning for fget is still in the preliminary stage. ## conclusion and future work in this paper, to learn effective fget models for those low-resource languages, we propose an effective cross-lingual contrastive learning framework cross-c to transfer the typing knowledge from high-resource languages to low-resource languages. specifically, the framework cross-c uses a multi-lingual plm -m-bert as the framework backbone, which can simultaneously capture multi-lingual semantics in a unified feature space. furthermore, to bridge the gap between high-resource languages and low-resource languages, we introduce entity-pair-oriented heuristic rules as well as machine translation to automatically obtain high-quality cross-lingual data, and then apply cross-lingual contrastive learning on this distantly-supervised data to enhance the backbone plm. the experimental results show that by applying cross-c, the typing knowledge can be transferred from high-resource languages to low-resource languages, and we can learn effective fget models without any language-specific human-labeled data for those low-resource languages. in the future: