How Can Cross-lingual Knowledge Contribute Better to Fine-Grained Entity Typing? (2022)

Abstract: Cross-lingual Entity Typing (CLET) aims at improving the quality of entity type prediction by transferring semantic knowledge learned from rich-resourced languages to low-resourced languages. In this paper, by utilizing multilingual transfer learning via the mixture-of-experts approach, our model dynamically capture the relationship between target language and each source language, and effectively generalize to predict types of unseen entities in new languages. Extensive experiments on multi-lingual datasets show that our method significantly outperforms multiple baselines and can robustly handle negative transfer. We questioned the relationship between language similarity and the performance of CLET. A series of experiments refute the commonsense that the more source the better, and suggest the Similarity Hypothesis for CLET.

## introduction fine-grained entity typing (fet) aims at labeling entity mentions in a particular context with one or more specific types organized in a type hierarchy. for example, donald trump is classified as having the path of following types: president, politician, person. president is a subclass of politician that in turn is a subclass of person. fet provides accurate type information, and is therefore quite useful for various downstream nlp tasks, such as entity linking @xcite @xcite , relation extraction @xcite ), text generation @xcite , and so on. supervised learning approaches to fet need huge amount of labeled training data @xcite @xcite , and can be applied for a few rich-resourced languages, e.g., english, which have enough qualified labeled data. figure 1 : example of cross-lingual entity typing. we use knowledge from source language (english) to help with entity typing task in target language (chinese). for the vast majority of low-resourced languages, we have insufficient training data, or even do not have labeled data at all. however, languages are not independent, instead, some are more similar than others, and form a family tree. for example, portuguese is similar to spanish; dutch can even be thought of as half way between german and english. this motivates us to utilize the knowledge from rich-resourced languages (source) to help to predict missing types in a low-resourced language (target), which is called the cross-lingual entity typing (clet). previous research showed that transferring knowledge from multiple source languages could improve the performance of entity typing @xcite . recent research proposed a unified clet model, trained with four rich source languages (english, finnish, german, and spanish), is able to accept over 100 different languages @xcite . behind such unified models is the assumption that the more richresourced languages a model has, the better the performance will be. this leads to the search of the best unified model for all low-resourced languages. here, we raise the question: how will the similarity between the source and the target languages affect the performance? to this end, we carefully select six languages as follows: german, english, and dutch in the west germanic family, russian in the slavic family, and spanish in the romance family. this five languages are in the european family 1 . we select chinese in the sino-tibetan family, which is totally different from the other five languages, as illustrated in figure 2 . the more similar two languages are, the higher is their lowest common ancestor located in the language tree. following the unsupervised multilingual transfer-learning setup, we use labeled data from source languages and unlabeled data from the target language, leverage multilingual bert as feature encoder to produce language-independent features @xcite , and use mixtureof-experts (moe) approach @xcite to capture the correlations between the target language and each source language. for each target example, the predicted posterior is a weighted combination of all the experts' predictions. experts' weights reflect the proximity of the example to each source language. to further improve transfer quality, we apply a language discriminator to extract more language-invariant features from both source and target languages via adversarial learning. extensive experiments show that our proposed method significantly outperforms multiple state-of-the-art monolingual methods. in contrast to other cross-lingual fet researches, our work explores how the similarity between source and target languages would affect the fet performance. our experiment results surprisingly refute the commonly accepted assumption that the more and the richer the source languages are, the better performance it will be. our results suggests the importance of the similarity between source and target languages. the more similar the source and the target are and the richer the source is, the 1 https://thelanguagenerds.com/2019/ feast-your-eyes-on-magnificent-linguistic-family-tree/ better performance it will be. adding a rich but dissimilar source may reduce the performance. this observation refutes the existence of the best unified model for all target languages. the best crosslingual source languages shall be rich and selected among the cluster of the most similar languages to the target language. the rest of this paper is organized as follows. section 2 formally defines the problem of crosslingual fine-grained entity typing. section 3 describes our approach. section 4 reports two groups of experiments, one to evaluate our method, the other to explore the relation between language similarity and the performance of type prediction. section 5 reviews some related works. section 6 concludes our work. ## problem formulation we use @xmath0, in which n is the number of source languages, t as the target language. types are organized into a treestructured hierarchy y, shared by all languages. based on the assumption that each mention can only have one type-path depending on the context, we represent each type-path uniquely by the terminal type (which might not be a leaf node). for example, type-path root-person-athlete can be represented as just athlete, while root-person can be unambiguously represented as the non-leaf person. for each source language s @xmath1, we have a set of training data s @xmath2, y t )} |s i | @xmath3. x @xmath4, c t ) contains two parts, m @xmath5, . . . , w r is an entity mention and c @xmath6, . . . , w l is its context, both m t and c t are word sequences, where l is the context length and 1 < @xmath7. y t is the most specific type of m t , corresponding to a unique typepath in y. for target language t , we create a set of unlabeled data @xmath8 ## experiments a series of experiments are conducted to evaluate our clet method and to examine how the language similarity affects the performance of clet. our source code is available foot_0 for reference. ## related work fine-grained entity typing. fet research targets at utilising sentence-level context for making predictions @xcite and @xcite , in which they created the commonly used figer and ontonotes datasets. @xcite proposed an attentive lstm network model to encode an entity context, and proposed an attention mechanism to allow the model to focus on relevant expressions in a context. @xcite studied two kinds of noises, namely, outof-context noise and overly-specific noise in training data. @xcite leveraged a novel cost function to jointly model the correlation among hierarchical types and label noises. @xcite presented an effective method to impose label-relational inductive bias on fine-grained entity typing models. @xcite investigated the problem of denoising distant training data for entity typing tasks. @xcite regularized distantly supervised models with compact latent space clustering (clsc) to effectively utilize noisy data. @xcite employed contextualized word representations to further boosts the performance. ## conclusions and discussions we address the problem of cross-lingual entity typing (clet) in an unsupervised setting, and propose a mixture-of-experts (moe) approach to dynamically capture the relation between the target language and each source language, which enables to acquire more knowledge from source languages. experiments on multi-lingual datasets show that this approach outperforms various baselines and can effectively predict types of unseen entities in new languages. the presented work is the first to investigate how language similarity affects the performance of clet, and propose the similarity hypothesis. this will be helpful for the empirical selection of source languages, and raises new questions, such as how we can quantitatively compute and compare similarities among languages.