Cross-lingual Entity Alignment with Incidental Supervision (2021)

Abstract: Much research effort has been put to multilingual knowledge graph (KG) embedding methods to address the entity alignment task, which seeks to match entities in different languagespecific KGs that refer to the same real-world object. Such methods are often hindered by the insufficiency of seed alignment provided between KGs. Therefore, we propose a new model, JEANS , which jointly represents multilingual KGs and text corpora in a shared embedding scheme, and seeks to improve entity alignment with incidental supervision signals from text. JEANS first deploys an entity grounding process to combine each KG with the monolingual text corpus. Then, two learning processes are conducted: (i) an embedding learning process to encode the KG and text of each language in one embedding space, and (ii) a self-learning based alignment learning process to iteratively induce the correspondence of entities and that of lexemes between embeddings. Experiments on benchmark datasets show that JEANS leads to promising improvement on entity alignment with incidental supervision, and significantly outperforms state-of-the-art methods that solely rely on internal information of KGs.

## introduction a multilingual knowledge base (kb) such as dbpedia @xcite , conceptnet @xcite and yago @xcite stores multiple language-specific knowledge graphs (kgs) that express relations of many concepts and real-world entities. as each kg thereof is either extracted independently from monolingual corpora @xcite or contributed by speakers of the language @xcite ; mitchell et al., 2018), it is common for different kgs to constitute complementary knowledge @xcite . hence, aligning and synchronizing language-specific kgs support ai systems with more comprehensive commonsense reasoning @xcite @xcite , and benefit various knowledge-driven nlp tasks, including machine translation @xcite , narrative prediction @xcite and dialogue agents @xcite . learning to align multilingual kgs is a nontrivial task, as kgs with distinct surface forms, heterogeneous schemata and inconsistent structures easily cause traditional symbolic methods to fall short @xcite @xcite . recently, much attention has been paid to methods based on multilingual kg embeddings @xcite @xcite @xcite . those methods seek to separately encode the structure of each language-specific kg in an embedding space. then, based on some seed entity alignment, the entity counterparts in different kgs can be easily matched via distances or transformations of embedding vectors. the principle is that entities with relevant neighborhood information can be characterized with similar embedding representations. such representations particularly are tolerant to the aforementioned heterogeneity of surface forms and schemata in language-specific kgs @xcite @xcite . while multilingual kg embeddings provide a general and tractable way to align kgs, it still remains challenging for related methods to precisely infer the cross-lingual correspondence of entities. the challenge is that the seed entity alignment, which serves as the essential training data to learn the connection between language-specific kg embeddings, is often limitedly provided in kbs @xcite . hence, the lack of supervision often hinders the precision of inferred entity counterparts, and affects even more significantly when kgs scale up and become inconsistent in contents and density @xcite . several methods also gain auxiliary supervision from profile information of entities, including descriptions @xcite @xcite and numerical attributes @xcite . however, such profile information is not available in many kgs @xcite @xcite , therefore causing these methods to be not generally applicable. unlike existing models that rely on internal information of kgs, we seek to create embeddings that incorporate both kgs and freely available text corpora, and exploit incidental supervision signals @xcite ) from text corpora to enhance the alignment learning on kgs (figure 1 ). in this paper, we propose a novel embedding model jeans (joint embedding based entity alignment with incidental supervision; ). particularly, jeans first performs a grounding process @xcite to link entity mentions in each monolingual text corpus to the kg of the same language. based on the kgs and grounded text in a pair of languages, jeans conducts two learning processes, i.e. embedding learning and alignment learning. the embedding learning process distributes entities, relations and lexemes of each language in its embedding space, in which a kg embedding model and a language model for that language are jointly trained. this process seeks to leverage text contexts to help capture the proximity of entities. on top of that, alignment learning captures cross-lingual correspondence for entities and lexemes in a self-learning manner @xcite . starting from a small amount of seed entity alignment, this process iteratively induces a transformation between language-specific embedding spaces, and infers more alignment of entity and lexemes at each iteration to improve the learning at the next one. moreover, we also employ the closed-form procrustes solution @xcite to strengthen alignment induction within each iteration. experimental results on two benchmarks confirm the effectiveness of jeans in leveraging incidental supervision, leading to significant improvement to entity alignment and drastically outperforming existing methods. ## related work we discuss relevant works in four topics. each of them has a large body of work which we can only provide as a highly selected summary. entity alignment. entity alignment in kbs has been a long-standing problem @xcite . aside from earlier approaches based on symbolic or schematic similarity of entities @xcite @xcite , more recent research addresses this task with multilingual kg embeddings. a representative method of such is mtranse @xcite . mtranse jointly learns two model components. there are a translational embedding model @xcite that distributes the facts in language-specific kgs into separate embeddings, and a transformation-based alignment model that maps between entity counterparts across embedding spaces. following the general principle of mtranse, later approaches are developed through the following three lines. one is to incorporate various embedding learning techniques for kgs. besides translational techniques, some models employ alternative relation modeling techniques to encode relation facts, such as circular correlation @xcite , hadamard product @xcite and recurrent skipping networks @xcite . others encode entities with neighborhood aggregation techniques, including gcn @xcite @xcite @xcite , rgcn @xcite and gat @xcite . their benefits are mainly to produce entity representations capturing high-order proximity, so as to better suit the alignment task. a few works follow the second line to enhance the alignment learning with semi-supervised learning techniques. representative ones include co-training @xcite , optimal transport @xcite and bootstrapping @xcite , which improve the preciseness of alignment captured with limited supervision. the third line of research seeks to obtain additional supervision from entity profiles, including descriptions @xcite , attributes @xcite @xcite and kg schemata @xcite . while those alternative views of entities can effectively bridge the embeddings, the limitation of such methods lies in the unavailability of those views in many kgs @xcite @xcite . a survey on the entity alignment problem by @xcite has provided a more comprehensive summarization of recent advances in these lines. our method is mainly related to the third line of research. while instead of leveraging specific intra-kb information, our method introduces supervision signals from text contexts that are freely accessible to almost any kbs with the aid of grounding techniques. meanwhile, our paper also follows the second line to improve alignment learning techniques, and couples two mainstream techniques for embedding learning. joint embeddings of entities and text. fewer efforts have been put to jointly characterize entities and text as embeddings. @xcite propose to connect a translational embedding of freebase @xcite to a english word embedding based on wikipedia anchors, therefore providing a joint embedding to enhance link prediction in the kg. @xcite generalize the approach by @xcite with distant supervision based on entity descriptions and text corpora. @xcite extract dependency paths from sentences and jointly embed them with a kg using distmult @xcite to support the relation extraction task. several other approaches focus on jointly embedding words, entities @xcite @xcite and entity types @xcite appearing in the same textual contexts without considering relational structure of a kg. these ap-proaches are employed in monolingual nlp tasks including entity linking @xcite , entity abstraction @xcite and factoid qa @xcite . as they focus on a monolingual and supervised scenario, they are essentially different from our goal to help cross-lingual kg alignment with incidental supervision from unparalleled corpora. multilingual word embeddings. our model component of alignment induction from text is closely connected to multilingual word embeddings. earlier approaches in this line, regardless of being supervised or weakly supervised, based on seed lexicon @xcite or parallel corpora @xcite , are systematically summarized in a recent survey @xcite . while a number of methods in this line can be employed in our model to gain addition supervision for entity alignment, we choose to use a combination of procrustes solution @xcite with selflearning to offer precise inference of cross-lingual alignment based on limited seed alignment. note that recent contextualized embeddings such m-bert @xcite and xlm-r @xcite do not directly suit our problem setting, since contextualization could cause ambiguity to entity representations, therefore impairing the alignment of entities across embedding spaces. ## method we hereby begin introducing our method with the formalization of learning resources. in a kb, l denotes the set of languages, and l 2 unordered language pairs. g l is the languagespecific kg of language @xmath0. e l and r l respectively denote the corresponding vocabularies of entities and relations. @xmath1, r, t) denotes a triple in g l such that h, @xmath2. boldfaced h, r, t represent the embedding vectors of head h, relation r, and tail t respectively. for a language pair (l 1 , l 2 ) ∈ l 2 , i e (l 1 , l 2 ) denotes a set of entity alignments between l 1 and l 2 , such that e 1 ∈ e l 1 and e 2 ∈ e l 2 for each entity pair (e 1 , e 2 ) ∈ i e (l 1 , l 2 ). following the convention of previous work @xcite @xcite , we assume the entity pairs to have a 1-to-1 mapping and it is specified in i e (l 1 , l 2 ). this assumption is congruent to the design of mainstream kbs @xcite where disambiguation of entities is granted. besides the definition of multilingual kgs, we use d l to denote the text corpus of language l. d l is a set of documents {d l }, where each document d @xmath3, w 2 , ..., w l ] is a sequence of tokens from the monolingual lexicon w l . each token w i thereof is originally a lexeme, but may also be an entity surface form after the ground process, and we also use boldfaced w i to denote its vector. i w (l 1 , l 2 ) denotes the seed lexicon between (l 1 , l 2 ), such that w 1 ∈ w l 1 and w 2 ∈ w l 2 for each lexeme pair (w 1 , w 2 ) ∈ i w (l 1 , l 2 ). note that i w only include the alignment between lexemes, and may optionally serve as external supervision data. to be consistent with previous problem settings of entity alignment @xcite @xcite , i w is not necessarily provided to training, but is defined to be compatible with the scenarios where it is available. jeans addresses entity alignment in three consecutive processes. (i) a grounding process first link entities of each kg to possible mentions of them in the corresponding monolingual corpus, therefore connecting entities and text tokens of the same language into a shared vocabulary. (ii) an embedding learning process characterizes the kg and text of each language into a separate embedding space. in this process, we couple both the translational technique @xcite @xcite and the neighborhood aggregation technique @xcite , which are two representative techniques to characterize a kg. simultaneously, the monolingual text tokens are encoded with a skip-gram language model @xcite . (iii) on top of the embeddings, starting from a small amount of seed entity alignment and optional seed lexicon, the alignment learning process iteratively infers more alignment both on kgs and text using selflearning and procrustes solution @xcite . the processes of jeans's learning is consistent to figure 1 . the rest of this section introduces the technical details of each process. ## experiment in this section, we evaluate jeans on two benchmark datasets for cross-lingual entity alignment, and compare against a wide selection of recent baseline methods. we also provide detailed ablation study on model components of jeans. ## conclusion this paper introduces jeans for entity alignment. different from previous methods that leverage only internal information of kgs, jeans extends the learning on any text corpora that may contain the kg entities. for each language, a noisy grounding process first connects both data modalities, followed by an embedding learning process coupling gcn with relational modeling, and an self-learning based alignment process. without introducing additional labeled data, jeans offers significantly improved performance over sota models on benchmarks. hence, it shows the effectiveness and feasibility of exploiting incidental supervision from free text for entity alignment. for future work, aside from experimenting with other embedding learning techniques for kgs and text, we plan to extend jeans to learn associations on kgs with different specificity @xcite . we also seek to extend the representation scheme in hyperbolic spaces @xcite along with the incorporation of hyperbolic lexical embedding techniques @xcite , aiming at better capturing the associations for hierarchical ontologies.