Align-then-Enhance: Multilingual Entailment Graph Enhancement with Soft Predicate Alignment (2023)

Abstract: Entailment graphs (EGs) with predicates as nodes and entailment relations as edges are typically incomplete, while EGs in different languages are often complementary to each other. In this paper, we propose a new task, multilingual entailment graph enhancement, which aims to utilize the entailment information from one EG to enhance another EG in a different language. The ultimate goal is to obtain an enhanced EG containing richer and more accurate entailment information. We present an align-then-enhance framework (ATE) to achieve accurate multilingual entailment graph enhancement, which first exploits a cross-graph guided interaction mechanism to automatically discover potential equivalent predicates between different EGs and then constructs more accurate enhanced entailment graphs based on soft predicate alignments. Extensive experiments show that ATE achieves better and more robust predicate alignment results between different EGs, and the enhanced entailment graphs generated by ATE outperform the original graphs for entailment detection.

## introduction predicate entailment detection is the task to determine if the meaning of one predicate is entailed (can be inferred) from the other predicate, which benefits many core natural language processing tasks such as question answering and semantic parsing. usually, a question like "did lionel messi appear at the fifa world cup stadium gelsenkirchen in 2006" might be answered by a sentence that does not directly correspond to the question, but is an expression like "lionel messi made his debut at the fifa world cup stadium gelsenkirchen in 2006", since the predicate "make one's debut at" entails predicate "appear at". to bridge such semantic gap between queries and answers, recent approaches @xcite @xcite @xcite have looked into learning typed entailment graphs (egs) with predicates as nodes and entailment relations as edges. these methods first extract predicate-argument (entity) relation triples from large text corpora, and construct typed entailment graphs based on the distributional inclusion hypothesis @xcite @xcite . predicates are then grouped into typed entailment subgraphs based on the types of entity arguments they take. such an eg is in an effective and machine-readable form to organize the context-independent entailment relations between predicates and can facilitate reasoning without extra context or resources, which can be regarded as a special kind of knowledge graph (kg) for natural language understanding. figure 1 shows excerpts from two typed egs in different languages with arguments of @xmath0, location>. however, egs frequently suffer from incompleteness, and egs in different languages are often complementary to each other. this makes a compelling case for developing a method that can integrate entailment information from egs in different languages and construct an enhanced eg containing richer and more accurate entailment information. for instance, to answer the aforementioned question "did lionel messi appear at the fifa world cup stadium gelsenkirchen in 2006", given [person]*抵达*[location] recently, a few efforts have been made to improve the quality of an eg by integrating entailment information from another eg. @xcite have tried to align the english and german egs by learning the predicate representations towards alignment through a link prediction model and showed that the stronger english eg can aid in german entailment detection. whereas @xcite only proves that an eg in a higher resource language can improve the quality of an eg in a lower resource language, @xcite further demonstrates that the cross-lingual complementarity between different egs can be used in both directions by ensembling the predictions from the two graphs. however, they did not really realize the alignment of egs. in order to achieve an ensemble, their model needs the parallel chinese translations of the english questions to query the chinese and english egs separately. in this paper, we propose a new task, multilingual entailment graph enhancement (mege), which aims to automatically align egs in different languages and utilize the entailment information from one eg to enhance the other. we emphasize that the enhancement should be effective in both directions, which is demonstrated and discussed in section 6.2. for the multilingual entailment graph enhancement task, we present an align-then-enhance framework, ate, which first automatically discovers equivalent predicates between egs in different languages and then constructs more accurate enhanced egs based on soft predicate alignments. in order to achieve accurate predicate alignment, an effective approach is to learn better predicate representations toward alignment. however, since egs usually suffer from severe sparsity issues, it is difficult to directly embed eg structures to learn effective predicate representations. for example, 79% of nodes in the popular english eg released by hosseini et al. @xcite have degrees no more than 2. to tackle this issue, we introduce rich context information of predicates extracted from the large opendomain encyclopedia kg wikidata @xcite into original eg to assist in learning better predicate representations, constructing a predicate-centric graph. simultaneously, we also build the entity-centric graph, aiming to improve the predicate representations by utilizing the information of entities closely associated to the predicates. and we propose a cross-graph guided interaction (cgi) mechanism to encourage sufficient interaction between the predicate-centric graph and the entity-centric graph and learn better predicate representations for alignment. to our best knowledge, there is no standard dataset to directly evaluate the predicate alignment performance for entailment graphs. we thus build a new alignment evaluation dataset egalign. experiments on egalign indicate that our model achieves the state-of-the-art performance on predicate alignment between different egs. the key technical contributions of this paper are as follows: we present an align-then-enhance framework for this task. ## related work most previous works on entailment graphs only focused on constructing an entailment graph in a single language, and usually ignore the complementarity between different egs. similar to the idea of entailment graph enhancement, @xcite do not construct an enhanced entailment graph but learned clusters of semantically equivalent english and french predicates based on their named-entity arguments. they create predicate representations and align the predicates by the cosine similarity between representations. @xcite solve the problem of paraphrasing in a multilingual context by creating aligned paraphrase clusters. they take the wikipedia articles describing the same topic as parallel texts, and use the wikipedia inter-language links between named entities to align predicates in different languages. the study on creating paraphrase clusters lays a foundation for the construction of entailment graphs. @xcite first propose typed predicate entailment graphs but can not be scalable to a large amount of data. to tackle this problem, @xcite present a scalable method that learns globally consistent similarity scores for entailment graph construction. based on the typed entailment graphs, @xcite are the first to try to align an english entailment graph with a german entail-ment graph. however, as discussed in section 1, the serious sparsity issues of egs hinder their alignment performance and they only perform alignment in one direction. most recently, @xcite demonstrate the cross-lingual complementarity between an english eg and a chinese eg in both directions. however, instead of really aligning the two egs, they ensemble the predictions from the two graphs, which needs the parallel translations between english and chinese questions for querying the english and chinese egs separately. ## problem formulation let p be the set of all typed predicates and t be the set of types, v(t 1 , t 2 ) denotes the set of typed predicates p with unordered argument types t 1 and t 2 , where @xmath1, t 2 ∈ t . the argument types of each predicate are determined by the types of entities that instantiate the argument slots. formally, we represent a typed eg as g(t 1 , t 2 ) =< v(t 1 , t 2 ), e(t 1 , t 2 ) >, where v(t 1 , t 2 ) is the set of typed predicate nodes and e(t 1 , t 2 ) denotes the set of weighted edges. we represent the edges as the sparse score matrix w (t 1 , t 2 ) ∈ [0, 1] |v(t 1 ,t 2 )|×|v(t 1 ,t 2 )| , containing the entailment scores between predicates of types t 1 and t 2 . without loss of generality, we consider the entailment graph enhancement task between two typed egs, the goal of this task is to utilize the entailment information (i.e., the entailment scores between typed predicates) from one eg to enhance the other. in this paper, we achieve this in two steps: 1) given a set of pre-aligned typed predicate pairs ## our approach: ate to obtain an enhanced eg, we propose a model, ate. as depicted in figure 2 , ate takes two typed egs as input, and follows a two-stage pipeline: predicate alignment and entailment graph enhancement, and finally output an enhanced eg with richer and more accurate entailment information. ## conclusion we present a new task, multilingual entailment graph enhancement, aiming to enhance the quality of one eg with another graph in a different language. we design an align-then-enhance method for this task, which utilizes a cross-graph guided interaction mechanism to tackle the sparsity issues of egs and achieves eg enhancement based on soft predicate alignment between different egs. experiments show that our align-then-enhance framework can effectively mine equivalent predicates in other egs through sufficient cross-graph interaction and better achieve accurate enhancement. we build a new dataset egalign to evaluate the predicate alignment performance of our model, and the results show that our model achieves the best performance. furthermore, we show that the enhanced egs outperform the original graph as well as bert on entailment detection.