Entailment Graph Learning with Textual Entailment and Soft Transitivity (2022)

Abstract: Typed entailment graphs try to learn the entailment relations between predicates from text and model them as edges between predicate nodes. The construction of entailment graphs usually suffers from severe sparsity and unreliability of distributional similarity. We propose a two-stage method, Entailment Graph with Textual Entailment and Transitivity (EGT2). EGT2 learns the local entailment relations by recognizing the textual entailment between template sentences formed by typed CCG-parsed predicates. Based on the generated local graph, EGT2 then uses three novel soft transitivity constraints to consider the logical transitivity in entailment structures. Experiments on benchmark datasets show that EGT2 can well model the transitivity in entailment graph to alleviate the sparsity, and leads to signifcant improvement over current state-of-the-art methods.

## introduction entailment, as an important relation in natural language processing (nlp), is critical to semantic understanding and natural language inference (nli). entailment relation has been widely applied in different nlp tasks such as question answering @xcite , machine translation @xcite and knowledge graph completion @xcite . when coming across a question that "which medicine cures the infection?", one can recognize the information "griseofulvin is preferred for the infection," in the corpus and appropriately write down the answer with the knowledge that "is preferred for" entails "cures" when their arguments are medicines and diseases, although the surface form of predicate "cures" does not exactly appear in the corpus. there are many ways to present one question, and it is impossible to handle them without understanding however, existing eg construction methods still face challenges in both local and global stages. the distributional inclusion hypothesis (dih) about entailment assumes that given a predicate (rela-tion) p, it can be replaced in any context by another predicate (relation) q if and only if p entails q @xcite . most local methods in previous works are guided by dih, thus rely on the distributional co-occurrences from corpora, including named entities, entity pairs and context, as features to compute the local entailment scores. since different predicate pairs are processed independently, the locally built graphs suffer from severe data sparsity. that is, there are many entailment relations missing (as edges) in the graphs if the predicate pairs do not co-occur in the corpus. furthermore, predictions from local models may not be coherent with each other, for example, a local model may output three predictions like, a entails b, b entails c and c entails a at the same time, which actually indicate possible errors among the local predictions. to overcome the challenges faced by local models, different global approaches are used to take the interactions and dependencies between entailment relations into consideration. the first discussed global dependency is the logical transitivity among different predicates, that is, predicate a entails predicate c if there is another predicate b making both "a entails b" and "b entails c" hold simultaneously. @xcite uses the integer linear programming (ilp) to ensure the transitivity constraints on the entailment graphs, which is , unfortunately, not scalable on large graphs with thousands of nodes. @xcite models the structural similarity across graphs and paraphrasing relations within graphs to learn the global consistency, but does not gain further improvement due to the lack of high-quality local graphs and proper transitivity modeling. in order to deal with the problems in the local and global stages, we propose a novel entailment graph learning approach, entailment graph with textual entailment and transitivity (egt2). egt2 builds high-quality local entailment graphs by inputting predicates as sentences into a transformer-based language model fine-tuned on an rte task to avoid the unreliability of distributional scores, and models the global transitivity on these scores through carefully designed soft constraint losses, which alleviate the data sparsity and are feasible on large-scale local graphs. our key insight is that the entailment relation @xmath0: (1) the appropriate constraint scalable on large graphs containing rich information, and (2) the reliability of local graphs offering the premise @xmath1, which is impractical for previous distributional approaches, but may be available for the models well-behaved on rte tasks. specifically, the input sentences fed to transformer-based language models are formed without context, which makes our method accessible to those predicates not appearing in the corpus. the transitivity implication is confined to entailment relations with high confidence, which improves the quality of implied edges and cuts down the computational overheads. in a word, this paper makes the following contributions: ## related work based on dih, previous works extract feature vectors for typed predicates to compute the local distributional similarity. the set of entity argument pair strings, like "griseofulvin-infection" in the example of section 1, are used as the features weighted by pointwise mutual information @xcite . given the feature vectors for a predicate pair, different similarity scores, like cosine similarity, lin @xcite , dirt @xcite , weeds @xcite and balanced inclusion @xcite , are calculated as the local similarities. hosseini et al. (2019) and hosseini et al. (2021) use markov chain on an entity-predicate bipartite graph weighted by link prediction scores to calculate the transition probability between two predicates as the local score. they rely on the link predication model to generate the features in fact. guillou et al. (2020) adds temporal information into entailment graphs by extracting entity pairs within a limited temporal window as predicate features. mckenna et al. (2021) extends the graphs to include entailment relations between predicates with different numbers of arguments by splitting the features from argument pairs into independent entity slots, which impairs the representation ability of features when unary predicates are involved. as mentioned in section 1, entailment graphs are generally learned by imposing global constraints on the local entailment relations about extracted predicates. the transitivity in entailment graphs is modeled by the integer linear programming (ilp) in @xcite , which selects a transitive sub-graph of a local weighted graph to maximize the summation over the weights of its edges. their work is limited to a few hundreds of predicates due to the computational complexity of ilp. for better scalability, @xcite and @xcite make a strong frg-assumption that if predicate a entails predicates b and c, b and c entail each other, and an approximation method, called tree-node-fix (tnf). obviously, the assumption is too strong to be satisfied by real cases. since the hard constraints are difficult to work well on large-scale entailment graphs, @xcite propose two global soft constraints that maintain the similarity between paraphrasing predicates within typed graphs and between predicates with the same names in graphs with different argument types. their soft constraints are also used in @xcite and @xcite . the similarity between paraphrasing predicates, which ensures (@xmath2, implicitly takes the transitivity between paraphrasing predicates and third predicate into consideration. but it ignores the transitivity in more common cases, and leads to a limited improvement on performance. meanwhile, the transformer-based language model (lm), although proved to be effective in rte tasks @xcite @xcite , has received less attention in entailment graph learning. schmitt and schütze (2021a) uses pretrained lm on the lexical inference in context (liic) task, which is closely related to entailment graph learning. @xcite uses pretrained bert to initialize the contextualized embeddings in their contextualized link prediction and entailment score calculation. higher scores are assigned to the entailed predicates in the context of their premises, which is one implicit expression form of dih and different from our direct utilization of lm on textual entailment. ## conclusions in this paper, we propose a novel typed entailment graph learning framework, egt2, which uses language models fine-tuned on textual entailment tasks to calculate local entailment scores and applies soft transitivity constraints to learn global entailment graphs in gradient-based method. the transitivity constraints are achieved by carefully designed loss functions, and effectively boost the quality of local entailment graphs. by using the fine-tuned local lm and global soft constraints, egt2 does not rely on distributional features, and can be easily applied to large-scale graphs. experiments on standard benchmark datasets show that egt2 achieves significantly better performance than existing state-of-the-art entailment graph methods.