From the One, Judge of the Whole: Typed Entailment Graph Construction with Predicate Generation (2023)

Abstract: Entailment Graphs (EGs) have been constructed based on extracted corpora as a strong and explainable form to indicate context-independent entailment relation in natural languages. However, EGs built by previous methods often suffer from the severe sparsity issues, due to limited corpora available and the long-tail phenomenon of predicate distributions. In this paper, we propose a multi-stage method, Typed Predicate-Entailment Graph Generator (TP-EGG), to tackle this problem. Given several seed predicates, TP-EGG builds the graphs by generating new predicates and detecting entailment relations among them. The generative nature of TP-EGG helps us leverage the recent advances from large pretrained language models (PLMs), while avoiding the reliance on carefully prepared corpora. Experiments on benchmark datasets show that TP-EGG can generate high-quality and scale-controllable entailment graphs, achieving significant in-domain improvement over state-of-the-art EGs and boosting the performance of down-stream inference tasks.

## introduction the entailment relation between textual predicates plays a critical role in natural language inference and natural language understanding tasks, including question answering @xcite and knowledge graph completion @xcite @xcite . to detect entailment relations, previous works pay attention to the recognizing textual entailment (rte) task, which takes a pair of sentences as input and predicts whether one sentence entails the other @xcite @xcite . current rte models perform well on rte benchmarks, but most of them are lacking in explainability, as they make use of the black-box language models (lm) without providing any explainable clues. recent works focus on learning the entailment graph (eg) structure, which organizes typed predicates in directional graphs with entailment relations as the edges @xcite @xcite , as shown in figure 1 . with the explicit graph structure containing predicates and their entailment relations, similar to knowledge graphs (kgs), using egs becomes an explainable and context-independent way to represent the knowledge required in natural language inference and other nlp tasks. most existing egs are constructed with the distributional inclusion hypothesis (dih), which suggests that all typical context features of a predicate v can also occur with another predicate w if v entails w @xcite . constructing egs with dih requires distributional cooccurrences of contextual features from large corpora to calculate the semantic similarity between predicates @xcite . however, the egs constructed from large corpora often suffer from two different kinds of sparsity issues: the predicate sparsity and the edge sparsity. existing corpora used for eg construction are mainly collected from specific resources @xcite , such as news articles. as a result, entailment relations could not be learned between those predicates that do not appear in the corpora, which leads to the predicate sparsity issue. meanwhile, if two predicates scarcely appear around similar contexts in the given corpora, the dih could not indicate the potential entailment relationship between them. it leads to the edge sparsity of egs as the corresponding edges may be missing due the limited coverage of the corpora. to tackle the sparsity issues, previous works pay attention to learning global graph structures to mine latent entailment relations and alleviate the edge sparsity @xcite @xcite , but predicate sparsity is still holding back the improvement of egs. solving predicate sparsity by simply scaling up the distributional feature extraction is impracticable, due to the long-tail phenomenon of predicate distribution @xcite . the shortcomings of extractive methods come in quest for non-extraction way to overcome. recent progress in deep generative lms, including gpt-3 @xcite and t5 @xcite , makes it possible to produce predicates and entailment relations by generative methods. inspired by the commonsense transformer @xcite , we propose a novel generative multi-stage eg construction method, called typed predicate-entailment graph generator (tp-egg). as shown in figure 2 , tp-egg takes several seed predicates as input of the lm-based predicate generator to depict the domain of predicates and generate more in-domain predicates. with generated predicates, tp-egg uses a novel transitivity-ensured edge se-lector by representing predicates as spheres in the vector space, to pick out the potential entailment relations among generated predicates. then tp-egg calculates the corresponding edge weights by the lm-based edge calculator. our key insight is that by re-modeling the predicate extraction process as a generation process, we can leverage the underlying knowledge about natural language inference inside the lms to avoid the data sparsity issues of extractive methods. by choosing appropriate seed predicates and setting the parameters of tp-egg, one can generate egs containing knowledge from a specific domain in arbitrary scales to fit the downstream requirement, without limitations from the uncontrollable distribution in domain-independent corpora. since almost all the eg construction modules in tp-egg is controlled by pre-trained lms, the output egs can be seen as explicit representations of the knowledge in lms and used in downstream tasks, such as rte in our experiments. by counts or pointwise mutual information @xcite , and compute the distribution similarity guided by dih. for a predicate pair, different similarities are calculated, such as cosine similarity, lin @xcite , weed @xcite , and balanced inclusion @xcite . markov chain of predicate-argument transition @xcite and temporal information from extracted corpora @xcite are also used in egs construction. these methods independently calculate the entailment relations for each pair, called local methods. besides, global constraints are used to detect new entailment relations beyond local relations. the transitivity in egs, which means a entails b and b entails c indicate a entails c for three predicates a, b and c, is the most widely used in previous works as hard constraints @xcite or soft loss functions @xcite . the weight similarity constraints between different typed egs and similar predicates are also taken into consideration @xcite . as one of the most important areas of nlp, text generation, or natural language generation (nlg), has also been advanced by the surgent development of pre-trained lms. bart @xcite uses encoder-decoder transformer architecture to re-correct the corrupted data in pre-training phase; gpt-3 @xcite uses transformer decoder to achieve in-context learning with massive multi-task unsupervised data. t5 @xcite unifies different tasks into natural language prefixes and solves them by text generation. pre-trained lms are also applied in recent eg methods. cnce @xcite initializes the contextualized embeddings of entity-relation triplets by bert @xcite and uses random walk to get the entailment probability; egt2 @xcite fine-tunes a patternadapted lm on the predicate sentences and recalculates high-quality edge weights for global constraints; @xcite applies roberta @xcite as predicate encoder and matches missing predicates in egs with k-nearest neighbor algorithm to alleviate the predicate sparsity. as far as we are concerned, our method is the first attempt to use generative lm in eg construction and directly generate egs without the distributional features from large corpora. ## our approach egs store predicates as nodes and entailment relations between them as edges in graph structures. following previous eg methods @xcite @xcite , we use the neo-davisonian semantic form of binary relation @xcite to indicate typed predicates, whose types are defined by the combination of argument types. predicate p connecting two arguments a 1 , a 2 with types t 1 , t 2 can be represented as @xmath0.i 1 , w 2 .i 2 , t 1 , t 2 ), where w j is the center relation tokens (and perhaps prepositions) about a j , and i j is corresponding argument order of a j in w j . for example, the event "the government is elected in 1910 and adored by natives" contains two predicates (elect.2, elect.in.2, government, time) and (adore.1, adore.2, person, government). we denote p as the collection of all typed predicates, t as the collection of all argument types, and τ 1 , τ 2 : @xmath1, where τ 1 (p) = t 1 and τ 2 (p) = t 2 for any predicate @xmath2.i 1 , w 2 .i 2 , t 1 , t 2 ). we formally define that a typed entailment graph g(t 1 , t 2 )=<p (t 1 , t 2 ), e(t 1 , t 2 )> includes the collection of typed predicates p (t 1 , t 2 ) = {p|(τ 1 (p), τ 2 (p)) ∈ {(t 1 , t 2 ), (t 2 , t 1 )}}, and the directional weighted edge set e(t 1 , t 2 ), which can be represented as an adjacent matrix w (t 1 , t 2 ) ∈ [0, 1] |p (t 1 ,t 2 )|×|p (t 1 ,t 2 )| . for those g(t 1 , t 2 ) whose t 1 ̸ = t 2 , the order of types t1, t2 is naturally determined. when t 1 = t 2 = t, argument types are ordered such that g(t, t) can determine the order of types like "thing a" and "thing b" to distinguish predicates like "thing a eat thing b" and "thing b eat thing a". this order obviously affect the meaning of predicates, as "thing a eats thing b" entails "thing b is eaten by thing a", but "thing eats thing" is doubtful to entail "thing is eaten by thing". ## experimental setup datasets. following previous works @xcite @xcite , we include levy/holt dataset (levy and dagan, 2016; holt, 2018) and berant dataset @xcite into eg evaluation datasets. besides, we reorganize the sherliic dataset @xcite , a dataset for lexical inference in context (liic), into an eg benchmark. we further reannotate conflicting pairs in levy/holt, referred as levy/holt-r dataset. dataset statistics are shown in