Multivalent Entailment Graphs for Question Answering (2021)

Abstract: Drawing inferences between open-domain natural language predicates is a necessity for true language understanding. There has been much progress in unsupervised learning of entailment graphs for this purpose. We make three contributions: (1) we reinterpret the Distributional Inclusion Hypothesis to model entailment between predicates of different valencies, like DEFEAT(Biden, Trump) entails WIN(Biden); (2) we actualize this theory by learning unsupervised Multivalent Entailment Graphs of open-domain predicates; and (3) we demonstrate the capabilities of these graphs on a novel question answering task. We show that directional entailment is more helpful for inference than non-directional similarity on questions of fine-grained semantics. We also show that drawing on evidence across valencies answers more questions than by using only the same valency evidence.

## introduction we are reading a mystery about a dark and foreboding manor and have one question: "is mr. boddy dead?" 1 our text might say "colonel mustard killed mr. boddy," or "mr. boddy was murdered in the kitchen with a candlestick," either of which answers the question, but only via natural language inference. an entailment graph (eg) is a structure of meaning postulates supporting these inferences such as "if a kills b, then b is dead." entailment graphs contain vertices of opendomain natural language predicates and entailments between them are represented as directed edges. previous models learn predicates of a single valency, the number and types of arguments controlled by the predicate. commonly these are binary graphs, which cannot model single-argument predicates like the entity states "is dead" or "is an author." this means they miss a variety of entailments in text that could be used to answer questions such as our example. the distributional inclusion hypothesis (dih) @xcite is a theory which has been used effectively in unsupervised learning of these same-valency entailment graphs @xcite @xcite . in this work the dih is reinterpreted in a way which supports learning entailments between predicates of different valencies such as kill(mustard, boddy) die(boddy). we extend the work of hosseini et al. @xcite and develop a new multivalent entailment graph (mgraph) where vertices may be predicates of different valencies. this results in new kinds of entailments that answer a broader range of questions including entity state. we further pose a true-false question answering task generated automatically from news text. our model draws inferences across propositions of different valencies to answer more questions than using same-valence entailment graphs. we also compare with several baselines, including unsupervised pretrained language models, and show that our directional entailment graphs succeed over non-directional similarity measures in answering questions of fine-grained semantics. advantageously, egs are structures designed to be queried, so they are inherently explainable. this research is conducted in english, but as an unsupervised algorithm it may be applied to other languages given a parser and named entity linker. ## background the task of recognizing textual entailment @xcite requires models to predict a relation between a text t and hypothesis h; "t entails h if, typically, a human reading t would infer that h is most likely true." from here, research has moved in several directions. we study predicates, including verbs and phrases that apply to arguments. research in predicate entailment graphs has evolved from "local" learning of entailment rules @xcite to later work on joint learning of "globalized" rules, overcoming sparsity in local graphs @xcite . these graphs frequently rely on the dih for the local learning step to learn initial predicate entailments. the dih states that for some predicates p and q, if the contextual features of p are included in those of q, then p entails q @xcite . in previous work predicate arguments are successfully used as these contextual features, but only predicates of the same valency are considered (e.g. binary predicates entail binary; unary entail unary), and further research computes additional edges in these same-valency graphs such as with link prediction @xcite . however, this leaves out crucial inferences that cross valencies such as the kill/die example, which are easy for humans. we generalize the dih to learn entailments within and across valencies. typing is very helpful for entailment graph learning @xcite @xcite . inducing a type for each entity such as "person," "location," etc. enables generalized learning across instances and disambiguates word sense, e.g. "running a company" has different entailments than "running code." we compare our model to several baselines, including strong pretrained language models in an unsupervised setting using similarity. bert @xcite generates impressive word representations, even unsupervised @xcite , which we compare with on a task of predicate inference. we further test roberta @xcite to show the impact of robust in-domain pretraining on the same architecture. these non-directional similarity models provide a strong baseline for evaluating directional entailment graphs. ## multivalent distributional inclusion hypothesis we pose a new, multivalent interpretation of the dih (the mdih) which models the entailment of predicates across valencies. the intuition comes from observing eventualities @xcite which occur in the world. neo-davidsonian semantics @xcite explains that a textual predicate, its arguments, and adjuncts, are all properties of an underlying event variable. en-tailments about one or more of the arguments arise from their roles in this eventuality. we may infer that "mr. boddy died" due to his role as a direct object in the killing/murdering event. no other information is needed, including who murdered mr. boddy, where, or with what instrument. boddy is dead simply because he was murdered. we build on this insight to develop the mdih. here, a predicate is represented (as in §2) by features which are the argument tuples it appears with. we recognize a tuple as a proxy for a world event, e.g. visit(obama, hawaii) identifies one instance of a real visit event. our method learns by tracking entity tuples across events in the world. the mdih signals an entailment from a premise p to hypothesis h if, distributionally, subtuples of p are always found amongst tuples of h. crucially, we allow h to drop in valency so that we learn entailments about subsets of p's arguments. we now formalize the mdih and then illustrate with an example. we define the argument tuple structures for a premise and hypothesis predicate: p is a set of m argument tuples (each of size i) which correspond to instances of a premise predicate p. h is a set of n argument tuples (each of size j) representing the same for hypothesis h. we limit @xmath0, e.g. we learn about relations on realized entity subsets. we do not learn entailments to higher valencies (such as a unary entailing a binary) because additional arguments must be existential, not real. we leave this to future work. to select argument subtuples from tuples in p , we define a vector of indices j with length j, which selects arguments by position. for example, with @xmath1, 3], perform p [:, j]. for each argument tuple in p , select just the 2nd and 3rd arguments, forming a new set of 2-tuples. we define the multivalent distributional inclusion hypothesis: if p [:, j] ⊆ h[:, m(j)], then p h here m : n @xmath2. an example where m is needed for argument swapping is "x bought y" entails "y sold to x." we illustrate by working the kill/die example on a hypothetical corpus. we might find that kill(x, y) die(y) by trying @xmath3. we start with p , all 2-tuples of killings, and h, all 1-tuples of dyings and apply j and m. we may find that selecting arg 2 from all tuples in p forms a subset of the selection of arg 1 from tuples in h. though dyings may happen in many ways, we observe that arg 2 of a killing often occurs elsewhere in the corpus with a dying, and thus we infer the entailment between predicates. intuitively this is true for arbitrarily large valencies: murder(mustard, boddy, kitchen, candlestick) entails kill(mustard, boddy) and both entail die(boddy). though arguments may be dropped from the premise, they still influence entailments. this is because the mdih tracks eventualities. "person writing a book" is a different kind of event than "person writing software" with a different distribution of argument tuples, so we learn that the former entails being an author, while the latter entails being a programmer. ## learning multivalent graphs we define an entailment graph as a directed graph of predicates and their entailments, @xmath4, e). the vertices v are the set of predicates, where each argument has a type from the set of 49 figer base types t , e.g. travel.to(:person, :location) ∈ v , and :person, :@xmath5. the directed edges are in multivalent entailment graphs we expand v to contain predicates of both 1-and 2-valency, and e to edges between these vertices, described as follows. let b i , b @xmath6, u @xmath7. define e as the set of all entities in the world, and some particular entities x, @xmath8. e contains these patterns of entailment: predicates with @xmath9, but are also included in the mgraph by decomposing them into binary relations between pairs of entities. this is another application of our multivalent dih. we maintain argument roles, so each binary is a window into its higher-valency predicate, allowing higher-valency predicates to entail lower binaries and unaries. to learn these new kinds of connections we develop a method of local entailment rule learning using the mdih. as in §2, the local step learns the initial directed edges of the entailment graph, which are further improved with global learning. our local step learns entailments by machine-reading the newsspike corpus (2.3gb), which contains 550k news articles, or over 20m sentences @xcite . newsspike consists of multi-source news articles collected within a fixed timeframe, and due to these properties the articles frequently discuss the same events but phrased in different ways, providing appropriate training evidence. ## evaluation: question answering we pose an automatically generated qa task to evaluate our model explicitly for directional inference between binary and unary predicates, as we are not aware of any standard datasets for this problem. our task is to answer true-false questions about real events that are discussed in the news, for example, "was biden elected?" these types of questions are surprisingly difficult and frequently require inference to answer @xcite . for this, entailment is especially useful: we must decide if the question (hypothesis) is true given a list of propositions from limited news text (premises), which are all likely to be phrased differently. this task is designed independently of the mgraph as a challenge in information retrieval. positive questions made from binary and unary predicates are selected directly from the news text using special criteria, and are then removed. from these positives we automatically generate false events to use as negatives, which are designed to mimic real, newsworthy events. the remaining news text is used to answer the questions. we at-tempt to make every question answerable, but since they are generated automatically there is no guarantee. however, the task is fair as all models are given the same information. the additive effects of multivalent entailment should be demonstrated: with more kinds of entailment, the mgraph should find more textual support and answer more questions. the task is presented on a text sample from newscrawl, a multi-source corpus of news articles, to be published separately. a test set is extracted which contains 700k sentences from articles over a period of several months, and also a development set from a further 500k sentences. we generate questions balanced to a ratio of 50% binary questions / 50% unary; and within each 50% positive / 50% negative. ## question answering results the models produce a gradation of judgement scores between 0 (false) and 1 (true). as in earlier work, we slide a classification threshold over the score range to produce a precision-recall curve for each model. results are in figure 3 (left). multivalent graph performance is shown incrementally. the bb model can answer a portion of binary questions; the uu model can answer more unary questions; adding the bu model can answer still more unary questions using binary evidence. we observe successful inference of our kill/die example and others. "obama was elected to office" affirms the question "was obama a candidate?" and "zach randolph returned" affirms "did zach randolph arrive?" our test set is from multiple sources over the same time period. the exact-match baseline shows the limitations of answering questions simply by collecting more data; most questions require inference to answer. the complete mgraph achieves ~3x this recall by drawing inferences. our model achieves higher precision than bert and roberta similarity models in the low recall range. the similarity models perform well, achieving full recall by generalizing for rarer predicates. we note that roberta bests bert due to extensive in-domain pretraining. the bb model appears to struggle. in fact 90.5% of unary questions have a vertex in the graph, but only 64.1% of binaries do. the bb model frequently cannot answer questions because the question predicate wasn't seen in training. this difference is because binary predicates are more diverse so suffer more from sparsity: they are often multiword expressions and have a second, typed argument. indeed, most binary predicate research (in symbolic methods) focuses on only the top 50% of recall in several datasets @xcite @xcite . for an even comparison we create a filtered question set. from all questions we remove those without a vertex in the mgraph, then balance them as in §5, resulting in 20,519 questions (10,273 unary and 10,246 binary). this filtered test directly compares the models, since both the entailment graphs and the similarity models have a chance to answer all the questions. results are shown in figure 3 (right), with a very different outcome. head-to-head, the mgraph offers substantially better precision across all recall levels. at 50% recall, the mgraph has 76% precision with roberta at 65%. notably, on both tests, more unary questions are answered using both unary and binary predicate evidence than just using unary evidence alone. on the filtered test, the bu model increases max recall from 54% to 70%. finally, we note ppdb's poor performance (highest recall shown), only 1% higher recall than the exact-match baseline despite having entries for 88% of questions. though ppdb features many directional entailments, this sparsity of edges useful for the task is likely because bilingual pivoting excels at detecting near-paraphrases, not relations between distinct eventualities, e.g. it can't learn "getting elected" entails "being a candidate." advantageously, our method learns this open-domain knowledge by tracking entities across all the events they participate in. we show a breakdown of the filtered test results in ## error analysis we sample 300 false positives (100 for each model) and report analyses in ## conclusions the mdih is shown as an effective theory of unsupervised, open-domain predicate entailment, which crosses valencies by respecting argument roles. our multivalent entailment graph's performance has been demonstrated on a question answering task requiring fine-grained semantic understanding. our method is able to answer a broader variety of questions than earlier entailment graphs, aided by drawing on evidence across valencies. we outperform baseline models including a strong similarity measure using unsupervised bert and roberta, while using far less training data. this shows that directional entailment is more helpful for inference on such a task than non-directional similarity, even with robust, in-domain pretraining. we also noted a complementarity between unsupervised methods. our symbolic graph method achieves high precision for learned predicates, while sub-symbolic neural models achieve high recall by generalizing to unseen predicates. future work may leverage our mdih signal to train a directional neural classifier and combine benefits.