Blindness to Modality Helps Entailment Graph Mining (2021)

Abstract: Understanding linguistic modality is widely seen as important for downstream tasks such as Question Answering and Knowledge Graph Population. Entailment Graph learning might also be expected to benefit from attention to modality. We build Entailment Graphs using a news corpus filtered with a modality parser, and show that stripping modal modifiers from predicates in fact increases performance. This suggests that for some tasks, the pragmatics of modal modification of predicates allows them to contribute as evidence of entailment.

## introduction the ability to recognise textual entailment and paraphrase is crucial in many downstream tasks, including open domain question answering from text. for example, if we pose the question "did joe biden run for president?" and the text states that "joe biden was elected president", producing the correct answer (yes) necessitates understanding that being elected president entails running for president 1 . entailment graphs, constructed via unsupervised learning techniques over large text corpora, provide a solution to this problem. they consist of nodes representing predicates and directed edges representing entailment relations between them. given the importance of detecting uncertainty for other downstream nlp tasks such as information extraction @xcite , information retrieval @xcite , machine reading @xcite , and question answering @xcite one might expect that it would also be useful in learning entailment graphs. that is, they would be more reliable if learned from data in which predications * equal contribution 1 assuming a democratic election. we use the typical definition of the premise most likely entailing the hypothesis @xcite are asserted as actually happening, rather than data with uncertain predications under scope of various types of modality. we investigate whether this is the case. the entailment graph-learning algorithm depends on descriptions of eventualities in the news, observing directional co-occurrences of typed predicates and their arguments. for example, we expect to observe all the arguments of being president, such as biden and obama, also to be encountered in a sufficiently large multiply-sourced body of text as arguments of running for president, but not the other way around (hillary clinton will run but not be president). however, if all the reports of clinton might be president are extracted as be_president(clinton), one might expect the learning signal to be confusing to the algorithm. we use the method of hosseini et al. (2018) combined with a modality parser (bijl de @xcite to construct typed entailment graphs from raw text corpora under two different settings. modality-aware: modal predications are removed from the data entirely, and modality-unaware: the model learns from both asserted and modal predications. our contributions are 1) a comparison of entailment graphs learned from modal and nonmodal data, showing (counterintuitively) that ignoring modal distinctions in fact improves entailment graph-learning, and 2) insights as to whether this effect applies uniformly across different subdomains. ## background entailment rules specify directional inferences between linguistic predicates @xcite , and can be stored in an entailment graph, whose global structural properties can be used to learn more accurately @xcite . they are defined as a directed graph @xmath0, e}, in which the nodes n are typed predicates and edges e represent the entailment relation. the lex-category example ∅ protesters attacked the police modal operator protesters may have attacked the police conditional if protesters attack the police... counterfactual had protesters attacked the police... propositional journalists said that attitude protesters attacked the police a subgraph containing predicates of a type-pair (e.g. person-location) can be learned in an unsupervised way from collections of multiplysourced text. a vector of argument-pair counts for every predicate is first machine read from the corpus. typically, relation extraction systems used for reading these corpora ignore modal modifiers, possibly introducing noise in the graph. next, a (directed) similarity score (e.g. dirt @xcite ), weed's score @xcite or binc @xcite ) is computed between the vectors, producing a local entailment score between each predicate pair. then a globalisation process such as the soft constraints algorithm of @xcite , which transfers information both within and between type-pair subgraphs, can be used to refine these local scores. when using the graph in practice, all edges with a score above a chosen threshold can be considered an entailment. there are various semantic phenomena a speaker can use to mark veridicality (see these phenomena have been investigated for various nlp tasks, including uncertainty detection @xcite , hedge detection @xcite and modality annotation @xcite . capturing this information is valuable to tasks such as information extraction, question answering and knowledge base population @xcite . early approaches to detecting modality focused on lexicon design @xcite @xcite , with later approaches using machine learning over annotated corpora @xcite @xcite . recently, bijl de vroe et al. (2021) designed a parser similar to that by @xcite , to cover a wider range of phenomena, including conditionality and propositional attitude. while modality annotation is clearly useful for recognising entailment from a given text @xcite , to our knowledge no research has been conducted on its effect on learning entailment graphs. ## methods we extend relation extraction to pay attention to modality, so that we can distinguish modal and nonmodal relations in the entailment graph mining algorithm. this allows us to investigate the impact of modalised predicate data on the accuracy of learned entailment edges. we extract binary relations of the form arg1predicate-arg2 using montee, an open-domain modality-aware relation extraction system @xcite . montee uses the rotat-ingccg parser (stanojević and steedman, 2019) as the basis for extracting binary relations and a modality lexicon to identify modality triggers. a relation is tagged as modal (mod), propositional attitude (att_say, att_think) or conditional (cond) if the ccg dependency graph contains a path between a relation node and a node matching an entry in the montee lexicon. counterfactuals (count) are tagged according to hand-crafted rules. since we focus on uncertainty and not negation, lexical negation (lneg) tagging is ignored. in the modality-aware setting, we remove relations tagged by montee as any kind of modal ({mod, att_say, att_think, count, cond}). in local learning, learned entailment edges then have access only to non-modal evidence: eventualities that were asserted as actually happening. for example, the edge between win and lose should now be learned only from non-modal descriptions such as a won today against b or a has been defeated by b, leaving out modal descriptions (a could beat b). the local and globalisation parts of the algorithm are otherwise unchanged. ## experimental setup using montee foot_0 , we extract 40,669,812 binary relation triples from the newsspike corpus @xcite . of these, 14.57% are tagged; 10.04% mod, 3.51% rep_say, 0.38% rep_think, 0.61% cond, and 0.03% count. we then construct three different datasets and build an entailment graph with each. the modalityunaware baseline, baselinelarge, is trained on the complete set of relations with modality tags removed. this corresponds to the data and model in @xcite . for the modality-aware asserted graph, we extract only the set of 34,744,216 asserted relations (∼85% of the relations), i.e. all modal relations are excluded. to rule out effects of data size, we construct baselinesmall, which is trained on a random sample of 85% relations from the total set. comparing asserted to base-linelarge shows us whether it is worth filtering out modal data, and comparing asserted to baselinesmall shows whether asserted data or mixed data (i.e. asserted and modal) is more effective for learning entailment relations. we follow the example of hosseini et al. (2018) and construct typed graphs for all possible type pairs (e.g. person-location). relation arguments are typed by linking to a named entity freebase identifier @xcite using the aida-light linker @xcite , and mapping these identifiers to a type in the figer hierarchy @xcite . the typed relations become the input to the graph learning step of the entailment graph mining algorithm. following previous research, we use the binc similarity score @xcite to compute entailment scores. we first construct local typed entailment graphs and then globalise the scores across graphs as in @xcite . we evaluate the entailment graphs on two datasets. the first is the levy/holt entailment dataset, a set of 18,407 entailment pairs for the general domain @xcite . as our training method is unsupervised and we do not tune hyperparameters, we evaluate on the complete levy/holt dataset rather than the dev/test split. we also evaluate on the sports entailment dataset @xcite , focusing on the sub- all three entailment graphs cover this range and predictions with precision higher than random are important for downstream applications. on the levy/holt dataset (all examples), base-linelarge performs best overall. the strong performance of baselinelarge compared to asserted is in itself surprising, and indicates that it is usually not beneficial to distinguish modality when building entailment graphs. this can be understood as a data size issue: filtering out data is harmful as it introduces sparsity, and modal data is useful enough to provide a learning signal. more counterintuitive, however, is that even baselinesmall, which controls for training dataset size, outperforms asserted. to understand why, we measured the size of each graph in terms of the number of nodes (predicates) and edges (entailment relations) it contained, and the percentage of predicates in the levy/holt dataset that were present in the graph (see another explanation might be that this richer predicate coverage allows baselinesmall to accurately correlate more of the common paraphrase examples in the levy/holt dataset. to this end we investigated the directional portion of the levy/holt dataset, which contains 2,414 examples of both the entailment pair and its reverse, where the entailment is true in one direction and false in the other. as noted by @xcite this task is much harder than that represented by the original dataset. however, the baselines both outperform the asserted graph on the directional entailment task. we also observe a similar pattern in the percentage of predicates covered (see last column in performance on the sports entailment dataset (figure 2 ) reveals a different pattern. base-linelarge outperforms baselinesmall as expected, but asserted performs best, despite lower coverage of the predicates in the sports entailment dataset (see ## discussion & future work another appealing intuition for the usefulness of modal relations is that they might generally be expressed in text when the prior probability of the main predicate is already high. this would lead the distributions for the main predicates to be improved in spite of the uncertainty of the evidence. additionally, if the probability of a premise is high enough to be worth mentioning, then in general that of its entailments will be too. however, this may not hold for the sports scenario because the outcomes are widely speculated upon despite being highly uncertain. indeed it is easy to find examples in the news corpus to support these intuitions. in the general domain we observe examples of eventualities initially being discussed with uncertainty, and later mentioned as asserted. an example of this is the acquisition of dell by michael dell: on february 5th, 2013 we observe "... founder and ceo michael dell and investment firm silver lake partners will buy dell.", and subsequently, on february 6th, 2013 we read "so michael dell and a private equity group have bought dell and taken it private.". we also observe the reverse scenario in the sports domain. for example, on january 10th, 2013 we observe "the popular opinion on this game seems to be seattle beating atlanta because...", while shortly afterwards we are informed that "falcons come back to beat seahawks". the latter is likely rather domain-specific, and we may expect to find a similar effect for other domains that share the disjunctive outcome property, for example elections, court cases and battles, where modals are used when speculating about potential and counterfactual outcomes. we will explore ways to leverage this information and consider other sub-domains for which it is useful to retain or remove modal data. this may involve creating more domain-specific datasets. it is also worth investigating the effects of negation, which shares similar properties to modality, on learning entailment graphs. relatedly, we could retain predicates under specific modal modifiers, as these correspond to different prior probabilities of eventualities, carrying a different epistemic commitment from the writer. eventualities that happen "undoubtedly" might be preferred over those that are "unlikely", for instance, and the modality parser can output specific categories of modality, allowing us to choose the subsets that should be kept. finally, we will experiment with learning entailment graphs with modal predicate nodes, by retaining modal relations with tags attached as input. many of these entailments are trivial, because any entailment of a consequence can be reproduced under modal scope (if @xmath1, then also @xmath2. more notably, we might recover that following an entailment in the reverse direction can produce a modal entailment (e.g. if @xmath3, then we know @xmath4, and many preconditions will behave interestingly (e.g. @xmath5, but also @xmath6. to evaluate this idea, we will design a dataset of modal entailments, drawing inspiration from previous research on veridicality in entailment datasets @xcite . ## conclusion we have investigated the role of modally modified relations in entailment graph mining, and shown that, contrary to results from other tasks, uncertain predications actually constitute a valuable learning signal overall. further analysis shows that there are specific predicate domains in which removing modal data is beneficial.