Knowledge-Enhanced Evidence Retrieval for Counterargument Generation (2021)

Abstract: Finding counterevidence to statements is key to many tasks, including counterargument generation. We build a system that, given a statement, retrieves counterevidence from diverse sources on the Web. At the core of this system is a natural language inference (NLI) model that determines whether a candidate sentence is valid counterevidence or not. Most NLI models to date, however, lack proper reasoning abilities necessary to find counterevidence that involves complex inference. Thus, we present a knowledge-enhanced NLI model that aims to handle causality- and example-based inference by incorporating knowledge graphs. Our NLI model outperforms baselines for NLI tasks, especially for instances that require the targeted inference. In addition, this NLI model further improves the counterevidence retrieval system, notably finding complex counterevidence better.

## introduction generating counterarguments is key to many applications, such as debating systems @xcite , essay feedback generation @xcite , and legal decision making @xcite . in nlp, many prior studies have focused on generating counterarguments to the main conclusions of long arguments, usually motions. although such counterarguments are useful, argumentative dialogue is usually interactive and synchronous, and one often needs to address specific statements in developing argument. for instance, in the change-myview (cmv) subreddit, challengers often quote and counter specific statements in the refuted argument, where 41% of these attacks are about factual falsehood, such as exceptions, feasibility, and lack of evidence @xcite . hence, the scope of our work is narrower than most prior work. instead 1 source code and data are available at https:// github.com/yohanjo/kenli . of generating a counterargument to a complete argument, we aim to find counterevidence to specific statements in an argument. this counterevidence may serve as essential building blocks for developing a larger counterargument and also allow for more interactive development of argumentation. we adopt a popular fact-verification framework @xcite : given a statement to refute, we retrieve relevant documents from the web and select counterevidence (figure 1 ). at the core of this framework is a module that determines whether a candidate sentence is valid counterevidence to the given statement. a natural choice for this module is a natural language inference (nli) model. but nli models to date have shown a lack of reasoning abilities @xcite , which is problematic because counterarguments often involve complex inference. to overcome this limitation, we enhance nli by focusing on two types of inference informed by argumentation theory @xcite . the first is argument from examples, as in: claim: vegan food reduces the risk of diseases. counterevidence: legume protein sources can result in phytohemagglutinin poisoning. the inference is based on the fact that "legume protein sources" and "phytohemagglutinin poisoning" are examples of "vegan food" and "diseases", respectively. the second type of inference is argument from cause-to-effect, as in: claim: veganism reduces the risk of diabetes. counterevidence: vegan diets suffer from poor nutrition. the inference is based on the fact that poor nutrition can cause diabetes. in order to handle causality-and example-based inference, we develop a knowledge-enhanced nli model ( §3). by incorporating two knowledge graphs-causenet and wikidata-into the model while training on public nli datasets, the accuracy of the nli model improves across nli datasets, especially for challenging instances that require the targeted inference. we integrate this nli model into the entire retrieval system to find counterevidence to argumentative statements from two online argument platforms, changemyview (cmv) and kialo ( §4). we demonstrate that our knowledge-enhanced nli model improves the system, finding more complex counterevidence. we also conduct in-depth analyses of the utility of different types of source documents and document search methods (wikipedia, bing, and google). our contributions are as follows: ## knowledge-enhanced nli a natural language inference (nli) model is the core of our entire system (figure 1 ). given a statement to refute, the system retrieves and ranks relevant documents, and then obtains a set of candidate sentences for counterevidence. for each candidate, the nli model decides whether it entails, contradicts, or neither the statement. in this section, we first motivate the model design and explain our knowledge-enhanced nli model (kenli), followed by evaluation settings and results. ## retrieval of counterevidence our system for counterevidence retrieval builds on deseption @xcite , a state-of-theart system for the fact extraction and verification (fever) task @xcite . as figure 1 shows, given a statement to verify, it retrieves and ranks relevant documents, ranks candidate evidence sentences, and predicts whether the statement is supported, refuted, or neither. we adapt deseption to suit our task, where the main contribution is to strengthen the last stage via our knowledge-enhanced nli model (a detailed comparison between our system and deseption is in appendix c). we first explain individual stages and then describe evaluation settings and results. ## conclusion in this paper, we built a counterevidence retrieval system. to retrieve counterevidence that involves complex inference, we presented a knowledgeenhanced nli model with specific focus on causality-and example-based inference. the nli model demonstrates improved performance for nli tasks, especially for instances that require the targeted inference. integrating the nli model into the retrieval system further improves counterevidence retrieval performance, especially recall, showing the effectiveness and utility of our method of incorporating knowledge graphs in nli. for all annotation tasks, we recruited annotators on amazon mechanical turk (mturk). participants should meet the following qualifications: (1) residents of the u.s., (2) at least 500 hits approved, and (3) hit approval rate greater than 97%. each hit includes several questions and one attention question. the attention question asks the annotator to select a specific option, and we rejected and discarded a hit if the annotator failed the attention question. for all annotation tasks, the annotation manuals are publicly available.