Team Papelo at FEVEROUS : Multi-hop Evidence Pursuit (2021)

Abstract: We develop a system for the FEVEROUS fact extraction and verification task that ranks an initial set of potential evidence and then pursues missing evidence in subsequent hops by trying to generate it, with a “next hop prediction module” whose output is matched against page elements in a predicted article. Seeking evidence with the next hop prediction module continues to improve FEVEROUS score for up to seven hops. Label classification is trained on possibly incomplete extracted evidence chains, utilizing hints that facilitate numerical comparison. The system achieves .281 FEVEROUS score and .658 label accuracy on the development set, and finishes in second place with .259 FEVEROUS score and .576 label accuracy on the test set.

## introduction the 2021 feverous (fact extraction and verification over unstructured and structured information) task @xcite introduces several challenges not seen in the 2018 fever task @xcite . tabular information, lists, and captions now appear as evidence, in addition to natural text sentences. most claims now require multiple pieces of supporting evidence to support or refute them. even claims that cannot be fully verified now require the submission of supporting evidence for aspects of the claim that can be verified. counting and numerical reasoning skills are needed to verify many claims. annotators for feverous differed in their interpretation of what constituted necessary evidence, and often added duplicate evidence that should be in an alternative reasoning chain to a main reasoning chain. for this reason it is dangerous to target a precise, minimal set of evidence as in fever for high evidence f1 @xcite , and we instead fill the full set of five sentences and 25 table cells permitted for submission. thus we focus on solving the evidence retrieval problem and first assemble a set of preliminary set of relevant facts. several of these facts may be combined to determine the veracity of the claim. @xcite define multi-hop reasoning as reasoning with information taken from more than one document to arrive at an answer, so using the preliminary evidence set could already be multihop reasoning, but from the perspective of retrieval we consider retrieving the initial evidence set to be a first "hop." where multi-hop reasoning is required, it may be necessary to retrieve additional documents after reading the preliminary evidence, which could not be searched for using the claim alone. we support this functionality by predicting whether evidence chains are complete and generating additional search queries based on the preliminary evidence. this next hop prediction module can be applied as many as seven times to update the evidence chains, each time improving the fever-ous score. on the final evidence chains, the label ("supports", "refutes", or "not enough information") is predicted by a module trained on extracted evidence chains. because "not enough information" (nei) labels are scarce, we alternatively can decide whether to give an nei label based on whether the next hop prediction module is still seeking more evidence for the claim. inputs are carefully represented to facilitate numerical comparisons for the final label decision and to allow the use of other contextual information by every module. the described system attains a feverous score of .281 on the development set with label accuracy of .658. ## context and structured information downstream classifiers usually classify page elements in isolation, but the meaning of these elements sometimes is not clear without contextual information. in the fever task, attaching a prefix to each sentence consisting of the page title type example sentence [ mississippi river ] when measured from its traditional source at lake itasca, the mississippi has a length of 2,320 miles (3,730 km). list item [ temple tower ] list context cast value marceline day as patricia verney table cell [ temple tower ] value release date {{ key temple tower value april 13, 1930 }} table cell [ l-arabinose operon ] caption catabolism of arabinose in e. coli {{ key substrate value l-arabinose }} key enzyme(s) value araa key function value isomerase key reversible value yes key product value l-ribulose in brackets improved performance @xcite , for example by providing hints about what pronouns might refer to. we continue this practice for feverous. for list elements, we take the page element immediately preceding the list as context. this often is a sentence indicating what is in the list. then the list element is represented by "[ title ] con-text context value list item", so that the list element and what the list is about may be seen simultaneously. for table cells, we represent the entire row containing the cell. if a cell in a row above has an is_header attribute, the cells are prefixed with "key header". this is followed by the actual value from the current row, in the form "value header". thus each cell in a row looks like a combination of key/value pairs (or simply values if there is no header). this representation is similar to the one used by @xcite . all the cells in a row would look alike if we simply followed this procedure, so we distinguish the key/value pair corresponding to the current cell by enclosing it in double braces. finally, the title is prepended, and if there is a caption, it is prepended as "caption caption". examples of the table cell, list element, and sentence formats are shown in ## preliminary evidence retrieval we follow the baseline system @xcite to select an initial set of documents for downstream analysis. this module retrieves documents whose titles match named entities that appear in the claim, plus documents with a high tf-idf score against the claim, up to five total documents. following @xcite , we also considered the use of genre @xcite to identify more wikipedia page titles from entities that were not quite exact matches. (we preferred an exact match if present.) the use of these entities actually drove feverous score down, perhaps by crowding out the tf-idf documents, so we reverted to the baseline approach. given a set of documents, we rank page elements using models trained to predict the set of evidence elements. one model is trained on sentences, list elements, and table captions, and the other is trained on table cells. we use a roberta base model @xcite and follow a training approach similar to the dense passage retriever @xcite . given a positive training pair consisting of a claim c and a piece of evidence e, we collect six negative pairs (c, x i ). for four of the negatives we take x i to be the highest tf-idf matches returned by the baseline system that are not part of the gold evidence. for the other two negatives we take x i to be part of the gold evidence for a different claim, randomly chosen. the multiple choice classification head of roberta outputs a scalar f (c, x) for each pair, and the batch of seven pairs is trained as one example with the cross-entropy loss (top 25 cells and five non-cell page elements) to these modules. this is computed by taking the union of all page elements (cells or non-cells) in all evidence chains in all claims, and considering the fraction that belong to one of our predicted evidence sets for the corresponding claims. we recall more relevant table cells, but surprisingly, fewer relevant sentences. in development, we mistakenly benchmarked the ranking models on a set which made gold evidence available for ranking even if it was not in a retrieved document, and on this basis, it appeared that ranking the sentences was advantageous. therefore we used not only the table cell ranking module but also the sentence ranking module in our submitted system. ## next hop prediction the use of the evidence ranking model is not sufficient to solve problems that require more difficult kinds of multi-hop reasoning. though evidence chains are typically rooted in entities and concepts that appear in the claim, as one progresses down the chain it may be necessary to retrieve information about an entity mentioned in a previous piece of evidence. such information would be difficult to query based on the claim alone. to support this scenario, we introduce a next hop prediction module, as shown in figure 1 . hop 1 consists of the evidence retrieved by the evidence ranking module. given an evidence set produced in hop n, the next hop prediction module attempts to imagine information that is still needed but not retrieved yet. it generates a string consisting of the title of the needed article and the sentence or table cell (in the same format as before) that it wants to retrieve from that article. if available, the article with that title is retrieved; otherwise, sentences from previously retrieved articles will be searched. then we choose one sentence and two table cells with the best word overlap against the imagined evidence. the bottom ranked elements of the evidence set for hop n are pushed out, and these chosen elements are pushed to the top of the evidence set for hop n + 1. the evidence ranking module was found not to be helpful in ranking newly retrieved evidence, often because it strayed too far from the original claim. 1 the next hop prediction module is implemented by a t5 base sized model @xcite . t5 consists of a text-to-text encoder-decoder transformer architecture, and its pre-training mixes mul-tiple unsupervised objectives on the colossal clean crawled corpus with supervised nlu tasks including abstractive summarization, question answering, glue text classification, and translation, cast into a text to text format. we train the model for three epochs on maximum sequence length 512, using huggingface default parameters @xcite . in our task, each input begins with the task identifier "missing: " and a list of the pages retrieved already, followed by the string [hyp] and then the claim being classified. then the elements of the current evidence set (each beginning with a page title in brackets) are concatenated. training is based on the gold evidence chains in the training set, and the set of documents retrieved by the baseline model. every example with evidence from a missing document is used as an example, with the current evidence set being the gold evidence in the retrieved documents and the target evidence being the first piece of evidence from a missing document. for half of the remaining examples (those with no missing documents) including all nei examples with multiple pieces of evidence, a piece of evidence is randomly left out from the current evidence set, and that evidence is to be predicted as the target. in the other examples, the word "none" is to be predicted, indicating that the evidence chain is complete. the target output strings are the word "supports" or "refutes," followed by the target evidence in the usual format or "none." for nei examples, "supports" is to be predicted, indicating a partial evidence chain with no contradictions yet. thus the log likelihood objective on the target output string amounts to a multi-task objective, combining a prediction of missing evidence with a prediction of the label based on partial information. because missing evidence should be helpful for label prediction, we hope that co-training on the task of label prediction improves the features used to generate the missing evidence. the existence of distracting evidence distinguishes the training setting from the testing setting. at test time, the module is always queried with a full set of five sentences and 25 cells, some of which may be irrelevant. for comparison, we trained a model with extracted evidence instead of gold evidence, but the model trained on gold chains achieved more complete chains in fewer hops. an example of next hop prediction is given in the appendix. ## label classification after the next hop predictor has been run for seven hops, our system uses a label classification module to predict the final label. another t5 base model is used for this problem, but here we train on the extracted evidence sets (including irrelevant evidence, and missing some gold evidence) that are collected for the training set. input strings are the same as for the next hop predictor module. the target strings are just "supports," "refutes," or "neutral." as nei instances only make up 3% of the training set, this label is never learned and the outputs are either "supports" or "refutes." 2 the label accuracy of this approach on the development set is compared to other approaches that are trained with gold evidence or a roberta model in