Original: Best of Both Worlds: A Hybrid Approach for Multi-Hop Explanation with Declarative Facts
Revised: Best of Both Worlds: A Hybrid Approach for Multi-Hop Explanation with Declarative Facts

Original: Abstract
Revised: Abstract

Original: Language-enabled AI systems can answer complex, multi- hop questions to high accuracy, but supporting answers with evidence is a more challenging task which is important for the transparency and trustworthiness to users.
Revised: Language-enabled AI systems can answer complex, multi- hop questions to high accuracy, but supporting answers with evidence is a more challenging task which is important for the transparency and trustworthiness to users.

Original: Prior work in this area typically makes a trade-off between efﬁciency and ac- curacy; state-of-the-art deep neural network systems are too cumbersome to be useful in large-scale applications, while the fastest systems lack reliability.
Revised: Prior work in this area typically makes a trade-off between efﬁciency and ac- curacy; state-of-the-art deep neural network systems are too cumbersome to be useful in large-scale applications, while the fastest systems lack reliability.

Original: In this work, we integrate fast syntactic methods with powerful semantic meth- ods for multi-hop explanation generation based on declarative facts.
Revised: In this work, we integrate fast syntactic methods with powerful semantic meth- ods for multi-hop explanation generation based on declarative facts.

Original: Our best system, which learns a lightweight opera- tion to simulate multi-hop reasoning over pieces of evidence and ﬁne-tunes language models to re-rank generated expla- nation chains, outperforms a purely syntactic baseline from prior work by up to 7% in gold explanation retrieval rate.
Revised: Our best system, which learns a lightweight opera- tion to simulate multi-hop reasoning over pieces of evidence and ﬁne-tunes language models to re-rank generated expla- nation chains, outperforms a purely syntactic baseline from prior work by up to 7% in gold explanation retrieval rate.

Original: Introduction
Revised: Introduction

Original: Efﬁcient and natural human communication relies on implicit shared knowledge and underlying reasoning processes.
Revised: Efﬁcient and natural human communication relies on implicit shared knowledge and underlying reasoning processes.

Original: Despite rapid progress in language-enabled AI agents for tasks like question answering and more, state-of-the-art systems still struggle to explain their decisions in natural language.
Revised: Despite rapid progress in language-enabled AI agents for tasks like question answering and more, state-of-the-art systems still struggle to explain their decisions in natural language.

Original: To improve their interpretability and robustness, a number of multi-hop explanation generation and identiﬁcation benchmarks based on large, unstructured corpora of facts have been created (Mihaylov et al. 2018; Khot et al.
Revised: To improve their interpretability and robustness, a number of multi-hop explanation generation and identiﬁcation benchmarks based on large, unstructured corpora of facts have been created (Mihaylov et al. 2018; Khot et al.

Original: 2020; Jhamtani and Clark 2020).
Revised: 2020; Jhamtani and Clark 2020).

Original: However, when generating explanation chains, powerful deep neural networks can be too cumbersome to be useful in large-scale applications, while the fastest systems lack reliability, as they rely on syntactic features and ignore semantic relations between concepts (Banerjee and Baral 2020; Jhamtani and Clark 2020).
Revised: However, when generating explanation chains, powerful deep neural networks can be too cumbersome to use in large-scale applications, while the fastest systems lack reliability, as they depend on syntactic features and ignore semantic relations between concepts (Banerjee and Baral 2020; Jhamtani and Clark 2020).

Original: In this work, we present novel approaches to integrate efﬁcient syntactic retrieval methods with ﬂexible semantic modeling methods for multi-hop explanation.
Revised: In this work, we present novel approaches to integrate efﬁcient syntactic retrieval methods with ﬂexible semantic modeling methods for multi-hop explanation.

Original: Our methods simulate a multi-hop reasoning process from the retrieval and synthesis of evidence to re-ranking candidate explanations
Revised: Our methods simulate a multi-hop reasoning process from the retrieval and synthesis of evidence to re-ranking candidate explanations.

Original: . Related Work
Revised: Related Work

Original: Recent work has focused on different aspects of multi-hop reasoning for question answering and related natural lan- guage understanding tasks.
Revised: Recent work has focused on different aspects of multi-hop reasoning for question answering and related natural language understanding tasks.

Original: One line of work has incorporated highly structured knowledge graphs into language understanding by combining graphical methods with language models (Lin et al.
Revised: One line of work has incorporated highly structured knowledge graphs into language understanding by combining graphical methods with language models (Lin et al.

Original: Ji et al.
Revised: Ji et al.

Original: 2020; Yasunaga et al. 2021)
Revised: 2020; Yasunaga et al. 2021)

Original: , augmenting language model inputs with relational knowledge (Zhang et al. 2019;
Revised: , augmenting language model inputs with relational knowledge (Zhang et al. 2019;

Original: Chen et al. 2020;
Revised: Chen et al. 2020;

Original: Xu et al. 2021), and applying language models to relational knowledge to infer multi-hop reasoning paths through knowledge graphs (Wang et al. 2020).
Revised: Xu et al. 2021), and applying language models to relational knowledge to infer multi-hop reasoning paths through knowledge graphs (Wang et al. 2020).

Original: Others have further explored training language models with semi-structured relational knowledge (Sap et al. 2019; Bosselut et al.
Revised: Others have further explored training language models with semi-structured relational knowledge (Sap et al. 2019; Bosselut et al.

Original: ; Mostafazadeh et al.
Revised: ; Mostafazadeh et al.

Original: 2020; Hwang et al.
Revised: 2020; Hwang et al.

Original: , i.e., where nodes are natural language sentences rather than canonicalized concepts, to later use for generating multi-hop explanations in natural language (Shwartz et al. 2020; Bosselut et al. 2021).
Revised: , i.e., where nodes are natural language sentences rather than canonicalized concepts, to later use for generating multi-hop explanations in natural language (Shwartz et al. 2020; Bosselut et al. 2021).

Original: For generating multi-hop explanations from entirely unstructured corpora, other work has explored using multi-step syntactic information retrieval methods (Jhamtani and Clark 2020), and modeling such corpora as knowledge graphs with relations induced by shared mentions of concepts between documents (Dhingra et al. 2020;
Revised: For generating multi-hop explanations from entirely unstructured corpora, other work has explored using multi-step syntactic information retrieval methods (Jhamtani and Clark 2020), and modeling such corpora as knowledge graphs with relations induced by shared mentions of concepts between documents (Dhingra et al. 2020;

Original: Lin et al. 2021).
Revised: Lin et al. 2021).

Original: While the former approach lacks the ability to capture semantic relationships between evidence sentences, the latter demands high time and space complexity both in generating a graph from corpora of millions of facts, and in everyday uses of adding or removing facts from the corpus.
Revised: While the former approach lacks the ability to capture semantic relationships between evidence sentences, the latter demands high time and space complexity both in generating a graph from corpora of millions of facts, and in everyday uses of adding or removing facts from the corpus.

Original: More recent work has used pre-trained word embeddings to add some lightweight semantic representation to syntactic evidence retrieval (Yadav, Bethard, and Surdeanu 2021).
Revised: More recent work has used pre-trained word embeddings to add some lightweight semantic representation to syntactic evidence retrieval (Yadav, Bethard, and Surdeanu 2021).

Original: Unlike these approaches, we present a ﬂexible and relatively lightweight pipeline to apply both syntactic and learned, contextualized semantic approaches in multi-hop explanation generation, including evidence retrieval, multi-hop reasoning over evidence, and re-ranking candidate explanations.
Revised: Unlike these approaches, we present a ﬂexible and relatively lightweight pipeline to apply both syntactic and learned, contextualized semantic approaches in multi-hop explanation generation, including evidence retrieval, multi-hop reasoning over evidence, and re-ranking candidate explanations.

Original: Problem Statement
Revised: Problem Statement

Original: In the research community, two types of explanation have been studied: introspective explanation and justiﬁcation explanation (Biran and Cotton 2017).
Revised: In the research community, two types of explanation have been studied: introspective explanation and justiﬁcation explanation (Biran and Cotton 2017).

Original: The former explicates how a decision is made, and the latter gathers evidence to support a decision.
Revised: The former explicates how a decision is made, and the latter gathers evidence to support a decision.

Original: In this study, we focus on the task of justiﬁcation explanation.
Revised: In this study, we focus on the task of justiﬁcation explanation.

Original: Speciﬁcally, we explore the problem of generating multi-hop explanations to support the answer to a natural language question, where the explanation chain is generated from an unstructured corpus of declarative facts.
Revised: Speciﬁcally, we explore the problem of generating multi-hop explanations to support the answer to a natural language question, where the explanation chain is generated from an unstructured corpus of declarative facts.

Original: Unstructured natural language corpora are suitable knowledge resources for human-AI interaction, as humans can easily support reasoning by providing their own commonsense knowledge in short, natural language statements.
Revised: Unstructured natural language corpora are suitable knowledge resources for human-AI interaction, as humans can easily support reasoning by providing their own commonsense knowledge in short, natural language statements.

Original: This carefully restricted problem of explanation generation consists of two key challenges.
Revised: This carefully restricted problem of explanation generation consists of two key challenges.

Original: First, we must solve the retrieval task to gather candidate supporting evidence from the corpus.
Revised: First, we must solve the retrieval task to gather candidate supporting evidence from the corpus.

Original: Second, we need to invoke a multi-hop reasoning process to connect pieces of evidence to form the most valid explanation to justify the answer to the question.
Revised: Second, we need to invoke a multi-hop reasoning process to connect pieces of evidence to form the most valid explanation to justify the answer to the question.

Original: Datasets
Revised: Datasets

Original: To explore this problem, we consider two datasets.
Revised: To explore this problem, we consider two datasets.

Original: First, the Question Answering via Sentence Composition (QASC) dataset providesa about 10,000 multiple-choice science questions (Khot et al. 2020).
Revised: First, the Question Answering via Sentence Composition (QASC) dataset provides about 10,000 multiple-choice science questions (Khot et al. 2020).

Original: QASC is a challenging problem, as each question requires composing two facts from a corpus of about 17 million declarative facts to connect the question and its correct answer.
Revised: QASC is a challenging problem, as each question requires composing two facts from a corpus of about 17 million declarative facts to connect the question and its correct answer.

Original: For example, given the question “ Differential heating of air can be harnessed for what?”
Revised: For example, given the question “ Differential heating of air can be harnessed for what?”

Original: and correct answer “ electricity production ,” the answer can be explained by composing the facts “ Differential heating of air produces wind” and “Wind is used for producing electricity ,” which connect the question and answer.
Revised: and correct answer “ electricity production ,” the answer can be explained by composing the facts “ Differential heating of air produces wind” and “Wind is used for producing electricity ,” which connect the question and answer.

Original: QASC includes a gold, human-curated 2-hop explanation from the corpus for each question-answer pair.
Revised: QASC includes a gold, human-curated 2-hop explanation from the corpus for each question-answer pair.

Original: Meanwhile, the Explainable QASC (eQASC) dataset adds 10 automatically generated explanations for each question-answer pair, each of which are labeled by annotators as valid or invalid (Jhamtani and Clark 2020).
Revised: Meanwhile, the Explainable QASC (eQASC) dataset adds 10 automatically generated explanations for each question-answer pair, each of which are labeled by annotators as valid or invalid (Jhamtani and Clark 2020).

Original: While the state-of-the-art accuracy on QASC has reached up to 90%,only 76% of questions have any valid explanation chains in eQASC.
Revised: While the state-of-the-art accuracy on QASC has reached up to 90%,only 76% of questions have any valid explanation chains in eQASC.

Original: This indicates that explaining the answers to questions in QASC is a more challenging problem than answering them.
Revised: This indicates that explaining the answers to questions in QASC is a more challenging problem than answering them.

Original: This motivates us to further explore the problem of generating multi-hop explanations for QASC.
Revised: This motivates us to further explore the problem of generating multi-hop explanations for QASC.

Original: Methods
Revised: Methods

Original: In our experiments toward multi-hop explanation generation, we consider syntactic and semantic multi-hop retrieval methods, then explore ways to re-rank retrieved explanations to reduce the pool of candidates.
Revised: In our experiments toward multi-hop explanation generation, we consider syntactic and semantic multi-hop retrieval methods, then explore ways to re-rank retrieved explanations to reduce the pool of candidates.

Original: Syntactic Methods
Revised: Syntactic Methods

Original: Syntactic information retrieval methods enable quick searching of millions of documents.
Revised: Syntactic information retrieval methods enable quick searching of millions of documents.

Original: eQASC was originally generated using ElasticSearch 2, a fast but primarily syntactic search engine based on keyword overlap.
Revised: eQASC was originally generated using ElasticSearch, 2 a fast but primarily syntactic search engine based on keyword overlap.

Original: After indexing the QASC corpus facts into an ElasticSearch index, Jhamtani and Clark (2020) used a simple procedure (shown in
Revised: After indexing the QASC corpus facts into an ElasticSearch index, Jhamtani and Clark (2020) used a simple procedure (shown in

Original: Figure 1) to generate a 2-hop explanation for each questionanswer pair from QASC.
Revised: Figure 1) to generate a 2-hop explanation for each questionanswer pair from QASC.

Original: First, query the corpus for N =candidate
Revised: First, query the corpus for N =

Original: ﬁrst facts.
Revised: [Figure]

Original: For each candidate ﬁrst fact, query the corpus for M = 4 candidate second facts, where each candidate second fact must contain a word that appears in the
Revised: [Figure] [Figure] ×

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  question-answer pair, and a word that appears in the ﬁrst fact.
Revised: candidate ﬁrst facts. For each candidate ﬁrst fact, query the corpus for M = 4 candidate second facts, where each candidate second fact must contain a word that appears in the question-answer pair, and a word that appears in the ﬁrst fact.

Original: The purpose of this restriction is to force the resulting chain of facts to connect concepts in the question and answer through intermediate concepts.
Revised: The purpose of this restriction is to force the resulting chain of facts to connect concepts in the question and answer through intermediate concepts.

Original: Lastly, from the set product of all candidate ﬁrst facts and all candidate second facts, select up to K = 10 candidate explanation chains, ranked by the sum of retrieval scores from the ElasticSearch engine.
Revised: Lastly, from the set product of all candidate ﬁrst facts and all candidate second facts, select up to K = 10 candidate explanation chains, ranked by the sum of retrieval scores from the ElasticSearch engine.

Original: Expanding syntactic retrieval.
Revised: Expanding syntactic retrieval.

Original: This is a simple, fast approach to generate a large number of candidate explanation chains.
Revised: This is a simple, fast approach to generate a large number of candidate explanations.

Original: To improve the likelihood that a valid explanation is generated, we can expand and diversify the search results by increasing N , M , and K .
Revised: To improve the likelihood of generating a valid explanation, we can expand and diversify the search results by increasing N , M , and K .

Original: Speciﬁcally, we increaseN , M , and K each  to 200.
Revised: Speciﬁcally, we increase each of them to 200.

Original: Semantic Methods
Revised: Semantic Methods

Original: Alternatively, semantic information retrieval methods can enable stronger meaning representation than syntactic methods with a trade-off of search speed.
Revised: Alternatively, semantic information retrieval methods can enable stronger meaning representation than syntactic methods with a trade-off of search speed.

Original: Typical approaches generate a semantic vector embedding for all documents in a corpus.
Revised: Typical approaches generate a semantic vector embedding for all documents in a corpus.

Original: They then generate a comparable embedding of the query, and usesimple vector similarity measures, e.g., inner product, to rank documents.
Revised: They then generate a comparable embedding of the query, and use vector similarity measures to rank documents.

Original: Dense passage retrieval.
Revised: Dense passage retrieval.

Original: Dense passage retrieval (DPR) is a recent approach to semantic information retrieval which learns dual encoders for queries and documents (Karpukhin et al. 2020).
Revised: Dense passage retrieval (DPR) is a recent approach to semantic information retrieval which learns dual encoders for queries and documents (Karpukhin et al. 2020).

Original: They are trained such that the query and document encoders generate similar embeddings for semantically similar queries and documents.
Revised: They are trained such that the query and document encoders generate similar embeddings for semantically similar queries and documents.

Original: Similarity is measured by inner product of vectors, and is maximized for matching queries and documents, but minimized for irrelevant queries and documents.
Revised: Similarity is measured by inner product of vectors, and is maximized for matching queries and documents, but minimized for irrelevant queries and documents.

Original: We can then use the document encoder to index the facts in a corpus, and efﬁciently query the index using the embedding from the query encoder.
Revised: We can then use the document encoder to index the facts in a corpus, and efﬁciently query the index using the embedding from the query encoder.

Original: For each question-answer pair in QASC and the two facts in its gold explanation chain, we can train a dense passage retriever to generate similar embeddings for the questionanswer pair (query) and these facts (documents), then encode all facts in the QASC corpus for search purposes.
Revised: For each question-answer pair in QASC and the two facts in its gold explanation chain, we can train a dense passage retriever to generate similar embeddings for the questionanswer pair (query) and these facts (documents), then encode all facts in the QASC corpus for search purposes.

Original: Multi-hop reasoning.
Revised: Multi-hop reasoning.

Original: To generate a multi-hop explanation using this approach, we need to facilitate reasoning over the facts and queries.
Revised: To generate a multi-hop explanation using this approach, we need to facilitate reasoning over the facts and queries.

Original: Given a question-answer pair from
Revised: Given a question-answer pair from

Original: QASC, we ﬁrst query the DPR index for N = 5 facts.
Revised: QASC, we ﬁrst query the DPR index for N = 5 facts.

Original: Toreduce error accumulation in generating the chain of facts, [Figure] [Figure]
Revised: To

Original: [Figure]
Revised: [Figure]

Original: f

Original: [Figure]
Revised: [Figure]

Original: f N … ff N … f 1M f NM … … … … 
Revised: Re-Encoder ff N … ff N … f 1M f NM … … … … reduce error accumulation in we then re-encode each fact into a new query embedding incorporating the candidate fact and the original query.

Original: The re-encoder is a lightweight feedforward network inspired by the fact-translating function proposed in Lin et al.
Revised: The re-encoder is a lightweight feedforward network inspired by a similar fact-translating function proposed in Lin et al.

Original: Given the embeddings q QA and d 1 for the questionanswer pair query and ﬁrst fact document respectively, we use the gold explanation chains from QASC to learn the re-encoder g ( q QA , d 1 ) .
Revised: Given the embeddings q QA and d 1 for the question-answer pair query and ﬁrst fact document respectively, we use the gold explanation chains from QASC to learn the re-encoder g ( q QA , d 1 ) .

Original: Speciﬁcally, if d 1 is the embedding of the ﬁrst fact in the gold explanation chain, we maximize the inner product between the re-encoded output q r and the document embedding d 2 for the second gold fact.
Revised: Speciﬁcally, if d 1 is the embedding of the ﬁrst fact in the gold explanation chain, we maximize the inner product between the re-encoded output q r and the document embedding d 2 for the second gold fact.

Original: Next, we query the DPR index again using q r to obtain M = 2 candidate second facts.
Revised: Next, we query the DPR index again using q r to obtain M = 2 candidate second facts.

Original: To reduce noise, we ﬁlter out any facts that mention no concepts in either the question or answer.
Revised: To reduce noise, we ﬁlter out any facts that mention no concepts in either the question or answer.

Original: Lastly, from the set product of all candidate ﬁrst facts and all candidate second facts, select up to K = 10 candidate explanation chains, ranked by the sum of retrieval scores, i.e., inner products when querying the DPR index.
Revised: Lastly, from the set product of all candidate ﬁrst facts and all candidate second facts, select up to K = 10 candidate explanation chains, ranked by the sum of retrieval scores, i.e., inner products when querying the DPR index.

Original: Our semantic explanation pipeline is shown in full in Figure 2.
Revised: Our semantic explanation pipeline is shown in full in Figure 2.

Original: It is worth noting that our lightweight re-encoder operation can extend to any number of hops.
Revised: It is worth noting that our lightweight re-encoder operation can extend to any number of hops.

Original: Re-Ranking Candidate Explanations
Revised: Re-Ranking Candidate Explanations

Original: Both of our syntactic and semantic multi-hop retrieval systems can quickly propose candidate explanation chains for questions in QASC.
Revised: Both of our syntactic and semantic multi-hop retrieval systems can quickly propose candidate explanation chains for questions in QASC.

Original: However, both approaches overgenerate candidates, and the high number of candidates (i.e., up to 200) limits the practical usefulness of the systems to an end user.
Revised: However, both approaches overgenerate candidates, and the high number of candidates (i.e., up to 200) limits the practical usefulness of the systems to an end user.

Original: As such, we lastly propose a re-ranker for candidates based on large-scale, pre-trained language models (Devlin et al. 2019;
Revised: As such, we lastly propose a re-ranker for candidates based on large-scale, pre-trained language models (Devlin et al. 2019;

Original: Liu et al. 2019).
Revised: Liu et al. 2019).

Original: Speciﬁcally, we use the gold explanation chains from QASC to ﬁne-tune a language model to the classiﬁcation task of whether or not a candidate explanation is valid for a question-answer pair.
Revised: Speciﬁcally, we use the gold explanation chains from QASC to ﬁne-tune a language model to the classiﬁcation task of whether or not a candidate explanation is valid for a question-answer pair.

Original: We then re-rank a pool of candidates based on the system’s estimated likelihood that each explanation chain is valid, and keep only the top K = 10 candidates for a direct comparison to the syntactic approach used to generate eQASC.
Revised: We then re-rank a pool of candidates based on the system’s estimated likelihood that each explanation chain is valid, and keep only the top K = 10 candidates for a direct comparison to the syntactic approach used to generate eQASC.

Original: Experimental Results
Revised: Experimental Results

Original: We next apply these approaches to the task of selecting 2hop reasoning chains for question-answer pairs in QASC, and directly compare our results to the original procedure to generate eQASC.
Revised: We next apply these approaches to the task of selecting 2hop reasoning chains for question-answer pairs in QASC, and directly compare our results to the original procedure to [Table]

Original: We compare systems by their individual
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  gold retrieval rate on the validation set for QASC, i.e., the percentage of question-answer pairs for which the gold explanation chain from QASC was successfully reproduced.
Revised: generate eQASC. We compare systems by their individual gold retrieval rate on the validation set for QASC, i.e., the percentage of question-answer pairs for which the gold explanation chain from QASC was successfully reproduced.

Original: This serves as an indicator of the quality of generated explanations, as it suggests that generated explanations tend to look more like those curated from the corpus by humans.
Revised: This serves as an indicator of the quality of generated explanations, as it suggests that generated explanations tend to look more like those curated from the corpus by humans.

Original: Expanded Syntactic Explanation
Revised: Expanded Syntactic Explanation

Original: As mentioned earlier, we ﬁrst expanded the ElasticSearchbased approach used to generate eQASC by increasing the search hyperparameters N , M , and K each to 200.
Revised: As mentioned earlier, we ﬁrst expanded the ElasticSearchbased approach used to generate eQASC by increasing the search hyperparameters N , M , and K each to 200.

Original: Selected results from adjusting these hyperparameters are listed in Table 1.
Revised: Selected results from this are listed in Table 1.

Original: By only increasing K (i.e., the number of candidate explanation chains considered) to 200, the retrieval rate increases from 31.1% to 37.0%.
Revised: By only increasing K (i.e., the number of candidate explanation chains considered) to 200, the retrieval rate increases from 31.1% to 37.0%.

Original: When increasing N and M (i.e., the number of candidate ﬁrst and second facts considered) also to 200, the retrieval rate further increases to 46.5%, a net 15.4% gain.
Revised: When increasing N and M (i.e., the number of candidate ﬁrst and second facts considered) also to 200, the retrieval rate further increases to 46.5%, a net 15.4% gain.

Original: Syntactic-Semantic Multi-Hop Explanation
Revised: Syntactic-Semantic Multi-Hop Explanation

Original: Next, we incorporate our semantic multi-hop retrieval process powered by DPR.
Revised: Next, we incorporate our semantic multi-hop retrieval process powered by DPR.

Original: Training details.
Revised: Training details.

Original: The dual encoders for DPR are learned starting from pre-trained BERT-base (Liu et al. 2019).
Revised: The dual encoders for DPR are learned starting from pre-trained BERT-base (Devlin et al. 2019).

Original: The best encoders are selected based on the mean squared error between embeddings for matching question-answer pairs and facts on the QASC validation set.
Revised: The best encoders are selected based on the mean squared error between embeddings for matching question-answer pairs and facts on the QASC validation set.

Original: The batch size is ﬁxed at 16, while learning rate and number of training epochs are
Revised: The batch size is ﬁxed at 16, while learning rate and number of training epochs are

Original: [Table]
Revised: [Table]

Original: selected based on a grid search.
Revised: selected based on a grid search.

Original: For the re-encoder, the training batch size, learning rate, and number of epochs are similarly selected based on a grid search, minimizing the mean squared error between the output re-encoded queries and target fact embeddings on the validation set.
Revised: For the re-encoder, the training batch size, learning rate, and number of epochs are similarly selected based on a grid search, minimizing the mean squared error between the output re-encoded queries and target fact embeddings on the validation set.

Original: Results.
Revised: Results.

Original: Table 2 compares the gold retrieval rate of various combinations of the syntactic and semantic approaches for multi-hop retrieval on the QASC validation and testing sets.
Revised: Table 2 compares the gold retrieval rate of various combinations of the syntactic and semantic approaches for multi-hop retrieval on the QASC validation and testing sets.

Original: 4 Our results show that while using only the semantic candidate explanation chains leads to a 13.9% gold retrieval rate at best, combining the expanded syntactic and semantic candidates gives us the best result of up to 51.1% gold retrieval rate, outperforming the case where only the expanded syntactic candidates are considered.
Revised: 4 Our results show that while using only the semantic candidate explanation chains leads to a 13.9% gold retrieval rate at best, combining the expanded syntactic and semantic candidates gives us the best result of up to 51.1% gold retrieval rate, outperforming the case where only the expanded syntactic candidates are considered.

Original: Thus, the semantic approach ﬁnds some of the missing gold explanations that the syntactic approach misses, suggesting that both syntactic and semantic approaches are needed for generating the best-quality explanations on QASC questions.
Revised: Thus, the semantic approach ﬁnds some of the missing gold explanations that the syntactic approach misses, suggesting that both syntactic and semantic approaches are needed for generating the best-quality explanations on QASC questions.

Original: LM Re-Ranking
Revised: LM Re-Ranking

Original: While our results improve the gold retrieval rate by a wide margin, recall that our multi-hop retrieval approaches for
Revised: While our results improve the gold retrieval rate by a wide margin, recall that our multi-hop retrieval approaches for

Original: QASC increase the number of candidate explanation chains
Revised: QASC increase the number of candidate explanation chains

Original: K to 200.
Revised: K to 200.

Original: Such a large set of candidates is not useful in practice, as a human user would have to sort through a cumbersome number of explanations in order to judge the machine’s understanding of the question and answer.
Revised: Such a large set of candidates is not useful in practice, as a human user would have to sort through a cumbersome number of explanations in order to judge the machine’s understanding of the question and answer.

Original: As such, we lastly present our experiments on re-ranking candidate explanation chains, which enables us to truncate our results to K = 10 top candidate explanation chains without massive performance drops, and consequently compare our approach directly to the original approach used to generate eQASC.
Revised: As such, we lastly present our experiments on re-ranking candidate explanation chains, which enables us to truncate our results to K = 10 top candidate explanation chains without massive performance drops, and consequently compare our approach directly to the original approach used to generate eQASC.

Original: Training details.
Revised: Training details.

Original: Using the re-ranking approach described earlier, we ﬁne-tune the BERT (Devlin et al. 2019) and R O BERT A (Liu et al. 2019)
Revised: Using the re-ranking approach described earlier, we ﬁne-tune the BERT (Devlin et al. 2019) and R O BERT A (Liu et al. 2019)

Original: pre-trained language models.
Revised: pre-trained language models.

Original: 5 Models are trained with a 1:2 ratio of gold and invalid explanation chains, with 3 unique invalid explanation chains
Revised: 5 Models are trained with a 1:2 ratio of gold and invalid explanation chains, with 3 unique invalid explanation chains

Original: Retrieval
Revised: Retrieval

Original: [Table]
Revised: [Table]

Original: randomly sampled from ElasticSearch results per gold explanation chain (forward and forms)
Revised: randomly sampled from ElasticSearch results per gold explanation chain (forward and forms)

Original: Models are se lected based on instances achieving the highest top-1 gold retrieval rate, i.e., proportion of question-answer pairs where the gold explanation chain is ranked highest, on the QASC validation set similarly redistributed in this way.
Revised: Models are se lected based on instances achieving the highest top-1 gold retrieval rate, i.e., proportion of question-answer pairs where the gold explanation chain is ranked highest, on the QASC validation set similarly redistributed in this way.

Original: Results discussion.
Revised: Results discussion.

Original: Table 3 compares the ﬁnal gold retrieval rates for the top K = 10 re-ranked candidates from various approaches.
Revised: Table 3 compares the ﬁnal gold retrieval rates for the top K = 10 re-ranked candidates from various approaches.

Original: While the original syntactic approach for generating eQASC achieves a respective 31.1% and 34.1% gold retrieval rate on the validation and testing sets, our expanded syntactic approach achieves up to 37.9% and 36.2% gold retrieval rate with R O BERT A .
Revised: While the original syntactic approach for generating eQASC achieves a respective 31.1% and 34.1% gold retrieval rate on the validation and testing sets, our expanded syntactic approach achieves up to 37.9% and 36.2% gold retrieval rate with R O BERT A .

Original: Again with R O BERT A , our syntactic-semantic multi-hop retrieval achieves the best results of 38.1% and 36.4% on the validation and testing sets, respectively, exceeding the baselines.
Revised: Again with R O BERT A , our syntactic-semantic multi-hop retrieval achieves the best results of 38.1% and 36.4% on the validation and testing sets, respectively, exceeding the baselines.

Original: While our best approach achieves up to anet 7.0% gain over the baseline syntactic approach used to generate eQASC, much of our performance gain was lost after re-ranking.
Revised: After narrowing down from 200 candidate explanations to 10 with the re-ranker, we retain up to a 7.0% net improvement of gold retrieval rate compared with the baseline.

Original: This is possibly because the  negative examples proposed by DPR were not incorporated in re-ranker training, which may cause generalization issues.Future work may further explore the re-ranking problem to better preserve the retrieval rates from the larger candidate pools explored earlier.
Revised: Given that the net gain was 13.9% with 200 candidate explanations, one future direction is to improve the re-ranker performance, so that we can retain more of this improvement. To achieve this, one option is to revisit the re-ranker training, which did not incorporate negative examples proposed by DPR, and may experience generalization issues.

Original: Conclusion
Revised: Conclusion

Original: In this work, by utilizing a small amount of ground truth supervision, we exploredseveral approaches to improve the generation of multi-hop explanations from a corpus of declarative facts.
Revised: In this work, by utilizing a small amount of ground truth supervision, we explored approaches to improve the generation of multi-hop explanations from a corpus of declarative facts.

Original: Our results show that both fast, syntactic methods and slow, semantic methods are useful for gathering relevant evidence for explanation.
Revised: We show that both fast, syntactic methods and slow, semantic methods are useful for gathering relevant evidence for explanation.

Original: To facilitate multi-hop reasoning from one piece of evidence to the next, we had some success in using a lightweight feedforward re-encoder, as opposed to state-of-the-art graph-based approaches that consume too much time and memory for practical online use.
Revised: To facilitate multi-hop reasoning from one piece of evidence to the next, we had some success in using a lightweight feedforward re-encoder, as opposed to state-ofthe-art graph-based approaches that consume too much time and memory for practical online use.

Original: As many of our results over-generate candidate multihop explanation chains, we lastly explored using pre-trained language models to re-rank and ﬁlter candidates to only the best ones.
Revised: As many of our approaches over-generate candidate explanations, we lastly explored using pre-trained language models to re-rank and ﬁlter candidates.

Original: However, our results show that this is a signiﬁcant challenge, and future work may further explore this problem.
Revised: Our results suggest this is a signiﬁcant challenge, and future work may further explore this problem.