Original: M ULTI - STEP R EASONING FOR  O PEN - DOMAIN Q UESTION A NSWERING
Revised: M ULTI - STEP R ETRIEVER -R EADER I NTERACTION FOR S CALABLE O PEN - DOMAIN Q UESTION A NSWERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper introduces a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other.
Revised: This paper introduces a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other.

Original: The framework is agnostic to the architecture of the machine reading model and the  retriever uses fast nearest neighbor search algorithms that allow it to scale to corpora containing millions of paragraphs.
Revised: The framework is agnostic to the architecture of the machine reading model provided it has access to the token-level hidden representations of the reader. The retriever uses fast nearest neighbor search algorithms that allow it to scale to corpora containing millions of paragraphs.

Revised: A gated recurrent unit updates the query at each step conditioned on the “state” of the reader and the “reformulated” query is used to re-rank the paragraphs by the retriever.

Original: We show the efﬁcacy of our architecture by achieving state-of-the-art results  on  large open domain datasets such asTriviaQA-unﬁltered (Joshi et al., 2017).
Revised: We show the efﬁcacy of our architecture by achieving state-of-the-art results (9.5% relative increase) on TriviaQA-unﬁltered and we achieve competitive performance on other large open domain datasets such as

Revised: Q UASAR -T, S EARCH QA, and SQ U AD-open.

Original: We also show that our multi-step-reasoning framework brings uniform improvements when applied to two different reader architectures.
Revised: We conduct analysis and show that iterative interaction helps in retrieving useful paragraphs from the corpus. Finally, we show that our multi-step-reasoning framework brings uniform improvements when applied to two widely used reader architectures – Dr.QA and BiDAF 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Open-domain question answering (QA) (Voorhees et al., 1999) involves a retriever for selecting relevant context from a large corpora of text (e.g. Wikipedia) and a reader for ‘reasoning’ on the retrieved context.
Revised: Open-domain question answering (QA) (Voorhees et al., 1999) involves a retriever for selecting relevant context from a large corpora of text (e g. Wikipedia) and a reader for ‘reasoning’ on the retrieved context.

Original: A lot of effort has been put into designing sophisticated neural machine reading architectures for reading short context (e.g. a single paragraph), with much success (Wang & Jiang, 2017; Seo et al., 2017; Xiong et al., 2017; Wang et al., 2018c; Yu et al., 2018, inter alia).
Revised: A lot of effort has been put into designing sophisticated neural machine reading architectures for reading short context (e g. a single paragraph), with much success (Wang & Jiang, 2017; Seo et al., 2017; Xiong et al., 2017; Wang et al., 2018c; Yu et al., 2018, inter alia).

Original: However, the performance of such systems degrade signiﬁcantly when combined with a retriever in open domain settings (Chen et al., 2017).
Revised: However, the performance of such systems degrade signiﬁcantly when combined with a retriever in open domain settings (Chen et al., 2017).

Original: For example, the exact match accuracy of DrQA (Chen et al., 2017), on the SQ U AD dataset (Rajpurkar et al., 2016) degrades from 69.5% to 28.4% in open-domain settings.
Revised: For example, the exact match accuracy of DrQA (Chen et al., 2017), on the SQ U AD dataset (Rajpurkar et al., 2016) degrades from 69.5% to 28.4% in open-domain settings.

Original: The primary reason for this degradation in performance is due to the retriever’s failure to ﬁnd the relevant paragraphs for the machine reading model (Htut et al., 2018).
Revised: The primary reason for this degradation in performance is due to the retriever’s failure to ﬁnd the relevant paragraphs for the machine reading model (Htut et al., 2018).

Original: We propose the following desiderata for a general purpose open-domain QA system - (a) The retriever model should be fast , since it has to ﬁnd the relevant context from a very large text corpora and give it to the more sophisticated and computationally expensive machine reading model (b)
Revised: We propose the following desiderata for a general purpose open-domain QA system - (a) The retriever model should be fast , since it has to ﬁnd the relevant context from a very large text corpora and give it to the more sophisticated and computationally expensive machine reading model (b)

Original: Secondly, the retriever and reader models should be interactive, i.e. if the reader model is unable to ﬁnd the answer from the initial retrieved context, the retriever should be able to learn to provide more relevant context to the reader.
Revised: Secondly, the retriever and reader models should be interactive, i.e. if the reader model is unable to ﬁnd the answer from the initial retrieved context, the retriever should be able to learn to provide more relevant context to the reader.

Original: Open-domain QA systems such as R 3 (Wang et al., 2018a) and DS-QA (Lin et al., 2018) have sophisticated retriever models where the reader and retriever are jointly trained.
Revised: Open-domain QA systems such as R 3 (Wang et al., 2018a) and DS-QA (Lin et al., 2018) have sophisticated retriever models where the reader and retriever are jointly trained.

Original: However, their retriever computes question-dependent paragraph representation which is then encoded by running an expensive recurrent neural network over the tokens in the paragraph.
Revised: However, their retriever computes question-dependent paragraph representation which is then encoded by running an expensive recurrent neural network over the tokens in the paragraph.

Original: Since the retriever has to rank a lot of paragraphs, this design would not scale well to large corporas.
Revised: Since the retriever has to rank a lot of paragraphs, this design would not scale well to large corporas.

Original: One the other hand, the retriever model of QA systems such as DrQA (Chen et al., 2017), Clark & Gardner (2018) are based on a tf-idf retriever, but they lack trainable parameters and are consequently unable to recover from mistakes.
Revised: One the other hand, the retriever model of QA systems such as DrQA (Chen et al., 2017), Clark & Gardner (2018) are based on a tf-idf retriever, but they lack trainable parameters and are consequently unable to recover from mistakes.

Original: This paper introduces an open domain architecture in which the retriever and reader iteratively interact with each other.
Revised: This paper introduces an open domain architecture in which the retriever and reader iteratively interact with each other.

Original: Our model ﬁrst pre-computes and caches representation of context (paragraph).
Revised: Our model ﬁrst pre-computes and caches representation of context (paragraph).

Original: These representations are independent of the query unlike recent architectures (Seo et al., 2017;
Revised: These representations are independent of the query unlike recent architectures (Seo et al., 2017;

Original: Xiong et al., 2017; Wang et al., 2018a; Lin et al., 2018) and hence can be computed and stored ofﬂine. 
Revised: Xiong et al., 2017; Wang et al., 2018a; Lin et al., 2018) and hence can be computed and stored ofﬂine. [Figure] E m b e d q u e r y [Figure]

Original: Given an input question, the retriever performs fast inner product search to ﬁnd the most relevant contexts.
Revised: Given an input question, the retriever performs fast inner product search to ﬁnd the most relevant contexts.

Original: The highest ranked contexts are then passed to the neural machine reader which usually is a multi-layer recurrent neural network with attention.
Revised: The highest ranked contexts are then passed to the neural machine reader which usually is a multi-layer recurrent neural network with attention.

Original: Our architecture is agnostic to the choice of the reader architecture and we show thatdoing multi-step-reasoning increases performance oftwo state-of-the-art machine reading architectures - DrQA (Chen et al., 2017) and BiDAF (Seo et al., 2017).
Revised: Our architecture is agnostic to the choice of the reader architecture and we show that multi-step-reasoning increases performance of state-of-the-art machine reading architectures - DrQA (Chen et al., 2017) and BiDAF (Seo et al., 2017).

Original: It is possible that the answer might not exist in the initial retrieved context or that the model would need to combine information across multiple contexts (Wang et al., 2018b).We equip the reader with [Figure]
Revised: It is possible that the answer might not exist in the initial retrieved context or that the model would need to combine information across multiple contexts (Wang et al., 2018b).

Original: E m b e d q u e r y [Figure] an additional gated recurrent unit (Cho et al., 2014; Chung et al., 2014) which takes in the state of the reader and the current query vector and generates a new query vector .
Revised: We equip the reader with an additional gated recurrent unit (Cho et al., 2014; Chung et al., 2014) which takes in the state of the reader and the current query vector and generates a new query vector .

Original: This new query vector is then used by the retriever model to re-rank the context.
Revised: This new query vector is then used by the retriever model to re-rank the context.

Original: This allows the model to read new contexts and combine evidence across multiple contexts, if required.
Revised: This allows the model to read new contexts and combine evidence across multiple contexts, if required.

Original: Since the retriever makes a ‘hard selection’ of context to send to the reader, we train the retriever and the reader jointly using reinforcement learning (RL).
Revised: Since the retriever makes a ‘hard selection’ of context to send to the reader, we train the retriever and the reader jointly using reinforcement learning (RL).

Original: Recently, Buck et al.
Revised: Recently, Buck et al.

Original: (2018) introduced the ActiveQA model in which a RL agent sits between the user and the QA system and learns to reformulate the natural language query so as to maximize performance.
Revised: (2018) introduced the ActiveQA model in which a RL agent sits between the user and the QA system and learns to reformulate the natural language query so as to maximize performance.

Original: However, reformulating the query in a natural language domain is a hard problem as we do not have a good generative model for natural language.
Revised: However, reformulating the query in a natural language domain is a hard problem as we do not have a good generative model for natural language.

Original: In our approach, we circumvent this issue by performing query reformulation in the continuous embedding space and not in the space of natural language.
Revised: In our approach, we circumvent this issue by performing query reformulation in the continuous embedding space and not in the space of natural language.

Original: Empirically, we ﬁnd that our model signiﬁcantly outperforms ActiveQA.
Revised: Empirically, we ﬁnd that our model signiﬁcantly outperforms ActiveQA.

Original: Our architecture draws inspiration from how students are instructed to take reading comprehension tests (Cunningham & Shablak, 1975; Bishop et al., 2006; Duggan & Payne, 2009).
Revised: Our architecture draws inspiration from how students are instructed to take reading comprehension tests (Cunningham & Shablak, 1975; Bishop et al., 2006; Duggan & Payne, 2009).

Original: Given a document containing multiple paragraphs and a set of questions which can be answered from the document, (a) the student quickly skims the paragraphs, (b) then for each question, she ﬁnds the most relevant paragraphs that she thinks will answer the question.
Revised: Given a document containing multiple paragraphs and a set of questions which can be answered from the document, (a) the student quickly skims the paragraphs, (b) then for each question, she ﬁnds the most relevant paragraphs that she thinks will answer the question.

Original: (c)
Revised: (c)

Original: She then carefully reads the chosen paragraph to answer the question (d)
Revised: She then carefully reads the chosen paragraph to answer the question (d)

Original: However, if the chosen paragraph does not answer the question, then given the question and the knowledge of what she has read till now, she decides which paragraph to read next.
Revised: However, if the chosen paragraph does not answer the question, then given the question and the knowledge of what she has read till now, she decides which paragraph to read next.

Original: Step (a) is akin to our model encoding and storing the question independent paragraph representations and step (b) corresponds to the inner product search to ﬁnd the relevant context.
Revised: Step (a) is akin to our model encoding and storing the question independent paragraph representations and step (b) corresponds to the inner product search to ﬁnd the relevant context.

Original: The reading of the context by the sophisticated neural machine reader corresponds to step (c) and the last step corresponds to the iterative (multi-step) interaction between the retriever and the reader.
Revised: The reading of the context by the sophisticated neural machine reader corresponds to step (c) and the last step corresponds to the iterative (multi-step) interaction between the retriever and the reader.

Original: To summarize, this paper makes the following contributions: (a) We introduce a new framework for open-domain QA in which the retriever and reader iteratively interact with each other via a novel multi-step-reasoning component which learns to reformulate the query vector.
Revised: To summarize, this paper makes the following contributions: (a) We introduce a new framework for open-domain QA in which the retriever and reader iteratively interact with each other via a novel multi-step-reasoning component which learns to reformulate the query vector.

Original: (b) Unlike several recent open-domain models, our paragraph vectors are independent of the query representations which makes our architecture highly scalable and we empirically demonstrate it by running large scale experiments over millions of paragraphs.
Revised: (b) Unlike several recent open-domain models, our paragraph vectors are independent of the query representations which makes our architecture highly scalable and we empirically demonstrate it by running large scale experiments over millions of paragraphs.

Original: (c) Lastly, our framework is agnostic to the architecture of the reader and we demonstrate the efﬁcacy of our framework on two different neural machine reading comprehension models.
Revised: (c) Lastly, our framework is agnostic to the architecture of the reader and we demonstrate the efﬁcacy of our framework on two different neural machine reading comprehension models.

Original:   M ODEL
Revised:  B ASIC C OMPONENTS OF OUR M ODEL

Original: The architecture of our model consists of three main components - (a) paragraph retriever - that computes a relevance score for each paragraph w.r.t a given query and ranks them according to the computed score.
Revised: The architecture of our model consists of three main components - (a) paragraph retriever - that computes a relevance score for each paragraph w.r.t a given query and ranks them according to the computed score.

Original: (b) reader - a more sophisticated neural machine reading model that receives few top-ranked paragraphs from the retriever and outputs a span of text as a possible answer to the query and (c) multi-step-reasoner - a gated recurrent unit that facilitates iterative interaction between the retriever and the reader.
Revised: (b) reader - a more sophisticated neural machine reading model that receives few top-ranked paragraphs from the retriever and outputs a span of text as a possible answer to the query and (c) multi-step-reasoner - a gated recurrent unit that facilitates iterative interaction between the retriever and the reader.

Original: Formally, the input to our model is a natural language question Q = q 1 , q 2 ,..., q n consisting of n tokens and a set of paragraphs P = { p 1 , p 2 , ...
Revised: Formally, the input to our model is a natural language question Q = q 1 , q 2 ,..., q n consisting of n tokens and a set of paragraphs P = { p 1 , p 2 , ...

Original: p K } .
Revised: p K } .

Original: Our model extracts a span of text a as the answer to the question from the paragraphs in P. Note that the set of paragraphs in P can be the paragraphs in a set of documents retrieved by a search engine (as in T RIVIA QA web-open (Joshi et al., 2017)) or it could be all the paragraphs in a large text corpus such as Wikipedia.
Revised: Our model extracts a span of text a as the answer to the question from the paragraphs in P. Note that the set of paragraphs in P can be the paragraphs in a set of documents retrieved by a search engine (as in T RIVIA QA web-open (Joshi et al., 2017)) or it could be all the paragraphs in a large text corpus such as Wikipedia.

Original: Next we describe each individual component of our model.
Revised: Next we describe each individual component of our model.

Original: 2. P ARAGRAPH R ETRIEVER
Revised: 2. P ARAGRAPH R ETRIEVER

Original: The paragraph retriever computes a score for how likely a paragraph is to contain an answer to a given question.
Revised: The paragraph retriever computes a score for how likely a paragraph is to contain an answer to a given question.

Original: The paragraph representations are computed independent of the query and once computed, they are not updated.
Revised: The paragraph representations are computed independent of the query and once computed, they are not updated.

Original: This allows us to cache the representations and store them ofﬂine.
Revised: This allows us to cache the representations and store them ofﬂine.

Original: The relevance score of a paragraph is computed as a inner product between the paragraph and the query vectors.
Revised: The relevance score of a paragraph is computed as a inner product between the paragraph and the query vectors.

Original: The paragraph and query representations are computed as follows.
Revised: The paragraph and query representations are computed as follows.

Original: Given a paragraph p = { p 1 , p 2 ,..., p m } consisting of m tokens, a multi-layer recurrent neural network encodes each tokens in the paragraph.
Revised: Given a paragraph p = { p 1 , p 2 ,..., p m } consisting of m tokens, a multi-layer recurrent neural network encodes each tokens in the paragraph — { p 1 , p 2 ,..., p m } =

Original: [Equation] where p j ∈ R 2 d encodes useful contextual information around the j -th token.
Revised: RNN ( { p 1 , p 2 ,..., p m } ) , where p j ∈ R 2 d encodes useful contextual information around the j -th token.

Original: Speciﬁcally we choose to use a multi-layer bidirectional long-short term memory network (LSTM) (Hochreiter & Schmidhuber, 1997) and take p j as the hidden units in the last layer of the RNN.
Revised: Speciﬁcally we choose to use a multilayer bidirectional long-short term memory network (LSTM) (Hochreiter & Schmidhuber, 1997) and take p j as the hidden units in the last layer of the RNN.

Original: We concatenate the representation computed by the forward and the backward LSTM.
Revised: We concatenate the representation computed by the forward and the backward LSTM.

Original: To compute a single paragraph vector p ∈ R 2 d from all the token representations, we combine them using weights b j ∈ R
Revised: To compute a single paragraph vector p ∈ R 2 d from all the token representations, we combine them using weights b j ∈ R

Original: [Equation]
Revised: [Equation]

Original: Here b j encodes the importance of each token and w ∈ R 2 d , W s ∈ R 2 d × 2 d are learned weights.
Revised: Here b j encodes the importance of each token and w ∈ R 2 d , W s ∈ R 2 d × 2 d are learned weights.

Original: This is similar to computing self-attention (Lin et al., 2017; Vaswani et al., 2017).
Revised: This is similar to computing self-attention (Lin et al., 2017; Vaswani et al., 2017).

Original: The query q = { q 1 , q 2 ,..., q n } is encoded by another network with the exact same architecture to obtain a query vector q ∈ R 2 d .
Revised: The query q = { q 1 , q 2 ,..., q n } is encoded by another network with the exact same architecture to obtain a query vector q ∈ R 2 d .

Original: Next the relevance score of a paragraph w.r.t the query ( score ( p , q ) ∈ R ) is computed by a simple inner product:
Revised: Next the relevance score of a paragraph w.r.t the query ( score ( p , q ) ∈ R ) is computed by a simple inner product — score ( p , q )

Original: [Equation]
Revised: = (cid:104) p , q (cid:105) .

Original: To return, we select k paragraphs corresponding the maximum score.
Revised: The paragraph retriever then returns the top scoring k paragraphs to the reader.

Original: Thus, our problem  reduces to maximum inner product search (MIPS).When number of candidate paragraphs is small, we can evaluate the score for each candidate paragraph on GPU to perform MIPS.
Revised: Fast inner product search. Inner product can be efﬁciently computed on a GPU, however in our experiments with large corpus with over million paragraphs, (a) the paragraph vectors require more memory than available in a single commodity GPU (b) it is computationally wasteful to compute inner products with all paragraph vectors and we can do better by leveraging fast nearest neighbor (NN) search algorithms. Since the retriever has to ﬁnd the k paragraphs with the highest inner products w.r.t query, our problem essentially reduces to maximum inner product search (MIPS).

Original: However, such approach of linear search is wasteful and not scalable when the number of candidate paragraphs is large, e.g. in case of full Wikipedia.
Revised: There exists data structures for fast NN search in metric spaces such as Cover Trees (Beygelzimer et al., 2006). But we cannot use them directly, since triangle equality does not hold in inner product space.

Original: Borrowing ideas from Bachrach et al.
Revised: Borrowing ideas from Bachrach et al.

Original: (2014); Anonymous (2018), we propose to useexcellent nearest neighbor (NN) search algorithms to perform the MIPS in sublinear time.
Revised: (2014); Anonymous (2018), we propose to use nearest neighbor (NN) search algorithms to perform the MIPS in sublinear time as follows.

Original: Assume, we can have an upper boundu for the L2 norm, i.e. u ≤ (cid:107) p (cid:107) , ∀ p .
Revised: Let u be an upper bound for the L2 norm, i.e. u ≥ (cid:107) p (cid:107) , ∀ p .

Original: Then, the trick is to perform NN search in L2 distance with modiﬁed query and paragraph vectors so that the search is equivalent to MIPS for original query.
Revised: Then, the trick is to perform NN search in L2 distance with modiﬁed query and paragraph vectors so that the search is equivalent to MIPS for original query.

Original: In this regard, deﬁne the augmented paragraph vectors as ˜ p
Revised: In this regard, deﬁne the augmented paragraph vectors as ˜ p

Original: i =
Revised: i =

Original: [ p i ; (cid:112) u 2 −(cid:107) p i (cid:107) 2 ] and augmented query vector as ˜ q =
Revised: [ p i ; (cid:112) u 2 −(cid:107) p i (cid:107) 2 ] and augmented query vector as ˜ q =

Original: [ q ;0 ] .
Revised: [ q ;0 ] .

Original: With this transformation, we can see that:
Revised: With this transformation, we can see that:

Original: [Equation]
Revised: [Equation]

Original: Many exact NN search algorithms can ﬁnd k-NN in time (sublinear) logarithmic in number of paragraphs (Beygelzimer et al., 2006; Anonymous, 2018), after an one-time preprocessing.
Revised: Many exact NN search algorithms can ﬁnd k-NN in time (sublinear) logarithmic in number of paragraphs (Beygelzimer et al., 2006; Anonymous, 2018), after an one-time preprocessing.

Original: In case of question answering, the preprocessing can be doneonly once because the set of all paragraphs is ﬁxed for all the queries.
Revised: In our architecture, the preprocessing can be done because the set of all paragraphs is ﬁxed for all the queries.

Original: We chose to use SGTree (Anonymous, 2018) to perform the NN/MIPS due to its fast construction time and competitive performance.
Revised: We chose to use SGTree (Anonymous, 2018) to perform the NN/MIPS due to its fast construction time and competitive performance reducing the search to logarithmic time per query.

Original: Note that our paragraph representations are independent of the given query and are never updated once they are trained.
Revised: Note that our paragraph representations are independent of the given query and are never updated once they are trained.

Original: After training is completed, we cache theW s p ∈ R 2 d vectors.
Revised: After training is completed, we cache the p ∈ R 2 d vectors.

Original: This is unlike many recent work in open-domain QA.
Revised: This is unlike many recent work in open-domain QA.

Original: For example, in R 3 (Wang et al., 2018a), the retriever uses
Revised: For example, in R 3 (Wang et al., 2018a), the retriever uses

Original: Match-LSTM model (Wang & Jiang, 2017) to compute question-matching representations for each token in the passage.
Revised: Match-LSTM model (Wang & Jiang, 2017) to compute question-matching representations for each token in the passage.

Original: The paragraph representations are then obtained by running a bi-directional
Revised: The paragraph representations are then obtained by running a bi-directional

Original: RNN over these matched representation.
Revised: RNN over these matched representation.

Original: Although powerful, this architecture design will not scale in open-domain settings where the retriever has to re-compute new representations for possibly millions of paragraphs for every new question.
Revised: Although powerful, this architecture design will not scale in open-domain settings where the retriever has to re-compute new representations for possibly millions of paragraphs for every new question.

Original: In contrast, once training is over, we cache the paragraph representation and use it for every query at test time.
Revised: In contrast, once training is over, we cache the paragraph representation and use it for every query at test time.

Original: Training - Following previous work (Htut et al., 2018; Lin et al., 2018; Wang et al., 2018a), we gather labels for paragraphs during training using distant supervision (Mintz et al., 2009).
Revised: Training - Following previous work (Htut et al., 2018; Lin et al., 2018; Wang et al., 2018a), we gather labels for paragraphs during training using distant supervision (Mintz et al., 2009).

Original: A paragraph that contains the exact ground truth answer string is labeled as an positive example.
Revised: A paragraph that contains the exact ground truth answer string is labeled as an positive example.

Original: For a positive (negative) labeled paragraph, we maximize (minimize) the log ( σ ( score ( p , q ))) .
Revised: For a positive (negative) labeled paragraph, we maximize (minimize) the log ( σ ( score ( p , q ))) .

Original: The number of layers of the bi-directional LSTM encoder is set to three and we use Adam (Kingma & Ba, 2014) for optimization.
Revised: The number of layers of the bi-directional LSTM encoder is set to three and we use Adam (Kingma & Ba, 2014) for optimization.

Original: Once training is done, we pre-compute and cache the paragraph representation of each dataset in our experiments.
Revised: Once training is done, we pre-compute and cache the paragraph representation of each dataset in our experiments.

Original: 2. M ACHINE R EADER
Revised: 2. M ACHINE R EADER

Original: The reader is a sophisticated neural machine reading comprehension (MRC) model that takes in the top few paragraphs sent by the retriever and outputs a span of answer text.
Revised: The reader is a sophisticated neural machine reading comprehension (MRC) model that takes in the top few paragraphs sent by the retriever and outputs a span of answer text.

Original: Our model is agnostic to the exact architecture of the reader and we perform experiments to show the efﬁcacy of our model on two state-of-the-art neural machine reading models - DrQA
Revised: Our model is agnostic to the exact architecture of the reader and we perform experiments to show the efﬁcacy of our model on two state-of-the-art neural machine reading models - DrQA

Original: (Chen et al., 2017) and BiDAF (Seo et al., 2017).
Revised: (Chen et al., 2017) and BiDAF (Seo et al., 2017).

Original: MRC models designed for SQ U AD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), etc operate on a single paragraph.
Revised: MRC models designed for SQ U AD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), etc operate on a single paragraph.

Original: However it has been shown that reading and aggregating evidence across multiple paragraphs is important when doing QA over larger evidence set (e.g. full Wikipedia document) (Swayamdipta et al., 2018; Clark & Gardner, 2018) and in open domain settings (Wang et al., 2018b).
Revised: However it has been shown that reading and aggregating evidence across multiple paragraphs is important when doing QA over larger evidence set (e.g. full Wikipedia document) (Swayamdipta et al., 2018; Clark & Gardner, 2018) and in open domain settings (Wang et al., 2018b).

Original: Most MRC models compute a start and end scores for each token in the paragraph that represents how likely a token is the start/end of an answer span.
Revised: Most MRC models compute a start and end scores for each token in the paragraph that represents how likely a token is the start/end of an answer span.

Original: To gather evidence across multiple paragraphs sent by the retriever, we normalize the start/end scores across the paragraphs.
Revised: To gather evidence across multiple paragraphs sent by the retriever, we normalize the start/end scores across the paragraphs.

Original: Furthermore a text span can appear multiple times in a paragraph.
Revised: Furthermore a text span can appear multiple times in a paragraph.

Original: To give importance to all answer spans in the text, our objective aggregates (sums) the log-probability of the score for each answer position.
Revised: To give importance to all answer spans in the text, our objective aggregates (sums) the log-probability of the score for each answer position.

Original: In other words, let I ( w , p ) denote the token start positions where the answer span appears in the paragraph p and let w S be the starting word of the answer span.
Revised: In other words, let I ( w , p ) denote the token start positions where the answer span appears in the paragraph p and let w S be the starting word of the answer span.

Original: Our model maximizes the sum of the following objective for the start and end word of an answer spans as follows.
Revised: Our model maximizes the sum of the following objective for the start and end word of an answer spans as follows.

Original: (For brevity, we only show the objective for the starting word (w s ) of the span.)
Revised: (For brevity, we only show the objective for the starting word (w s ) of the span.)

Original: [Equation]
Revised: [Equation]

Original: Here, P denotes the set of all top-ranked paragraphs by the retriever, n j denotes the number of tokens in paragraph j and score start ( k , j ) denotes the start score of the k -th token in the j -th paragraph.
Revised: Here, P denotes the set of all top-ranked paragraphs by the retriever, n j denotes the number of tokens in paragraph j and score start ( k , j ) denotes the start score of the k -th token in the j -th paragraph.

Original: A similar score aggregation strategy has been used in previous work (Kadlec et al., 2016; Clark &
Revised: A similar score aggregation strategy has been used in previous work (Kadlec et al., 2016; Clark &

Original: Gardner, 2018).
Revised: Gardner, 2018).

Original: During inference, following Chen et al.
Revised: During inference, following Chen et al.

Original: ; Seo et al.
Revised: ; Seo et al.

Original: (2017), the score of a span that starts at token position i and ends at position j of paragraph p is given by the sum of the score start ( i , p ) and score end ( j , p ) . 
Revised: (2017), the score of a span that starts at token position i and ends at position j of paragraph p is given by the sum of the score start ( i , p ) and score end ( j , p ) . During inference, we score spans up to a pre-deﬁned maximum length of 15.

Revised: Further, we add the scores of the spans across different paragraphs if they have the same surface form.

Revised: That is, if the same span (e.g. “Barack Obama”) occurs multiple times in several paragraphs, we sum the individual span scores.

Revised: However, to ensure tractability, we only consider the top 10 spans for aggregation.

Original: It should be noted that the changes we made to aggregate evidence over multiple paragraphs and mentions of spans needs no change in the original architecture of the machine reading model.
Revised: It should be noted that the changes we made to aggregate evidence over multiple paragraphs and mentions of spans needs no change in the original architecture of the machine reading model.

Original: 2.  M ULTI -S TEP -R EASONER
Revised:  M ULTI -S TEP -R EASONER

Original: A novel component of our open-domain architecture is the multi-step-reasoner which facilitates iterative interaction between the retriever and the machine reader.
Revised: A novel component of our open-domain architecture is the multi-step-reasoner which facilitates iterative interaction between the retriever and the machine reader.

Original: The multi-step-reasoner is a gated recurrent unit (Cho et al., 2014; Chung et al., 2014) which takes in the current state of the reader and the current query vector and does a gated update to produce a reformulated query.
Revised: The multi-step-reasoner is a gated recurrent unit (Cho et al., 2014; Chung et al., 2014) which takes in the current state of the reader and the current query vector and does a gated update to produce a reformulated query.

Original: The reformulated query is sent back to the retriever which uses it to re-rank the paragraphs in the corpus.
Revised: The reformulated query is sent back to the retriever which uses it to re-rank the paragraphs in the corpus.

Original: Since the gated update of the multi-step-reasoner, conditions on the current state of the machine reader, this multi-step interaction provides a way for the search engine (retriever) and the QA model (reader) to communicate with each other.
Revised: Since the gated update of the multi-step-reasoner, conditions on the current state of the machine reader, this multi-step interaction provides a way for the search engine (retriever) and the QA model (reader) to communicate with each other.

Original: This can be seen as an instance of two agents cooperating via communication to solve a task (Lazaridou et al., 2017; Lee et al., 2018; Cao et al., 2018).
Revised: This can be seen as an instance of two agents cooperating via communication to solve a task (Lazaridou et al., 2017; Lee et al., 2018; Cao et al., 2018).

Original: More formally, let q t ∈ R 2 d be the current query representation which was most recently used by the paragraph retriever to score the paragraphs in the corpus (equation 1).
Revised: More formally, let q t ∈ R 2 d be the current query representation which was most recently used by the paragraph retriever to score the paragraphs in the corpus.

Original: The multi-step-reasoner also has access to the reader state which is computed from the hidden memory vectors of the reader.
Revised: The multi-step-reasoner also has access to the reader state which is computed from the hidden memory vectors of the reader.

Original: The reader state captures the current information that the reader has encoded after reading the paragraphs that was sent by the retriever.
Revised: The reader state captures the current information that the reader has encoded after reading the paragraphs that was sent by the retriever.

Original: Next we show how the reader state is computed.
Revised: Next we show how the reader state is computed.

Revised: The span extractive machine reading architectures compute a hidden representation for each token in the paragraph.

Revised: A majority of them use RNN encoders, however recent successful models also use

Revised: CNN (Yu et al., 2018; Swayamdipta et al., 2018) and Transformer (Devlin et al., 2018) architectures.

Original:  Let m j ∈ R 2 p be the hidden vector associated with the j -th token in the paragraph as computed by the bi-directional multi-layer recurrent neural network encoder of the reader model 1 .
Revised: To compute the reader state, our framework needs access to the hidden representation for each token in the paragraph computed by the reader. Let m j ∈ R 2 p be the hidden vector associated with the j -th token in the paragraph.

Original: Let L ∈ R 2 p be the ﬁnal query representation of the reader model.
Revised: Let L ∈ R 2 p be the ﬁnal query representation of the reader model.

Original: L is usually created by some pooling operation on the hidden representation of each question token.
Revised: L is usually created by some pooling operation on the hidden representation of each question token.

Original: The reader state S ∈ R 2 p is computed from each of the hidden vectors m j and L by ﬁrst computing soft-attention weights between each paragraph token, followed by combining each m j with the soft attention weights.
Revised: The reader state S ∈ R 2 p is computed from each of the hidden vectors m j and L by ﬁrst computing soft-attention weights between each paragraph token, followed by combining each m j with the soft attention weights.

Original: [Equation]
Revised: [Equation]

Original: Finally, the new reformulated query q t + 1 for the paragraph retriever is calculated by a GRU update.
Revised: Finally, the new reformulated query q t + 1 for the paragraph retriever is calculated by a GRU update — q t + 1 =

Original: [Equation]
Revised: GRU ( q t , S ) .

Original: The gated update ensures that relevant information from q t is preserved and new and useful information from the reader state S is added to the reformulated query.
Revised: The gated update ensures that relevant information from q t is preserved and new and useful information from the reader state S is added to the reformulated query.

Original: Training - There exists no supervision for training the query reformulation of the multi-step-reasoner.
Revised: There exists no supervision for training the query reformulation of the multi-step-reasoner.

Original: Therefore, to train the parameters of the GRU network we employ reinforcement learning .
Revised: Therefore, to train the parameters of the GRU network we employ reinforcement learning .

Original: We deﬁne the problem as a deterministic ﬁnite horizon Partially Observed Markov decision process (POMDP).
Revised: We deﬁne the problem as a deterministic ﬁnite horizon Partially Observed Markov decision process (POMDP).

Original: The components of POMDP are described as follows:
Revised: The components of POMDP are described as follows:

Original: States .
Revised: States .

Original: A state in the state space consists of the entire text corpora, the query, the answer
Revised: A state in the state space consists of the entire text corpora, the query, the answer

Original: and k selected paragraphs.
Revised: and k selected paragraphs.

Original: The reader is part of the environment and is fully described given the current state, i.e. selected paragraphs and the query.
Revised: The reader is part of the environment and is fully described given the current state, i.e. selected paragraphs and the query.

Original: Observations .
Revised: Observations .

Original: The agent only observes a function of the current state.
Revised: The agent only observes a function of the current state.

Original: In particular, to the agent only the initial query vector and the memory of the reader model is shown, which are a function of the current state.
Revised: In particular, to the agent only the initial query vector and the memory of the reader model is shown, which are a function of the current state.

Original: Intuitively, this represents the information encoded by the machine reader model after reading the top k paragraphs sent by the retriever in current step.
Revised: Intuitively, this represents the information encoded by the machine reader model after reading the top k paragraphs sent by the retriever in current step.

Original: Actions .
Revised: Actions .

Original: The set of all paragraphs in the text corpora forms the action space.
Revised: The set of all paragraphs in the text corpora forms the action space.

Original: The retriever scores all paragraphs w.r.t the current query and selects the top k paragraphs to send to the reader model.
Revised: The retriever scores all paragraphs w.r.t the current query and selects the top k paragraphs to send to the reader model.

Original: We treat k as a hyper-parameter in our experiments.
Revised: We treat k as a hyper-parameter in our experiments.

Original: Reward .
Revised: Reward .

Original: At every step, the reward is measured by how well the answer extracted by the reader model matches to the ground-truth answer.
Revised: At every step, the reward is measured by how well the answer extracted by the reader model matches to the ground-truth answer.

Original: We use the F 1 score (calculated by word overlap betweenprediction and ground-truth) as the reward at each step.
Revised: We use the F 1 score (calculated by word overlap between

Original: Transition .
Revised: Transition .

Original: The environment evolves deterministically after reading the paragraphs sent by the retriever.
Revised: The environment evolves deterministically after reading the paragraphs sent by the retriever.

Original: Our policy  is parameterized by the GRU of the multi-step-reasoner and we directly optimize the parameters to maximize the expected reward.
Revised: Our policy π is parameterized by the GRU of the multi-step-reasoner. Let r t denote the reward ( Fscore) returned by the environment at time t . We directly optimize the parameters to maximize the expected reward given by — J ( θ ) = E π (cid:2)

Revised: ∑ Tt = 1 r t (cid:3) .

Revised: Here T denotes the number of steps of interactions between the retriever and the reader.

Original: We treat the reward at each step equally and do not apply any discounting.
Revised: We treat the reward at each step equally and do not apply any discounting.

Original: We employ the R EINFORCE (Williams, 1992) algorithm to train the GRU network.
Revised: Motivated by the R EINFORCE (Williams, 1992) algorithm, we compute the gradient of our objective as [Equation]

Original: 2. P UTTING IT ALL TOGETHER
Revised: Here, p t is the top-ranked paragraph returned by the retriever at time t and the probability that the current policy assigns to p t is computed by normalizing the scores with a softmax. It is usual practice to add a variance reduction baseline (e.g. average of rewards in the minibatch) for stable training, but we found this signiﬁcantly degrades the ﬁnal performance. We think this is because, in QA, a minibatch consists of questions of varying difﬁculty and hence the rewards in a batch itself have high variance. This is similar to ﬁndings by Shen et al.

Revised: (2017) for closed domain-QA (e.g. SQ U AD).

Original: Our open-domain architecture is summarized above in Algorithm 1.
Revised: Our open-domain architecture is summarized above in Algorithm 1.

Original: Given a large text corpora (as a list of paragraphs), the corresponding paragraph embeddings (which can be trained by the procedure in (§ 2.1)) and hyper-parameters (T , k ), our model M returns a text span a as answer.
Revised: Given a large text corpora (as a list of paragraphs), the corresponding paragraph embeddings (which can be trained by the procedure in (§ 2.1)) and hyper-parameters (T , k ), our model M returns a text span a as answer.

Original: The multi-step interaction between the retriever and reader can be best understood by the for loop in line 2 of algorithm 1.
Revised: The multi-step interaction between the retriever and reader can be best understood by the for loop in line 2 of algorithm 1.

Original: The initial query q 0 is ﬁrst used to rank all the paragraphs in the corpus (line 3), followed by which the top k paragraphs are sent to the reader (line 4).
Revised: The initial query q 0 is ﬁrst used to rank all the paragraphs in the corpus (line 3), followed by which the top k paragraphs are sent to the reader (line 4).

Original: The reader returns the answer span (with an associated score for the span) and also its internal state(§ 2.3) (line 5).
Revised: The reader returns the answer span (with an associated score for the span) and also its internal state (line 5).

Original: The GRU network then takes in the current query and the reader state to produce the updated query which is then passed to the retriever (line 6).
Revised: The GRU network then takes in the current query and the reader state to produce the updated query which is then passed to the retriever (line 6).

Original: The retriever uses this updated query to again re-rank the paragraphs and the entire process is repeated for T steps.
Revised: The retriever uses this updated query to again re-rank the paragraphs and the entire process is repeated for T steps.

Original: At the end of T steps, the model returns the span with the highest score returned by the reader model.
Revised: At the end of T steps, the model returns the span with the highest score returned by the reader model.

Original: The reader is trained using supervised learning (using the correct spans as supervision) and the parameters of the GRU network are trained using reinforcement learning.
Revised: The reader is trained using supervised learning (using the correct spans as supervision) and the parameters of the GRU network are trained using reinforcement learning.

Original: During training, we ﬁrst pre-train the reader model by setting the number of multi-step reasoning steps (T = 1).
Revised: During training, we ﬁrst pre-train the reader model by setting the number of multi-step reasoning steps (T = 1).

Original: After the training converges, we freeze the parameters of the reader model and train the parameters of the GRU network using policy gradients.
Revised: After the training converges, we freeze the parameters of the reader model and train the parameters of the GRU network using policy gradients.

Original: The output of the reader model is used to generate the reward which is used to train the policy network.
Revised: The output of the reader model is used to generate the reward which is used to train the policy network.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Open domain QA is a well-established task that dates back to few decades of reasearch.
Revised: Open domain QA is a well-established task that dates back to few decades of reasearch.

Original: For example the B ASEBALL system (Green Jr et al., 1961) aimed at answering open domain question albeit for a speciﬁc domain.
Revised: For example the B ASEBALL system (Green Jr et al., 1961) aimed at answering open domain question albeit for a speciﬁc domain.

Original: Open-domain QA has been popularized by the Trec-8 task (Voorhees et al., 
Revised: Open-domain QA has been popularized by the Trec-8 task (Voorhees et al., 1999).

Original: [Table]

Original: Research has still progressed in open-domain settings with the introduction of datasets such as SearchQA Dunn et al.
Revised: Research has still progressed in open-domain settings with the introduction of datasets such as SearchQA Dunn et al.

Original: Quasar (Dhingra et al., 2017), TriviaQA (Joshi et al., 2017) but performance is still unsatisfactory.
Revised: Quasar (Dhingra et al., 2017), TriviaQA (Joshi et al., 2017) but performance is still unsatisfactory.

Original: In many open-domain systems (Chen et al., 2017; Clark & Gardner, 2018), the retriever is a simple IR based system (e.g. tﬁdf retriever) with no trainable parameters and hence the retriever cannot overcome from its mistakes.
Revised: In many open-domain systems (Chen et al., 2017; Clark & Gardner, 2018), the retriever is a simple IR based system (e.g. tﬁdf retriever) with no trainable parameters and hence the retriever cannot overcome from its mistakes.

Original: Recent work such as R 3 (Wang et al., 2018a) use a trained retriever and have shown improvement in performance.
Revised: Recent work such as R 3 (Wang et al., 2018a) use a trained retriever and have shown improvement in performance.

Original: However R 3 forms query dependent paragraph representation and such architectures will not scale to full open-domain settings where the retriever has to rank millions of paragraphs.
Revised: However R 3 forms query dependent paragraph representation and such architectures will not scale to full open-domain settings where the retriever has to rank millions of paragraphs.

Original: The retriever and reader of R 3 and DS-QA Lin et al.
Revised: The retriever and reader of R 3 and DS-QA Lin et al.

Original: (2018) are trained jointly but they do not support iterative reasoning thereby failing to recover from any mistakes made by the ranker or where evidence needs to be aggregated across multiple paragraphs.
Revised: (2018) are trained jointly but they do not support iterative reasoning thereby failing to recover from any mistakes made by the ranker or where evidence needs to be aggregated across multiple paragraphs.

Revised: Lastly, we note that there has been work (Huang et al., 2013) to use deep neural network based models for better retrieval or web search but these model do not train a retriever and a QA model jointly.

Original: Models that do iterative reasoning (Shen et al., 2017; Liu et al., 2017) i.e. read a single paragraph multiple times, have proven to be effective in closed domain settings (e.g. SQ U AD).
Revised: Models that do iterative reasoning (Shen et al., 2017; Liu et al., 2017) i.e. read a single paragraph multiple times, have proven to be effective in closed domain settings (e.g. SQ U AD).

Original: Our model can be seen as doing iterative reasoning but in a more realistic, open-domain setting.
Revised: Our model can be seen as doing iterative reasoning but in a more realistic, open-domain setting.

Original: In fact, our architecture reduces to these models when the set of evidence consists of only one paragraph and hence can be seen as a strict generalization of these models.
Revised: In fact, our architecture reduces to these models when the set of evidence consists of only one paragraph and hence can be seen as a strict generalization of these models.

Original: E XPERIMENTS
Revised: Nogueira & Cho (2017) use query reformulation to maximize just the recall of a IR system whereas we use query reformulation to maximize the performance of a complete QA system. Perhaps the most related to our work is Active Question Answering (AQA)

Original:  We now present empirical studies for our modelin order to establish that (i) multi-step reasoning is effective (§ 4.1), (ii) the framework achieves  high accuracy on large open-domain datasets (§ 4.2), (iii) the framework is agnostic to reader architecture (§ 4.4), and (iv) the framework scales to corpora containing millions of paragraphs (§ 4.3)
Revised: (Buck et al., 2018), which use reformulation of the natural language query to improve the performance of a BiDAF reader model on S EARCH QA. We directly compare with AQA and show that our model achieves signiﬁcantly better performance. At a very high level, our framework can be viewed as an instance of a memory network (Sukhbaatar et al., 2015) where the multi-step-reasoner can be seen as the controller update step of a memory network type of inference.  E XPERIMENTS

Original: To illustrate these claims, we evaluate on the following open-domain question answering datasets:
Revised: [Table] We now present experiments to show the effectiveness of each component of our framework.

Original: TriviaQA-open – TriviaQA includes around 95,000 questions answers pairs authored by Trivia enthusiasts.
Revised: We experiment on the following large open-domain QA datasets — (a) T RIVIA QA-unﬁltered– a version of T RIVIA QA (Joshi et al., 2017) built for open-domain

Original: We use the unﬁltered TriviaQA corpus which contains a lot more  paragraphs than the web/wiki setting.
Revised: QA. It has much more number of paragraphs than the web/wiki setting of T RIVIA QA.

Original: Moreover there is no guarantee that  the evidence will have answer to a question.
Revised: Moreover, there is no guarantee that every document in the evidence will contain the answer making this setting more challenging.

Original: TriviaQA-full – This is a setting in which we combine all evidence for every question in the development set.
Revised: (b) T RIVIA QA-open– To test our framework for large scale setting, we combine all evidence for every question in the development set.

Original: This resulted in a corpus containing 1.68M paragraphs per question.
Revised: This resulted in a corpus containing 1.68M paragraphs per question.

Original: In other words, to answer a question, the retriever has to ﬁnd the correct context among 1.68M paragraphs.
Revised: In other words, to answer a question, our model has to ﬁnd the correct answer from 1.68M paragraphs.

Original: SearchQA – is another open-domain dataset which consists of question-answer pairs crawled from the J!
Revised: (c) S EARCH QA (Dunn et al., 2017) – is another open-domain dataset which consists of question-answer pairs crawled from the J!

Original: archive.
Revised: archive.

Original: The paragraphs are obtained from 50 webpages retrieved using the Google search API Quasar-T – consists of 43K open-domain questions where the paragraphs are obtained by retrieving sentences from the ClueWeb data source.
Revised: The paragraphs are obtained from 50 web

Original: Implementation All the algorithms are implemented on PyTorch.

Original: For all experiments, we use

Original: Adam optimizer (Kingma & Ba, 2014) with the default hyperparameter settings. We run our experiments on a commodity machine with Intel R (cid:13) Xeon R (cid:13) CPU E5-2630 v4 CPU, 256GB RAM, and NVidia R (cid:13) Titan X GPU.
Revised: [Table]

Original: Model
Revised: Model

Original: Quasar-T SearchQA TriviaQA EM
Revised: [Table]

Original: F EM F EM
Revised: P@

Original: F

Original: [Table]
Revised: [Table]

Original: Bidaf
Revised: P@ R 3 (Wang et al., 2018a)

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: Our Retriever 35.

Original: [Table]
Revised: [Table]

Revised: + multi-step (7 steps)

Original: [Table]
Revised: [Table]

Original: multi-step-reasoner 39. 4. E FFECTIVENESS OF M ULTI -S TEP R EASONING
Revised: based retriever of R 3 is a more powerful model than our intial retrieval model. However, we ﬁnd that after doing few steps of iterative interaction between retriever and reader leads to better re-ranking and improved results.

Original: [Figure]
Revised: [Figure]

Original: TriviaQA
Revised: [Table] [Figure] snippets retrieved using the Google search API. (d) Q UASAR -T

Original: SearchQA
Revised: (Dhingra et al., 2017) – consists of 43K open-domain questions where the paragraphs are obtained from the ClueWeb data source. (e) SQ U AD-open– We also experimented on the open domain version of the SQ U AD dataset. For fair comparison to baselines, our evidence corpus was created by retrieving the top-5 wikipedia documents as returned by the pipeline of Chen et al.

Original: [Figure]
Revised: The datasets are summarized in table 1.

Original: [Figure]
Revised: We ﬁrst investigate the performance of our paragraph retriever model (§2.1).

Original: [Figure]
Revised: Our retriever is based on inner product between the query and pre-computed paragraph vectors. This architecture, although more scalable, is less powerful than the retrievers of R 3 and DS-QA which compute query-dependent passage representation via soft alignment (attention) before encoding with a bi-LSTM. Results in table 3 indeed show that retriever of R 3 has better P@ k than our retriever. We also measure the performance of our retriever after few steps (#steps = 7) of interaction between the retriever and reader.

Original: We show how multi-step reasoning using RL, as presented in Section 2.3, improves the question answering accuracy.
Revised: As we can see from table 3, the query reformulation has resulted in better re-ranking and hence an overall improved performance of the retriever.

Original: For each dataset, we start with a reader model which has been trained using no multi-step reasoning steps. We freeze the parameters of the reader and then train the parameters of the GRU (policy) using reinforcement learning.
Revised: To test for scalability in open-domain settings, we conduct a synthetic experiment. We plot the wall-clock time (in seconds) taken by the retriever to score and rank paragraphs with a query. To test for scalability, we increase the number of paragraphs from 500 to 100 million and test on a single

Original: Different datasets have varying degree of complexity of the questions, thus we need different number of multi-step reasoning.
Revised: Titan-X GPU.

Original: For every dataset, we uniformly see improvements in performance over the base model. The improvement with multi-step reasoning over the base model is shown in
Revised: For our baseline, we use the GPU implementation of the retriever of DS-QA (Lin et al., 2018). For our model, we test on three variants — (a) inner product on GPU, (b) inner-product on

Original: Figure 2 and illustratives are shown in Figure 3.
Revised: CPU and (c) inner-product using SG-Tree. Figure 2 shows the results.

Original: 4. A CCURACY
Revised: Our retriever model on GPU is faster than DS-QA, since the latter perform much more computation compared to just inner-product in our case. Moreover, DS-QA quickly uses up all the available memory and throws a memory error (OOM) by just 100K paragraphs. The inner product operation scales up to 1M paragraph before going OOM. Our nearest neighbor approach (SG-Tree) shows very impressive performance even though it operates on CPU and consistently outperforms dot product operation on the CPU.

Original: Next, we demonstrate high accuracy ofmulti-step-reasoner on reported baselines on the three challenging open datasets.
Revised: Next we investigate the overall improvements in the QA performance because of the iterative interaction between the retriever and the reader. Performance on benchmark datasets.

Original: In Table 2, we compare multi-step-reasoner with various prior art using Exact Match (EM) and F1 scores.
Revised: Table 2 compares the performance of our model with various competitive baselines on four open-domain QA datasets.

Original: On the hidden test set of TriviaQA we obtain the highest score ever reported to the best of our knowledge. We also have competitive scores on other datasets.
Revised: The key highlights of the results in this table are (a) We perform comparably well on all the datasets and to the best of our knowledge, our model achieves the highest reported number on the hidden test set of T RIVIA QA-unﬁltered 2 .

Original: For example on SearchQA, we beat AQA (Buck et al., 2018) signiﬁcantly by more than 13 F1 points.
Revised: (b) Our framework achieves uniform boost in scores on two popular and widely used machine reading architectures — Dr.QA (Chen et al., 2017) and BiDAF (Seo et al., 2017).

Original: This shows that it is much more efﬁcient todo query reformulation in rich embedding space rather than in the space of natural language.
Revised: This show that our model is indeed agnostic to the neural architecture of the machine reading model provided it has access to the token level hidden representations of the reader.

Original: We also consistently outperform R 3 onall datasets demonstrating the effectiveness of iterative reasoning.
Revised: We are waiting on the ofﬁcial results of BiDAF on the hidden test set of T RIVIA QA-unﬁltered but we are hopeful that the trend will remain the same.

Original: [Table]
Revised: [Figure]

Original: 4. S CALABILITY
Revised: Performance w.r.t varying number of steps. Figure 3 shows the relative improvements our models achieve on Q UASAR -T, S EARCH QA and T RIVIA QA-unﬁltered with varying number of steps of interaction between the retriever and the reader.

Original: Next we show the scalability of our architecture.
Revised: The key takeaway of this experiment is that multiple steps of interaction uniformly increases performance over base model (with no interaction). Different datasets have varying level of difﬁculties, however, the performance reaches its peak around 5 to 7 steps and does not provide much beneﬁt on further increasing the number of steps. Large scale experiment on T RIVIA QA-open . The benchmark datasets for open-domain QA have on an average hundred paragraphs for each question (Table 1).

Original: For this experiment, we combine all  paragraphs in the development set resulting in 1.68M paragraphs. Since we do not update paragraph representations and we use fast nearest neighbor data structures, our architectures can scale to such large corpus size.
Revised: In real open-domain settings, the size of the evidence corpus might be much bigger. To test our model for the same, we created the T RIVIA QA-open setting, in which for each query we combined all the paragraphs in the development set, resulting in 1.6M paragraphs per query.

Original: Since no other baselines scaled to this size, we compare our multi-step-reasoner model with a reader model with no multi-steps.
Revised: We want to emphasize that the baseline models such as R 3 and DS-QA will not scale to this large setting. The baseline DrQA.

Original: The reader model got an exact match score of  37.45 and F1 score of 42.16, where as the model with reasoning steps 3 got a score of  39.76(Em) and 44.30 (F1).
Revised: model with no interaction between the retriever and reader get a score of EM = 37.45 and F1 = The same model withsteps of interaction achieves a score of EM = 39.76 and F1 =

Original: This also shows that multi-step reasoning is effective even inthe large scale setting.
Revised: The key takeaways from this experiment are — (a) our framework can scale to settings containing millions of paragraphs, (b) iterative interaction still increase performance even in large scale settings, (c) the overall performance has signiﬁcantly decreased (from 61 to 44.3 F1), suggesting that large context in open-domain QA is very challenging and should be an active area of research with signiﬁcant scope for improvement. [Table]

Original: 4. E XPERIMENTS ON B I DAF READER
Revised: 5. A NALYSIS OF R ESULTS

Original: Finally, we show that our framework can be used in exactly the same way for a different reader architecture.
Revised: This section does further analysis of the results of the model. Speciﬁcally we are interested in analyzing if our method is able to gather more relevant evidence as we increase the number of steps of interaction between the retriever and the reader. We conduct this analysis on the development set of S EARCH QA (containing 13,393 queries) and to simplify analysis, we make the retriever choose the top scored single paragraph to send to the reader, at each step.

Revised: The results are noted in table 4.

Original: We use an open source implementation of BiDAF (Seo et al., 2017) and modify it  to return reader state (with the scores).
Revised: As we can see from row 1 of table 4, on increasing steps of interaction, the quality of retrieved paragraphs becomes better, i.e. even though the correct paragraph (containing the answer string) was not retrieved in the ﬁrst step, the retriever is able to fetch relevant paragraphs later. It is also unsurprising to see, that when correct paragraphs are retrieved, the performance of the reader also increases. To check if the our policy was exploring and retrieving different paragraphs, we counted the mean number of unique paragraphs retrieved across all steps and found that it is high (around 2) when #steps of interaction is small as 3, suggesting that the policy chooses to explore initially. As the number of steps is increased, it increases at slower rate suggesting that if the policy has already found good paragraphs for the reader, it chooses to exploit them rather than choosing to explore.

Original: Without any tuning, we observed that multi-step-reasoning gave a boost inpeformance on the QuASAR-T dataset.
Revised: Figure 4 shows two instances where iterative interaction is helpful. In ﬁgure to the left, the retriever is initially unable to ﬁnd a paragraph that directly answers the question, however it ﬁnds a paragraph which gives a different name for the disease allowing it to ﬁnd a more relevant paragraph that directly answers the query. In the ﬁgure to the right, after the query reformulation, both the retrieved paragraphs have the correct answer string. Since we aggregate (sum) the scores of spans, this leads to an increase in the score of the right answer span (Demeter) to be the maximum.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other.
Revised: In this paper, we introduce a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other.

Original: The resulting framework improved accuracy in question answering task by 15% as compared to previous results establishing new stateof-the-art results on the  TriviaQA-unﬁltered dataset(Joshi et al., 2017).
Revised: The resulting framework improved accuracy in question answering task by 9.5% as compared to previous results establishing new state-of-the-art results on the challenging TriviaQA-unﬁltered dataset.

Original:  We also show that our  method can scale to corpora containing millions of paragraphs.
Revised: Our method also performs competitively on three other open-domain datasets namely — S EARCH QA, Q UASAR -T and SQ U ADopen. We also show that our fast retrieval method can scale upto tens of millions of paragraph, much beyond the current capability of existing open-domain systems with a trained retriever module. We also demonstrate that our multi-step interaction brings uniform increase in performance on all datasets.

Original: Lastly, we show that our framework brings uniform improvements to two  neural machine reading architectures - DrQA (Chen et al., 2017) and
Revised: Finally, we show that our method is agnostic to the architecture of the machine reading system provided we have access to the token level hidden representations of the reader. Our method brings an increase in performance to two popular and widely used neural machine reading architectures —

Original: BiDAF (Seo et al., 2017).
Revised: Dr.QA and BiDAF.