Original: A SK THE R IGHT Q UESTIONS : A CTIVE Q UESTION R EFORMULATION WITH R EIN -
Revised: A SK THE R IGHT Q UESTIONS : A CTIVE Q UESTION R EFORMULATION WITH R EIN -

Original: FORCEMENT L EARNING
Revised: FORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We frame Question Answering as a Reinforcement Learning task, an approach that we call Active Question Answering .
Revised: We frame Question Answering as a Reinforcement Learning task, an approach that we call Active Question Answering .

Original: We propose an agent that sits between the user and a black box question-answering system and which learns to reformulate questions to elicit the best possible answers.
Revised: We propose an agent that sits between the user and a black box question-answering system and which learns to reformulate questions to elicit the best possible answers.

Original: The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer.
Revised: The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer.

Original: The reformulation system is trained end-to-end to maximize answer quality using policy gradient.
Revised: The reformulation system is trained end-to-end to maximize answer quality using policy gradient.

Original: We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy! .
Revised: We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy! .

Original: Our agent improves F1 by 11.4% over a state-of-the-art base model that uses the original question/answer pairs.
Revised: Our agent improves F1 by 11.4% over a state-of-the-art base model that uses the original question/answer pairs.

Original: Based on a qualitative analysis of the language that the agent has learned while interacting with the question answering system, we propose that the agent has discovered basic information retrieval techniques such as term reweighting and stemming.
Revised: Based on a qualitative analysis of the language that the agent has learned while interacting with the question answering system, we propose that the agent has discovered basic information retrieval techniques such as term reweighting and stemming.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Web and social media have become primary sources of information.
Revised: Web and social media have become primary sources of information.

Original: Users’ expectations and information seeking activities co-evolve with the increasing sophistication of these resources.
Revised: Users’ expectations and information seeking activities co-evolve with the increasing sophistication of these resources.

Original: Beyond navigation, document retrieval, and simple factual question answering, users seek direct answers to complex and compositional questions.
Revised: Beyond navigation, document retrieval, and simple factual question answering, users seek direct answers to complex and compositional questions.

Original: Such search sessions may require multiple iterations, critical assessment and synthesis (Marchionini, 2006).
Revised: Such search sessions may require multiple iterations, critical assessment and synthesis (Marchionini, 2006).

Original: The productivity of natural language yields a myriad of ways to formulate a question.
Revised: The productivity of natural language yields a myriad of ways to formulate a question.

Original: In the face of complex information needs, humans overcome uncertainty by reformulating questions, issuing multiple searches, and aggregating responses.
Revised: In the face of complex information needs, humans overcome uncertainty by reformulating questions, issuing multiple searches, and aggregating responses.

Original: Inspired by humans’ ability to ask the right questions , we present an agent that learns to carry out this process for the user.
Revised: Inspired by humans’ ability to ask the right questions , we present an agent that learns to carry out this process for the user.

Original: The agent sits between the user and a backend QA system that we refer to as ‘the environment’.
Revised: The agent sits between the user and a backend QA system that we refer to as ‘the environment’.

Original: We call the agent AQA, as it implements an active question answering strategy.
Revised: We call the agent AQA, as it implements an active question answering strategy.

Original: AQA aims to maximize the chance of getting the correct answer by sending a reformulated question to the environment.
Revised: AQA aims to maximize the chance of getting the correct answer by sending a reformulated question to the environment.

Original: The agent seeks to ﬁnd the best answer by asking many questions and aggregating the returned evidence.
Revised: The agent seeks to ﬁnd the best answer by asking many questions and aggregating the returned evidence.

Original: The internals of the environment are not available to the agent, so it must learn to probe a black-box optimally using only question strings.
Revised: The internals of the environment are not available to the agent, so it must learn to probe a black-box optimally using only question strings.

Original: The key component of our solution, Figure 1, is a sequence-to-sequence model trained with reinforcement learning (RL) using a reward based on the answer returned by the environment.
Revised: The key component of our solution, Figure 1, is a sequence-to-sequence model trained with reinforcement learning (RL) using a reward based on the answer returned by the environment.

Original: The second component to AQA combines the evidence from interacting with the environment using a convolutional neural network to select an answer.
Revised: The second component to AQA combines the evidence from interacting with the environment using a convolutional neural network to select an answer.

Original: We evaluate on a dataset of Jeopardy!
Revised: We evaluate on a dataset of Jeopardy!

Original: questions, SearchQA (Dunn et al., 2017).
Revised: questions, SearchQA (Dunn et al., 2017).

Original: These questions are hard to answer by design because they use convoluted language, e.g., Travel doesn’t seem to be an issue for this sorcerer & onetime surgeon; astral projection & teleportation are no prob (answer:
Revised: These questions are hard to answer by design because they use convoluted language, e.g., Travel doesn’t seem to be an issue for this sorcerer & onetime surgeon; astral projection & teleportation are no prob (answer:

Original: Doctor Strange ).
Revised: Doctor Strange ).

Original: Thus SearchQA tests the ability of AQA to reformulate questions such that the QA system has the best chance of returning the correct answer.
Revised: Thus SearchQA tests the ability of AQA to reformulate questions such that the QA system has the best chance of returning the correct answer.

Original: AQA improves over the performance of a deep network built for QA, BiDAF (Seo et al., 2017a), which has produced state-of-the-art results on multiple tasks, by 11.4% absolute F1, a 32% relative F1 improvement.
Revised: AQA improves over the performance of a deep network built for QA, BiDAF (Seo et al., 2017a), which has produced state-of-the-art results on multiple tasks, by 11.4% absolute F1, a 32% relative F1 improvement.

Original: We also perform a qualitative analysis of the kind of reformulations generated by the AQA agent.
Revised: We also perform a qualitative analysis of the kind of reformulations generated by the AQA agent.

Original: We hypothesize that while optimizing its output, in the context of machine-machine communication with the QA system,
Revised: We hypothesize that while optimizing its output, in the context of machine-machine communication with the QA system,

Original: AQA discovers classic IR query operations such as term re-weighting, expansion and morphological
Revised: AQA discovers classic IR query operations such as term re-weighting, expansion and morphological

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: simpliﬁcation/stemming.
Revised: simpliﬁcation/stemming.

Original: A possible reason being that current machine comprehension tasks involve short textual snippets ranking, thus incentivizing relevance, more than deep language understanding.
Revised: A possible reason being that current machine comprehension tasks involve short textual snippets ranking, thus incentivizing relevance, more than deep language understanding.

Original:  A CTIVE Q UESTION A NSWERING M ODEL
Revised:  A CTIVE Q UESTION A NSWERING M ODEL

Original: Figure 1 shows the components of the Active Question Answering (AQA) agent-environment setup.
Revised: Figure 1 shows the components of the Active Question Answering (AQA) agent-environment setup.

Original: The AQA model interacts with a black-box environment.
Revised: The AQA model interacts with a black-box environment.

Original: AQA queries it with many versions of a question, and ﬁnally returning the best of the answers found.
Revised: AQA queries it with many versions of a question, and ﬁnally returning the best of the answers found.

Original: An episode starts with an original question q 0 .
Revised: An episode starts with an original question q 0 .

Original: The agent then generates a set of reformulations { q i } Ni =1 .
Revised: The agent then generates a set of reformulations { q i } Ni =1 .

Original: These are sent to the environment which returns answers { a i } Ni =1 .
Revised: These are sent to the environment which returns answers { a i } Ni =1 .

Original: The selection model then selects the best from these candidates.
Revised: The selection model then selects the best from these candidates.

Original: 2. Q UESTION -A NSWERING E NVIRONMENT
Revised: 2. Q UESTION -A NSWERING E NVIRONMENT

Original: For the QA environment we use a competitive neural question answering model, BiDirectional
Revised: For the QA environment we use a competitive neural question answering model, BiDirectional

Original: Attention Flow (BiDAF) (Seo et al., 2017a).
Revised: Attention Flow (BiDAF) (Seo et al., 2017a).

Original: BiDAF is an extractive QA system, it ﬁnds its answers in contiguous spans of a given document.
Revised: BiDAF is an extractive QA system, it ﬁnds its answers in contiguous spans of a given document.

Original: Given a question, the environment returns an answer and, during training, a reward.
Revised: Given a question, the environment returns an answer and, during training, a reward.

Original: The reward may be any quality metric for the returned answer, we use token-level F1 score.
Revised: The reward may be any quality metric for the returned answer, we use token-level F1 score.

Original: Note that the reward for each answer a i is computed against the original question q 0 .
Revised: Note that the reward for each answer a i is computed against the original question q 0 .

Original: We assume that the environment is opaque; the agent has no access to its parameters, activations or gradients.
Revised: We assume that the environment is opaque; the agent has no access to its parameters, activations or gradients.

Original: This setting enables us to also interact with other information sources.
Revised: This setting enables us to also interact with other information sources.

Original: However, without propagating gradients through the environment we lose information, feedback on the quality of the question reformulations is noisy, presenting a training challenge.
Revised: However, without propagating gradients through the environment we lose information, feedback on the quality of the question reformulations is noisy, presenting a training challenge.

Original: 2. R EFORMULATION M ODEL
Revised: 2. R EFORMULATION M ODEL

Original: The reformulator is a sequence-to-sequence model, as is popular for neural machine translation.
Revised: The reformulator is a sequence-to-sequence model, as is popular for neural machine translation.

Original: We build upon the implementation of Britz et al.
Revised: We build upon the implementation of Britz et al.

Original: The major departure from the standard
Revised: The major departure from the standard

Original: MT setting is that our model reformulates utterances in the same language.
Revised: MT setting is that our model reformulates utterances in the same language.

Original: Unlike in MT, there is little high quality training data available for monolingual paraphrasing.
Revised: Unlike in MT, there is little high quality training data available for monolingual paraphrasing.

Original: Effective training of highly-parametrized neural networks relies on an abundance of data.
Revised: Effective training of highly-parametrized neural networks relies on an abundance of data.

Original: We address this challenge by ﬁrst pre-training on a related task, multilingual translation, and then using signals produced during the interaction with the environment for adaptation.
Revised: We address this challenge by ﬁrst pre-training on a related task, multilingual translation, and then using signals produced during the interaction with the environment for adaptation.

Original: 2. A NSWER S ELECTION M ODEL
Revised: 2. A NSWER S ELECTION M ODEL

Original: During training, we have access to the reward for the answer returned for each reformulation q i .
Revised: During training, we have access to the reward for the answer returned for each reformulation q i .

Original: However, at test time we must predict the best answer a ∗ .
Revised: However, at test time we must predict the best answer a ∗ .

Original: The selection model selects the best answer from the set { a i } Ni =1 observed during the interaction by predicting the difference of the F1 score to the average F1 of all variants.
Revised: The selection model selects the best answer from the set { a i } Ni =1 observed during the interaction by predicting the difference of the F1 score to the average F1 of all variants.

Original: We use pre-trained embeddings for the tokens of query, rewrite, and answer.
Revised: We use pre-trained embeddings for the tokens of query, rewrite, and answer.

Original: For each, we add a 1D CNN followed by max-pooling.
Revised: For each, we add a 1D CNN followed by max-pooling.

Original: The three resulting vectors are then concatenated and passed through a feed-forward network which produces the output.
Revised: The three resulting vectors are then concatenated and passed through a feed-forward network which produces the output.

Original:  T RAINING 3. Q UESTION A NSWERING E NVIRONMENT
Revised:  T RAINING 3. Q UESTION A NSWERING E NVIRONMENT

Original: We train a model on the training set for the QA task at hand, see Section 4.4 for details.
Revised: We train a model on the training set for the QA task at hand, see Section 4.4 for details.

Original: Afterwards,
Revised: Afterwards,

Original: BiDAF becomes the black-box environment and its parameters are not updated further.
Revised: BiDAF becomes the black-box environment and its parameters are not updated further.

Original: In principle, we could train both the agent and the environment jointly to further improve performance.
Revised: In principle, we could train both the agent and the environment jointly to further improve performance.

Original: However, this is not our desired task: our aim is for the agent to learn to communicate using natural language with an environment over which is has no control.
Revised: However, this is not our desired task: our aim is for the agent to learn to communicate using natural language with an environment over which is has no control.

Original: 3. P OLICY G RADIENT T RAINING OF THE R EFORMULATION M ODEL
Revised: 3. P OLICY G RADIENT T RAINING OF THE R EFORMULATION M ODEL

Original: For a given question q 0 , we want to return the best possible answer a ∗ , maximizing a reward a ∗ = argmax a R ( a | q 0 ) .
Revised: For a given question q 0 , we want to return the best possible answer a ∗ , maximizing a reward a ∗ = argmax a R ( a | q 0 ) .

Original: Typically, R is the token level F1 score on the answer.
Revised: Typically, R is the token level F1 score on the answer.

Original: The answer a = f ( q ) is an unknown function of a question q , computed by the environment.
Revised: The answer a = f ( q ) is an unknown function of a question q , computed by the environment.

Original: The reward is computed with respect to the original question q 0 while the answer is provided for q .
Revised: The reward is computed with respect to the original question q 0 while the answer is provided for q .

Original: The question is generated according to a policy π θ where θ are the policy’s parameters q ∼ π θ ( · | q 0 ) .
Revised: The question is generated according to a policy π θ where θ are the policy’s parameters q ∼ π θ ( · | q 0 ) .

Original: The policy, in this case a sequence-to-sequence model, assigns a probability
Revised: The policy, in this case a sequence-to-sequence model, assigns a probability

Original: [Equation]
Revised: [Equation]

Original: 1) to any possible question q = w 1 , . . .
Revised: 1) to any possible question q = w 1 , . . .

Original: , w T , where T is the length of q with tokens w t ∈ V from a ﬁxed vocabulary V .
Revised: , w T , where T is the length of q with tokens w t ∈ V from a ﬁxed vocabulary V .

Original: The goal is to maximize the expected reward of the answer returned under the policy, E q ∼ π θ ( · | q 0 )
Revised: The goal is to maximize the expected reward of the answer returned under the policy, E q ∼ π θ ( · | q 0 )

Original: [ R ( f ( q ))] .
Revised: [ R ( f ( q ))] .

Original: We optimize the reward directly with respect to parameters of the policy using
Revised: We optimize the reward directly with respect to parameters of the policy using

Original: Policy Gradient methods (Sutton & Barto, 1998).
Revised: Policy Gradient methods (Sutton & Barto, 1998).

Original: The expected reward cannot be computed in closed form, so we compute an unbiased estimate with Monte Carlo sampling,
Revised: The expected reward cannot be computed in closed form, so we compute an unbiased estimate with Monte Carlo sampling,

Original: [Equation]
Revised: [Equation]

Original: To compute gradients for training we use REINFORCE (Williams & Peng, 1991),
Revised: To compute gradients for training we use REINFORCE (Williams & Peng, 1991),

Original: [Equation]
Revised: [Equation]

Original: This estimator is often found to have high variance, leading to unstable training (Greensmith et al., 2004).
Revised: This estimator is often found to have high variance, leading to unstable training (Greensmith et al., 2004).

Original: We reduce the variance by substracting the following baseline reward: B ( q 0 ) =
Revised: We reduce the variance by subtracting the following baseline reward: B ( q 0 ) =

Original: E q ∼ π θ ( · | q 0 )
Revised: E q ∼ π θ ( · | q 0 )

Original: [ R ( f ( q ))] .
Revised: [ R ( f ( q ))] .

Original: This expectation is also computed by sampling from the policy given q 0 .
Revised: This expectation is also computed by sampling from the policy given q 0 .

Original: We often observed collapse onto a sub-optimal deterministic policy.
Revised: We often observed collapse onto a sub-optimal deterministic policy.

Original: To address this we use entropy regularization
Revised: To address this we use entropy regularization

Original: [Equation]
Revised: [Equation]

Original: This ﬁnal objective is:
Revised: This ﬁnal objective is:

Original: [Equation]
Revised: [Equation]

Original: where λ is the regularization weight.
Revised: where λ is the regularization weight.

Original: 3. A NSWER S ELECTION
Revised: 3. A NSWER S ELECTION

Original: Unlike the reformulation policy, we train the answer With either beam search or sampling we can produce many rewrites of a single question from our reformulation system.
Revised: Unlike the reformulation policy, we train the answer With either beam search or sampling we can produce many rewrites of a single question from our reformulation system.

Original: We issue each rewrite to the QA environment, yielding a set of (query, rewrite, answer) tuples from which we need to pick the best instance.
Revised: We issue each rewrite to the QA environment, yielding a set of (query, rewrite, answer) tuples from which we need to pick the best instance.

Original: We train another neural network to pick the best answer from the candidates.
Revised: We train another neural network to pick the best answer from the candidates.

Original: We frame the task as binary classiﬁcation, distinguishing between above and below average performance.
Revised: We frame the task as binary classiﬁcation, distinguishing between above and below average performance.

Original: In training, we compute the F1 score of the answer for every instance.
Revised: In training, we compute the F1 score of the answer for every instance.

Original: If the rewrite produces an answer with an F1 score greater than the average score of the other rewrites the instance is assigned a positive label.
Revised: If the rewrite produces an answer with an F1 score greater than the average score of the other rewrites the instance is assigned a positive label.

Original: We ignore questions where all rewrites yield equally good/bad answers.
Revised: We ignore questions where all rewrites yield equally good/bad answers.

Original: We evaluated
Revised: We evaluated

Original: FFNNs, LSTMs and CNNs and found that the performance of all systems was comparable.
Revised: FFNNs, LSTMs and CNNs and found that the performance of all systems was comparable.

Original: Since the inputs are triples of variable length sequences the latter two allow us to incorporate the tokens directly without the need for feature engineering.
Revised: Since the inputs are triples of variable length sequences the latter two allow us to incorporate the tokens directly without the need for feature engineering.

Original: We choose a CNN for computational efﬁciency (cf. 2.3).
Revised: We choose a CNN for computational efﬁciency (cf. 2.3).

Original: 3. P RETRAINING OF THE R EFORMULATION M ODEL
Revised: 3. P RETRAINING OF THE R EFORMULATION M ODEL

Original: We pre-train the policy by building a paraphrasing Neural MT model that can translate English to
Revised: We pre-train the policy by building a paraphrasing Neural MT model that can translate English to

Original: English.
Revised: English.

Original: While parallel corpora are available for many language pairs, English-English corpora are scarce.
Revised: While parallel corpora are available for many language pairs, English-English corpora are scarce.

Original: We ﬁrst produce a multilingual translation system that translates between several languages (Johnson et al., 2016).
Revised: We ﬁrst produce a multilingual translation system that translates between several languages (Johnson et al., 2016).

Original: This allows us to use available bilingual corpora.
Revised: This allows us to use available bilingual corpora.

Original: Multilingual training requires nothing more than adding two special tokens to the data indicating the source and target languages.
Revised: Multilingual training requires nothing more than adding two special tokens to the data indicating the source and target languages.

Original: The encoder-decoder architecture of the translation model remains unchanged.
Revised: The encoder-decoder architecture of the translation model remains unchanged.

Original: As Johnson et al.
Revised: As Johnson et al.

Original: (2016) show, this model can be used for zero-shot translation , i.e. to translate between language pairs for which it has seen no training examples.
Revised: (2016) show, this model can be used for zero-shot translation , i.e. to translate between language pairs for which it has seen no training examples.

Original: For example after training
Revised: For example after training

Original: English-Spanish, English-French, French-English, and Spanish-English the model has learned a single encoder that encodes English, Spanish, and French and a decoder for the same three languages.
Revised: English-Spanish, English-French, French-English, and Spanish-English the model has learned a single encoder that encodes English, Spanish, and French and a decoder for the same three languages.

Original: Thus, we can use the same model for French-Spanish, Spanish-French and also English-English translation by adding the respective tokens to the source.
Revised: Thus, we can use the same model for French-Spanish, Spanish-French and also English-English translation by adding the respective tokens to the source.

Original: Johnson et al.
Revised: Johnson et al.

Original: (2016) note that zero-shot translation ususally performs worse than bridging, an approach that uses the model twice: ﬁrst, to translate into a pivot language, then into the target language.
Revised: (2016) note that zero-shot translation usually performs worse than bridging, an approach that uses the model twice: ﬁrst, to translate into a pivot language, then into the target language.

Original: However, the performance gap can be closed by running a few training steps for the desired language pair.
Revised: However, the performance gap can be closed by running a few training steps for the desired language pair.

Original: Thus, we ﬁrst train on multilingual data, then on a small corpus of monolingual data.
Revised: Thus, we ﬁrst train on multilingual data, then on a small corpus of monolingual data.

Original:  E XPERIMENTS 4. Q UESTION A NSWERING D ATA
Revised:  E XPERIMENTS 4. Q UESTION A NSWERING D ATA

Original: SearchQA (Dunn et al., 2017) is a recently-released dataset built starting from a set of Jeopardy!
Revised: SearchQA (Dunn et al., 2017) is a recently-released dataset built starting from a set of Jeopardy!

Original: clues.
Revised: clues.

Original: Clues are obfuscated queries such as This ‘Father of Our Country’ didn’t really chop down a cherry tree .
Revised: Clues are obfuscated queries such as This ‘Father of Our Country’ didn’t really chop down a cherry tree .

Original: Each clue is associated with the correct answer, e.g. George Washington , and a list of snippets from Google’s top search results.
Revised: Each clue is associated with the correct answer, e.g. George Washington , and a list of snippets from Google’s top search results.

Original: SearchQA contains over 140k question/answer pairs and 6.9M snippets.
Revised: SearchQA contains over 140k question/answer pairs and 6.9M snippets.

Original: We train our model on the pre-deﬁned training split, perform model selection and tuning on the validation split and report results on the validation and test splits.
Revised: We train our model on the pre-deﬁned training split, perform model selection and tuning on the validation split and report results on the validation and test splits.

Original: The training, validation and test sets contain 99,820, 13,393 and 27,248 examples, respectively.
Revised: The training, validation and test sets contain 99,820, 13,393 and 27,248 examples, respectively.

Original: 4. Q UESTION R EFORMULATOR T RAINING
Revised: 4. Q UESTION R EFORMULATOR T RAINING

Original: For the pre-training of the reformulator we use the multilingual United Nations Parallel Corpus v1.0 (Ziemski et al., 2016).
Revised: For the pre-training of the reformulator we use the multilingual United Nations Parallel Corpus v1.0 (Ziemski et al., 2016).

Original: This dataset contains 11.4M sentences which are fully aligned across six
Revised: This dataset contains 11.4M sentences which are fully aligned across six

Original: UN languages: Arabic, English, Spanish, French, Russian, and Chinese.
Revised: UN languages: Arabic, English, Spanish, French, Russian, and Chinese.

Original: From all bilingual pairs we produce a multilingual training corpus of 30 language pairs.
Revised: From all bilingual pairs we produce a multilingual training corpus of 30 language pairs.

Original: This yields 340M training examples which we use to train the zero-shot neural MT system (Johnson et al., 2016).
Revised: This yields 340M training examples which we use to train the zero-shot neural MT system (Johnson et al., 2016).

Original: We tokenize our data using 16k sentence pieces.
Revised: We tokenize our data using 16k sentence pieces.

Original: Following Britz et al.
Revised: Following Britz et al.

Original: we use a bidirectional LSTM as encoder and a 4-layer stacked LSTM with attention as decoder.
Revised: we use a bidirectional LSTM as encoder and a 4-layer stacked LSTM with attention as decoder.

Original: The model converged after training on 400M instances using the Adam optimizer with a learning rate of 0.001 and batch size of 128.
Revised: The model converged after training on 400M instances using the Adam optimizer with a learning rate of 0.001 and batch size of 128.

Original: The model trained as described above has poor quality.
Revised: The model trained as described above has poor quality.

Original: For example, for the question What month, day and year did Super Bowl 50 take place? , the top rewrite is What month and year goes back to the morning and year? .
Revised: For example, for the question What month, day and year did Super Bowl 50 take place? , the top rewrite is What month and year goes back to the morning and year? .

Original: To improve quality, we resume training on a smaller monolingual dataset, extracted from the Paralex database of question paraphrases (Fader et al., 2013).
Revised: To improve quality, we resume training on a smaller monolingual dataset, extracted from the Paralex database of question paraphrases (Fader et al., 2013).

Original: Unfortunately, this data contains many noisy pairs.
Revised: Unfortunately, this data contains many noisy pairs.

Original: We ﬁlter many of these pairs out by keeping only those where the Jaccard coefﬁcient between the sets of source and target terms is above 0.5.
Revised: We ﬁlter many of these pairs out by keeping only those where the Jaccard coefﬁcient between the sets of source and target terms is above 0.5.

Original: Further, since the number of paraphrases for each question can vary signiﬁcantly, we keep at most 4 paraphrases for each question.
Revised: Further, since the number of paraphrases for each question can vary signiﬁcantly, we keep at most 4 paraphrases for each question.

Original: After processing, we are left with about 1.5M pairs out of the original 35M.
Revised: After processing, we are left with about 1.5M pairs out of the original 35M.

Original: The reﬁned model has visibly better quality than the zero-shot one; for the example question above it generates What year did superbowl take place? .
Revised: The reﬁned model has visibly better quality than the zero-shot one; for the example question above it generates What year did superbowl take place? .

Original: We also tried training on the monolingual pairs alone.
Revised: We also tried training on the monolingual pairs alone.

Original: As in (Johnson et al., 2016), the quality was in between the multilingual and reﬁned models.
Revised: As in (Johnson et al., 2016), the quality was in between the multilingual and reﬁned models.

Original: After pre-training the reformulator, we switch the optimizer from Adam to SGD and train for 100 k
Revised: After pre-training the reformulator, we switch the optimizer from Adam to SGD and train for 100 k

Original: RL steps of batch size 64 with a low learning rate of 0 .
Revised: RL steps of batch size 64 with a low learning rate of 0 .

Original: We use an entropy regularization weight of λ = 0 . 001 .
Revised: We use an entropy regularization weight of λ = 0 . 001 .

Original: For a stopping criterion, we monitor the reward from the best single rewrite, generated via greedy decoding, on the validation set.
Revised: For a stopping criterion, we monitor the reward from the best single rewrite, generated via greedy decoding, on the validation set.

Original: In contrast to our initial training which we ran on GPUs, this training phase is dominated by latency of the QA system and we run inference and updates on
Revised: In contrast to our initial training which we ran on GPUs, this training phase is dominated by latency of the QA system and we run inference and updates on

Original: CPU and the BiDAF environment on GPU.
Revised: CPU and the BiDAF environment on GPU.

Original: 4. T RAINING THE A NSWER S ELECTOR
Revised: 4. T RAINING THE A NSWER S ELECTOR

Original: For the selction model we use supervised learning: ﬁrst, we train the reformulator, then we generate
Revised: For the selction model we use supervised learning: ﬁrst, we train the reformulator, then we generate

Original: N = 20 rewrites for each question in the SearchQA training and validation sets.
Revised: N = 20 rewrites for each question in the SearchQA training and validation sets.

Original: After sending these to the environment we have about 2M (question, rewrite, answer) triples.
Revised: After sending these to the environment we have about 2M (question, rewrite, answer) triples.

Original: We remove queries where all rewrites yield identical rewards, which removes about half of the training data.
Revised: We remove queries where all rewrites yield identical rewards, which removes about half of the training data.

Original: We use pre-trained 100-dimensional embeddings (Pennington et al., 2014) for the tokens.
Revised: We use pre-trained 100-dimensional embeddings (Pennington et al., 2014) for the tokens.

Original: Our CNN-based selection model encodes the three strings into 100 dimensional vectors using a 1D CNN with kernel width 3 and output dimension 100 over the embedded tokens, followed by max-pooling.
Revised: Our CNN-based selection model encodes the three strings into 100 dimensional vectors using a 1D CNN with kernel width 3 and output dimension 100 over the embedded tokens, followed by max-pooling.

Original: The vectors are then concatenated and passed through a feed-forward network which produces the binary output, indicating whether the triple performs below of above average, relative to the other reformulations and respective answers.
Revised: The vectors are then concatenated and passed through a feed-forward network which produces the binary output, indicating whether the triple performs below of above average, relative to the other reformulations and respective answers.

Original: We use the training portion of the SearchQA data thrice, ﬁrst for the initial training of the BiDAF model, then for the reinforcement-learning based tuning of the reformulator, and ﬁnally for training of the selector.
Revised: We use the training portion of the SearchQA data thrice, ﬁrst for the initial training of the BiDAF model, then for the reinforcement-learning based tuning of the reformulator, and ﬁnally for training of the selector.

Original: We carefully monitored that this didn’t cause severe overﬁtting.
Revised: We carefully monitored that this didn’t cause severe overﬁtting.

Original: BiDAF alone has a generalization gap between the training and validation set errors of 3.4 F1.
Revised: BiDAF alone has a generalization gap between the training and validation set errors of 3.4 F1.

Original: This gap remains virtually identical after training the rewriter.
Revised: This gap remains virtually identical after training the rewriter.

Original: After training the CNN, AQA-Full has a slightly larger gap of 3.
Revised: After training the CNN, AQA-Full has a slightly larger gap of 3.

Original: F1.
Revised: F1.

Original: We conclude that training AQA on BiDAF’s training set causes very little additional overﬁtting.
Revised: We conclude that training AQA on BiDAF’s training set causes very little additional overﬁtting.

Original: We use the test set only for evaluation of the ﬁnal model.
Revised: We use the test set only for evaluation of the ﬁnal model.

Original: 4. B ASELINES AND B ENCHMARKS
Revised: 4. B ASELINES AND B ENCHMARKS

Original: As a baseline, we repeat the results reported for a deep learning system developed for SearchQA (Dunn et al., 2017).
Revised: As a baseline, we repeat the results reported for a deep learning system developed for SearchQA (Dunn et al., 2017).

Original: This is a modiﬁed pointer network, called Attention Sum Reader.
Revised: This is a modiﬁed pointer network, called Attention Sum Reader.

Original: The BiDAF environment can be used without the reformulator to answer the original question.
Revised: The BiDAF environment can be used without the reformulator to answer the original question.

Original: This corresponds to the raw performance of BiDAF, and is our second baseline.
Revised: This corresponds to the raw performance of BiDAF, and is our second baseline.

Original: We train BiDAF directly on the SearchQA training data.
Revised: We train BiDAF directly on the SearchQA training data.

Original: In the SearchQA task, the answers are augmented with several snippets (50 on average) returned by a Google Search for the question.
Revised: In the SearchQA task, the answers are augmented with several snippets (50 on average) returned by a Google Search for the question.

Original: We join snippets to form the context from which BiDAF selects answer spans.
Revised: We join snippets to form the context from which BiDAF selects answer spans.

Original: For performance reasons, we limit the context to the top 10 snippets.
Revised: For performance reasons, we limit the context to the top 10 snippets.

Original: This corresponds to ﬁnding the answer on the ﬁrst page of Google results.
Revised: This corresponds to ﬁnding the answer on the ﬁrst page of Google results.

Original: The results are only mildly affected by this limitation, for 10% of the questions there is no answer in this shorter context.
Revised: The results are only mildly affected by this limitation, for 10% of the questions there is no answer in this shorter context.

Original: These datapoints are all counted as losses.
Revised: These datapoints are all counted as losses.

Original: We trained with the Adam optimizer for 4500 steps, using learning rate 0.001, batch size 60.
Revised: We trained with the Adam optimizer for 4500 steps, using learning rate 0.001, batch size 60.

Original: We present two benchmark performance levels.
Revised: We present two benchmark performance levels.

Original: The ﬁrst is human performance reported in (Dunn et al., 2017) based on a sample of the test set.
Revised: The ﬁrst is human performance reported in (Dunn et al., 2017) based on a sample of the test set.

Original: The second is ‘Oracle Ranking’, which provides an upper bound on the improvement that can be made by selection.
Revised: The second is ‘Oracle Ranking’, which provides an upper bound on the improvement that can be made by selection.

Original: For this, we replace the selection model with an oracle that picks the answer with highest F1 score from the set of those returned for the reformulations.
Revised: For this, we replace the selection model with an oracle that picks the answer with highest F1 score from the set of those returned for the reformulations.

Original: [Table]
Revised: [Table]

Original: 4. AQA V ARIANTS
Revised: 4. AQA V ARIANTS

Original: We evaluate several variants of AQA.
Revised: We evaluate several variants of AQA.

Original: For each query q in the evaluation we generate a list of reformulations q i , for i = 1 . . .
Revised: For each query q in the evaluation we generate a list of reformulations q i , for i = 1 . . .

Original: N , from the AQA reformulator trained as described in Section 3.
Revised: N , from the AQA reformulator trained as described in Section 3.

Original: We set N = 20 in these experiments.
Revised: We set N = 20 in these experiments.

Original: AQA Top Hyp.
Revised: AQA Top Hyp.

Original: First, we omit selection, and just select the top hypothesis generated by the sequence model, q 1 .
Revised: First, we omit selection, and just select the top hypothesis generated by the sequence model, q 1 .

Original: AQA Voting
Revised: AQA Voting

Original: In addition to the most likely answer span, BiDAF also reports a model score, which we use for a heuristic weighted voting scheme to implement deterministic selection.
Revised: In addition to the most likely answer span, BiDAF also reports a model score, which we use for a heuristic weighted voting scheme to implement deterministic selection.

Original: Let a be the answer returned by BiDAF for query q , with associated score s ( a ) .
Revised: Let a be the answer returned by BiDAF for query q , with associated score s ( a ) .

Original: We pick the answer according to argmax a (cid:80) a (cid:48)
Revised: We pick the answer according to argmax a (cid:80) a (cid:48)

Original: = a s ( a (cid:48) ) .
Revised: = a s ( a (cid:48) ) .

Original: AQA Max Conf.
Revised: AQA Max Conf.

Original: We implement a second heuristic that selects the answer with the single highest
Revised: We implement a second heuristic that selects the answer with the single highest

Original: BiDAF score across question reformulations.
Revised: BiDAF score across question reformulations.

Original: AQA Full Finally, we present the complete system with the learned CNN model described in
Revised: AQA Full Finally, we present the complete system with the learned CNN model described in

Original: Section 2.
Revised: Section 2.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Table 1 shows the results.
Revised: Table 1 shows the results.

Original: We report exact match and F1 metrics, computed on token level between the predicted answer and the gold answer.
Revised: We report exact match and F1 metrics, computed on token level between the predicted answer and the gold answer.

Original: We present results on the full validation and test sets (referred to as n -gram in (Dunn et al., 2017)).
Revised: We present results on the full validation and test sets (referred to as n -gram in (Dunn et al., 2017)).

Original: This includes questions that have both unigram and longer answers.
Revised: This includes questions that have both unigram and longer answers.

Original: SearchQA appears to be harder than other recent QA tasks such as SQuAD (Rajpurkar et al., 2016), for both machines and humans.
Revised: SearchQA appears to be harder than other recent QA tasks such as SQuAD (Rajpurkar et al., 2016), for both machines and humans.

Original: BiDAF’s performance drops by 40 F1 points on SearchQA compared to SQuAD.
Revised: BiDAF’s performance drops by 40 F1 points on SearchQA compared to SQuAD.

Original: However, BiDAF is still competitive on SeachQA, improving over the Attention Sum
Revised: However, BiDAF is still competitive on SeachQA, improving over the Attention Sum

Original: Reader network by 13.7 F1 points.
Revised: Reader network by 13.7 F1 points.

Original: Using the top hypothesis alone already yields an improvement of 2.2 F1 on test.
Revised: Using the top hypothesis alone already yields an improvement of 2.2 F1 on test.

Original: This improvement without answer selection demonstrates that the reformulator is able to produce questions more easily answered by the environment.
Revised: This improvement without answer selection demonstrates that the reformulator is able to produce questions more easily answered by the environment.

Original: Heuristic selection via both Voting and Max Conf yield a further performance boost.
Revised: Heuristic selection via both Voting and Max Conf yield a further performance boost.

Original: Both heuristics draw upon the intuition that when BiDAF is conﬁdent in its answer it is more likely to be correct, and that multiple instances of the same answer provide positive evidence (for Max Conf, the max operation implicitly rewards having an answer scored with respect to multiple questions).
Revised: Both heuristics draw upon the intuition that when BiDAF is conﬁdent in its answer it is more likely to be correct, and that multiple instances of the same answer provide positive evidence (for Max Conf, the max operation implicitly rewards having an answer scored with respect to multiple questions).

Original: Finally, a trained selection function improves performance further, yielding an absolute increase of 11.4 F1 points (32% relative) over BiDAF with the original questions.
Revised: Finally, a trained selection function improves performance further, yielding an absolute increase of 11.4 F1 points (32% relative) over BiDAF with the original questions.

Original: In terms of exact match score this more than closes half the gap between BiDAF and human performance.
Revised: In terms of exact match score this more than closes half the gap between BiDAF and human performance.

Original: [Table]
Revised: [Table]

Original: 4. Q UALITATIVE ANALYSIS
Revised: 4. Q UALITATIVE ANALYSIS

Original: We analyze the question reformulations on the dev partition of SearchQA.
Revised: We analyze the question reformulations on the dev partition of SearchQA.

Original: It is important to bear in mind that these Jeopardy!
Revised: It is important to bear in mind that these Jeopardy!

Original: clues have been preprocessed by lower-casing and stop word removal.
Revised: clues have been preprocessed by lower-casing and stop word removal.

Original: The effect is that the pre-processed clues are closer to keyword-based search queries than grammatical questions; e.g., Gandhi was deeply inﬂuenced by this count who wrote "War and Peace" is simpliﬁed to gandhi deeply inﬂuenced count wrote war peace .
Revised: The effect is that the pre-processed clues are closer to keyword-based search queries than grammatical questions; e.g., Gandhi was deeply inﬂuenced by this count who wrote "War and Peace" is simpliﬁed to gandhi deeply inﬂuenced count wrote war peace .

Original: Table 2 summarizes the results of this analysis.
Revised: Table 2 summarizes the results of this analysis.

Original: The questions contain 9.6 words on average.
Revised: The questions contain 9.6 words on average.

Original: They contain few repeated terms, computed as the mean term frequency (TF) per question.
Revised: They contain few repeated terms, computed as the mean term frequency (TF) per question.

Original: We also compute the mean and median document frequency (DF) per query, where a document is the context from which the answer is selected.
Revised: We also compute the mean and median document frequency (DF) per query, where a document is the context from which the answer is selected.

Original: This gives us a measure of how informative the question terms are.
Revised: This gives us a measure of how informative the question terms are.

Original: We ﬁrst consider the top hypothesis generated by the pre-trained (before RL) NMT reformulation system (Base-NMT).
Revised: We ﬁrst consider the top hypothesis generated by the pre-trained (before RL) NMT reformulation system (Base-NMT).

Original: The Base-NMT rewrites differ greatly from their sources.
Revised: The Base-NMT rewrites differ greatly from their sources.

Original: They are shorter, 6.
Revised: They are shorter, 6.

Original: words on average, have slightly fewer repeated terms (1.01 vs. 1.03) and are often syntactically wellformed questions.
Revised: words on average, have slightly fewer repeated terms (1.01 vs. 1.03) and are often syntactically wellformed questions.

Original: E.g., the example above becomes Who inﬂuenced count wrote war? .
Revised: E.g., the example above becomes Who inﬂuenced count wrote war? .

Original: 5 Base-NMT also recovers missing function words.
Revised: 5 Base-NMT also recovers missing function words.

Original: We veriﬁed the improved ﬂuency also using a large language model and found that the Base-NMT rewrites are 50% more likely than the original ones.
Revised: We veriﬁed the improved ﬂuency also using a large language model and found that the Base-NMT rewrites are 50% more likely than the original ones.

Original: These rewrites also involve fewer content terms, DF decreases.
Revised: These rewrites also involve fewer content terms, DF decreases.

Original: This is likely due to the domain mismatch between SearchQA and the NMT training corpus.
Revised: This is likely due to the domain mismatch between SearchQA and the NMT training corpus.

Original: We next consider the top hypothesis generated by AQA’s reformulator (AQA-R), after policy gradient training.
Revised: We next consider the top hypothesis generated by AQA’s reformulator (AQA-R), after policy gradient training.

Original: These are the rewrites evaluated as AQA Top Hyp.
Revised: These are the rewrites evaluated as AQA Top Hyp.

Original: in Table 1.
Revised: in Table 1.

Original: We use these rewrites instead
Revised: We use these rewrites instead

Original: AQA-full to avoid confounding effects due to the answer selection.
Revised: AQA-full to avoid confounding effects due to the answer selection.

Original: The rewrites look very different from both the Base-NMT and the SearchQA ones.
Revised: The rewrites look very different from both the Base-NMT and the SearchQA ones.

Original: For the example, above AQA-R’s top hypothesis is
Revised: For the example, above AQA-R’s top hypothesis is

Original: What is name gandhi gandhi inﬂuence wrote peace peace? .
Revised: What is name gandhi gandhi inﬂuence wrote peace peace? .

Original: Somewhat mysteriously, 99.8% start with the: preﬁx What is name .
Revised: Somewhat mysteriously, 99.8% start with the: preﬁx What is name .

Original: This is puzzling as it happens only for 9 Base-NMT rewrites, and never in the original SearchQA questions.
Revised: This is puzzling as it happens only for 9 Base-NMT rewrites, and never in the original SearchQA questions.

Original: We speculate it might be related to the fact that virtually all answers involve names of named entities (Micronesia) or generic concepts (pizza).
Revised: We speculate it might be related to the fact that virtually all answers involve names of named entities (Micronesia) or generic concepts (pizza).

Original: AQA-R’s rewrites are visibly less ﬂuent than both the originals and the Base-MT ones.
Revised: AQA-R’s rewrites are visibly less ﬂuent than both the originals and the Base-MT ones.

Original: However, they have more repeated terms (1.2 average TF), are signiﬁcantly longer (11.9) than the Base-NMT initialization and contain more context terms (higher DF).
Revised: However, they have more repeated terms (1.2 average TF), are signiﬁcantly longer (11.9) than the Base-NMT initialization and contain more context terms (higher DF).

Original: Finally, AQA-R’s reformulations contain morphological variants in 12.5% of cases.
Revised: Finally, AQA-R’s reformulations contain morphological variants in 12.5% of cases.

Original: The number of questions that contain multiple tokens with the same stem doubles from SearchQA to AQA-R.
Revised: The number of questions that contain multiple tokens with the same stem doubles from SearchQA to AQA-R.

Original: Singular forms are preferred over plurals.
Revised: Singular forms are preferred over plurals.

Original: Morphological simpliﬁcation is useful because it increases the chance that a word variants match the same embeddings.
Revised: Morphological simpliﬁcation is useful because it increases the chance that a word variants match the same embeddings.

Original: Recently, Lewis et al.
Revised: Recently, Lewis et al.

Original: (2017) trained chatbots that negotiate via language utterances in order to complete a task.
Revised: (2017) trained chatbots that negotiate via language utterances in order to complete a task.

Original: They report that the agents’ language can diverge from human language if there is no incentive to do so in the reward function.
Revised: They report that the agents’ language can diverge from human language if there is no incentive to do so in the reward function.

Original: Our ﬁndings seem related.
Revised: Our ﬁndings seem related.

Original: The fact that the questions reformulated by AQA bare little resemblance to natural language does not depend on the keyword-like inputs, since Base-NMT is capable of producing ﬂuent questions from the same input.
Revised: The fact that the questions reformulated by AQA bare little resemblance to natural language does not depend on the keyword-like inputs, since Base-NMT is capable of producing ﬂuent questions from the same input.

Original: We hypothesize instead that there is no incentive for the model to use human language because of the nature of the task.
Revised: We hypothesize instead that there is no incentive for the model to use human language because of the nature of the task.

Original: AQA learns to ask BiDAF questions by optimizing a language that increases the likelihood of
Revised: AQA learns to ask BiDAF questions by optimizing a language that increases the likelihood of

Original: BiDAF extracting the right answer.
Revised: BiDAF extracting the right answer.

Original: Jia & Liang (2017) argue that reading comprehension systems are not capable of signiﬁcant language understanding and fail easily in adversarial settings.
Revised: Jia & Liang (2017) argue that reading comprehension systems are not capable of signiﬁcant language understanding and fail easily in adversarial settings.

Original: We believe that current deep QA systems are primarily sophisticated ranking systems trained to sort snippets of text from the context.
Revised: We believe that current deep QA systems are primarily sophisticated ranking systems trained to sort snippets of text from the context.

Original: As such, they resemble document retrieval systems which incentivizes the discovery (by AQA) of IR techniques that have been successful for decades: term re-weighting, query expansion and stemming/morphological simpliﬁcation.
Revised: As such, they resemble document retrieval systems which incentivizes the discovery (by AQA) of IR techniques that have been successful for decades: term re-weighting, query expansion and stemming/morphological simpliﬁcation.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Query operations involving reformulation, expansion and term reweighting are classic topics in information retrieval (Baeza-Yates & Ribeiro-Neto, 1999, chap. 5).
Revised: Query operations involving reformulation, expansion and term reweighting are classic topics in information retrieval (Baeza-Yates & Ribeiro-Neto, 1999, chap. 5).

Original: In NLP literature, Lin & Pantel (2001) proposed to learn patterns of question variants by comparing dependency parsing trees.
Revised: In NLP literature, Lin & Pantel (2001) proposed to learn patterns of question variants by comparing dependency parsing trees.

Original: Later,
Revised: Later,

Original: Duboue & Chu-Carroll (2006) showed that MT-based paraphrases can be useful in principle by providing signiﬁcant headroom in oracle-based estimations of QA performance.
Revised: Duboue & Chu-Carroll (2006) showed that MT-based paraphrases can be useful in principle by providing signiﬁcant headroom in oracle-based estimations of QA performance.

Original: More recently,
Revised: More recently,

Original: Berant & Liang (2014) used paraphrasing to augment the training of a semantic parser by expanding through the paraphrases as a latent representation.
Revised: Berant & Liang (2014) used paraphrasing to augment the training of a semantic parser by expanding through the paraphrases as a latent representation.

Original: Bilingual corpora and MT have been used to generate paraphrases by pivoting through a second language.
Revised: Bilingual corpora and MT have been used to generate paraphrases by pivoting through a second language.

Original: Recent work uses neural translation models and multiple pivots (Mallinson et al., 2017).
Revised: Recent work uses neural translation models and multiple pivots (Mallinson et al., 2017).

Original: In contrast, our approach does not use pivoting and is, to our knowledge, the ﬁrst direct neural paraphrasing system.
Revised: In contrast, our approach does not use pivoting and is, to our knowledge, the ﬁrst direct neural paraphrasing system.

Original: Riezler et al.
Revised: Riezler et al.

Original: (2007) propose phrase-based paraphrasing for query expansion.
Revised: (2007) propose phrase-based paraphrasing for query expansion.

Original: In contrast with this line of work our goal is to generate full question reformulations while optimizing directly the end-to-end target quality metrics.
Revised: In contrast with this line of work our goal is to generate full question reformulations while optimizing directly the end-to-end target quality metrics.

Original: Reinforcement learning is gaining traction in natural language understanding across many problems.
Revised: Reinforcement learning is gaining traction in natural language understanding across many problems.

Original: For example, Narasimhan et al.
Revised: For example, Narasimhan et al.

Original: (2015) use RL to learn control policies for multi-user dungeon games where the state of the game is summarized by a textual description, and Li et al.
Revised: (2015) use RL to learn control policies for multi-user dungeon games where the state of the game is summarized by a textual description, and Li et al.

Original: (2017) use RL for dialogue generation.
Revised: (2017) use RL for dialogue generation.

Original: Policy gradient methods have been investigated recently for MT and other sequence-to-sequence problems.
Revised: Policy gradient methods have been investigated recently for MT and other sequence-to-sequence problems.

Original: They alleviate limitations inherent to the word-level optimization of the cross-entropy loss, allowing use of sequence-level reward functions, like BLEU.
Revised: They alleviate limitations inherent to the word-level optimization of the cross-entropy loss, allowing use of sequence-level reward functions, like BLEU.

Original: Reward functions based on language models and reconstruction errors are used to bootstrap MT with fewer resources (Xia et al., 2016).
Revised: Reward functions based on language models and reconstruction errors are used to bootstrap MT with fewer resources (Xia et al., 2016).

Original: RL training can also prevent exposure bias ; an inconsistency between training and inference time stemming from the fact that the model never sees its own mistakes during training (Ranzato et al., 2015).
Revised: RL training can also prevent exposure bias ; an inconsistency between training and inference time stemming from the fact that the model never sees its own mistakes during training (Ranzato et al., 2015).

Original: We also use policy gradient to optimize our agent, however, we use end-to-end question answering quality as the reward.
Revised: We also use policy gradient to optimize our agent, however, we use end-to-end question answering quality as the reward.

Original: Uses of policy gradient for QA include Liang et al.
Revised: Uses of policy gradient for QA include Liang et al.

Original: (2017), who train a semantic parser to query a knowledge base, and Seo et al.
Revised: (2017), who train a semantic parser to query a knowledge base, and Seo et al.

Original: (2017b) who propose query reduction networks that transform a query to answer questions that involve multi-hop common sense reasoning.
Revised: (2017b) who propose query reduction networks that transform a query to answer questions that involve multi-hop common sense reasoning.

Original: The work of Nogueira &
Revised: The work of Nogueira &

Original: Cho (2016) is most related to ours.
Revised: Cho (2016) is most related to ours.

Original: They identify a document containing an answer to a question by following links on a graph.
Revised: They identify a document containing an answer to a question by following links on a graph.

Original: Evaluating on a set of questions from the game Jeopardy!
Revised: Evaluating on a set of questions from the game Jeopardy!

Original: , they learn to walk the Wikipedia graph until they reach the predicted answer.
Revised: , they learn to walk the Wikipedia graph until they reach the predicted answer.

Original: In a follow-up Nogueira & Cho (2017) improve document retrieval with an approach inspired by relevance feedback in combination with RL.
Revised: In a follow-up Nogueira & Cho (2017) improve document retrieval with an approach inspired by relevance feedback in combination with RL.

Original: They reformulate a query by adding terms from documents retrieved from a search engine for the original query.
Revised: They reformulate a query by adding terms from documents retrieved from a search engine for the original query.

Original: Our work differs in that we generate full reformulations via sequence-to-sequence modeling rather than adding single terms, and we target question-answering, rather than document retrieval.
Revised: Our work differs in that we generate full reformulations via sequence-to-sequence modeling rather than adding single terms, and we target question-answering, rather than document retrieval.

Original: Active QA is also related to recent research on fact-checking: Wu et al.
Revised: Active QA is also related to recent research on fact-checking: Wu et al.

Original: (2017) propose to perturb database queries in order to estimate the support of quantitative claims.
Revised: (2017) propose to perturb database queries in order to estimate the support of quantitative claims.

Original: In Active QA questions are perturbed with a similar purpose, although directly at the surface natural language form.
Revised: In Active QA questions are perturbed with a similar purpose, although directly at the surface natural language form.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose active question answering (AQA), a framework to improve answering by systematically perturbing input questions.
Revised: We propose active question answering (AQA), a framework to improve answering by systematically perturbing input questions.

Original: We investigated a ﬁrst system of this kind that has three components: a question reformulator, a black box QA system, and a candidate answer selection model.
Revised: We investigated a ﬁrst system of this kind that has three components: a question reformulator, a black box QA system, and a candidate answer selection model.

Original: The reformulator and selector form a trainable agent that tries to elicit the best answers from the QA system.
Revised: The reformulator and selector form a trainable agent that tries to elicit the best answers from the QA system.

Original: Importantly, the agent may only query the environment with question strings.
Revised: Importantly, the agent may only query the environment with question strings.

Original: Experimental results prove that the approach is highly effective.
Revised: Experimental results prove that the approach is highly effective.

Original: We improve a sophisticated Deep QA system by 11.4% absolute F1, 32% relative F1, on a difﬁcult dataset of long, semantically complex, questions.
Revised: We improve a sophisticated Deep QA system by 11.4% absolute F1, 32% relative F1, on a difﬁcult dataset of long, semantically complex, questions.

Original: For future work we will investigate the sequential, iterative aspect of QA, and frame the problem as an end-to-end RL task, thus, closing the loop between the reformulator and the selector.
Revised: For future work we will investigate the sequential, iterative aspect of QA, and frame the problem as an end-to-end RL task, thus, closing the loop between the reformulator and the selector.