E VIDENCE A GGREGATION FOR A NSWER R E -R ANKING IN O PEN -D OMAIN Q UESTION A NSWERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Very recently, it comes to be a popular approach for answering open-domain questions by ﬁrst searching question-related passages, then applying reading comprehension models to extract answers.
Existing works usually extract answers from single passages independently, thus not fully make use of the multiple searched passages, especially for the some questions requiring several evidences, which can appear in different passages, to be answered.
The above observations raise the problem of evidence aggregation from multiple passages.
In this paper, we deal with this problem as answer re-ranking.
Speciﬁcally, based on the answer candidates generated from the existing state-of-the-art QA model, we propose two different re-ranking methods, strength-based and coverage-based re-rankers, which make use of the aggregated evidences from different passages to help entail the ground-truth answer for the question.
Our model achieved state-of-the-arts on three public open-domain QA datasets, Quasar-T, SearchQA and the open-domain version of TriviaQA, with about 8% improvement on the former two datasets.
I NTRODUCTION
Open-domain question answering (QA) aims to answer questions from a broad range of domains by effectively retrieving evidence from one or more huge open-domain knowledge sources and by comprehending the evidence to infer the correct answers.
Such resources can be Wikipedia (Chen et al., 2017), the whole web (Ferrucci et al., 2010), structured knowledge bases (Berant et al., 2013;
Yu et al., 2017) or combinations of the above (Baudi ˇ s & ˇ Sediv ` y, 2015).
Recent work on open-domain QA focuses on using unstructured text from the web as sources, with the help from machine comprehension models (Chen et al., 2017; Dhingra et al., 2017b; Wang et al., 2017).
These studies adopt a two-step-process: an information retrieval (IR) model to coarsely select relevant passages to a question, followed by a reading comprehension (RC) model (Wang &
Jiang, 2017; Seo et al., 2017; Chen et al., 2017) to infer an answer from the passages.
These works use each passage independently to train RC model, thus fail to fully explore the beneﬁts of using multiple passages.
Information from different passages can provide enhanced or complementary evidence.
We propose to improve open-domain QA by explicitly aggregating evidence from multiple passages.
Our method is inspired by two signiﬁcant observations from previous open-domain QA result analysis:
• Firstly, the correct answer is often suggested by more passages repeatedly compared to the incorrect ones.
For example, in Figure 1(a), the correct answer “ danny boy ” has more passages providing evidence relevant to the question compared to the incorrect one.
This observation can be seen as multiple passages collaboratively enhancing the evidence for the correct answer.
Secondly, sometimes the question covers multiple answer aspects, which spreads over multiple passages.
In order to infer the correct answer, one has to ﬁnd ways to aggregate those multiple passages in an effective yet sensible way to try and cover all aspects.
In Figure 1(b), for example, the correct answer “ Galileo Galilei ” at the bottom has passages P1, “ Galileo was a physicist ... ” and P2, “ Galileo discovered the ﬁrst 4 moons of Jupiter ”, mentioning two pieces of evidence to match the question. In this case, the aggregation of these two pieces of evidence from the two passages can help entail the ground-truth answer “ Galileo Galilei ” for the question. In comparison, the incorrect answer “ Isaac Newton ” has passages providing partial evidence on only
Question1: What is the more popular name for the londonderry air ？
A1: tune from county
P1 :the best known title for this melody is londonderryair - lrb- sometimes also called the tune from county derry-rrb- .
A2: danny boy
P1 :londonderryair words : this melody is more commonly known with the words `` dannyboy ''
P2 :londonderryair dannyboy music ftse londoni love you .
P3 : dannyboy limavady is most famous for the tune londonderryair collected by jane ross in the mid-19th century from a local fiddle player .
P4 :it was here that jane ross noted down the famous londonderryair -lrb- ` dannyboy ' -rrb- from a passing fiddler .
Question2: Which physicist , mathematician and astronomer discovered the first 4 moons of Jupiter
P1 : Sir Isaac Newton was an English physicist , mathematician , astronomer , natural philosopher , alchemist and theologian…
P2 : Sir Isaac Newton was an English mathematician, astronomer, and physicist who is widely recognized as one of the most influential scientists …
A1: Isaac Newton
Question2: Which physicist , mathematician and astronomer discovered the first 4 moons of Jupiter
P1 : Galileo Galilei was an Italian physicist , mathematician , astronomer , andphilosopher who played a major role in the Scientific
Revolution .
P2 : Galileo Galilei is credited with discovering the first four moons of Jupiter .
A2: Galileo Galilei
[Figure]
“ physicist , mathematician and astronomer ”.
This observation can be seen as multiple passages providing complementary evidence to the correct answer; and a method to take the union of the evidence for question answering will be helpful.
Such a method is important especially for cases where the question is complex and a single passage is not sufﬁcient to answer the question.
In this case it is necessary to turn to complementary evidence from other passages.
To provide more accurate answers for open-domain QA, we hope to make better use of multiple passages for the same question by aggregating both the strengthened and the complementary evidence from all the passages.
We formulate the above evidence aggregation as an answer re-ranking problem.
Re-ranking has been commonly used in NLP problems, such as in parsing and translation, in order to make use of high-order or global features that are too expensive for decoding algorithms (Collins & Koo, 2005;
Shen et al., 2004; Huang, 2008; Dyer et al., 2016).
Here we apply the idea, for each answer candidate, to efﬁciently incorporate global information from multiple pieces of textual evidence, without signiﬁcantly increasing the complexity of the prediction of the
RC model: we ﬁrst collect the top- K candidate answers based on their probabilities computed by a standard RC/QA system, and then we use two proposed re-rankers to re-score the answer candidates by aggregating each candidate’s evidence in different ways. The re-rankers are:
• A strength-based re-ranker , which ranks the answer candidates according to how often their evidence occurs in different passages.
The re-ranker is based on the ﬁrst observation, that if one answer candidate has multiple pieces of evidence, and each passage containing evidence will tend to predict it as the answer with a relatively high score (although it may not be the top score), the candidate is more likely to be correct.
The passage count of each candidate, and the aggregated probabilities for the candidate, reﬂects how strong its evidence is, thus in turn suggests how likely the candidate is the corrected answer.
• A coverage-based re-ranker , which aims to rank a answer candidate higher if the union of all its contexts in different passages could cover more aspects asked in the question.
To achieve this, for each answer we concatenate all the passages that contain the answer together.
The result is a new context that aggregates all the evidence necessary to entail the answer for the question.
We then treat the new context as one sequence to represent the answer, and build an attentionbased match-LSTM model (Wang & Jiang, 2017) between the sequence and the question to measure how well the new aggregated context could entail the question.
Overall, our contributions are as follows: 1) We propose a re-ranking-based framework to make use of the evidence from multiple passages in open-domain QA, and two re-rankers to address the common cases of coherent and complementary evidence aggregation in existing open-domain QA datasets.
We ﬁnd out the latter performs better on two of the three public datasets.
Our proposed we achieved to 8%
M ETHOD
Given an open-domain question q , the problem we are solving is to ﬁnd the correct answer a g to q based on all the information from the web.
To retrieve all the contexts most related to the question, an IR model (with the help of a search engine such as Google or Bing) is used to provide the top- N candidate passages, p 1 , p 2 , ..., p N , from the web.
Then a reading comprehension (RC) model is used to extract the answer from these passages.
This setting is different from closed-domain QA (Rajpurkar et al., 2016), where a single ﬁxed passage is given, from which the answer is to be extracted.
When developing a closed-domain QA system, we can use the speciﬁc positions of the answer sequence in the given passage for training.
In comparison, in the open-domain setting, the RC models are usually trained under distant supervision (Chen et al., 2017; Dhingra et al., 2017b;
Joshi et al., 2017).
Speciﬁcally, as the training data doesn’t label the positions of the answer spans in passages, during the training stage, the RC model will match all passages that contain the ground-truth answer with the question one by one.
Throughout the paper, we will ﬁrst directly apply an existing RC model called R 3 (Wang et al., 2017) to extract the candidate answers.
After the candidate answers are extracted, we start to aggregate evidence from multiple passages for better answer prediction by adopting a re-ranking approach over answer candidates.
Given a question q , suppose we have a baseline open-domain QA system that could generate the top- K answer candidates a 1 , . .
. , a K , each being a text span in some passage p i .
The goal of the re-ranker is to rank this list of candidates such that the top-ranked candidates have more chance to be the correct answer a g .
This re-ranking step considers more powerful features and more context information that cannot be easily handled by the baseline system.
In this paper, we investigate two types of such more powerful information based on the multiple pieces of evidence for each answer: evidence strength and evidence coverage , as aggregated from multiple passages.
An overview of our method is shown in Figure 2.
2. E VIDENCE A GGREGATION FOR S TRENGTH - BASED R E - RANKER
For open-domain QA, unlike the closed-domain setting, we have more passages retrieved by the IR model and the ground-truth answer may appear in different passages, which means different answer spans may be referring to the same answer.
Considering this property, we provide two features to further re-rank the Top- K answers generated by the RC model.
Measuring Strength by Count For this re-ranking method, we hypothesize that the more passages we have that could entail a particular answer for the question, the stronger the evidence is for that answer.
In the top- K answer spans generated by the RC model, each time an answer appears in the top- K answer list, we increase the counter for that answer by one.
In this way, in the end the answer with the highest count will be predicted to be the ﬁnal answer.
Measuring Strength by Probability Since we can get the probability of each answer span in a passages based on the RC model, we can also sum up the probabilities of the answer spans that are referring to the same answer.
Then the answer with the highest probability would be the ﬁnal prediction 1 .
In the re-ranking scenario, it is not necessary to exhaustively consider all the probabilities of all the spans in the passages, as there may be a large number of different answer spans and most of them are irrelevant to the ground-truth answer.
Remark:
Note that neither method above requires new model training.
Both methods just take the candidate predictions from the baseline QA system and then perform counting or probability calculation.
During testing the time complexity of strength-based re-ranking is negligible.
2. E VIDENCE A GGREGATION FOR C OVERAGE - BASED R E - RANKER
This section focuses on another type of evidence aggregation which needs to take the union of complementary information from different evidence passages.
As shown in Figure 1, the two answer candidates both have evidence matching the ﬁrst half of the question.
However, only the correct answer has evidence that could also match the second half.
In this case, the strength-based re-ranker will treat both answer candidates the same due to the equal amount of supporting evidence, while actually the second answer has complementary evidence satisfying all aspects of the question.
To handle such case, we propose a coverage-based re-ranker that could rank the answer candidates according to whether the unions of their evidence from different passages could well cover the question.
In order to take the union of evidence into consideration, we ﬁrst concatenate the passages containing the answer into a single pseudo passage.
Then we measure how well the new passage could entail the answer for the question.
As examples shown in Figure 1(b), we hope the textual entailment model could reﬂect (i) how each aspect of the question is matched by the union of multiple passages, where we simply deﬁne the aspect to be one hidden state of bi-directional LSTM (Hochreiter &
Schmidhuber, 1997); (ii) whether all the aspects of the question can be matched by the union of multiple passages.
The match-LSTM (Wang & Jiang, 2016) model is one way to achieve the above effect in entailment.
Therefore we build our coverage-based re-ranker on top of the concatenated pseudo passages through Match-LSTM.
The detailed method is described below.
Passage Aggregation
We consider the top- K answers, a 1 , . .
. , a K , provided by the baseline QA system.
For each answer a k , k ∈ [1 , K ] , we concatenate all the passages that contain a k , { p n | a k ∈ p n , n ∈ [1 , N ] } , to form the union passage ˆp k .
Our further model is to identify which union passage, e.g. ˆp k , could better entail its answer, e.g. a k , for the question.
Measuring Aspect(Word)-Level Matching As discussed earlier, the ﬁrst mission of the coverage-based re-ranker is to measure how each aspect of the question is matched by the union of multiple passages.
We achieve this with word-by-word attention followed by a comparison module.
First, we write the answer candidate a , question q and the union passage ˆp of a as matrices A , Q , ˆP , with each column being the embedding of a word in the sequence.
We then feed them to the bidirectional LSTM as follows:
[Equation]
where H a ∈ R l × A , H q ∈ R l × Q and H p ∈ R l × P are the hidden states for the answer candidate, question and passage respectively; l is the dimension of the hidden states, and A , Q and P are the length of the three sequences, respectively.
Next, we enhance the question representation H q with H a :
[Equation]
where [ · ; · ] is the concatenation of two matrices in row and H aq ∈ R l × ( A + Q ) .
As most of the answer candidates do not appear in the question, this is for better matching with the passage and ﬁnding more answer-related information from the passage.
Now we can view each aspect of the question as a column vector (i.e. a hidden state at each word position in the answer-question concatenation) in the enhanced question representation H aq .
Then the task becomes to measure how well each column vector can be matched by the union passage; and we achieve this by computing the attention vector Parikh et al.
(2016) for each hidden state of sequences a and q as follows:
[Equation]
where α ∈ R P × ( A + Q ) is the attention weight matrix which is normalized in column through softmax.
H aq ∈ R l × ( A + Q ) are the attention vectors for each word of the answer and the question by weighted summing all the hidden states of the passage ˆp .
Now in order to see whether the aspects in the question can be matched by the union passage, we use the following matching function:
[Equation]
where ·⊗ e ( A + Q ) is to repeat the vector (or scalar) on the left A + Q times; ( · (cid:74) · ) and ( ·−· ) are the element-wise operations for checking whether the word in the answer and question can be matched by the evidence in the passage.
We also concatenate these matching representations with the hidden state representations H aq
and H aq , so that some stop words or punctuation representations may not affect the ﬁnal aspect-level matching representations M ∈ R 2 l × ( A + Q ) , which is computed through the non-linear transformation on four different representations with parameters W m ∈ R 2 l × 4 l and b m ∈ R l .
Measuring the Entire Question Matching Next, in order to measure how the entire question is matched by the union passage ˆp by taking into consideration of the matching result at each aspect, we add another bi-directional LSTM on top of it to aggregate the aspect-level matching information 2 :
[Equation]
where H m ∈ R l × ( A + Q ) is to denote all the hidden states and h s ∈ R l , the max of pooling of each dimension of H m , is the entire matching representation which reﬂects how well the evidences in questions could be matched by the union passage.
Re-ranking Objective Function Our re-ranking is based on the entire matching representation.
For each candidate answer a k , k ∈ [1 , K ] , we can get a matching representation h s k between the answer a k , question q and the union passage ˆp k through Eqn.
Then we transform all representations into scalar values followed by a normalization process for ranking as follows:
[Equation]
where we concatenate the match representations for each answer in row through [ · ; · ] , and do a non-linear transformation by parameters W r ∈ R l × l and b r ∈ R l to get hidden representation
R ∈ R l × K .
Finally, we map the transformed matching representations into scalar values through parameters w o ∈ R l and w o ∈ R .
o ∈ R K is the normalized probability for the candidate answers to be ground-truth.
Due to the aliases of the ground-truth answer, there may be multiple answers in the candidates are ground-truth, we use KL distance as our objective function as follows:
[Equation]
where y k indicates whether a k the ground-truth answer or not and is normalized by (cid:80) Kk
=1 y k
and o k is the ranking output of our model for a k .
2. C OMBINATION OF D IFFERENT T YPES OF A GGREGATIONS
Although the coverage-based re-ranker tries to deal with more difﬁcult cases compared to the strength-based re-ranker, the strength-based re-ranker works on more common cases according to the distributions of most open-domain QA datasets.
As a result we hope to make the best of both methods.
We achieve our full re-ranker by weighted combination of the outputs of the above different rerankers without further training.
Speciﬁcally, we ﬁrst use softmax to re-normalize the top-5 answer scores provided by the two strength-based rankers and the one coverage-based re-ranker; we then weighted sum up the scores for the same answer and select the answer with the largest score as the ﬁnal prediction.
E XPERIMENTAL S ETTINGS
We have experiments on three public open-domain QA datasets, Quasar-T (Dhingra et al., 2017b),
SearchQA (Dunn et al., 2017) and TriviaQA (Joshi et al., 2017), to evaluate our model.
The passages retrieved for all the answers based on search engine, such as Google and Bing, are given.
And we will not retrieve more passages for the experiment.
3. D ATASETS
The statistics of the three datasets are shown in Table 1.
Quasar-T 3 (Dhingra et al., 2017b) is based on a trivia question set and makes use of “lucene index” as IR model to collect 100 sentence-level passages for each question from ClueWeb09 data source.
The human performance is evaluated in an open-book setting, i.e. the people had access to the passages retrieved by the same IR model and tried to ﬁnd the answers from the passages.
SearchQA 4 (Dunn et al., 2017) is based on a Jeopardy!
questions and use search engine Google to collect about 50 web page snippets as passages for each question.
The human performance is evaluated in a similar way to Quasar-T dataset.
TriviaQA (Open-Domain Setting) 5 (Joshi et al., 2017) collects trivia questions coming fromtrivia and quiz-league websites, and makes use of the Bing Web search API to collect the top
[Table]
webs most related to the questions.
We focus on the open domain setting of the dataset 6 and our model will use all the IR retrieved information.
3. B ASELINES
Our baseline models 7 include: GA (Dhingra et al., 2017a;b), a reading comprehension model with gated-attention; BiDAF (Seo et al., 2017), a RC model with bidirectional attention ﬂow; AQA (Buck et al., 2017), a reinforced system learning to aggregate the answers generated by the re-written questions; R 3 (Wang et al., 2017), a reinforced model making use of a ranker for selecting passages to train RC model.
As R 3 is the ﬁrst step of our system for generating candidate answers, the improvement our re-ranking methods can be directly compared to this baseline.
3. I MPLEMENTATION D ETAILS
We ﬁrst use a pre-trained R 3 model (Wang et al., 2017), which gets the stat-of-the-art performance on the 3 public datasets we consider, to generate top 50 candidate spans for both training, dev and test datasets, and we use them for further ranking.
During training, if the ground-truth answer doesn’t appear in the answer candidates, we will manually add it into the answer candidate list.
For the details of the Coverage-based Re-Ranker, we use Adamax (Kingma & Ba, 2015) to optimize the model.
Word embeddings are initialized by GloVe (Pennington et al., 2014) and are not updated during training.
We set all the words beyond Glove as zero vector.
We set l to 300, batch size to 30, learning rate to 0.002.
We tune the dropout probability fro 0 to 0.5 and the number of candidate answers for re-ranking ( K ) in [3, 5, 10] 8 .
R ESULTS AND A NALYSIS
In this section, we will show the results of our different re-ranking methods on three different public and the further analysis about our models.
4. O VERALL R ESULTS
The performance of our models are shown in Table 2.
We use F1 score and Exact Match (EM) as our evaluation metrics 9 .
From the results, we can clearly see that our full re-ranker, the combination of different re-rankers, can signiﬁcantly outperform the previous best performance in a large margin, especially on the datasets of Quasar-T and SearchQA. Moreover, our model is much better than
[Table]
Human Performance
[Table]
the human performance on the SearchQA dataset.
Besides, we can see that our coverage-based reranker performs a generally well performance on the three datasets, even though gets 1% lower than strength-based re-ranker on SearchQA dataset.
4. A NALYSIS
In this subsection, we will analyze the beneﬁts of re-ranking models.
Re-ranking performance versus answer lengths and question types Figure 3 decomposes the performance according to the length of golden answers and the types of questions on TriviaQA and
Quasar-T.
We ignore the analysis on SearchQA because the Jeopardy!
style questions are more difﬁcult to distinguish the questions types, and the range of answer length is smaller.
According to the results, the coverage-based re-ranker could outperform the baseline in different length of answers and different types of questions.
The strength-based re-ranker (counting) also gives improvement but is less stable across different datasets.
While the strength-based re-ranker (probability) tends to have results and trends that are close to the baseline curves, which is probably because the method is dominated by the probabilities predicted by the baseline.
[Table]
[Table]
The coverage-based re-ranker and the strength-based re-ranker (counting) have similar trends on most of the question types.
The only exception is that the strength-based re-ranker performs significantly worse compared to the coverage-based re-ranker on the “ why ” questions.
This is possibly because those questions usually have non-factoid answers, which are less likely to have exactly the same text spans predicted on different passages by the baseline.
Potential improvement of re-rankers Table 3 shows the percentage of time when the correct answer is included in the top- K answer predictions of the baseline R 3 method.
More concretely, the scores are computed by selecting the answer from the top- K predictions with the best EM/F score.
Therefore the ﬁnal top- K EM and F1 can be viewed as recall or upper bound of the top- K prediction.
From the results, although the top-1 prediction of R 3 is not very accurate, there is high chance that a moderate top- K list could cover the correct answer.
This explains why our re-ranking approach could achieve large improvement.
Also by comparing the upper bound performance of top- 5 and our re-ranking performance in Table 2, we can see there is still a clear gap of about 10% on both datasets and on both F1 and EM, showing the great potential improvement for the re-ranking model in future work.
Effect of the selection of K for the coverage-based re-ranker As shown in Table 3, with the increase of K from 1 to 10, the recall of top- K predictions from the baseline R 3 system increase signiﬁcantly.
Ideally, if we use a larger K , then the candidate lists will be more likely to contain good answers.
At the same time, the lists to be ranked are longer thus the re-ranking problem is harder.
Therefore, there is a trade-off between the coverage of rank lists and the difﬁculty of reranking; and selecting an appropriate K becomes important.
Table 4 shows the effects of K on the performance of coverage-based re-ranker.
We train and test the coverage-based re-ranker on top- K predictions from the baseline, here K ∈ { 3 , 5 , 10 } .
The upper bound results are the same ones from Table 3.
The results show that when K is small, like K =3, the performance is not very good due to the low coverage (thus low upper bound) of the candidate list.
With the increase of K , the performance becomes better, but the top-5 and top-10 results are on par with each other.
This is because the higher upper bound of top-10 results counteracts the harder problem of re-ranking longer lists.
Since there is no signiﬁcant advantage of the usage of K =10 while the computation is slower, we report all testing results with K =5.
Effect of the selection of K for the strength-based re-ranker Similar to Table 4, we conduct experiments to show the effects of K on the performance of strength-based re-ranker.
We run the strength-based re-ranker (counting) on top- K predictions from the baseline, here K ∈ { 10 , 50 , 100 , 200 } , as well as evaluate the upper bound results for these K s. Note that the strengthbased re-ranker runs very fast and the different number of K does not affect the computation speed
[Table] [Table]
very much compared to the other QA components.
The results are shown in Table [Table] where we achieve the best results when K =50.
The performance drops signiﬁcantly when K increases to 200.
This is because the ratio of incorrect answers increases a lot, making incorrect answers also likely to have high counts.
When K is smaller, such incorrect answers appear less because statistically they have lower prediction scores.
We report all testing results with K =50.
Examples Table 6 shows an example from Quasar-T where the re-ranker successfully correct the wrong answer predicted by the baseline.
This is a case where the coverage-based re-ranker helps: the correct answer “ Sesame Street ” has evidence from different passages that covers the aspects “ Emmy
Award ” and “ children ’s television shows ”.
Although it still doesn’t fully match all the facts in the question, it still helps to rank the correct answer higher than the top-1 prediction “ Great Dane ” from the R 3 baseline, which only has evidence covering “ TV ” and “ 1969 ” in the question.
R ELATED W ORK
Open Domain Question Answering The task of Open domain question answering dates back to as early as (Green Jr et al., 1961) and was popularized by TREC-8 (Voorhees, 1999).
The task is to produce the answer to a question by exploiting resources such as documents (Voorhees, 1999), webpages (Kwok et al., 2001) or structured knowledge bases (Berant et al., 2013; Bordes et al., 2015; Yu et al., 2017).
Recent efforts (Chen et al., 2017; Dunn et al., 2017; Dhingra et al., 2017b; Wang et al., 2017) beneﬁt from the advance of machine reading comprehension (RC) and follow the search-and-read
QA direction.
These deep learning based methods usually rely on a document retrieval module to retrieve a list of passages for RC models to extract answers.
As there is no passage-level annotation about which passages entails the answer, the model has to ﬁnd proper ways to handle the noise introduced in the IR step.
Chen et al.
(2017) uses bi-gram passage index to improve the retrieval step; and Dunn et al.
Dhingra et al.
(2017b) propose to reduce the length of the retrieved passages.
Wang et al.
(2017) focus more on noise reduction in the passage ranking step, in which a ranker module is jointly trained with the RC model with reinforcement learning.
To the best of our knowledge, our work is the ﬁrst to improve neural open-domain QA systems by beneﬁting from multiple passages for evidence aggregation.
Multi-Step Approaches for Reading Comprehension
We are the ﬁrst to introduce re-ranking methods to neural open-domain QA and multi-passage RC.
In the meanwhile, our two-step fashion approach shares some similarities to the previous multi-step approaches proposed for standard single-passage RC, in terms of the purposes of either using additional information or re-ﬁning answer predictions that are not easily handled by the standard answer extraction models for RC.
On cloze-test tasks (Hermann et al., 2015), Epireader Trischler et al.
(2016) relates to our work in the sense that it is a two-step extractor-reasoner model, which ﬁrst extracts K most probable singletoken answer candidates, then constructs a hypothesis by combining each answer candidate to the question and compares the hypothesis with all the sentences in the passage.
Their model differs from ours in several aspects: (i) Epireader matches a hypothesis to every single sentence, including all the “noisy” ones that does not contain the answer, that makes the model inappropriate for open-domain
QA setting; (ii) The sentence matching is based on the sentence embedding vectors computed by a convolutional neural network, which makes it hard to distinguish redundant and complementary evidence in aggregation; and (iii) Epireader passes the probabilities predicted by the extractor to the reasoner directly to sustain differentiability, which can not be easily adapted to our problem to handle phrases as answers or to use part of passages.
Similarly, (Cui et al., 2017) also combined answer candidates to the question to form hypotheses, and then explicitly use language models trained on documents to re-rank the hypotheses.
This method beneﬁts from the consistency between the documents and gold hypotheses (which are titles of the documents) in cloze-test datasets, but does not handle multiple evidence aggregation like our work.
S-Net (Tan et al., 2017) proposes a two-step approach to achieve generative QA.
The model ﬁrst extracts an text span as the answer clue and then generates the answer according to the question, passage and the text span.
Besides the different goal on answer generation instead of re-ranking like this work, their approach also differ from ours on that it extracts only one text span from a single selected passage.
C ONCLUSION
We propose to improve open-domain QA with evidence aggregation from multiple passages.
The aggregation process is formulated as answer re-ranking; and two re-rankers are proposed to deal with two types of important evidence aggregation in open-domain QA, aggregation on pieces of evidence strengthening each other and on pieces of evidence complementary to each other.
Both re-rankers helped to signiﬁcant improvement our results and greatly advances the state-of-the-arts on three open-domain QA datasets