M ULTI -M ENTION L EARNING FOR R EADING C OMPREHENSION WITH N EURAL C ASCADES
Anonymous authors
Paper under double-blind review
A BSTRACT
Reading comprehension is a challenging task, especially when executed across longer or across multiple evidence documents, where the answer is likely to reoccur.
Existing neural architectures typically do not scale to the entire evidence, and hence, resort to selecting a single passage in the document (either via truncation or other means), and carefully searching for the answer within that passage.
However, in some cases, this strategy can be suboptimal, since by focusing on a speciﬁc passage, it becomes difﬁcult to leverage multiple mentions of the same answer throughout the document.
In this work, we take a different approach by constructing lightweight models that are combined in a cascade to ﬁnd the answer.
Each submodel consists only of feed-forward networks equipped with an attention mechanism, making it trivially parallelizable.
We show that our approach can scale to approximately an order of magnitude larger evidence documents and can aggregate information from multiple mentions of each answer candidate across the document.
Empirically, our approach achieves state-of-the-art performance on both the Wikipedia and web domains of the TriviaQA dataset, outperforming more complex, recurrent architectures.
I NTRODUCTION
Reading comprehension, the task of answering questions based on a set of one more documents, is a key challenge in natural language understanding.
While data-driven approaches for the task date back to Hirschman et al.
(1999), much of the recent progress can be attributed to new largescale datasets such as the CNN/Daily Mail Corpus (Hermann et al., 2015), the Children’s Book Test
Corpus (Hill et al., 2015) and the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016).
These datasets have driven a large body of neural approaches (Wang & Jiang, 2016; Lee et al., 2016; Seo et al., 2016;
Xiong et al., 2016, inter alia ) that build complex deep models typically driven by long short-term memory networks (Hochreiter & Schmidhuber, 1997).
These models have given impressive results on SQuAD where the document consists of a single paragraph and the correct answer span is typically only present once.
However, they are computationally intensive and cannot scale to large evidence texts such as those in the recently released TriviaQA (Joshi et al., 2017) dataset, which provides evidence as entire webpages.
So far, progress on the TriviaQA dataset has leveraged existing approaches on the SQuAD dataset by truncating documents and focusing on the ﬁrst 800 words (Joshi et al., 2017; Pan et al., 2017).
This has the obvious limitation that the truncated document may not contain the evidence required to answer the question.
Furthermore, in TriviaQA there is often useful evidence spread through the supporting documents.
This cannot be harnessed by approaches such as Choi et al.
(2017) that greedily search for the best 1-2 sentences in a document.
For example, in Fig.1 the answer does not appear in the ﬁrst 800 words.
The passage starting at token 4089 does contain all of the information required to infer the answer but this inference requires us to resolve the two complex co-referential phrases in ‘In the summer of that year they got married in a church’.
Both of these are easier to resolve if we have access to the other mentions of Kramer and Pollock that describe their relationship in the 1940’s and the fact that they were married in a church.
In this paper we present a novel cascaded approach to extractive question answering that can accumulate evidence from an order of magnitude more text than previous approaches, and which achieves state-of-the-art performance on all tasks and metrics in the TriviaQA evaluation.
The model ( § 3) is
Question :
Which US artist married Lee Krasner in 1945?
Answer :
Jackson Pollock
Document :
Wikipedia entry for Lee Krasner (Excerpts shown)
Start
Passage
She lost interest in their usage of hard-edge geometric style after her relationship with Pollock began.
Pollock ’s influence.
Although many people believe that Krasner stopped working in the 1940s in order to nurture Pollock ’s home life and career, she never stopped creating art.
Relationship with Jackson Pollock
Lee Krasner and Jackson Pollock established a relationship with one another in 1942 after they both exhibited at the McMillen
Gallery.
Krasner was intrigued by his work and the fact she did not know who he was since she knew many abstract painters in New York.
She went to his apartment to meet him.
By 1945, they moved to The Springs on the outskirts of East Hampton.
In the summer of that year, they got married in a church with two witnesses present.
While she married Pollock in a church, Krasner continued to identify herself as Jewish but decided to not practice the religion.
split into three levels that consist of a feed-forward network applied to an embedding of the candidate answer span in the document.
The ﬁrst level submodels use simple bag-of-embeddings representations of the question, span, and the words surrounding the span in the document (the context).
The second level submodel uses the representation built by the ﬁrst level, along with an attention mechanism (Bahdanau et al., 2014) that aligns question words with words in the sentence that contains the candidate span.
Finally, for answer candidates that are mentioned multiple times in the evidence document, the third level submodel aggregates the mention level representations from the second level to build a single answer representation.
At inference time, predictions are made using the output of the third level classiﬁer only.
However, in training, as opposed to using a single loss, all the classiﬁers are trained using the the multi-loss framework of Al-Rfou et al.
(2016), with gradients ﬂowing from down from higher to lower submodels.
The motivation behind the separation into submodels and multi-loss objective is the prevention of adaptation between features (Hinton et al., 2012).
This is particularly important in our case where the higher level, more complex submodels could subsume the weaker, lower level models c.f. Al-Rfou et al.
In Section § 4 we show that the extension to longer documents, the inclusion of the multi-mention aggregation, and the use of a multi-loss objective are all essential in helping our model achieve state of the art performance.
Since we use only feed-forward networks along with ﬁxed length window representations of the question, answer candidate, and answer context, the vast majority of computation required by our model is trivially parallelizable, in contrast to other efforts that use recurrent models.
R ELATED A PPROACHES
Most existing approaches to reading comprehension (Wang & Jiang, 2016; Lee et al., 2016;
Seo et al., 2016; Xiong et al., 2016; Wang et al., 2017, inter alia ) involve using recurrent neural nets (LSTMs or memory nets) along with various ﬂavors of the attention mechanism Bahdanau et al.
(2014) to align the question with the passage.
In preliminary experiments in the original TriviaQA paper, Joshi et al.
(2017) explored one such approach, the BiDAF architecture (Seo et al., 2016), for their dataset.
However, BiDAF being designed for SQuAD, where the evidence passage is much shorter (122 tokens on an average), does not scale to the entire document (2895 tokens on an average) for TriviaQA; hence the document is truncated to the ﬁrst 800 tokens.
Pointer networks with multihop reasoning, and syntactic and NER features, have been used recently in three architectures —
Smarnet (Chen et al., 2017b), Reinforced Mnemonic Reader (Hu et al., 2017) and MEMEN (Pan et al., 2017) for both SQuAD and TriviaQA.
Most of the above also use document truncation.
Approaches such as Choi et al.
(2017) that ﬁrst select the top (1-2) sentences using a very coarse model and then run a recurrent architecture on these sentences to ﬁnd the correct span.
Chen et al.
(2017a)
propose scoring spans in each paragraph with a recurrent network architecture separately and then take taking the span with the highest score.
Our approach is different from existing question-answering architectures in the following aspects.
First, instead of using one monolithic architecture, we employ a cascade of simpler models that enables us to analyze larger parts of the document.
Seondly, instead of recurrent neural networks, we use only feed-forward networks to improve scalability.
Further, our approach aggregates information from different mentions of the same candidate answer at the representation level.
Finally, our learning problem also leverages the presence of several correct answer spans in the document.
M ODEL
For the reading comprehension task ( § 3.1), we propose a cascaded model architecture arranged in three different levels ( § 3.2).
Submodels in the lower levels ( § 3.3) use simple features to score candidate answer spans.
Higher level submodels select the best answer candidate using more expensive attention-based features ( § 3.4) and by aggregating information from different occurrences of each answer candidate in the document ( § 3.5).
The submodels score all the potential answer span candidates in the evidence document 1 , each represented using simple bags-of-embeddings.
Each submodel is associated with its own objective; the ﬁnal objective is given by a linear combination of all the objectives ( § 3.6).
We conclude this section with a runtime analysis of the model ( § 3.7).
3. T ASK
We take as input a sequence question word embeddings q = { q 1 . . .
q m } , and document word embeddings d = { d 1 . . .
d n } , obtained from a dictionary of pre-trained word embeddings.
Each candidate answer span s = { d s 1 . . .
d s o } is a collection of o ≤ l word embeddings where l is the maximum length of a span.
The set of all candidate answer spans is S :
= { s i } nli =1 2 .
Since the same spans of text can occur multiple times in each document, we also maintain the set of unique answer candidate spans, u ∈ S u , and a mapping between each span and the unique answer candidate that it corresponds to, s (cid:16) u .
In TriviaQA, each answer can have multiple alternative forms, as shown in Fig.1.
The set of correct answer strings is S ∗ and our task is to predict a single answer candidate ˆ u ∈ S.
3. O VERVIEW OF M ETA -A RCHITECTURE
We ﬁrst describe our meta-architecture, which is a collection of simple submodels M k ( · ) organized in a cascade.
The idea of modeling separate classiﬁers that use complementary features comes from Al-Rfou et al.
(2016) who found this gave considerable gains over combining all the features into a single model for a conversational system.
As shown in Figure 2, our architecture consists of two submodels M 1 , M 2 at the ﬁrst level, one submodel M 3 at the second, and one submodel M 4 at the third level.
Each submodel M k returns a score, φ ( k ) s as well as a vector, h ( k
) s that is fed as input
[Figure]
to the submodel at the next level.
[Equation]
LEVEL
[Equation]
LEVEL
[Equation]
LEVEL
Using their respective scores, φ ( k ) s , the models M 1 ...M 3 deﬁne a distribution over all spans, while
M 4 uses φ (4) u to deﬁne a distribution over all unique candidates, as follows:
[Equation]
In training, our total loss is given by an interpolation of losses for each of M 1 , .., M 4 .
However, during inference we make a single prediction, simply computed as ˆ u = arg max u ∈ D u φ (4) u .
3. L EVEL 1: A VERAGED B AGS OF E MBEDDINGS
The ﬁrst submodel is the simplest, and it takes as input only bags of embeddings.
This submodel has two parts, one that looks at the span and question together, and another that looks at the span along with the local context in which the span appears.
First we deﬁne the span representation used by both of these parts.
Then we will describe the question and context representations, and how they are used to predict which span is the correct answer.
Span: The length o span, s is represented using averaged embeddings ˜s =
1 o (cid:80) oj =1 d s j .
We also include a binary feature
γ qs i indicating if any question words are in the span, following (Chen et al., 2017a).
The question-in-span feature is motivated by the observation that questions rarely contain the answers that they are seeking and it helps the model avoid answers that are overcomplete — containing information already known by the questioner.
3.3. Q UESTION + S PAN ( M 1 )
The question + span component of the level 1 submodel predicts the correct answer using a feedforward neural network on top of ﬁxed length question and span representations.
The span representation is taken from above and we represent the question with a weighted average of the question embeddings.
Question: Motivated by Lee et al.
we learn a weight δ q i for each of the words in the question using the parameterized function deﬁned below.
This allows the model to learn to focus on the important words in the question.
δ q i is generated with a two-layer feed-forward net with rectiﬁed linear unit (ReLU) activations (Nair & Hinton, 2010; Glorot et al., 2011).
[Equation]
where U , V , w , z , a and b are parameters of the feed-forward network.
Since all three submodels rely on identically structured feed-forward networks and linear prediction layers, from now on we will use the abbreviations ﬀnn and linear as shorthand for the functions deﬁned above.
The scores, δ q i are normalized and used to generate an aggregated question vector
˜q as follows.
[Equation]
Now that we have representations of both the question and the span, the question + span model computes a hidden layer representation of the question, span pair as well as a scalar score for the span candidate as follows:
[Equation]
where [ a ; b ] represents the concatenation of a and b .
3.3. S PAN
[Equation] HORT C ONTEXT ( M 2 )
The span + short context component builds a representation of each span in its local linguistic context.
We hypothesize that words in the left and right context of a candidate span are important for the modeling of the span’s semantic category (e.g. person, place, date).
Context: We represent the left context c Ls , and right context c Rs using averaged embeddings
[Equation]
and use these to generate a span-in-context representation h (2) s and answer score φ (2)
s
[Equation]
3. L EVEL 2: A TTENDING TO THE C ONTEXT ( M 3 )
Unlike level 1 which builds a question representation independently of the document, the levelsubmodel considers the the question in the context of each sentence in the document.
This level aligns the question embeddings with the embeddings in each sentence using neural attention (Bahdanau et al., 2014), speciﬁcally the attend and compare steps from the decomposable attention model of Parikh et al.
and these aligned representations are used to predict if the sentence could contain the answers.
We apply this attention to sentences, not individual span contexts, to prevent our computational requirements from exploding with the number of spans.
Subsequently, it is only because level 2 includes the level 1 representations h (1) s and h (2) s that it can assign different scores to different spans in the same sentence.
Sentence: We deﬁne µ s = { d µ s1 . . .
d µ s (cid:96) } to be the embeddings of the (cid:96) words of the sentence that contains s .
First, we measure the similarity between every pair of question and sentence embeddings by passing them through a feed-forward net, ﬀnn att 1 and using the resulting hidden layers to compute a similarity score, η .
Taking into account this similarity, attended vectors ¯ q i and ¯ d µ sj are computed for each question and sentence token, respectively
[Equation]
The original vector and its corresponding attended vector are concatenated and passed through another feed-forward net, ﬀnn att 2 the ﬁnal layers from which are summed to obtain a question-aware sentence vector ¯ µ s , and a sentence context-aware question vector, ¯ q .
[Equation]
Using these attended vectors as features, along with the hidden layers from level 1 and the questionspan feature
[Equation]
new scores and hidden layers are computed for level 2:
[Equation]
3. L EVEL 3: A GGREGATING M ULTIPLE M ENTIONS ( M 4 )
In this level, we aggregate information from all the candidate answer spans which occur multiple times throughout the document.
The hidden layers of every span from level 2 (along with the question-in-span feature) are passed through a feed-forward net, and then summed if they correspond to the same unique span, using the s (cid:16) u map.
The sum, h u is then used to compute the score and the hidden layer 3 for each unique span, u in the document.
[Equation]
3. O VERALL OBJECTIVE AND LEARNING
To handle distant supervision, previous approaches use the ﬁrst mention of the correct answer span (or any of its aliases) in the document as gold (Joshi et al., 2017).
Instead, we leverage the existence of multiple correct answer occurrences by maximizing the probability of all such occurrences.
Using Equation 1, the overall objective, (cid:96) ( U ∗ , V ∗ , w ∗ , z ∗ , a ∗ , b ∗ ) is given by the total negative log likelihood of the correct answer spans under all submodels:
[Equation]
where λ 1 , .., .λ 4 are hyperparameters that weigh the contribution of each loss term.
3. C OMPUTATIONAL C OMPLEXITY
We brieﬂy discuss the asymptotic complexity of our approach.
For simplicity assume all hidden dimensions and the embedding dimension are ρ and that the complexity of matrix( ρ × ρ )-vector( ρ × 1 ) multiplication is O ( ρ 2 ) .
Thus, each application of a feed-forward network has O ( ρ 2 ) complexity.
Recall that m is the length of the question, n is the length of the document, and l is the maximum length of each span.
We then have the following complexity for each submodel:
• Level 1 (Question + Span) :
Building the weighted representation of each question takes O ( mρ 2 ) and running the feed forward net to score all the spans requires O ( nld 2 ) , for a total of O ( mρ 2 + Onlρ 2 ) .
• Level 1 (Span + Short Context) :
This requires O ( nlρ 2 ) .
• Level 2 : Computing the alignment between the question and each sentence in the document takes O ( nρ 2 + mρ 2 + nmρ ) and then scoring each span requires O ( nlρ 2 ) .
This gives a total complexity of O ( nlρ 2 + nmρ ) , since we can reasonably assume that m < n .
• Level 3 : This requires O ( nlρ 2 ) .
Thus, the total complexity of our approach is O ( nlρ 2 + mnρ ) .
While the nl and mn terms can seem expensive at ﬁrst glance, a key advantage of our approach is that each sub-model is trivially parallelizable over the length of the document ( n ) and thus very amenable to GPUs.
E XPERIMENTS AND R ESULTS 4. T RIVIA QA D ATASET
The TriviaQA dataset Joshi et al.
(2017) contains a collection of 95k trivia question-answer pairs from several online trivia sources.
To provide evidence for answering these questions, documents are collected independently, from the web and from Wikipedia.
In addition to the answers from the trivia sources, aliases for the answers are collected from DBPedia; on an average, there aresuch aliases per question-answer pair.
Answers and their aliases can occur multiple times in the document; the average occurrence frequency is almost 15 times per document in either domain.
The dataset also provides a subset on the development and test splits which only contain examples determined by annotators to have enough evidence in the document to support the answer.
4. E XPERIMENTAL S ETTINGS
Data preprocessing: All documents are tokenized using the NLTK 4 tokenizer.
Each document is truncated to contain at most 6000 words and at most 1000 sentences.
Sentences are truncated to a maximum length of 50.
Spans only up to length l = 5 are considered; we discard cross-sentence spans.
To be consistent with the evaluation setup of Joshi et al.
(2017), for the Wikipedia domain we create a training instance for every question (with all its associated documents), while on the web domain we create a training instance for every question-document pair.
Hyperparameters: We use GloVe embeddings (Pennington et al., 2014) of dimension 300 (trained on a corpus of 840 billion words) that are kept ﬁxed during training.
Each embedding vector was normalized to have (cid:96) -2 norm of 1.
Out-of-vocabulary words are hashed to one of 1000 random embeddings each initialized to zero mean and one variance.
Dropout regularization (Srivastava et al., 2014) is applied to all ReLU layers (but not for the linear layers).
We additionally tuned the following hyperparameters using grid search: network size (2-layers, each with 300 neurons), dropout ratio (0.1), learning rate (0.05), context size (1), and loss weights ( λ 1 = λ 2 = 0 .
35 , λ 3 = 0 .
2 , λ 4 = 0 .
1 ) where the optimal values are indicated in parentheses.
We use Adagrad (Duchi et al., 2011) for optimization with a default initial accumulator value of 0.1 and a batch size of 1.
Each hyperparameter setting took 2-3 days to train on a single NVIDIA P100 GPU.
The model was implemented in Tensorﬂow (Abadi et al., 2016).
4. R ESULTS
Table 1 presents our results on both the full test sets as well as the veriﬁed subsets, using the exact match (EM) and F 1 metrics.
Our approach achieves state-of-the-art performance on both the
Wikipedia and web domains outperforming considerably more complex models.
In the Wikipedia domain, we surpass the leaderboard-best 5 (as of 24th October, 2017) on all metrics.
In the web
Since these approaches might not be published as yet, it is not possible for us to draw comparisons with their models.
[Table]
domain, except for the veriﬁed F 1 scores, we see a similar trend.
Surprisingly, we outperform approaches which use multi-layer recurrent pointer networks with specialized memories Chen et al
[Table]
Table 2 shows some ablations that give more insight into the different contributions of our model components.
Our ﬁnal approach (3-Level Cascade, Multi-loss) achieves the best performance.
Training with only a single loss in Level 3 (3-Level
Cascade, Single Loss) leads to a considerable decrease in performance, signifying the effect of using a multi-loss architecture.
It is unclear if combining the two submodels in Level 1 into a single feedforward network that is a function of the question, span and short context (3-Level
Cascade, Combined Level 1) is signiﬁcantly better than our ﬁnal model.
Although we found it could obtain high results, it was less consistent across different runs and gave lower scores on average (49.30) compared to our approach averaged over 4 runs (51.03).
Finally, the last three rows show the results of using only smaller components of our model instead of the entire architecture.
In particular, our model without the aggregation submodel (Level 1 + Level 2 Only) performs considerably worse, showing the value of aggregating multiple mentions of the same span across the document.
The effect of truncation on accuracy in Figure 4.3 (left) indicates that more scalable approaches that can take advantage of longer documents have the potential to perform better on the TriviaQA task.
TriviaQA answers and their aliases typically reoccur in the document (15 times per document on an average).
To verify whether our model is able to predict answers which occur frequently in the document, we look at the frequencies of the predicted answer spans in Fig 4.3 (right).
This distribution follows the distribution of the gold answers very closely, showing that our model learns the frequency of occurrence of answer spans in the document.
C ONCLUSION
We presented a 3-level cascaded model for TriviaQA reading comprehension.
Our approach, through the use of feed-forward networks and bag-of-embeddings representations, can handle longer evidence documents and aggregated information from multiple occurrences of answer spans
[Figure]
i k i
p
e d i a D e v E x ac t M a t c h
[Figure]
Cumulative Frequency
[Figure]
Answer
[Figure]
u
[Figure]
e r o f
E x a m p l e
[Figure]
predictedgold
[Figure]
throughout the document.
We achieved state-of-the-art performance on both Wikipedia and web domains, outperforming several complex recurrent architectures