P HASE C ONDUCTOR ON M ULTI - LAYERED A TTENTIONS FOR M ACHINE C OMPREHENSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Attention models have been intensively studied to improve NLP tasks such as machine comprehension via both question-aware passage attention model and selfmatching attention model.
Our research proposes phase conductor
(PhaseCond
) for attention models in two meaningful ways.
First, PhaseCond an architecture of multi-layered attention models, consists of multiple phases each implementing a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information ﬂow.
Second
, we extend and improve the dot-product attention function for PhaseCond by simultaneously encoding multiple question and passage embedding layers from different perspec
- tives.
We demonstrate the effectiveness of our proposed model PhaseCond on the SQuAD dataset, showing that our model signiﬁcantly outperforms both stateof-the-art single-layered and multiple-layered attention models.
We deepen our results with new ﬁndings via both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models
I NTRODUCTION
Attention-based neural networks have demonstrated success in a wide range of NLP tasks ranging from neural machine translation (Bahdanau et al., 2015), image captioning (Xu et al., 2015), and speech recognition (Chorowski et al., 2015).
Beneﬁting from the availability of large-scale benchmark datasets such as SQuAD (Rajpurkar et al., 2016), the attention-based neural networks has spread to machine comprehension and question answering tasks to allow the model to attend over past output vectors (Wang & Jiang, 2017; Seo et al., 2017; Xiong et al., 2017; Hu et al., 2017;
Wang et al., 2017; Pan et al., 2017).
Wang & Jiang (2017) uses attention mechanism in Pointer Network to detect an answer boundary by predicting the start and the end indices in the passage.
Seo et al.
(2017) introduces a bi-directional attention ﬂow network that attention models are decoupled from the recurrent neural networks.
Xiong et al.
(2017) employs a coattention mechanism that attends to the question and document together.
Wang et al.
(2017) uses a gated attention network that includes both question and passage match and self-matching attentions.
Both Pan et al.
(2017) and Hu et al.
(2017) employs the structure of multi-hops or iterative aligner to repeatedly fuse the passage representation with the question representation as well as the passage representation itself.
Inspired by the above-mentioned works, we are proposing to introduce a general framework PhaseCond for the use of multiple attention layers.
There are two motivations.
First, previous research on the self-attention model is to purely capture long-distance dependencies (Vaswani et al., 2017), and therefore a multi-hops architecture (Hu et al., 2017; Pan et al., 2017) is used to alternatively captures question-aware passage representations and reﬁnes the results by using a self-attention model.
In contrast to the multi-hops architecture, our motivation of using the self-attention model for machine comprehension is to propagate answer evidence which is derived from the preceding question-passage representation layers, without using an interactive manner.
This perspective leads to a different attention-based architecture containing two sequential phases, question-aware passage representation phase and evidence propagation phase.
Second, unlike the domains such as machine translation (Bahdanau et al., 2015) which jointly align and translate words, question-passage attention models for machine comprehension and question answering calculate the alignment matrix corresponding to all question and passage word pairs (Wang & Jiang, 2017; Seo et al., 2017).
Despite the attention models’ success on the machine comprehen-
[Table]
sion task, there has not been any other work exploring learning to encode multiple representations of question or passage from different perspectives for different parts of attention functions.
More speciﬁcally, most approaches use two same question representations U for the question-passage attention model
α ( H, U ) U , where H is the passage representation.
Our hypothesis is that attention models can be more effective by learning different encoders for a question representation U and a question representation V from different aspects.
The key differences between our proposed model and competing approaches are summarized at Table
[Table]
Our contributions are threefold: 1) we proposed a phase conductor for attention models containing multiple phases, each with a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information ﬂow, 2) we present an improved attention function for question-passage attention based on two kinds of encoders: an independent question encoder and a weight-sharing encoder jointly considering the question and the passage, as opposed to most previous works which only using the same encoder for one attention model
[Table]
and 3) we provide both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models.
Experimental results show that our proposed PhaseCond lead to signiﬁcant performance improvements over the state-of-the-art single-layered and multilayered attention models.
Moreover, we observe several meaningful trends: a) during the questionpassage attention phase, repeatedly attending the passage with the same question representation “forces” each passage word to become increasingly closer to the original question representation, and therefore increasing the number of layers has a risk of degrading the network performance, b) during the self-attention phase, the self-attention’s alignment weights of the second layer become noticeably “sharper” than the ﬁrst layer, suggesting the importance of fully propagating evidence through the passage itself.
M ODEL A RCHITECTURE
We proposed phased conductor model (or PhaseCond), which consisting of multiple phases and each phase has two parts, a stack of attention layers L and a stack of fusion layers F controlling information ﬂow.
In our model, a fusion layer F can be an inner fusion layer F inner inside of a stack of attention layers, or an outer fusion layer F outer immediately following a stack of attention layers.
Without loss of generality, PhaseCond’s conﬁgurable computational path for two-phase, a question-passage attention phase containing N question-passage attention layers L Q , and a selfattention phase containing K self-attention layers L S , can be deﬁned as {L Q → F inner } × N → F outer → {L S → F inner } × L → F outer .
Figure 1 gives an concrete example of building PhaseCond based network for the machine comprehension task.
The network contains encoding layers, question-passage attention layers, self-attention layers and output layers.
The encoding layer maps various groups of features, such as character features and word features, to their corresponding embeddings.
Those raw embeddings are then fed into an outer fusion layer to encode these embeddings as passage or question representations in Section 2.1.
Next, the representations are sent to question-passage attention layers to align and represent passage representation with the whole question representation in Section 2.3.
The output of each layer is concatenated and regularized by a stack of fusion layers in Section 2.2.1.
After that, the question-attended passage representation is directly matching against itself, for the purpose of propagating information through the whole passage detailed in Section 2.3.
For each self-attention layer, we conﬁgure an inner fusion layer to obtain a gated representation that is learned to decide how much of the current output is fused by the input from the previous layer detailed in Section 2.3.1.
Finally, the fused vectors are sent to the output layer to predict the boundary of the answer span described in Section 2.4.
2. E NCODER L AYERS
The concatenation of raw features as inputs are processed in fusion layers followed by encoder layers to form more abstract representations.
Here we choose a bi-directional Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) to obtain more abstract representations for words in passages and questions.
Different from the commonly used approaches that every single model has exactly one question and passage encoder (Seo et al., 2017; Wang et al., 2017;
Hu et al., 2017), our encoder layers simultaneously calculate multiple question and passage representations, for the purpose of serving different parts of attention functions of different phases.
We use two types of encoders, independent encoder and shared encoder .
In terms of independent encoder, a bi-directional LSTM is used to produce new representation v Q 1 , . .
. , v Qm of all words in the question,
[Equation]
where v Qj ∈ R 2 d are concatenated hidden states of two independent BiLSTM for the j -th question word and d is the hidden size.
In terms of shared encoder, we jointly produce new representation h P 1 , . .
, h Pn and u Q 1 , . .
, u Qm for the passage and question via a shared bi-directional LSTM,
[Equation]
where h Pi ∈ R 2 d and u Qj ∈ R 2 d are concatenated hidden states of BiLSTM for the i -th passage word and j -th question word, sharing the same trainable BiLSTM parameters.
2. Q UESTION -P ASSAGE A TTENTION L AYERS
The process of representing a passage with a question essentially includes two sub-tasks: 1) calculating the similarity between the question and different parts of the passage, and 2) representing the passage part with the given question depending on how similar they are.
A single question-passage attention layer is illustrated in Figure 2.
In this model, at the t -th layer an alignment matrix
A t ∈ R m , whose shape equals the number of words n in a passage multiplied by the number of words m in a question, is derived by aligning the passage representation at the t −layer with the shared weight question representation,
[Equation]
where h t − 1 i is the the i -th passage word representation at the t − 1 layer,
h
i equals to h Pi calculated from Eq 2, u Qj calculated from Eq 3 is the same for all the layers, the alignment matrix element
A t ( i, j ) is a scalar, denoting the similarity between the i -th passage word and the j -th question word by using dot product of the passage word vector and the question word vector.
Given the alignment matrix element as weights, we compute the new passage representation h ti for the t -th layer by using weighted average over all the independent question representation v Q calculated from Eq 1, as shown in the following.
[Equation]
where h t − 1 i ∈ R 2 d .
Note the independent representation v Qk for k -th question word is different with the shared weight question representation u
Qk .
2.2. O UTER F USION L AYERS
For each question-passage attention layer, its output of h ti , where t ∈ N , is concatenated to form the ﬁnal output vector to represent the i -th passage word C 0 i
=
[ h 1 i ; . .
; h Ni ] .
Increasing the number of layers N allows an increasingly more complex representation for a passage word.
In order to regulate the ﬂow of N question-passage attention layers and to prevent the over-ﬁtting problem, we use fusion layers, which is highway networks (Srivastava et al., 2015) using of GRUlike gating units and taking C
i as its input:
[Equation]
where t ∈ K , K is the number of fusion layers, W tC , W tz are the weights, b tC , b tz are the bias of t -th fusion layer, and the transform gate z t is a non-linear activation function.
The ﬁnal result of fusion layers C N i ∈ R 2 Nd is sent to self-attention models as input for processing.
2. S ELF -A TTENTION L AYERS
Following the question-passage attention layers, self-attention layers propagate evidence through the passage context.
This process is similar in spirit to the steps of exploring similarity or redundancy between answer candidates (e.g., ”J.F.K” and ”Kennedy” can, in fact, be equivalent despite their different surface forms) that have been shown to be very effective during answer merging stage (Ferrucci et al., 2010).
More generally, propagating evidence among the passage words allows correct answers to have better evidence for the question than the rest part of the passage.
For a single self-attention layer, we ﬁrst compute a self alignment matrix S t ∈ R n × n by comparing the passage representation itself,
[Equation]
where h t − 1 i is the i -th passage word as input for the t -th self-attention layer, initial value
h
i is deﬁned as the ﬁnal fused result C Ni from question-passage attention model in section 2.2.1.
Given the alignment matrix element as weights, evidences are propagate from the previous layer to the next to produce the new passage representation h t i by using the weighted average over all the t − 1 layer passage representation:
[Equation]
where h t − 1 k is the passage representation for the k -th word at the t − 1 self-attention layer,
B ti ∈ R 2 Nd is the output the self-attention layer and it will be sent to a fusion layer, described in section 2.3.1, to obtain the t -th layer passage representation h ti .
2.3. I NNER F USION L AYERS
To efﬁciently propagate evidence through the passage, we reﬁne the self-attended representations by using multiple layers.
At the end of each self-attention layer, a GRU-like gating mechanism (Hu et al., 2017) is used to decide what information to store and send to the next self-attention layer, by merging the newly produced representation of the current layer and the input representation from the previous layer,
[Equation]
where W tB , W tf are the weights, b tB , b tf are the bias of t -th fusion layer, and f t is a non-linear activation function.
The output h ti , whose dimensions are the same as its input vector B ti , is then sent to the next layer of self-attention model as input to calculate Eq 9 and Eq 10.
2. O UTPUT L AYERS
We directly follow Hu et al.
and use a memory-based answer pointer networks to predict boundary of the answer.
The memory-based answer pointer network contains multiple hops.
For the t -th hop, the pointer network produces the probability distribution of the start index p ts and the end index p te using a pointer network (Vinyals et al., 2015) respectively.
If the t -th hop is not the last hop, then the hidden states for the start and end indices are transformed and fed into the next-hop prediction.
The training loss is deﬁned as the sum of the negative log probabilities of the last hop start and end indices averaged over all examples.
E XPERIMENTS AND A NALYSIS
This paper focuses on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) to train and evaluate our model.
SQuAD, which has gained a signiﬁcant attention recently, is a [Table] largescale dataset consisting of more than 100,000 questions manually created through crowdsourcing on 536 Wikipedia articles.
The dataset is randomly partitioned into a training set (80%), a development set (10%), and a blinded test set (10%).
Two metrics are used to perform evaluation: Exact Match (EM) score which calculates the ratio of questions that are answered correctly by exact string match, and F1 score which calculates the harmonic mean of the precision and recall between predicted answers and ground true answers at the character level.
3. T RAINING D ETAILS
Our input for the encoding layer in Section 2.1 includes a list of commonly used features.
We use pre-trained GloVe 100-dimensional word vectors (Pennington et al., 2014), parts-of-speech tag features, named-entity tag feature, and binary features of exact matching (Chen et al., 2017) which indicate if a passage word can be exactly matched to any question word and vice versa.
Following Hu et al.
(2017), we also use question type (what, how, who, when, which, where, why, be, and other) features (Zhang et al., 2017) where each type is represented by a trainable embedding.
We use CNN with 100 one-dimensional ﬁlters with width 5 to encode character level embedding.
The hidden size is set as 128 for all the LSTM layers.
Dropout (Srivastava et al., 2014) are used for all the learnable parameters with a ratio as 0.2.
We use the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of 0.0006, which is halved when a bad checkpoint is met.
3. M AIN RESULTS OF MODEL COMPARISON
We compare our proposed model PhaseCond with a multi-layered attention model, the Iterative
Aligner, as well as various other recently published systems, which include a single-layered model,
BIDAF (Seo et al., 2017), and a single-layered model containing both the question-passage attention and self-attention, RNET (Wang et al., 2017).
We ﬁrst compare our proposed model PhaseCond with Iterative Aligner, which is employed by two top ranked systems MEMEN (Pan et al., 2017) and MReader (Hu et al., 2017) on the SQuAD leaderboard 1 .
Since our goal is to show the effectiveness of our proposed model PhaseCond, we use a baseline system implementing MReader for the direct comparison.
All the experiment settings are the same for PhaseCond and Iterative Aligner including the number of attention layers, input features, optimizer and learning rate, number of training steps and etc.
As shown in Table 2 which summarizes the performance of single models, we achieve steady improvements when 1) additional question encoders are used to extend the passage-question attention function, denoted as QPAtt+, as detailed in Section 2.1 and Section 2.2, and 2) on top of that, using PhaseCond making our model better than using Iterative Aligner.
Specifically, PhaseCond’s computational path for two question-aware passage attention layers L Q and two self-attention layers L S goes from L Q 1 → L Q 2 → F outer → L S 1 → F inner → L S 2 → F inner .
On the other hand, Iterative Aligner builds path in turn through different kinds of attention layers:
[Table]
As shown in Table 3, in the single model setting, our model PhaseCond is clearly more effective than all the single-layered models (BiDAF and RNET) and multi-layered models (MReader and Iterative
Aligner).
We draw the same conclusion for the ensemble model setting, despite that the RNET works better on the Dev EM measure.
The EM result of our baseline Iterative Aligner is lower
[Table] [Table]
than RNET, conﬁrming that the problem is not caused by our proposed model. Our explanations is that 1) RNET uses a different feature set (e.g., GloVe 300 dimensional word vectors are employed) and different encoding steps (e.g., three GRU layers are used for encoding question and passage representations), and 2) RNET uses a different ensemble method from our implementation.
3. A NALYSIS ON A TTENTION L AYERS
Table 4 shows the performance with different number of layers for both question-passage attention phase and self-attention phase.
We change the layer number separately to compare the performance.
For the question-passage attention phase, using single layer doesn’t degrade the performance signiﬁcantly from the default setting of two layers, resulting in a different conclusion from Hu et al.
Xiong et al.
Intuitively, this is largely expected because representing the passage repeatedly with the same question doesn’t constantly add more information.
In contrast, multiple stacking layers are needed to allow the evidence fully propagated through the passage.
This is exactly what we observed in two stacking layered self-attention phase.
In Figure 3, we visualize the attention matrices for each layer to show dynamic attention changes.
The model is based on the main setting which has two question-passage layers and two self-attention layers.
We observed several critical trends.
First, the ﬁrst layer of the question-passage attention phase can successfully align question keywords with the corresponding passage keywords, as shown in Figure 3a.
For example, the question keyword “represented” have been successfully aligned with related passage keywords “champion”, “defeated”, and “earned”.
Second, patterns of striped color in Figure 3a indicate similar weights among all the passage words, meaning that it becomes indistinguishable among passage words, and therefore adding another layer of question-passage attention model degrades the alignment quality dramatically.
This observation is meaningful which shows that repeatedly representing a passage word regarding the same question representation can make the passage embedding become closer to the original question representation.
Third, when comparing Figure 3c and Figure 3d, we observed that the color is diluted for most of the weights in the second layer of self-attention phase, meanwhile a small portion of weights is strengthened,
[Table]
[Table] [Table]
[Table] [Table]
[Figure]
Generally, the darker the color is the higher the weight is (the only exception is Figure 3b which contains negative values).
Given the question “Which NFL team represented the AFC at Super
Bowl 50?”, the system correctly detects the answer “Denver Broncos” from the passage part “The
American Football Conference (AFC) champion Denver Broncos defeated the National Football
Conference (NFC) champion Carolina Panthers 2410 to earn their third Super Bowl title.” suggesting that information propagation is converging.
For example, in Figure 3d as the last attention layer, the phrase “Denver Broncos” becomes more concentrated on the phrase “Carolina Panthers” than that of Figure 3c.
In contrast, “Denver Broncos” becomes less focused on the other keywords (e.g., “champion” and “title”) of the same passage.
C ONCLUSION
In this paper, we introduce a general framework PhaseCond, on multi-layered attention models with two phases including a question-aware passage representation phase and an evidence propagation phase.
The question-aware passage representation phase has a stack of question-aware passage attention models, followed by outer fusion layers that regularize concatenated passage representations.
The evidence propagation phase has a stack of self-attention layers, each of which is followed by inner fusion layers that control the information to propagate and output.
Also, an improved attention mechanism for PhaseCond is proposed based on a popular dot-product attention function by simultaneously encoding both the independent question embedding layers, the weight-sharing question embedding layer and weight-sharing passage embedding layer.
The experimental results show that our model signiﬁcantly outperforms single-layered or multiple-layered attention networks on blinded test data of SQuAD.
Moreover, our in-depth quantitative analysis and visualizations provide meaningful ﬁndings for both question-aware passage attention mechanism and self-matching attention mechanism.