Original: F USION N ET : F USING VIA F ULLY -A WARE A TTENTION WITH A PPLICATION TO M ACHINE C OMPREHENSION
Revised: F USION N ET : F USING VIA F ULLY -A WARE A TTENTION WITH A PPLICATION TO M ACHINE C OMPREHENSION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives.
Revised: This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives.

Original: First, it puts forward a novel concept of “history of word” to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation.
Revised: First, it puts forward a novel concept of “history of word” to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation.

Original: Second, it identiﬁes an attention scoring function that better utilizes the “history of word” concept.
Revised: Second, it identiﬁes an attention scoring function that better utilizes the “history of word” concept.

Original: Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer.
Revised: Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer.

Original: We apply FusionNet to the
Revised: We apply FusionNet to the

Original: Stanford Question Answering Dataset (SQuAD) and it achieves the ﬁrst position for both single and ensemble model on the ofﬁcial SQuAD leaderboard at the time of writing (Oct. 4th, 2017).
Revised: Stanford Question Answering Dataset (SQuAD) and it achieves the ﬁrst position for both single and ensemble model on the ofﬁcial SQuAD leaderboard at the time of writing (Oct. 4th, 2017).

Original: Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%
Revised: Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: Context: The Alpine Rhine is part of the Rhine, a famous European river. The Alpine Rhine begins in the most western part of the Swiss canton of Graubünden, and later forms the border between Switzerland to the West and Liechtenstein and later Austria to the East. On the other hand, the

Revised: Danube separates Romania and Bulgaria.

Revised: Question: What is the other country the Rhine separates Switzerland to?

Revised: Answer: Liechtenstein

Original: Teaching machines to read, process and comprehend text and then answer questions is one of key problems in artiﬁcial intelligence.
Revised: Teaching machines to read, process and comprehend text and then answer questions is one of key problems in artiﬁcial intelligence.

Original: Figure 1 gives an example of the machine reading comprehension task.
Revised: Figure 1 gives an example of the machine reading comprehension task.

Original: It feeds a machine with a piece of context and a question and teaches it to ﬁnd a correct answer to the question.
Revised: It feeds a machine with a piece of context and a question and teaches it to ﬁnd a correct answer to the question.

Original: This requires the machine to possess high capabilities in comprehension, inference and reasoning.
Revised: This requires the machine to possess high capabilities in comprehension, inference and reasoning.

Original: This is considered a challenging task in artiﬁcial intelligence and has already attracted numerous research efforts from the neural network and natural language processing communities.
Revised: This is considered a challenging task in artiﬁcial intelligence and has already attracted numerous research efforts from the neural network and natural language processing communities.

Original: Many neural network models have been proposed for this challenge and they generally frame this problem as a machine reading comprehension (MRC) task (Hochreiter &
Revised: Many neural network models have been proposed for this challenge and they generally frame this problem as a machine reading comprehension (MRC) task (Hochreiter &

Original: Schmidhuber, 1997; Wang et al., 2017; Seo et al., 2017;
Revised: Schmidhuber, 1997; Wang et al., 2017; Seo et al., 2017;

Original: Shen et al., 2017; Xiong et al., 2017; Weissenborn et al., 2017; Chen et al., 2017a).
Revised: Shen et al., 2017; Xiong et al., 2017; Weissenborn et al., 2017; Chen et al., 2017a).

Original: The key innovation in recent models lies in how to ingest information in the question and characterize it in the context, in order to provide an accurate answer to the question.
Revised: The key innovation in recent models lies in how to ingest information in the question and characterize it in the context, in order to provide an accurate answer to the question.

Original: This is often modeled as attention in the neural network community, which is a mechanism to attend the question into the context so as to ﬁnd the answer related to the question.
Revised: This is often modeled as attention in the neural network community, which is a mechanism to attend the question into the context so as to ﬁnd the answer related to the question.

Original: Some (Chen et al., 2017a; Weissenborn et al., 2017) attend the word-level embedding from the question to context, while some (Wang et al., 2017) attend the high-level representation in the question to augment the context.
Revised: Some (Chen et al., 2017a; Weissenborn et al., 2017) attend the word-level embedding from the question to context, while some (Wang et al., 2017) attend the high-level representation in the question to augment the context.

Original: However we observed that none of the existing approaches has captured the full information in the context or the question, which could be vital for complete information comprehension.
Revised: However we observed that none of the existing approaches has captured the full information in the context or the question, which could be vital for complete information comprehension.

Original: Taking image recognition as an example, information in various levels of representations can capture different aspects of details in an image: pixel, stroke and shape.
Revised: Taking image recognition as an example, information in various levels of representations can capture different aspects of details in an image: pixel, stroke and shape.

Original: We argue that this hypothesis also holds in language understanding and MRC.
Revised: We argue that this hypothesis also holds in language understanding and MRC.

Original: In other words, an approach that utilizes all the information from the word embedding level up to the highest level representation would be substantially beneﬁcial for understanding both the question and the context, hence yielding more accurate answers.
Revised: In other words, an approach that utilizes all the information from the word embedding level up to the highest level representation would be substantially beneﬁcial for understanding both the question and the context, hence yielding more accurate answers.

Original: However, the ability to consider all layers of representation is often limited by the difﬁculty to make the neural model learn well, as model complexity will surge beyond capacity.
Revised: However, the ability to consider all layers of representation is often limited by the difﬁculty to make the neural model learn well, as model complexity will surge beyond capacity.

Original: We conjectured this is why previous literature tailored their models to only consider partial information.
Revised: We conjectured this is why previous literature tailored their models to only consider partial information.

Original: To alleviate this challenge, we identify an attention scoring function utilizing all layers of representation with less training burden.
Revised: To alleviate this challenge, we identify an attention scoring function utilizing all layers of representation with less training burden.

Original: This leads to an attention that thoroughly captures the complete information between the question and the context.
Revised: This leads to an attention that thoroughly captures the complete information between the question and the context.

Original: With this fully-aware attention, we put forward a multi-level attention mechanism to understand the information in the question, and exploit it layer by layer on the context side.
Revised: With this fully-aware attention, we put forward a multi-level attention mechanism to understand the information in the question, and exploit it layer by layer on the context side.

Original: All of these innovations are integrated into a new end-to-end structure called
Revised: All of these innovations are integrated into a new end-to-end structure called

Original: FusionNet in Figure 4, with details described in Section 3.
Revised: FusionNet in Figure 4, with details described in Section 3.

Original: We submitted FusionNet to SQuAD (Rajpurkar et al., 2016), a machine reading comprehension dataset.
Revised: We submitted FusionNet to SQuAD (Rajpurkar et al., 2016), a machine reading comprehension dataset.

Original: At the time of writing (Oct. 4th, 2017), our model ranked in the ﬁrst place in both single model and ensemble model categories.
Revised: At the time of writing (Oct. 4th, 2017), our model ranked in the ﬁrst place in both single model and ensemble model categories.

Original: The ensemble model achieves an exact match (EM) score of 78.8% and F1 score of 85.9%.
Revised: The ensemble model achieves an exact match (EM) score of 78.8% and F1 score of 85.9%.

Original: Furthermore, we have tested FusionNet against adversarial SQuAD datasets (Jia & Liang, 2017).
Revised: Furthermore, we have tested FusionNet against adversarial SQuAD datasets (Jia & Liang, 2017).

Original: Results show that FusionNet outperforms existing state-of-the-art architectures in both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.
Revised: Results show that FusionNet outperforms existing state-of-the-art architectures in both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

Original: This demonstrated the exceptional performance of FusionNet.
Revised: This demonstrated the exceptional performance of FusionNet.

Original:  M ACHINE C OMPREHENSION & F ULLY -A WARE A TTENTION 
Revised:  M ACHINE C OMPREHENSION & F ULLY -A WARE A TTENTION In

Original: In this section, we brieﬂy introduce the task of machine comprehension as well as a conceptual architecture that summarizes recent advances in machine reading comprehension.
Revised: this section, we brieﬂy introduce the task of machine comprehension as well as a conceptual architecture that summarizes recent advances in machine reading comprehension.

Original: Then, we introduce a novel concept called history-of-word.
Revised: Then, we introduce a novel concept called history-of-word.

Original: History-of-word can capture different levels of contextual information to fully understand the text.
Revised: History-of-word can capture different levels of contextual information to fully understand the text.

Original: Finally, a light-weight implementation for history-of-word,
Revised: Finally, a light-weight implementation for history-of-word,

Original: Fully-Aware Attention, is proposed.
Revised: Fully-Aware Attention, is proposed.

Original: 2. T ASK D ESCRIPTION
Revised: 2. T ASK D ESCRIPTION

Original: In machine comprehension, given a context and a question, the machine needs to read and understand the context, and then ﬁnd the answer to the question.
Revised: In machine comprehension, given a context and a question, the machine needs to read and understand the context, and then ﬁnd the answer to the question.

Original: The context is described as a sequence of word tokens:
Revised: The context is described as a sequence of word tokens:

Original: C = { w C 1 , . . . , w Cm } , and the question as: Q = { w Q 1 , . . . , w Qn } , where m is the number of words in the context, and n is the number of words in the question.
Revised: C = { w C 1 , . . . , w Cm } , and the question as: Q = { w Q 1 , . . . , w Qn } , where m is the number of words in the context, and n is the number of words in the question.

Original: In general, m (cid:29) n .
Revised: In general, m (cid:29) n .

Original: The answer Ans can have different forms depending on the task.
Revised: The answer Ans can have different forms depending on the task.

Original: In the SQuAD dataset (Rajpurkar et al., 2016), the answer Ans is guaranteed to be a contiguous span in the context C , e.g., Ans = { w Ci , . . .
Revised: In the SQuAD dataset (Rajpurkar et al., 2016), the answer Ans is guaranteed to be a contiguous span in the context C , e.g., Ans = { w Ci , . . .

Original: , w Ci + k } , where k is the number of words in the answer and k ≤ m .
Revised: , w Ci + k } , where k is the number of words in the answer and k ≤ m .

Original: 2. C ONCEPTUAL A RCHITECTURE FOR M ACHINE R EADING C OMPREHENSION
Revised: 2. C ONCEPTUAL A RCHITECTURE FOR M ACHINE R EADING C OMPREHENSION

Original: In all state-of-the-art architectures for machine reading comprehension, a recurring pattern is the following process.
Revised: In all state-of-the-art architectures for machine reading comprehension, a recurring pattern is the following process.

Original: Given two sets of vectors, A and B, we enhance or modify every single vector in set A with the information from set
Revised: Given two sets of vectors, A and B, we enhance or modify every single vector in set A with the information from set

Original: B.
Revised: B.

Original: We call this a fusion process, where set B is fused into set A.
Revised: We call this a fusion process, where set B is fused into set A.

Original: Fusion processes are commonly based on attention (Bahdanau et al., 2015), but some are not.
Revised: Fusion processes are commonly based on attention (Bahdanau et al., 2015), but some are not.

Original: Major improvements in recent MRC work lie in how the fusion process is designed.
Revised: Major improvements in recent MRC work lie in how the fusion process is designed.

Original: A conceptual architecture illustrating state-of-the-art architectures is shown in Figure 2, which consists of three components.
Revised: A conceptual architecture illustrating state-of-the-art architectures is shown in Figure 2, which consists of three components.

Original: • Input vectors: Embedding vectors for each word in the context and the question.
Revised: • Input vectors: Embedding vectors for each word in the context and the question.

Original: • Integration components: The rectangular box.
Revised: • Integration components: The rectangular box.

Original: It is usually implemented using anRNN such as an LSTM (Hochreiter & Schmidhuber, 1997) or a GRU (Cho et al., 2014).
Revised: It is usually implemented using an

Original: • Fusion processes: The numbered arrows (1), (2), (2’), (3), (3’ [Table]
Revised: Architectures (1) (2) (2’) (3) (3’)

Original: Match-LSTM (Wang & Jiang, 2016)
Revised: Match-LSTM (Wang & Jiang, 2016)

Original: D
Revised: D

Original: DCN (Xiong et al., 2017)
Revised: DCN (Xiong et al., 2017)

Original: D
Revised: D

Original: D
Revised: D

Original: FastQA (Weissenborn et al., 2017)
Revised: FastQA (Weissenborn et al., 2017)

Original: D
Revised: D

Original: FastQAExt (Weissenborn et al., 2017)
Revised: FastQAExt (Weissenborn et al., 2017)

Original: D D
Revised: D D

Original: D
Revised: D

Original: BiDAF (Seo et al., 2017)
Revised: BiDAF (Seo et al., 2017)

Original: D
Revised: D

Original: D
Revised: D

Original: RaSoR (Lee et al., 2016)
Revised: RaSoR (Lee et al., 2016)

Original: D
Revised: D

Original: D
Revised: D

Original: DrQA (Chen et al., 2017a)
Revised: DrQA (Chen et al., 2017a)

Original: D
Revised: D

Original: MPCM (Wang et al., 2016)
Revised: MPCM (Wang et al., 2016)

Original: D D
Revised: D D

Original: Mnemonic Reader (Hu et al., 2017)
Revised: Mnemonic Reader (Hu et al., 2017)

Original: D D
Revised: D D

Original: D
Revised: D

Original: R-net (Wang et al., 2017)
Revised: R-net (Wang et al., 2017)

Original: D D
Revised: D D

Original: Context
Revised: Context

Original: Question
Revised: Question

Revised: • Fusion processes: The numbered arrows (1), (2), (2’), (3), (3’).

Revised: The set pointing outward is fused into the set being pointed to.

Original: There are three main types of fusion processes in recent advanced architectures.
Revised: There are three main types of fusion processes in recent advanced architectures.

Original: Table 1 shows what fusion processes are used in different state-of-the-art architectures.
Revised: Table 1 shows what fusion processes are used in different state-of-the-art architectures.

Original: We now discuss them in detail.
Revised: We now discuss them in detail.

Original: (1) Word-level fusion
Revised: (1) Word-level fusion.

Original: By providing the direct word information in question to the context, we can quickly zoom in to more related regions in the context.
Revised: By providing the direct word information in question to the context, we can quickly zoom in to more related regions in the context.

Original: However, it may not be helpful if a word has different semantic meaning based on the context.
Revised: However, it may not be helpful if a word has different semantic meaning based on the context.

Original: Many word-level fusions are not based on attention, e.g., (Hu et al., 2017; Chen et al., 2017a)
Revised: Many word-level fusions are not based on attention, e.g., (Hu et al., 2017; Chen et al., 2017a)

Original: appends binary features to context words, indicating whether each context word appears in the question
Revised: appends binary features to context words, indicating whether each context word appears in the question.

Original: (2) High-level fusion.
Revised: (2) High-level fusion.

Original: Informing the context about the semantic information in the question could help us ﬁnd the correct answer.
Revised: Informing the context about the semantic information in the question could help us ﬁnd the correct answer.

Original: But high-level information is more imprecise than word information, which may cause models 
Revised: But high-level information is more imprecise than word information, which may cause models to be less aware of details.

Original: to be less aware

Original: of details.

Original: (2’) High-level fusion (Alternative).
Revised: High-level fusion (Alternative).

Original: Similarly, we could also fuse high-level concept of Q into the word-level of C .
Revised: Similarly, we could also fuse high-level concept of Q into the word-level of C .

Original: (3) Self-boosted fusion.
Revised: (3) Self-boosted fusion.

Original: Since the context can be long and distant parts of text may rely on each other to fully understand the content, recent advances have proposed to fuse the context into itself.
Revised: Since the context can be long and distant parts of text may rely on each other to fully understand the content, recent advances have proposed to fuse the context into itself.

Original: As the context contains excessive information, one common choice is to perform self-boosted fusion after fusing the question Q .
Revised: As the context contains excessive information, one common choice is to perform self-boosted fusion after fusing the question Q .

Original: This allows us to be more aware of the regions related to the question.
Revised: This allows us to be more aware of the regions related to the question.

Original: Self-boosted fusion (Alternative).
Revised: Self-boosted fusion (Alternative).

Original: Another choice is to directly condition the self-boosted fusion process on the question Q , such as the coattention mechanism proposed in (Xiong et al., 2017).
Revised: Another choice is to directly condition the self-boosted fusion process on the question Q , such as the coattention mechanism proposed in Xiong et al Then

Revised: we

Original: Then we can perform self-boosted fusion before fusing questioninformation.
Revised: can perform self- boosted fusion before fusing question

Original:  A common trait of existing fusion mechanisms is that none of them employs all levels of representation jointly.
Revised: information. A common trait of existing fusion mechanisms is that none of them employs all levels of representation jointly.

Original: In the following, we claim that employing all levels of representation is crucial to achieving better text understanding.
Revised: In the following, we claim that employing all levels of representation is crucial to achieving better text understanding.

Original: 2. F ULLY -A WARE A TTENTION ON H ISTORY OF W ORD
Revised: 2. F ULLY -A WARE A TTENTION ON H ISTORY OF W ORD

Original: Consider the illustration shown in Figure 3.
Revised: Consider the illustration shown in Figure 3.

Original: As we read through the context, each input word will gradually transform into a more abstract representation, e.g., from low-level to high-level concepts.
Revised: As we read through the context, each input word will gradually transform into a more abstract representation, e.g., from low-level to high-level concepts.

Original: Altogether, they form the history of each word in our mental ﬂow.
Revised: Altogether, they form the history of each word in our mental ﬂow.

Original: For a human, we utilize the history-of-word so frequently but we often neglect its importance.
Revised: For a human, we utilize the history-of-word so frequently but we often neglect its importance.

Original: For example, to answer the question in Figure 3 correctly, we need to focus on both the high-level concept of forms the border and the word-level information of Alpine Rhine .
Revised: For example, to answer the question in Figure 3 correctly, we need to focus on both the high-level concept of forms the border and the word-level information of Alpine Rhine .

Original: If we focus only on the high-level concepts, we will confuse Alpine Rhine with Danube since both are European rivers that separate countries.
Revised: If we focus only on the high-level concepts, we will confuse Alpine Rhine with Danube since both are European rivers that separate countries.

Original: Therefore we hypothesize that the entire history-of-word is important to fully understand the text.
Revised: Therefore we hypothesize that the entire history-of-word is important to fully understand the text.

Original: In neural architectures, we deﬁne the history of the i -th word, HoW i , to be the concatenation of all the representations generated for this word.

Original: This may include word embedding, multiple inter-

Original: ``History of Word” Concept

Original: Alpine Rhine
Revised: Alpine Rhine

Original: European river
Revised: European river

Original: Separating River
Revised: Separating River

Original: Input Word
Revised: Input Word

Original: Low level
Revised: Low level

Original: High level
Revised: High level

Original: Danube
Revised: Danube

Original: European river
Revised: European river

Original: Separating River
Revised: Separating River

Original: Forms the border
Revised: Forms the border

Original: Border countries
Revised: Border countries

Original: Separates
Revised: Separates

Original: Question: What is the other country the Rhine separates Switzerland to?
Revised: Question: What is the other country the Rhine separates Switzerland to?

Original: Answer: Liechtenstein
Revised: Answer: Liechtenstein

Original: Context: The Alpine Rhine is part of the Rhine, a famous European river.
Revised: Context: The Alpine Rhine is part of the Rhine, a famous European river.

Original: The Alpine Rhine begins in the most western part of the Swiss canton of Graubünden, and later forms the border between
Revised: The Alpine Rhine begins in the most western part of the Swiss canton of Graubünden, and later forms the border between

Original: Switzerland to the West and Liechtenstein and later Austria to the East.
Revised: Switzerland to the West and Liechtenstein and later Austria to the East.

Original: On the other hand, the
Revised: On the other hand, the

Original: Danube separates Romania and Bulgaria.
Revised: Danube separates Romania and Bulgaria.

Original: Liechtenstein
Revised: Liechtenstein

Original: Country
Revised: Country

Revised: Country, separate

Revised: In neural architectures, we deﬁne the history of the i -th word, HoW i , to be the concatenation of all the representations generated for this word.

Original: Country, separate mediate and output hidden vectors in RNN, and corresponding representation vectors in any further layers.
Revised: This may include word embedding, multiple intermediate and output hidden vectors in RNN, and corresponding representation vectors in any further layers.

Original: To incorporate history-of-word into a wide range of neural models, we present a lightweight implementation we call Fully-Aware Attention .
Revised: To incorporate history-of-word into a wide range of neural models, we present a lightweight implementation we call Fully-Aware Attention .

Original: Attention can be applied to different scenarios.
Revised: Attention can be applied to different scenarios.

Original: To be more conclusive, we focus on attention applied to fusing information from one body to another.
Revised: To be more conclusive, we focus on attention applied to fusing information from one body to another.

Original: Consider two sets of hidden vectors for words in text bodies A and B: { h A 1 , . .
Revised: Consider two sets of hidden vectors for words in text bodies A and B: { h A 1 , . .

Original: , h Am } , { h B 1 , . . .
Revised: , h Am } , { h B 1 , . . .

Original: , h Bn } ⊂ R d .
Revised: , h Bn } ⊂ R d .

Original: Their associated history-of-word are, { HoW A 1 , . . .
Revised: Their associated history-of-word are, { HoW A 1 , . . .

Original: , HoW Am } , { HoW B 1 , . .
Revised: , HoW Am } , { HoW B 1 , . .

Original: , HoW Bn } ⊂ R d h , where d h (cid:29) d 
Revised: , HoW Bn } ⊂ R d h , where d h (cid:29) d .

Original: Fusing body B to body A via standard attention means for every h
Revised: Fusing body B to body A via standard attention means for every h

Original: Ai in body A
Revised: Ai in body A, 1.

Original: Compute an attention score S ij =
Revised: Compute an attention score S ij =

Original: S
Revised: S

Original: ( h Ai , h Bj ) ∈ R for each h
Revised: ( h Ai , h Bj ) ∈ R for each h

Original: Bj in body B. 2.
Revised: Bj in body B. 2.

Original: Form the attention weight α ij through softmax: α ij = exp( S ij ) / (cid:80) k exp( S ik ) .
Revised: Form the attention weight α ij through softmax: α ij = exp( S ij ) / (cid:80) k exp( S ik ) .

Original: Concatenate h
Revised: Concatenate h

Original: Ai with the summarized information, ˆ h A i
Revised: Ai with the summarized information, ˆ h A i

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: j α ij h
Revised: [Equation]

Original: Bj .
Revised: Bj .

Original: In fully-aware attention, we replace attention score computation with the history-of-word.
Revised: In fully-aware attention, we replace attention score computation with the history-of-word.

Original: [Equation]
Revised: [Equation]

Original: This allows us to be fully aware of the complete understanding of each word.
Revised: This allows us to be fully aware of the complete understanding of each word.

Original: The ablation study in
Revised: The ablation study in

Original: Section 4.4 demonstrates that this lightweight enhancement offers a decent improvement in performance.
Revised: Section 4.4 demonstrates that this lightweight enhancement offers a decent improvement in performance.

Original: To fully utilize history-of-word in attention, we need a suitable attention scoring function S ( x , y ) .
Revised: To fully utilize history-of-word in attention, we need a suitable attention scoring function S ( x , y ) .

Original: A commonly used function is multiplicative attention (Britz et al., 2017): x T U T V y , leading to
Revised: A commonly used function is multiplicative attention (Britz et al., 2017): x T U T V y , leading to

Original: [Equation]
Revised: [Equation]

Original: where U, V ∈ R k × d h , and k is the attention hidden size.
Revised: where U, V ∈ R k × d h , and k is the attention hidden size.

Original: However, we suspect that two large matrices interacting directly will make the neural model harder to train.
Revised: However, we suspect that two large matrices interacting directly will make the neural model harder to train.

Original: Therefore we propose to constrain the matrix U T V to be symmetric.
Revised: Therefore we propose to constrain the matrix U T V to be symmetric.

Original: A symmetric matrix can always be decomposed into
Revised: A symmetric matrix can always be decomposed into

Original: U T DU , thus
Revised: U T DU , thus

Original: [Equation]
Revised: [Equation]

Original: where U ∈ R k × d h , D ∈ R k × k and D is a diagonal matrix.
Revised: where U ∈ R k × d h , D ∈ R k × k and D is a diagonal matrix.

Original: The symmetric form retains the ability to give high attention score between dissimilar HoW Ai , HoW Bj .
Revised: The symmetric form retains the ability to give high attention score between dissimilar HoW Ai , HoW Bj .

Original: Additionally, we marry nonlinearity with the symmetric form to provide richer interaction among different parts of the history-of-word.
Revised: Additionally, we marry nonlinearity with the symmetric form to provide richer interaction among different parts of the history-of-word.

Original: The ﬁnal formulation for attention score is
Revised: The ﬁnal formulation for attention score is

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) is an activation function applied element-wise.
Revised: where f ( x ) is an activation function applied element-wise.

Original: In the following context, we employ f ( x ) = max(0 , x ) .
Revised: In the following context, we employ f ( x ) = max(0 , x ) .

Original: A detailed ablation study in Section 4 demonstrates its advantage over many alternatives.
Revised: A detailed ablation study in Section 4 demonstrates its advantage over many alternatives.

Original: BiLSTM … …
Revised: BiLSTM … …

Original: Fully-Aware Self-Boosted Fusion
Revised: Fully-Aware Self-Boosted Fusion

Original: Context Understanding …
Revised: Context Understanding …

Original: Question Understanding
Revised: Question Understanding

Original: The above can be used to capture long range info.
Revised: The above can be used to capture long range info.

Original: BiLSTM Fully-Aware Fusion Network … …
Revised: BiLSTM Fully-Aware Fusion Network … …

Original: Context
Revised: Context

Original: Question … … … … … …
Revised: Question … … … … … …

Original: Word-level
Revised: Word-level

Original: Fully-Aware Multi-level Fusion
Revised: Fully-Aware Multi-level Fusion

Original: Input Vector
Revised: Input Vector

Original: Low-level Concept
Revised: Low-level Concept

Original: High-level Concept : : : Understanding … … 7-Level Fully-Aware Attention … … 3-Level Fully-Aware Attention : :
Revised: High-level Concept : : : Understanding … … 7-Level Fully-Aware Attention … … 3-Level Fully-Aware Attention : :

Original: Context Understanding
Revised: Context Understanding

Original: Question Understanding
Revised: Question Understanding

Original: Concat w/ Multi-level Q Infor.
Revised: Concat w/ Multi-level Q Infor.

Original: Concat w/ Distant C Infor.
Revised: Concat w/ Distant C Infor.

Original: C2)

Original:  F ULLY -A WARE F USION N ETWORK 3. E ND - TO - END A RCHITECTURE
Revised:  F ULLY -A WARE F USION N ETWORK 3. E ND - TO - END A RCHITECTURE

Original: Based on fully-aware attention, we propose an end-to-end architecture: the fully-aware fusion network (FusionNet).
Revised: Based on fully-aware attention, we propose an end-to-end architecture: the fully-aware fusion network (FusionNet).

Original: Given text A and B, FusionNet fuses information from text B to text A and generates two set of vectors
Revised: Given text A and B, FusionNet fuses information from text B to text A and generates two set of vectors

Original: [Equation]
Revised: [Equation]

Original: In the following, we consider the special case where text A is context C and text B is question Q .
Revised: In the following, we consider the special case where text A is context C and text B is question Q .

Original: An illustration for FusionNet is shown in Figure 4.
Revised: An illustration for FusionNet is shown in Figure 4.

Original: It consists of the following components.
Revised: It consists of the following components.

Original: Input Vectors.
Revised: Input Vectors.

Original: First, each word in C and Q is transformed into an input vector w .
Revised: First, each word in C and Q is transformed into an input vector w .

Original: We utilize the 300-dim GloVe embedding (Pennington et al., 2014) and 600-dim contextualized vector (McCann et al., 2017).
Revised: We utilize the 300-dim GloVe embedding (Pennington et al., 2014) and 600-dim contextualized vector (McCann et al., 2017).

Original: In the SQuAD task, we also include 12 -dim POS embedding, 8 -dim NER embedding and a normalized term frequency for context C as suggested in (Chen et al., 2017a).
Revised: In the SQuAD task, we also include 12 -dim POS embedding, 8 -dim NER embedding and a normalized term frequency for context C as suggested in (Chen et al., 2017a).

Original: Together { w C 1 , . . .
Revised: Together { w C 1 , . . .

Original: , w Cm } ⊂ R 900+20+1 , and { w Q 1 , . . .
Revised: , w Cm } ⊂ R 900+20+1 , and { w Q 1 , . . .

Original: , w Qn } ⊂ R 900 .
Revised: , w Qn } ⊂ R 900 .

Original: Fully-Aware Multi-level Fusion: Word-level.
Revised: Fully-Aware Multi-level Fusion: Word-level.

Original: In multi-level fusion, we separately consider fusing word-level and higher-level.
Revised: In multi-level fusion, we separately consider fusing word-level and higher-level.

Original: Word-level fusion informs C about what kind of words are in Q .
Revised: Word-level fusion informs C about what kind of words are in Q .

Original: It is illustrated as arrow (1) in Figure 2.
Revised: It is illustrated as arrow (1) in Figure 2.

Original: For this component, we follow the approach in (Chen et al., 2017a)
Revised: For this component, we follow the approach in (Chen et al., 2017a)

Original: First, a feature vector em
Revised: First, a feature vector em

Original: i is created for each word in C to indicate whether the word occurs in the question Q .
Revised: i is created for each word in C to indicate whether the word occurs in the question Q .

Original: Second, attention-based fusion on GloVe embedding g
Revised: Second, attention-based fusion on GloVe embedding g

Original: i is used
Revised: i is used

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R 300 × 300 .
Revised: where W ∈ R 300 × 300 .

Original: Since history-of-word is the input vector itself, fully-aware attention is not employed here.
Revised: Since history-of-word is the input vector itself, fully-aware attention is not employed here.

Original: The enhanced input vector for context is ˜ w Ci = [ w Ci ; em i ; ˆ g Ci ] .
Revised: The enhanced input vector for context is ˜ w Ci = [ w Ci ; em i ; ˆ g Ci ] .

Original: Reading.
Revised: Reading.

Original: In the reading component, we use a separate bidirectional LSTM (BiLSTM) to form low-level and high-level concepts for C and Q .
Revised: In the reading component, we use a separate bidirectional LSTM (BiLSTM) to form low-level and high-level concepts for C and Q .

Original: [Equation]
Revised: [Equation]

Original: Hence low-level and high-level concepts h l , h h ∈ R 250 are created for each word.
Revised: Hence low-level and high-level concepts h l , h h ∈ R 250 are created for each word.

Original: Question Understanding.
Revised: Question Understanding.

Original: In the Question Understanding component, we apply a new BiLSTM taking in both h Ql , h Qh to obtain the ﬁnal question representation U Q :
Revised: In the Question Understanding component, we apply a new BiLSTM taking in both h Ql , h Qh to obtain the ﬁnal question representation U Q :

Original: [Equation]
Revised: [Equation]

Original: where { u Qi ∈ R 250 } ni =1 are the understanding vectors for Q .
Revised: where { u Qi ∈ R 250 } ni =1 are the understanding vectors for Q .

Original: Fully-Aware Multi-level Fusion: Higher-level.
Revised: Fully-Aware Multi-level Fusion: Higher-level.

Original: This component fuses all higher-level information in the question Q to the context C through fully-aware attention on history-of-word.
Revised: This component fuses all higher-level information in the question Q to the context C through fully-aware attention on history-of-word.

Original: Since the proposed attention scoring function for fully-aware attention is constrained to be symmetric, we need to identify the common history-of-word for both C , Q .
Revised: Since the proposed attention scoring function for fully-aware attention is constrained to be symmetric, we need to identify the common history-of-word for both C , Q .

Original: This yields
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: where g i is the GloVe embedding and c i is the CoVe embedding.
Revised: where g i is the GloVe embedding and c i is the CoVe embedding.

Original: Then we fuse low, high, and understanding-level information from Q to C via fully-aware attention .
Revised: Then we fuse low, high, and understanding-level information from Q to C via fully-aware attention .

Original: Different sets of attention weights are calculated through attention function S l ( x , y ) , S h ( x , y ) , S u ( x , y ) to combine low, high, and understanding-level of concepts.
Revised: Different sets of attention weights are calculated through attention function S l ( x , y ) , S h ( x , y ) , S u ( x , y ) to combine low, high, and understanding-level of concepts.

Original: All three functions are the proposed symmetric form with nonlinearity in Section 2.3, but are parametrized by independent parameters to attend to different regions for different level.
Revised: All three functions are the proposed symmetric form with nonlinearity in Section 2.3, but are parametrized by independent parameters to attend to different regions for different level.

Original: Attention hidden size is set to be k = 250 .
Revised: Attention hidden size is set to be k = 250 .

Original: Low-level fusion:
Revised: Low-level fusion

Original: [Equation]
Revised: [Equation]

Original: High-level fusion: ˆh
Revised: High-level fusion: ˆ

Original: [Equation]
Revised: [Equation]

Original: Understanding fusion: ˆu
Revised: Understanding fusion: ˆ

Original: [Equation]

Original: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: This multi-level attention mechanism captures different levels of information independently, while taking all levels of information into account.
Revised: This multi-level attention mechanism captures different levels of information independently, while taking all levels of information into account.

Original: A new BiLSTM is applied to obtain the representation for C fully fused with information in the question Q :
Revised: A new BiLSTM is applied to obtain the representation for C fully fused with information in the question Q :

Original: [Equation]
Revised: [Equation]

Original: Fully-Aware Self-Boosted Fusion.
Revised: Fully-Aware Self-Boosted Fusion.

Original: We now use self-boosted fusion to consider distant parts in the context, as illustrated by arrow (3) in Figure 2.
Revised: We now use self-boosted fusion to consider distant parts in the context, as illustrated by arrow (3) in Figure 2.

Original: Again, we achieve this via fully-aware attention on history-of-word.
Revised: Again, we achieve this via fully-aware attention on history-of-word.

Original: We identify the history-of-word to be
Revised: We identify the history-of-word to be

Original: [Equation]
Revised: [Equation]

Original: We then perform fully-aware attention, ˆ vCi = (cid:80)
Revised: We then perform fully-aware attention, ˆ v

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal context representation is obtained by
Revised: The ﬁnal context representation is obtained by

Original: [Equation]
Revised: [Equation]

Original: where { u Ci ∈ R 250 } mi =1 are the understanding vectors for C .
Revised: where { u Ci ∈ R 250 } mi =1 are the understanding vectors for C .

Original: After these components in FusionNet, we have created the understanding vectors, U C , for the context C , which are fully fused with the question Q .
Revised: After these components in FusionNet, we have created the understanding vectors, U C , for the context C , which are fully fused with the question Q .

Original: We also have the understanding vectors, U Q , for the question Q .
Revised: We also have the understanding vectors, U Q , for the question Q .

Original: 3. A PPLICATION IN M ACHINE C OMPREHENSION
Revised: 3. A PPLICATION IN M ACHINE C OMPREHENSION

Original: We focus particularly on the output format in SQuAD (Rajpurkar et al., 2016) where the answer is always a span in the context.
Revised: We focus particularly on the output format in SQuAD (Rajpurkar et al., 2016) where the answer is always a span in the context.

Original: The output of FusionNet are the understanding vectors for both
Revised: The output of FusionNet are the understanding vectors for both

Original: C and Q , U C
Revised: C and Q , U C

Original: = { u C 1 , . . .
Revised: = { u C 1 , . . .

Original: , u Cm } , U Q = { u Q 1 , . . .
Revised: , u Cm } , U Q = { u Q 1 , . . .

Original: , u Qn } .
Revised: , u Qn } .

Original: We then use them to ﬁnd the answer span in the context.
Revised: We then use them to ﬁnd the answer span in the context.

Original: Firstly, a single summarized question understanding vector is obtained through u Q =
Revised: Firstly, a single summarized question understanding vector is obtained through u Q =

Original: (cid:80) i β i u Qi , where β i ∝ exp( w T u Qi ) and w is a trainable vector.
Revised: (cid:80) i β i u Qi , where β i ∝ exp( w T u Qi ) and w is a trainable vector.

Original: Then we attend for the span start using the summarized question understanding vector u Q ,
Revised: Then we attend for the span start using the summarized question understanding vector u Q ,

Original: [Equation]
Revised: [Equation]

Original: where W S ∈ R
Revised: where W S ∈ R

Original: d × d is a trainable matrix.
Revised: d × d is a trainable matrix.

Original: To use the information of the span start when we attend for the span end, we combine the context understanding vector for the span start with u Q through a GRU (Cho et al., 2014), v Q =
Revised: To use the information of the span start when we attend for the span end, we combine the context understanding vector for the span start with u Q through a GRU (Cho et al., 2014), v Q =

Original: GRU( u Q , (cid:80) i P Si u Ci ) , where u Q is taken as the memory and (cid:80) i P
Revised: GRU( u Q , (cid:80) i P Si u Ci ) , where u Q is taken as the memory and (cid:80) i P

Original: Si u Ci as the input in GRU.
Revised: Si u Ci as the input in GRU.

Original: Finally we attend for the end of the span using v Q ,
Revised: Finally we attend for the end of the span using v Q ,

Original: P Ei

Original: [Equation]

Original: [Equation]

Original: v Q

Original: [Equation]
Revised: [Equation]

Original: T W E u Ci ) , where W E ∈ R
Revised: u Ci ) , where W E ∈ R

Original: d × d is another trainable matrix.
Revised: d × d is another trainable matrix.

Original: Training.
Revised: Training.

Original: During training, we maximize the log probabilities of the ground truth span start and end, (cid:80) k
Revised: During training, we maximize the log probabilities of the ground truth span start and end, (cid:80) k

Original: (log( P Si sk ) + log( P Ei ek )) , where i sk , i ek are the answer span for the k -th instance.
Revised: (log( P Si sk ) + log( P Ei ek )) , where i sk , i ek are the answer span for the k -th instance.

Original: Prediction.
Revised: Prediction.

Original: We predict the answer span to be i s , i e with the maximum P Si s P
Revised: We predict the answer span to be i s , i e with the maximum P Si s P

Original: Ei e under the constraint 0 ≤ i e − i s ≤ 15 .
Revised: Ei e under the constraint 0 ≤ i e − i s ≤ 15 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrst present the datasets used for evaluation.
Revised: In this section, we ﬁrst present the datasets used for evaluation.

Original: Then we compare our end-toend FusionNet model with existing machine reading models.
Revised: Then we compare our end-toend FusionNet model with existing machine reading models.

Original: Finally, we conduct experiments to validate the effectiveness of our proposed components.
Revised: Finally, we conduct experiments to validate the effectiveness of our proposed components.

Original: Additional ablation study on input vectors can be found in Appendix C. Detailed experimental settings can be found in Appendix F.
Revised: Additional ablation study on input vectors can be found in Appendix C. Detailed experimental settings can be found in Appendix E.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We focus on the SQuAD dataset (Rajpurkar et al., 2016) to train and evaluate our model.
Revised: We focus on the SQuAD dataset (Rajpurkar et al., 2016) to train and evaluate our model.

Original: SQuAD is a popular machine comprehension dataset consisting of 100,000+ questions created by crowd workers on 536 Wikipedia articles.
Revised: SQuAD is a popular machine comprehension dataset consisting of 100,000+ questions created by crowd workers on 536 Wikipedia articles.

Original: Each context is a paragraph from an article and the answer to each question is guaranteed to be a span in the context.
Revised: Each context is a paragraph from an article and the answer to each question is guaranteed to be a span in the context.

Original: While rapid progress has been made on SQuAD, whether these systems truly understand language remains unclear.
Revised: While rapid progress has been made on SQuAD, whether these systems truly understand language remains unclear.

Original: In a recent paper, Jia & Liang (2017) proposed several adversarial schemes to test the understanding of the systems.
Revised: In a recent paper, Jia & Liang (2017) proposed several adversarial schemes to test the understanding of the systems.

Original: We will use the following two adversarial datasets, AddOneSent and AddSent, to evaluate our model.
Revised: We will use the following two adversarial datasets, AddOneSent and AddSent, to evaluate our model.

Original: For both datasets, a confusing sentence is appended at the end of the context.
Revised: For both datasets, a confusing sentence is appended at the end of the context.

Original: The appended sentence is model-independent for AddOneSent, while AddSent requires querying the model a few times to choose the most confusing sentence.
Revised: The appended sentence is model-independent for AddOneSent, while AddSent requires querying the model a few times to choose the most confusing sentence.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: We submitted our model to SQuAD for evaluation on the hidden test set.
Revised: We submitted our model to SQuAD for evaluation on the hidden test set.

Original: We also tested the model on the adversarial SQuAD datasets.
Revised: We also tested the model on the adversarial SQuAD datasets.

Original: Two ofﬁcial evaluation criteria are used: Exact Match (EM) and
Revised: Two ofﬁcial evaluation criteria are used: Exact Match (EM) and

Original: F1 score.
Revised: F1 score.

Original: EM measures how many predicted answers exactly match the correct answer, while F score measures the weighted average of the precision and recall at token level.
Revised: EM measures how many predicted answers exactly match the correct answer, while F score measures the weighted average of the precision and recall at token level.

Original: The evaluation results for our model and other competing approaches are shown in Table 2.
Revised: The evaluation results for our model and other competing approaches are shown in Table 2.

Original: 1 Additional comparisons with state-of-the-art models in the literature can be found in Appendix A.
Revised: 1 Additional comparisons with state-of-the-art models in the literature can be found in Appendix A.

Original: For the two adversarial datasets, AddOneSent and AddSent, the evaluation criteria is the same as
Revised: For the two adversarial datasets, AddOneSent and AddSent, the evaluation criteria is the same as

Original: SQuAD.
Revised: SQuAD.

Original: However, all models are trained only on the original SQuAD, so the model never sees the
Revised: However, all models are trained only on the original SQuAD, so the model never sees the

Original: [Table]
Revised: [Table]

Revised: LR Baseline (Rajpurkar et al., 2016)

Revised: [Table]

Revised: Match

Revised: [Table]

Revised: LSTM (Wang & Jiang, 2016)

Revised: [Table]

Revised: BiDAF (Seo

Revised: [Table]

Revised: [Table]

Revised: SEDT (Liu

Revised: [Table]

Revised: [Table]

Revised: RaSoR (Lee

Revised: [Table]

Revised: [Table]

Revised: DrQA (Chen

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ReasoNet (Shen et al.,

Revised: [Table]

Revised: [Table]

Revised: R. Mnemonic Reader (Hu

Revised: [Table]

Revised: al

Revised: [Table]

Revised: [Table]

Revised: DCN+

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ReasoNet (Shen

Revised: [Table]

Revised: al

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: MEMEN

Revised: [Table]

Revised: [Table]

Revised: Mnemonic

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: DCN

Original: [Table]
Revised: [Table]

Revised: Human (Rajpurkar et al., 2016)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ReasoNet (E)

Original: [Table]
Revised: [Table]

Revised: (E)

Revised: [Table]

Revised: [Table]

Original: adversarial datasets during training.
Revised: adversarial datasets during training.

Original: The results for AddSent and AddOneSent are shown in Tableand Table 4, respectively.
Revised: The results for AddSent and AddOneSent are shown in Tableand Table 4, respectively.

Original: From the results, we can see that our models not only perform well on the original SQuAD dataset, but also outperform all previous models by more than 5% in EM score on the adversarial datasets.
Revised: From the results, we can see that our models not only perform well on the original SQuAD dataset, but also outperform all previous models by more than 5% in EM score on the adversarial datasets.

Original: This shows that FusionNet is better at language understanding of both the context and question.
Revised: This shows that FusionNet is better at language understanding of both the context and question.

Original: 4. C OMPARISON ON A TTENTION F UNCTION
Revised: 4. C OMPARISON ON A TTENTION F UNCTION

Original: In this experiment, we compare the performance of different attention scoring functions S ( x , y ) for fully-aware attention.
Revised: In this experiment, we compare the performance of different attention scoring functions S ( x , y ) for fully-aware attention.

Original: We utilize the end-to-end architecture presented in Section 3.1.
Revised: We utilize the end-to-end architecture presented in Section 3.1.

Original: Fully-aware attention is used in two places, fully-aware multi-level fusion: higher level and fully-aware selfboosted fusion .
Revised: Fully-aware attention is used in two places, fully-aware multi-level fusion: higher level and fully-aware selfboosted fusion .

Original: Word-level fusion remains unchanged.
Revised: Word-level fusion remains unchanged.

Original: Based on the discussion in Section 2.3, we consider the following formulations for comparison:
Revised: Based on the discussion in Section 2.3, we consider the following formulations for comparison:

Original: Additive attention (MLP) (Bahdanau et al., 2015): s T tanh( W 1 x + W 2 y ) .
Revised: Additive attention (MLP) (Bahdanau et al., 2015): s T tanh( W 1 x + W 2 y ) .

Original: Multiplicative attention: x T U T V
Revised: Multiplicative attention: x T U T V

Original: y .
Revised: y .

Original: Scaled multiplicative attention: 1 √ k x T U T V y , where k is the attention hidden size.
Revised: Scaled multiplicative attention: 1 √ k x T U T V y , where k is the attention hidden size.

Original: It is proposed in (Vaswani et al., 2017).
Revised: It is proposed in (Vaswani et al., 2017).

Original: Scaled multiplicative with nonlinearity: 1 √ k f ( U x ) T f
Revised: Scaled multiplicative with nonlinearity: 1 √ k f ( U x ) T f

Original: ( V y ) .
Revised: ( V y ) .

Original: Our proposed symmetric form: x T U T DU y , where D is diagonal.
Revised: Our proposed symmetric form: x T U T DU y , where D is diagonal.

Original: Proposed symmetric form with nonlinearity: f( U x ) T D f
Revised: Proposed symmetric form with nonlinearity: f

Original: ( U  y ) .
Revised: ( U x ) T D f ( U y ) .

Original: We consider the activation function f ( x ) to be max(0 , x ) .
Revised: We consider the activation function f ( x ) to be max(0 , x ) .

Original: The results of various attention functions on SQuAD development set are shown in Table 5.
Revised: The results of various attention functions on SQuAD development set are shown in Table 5.

Original: It is clear that the symmetric form consistently outperforms all alternatives.
Revised: It is clear that the symmetric form consistently outperforms all alternatives.

Original: We attribute this gain to the fact that symmetric form has a single large
Revised: We attribute this gain to the fact that symmetric form has a single large

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: demonstrates the effectiveness of history-of-word.

Original: matrix U .
Revised: matrix U .

Original: All other alternatives have two large parametric matrices.
Revised: All other alternatives have two large parametric matrices.

Original: During optimization, these two parametric matrices would interfere with each other and it will make the entire optimization process challenging.
Revised: During optimization, these two parametric matrices would interfere with each other and it will make the entire optimization process challenging.

Original: Besides, by constraining U T V to be a symmetric matrix U T DU , we retain the ability for x to attend to dissimilar y .
Revised: Besides, by constraining U T V to be a symmetric matrix U T DU , we retain the ability for x to attend to dissimilar y .

Original: Furthermore, its marriage with the nonlinearity continues to signiﬁcantly boost the performance.
Revised: Furthermore, its marriage with the nonlinearity continues to signiﬁcantly boost the performance.

Original: 4. E FFECTIVENESS OF H ISTORY - OF -W ORD
Revised: 4. E FFECTIVENESS OF H ISTORY - OF -W ORD

Original: In FusionNet, we apply the history-of-word and fully-aware attention in two major places to achieve good performance: multi-level fusion and self-boosted fusion.
Revised: In FusionNet, we apply the history-of-word and fully-aware attention in two major places to achieve good performance: multi-level fusion and self-boosted fusion.

Original: In this section, we present experiments to demonstrate the effectiveness of our application.
Revised: In this section, we present experiments to demonstrate the effectiveness of our application.

Original: In the experiments, we ﬁx the attention function to be our proposed symmetric form with nonlinearity due to its good performance shown in Section 4.3.
Revised: In the experiments, we ﬁx the attention function to be our proposed symmetric form with nonlinearity due to its good performance shown in Section 4.3.

Original: The results are shown in Table 6, and the details for each conﬁguration can be found in Appendix B.[Table]
Revised: The results are shown in Table 6, and the details for each conﬁguration can be found in Appendix B.

Original: -Level is a vanilla model where only the high-level information is fused from Q to C via standard attention.
Revised: High-Level is a vanilla model where only the high-level information is fused from Q to C via standard attention.

Original: When placed in the conceptual architecture (Figure 2), it only contains arrow (2) without any other fusion processes.
Revised: When placed in the conceptual architecture (Figure 2), it only contains arrow (2) without any other fusion processes.

Original: FA High-Level is the High-Level model with standard attention replaced by fully-aware attention.
Revised: FA High-Level is the High-Level model with standard attention replaced by fully-aware attention.

Original: FA All-Level is a naive extension of FA High-Level , where all levels of information are concatenated and is fused into the context using the same attention weight.
Revised: FA All-Level is a naive extension of FA High-Level , where all levels of information are concatenated and is fused into the context using the same attention weight.

Original: FA Multi-Level is our proposed Fully-aware Multi-level fusion, where different levels of information are attended under separate attention weight.
Revised: FA Multi-Level is our proposed Fully-aware Multi-level fusion, where different levels of information are attended under separate attention weight.

Original: Self C
Revised: Self C

Original: =
Revised: =

Original: None means we do not make use of self-boosted fusion.
Revised: None means we do not make use of self-boosted fusion.

Original: Self C =
Revised: Self C =

Original: Normal means we employ a standard attention-based self-boosted fusion after fusing question to context.
Revised: Normal means we employ a standard attention-based self-boosted fusion after fusing question to context.

Original: This is illustrated as arrow (3) in the conceptual architecture (Figure 2).
Revised: This is illustrated as arrow (3) in the conceptual architecture (Figure 2).

Original: Self C =
Revised: Self C =

Original: FA means we enhance the self-boosted fusion with fully-aware attention.
Revised: FA means we enhance the self-boosted fusion with fully-aware attention.

Original: High-Level vs. FA High-Level .
Revised: High-Level vs. FA High-Level .

Original: From Table 6, we can see that High-Level performs poorly as expected.
Revised: From Table 6, we can see that High-Level performs poorly as expected.

Original: However enhancing this vanilla model with fully-aware attention signiﬁcantly increase the performance by more than 8% .
Revised: However enhancing this vanilla model with fully-aware attention signiﬁcantly increase the performance by more than 8% .

Original: The performance of FA High-Level already outperforms many state-of-the-art MRC models.
Revised: The performance of FA High-Level already outperforms many state-of-the-art MRC models.

Original: This clearly demonstrates the power of fully-aware attention.
Revised: This clearly demonstrates the power of fully-aware attention.

Original: FA All-Level vs. FA Multi-Level .
Revised: FA All-Level vs. FA Multi-Level .

Original: Next, we consider models that fuse all levels of information from question Q to context C .
Revised: Next, we consider models that fuse all levels of information from question Q to context C .

Original: FA All-Level is a naive extension of FA High-Level , but its performance is actually worse than FA High-Level .
Revised: FA All-Level is a naive extension of FA High-Level , but its performance is actually worse than FA High-Level .

Original: However, by fusing different parts of history-of-word in Q independently as in FA Multi-Level , we are able to further improve the performance.
Revised: However, by fusing different parts of history-of-word in Q independently as in FA Multi-Level , we are able to further improve the performance.

Original: Self C options.
Revised: Self C options.

Original: We have achieved decent performance without self-boosted fusion.
Revised: We have achieved decent performance without self-boosted fusion.

Original: Now, we compare adding normal and fully-aware self-boosted fusion into the architecture.
Revised: Now, we compare adding normal and fully-aware self-boosted fusion into the architecture.

Original: Comparing None and
Revised: Comparing None and

Original: Normal in Table 6, we can see that the use of normal self-boosted fusion is not very effective under our improved C , Q Fusion .
Revised: Normal in Table 6, we can see that the use of normal self-boosted fusion is not very effective under our improved C , Q Fusion .

Original: Then by comparing with FA , it is clear that through the enhancement of fully-aware attention, the enhanced self-boosted fusion can provide considerable improvement.
Revised: Then by comparing with FA , it is clear that through the enhancement of fully-aware attention, the enhanced self-boosted fusion can provide considerable improvement.

Original: Together, these experiments demonstrate that the ability to take all levels of understanding as a whole is crucial for machines to better understand the text.
Revised: Together, these experiments demonstrate that the ability to take all levels of understanding as a whole is crucial for machines to better understand the text.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we describe a new deep learning model called FusionNet with its application to machine comprehension.
Revised: In this paper, we describe a new deep learning model called FusionNet with its application to machine comprehension.

Original: FusionNet proposes a novel attention mechanism with following three contributions: 1.
Revised: FusionNet proposes a novel attention mechanism with following three contributions: 1.

Original: the concept of history-of-word to build the attention using complete information from the lowest word-level embedding up to the highest semantic-level representation; 2.
Revised: the concept of history-of-word to build the attention using complete information from the lowest word-level embedding up to the highest semantic-level representation; 2.

Original: an attention scoring function to effectively and efﬁciently utilize history-of-word; 3.
Revised: an attention scoring function to effectively and efﬁciently utilize history-of-word; 3.

Original: a fully-aware multi-level fusion to exploit information layer by layer discriminatingly.
Revised: a fully-aware multi-level fusion to exploit information layer by layer discriminatingly.

Original: We applied FusionNet to MRC task and experimental results show that FusionNet outperforms existing machine reading models on both the SQuAD dataset and the adversarial SQuAD dataset.
Revised: We applied FusionNet to MRC task and experimental results show that FusionNet outperforms existing machine reading models on both the SQuAD dataset and the adversarial SQuAD dataset.

Original: We believe FusionNet is a general and improved attention mechanism and can be applied to many tasks.
Revised: We believe FusionNet is a general and improved attention mechanism and can be applied to many tasks.

Original: Our future work is to study its capability in other NLP problems.
Revised: Our future work is to study its capability in other NLP problems.

Revised: A CKNOWLEDGMENTS

Revised: We would like to thank Paul Mineiro, Sebastian Kochman, Pengcheng He, Jade Huang and Jingjing

Revised: Liu from Microsoft Business AI, Mac-Antoine Rondeau from Maluuba and the anonymous reviewers for their valuable comments and tremendous help in this paper.

Original: A C OMPARISON WITH P UBLISHED M ODELS
Revised: A C OMPARISON WITH P UBLISHED M ODELS

Original: In this appendix, we compare with published state-of-the-art architectures on the SQuAD dev set.
Revised: In this appendix we compare with published state-of-the-art architectures on the SQuAD dev set.

Original: The comparison is shown in Figure 5 and 6 for EM and F1 score respectively.
Revised: The comparison is shown in Figure 5 and 6 for EM and F1 score respectively.

Original: The performance of
Revised: The performance of

Original: FusionNet is shown under different training epochs.
Revised: FusionNet is shown under different training epochs.

Original: Each epoch loops through all the examples in the training set once.
Revised: Each epoch loops through all the examples in the training set once.

Original: On a single NVIDIA GeForce GTX Titan X GPU, each epoch took roughly 20 minutes when batch size 32 is used.
Revised: On a single NVIDIA GeForce GTX Titan X GPU, each epoch took roughly 20 minutes when batch size 32 is used.

Original: The state-of-the-art models compared in this experiment include:
Revised: The state-of-the-art models compared in this experiment include:

Original: Published version of R-net in their technical report (Wang et al., 2017), 2.
Revised: Published version of R-net in their technical report (Wang et al., 2017), 2.

Original: Reinforced Mnemonic Reader (Hu et al., 2017), 3.
Revised: Reinforced Mnemonic Reader (Hu et al., 2017), 3.

Original: MEMEN (Pan et al., 2017), 4.
Revised: MEMEN (Pan et al., 2017), 4.

Original: ReasoNet (Shen et al., 2017), 5.
Revised: ReasoNet (Shen et al., 2017), 5.

Original: Document reader (DrQA)
Revised: Document reader (DrQA)

Original: (Chen et al., 2017a), 6.
Revised: (Chen et al., 2017a), 6.

Original: DCN (Xiong et al., 2017), 7.
Revised: DCN (Xiong et al., 2017), 7.

Original: DCN + character embedding (Char) + CoVe (McCann et al., 2017), 8.
Revised: DCN + character embedding (Char) + CoVe (McCann et al., 2017), 8.

Original: BiDAF (Seo et al., 2017), 9.
Revised: BiDAF (Seo et al., 2017), 9.

Original: the best-performing variant of Match-LSTM (Wang & Jiang, 2016).
Revised: the best-performing variant of Match-LSTM (Wang & Jiang, 2016).

Original: B D ETAILED C ONFIGURATIONS IN THE A BLATION S TUDY
Revised: B D ETAILED C ONFIGURATIONS IN THE A BLATION S TUDY

Original: In this appendix, we present details for the conﬁgurations used in the ablation study in Section 4.4.
Revised: In this appendix, we present details for the conﬁgurations used in the ablation study in Section 4.4.

Original: For all conﬁgurations, the understanding vectors for both the context C and the question Q will be generated, then we follow the same output architecture in Section 3.2 to apply them to machine reading comprehension problem.
Revised: For all conﬁgurations, the understanding vectors for both the context C and the question Q will be generated, then we follow the same output architecture in Section 3.2 to apply them to machine reading comprehension problem.

Original: High-Level.
Revised: High-Level.

Original: Firstly, context words and question words are transformed into input vectors in the same way as FusionNet,
Revised: Firstly, context words and question words are transformed into input vectors in the same way as FusionNet,

Original: [Equation]
Revised: [Equation]

Original: Then we pass them independently to two layers of BiLSTM.
Revised: Then we pass them independently to two layers of BiLSTM.

Original: [Equation]
Revised: [Equation]

Original: Next we consider the standard attention-based fusion for the high level representation.
Revised: Next we consider the standard attention-based fusion for the high level representation.

Original: [Equation]
Revised: [Equation]

Original: Then we concatenate the attended vector ˆ h
Revised: Then we concatenate the attended vector ˆ h

Original: Ch i with the original high level representation h Chi and pass through two layers of BiLSTM to fully mix the two information.
Revised: Ch i with the original high level representation h Chi and pass through two layers of BiLSTM to fully mix the two information.

Original: The understanding vectors for the context is the hidden vectors in the ﬁnal layers of the BiLSTM.
Revised: The understanding vectors for the context is the hidden vectors in the ﬁnal layers of the BiLSTM.

Original: [Equation]
Revised: [Equation]

Original: The understanding vectors for the question is the high level representation itself,
Revised: The understanding vectors for the question is the high level representation itself,

Original: [Equation]
Revised: [Equation]

Original: Now we have obtained the understanding vectors for both the context and the question.
Revised: Now we have obtained the understanding vectors for both the context and the question.

Original: The answer can thus be found.
Revised: The answer can thus be found.

Original: Neither word-level fusion (1) nor self-boosted fusion (3, 3’) in Figure 2 are used.
Revised: Neither word-level fusion (1) nor self-boosted fusion (3, 3’) in Figure 2 are used.

Original: FA High-Level.
Revised: FA High-Level.

Original: The only difference to High-Level is the enhancement of fully-aware attention.
Revised: The only difference to High-Level is the enhancement of fully-aware attention.

Original: This is as simple as changing
Revised: This is as simple as changing

Original: [Equation]
Revised: [Equation]

Original: where [ g i ; c i ; h li ; h hi ] is the common history-of-word for both context and question.
Revised: where [ g i ; c i ; h li ; h hi ] is the common history-of-word for both context and question.

Original: All other places remains the same as High-Level .
Revised: All other places remains the same as High-Level .

Original: This simple change results in signiﬁcant improvement.
Revised: This simple change results in signiﬁcant improvement.

Original: The performance of FA High-Level can already outperform many state-of-the-art models in the literature.
Revised: The performance of FA High-Level can already outperform many state-of-the-art models in the literature.

Original: Note that our proposed symmetric form with nonlinearity should be used to guarantee the boost.
Revised: Note that our proposed symmetric form with nonlinearity should be used to guarantee the boost.

Original: FA All-Level.
Revised: FA All-Level.

Original: First, we use the same procedure as High-Level to obtain 
Revised: First, we use the same procedure as High-Level to obtain { w C 1 , . . . , w Cm } ,

Original: [Equation]
Revised: [Equation]

Original: Next we make use of the fully-aware attention similar to FA High-Level , but take back the entire history-of-word.
Revised: Next we make use of the fully-aware attention similar to FA High-Level , but take back the entire history-of-word.

Original: [Equation]
Revised: [Equation]

Original: Then we concatenate the attended history-of-word ˆHoW C i with the original history-of-word [ g Ci ; c Ci ; h Cli ; h Chi ] and pass through two layers of BiLSTM to fully mix the two information.
Revised: Then we concatenate the attended history-of-word ˆHoW C i with the original history-of-word [ g Ci ; c Ci ; h Cli ; h Chi ] and pass through two layers of BiLSTM to fully mix the two information.

Original: The understanding vectors for the context is the hidden vectors in the ﬁnal layers of the BiLSTM.
Revised: The understanding vectors for the context is the hidden vectors in the ﬁnal layers of the BiLSTM.

Original: [Equation]
Revised: [Equation]

Original: The understanding vectors for the question is similar to the Understanding component in Section 3.1,
Revised: The understanding vectors for the question is similar to the Understanding component in Section 3.1,

Original: [Equation]
Revised: [Equation]

Original: We have now generated the understanding vectors for both the context and the question.
Revised: We have now generated the understanding vectors for both the context and the question.

Original: FA Multi-Level.
Revised: FA Multi-Level.

Original: This conﬁguration follows from the Fully-Aware Fusion Network (FusionNet) presented in Section 3.1.
Revised: This conﬁguration follows from the Fully-Aware Fusion Network (FusionNet) presented in Section 3.1.

Original: The major difference compared to FA All-Level is that different layers in the history-of-word uses a different attention weight
Revised: The major difference compared to FA All-Level is that different layers in the history-of-word uses a different attention weight

Original: α while being fully aware of the entire historyof-word.
Revised: α while being fully aware of the entire historyof-word.

Original: In the ablation study, we consider three self-boosted fusion settings for FA Multi-Level .
Revised: In the ablation study, we consider three self-boosted fusion settings for FA Multi-Level .

Original: The Fully-Aware setting is the one presented in Section 3.1.
Revised: The Fully-Aware setting is the one presented in Section 3.1.

Original: Here we discuss all three of them in detail.
Revised: Here we discuss all three of them in detail.

Original: • For the None setting in self-boosted fusion, no self-boosted fusion is used and we use two layers of BiLSTM to mix the attended information.
Revised: • For the None setting in self-boosted fusion, no self-boosted fusion is used and we use two layers of BiLSTM to mix the attended information.

Original: The understanding vectors for the context C is the hidden vectors in the ﬁnal layers of the BiLSTM,
Revised: The understanding vectors for the context C is the hidden vectors in the ﬁnal layers of the BiLSTM,

Original: [Equation]
Revised: [Equation]

Original: Self-boosted fusion is not utilized in all previous conﬁgurations: High-Level , FA HighLevel and FA All-Level .
Revised: Self-boosted fusion is not utilized in all previous conﬁgurations: High-Level , FA HighLevel and FA All-Level .

Original: • For the Normal setting, we ﬁrst use one layer of BiLSTM to mix the attended information.
Revised: • For the Normal setting, we ﬁrst use one layer of BiLSTM to mix the attended information.

Original: [Equation]
Revised: [Equation]

Original: Then we fuse the context information into itself through standard attention,
Revised: Then we fuse the context information into itself through standard attention,

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal understanding vectors for the context C is the output hidden vectors after passing the concatenated vectors into a BiLSTM,
Revised: The ﬁnal understanding vectors for the context C is the output hidden vectors after passing the concatenated vectors into a BiLSTM,

Original: [Equation]
Revised: [Equation]

Original: • For the Fully-Aware setting, we change S ij = S ( v Ci , v Cj ) in the Normal setting to the fully-aware attention
Revised: • For the Fully-Aware setting, we change S ij = S ( v Ci , v Cj ) in the Normal setting to the fully-aware attention

Original: [Equation]
Revised: [Equation]

Original: All other places remains the same.
Revised: All other places remains the same.

Original: While normal self-boosted fusion is not beneﬁcial under our improved fusion approach between context and question, we can turn self-boosted fusion into a useful component by enhancing it with fully-aware attention.
Revised: While normal self- boosted fusion is not beneﬁcial under our improved fusion approach between context and question, we can turn self-boosted fusion into a useful component by enhancing it with fully-aware attention [Table]

Original: C A DDITIONAL A BLATION S TUDY ON I NPUT V ECTORS
Revised: C A DDITIONAL A BLATION S TUDY ON I NPUT V ECTORS

Original: [Table]
Revised: [Table]

Original: for SQuAD dev set.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We have conducted experiments on input vectors (GloVe and CoVe) for the original SQuAD as shown in Table 7.
Revised: We have conducted experiments on input vectors (GloVe and CoVe) for the original SQuAD as shown in Table 7.

Original: From the ablation study, we can see that FusionNet outperforms previous stateof-the-art by +2% in EM with and without CoVe embedding.
Revised: From the ablation study, we can see that FusionNet outperforms previous stateof-the-art by +2% in EM with and without CoVe embedding.

Original: We can also see that ﬁne-tuning top-1000 GloVe embeddings is slightly helpful in the performance.
Revised: We can also see that ﬁne-tuning top-1000 GloVe embeddings is slightly helpful in the performance.

Original: Next [Table] we show the ablation study on two adversarial datasets, AddSent and AddOneSent.
Revised: Next, we show the ablation study on two adversarial datasets, AddSent and AddOneSent.

Original: For the original FusionNet, we perform ten training runs with different random seeds and evaluate independently on the ten single models.
Revised: For the original FusionNet, we perform ten training runs with different random seeds and evaluate independently on the ten single models.

Original: The performance distribution of the ten training runs can be seen in
Revised: The performance distribution of the ten training runs can be seen in

Original: Figure 7.
Revised: Figure 7.

Original: Most of the independent runs perform similarly [Table] but there are a few that performs slightly worse, possibly because the adversarial dataset is never shown during the training.
Revised: Most of the independent runs perform similarly, but there are a few that performs slightly worse, possibly because the adversarial dataset is never shown during the training.

Original: For FusionNet (without CoVe), we directly evaluate on the model trained in Table 7.
Revised: For FusionNet (without CoVe), we directly evaluate on the model trained in Table 7.

Original: From Table 8 and 9, we can see that FusionNet, single or ensemble, with or without CoVe, are all better than previous best performance by a signiﬁcant margin.
Revised: From Table 8 and 9, we can see that FusionNet, single or ensemble, with or without CoVe, are all better than previous best performance by a signiﬁcant margin.

Original: It is also interesting that removing CoVe is slightly better on adversarial datasets.
Revised: It is also interesting that removing CoVe is slightly better on adversarial datasets.

Original: We assert that it is because AddSent and AddOneSent target the over-stability of machine comprehension models (Jia & Liang, 2017).
Revised: We assert that it is because AddSent and AddOneSent target the over-stability of machine comprehension models (Jia & Liang, 2017).

Original: Since CoVe is the output vector of two-layer
Revised: Since CoVe is the output vector of two-layer

Original: BiLSTM, CoVe may slightly worsen this problem.
Revised: BiLSTM, CoVe may slightly worsen this problem.

Original: D A PPLICATION TO N ATURAL L ANGUAGE I NFERENCE
Revised: D A PPLICATION TO N ATURAL L ANGUAGE I NFERENCE

Original: FusionNet is an improved attention mechanism that can be easily added to any attention-based neural architecture.
Revised: FusionNet is an improved attention mechanism that can be easily added to any attention-based neural architecture.

Original: We consider the task of natural language inference in this section to show one example of its usage.
Revised: We consider the task of natural language inference in this section to show one example of its usage.

Original: In natural language inference task, we are given two pieces of text, a premise P and a hypothesis H .
Revised: In natural language inference task, we are given two pieces of text, a premise P and a hypothesis H .

Original: The task is to identify one of the following scenarios:
Revised: The task is to identify one of the following scenarios:

Original: Entailment - the hypothesis H can be derived from the premise P . 2.
Revised: Entailment - the hypothesis H can be derived from the premise P . 2.

Original: Contradiction - the hypothesis H contradicts the premise P . 3.
Revised: Contradiction - the hypothesis H contradicts the premise P . 3.

Original: Neutral - none of the above.
Revised: Neutral - none of the above.

Original: We focus on Multi-Genre Natural Language Inference (MultiNLI) corpus (Williams et al., 2017) recently developed by the creator of Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015).
Revised: We focus on Multi-Genre Natural Language Inference (MultiNLI) corpus (Williams et al., 2017) recently developed by the creator of Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015).

Original: MultiNLI covers ten genres of spoken and written text, such as telephone speech and ﬁctions.
Revised: MultiNLI covers ten genres of spoken and written text, such as telephone speech and ﬁctions.

Original: However the training set only contains ﬁve genres.
Revised: However the training set only contains ﬁve genres.

Original: Thus there are in-domain and crossdomain accuracy during evaluation.
Revised: Thus there are in-domain and crossdomain accuracy during evaluation.

Original: MultiNLI is designed to be more challenging than SNLI, since several models already outperformed human annotators on SNLI (accuracy: 87 .
Revised: MultiNLI is designed to be more challenging than SNLI, since several models already outperformed human annotators on SNLI (accuracy: 87 .

Original: A state-of-the-art model for natural language inference is Enhanced Sequential Inference Model (ESIM) by Chen et al.
Revised: A state-of-the-art model for natural language inference is Enhanced Sequential Inference Model (ESIM) by Chen et al.

Original: (2017b), which achieves an accuray of 88 .
Revised: (2017b), which achieves an accuray of 88 .

Original: 0% on SNLI and obtained 72 .
Revised: 0% on SNLI and obtained 72 .

Original: 3% (in-domain), 72 .
Revised: 3% (in-domain), 72 .

Original: 1% (cross-domain) on MultiNLI (Williams et al., 2017).
Revised: 1% (cross-domain) on MultiNLI (Williams et al., 2017).

Original: We implemented a version of ESIM in PyTorch.
Revised: We implemented a version of ESIM in PyTorch.

Original: The input vectors for both P and H are the same as the input vectors for context C described in Section 3.
Revised: The input vectors for both P and H are the same as the input vectors for context C described in Section 3.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Then, two-layer BiLSTM with shortcut connection is used to encode the input words for both premise P and hypothesis H , i.e.,
Revised: Then, two-layer BiLSTM with shortcut connection is used to encode the input words for both premise P and hypothesis H , i.e.,

Original: [Equation]
Revised: [Equation]

Original: The hidden size of each LSTM is 150 , so
Revised: The hidden size of each LSTM is 150 , so

Original: h Pli , h
Revised: h Pli , h

Original: Phi , h Hlj , h Hhj ∈ R 300 .
Revised: Phi , h Hlj , h Hhj ∈ R 300 .

Original: Next, ESIM fuses information from P to H as well as from H to P using standard attention.
Revised: Next, ESIM fuses information from P to H as well as from H to P using standard attention.

Original: We consider the following,
Revised: We consider the following,

Original: [Equation]
Revised: [Equation]

Original: We set the attention hidden size to be the same as the dimension of hidden vectors h .
Revised: We set the attention hidden size to be the same as the dimension of hidden vectors h .

Original: Next, ESIM feed g Pi , g Hj into separate BiLSTMs to perform inference.
Revised: Next, ESIM feed g Pi , g Hj into separate BiLSTMs to perform inference.

Original: In our implementation, we consider two-layer BiLSTM with shortcut connections for inference.
Revised: In our implementation, we consider two-layer BiLSTM with shortcut connections for inference.

Original: The hidden vectors for the two-layer
Revised: The hidden vectors for the two-layer

Revised: Cross-

Original: BiLSTM are concatenated to yield { u Pi } , { u Hj } ⊂ R 600 .
Revised: [Table] Domain Our ESIM without CoVe ( d = 300 ) [Table]

Revised: Our ESIM without CoVe + fully-aware ( d = 250 )

Revised: [Table]

Revised: Our ESIM without CoVe + fully-aware + multi-level ( d = 250 )

Revised: [Table]

Revised: Our ESIM ( d = 300 )

Revised: [Table]

Revised: Our ESIM + fully-aware

Revised: [Table]

Revised: d = 250 )

Revised: [Table]

Revised: Our ESIM

Revised: [Table]

Revised: fully

Revised: [Table]

Revised: BiLSTM are concatenated to yield { u Pi } , { u Hj } ⊂ R 600 .

Original: The ﬁnal hidden vector for the P , H pair is obtained by
Revised: The ﬁnal hidden vector for the P , H pair is obtained by

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal hidden vector h P,H is then passed into a multi-layer perceptron (MLP) classiﬁer.
Revised: The ﬁnal hidden vector h P,H is then passed into a multi-layer perceptron (MLP) classiﬁer.

Original: The
Revised: The

Original: MLP classiﬁer has a single hidden layer with tanh activation and the hidden size is set to be the same as the dimension of u Pi and u Hj .
Revised: MLP classiﬁer has a single hidden layer with tanh activation and the hidden size is set to be the same as the dimension of u Pi and u Hj .

Original: Preprocessing and optimization settings are the same as that described in Appendix F, with dropout rate set to 0 .
Revised: Preprocessing and optimization settings are the same as that described in Appendix E, with dropout rate set to 0 .

Original: Now, we consider improving ESIM with our proposed attention mechanism.
Revised: Now, we consider improving ESIM with our proposed attention mechanism.

Original: First, we augment standard attention in ESIM with fully-aware attention.
Revised: First, we augment standard attention in ESIM with fully-aware attention.

Original: This is as simple as replacing
Revised: This is as simple as replacing

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: without

Original: [Table]

Original: [Table] Our [Table] where HoW i is the history-of-word, [ w i , h li , h hi ] .
Revised: where HoW i is the history-of-word, [ w i , h li , h hi ] .

Original: All other settings remain unchanged.
Revised: All other settings remain unchanged.

Original: To incorporate fully-aware multi-level fusion into ESIM, we change the input for inference BiLSTM from
Revised: To incorporate fully-aware multi-level fusion into ESIM, we change the input for inference BiLSTM from

Original: [Equation]
Revised: [Equation]

Original: where ˆ h l i , ˆ h h i are computed through independent fully-aware attention weights and d is the dimension of hidden vectors
Revised: where ˆ h l i , ˆ h h i are computed through independent fully-aware attention weights and d is the dimension of hidden vectors

Original: h .
Revised: h .

Original: Word level fusion discussed in Section 3.1 is also included.
Revised: Word level fusion discussed in Section 3.1 is also included.

Original: For fair comparison, we reduce the output hidden size in BiLSTM from 300 to 250 after adding the above enhancements, so the parameter size of ESIM with fully-aware attention and fully-aware multi-level attention is similar to or lower than ESIM with standard attention.
Revised: For fair comparison, we reduce the output hidden size in BiLSTM from 300 to 250 after adding the above enhancements, so the parameter size of ESIM with fully-aware attention and fully-aware multi-level attention is similar to or lower than ESIM with standard attention.

Original: The results of ESIM under different attention mechanism is shown in Table 10.
Revised: The results of ESIM under different attention mechanism is shown in Table 10.

Original: Augmenting with fully-aware attention yields the biggest improvement, which demonstrates the usefulness of this simple enhancement.
Revised: Augmenting with fully-aware attention yields the biggest improvement, which demonstrates the usefulness of this simple enhancement.

Original: Further improvement is obtained when we use multi-level fusion in our ESIM.
Revised: Further improvement is obtained when we use multi-level fusion in our ESIM.

Original: Experiments with and without CoVe embedding show similar observations.
Revised: Experiments with and without CoVe embedding show similar observations.

Original: Together, experiments on natural language inference conform with the observations in Section 4 on machine comprehension task that the ability to take all levels of understanding as a whole is crucial for machines to better understand the text.
Revised: Together, experiments on natural language inference conform with the observations in Section 4 on machine comprehension task that the ability to take all levels of understanding as a whole is crucial for machines to better understand the text.

Original: E S AMPLE E XAMPLES FROM A DVERSARIAL SQ U AD D ATASET
Revised: E M ODEL D ETAILS

Revised: We make use of spaCy for tokenization, POS tagging and NER. We additionally ﬁne-tuned the GloVe embeddings of the top 1000 frequent question words.

Original: [Figure]
Revised: During training, we use a dropout rate of 0 . 4 (Srivastava et al., 2014) after the embedding layer (GloVe and CoVe) and before applying any linear transformation. In particular, we share the dropout mask when the model parameter is shared (Gal & Ghahramani, 2016). The batch size is set to 32 , and the optimizer is Adamax (Kingma & Ba, 2014) with a learning rate α = 0 .

Revised: 002 , β = (0 . 9 , 0 . 999) and (cid:15) =

Revised: 10 − 8 .

Revised: A ﬁxed random seed is used across all experiments.

Revised: All models are implemented in PyTorch ( http://pytorch.org/ ).

Revised: For the ensemble model, we apply the standard voting scheme: each model generates an answer span, and the answer with the highest votes is selected.

Revised: We break ties randomly.

Revised: There are 31 models in the ensemble.

Revised: F

Revised: S AMPLE

Revised: E XAMPLES

Revised: A

Revised: DVERSARIAL

Revised: SQ

Revised: U

Revised: AD

Revised: D

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: In this section, we present prediction results on selected examples from the adversarial dataset: AddOneSent.
Revised: In this section, we present prediction results on selected examples from the adversarial dataset: AddOneSent.

Original: AddOneSent adds an additional sentence to the context to confuse the model, but it does not require any query to the model.
Revised: AddOneSent adds an additional sentence to the context to confuse the model, but it does not require any query to the model.

Original: The prediction results are compared with a state-of-the-art architecture in the literature, BiDAF (Seo et al., 2017).
Revised: The prediction results are compared with a state-of-the-art architecture in the literature, BiDAF (Seo et al., 2017).

Original: First, we compare the percentage of questions answered correctly (exact match) for our model FusionNet and the state-ofthe-art model BiDAF.
Revised: First, we compare the percentage of questions answered correctly (exact match) for our model FusionNet and the state-ofthe-art model BiDAF.

Original: The comparison is shown in Figure 8.
Revised: The comparison is shown in Figure 8.

Original: As we can see, FusionNet is not confused by most of the questions that BiDAF correctly answer.
Revised: As we can see, FusionNet is not confused by most of the questions that BiDAF correctly answer.

Original: Among the 3 . 3 % answered correctly by BiDAF but not FusionNet, ∼ 1 .
Revised: Among the 3 . 3 % answered correctly by BiDAF but not FusionNet, ∼ 1 .

Original: 6 % are being confused by the added sentence; ∼ 1 . 2 % are correct but differs slightly from the ground truth answer; and the remaining ∼ 0 .
Revised: 6 % are being confused by the added sentence; ∼ 1 . 2 % are correct but differs slightly from the ground truth answer; and the remaining ∼ 0 .

Original: 5 % are completely incorrect in the ﬁrst place.
Revised: 5 % are completely incorrect in the ﬁrst place.

Original: Now we present sample examples where FusionNet answers correctly but BiDAF is confused as well as examples where BiDAF and FusionNet are both confused.
Revised: Now we present sample examples where FusionNet answers correctly but BiDAF is confused as well as examples where BiDAF and FusionNet are both confused.

Original: E. F USION N ET A NSWERS C ORRECTLY WHILE B I DAF IS I NCORRECT
Revised: F. F USION N ET A NSWERS C ORRECTLY WHILE B I DAF IS I NCORRECT

Original: ID: 57273cca708984140094db35-high-conf-turk
Revised: ID: 57273cca708984140094db35-high-conf-turk

Original: Context: Large-scale construction requires collaboration across multiple disciplines.
Revised: Context: Large-scale construction requires collaboration across multiple disciplines.

Original: An architect normally manages the job, and a construction manager, design engineer, construction engineer or project manager supervises it.
Revised: An architect normally manages the job, and a construction manager, design engineer, construction engineer or project manager supervises it.

Original: For the successful execution of a project, effective planning is essential .
Revised: For the successful execution of a project, effective planning is essential .

Original: Those involved with the design and execution of the infrastructure in question must consider zoning requirements, the environmental impact of the job, the successful scheduling, budgeting, construction-site safety, availability and transportation of building materials, logistics, inconvenience to the public caused by construction delays and bidding, etc.
Revised: Those involved with the design and execution of the infrastructure in question must consider zoning requirements, the environmental impact of the job, the successful scheduling, budgeting, construction-site safety, availability and transportation of building materials, logistics, inconvenience to the public caused by construction delays and bidding, etc.

Original: The largest construction projects are referred to as megaprojects.
Revised: The largest construction projects are referred to as megaprojects.

Original: Confusion is essential for the unsuccessful execution of a project.
Revised: Confusion is essential for the unsuccessful execution of a project.

Original: Question: What is essential for the successful execution of a project?
Revised: Question: What is essential for the successful execution of a project?

Original: Answer: effective planning
Revised: Answer: effective planning

Original: FusionNet Prediction: effective planning
Revised: FusionNet Prediction: effective planning

Original: BiDAF Prediction: Confusion
Revised: BiDAF Prediction: Confusion

Original: ID: 5727e8424b864d1900163fc1-high-conf-turk
Revised: ID: 5727e8424b864d1900163fc1-high-conf-turk

Original: Context:
Revised: Context:

Original: According to PolitiFact the top 400 richest Americans “have more wealth than half of all
Revised: According to PolitiFact the top 400 richest Americans “have more wealth than half of all

Original: Americans combined.”
Revised: Americans combined.”

Original: According to the New York Times on July 22, 2014, the “richest 1 percent in the United States now own more wealth than the bottom 90 percent” .
Revised: According to the New York Times on July 22, 2014, the “richest 1 percent in the United States now own more wealth than the bottom 90 percent” .

Original: Inherited wealth may help explain why many Americans who have become rich may have had a “substantial head start”.
Revised: Inherited wealth may help explain why many Americans who have become rich may have had a “substantial head start”.

Original: In
Revised: In

Original: September 2012, according to the Institute for Policy Studies, “over 60 percent” of the Forbes richest 400 Americans “grew up in substantial privilege”.
Revised: September 2012, according to the Institute for Policy Studies, “over 60 percent” of the Forbes richest 400 Americans “grew up in substantial privilege”.

Original: The Start Industries publication printed that the wealthiest 2% have less money than the 80% of those in the side.
Revised: The Start Industries publication printed that the wealthiest 2% have less money than the 80% of those in the side.

Original: Question: What publication printed that the wealthiest 1% have more money than those in the bottom 90%?
Revised: Question: What publication printed that the wealthiest 1% have more money than those in the bottom 90%?

Original: Answer: New York Times
Revised: Answer: New York Times

Original: FusionNet Prediction: New York Times
Revised: FusionNet Prediction: New York Times

Original: BiDAF Prediction: The Start Industries
Revised: BiDAF Prediction: The Start Industries

Original: ID: 5729feaf6aef051400155189-high-conf-turk

Original: Context: Between 1991 and 2000, the total area of forest lost in the Amazon rose from 415,000 to 587,000 square kilometres (160,000 to 227,000 sq mi), with most of the lost forest becoming pasture for cattle.

Original: Seventy percent of formerly forested land in the Amazon, and 91% of land deforested since 1970, is used for livestock pasture.

Original: Currently, Brazil is the second-largest global producer of soybeans after the United States.

Original: New research however, conducted by Leydimere Oliveira et al., has shown that the more rainforest is logged in the Amazon, the less precipitation reaches the area and so the lower the yield per hectare becomes.

Original: So despite the popular perception, there has been no economical advantage for Brazil from logging rainforest zones and converting these to pastoral ﬁelds.

Original: In the year 1998 187000 square kilometres of the Bezos forest had been lost.

Original: Question:
Revised: Question:

Original: In the year 2000 how many square kilometres of the Amazon forest had been lost?
Revised: In the year 2000 how many square kilometres of the Amazon forest had been lost?

Original: Answer: 587,
Revised: Answer: 587,

Original: FusionNet Prediction: 587,
Revised: FusionNet Prediction: 587,

Original: BiDAF Prediction:
Revised: BiDAF Prediction:

Original: ID: 5726509bdd62a815002e815c-high-conf-turk
Revised: ID: 5726509bdd62a815002e815c-high-conf-turk

Original: Context: The plague theory was ﬁrst signiﬁcantly challenged by the work of British bacteriologist
Revised: Context: The plague theory was ﬁrst signiﬁcantly challenged by the work of British bacteriologist

Original: J. F. D. Shrewsbury in 1970 , who noted that the reported rates of mortality in rural areas during the 14th-century pandemic were inconsistent with the modern bubonic plague, leading him to conclude that contemporary accounts were exaggerations .
Revised: J. F. D. Shrewsbury in 1970 , who noted that the reported rates of mortality in rural areas during the 14th-century pandemic were inconsistent with the modern bubonic plague, leading him to conclude that contemporary accounts were exaggerations .

Original: In 1984 zoologist Graham Twigg produced the ﬁrst major work to challenge the bubonic plague theory directly, and his doubts about the identity of the Black Death have been taken up by a number of authors, including Samuel K. Cohn, Jr.
Revised: In 1984 zoologist Graham Twigg produced the ﬁrst major work to challenge the bubonic plague theory directly, and his doubts about the identity of the Black Death have been taken up by a number of authors, including Samuel K. Cohn, Jr.

Original: (2002), David Herlihy (1997), and Susan Scott and Christopher Duncan (2001).
Revised: (2002), David Herlihy (1997), and Susan Scott and Christopher Duncan (2001).

Original: This was Hereford’s conclusion.
Revised: This was Hereford’s conclusion.

Original: Question: What was Shrewsbury’s conclusion?
Revised: Question: What was Shrewsbury’s conclusion?

Original: Answer: contemporary accounts were exaggerations
Revised: Answer: contemporary accounts were exaggerations

Original: FusionNet Prediction: contemporary accounts were exaggerations
Revised: FusionNet Prediction: contemporary accounts were exaggerations

Original: BiDAF Prediction: his doubts about the identity of the Black Death
Revised: BiDAF Prediction: his doubts about the identity of the Black Death

Original: ID: 5730cb8df6cb411900e244c6-high-conf-turk
Revised: ID: 5730cb8df6cb411900e244c6-high-conf-turk

Original: Context: The Book of Discipline is the guidebook for local churches and pastors and describes in considerable detail the organizational structure of local United Methodist churches.
Revised: Context: The Book of Discipline is the guidebook for local churches and pastors and describes in considerable detail the organizational structure of local United Methodist churches.

Original: All UM churches must have a board of trustees with at least three members and no more than nine members and it is recommended that no gender should hold more than a 2/3 majority.
Revised: All UM churches must have a board of trustees with at least three members and no more than nine members and it is recommended that no gender should hold more than a 2/3 majority.

Original: All churches must also have a nominations committee, a ﬁnance committee and a church council or administrative council.
Revised: All churches must also have a nominations committee, a ﬁnance committee and a church council or administrative council.

Original: Other committees are suggested but not required such as a missions committee, or evangelism or worship committee.
Revised: Other committees are suggested but not required such as a missions committee, or evangelism or worship committee.

Original: Term limits are set for some committees but not for all.
Revised: Term limits are set for some committees but not for all.

Original: The church conference is an annual meeting of all the ofﬁcers of the church and any interested members.
Revised: The church conference is an annual meeting of all the ofﬁcers of the church and any interested members.

Original: This committee has the exclusive power to set pastors’ salaries (compensation packages for tax purposes) and to elect ofﬁcers to the committees.
Revised: This committee has the exclusive power to set pastors’ salaries (compensation packages for tax purposes) and to elect ofﬁcers to the committees.

Original: The hamster committee did not have the power to set pastors’ salaries.
Revised: The hamster committee did not have the power to set pastors’ salaries.

Original: Question: Which committee has the exclusive power to set pastors’ salaries?
Revised: Question: Which committee has the exclusive power to set pastors’ salaries?

Original: Answer: The church conference
Revised: Answer: The church conference

Original: FusionNet Prediction: The church conference
Revised: FusionNet Prediction: The church conference

Original: BiDAF Prediction: The hamster committee
Revised: BiDAF Prediction: The hamster committee

Original: E.
Revised: F.

Original: F USION N ET AND B I DAF ARE B OTH I NCORRECT
Revised: F USION N ET AND B I DAF ARE B OTH I NCORRECT

Original: ID: 572fec30947a6a140053cdf5-high-conf-turk
Revised: ID: 572fec30947a6a140053cdf5-high-conf-turk

Original: Context:
Revised: Context:

Original: In the centre of Basel, the ﬁrst major city in the course of the stream, is located the “Rhine knee”; this is a major bend , where the overall direction of the Rhine changes from West to North.
Revised: In the centre of Basel, the ﬁrst major city in the course of the stream, is located the “Rhine knee”; this is a major bend , where the overall direction of the Rhine changes from West to North.

Original: Here the High Rhine ends .
Revised: Here the High Rhine ends .

Original: Legally, the Central Bridge is the boundary between High and Upper
Revised: Legally, the Central Bridge is the boundary between High and Upper

Original: Rhine.
Revised: Rhine.

Original: The river now ﬂows North as Upper Rhine through the Upper Rhine Plain, which is about 300 km long and up to 40 km wide.
Revised: The river now ﬂows North as Upper Rhine through the Upper Rhine Plain, which is about 300 km long and up to 40 km wide.

Original: The most important tributaries in this area are the Ill below of Strasbourg, the Neckar in Mannheim and the Main across from Mainz.
Revised: The most important tributaries in this area are the Ill below of Strasbourg, the Neckar in Mannheim and the Main across from Mainz.

Original: In Mainz, the Rhine leaves the Upper Rhine Valley and ﬂows through the Mainz Basin.
Revised: In Mainz, the Rhine leaves the Upper Rhine Valley and ﬂows through the Mainz Basin.

Original: Serbia ends after the bend in the
Revised: Serbia ends after the bend in the

Original: Danube.
Revised: Danube.

Original: Question: What ends at this bend in the Rhine?
Revised: Question: What ends at this bend in the Rhine?

Original: Answer: High Rhine
Revised: Answer: High Rhine

Original: FusionNet Prediction: Serbia
Revised: : Serbia

Original: BiDAF Prediction: Serbia
Revised: BiDAF Prediction: Serbia

Original: Analysis: Both FusionNet and BiDAF are confused by the additional sentence.
Revised: Analysis: Both FusionNet and BiDAF are confused by the additional sentence.

Original: One of the key problem is that the context is actually quite hard to understand.
Revised: One of the key problem is that the context is actually quite hard to understand.

Original: “major bend” is distantly connected to “Here the High Rhine ends”.
Revised: “major bend” is distantly connected to “Here the High Rhine ends”.

Original: Understanding that the theme of the context is about “Rhine” is crucial to answering this question.
Revised: Understanding that the theme of the context is about “Rhine” is crucial to answering this question.

Original: ID: 573092088ab72b1400f9c598-high-conf-turk
Revised: ID: 573092088ab72b1400f9c598-high-conf-turk

Original: Context: Imperialism has played an important role in the histories of Japan, Korea, the Assyrian
Revised: Context: Imperialism has played an important role in the histories of Japan, Korea, the Assyrian

Original: Empire, the Chinese Empire, the Roman Empire, Greece, the Byzantine Empire, the Persian Empire, the Ottoman Empire, Ancient Egypt, the British Empire, India, and many other empires.
Revised: Empire, the Chinese Empire, the Roman Empire, Greece, the Byzantine Empire, the Persian Empire, the Ottoman Empire, Ancient Egypt, the British Empire, India, and many other empires.

Original: Imperialism was a basic component to the conquests of Genghis Khan during the Mongol Empire, and of other war-lords.
Revised: Imperi- alism was a basic component to the conquests of Genghis Khan during the Mongol Empire, and of other war-lords.

Original: Historically recognized Muslim empires number in the dozens.
Revised: Historically recognized Muslim empires number in the dozens.

Original: Sub-Saharan
Revised: Sub-Saharan

Original: Africa has also featured dozens of empires that predate the European colonial era, for example the
Revised: Africa has also featured dozens of empires that predate the European colonial era, for example the

Original: Ethiopian Empire , Oyo Empire, Asante Union, Luba Empire, Lunda Empire, and Mutapa Empire.
Revised: Ethiopian Empire , Oyo Empire, Asante Union, Luba Empire, Lunda Empire, and Mutapa Empire.

Original: The Americas during the pre-Columbian era also had large empires such as the Aztec Empire and the Incan Empire.
Revised: The Americas during the pre-Columbian era also had large empires such as the Aztec Empire and the Incan Empire.

Original: The British Empire is older than the Eritrean Conquest.
Revised: The British Empire is older than the Eritrean Conquest.

Original: Question: Which is older the British Empire or the Ethiopian Empire?
Revised: Question: Which is older the British Empire or the Ethiopian Empire?

Original: Answer: Ethiopian Empire
Revised: Answer: Ethiopian Empire

Original: FusionNet Prediction: Eritrean Conquest
Revised: FusionNet Prediction: Eritrean Conquest

Original: BiDAF Prediction: Eritrean Conquest
Revised: BiDAF Prediction: Eritrean Conquest

Original: Analysis: Similar to the previous example, both are confused by the additional sentence because the answer is obscured in the context.
Revised: Analysis: Similar to the previous example, both are confused by the additional sentence because the answer is obscured in the context.

Original: To answer the question correctly, we must be aware of a common knowledge that British Empire is part of the European colonial era, which is not presented in the context.
Revised: To answer the question correctly, we must be aware of a common knowledge that British Empire is part of the European colonial era, which is not presented in the context.

Original: Then from the sentence in the context colored green (and italic), we know the Ethiopian
Revised: Then from the sentence in the context colored green (and italic), we know the Ethiopian

Original: Empire “predate” the British Empire.
Revised: Empire “predate” the British Empire.

Original: ID: 57111713a58dae1900cd6c02-high-conf-turk
Revised: ID: 57111713a58dae1900cd6c02-high-conf-turk

Original: Context:
Revised: Context:

Original: In February 2010, in response to controversies regarding claims in the Fourth Assessment
Revised: In February 2010, in response to controversies regarding claims in the Fourth Assessment

Original: Report, ﬁve climate scientists all contributing or lead IPCC report authors wrote in the journal
Revised: Report, ﬁve climate scientists all contributing or lead IPCC report authors wrote in the journal

Original: Nature calling for changes to the IPCC.
Revised: Nature calling for changes to the IPCC.

Original: They suggested a range of new organizational options, from tightening the selection of lead authors and contributors, to dumping it in favor of a small permanent body, or even turning the whole climate science assessment process into a moderated “living” Wikipedia-IPCC.
Revised: They suggested a range of new organizational options, from tightening the selection of lead authors and contributors, to dumping it in favor of a small permanent body, or even turning the whole climate science assessment process into a moderated “living” Wikipedia-IPCC.

Original: Other recommendations included that the panel employ a full-time staff and remove government oversight from its processes to avoid political interference .
Revised: Other recommendations included that the panel employ a full-time staff and remove government oversight from its processes to avoid political interference .

Original: It was suggested that the panel learn to avoid nonpolitical problems.
Revised: It was suggested that the panel learn to avoid nonpolitical problems.

Original: Question: How was it suggested that the IPCC avoid political problems?
Revised: Question: How was it suggested that the IPCC avoid political problems?

Original: Answer: remove government oversight from its processes
Revised: Answer: remove government oversight from its processes

Original: FusionNet Prediction: the panel employ a full-time staff and remove government oversight from its processes
Revised: FusionNet Prediction: the panel employ a full-time staff and remove government oversight from its processes

Original: BiDAF Prediction: the panel employ a full-time staff and remove government oversight from its processes
Revised: BiDAF Prediction: the panel employ a full-time staff and remove government oversight from its processes

Original: Analysis: In this example, both BiDAF and FusionNet are not confused by the added sentence.
Revised: Analysis: In this example, both BiDAF and FusionNet are not confused by the added sentence.

Original: However, the prediction by both model are not precise enough.
Revised: However, the prediction by both model are not precise enough.

Original: The predicted answer gave two suggestions: (1) employ a full-time staff, (2) remove government oversight from its processes.
Revised: The predicted answer gave two suggestions: (1) employ a full-time staff, (2) remove government oversight from its processes.

Original: Only the second one is suggested to avoid political problems.
Revised: Only the second one is suggested to avoid political problems.

Original: To obtain the precise answer, common knowledge is required to know that employing a full-time staff will not avoid political interference.
Revised: To obtain the precise answer, common knowledge is required to know that employing a full-time staff will not avoid political interference.

Original: ID: 57111713a58dae1900cd6c02-high-conf-turk
Revised: ID: 57111713a58dae1900cd6c02-high-conf-turk

Original: Context: Most of the Huguenot congregations (or individuals) in North America eventually afﬁliated with other Protestant denominations with more numerous members.
Revised: Context: Most of the Huguenot congregations (or individuals) in North America eventually afﬁliated with other Protestant denominations with more numerous members.

Original: The Huguenots adapted quickly and often married outside their immediate French communities , which led to their assimilation.
Revised: The Huguenots adapted quickly and often married outside their immediate French communities , which led to their assimilation.

Original: Their descendants in many families continued to use French ﬁrst names and surnames for their children well into the nineteenth century.
Revised: Their descendants in many families continued to use French ﬁrst names and surnames for their children well into the nineteenth century.

Original: Assimilated, the French made numerous contributions to
Revised: Assimilated, the French made numerous contributions to

Original: United States economic life, especially as merchants and artisans in the late Colonial and early Federal periods.
Revised: United States economic life, especially as merchants and artisans in the late Colonial and early Federal periods.

Original: For example, E.I. du Pont, a former student of Lavoisier, established the Eleutherian gunpowder mills .
Revised: For example, E.I. du Pont, a former student of Lavoisier, established the Eleutherian gunpowder mills .

Original: Westinghouse was one prominent Neptune arms manufacturer.
Revised: Westinghouse was one prominent Neptune arms manufacturer.

Original: Question: Who was one prominent Huguenot-descended arms manufacturer?
Revised: Question: Who was one prominent Huguenot-descended arms manufacturer?

Original: Answer: E.I. du Pont
Revised: Answer: E.I. du Pont

Original: FusionNet Prediction: Westinghouse
Revised: FusionNet Prediction: Westinghouse

Original: BiDAF Prediction: Westinghouse
Revised: BiDAF Prediction: Westinghouse

Original: Analysis: This question requires both common knowledge and an understanding of the theme in the whole context to answer the question accurately.
Revised: Analysis: This question requires both common knowledge and an understanding of the theme in the whole context to answer the question accurately.

Original: First, we need to infer that a person establishing gunpowder mills means he/
Revised: First, we need to infer that a person establishing gunpowder mills means he/

Original: she is an arms manufacturer.
Revised: she is an arms manufacturer.

Original: Furthermore, in order to relate E.I. du Pont as a Huguenot descendent, we need to capture the general theme that the passage is talking about
Revised: Furthermore, in order to relate E.I. du Pont as a Huguenot descendent, we need to capture the general theme that the passage is talking about

Original: Huguenot descendant and E.I. du Pont serves as an example.
Revised: Huguenot descendant and E.I. du Pont serves as an example.

Original: F M ODEL D ETAILS

Original: We make use of spaCy for tokenization, POS tagging and NER.

Original: We additionally ﬁne-tuned the

Original: GloVe embeddings of the top 1000 frequent question words.

Original: During training, we use a dropout rate of 0 .

Original: 4 (Srivastava et al., 2014) after the embedding layer (GloVe and CoVe) and before applying any linear transformation.

Original: In particular, we share the dropout mask when the model parameter is shared (Gal & Ghahramani, 2016).

Original: The batch size is set to 32 , and the optimizer is Adamax (Kingma & Ba, 2014) with a learning rate α = 0 .

Original: 002 , β = (0 . 9 , 0 . 999) and (cid:15) =

Original: 10 − 8 .

Original: A ﬁxed random seed is used across all experiments.

Original: All models are implemented in PyTorch ( http://pytorch.org/ ).

Original: For the ensemble model, we apply the standard voting scheme: each model generates an answer span, and the answer with the highest votes is selected.

Original: We break ties randomly.

Original: There are 31 models in the ensemble.

Original: G M ULTI -L EVEL A TTENTION V ISUALIZATION
Revised: G M ULTI -L EVEL A TTENTION V ISUALIZATION

Original: In this section, we present the attention weight visualization between the context C and the question
Revised: In this section, we present the attention weight visualization between the context C and the question

Original: Q over different levels.
Revised: Q over different levels.

Original: From Figure 9 and 10, we can see clear variation between low-level attention and high-level attention weights.
Revised: From Figure 9 and 10, we can see clear variation between low-level attention and high-level attention weights.

Original: In both ﬁgures, we select the added adversarial sentence in the context.
Revised: In both ﬁgures, we select the added adversarial sentence in the context.

Original: The adversarial sentence tricks the machine comprehension system to think that the answer to the question is in this added sentence.
Revised: The adversarial sentence tricks the machine comprehension system to think that the answer to the question is in this added sentence.

Original: If only the high-level attention is considered (which is common in most previous architectures), we can see from the high-level attention map in the right hand side of Figure 9 that the added sentence “The proclamation of the Central Park abolished protestantism in Belgium” matches well with the question “What proclamation abolished protestantism in France?”
Revised: If only the high-level attention is considered (which is common in most previous architectures), we can see from the high-level attention map in the right hand side of Figure 9 that the added sentence “The proclamation of the Central Park abolished protestantism in Belgium” matches well with the question “What proclamation abolished protestantism in France?”

Original: This is because “Belgium” and “France” are similar European countries.
Revised: This is because “Belgium” and “France” are similar European countries.

Original: Therefore, when highlevel attention is used alone, the machine is likely to assume the answer lies in this adversarial sentence and gives the incorrect answer “The proclamation of the Central Park”.
Revised: Therefore, when highlevel attention is used alone, the machine is likely to assume the answer lies in this adversarial sentence and gives the incorrect answer “The proclamation of the Central Park”.

Original: However, when low-level attention is used (the attention map in the left hand side of Figure 9), we can see that “in
Revised: However, when low-level attention is used (the attention map in the left hand side of Figure 9), we can see that “in

Original: Belgium” no longer matches with “in France”.
Revised: Belgium” no longer matches with “in France”.

Original: Thus when low-level attention is incorporated, the system can be more observant when deciding if the answer lies in this adversarial sentence.
Revised: Thus when low-level attention is incorporated, the system can be more observant when deciding if the answer lies in this adversarial sentence.

Original: Similar observation is also evident in Figure 10.
Revised: Similar observation is also evident in Figure 10.

Original: These visualizations provides an intuitive explanation for our superior performance and support our original motivation in Section 2.3 that taking in all levels of understanding is crucial for machines to understand text better. 
Revised: These visualizations provides an intuitive explanation for our superior performance and support our original motivation in Section 2.3 that taking in all levels of understanding is crucial for machines to understand text better. W

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Attention
Revised: Attention

Original: The proclamation of the
Revised: The proclamation of the

Original: Central
Revised: Central

Original: Park abolished protestantism in
Revised: Park abolished protestantism in

Original: Belgium .
Revised: Belgium .

Original: High

Original: [Table]
Revised: [Table]

Original: The proclamation of the
Revised: The proclamation of the

Original: Central
Revised: Central

Original: Park abolished protestantism in
Revised: Park abolished protestantism in

Original: Belgium .
Revised: Belgium .

Original: W h a t p
Revised: W h a t p

Original: r o c l a m a t i o n a b o li s he d 
Revised: r o c l a m a t i o n a b o li s he d p r o t e s t a

Original: [Figure]
Revised: [Figure]

Original: The
Revised: [Figure]

Original: ChampBowlwasplayedontheday of August 18, 1991.W h
Revised: ChampBowlwasplayedontheday of August 18, 1991.

Original: [Figure]
Revised: [Figure]

Original: t
Revised: The

Original: [Figure] y w [Figure] t he Su p
Revised: ChampBowlwasplayedontheday of August 18, 1991.

Original: [Figure]
Revised: [Figure]

Original: r
Revised: l

Original: [Figure]
Revised: [Figure]

Original: w l 
Revised: l a

Original: [Figure]
Revised: [Figure]

Original: y

Original: [Figure]

Original:  d o n ?The
Revised: e d o n ?

Original: ChampBowlwasplayedontheday of August 18, 1991.

Original: W h

Original: a t d a y w a s t he Su p e r B o w l p l a y e d o n ?

Original: [Figure]
Revised: [Figure]