Original: N ATURAL L ANGUAGE I NFERENCE OVER I NTERACTION S PACE
Revised: N ATURAL L ANGUAGE I NFERENCE OVER I NTERACTION S PACE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Natural Language Inference (NLI) task requires an agent to determine the logical relationship between a natural language premise and a natural language hypothesis.
Revised: Natural Language Inference (NLI) task requires an agent to determine the logical relationship between a natural language premise and a natural language hypothesis.

Original: We introduce Interactive Inference Network (IIN), a novel class of neural network architectures that is able to achieve high-level understanding of the sentence pair by hierarchically extracting semantic features from interaction space.
Revised: We introduce Interactive Inference Network (IIN), a novel class of neural network architectures that is able to achieve high-level understanding of the sentence pair by hierarchically extracting semantic features from interaction space.

Original: We show that an interaction tensor (attention weight) contains semantic information to solve natural language inference, and a denser interaction tensor contains richer semantic information.
Revised: We show that an interaction tensor (attention weight) contains semantic information to solve natural language inference, and a denser interaction tensor contains richer semantic information.

Original: One instance of such architecture, Densely Interactive Inference Network (DIIN), demonstrates the state-of-the-art performance on large scale NLI copora and large-scale NLI alike corpus.
Revised: One instance of such architecture, Densely Interactive Inference Network (DIIN), demonstrates the state-of-the-art performance on large scale NLI copora and large-scale NLI alike corpus.

Original: It’s noteworthy that DIIN achieve a greater than 20% error reduction on the challenging Multi-Genre NLI (MultiNLI; Williams et al. 2017) dataset with respect to the strongest published system.
Revised: It’s noteworthy that DIIN achieve a greater than 20% error reduction on the challenging Multi-Genre NLI (MultiNLI; Williams et al. 2017) dataset with respect to the strongest published system.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural Language Inference (NLI also known as recognizing textual entiailment, or RTE) task requires one to determine whether the logical relationship between two sentences is among entailment (if the premise is true, then the hypothesis must be true), contradiction (if the premise is true, then the hypothesis must be false) and neutral (neither entailment nor contradiction).
Revised: Natural Language Inference (NLI also known as recognizing textual entiailment, or RTE) task requires one to determine whether the logical relationship between two sentences is among entailment (if the premise is true, then the hypothesis must be true), contradiction (if the premise is true, then the hypothesis must be false) and neutral (neither entailment nor contradiction).

Original: NLI is known as a fundamental and yet challenging task for natural language understanding(Williams et al., 2017), not only because it requires one to identify the language pattern, but also to understand certain common sense knowledge.
Revised: NLI is known as a fundamental and yet challenging task for natural language understanding(Williams et al., 2017), not only because it requires one to identify the language pattern, but also to understand certain common sense knowledge.

Original: In Table 1, three samples from MultiNLI corpus show solving the task requires one to handle the full complexity of lexical and compositional semantics.
Revised: In Table 1, three samples from MultiNLI corpus show solving the task requires one to handle the full complexity of lexical and compositional semantics.

Original: Entailment is related to a broad range of tasks: in abstractive summarization, the generated summarization text should be entailed by the text; in paraphrase identiﬁcation, the paraphrased sentences entail each other; in information retrieval, the retrieved text is entailed by the source context(Bos & Markert, 2005).
Revised: The previous work on

Original: The previous work on NLI (or RTE) has extensively researched on conventional approaches(Fyodorov et al., 2000; Bos & Markert, 2005; MacCartney & Manning, 2009).
Revised: NLI (or RTE) has extensively researched on conventional approaches(Fyodorov et al., 2000; Bos & Markert, 2005; MacCartney & Manning, 2009).

Original: Recent progress on NLI is enabled by the availability of 570k human annotated dataset(Bowman et al., 2015) and the advancement of representation learning technique.
Revised: Recent progress on NLI is enabled by the availability of 570k human annotated dataset(Bowman et al., 2015) and the advancement of representation learning technique.

Original: Among the core representation learning techniques, attention mechanism is broadly applied in many
Revised: Among the core representation learning techniques, attention mechanism is broadly applied in many

Original: NLU tasks since its introduction: machine
Revised: NLU tasks since its introduction: machine

Original: translation(Bahdanau et al., 2014), abstractive summarization(Rush et al., 2015), Reading Comprehension(Hermann et al., 2015), dialog system(Mei et al., 2016), etc.
Revised: translation(Bahdanau et al., 2014), abstractive summarization(Rush et al., 2015), Reading Comprehension(Hermann et al., 2015), dialog system(Mei et al., 2016), etc.

Original: As described by Vaswani et al.
Revised: As described by Vaswani et al.

Original: (2017), “ An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.
Revised: (2017), “ An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.

Original: The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key ”.
Revised: The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key ”.

Original: Attention mechanism is known for its alignment between representations, focusing one part of representation over another, and modeling the dependency regardless of sequence length.
Revised: Attention mechanism is known for its alignment between representations, focusing one part of representation over another, and modeling the dependency regardless of sequence length.

Original: The attention weight follows softmax layer is the essential component of attention(Bahdanau et al., 2014).
Revised: The attention weight follows softmax layer is the essential component of attention(Bahdanau et al., 2014).

Original: The singlechannel attention weight can be viewed as a single-channel interaction tensor.
Revised: The single-channel attention weight can be viewed as a single-channel interaction tensor.

Original: An single-channel interaction tensor represents the word-by-word interaction between two sentences in one dimension.
Revised: An singlechannel interaction tensor represents the word-by-word interaction between two sentences in one dimension.

Original: On the other hand, A multi-channel attention weight, applied on multi-head attention (Vaswani et al., 2017) to align sentences in different representation subspace, can be viewed as a multi-channel interaction tensor.
Revised: On the other hand, A multi-channel attention weight, applied on multi-head attention (Vaswani et al., 2017) to align sentences in different representation subspace, can be viewed as a multi-channel interaction tensor.

Original: Observing attention’s powerful capability, We hypothesize that interaction tensor 
Revised: Observing attention’s powerful capability, We hypothesize that interaction tensor contains the necessary semantic information required for understanding the text. In this work, we demonstrate the feasibility that natural language inference task can be tackled directly through extracting semantic feature in interaction space. By incorporating the powerful feature ex-

Original: Premise: The FCC has created two tiers of small business for this service with the approval of the SBA.
Revised: Premise: The FCC has created two tiers of small business for this service with the approval of the SBA.

Original: Hypothesis: The SBA has given the go-ahead for the FCC to divide this service into two tiers of small business.
Revised: Hypothesis: The SBA has given the go-ahead for the FCC to divide this service into two tiers of small business.

Original: Label: entailment Premise: He was crying like his mother had just walloped him.
Revised: Label: entailment Premise: He was crying like his mother had just walloped him.

Original: Hypothesis:
Revised: Hypothesis:

Original: He was crying like his mother hit him with a spoon.
Revised: He was crying like his mother hit him with a spoon.

Original: Label:
Revised: Label:

Original: Neutral Premise: Later, Tom testiﬁed against John so as to avoid the electric chair.Hypothesis: Tom refused to
Revised: Neutral Premise: Later, Tom testiﬁed against John so as to avoid the electric chair.

Original: turn

Original: on

Original:  his friend, even though he was slated to be executed.
Revised: Hypothesis: Tom refused to turn on his friend, even though he was slated to be executed.

Original: [Table]
Revised: [Table]

Original: contains the necessary semantic information required for understanding the text.

Original: In this work, we demonstrate the feasibility that natural language inference task can be tackled directly through extracting semantic feature in interaction space.

Original: By incorporating the powerful feature extractor such as deep 2-D convolutional neural network architectures, we can extract n-gram pair semantic interaction features from interaction tensor.
Revised: tractor such as deep 2-D convolutional neural network architectures, we can extract n-gram pair semantic interaction features from interaction tensor.

Original: Our Interactive Inference Network (IIN) architecture is fully compatible with convolutional feature extractor that works well on CIFAR100(Krizhevsky, 2009) or ImageNET(Russakovsky et al., 2015) with minor adaptation.

Original: It builds up a bridge between NLU and computer vision. By hierarchically stacking the feature extractor, the model can understand the text from word level, phrase level to sentence level.
Revised: By hierarchically stacking the feature extractor, our Interactive Inference Network (IIN) architecture can understand the text from word level, phrase level to sentence level.

Original: The goal of reducing sequential computation lays the foundation of several recent work such as Extended Neural GPU(Kaiser & Bengio, 2016), ByteNet(Kalchbrenner et al., 2016), ConvS2S(Gehring et al., 2017), Transformer(Vaswani et al., 2017).
Revised: The goal of reducing sequential computation lays the foundation of several recent work such as Extended Neural GPU(Kaiser & Bengio, 2016), ByteNet(Kalchbrenner et al., 2016), ConvS2S(Gehring et al., 2017), Transformer(Vaswani et al., 2017).

Original: Recurrent structure generate a sequence of hidden state h t , as the function of previous hidden state h t − 1 and the input for position t .
Revised: Recurrent structure generate a sequence of hidden state h t , as the function of previous hidden state h t − 1 and the input for position t .

Original: The hard constraint precludes the parallelization within training, results in higher computational time complexity thus delays training(Vaswani et al., 2017; Gehring et al., 2017).
Revised: The hard constraint precludes the parallelization within training, results in higher computational time complexity thus delays training(Vaswani et al., 2017; Gehring et al., 2017).

Original: To tackle the problem, we propose a simple encoder without using any recurrent or recursive structure in this paper.
Revised: To tackle the problem, we propose a simple encoder without using any recurrent or recursive structure in this paper.

Original: Our experiments show that one instance of Interactive Inference Network, Densely Interactive Inference Network, achieves new state-of-the-art performance on both SNLI and MultiNLI copora.
Revised: Our experiments show that one instance of Interactive Inference Network, Densely Interactive Inference Network, achieves new state-of-the-art performance on both SNLI and MultiNLI copora.

Original: To test the generality of our architecture, we interpret the paraphrase identiﬁcation task as natural language inference task where matching as entailment, not matching as neutral.
Revised: To test the generality of our architecture, we interpret the paraphrase identiﬁcation task as natural language inference task where matching as entailment, not matching as neutral.

Original: We test the model on Quora Question Pair dataset, which contains over 400k real world question pair, and achieves new state-of-the-art performance.
Revised: We test the model on Quora Question Pair dataset, which contains over 400k real world question pair, and achieves new state-of-the-art performance.

Original: We introduce the related work in Section 2, and discuss the general framework of IIN along with a speciﬁc instance that enjoys state-of-the-art performance on multiple datasets in Section 3.
Revised: We introduce the related work in Section 2, and discuss the general framework of IIN along with a speciﬁc instance that enjoys state-of-the-art performance on multiple datasets in Section 3.

Original: We describe experiments and analysis in Section 4.
Revised: We describe experiments and analysis in Section 4.

Original: Finally, we conclude and discuss future work in
Revised: Finally, we conclude and discuss future work in

Original: Section 5.
Revised: Section 5.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The early exploration on NLI mainly rely on conventional methods and small scale datasets(Marelli et al., 2014).
Revised: The early exploration on NLI mainly rely on conventional methods and small scale datasets(Marelli et al., 2014).

Original: The availability of SNLI dataset with 570k human annotated sentence pairs has enabled a good deal of progress on natural language understanding.
Revised: The availability of SNLI dataset with 570k human annotated sentence pairs has enabled a good deal of progress on natural language understanding.

Original: The essential representation learning techniques for NLU such as attention(Wang & Jiang, 2015), memory(Munkhdalai & Yu, 2016) and the use of parse structure(Bowman et al., 2016; Mou et al., 2015) are studied on the SNLI which serves as an important benchmark for sentence understanding.
Revised: The essential representation learning techniques for NLU such as attention(Wang & Jiang, 2015), memory(Munkhdalai & Yu, 2016) and the use of parse structure(Bowman et al., 2016; Mou et al., 2015) are studied on the SNLI which serves as an important benchmark for sentence understanding.

Original: The models trained on NLI task can be divided into two categories: (i) sentence encoding-based model which aims to ﬁnd vector representation for each sentence and classiﬁes the relation by using the concatenation of two vector representation along with their absolute element-wise difference and element-wise product(Bowman et al., 2016; Vendrov et al., 2015; Mou et al., 2015; Liu et al., 2016; Munkhdalai & Yu, 2016).
Revised: The models trained on NLI task can be divided into two categories: (i) sentence encoding-based model which aims to ﬁnd vector representation for each sentence and classiﬁes the relation by using the concatenation of two vector representation along with their absolute element-wise difference and element-wise product(Bowman et al., 2016; Vendrov et al., 2015; Mou et al., 2015; Liu et al., 2016; Munkhdalai & Yu, 2016).

Original: (ii) Joint feature models which use the cross sentence feature or attention from one sentence to another(
Revised: (ii) Joint feature models which use the cross sentence feature or attention from one sentence to another(Rockt¨aschel et al., 2015; Wang & Jiang, 2015; Cheng et al., 2016; Parikh et al., 2016; Wang et al., 2017; Yu & Munkhdalai, 2017; Sha et al., 2016).

Original: After neural attention mechanism is successfully applied on the machine translation task, such technique has became widely used in both natural language process and computer vision domains.
Revised: After neural attention mechanism is successfully applied on the machine translation task, such technique has became widely used in both natural language process and computer vision domains.

Original: Many variants of attention technique such as hard-attention(Xu et al., 2015), self-attention(Parikh et al., 2016), multi-hop attention(Gong & Bowman, 2017), bidirectional attention(Seo et al., 2016) and multi-head attention(Vaswani et al., 2017) are also introduced to tackle more complicated tasks.
Revised: Many variants of attention technique such as hard-attention(Xu et al., 2015), self-attention(Parikh et al., , multi-hop attention(Gong & Bowman, 2017), bidirectional attention(Seo et al., 2016) and multi-head attention(Vaswani et al., 2017) are also introduced to tackle more complicated tasks.

Original: Before this work, neural attention mechanism is mainly used to make alignment, focusing on speciﬁc part of the representation.
Revised: Before this work, neural attention mechanism is mainly used to make alignment, focusing on speciﬁc part of the representation.

Original: In this work, we want to show that attention weight contains rich semantic information required for understanding the logical relationship between sentence pair.
Revised: In this work, we want to show that attention weight contains rich semantic information required for understanding the logical relationship between sentence pair.

Original: Though RNN or LSTM are very good for variable length sequence modeling, using Convolutional neural network in NLU tasks is very desirable because of its parallelism in computation.
Revised: Though RNN or LSTM are very good for variable length sequence modeling, using Convolutional neural network in NLU tasks is very desirable because of its parallelism in computation.

Original: Convolutional structure has been successfully applied in various domain such as machine translation(Gehring et al., 2017), sentence classiﬁcation(Kim, 2014), text matching(Hu et al., 2014) and sentiment analysis(Kalchbrenner et al., 2014), etc.
Revised: Convolutional structure has been successfully applied in various domain such as machine translation(Gehring et al., 2017), sentence classiﬁcation(Kim, 2014), text matching(Hu et al., 2014) and sentiment analysis(Kalchbrenner et al., 2014), etc.

Original: The convolution structure is also applied on different level of granularity such as byte(Zhang & LeCun, 2017), character(Zhang et al., 2015),
Revised: The convolution structure is also applied on different level of granularity such as byte(Zhang & LeCun, 2017), character(Zhang et al., 2015),

Original: word(Gehring et al., 2017) and sentences(Mou et al., 2015) levels.
Revised: word(Gehring et al., 2017) and sentences(Mou et al., 2015) levels.

Original:  M ODEL 3. I NTERACTIVE I NFERENCE N ETWORK
Revised:  M ODEL 3. I NTERACTIVE I NFERENCE N ETWORK

Original: The Interactive Inference Network (IIN) is a hierarchical multi-stage process and consists of ﬁve components.
Revised: The Interactive Inference Network (IIN) is a hierarchical multi-stage process and consists of ﬁve components.

Original: Each of the components is compatible with different type of implementations.
Revised: Each of the components is compatible with different type of implementations.

Original: Potentially all exiting approaches in machine learning, such as decision tree, support vector machine and neural network approach, can be transfer to replace certain component in this architecture.
Revised: Potentially all exiting approaches in machine learning, such as decision tree, support vector machine and neural network approach, can be transfer to replace certain component in this architecture.

Original: We focus on neural network approaches below.
Revised: We focus on neural network approaches below.

Original: Figure 1 provides a visual illustration of Interactive Inference
Revised: Figure 1 provides a visual illustration of Interactive Inference

Original: Network.
Revised: Network.

Original: Embedding Layer converts each word or phrase to a vector representation and construct the representation matrix for sentences.
Revised: Embedding Layer converts each word or phrase to a vector representation and construct the representation matrix for sentences.

Original: In embedding layer, a model can map tokens to vectors with the pre-trained word representation such as GloVe(Pennington et al., 2014), word2Vec(Mikolov et al., 2013) and fasttext(Joulin et al., 2016).
Revised: In embedding layer, a model can map tokens to vectors with the pre-trained word representation such as GloVe(Pennington et al. , word2Vec(Mikolov et al., 2013) and fasttext(Joulin et al.

Original: It can also utilize the preprocessing tool, e.g. named entity recognizer, part-of-speech recognizer, lexical parser and coreference identiﬁer etc., to incorporate more lexical and syntactical information into the feature vector. 2.
Revised: It can also utilize the preprocessing tool, e.g. named entity recognizer, part-of-speech recognizer, lexical parser and coreference identiﬁer etc., to incorporate more lexical and syntactical information into the feature vector. 2.

Original: Encoding Layer encodes the representations by incorporating the context information or enriching the representation with desirable features for future use.
Revised: Encoding Layer encodes the representations by incorporating the context information or enriching the representation with desirable features for future use.

Original: For instance, a model can adopt bidirectional recurrent neural network to model the temporal interaction on both direction, recursive neural network(Socher et al., 2011b) (also known as TreeRNN) to model the compositionality and the recursive structure of language, or self-attention to model the long-term dependency on sentence.
Revised: For instance, a model can adopt bidirectional recurrent neural network to model the temporal interaction on both direction, recursive neural network(Socher et al., 2011b) (also known as TreeRNN) to model the compositionality and the recursive structure of language, or self-attention to model the long-term dependency on sentence.

Original: Different components of encoder can be combined to obtain a better sentence matrix representation.
Revised: Different components of encoder can be combined to obtain a better sentence matrix representation.

Original: Interaction Layer creates an word-by-word interaction tensor by both premise representation and hypothesis representation matrix.
Revised: Interaction Layer creates an word-by-word interaction tensor by both premise representation and hypothesis representation matrix.

Original: In a TreeRNN setting, the interaction layer models the interaction between each node pair(Socher et al., 2011a).
Revised: In a TreeRNN setting, the interaction layer models the interaction between each node pair(Socher et al., 2011a).

Original: The interaction can be modeled in different ways.
Revised: The interaction can be modeled in different ways.

Original: A common approach is to compute the cosine similarity or dot product between each pair of feature vector.
Revised: A common approach is to compute the cosine similarity or dot product between each pair of feature vector.

Original: On the other hand, a denser interaction tensor can be obtained by using linear layers to scale down element-wise product between each pair of feature vector.
Revised: On the other hand, a denser interaction tensor can be obtained by using linear layers to scale down element-wise product between each pair of feature vector.

Original: In the interaction tensor, one channel of the tensor represents the words interact in one dimension(perspective) and therefore having d channels shows that the model understands the sentences with a implicit world representation of d dimensions(perspectives).
Revised: In the interaction tensor, one channel of the tensor represents the words interact in one dimension(perspective) and therefore having d channels shows that the model understands the sentences with a implicit world representation of d dimensions(perspectives).

Original: Feature Extraction Layer adopts feature extractor to extract the semantic feature from interaction tensor.
Revised: Feature Extraction Layer adopts feature extractor to extract the semantic feature from interaction tensor.

Original: The convolutional feature extractors, such  et al., 2016) and DenseNet(Huang et al., 2016), proven work well on image recognition are completely compatible under such architecture.The advance in computer vision is now transferable to natural language understanding.
Revised: The convolutional feature extractors, such as AlexNet(Krizhevsky et al., 2012), VGG(Simonyan & Zisserman, 2014), Inception(Szegedy et al., 2014), ResNet(He et al., 2016) and DenseNet(Huang et al., 2016), proven work well on image recognition are completely compatible under such architecture.

Original: Unlike the work (Kim, 2014; Zhang et al., 2015) who employs 1-D sliding window, our CNN architecture allows 2-D kernel to extract semantic interaction feature from the word-by-word interaction between n-gram pair.
Revised: Unlike the work (Kim, 2014; Zhang et al., 2015) who employs 1-D sliding window, our CNN architecture allows 2-D kernel to extract semantic interaction feature from the word-by-word interaction between n-gram pair.

Original: Sequential or tree-like feature extractors are also applicable in the feature extraction layer.
Revised: Sequential or tree-like feature extractors are also applicable in the feature extraction layer.

Original: Output Layer decodes the acquired feature to give prediction.Under the setting
Revised: Output Layer decodes the acquired feature to give prediction.

Original:  of NLI it generates the conﬁdence on each class.
Revised: Under the setting of NLI, it generates the conﬁdence on each class.

Original: 3. D ENSELY I NTERACTIVE I NFERENCE N ETWORK
Revised: 3. D ENSELY I NTERACTIVE I NFERENCE N ETWORK

Original: One example of IIN is Densely Interactive Inference Network (DIIN), a relatively simple structure but produces state-of-the-art performance on multiple datasets.
Revised: One example of IIN is Densely Interactive Inference Network (DIIN), a relatively simple structure but produces state-of-the-art performance on multiple datasets.

Original: Embedding Layer:
Revised: Embedding Layer:

Original: For DIIN, we use the concatenation of word embedding, character feature and syntactical features.
Revised: For DIIN, we use the concatenation of word embedding, character feature and syntactical features.

Original: The word embedding is obtained by mapping token to high dimensional vector space by pre-trained word vector (840B GloVe).
Revised: The word embedding is obtained by mapping token to high dimensional vector space by pre-trained word vector (840B GloVe).

Original: Word embedding is updated during training.
Revised: Word embedding is updated during training.

Original: As in (Kim et al., 2016; Lee et al., 2016), The character feature is obtained by using a  convolutional neural network followed by a max pooling over the learned character vectors.
Revised: As in (Kim et al., 2016; Lee et al., 2016), The character feature is obtained by applying a 1-D convolutional neural network followed by a max pooling upon the learned character embedding.

Original: Syntactical features include one-hot part-of-speech tagging feature and binary exact match feature.
Revised: Syntactical features include one-hot part-of-speech tagging feature and binary exact match feature.

Original: The exact match value is activated if there are tokens with same stem or lemma in the other sentence as the corresponding token.
Revised: The exact match value is activated if there are tokens with same stem or lemma in the other sentence as the corresponding token.

Original: The exact match feature is simple while foundextremely useful  in reading comprehension task (Chen et al., 2017a).
Revised: The exact match feature is simple while found useful as in reading comprehension task (Chen et al., 2017a).

Original: It helps to speed up the convergence speed in NLI task.

Original: Now we have premise representation P ∈ R
Revised: Now we have premise representation P ∈ R

Original: p × d and hypothesis representation H ∈ R h × d , where p refers to the sequence length of premise, h refers to the sequence length of hypothesis and d means the dimension of both representation.
Revised: p × d and hypothesis representation H ∈ R h × d , where p refers to the sequence length of premise, h refers to the sequence length of hypothesis and d means the dimension of both representation.

Original: The 1-D convolutional neural network and character features weights share the same set of parameters between premise and hypothesis.
Revised: The 1D convolutional neural network and character features weights share the same set of parameters between premise and hypothesis.

Original: Encoding Layer:
Revised: Encoding Layer:

Original: In the encoding layer,  premise representation P and  hypothesis representation 
Revised: In the encoding layer, the premise representation P and the hypothesis representation H are passed through a two-layer highway network, thus having P hw ∈ R p × d

Original: H are passed through a two-layer highway network, thus having P hw ∈ R p × d and H hw ∈ R h × d for new premise representation and new hypothesis representation.
Revised: and H hw ∈ R h × d for new premise representation and new hypothesis representation.

Original: These new representation are then passed to intra-attention (self-attention) layer to take into account the word order and context information.
Revised: These new representation are then passed to intra-attention (self-attention) layer to take into account the word order and context information.

Original: Take premise as example, we model intra-attention by
Revised: Take premise as example, we model intra-attention by

Original: [Equation]
Revised: [Equation]

Original: where P itrAtti is a weighted summation of P hw .
Revised: where P itrAtt i is a weighted summation of P hw .

Original: We choose α ( a , b ,, w itrAtt ) =
Revised: We choose α ( a , b , w itrAtt ) =

Original: w (cid:62) itrAtt [ a ; b ; a ◦ b ] , where w itrAtt ∈ R 3 d is a trainable weight, ◦ is element-wise multiplication, [;] is vector concatenation across row, and the implicit multiplication is matrix multiplication.
Revised: w (cid:62) itrAtt [ a ; b ; a ◦ b ] , where w itrAtt ∈ R 3 d is a trainable weight, ◦ is element-wise multiplication, [;] is vector concatenation across row, and the implicit multiplication is matrix multiplication.

Original: Then both P hw and P itrAtt are fed into a semantic composite fuse gate (fuse gate in short), which acts as a skip connection.
Revised: Then both P hw and P itrAtt are fed into a semantic composite fuse gate (fuse gate in short), which acts as a skip connection.

Original: The fuse gate is implemented as
Revised: The fuse gate is implemented as

Original: [Equation]
Revised: [Equation]

Original: where W 1 , W 2 , W 3 ∈ R 2 d × d and b 1 b 2 , b 3 ∈ R
Revised: where W 1 , W 2 , W 3 ∈ R 2 d × d and b 1 b 2 , b 3 ∈ R

Original: d are trainable weights, σ is sigmoid nonlinear operation.
Revised: d are trainable weights, σ is sigmoid nonlinear operation.

Original: We do the same operation on hypothesis representation, thus having H enc .
Revised: We do the same operation on hypothesis representation, thus having H enc .

Original: The weights of intraattention and fuse gate for premise and hypothesis are not shared, but the difference between the weights of are penalized.
Revised: The weights of intraattention and fuse gate for premise and hypothesis are not shared, but the difference between the weights of are penalized.

Original: The penalization aims to ensure the parallel structure learns the similar functionality but is aware of the subtle semantic difference between premise and hypothesis.
Revised: The penalization aims to ensure the parallel structure learns the similar functionality but is aware of the subtle semantic difference between premise and hypothesis.

Original: Interaction Layer:
Revised: Interaction Layer:

Original: The interaction layer models the interaction between premise encoded representation P enc and hypothesis encoded representation H enc as follows:
Revised: The interaction layer models the interaction between premise encoded representation P enc and hypothesis encoded representation H enc as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where P enci is the i -th row vector of P enc , and H encj is the j -th row vector of H enc .
Revised: where P enci is the i -th row vector of P enc , and H encj is the j -th row vector of H enc .

Original: Though there are many implementations of interaction, we ﬁnd β ( a, b )
Revised: Though there are many implementations of interaction, we ﬁnd β ( a, b )

Original: = a ◦ b very useful.
Revised: = a ◦ b very useful.

Original: Feature Extraction Layer:
Revised: Feature Extraction Layer:

Original: We adopt DenseNet(Huang et al., 2016) as convolutional feature extractor in DIIN.
Revised: We adopt DenseNet(Huang et al., 2016) as convolutional feature extractor in DIIN.

Original: Though our experiments show ResNet(He et al., 2016) works well in the architecture, we choose DenseNet because it is effective in saving parameters.
Revised: Though our experiments show ResNet(He et al., 2016) works well in the architecture, we choose DenseNet because it is effective in saving parameters.

Original: One interesting observation with ResNet is that if we remove the skip connection in residual structure, the model does not converge at all.
Revised: One interesting observation with ResNet is that if we remove the skip connection in residual structure, the model does not converge at all.

Original: We found batch normalization delays convergence without contributing to accuracy, therefore we does not use it in our case.
Revised: We found batch normalization delays convergence without contributing to accuracy, therefore we does not use it in our case.

Original: A ReLU activation function is applied after all convolution unless otherwise noted.
Revised: A ReLU activation function is applied after all convolution unless otherwise noted.

Original: Once we have the interaction tensor I , we use a convolution with 1 ×kernel to scale down the tensor in a ratio, FSDR , without following ReLU.
Revised: Once we have the interaction tensor I , we use a convolution with 1 ×kernel to scale down the tensor in a ratio, FSDR , without following ReLU.

Original: If the input channel is k then the output channel is floor ( k × FSDR ) .
Revised: If the input channel is k then the output channel is floor ( k × FSDR ) .

Original: Then the generated feature map is feed into three sets of Dense block(Huang et al., 2016) and transition block pair.
Revised: Then the generated feature map is feed into three sets of Dense block(Huang et al., 2016) and transition block pair.

Original: The DenseNet block contains n layers of 3 × 3 convolution layer with growth rate of GR .
Revised: The DenseNet block contains n layers of 3 × 3 convolution layer with growth rate of GR .

Original: The transition layer has a convolution layer with 1 × 1 kernel for scaling down purpose, followed by a max pooling layer with stride 2 .
Revised: The transition layer has a convolution layer with 1 × 1 kernel for scaling down purpose, followed by a max pooling layer with stride 2 .

Original: The transition scale down ratio in transition layer is TSDR .
Revised: The transition scale down ratio in transition layer is TSDR .

Original: Output Layer:
Revised: Output Layer:

Original: DIIN uses a linear layer to classify ﬁnal ﬂattened feature representation to three classes.
Revised: DIIN uses a linear layer to classify ﬁnal ﬂattened feature representation to three classes.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present the evaluation of our model.
Revised: In this section, we present the evaluation of our model.

Original: We ﬁrst perform quantitative evaluation, comparing our model with other competitive models.
Revised: We ﬁrst perform quantitative evaluation, comparing our model with other competitive models.

Original: We then conduct some qualitative analyses to understand how DIIN achieve the high level understanding through interaction.
Revised: We then conduct some qualitative analyses to understand how DIIN achieve the high level understanding through interaction.

Original: 4. D ATA
Revised: 4. D ATA

Original: Here we introduce three datasets we evaluate our model on.
Revised: Here we introduce three datasets we evaluate our model on.

Original: The evaluation metric for all dataset is accuracy.
Revised: The evaluation metric for all dataset is accuracy.

Original: SNLI Stanford Natural Language Inference (SNLI; Bowman et al. 2015) has 570k human annotated sentence pairs.
Revised: SNLI Stanford Natural Language Inference (SNLI; Bowman et al. 2015) has 570k human annotated sentence pairs.

Original: The premise data is draw from the captions of the Flickr30k corpus, and the hypothesis data is manually composed.
Revised: The premise data is draw from the captions of the Flickr30k corpus, and the hypothesis data is manually composed.

Original: The labels provided in are “entailment”, “neutral’, “contradiction” and “-”.
Revised: The labels provided in are “entailment”, “neutral’, “contradiction” and “-”.

Original: “-” shows that annotators cannot reach consensus with each other, thus removed during training and testing as in other works.
Revised: “-” shows that annotators cannot reach consensus with each other, thus removed during training and testing as in other works.

Original: We use the same data split as in Bowman et al.
Revised: We use the same data split as in Bowman et al.

Original: MultiNLI Multi-Genre NLI Corpus (MultiNLI; Williams et al. 2017) has 433k sentence pairs, whose collection process and task detail are modeled closely to SNLI.
Revised: MultiNLI Multi-Genre NLI Corpus (MultiNLI; Williams et al. 2017) has 433k sentence pairs, whose collection process and task detail are modeled closely to SNLI.

Original: The premise data is collected from maximally broad range of genre of American English such as written non-ﬁction genres (SLATE, OUP, GOVERNMENT, VERBATIM, TRAVEL), spoken genres (TELEPHONE, FACETO-FACE), less formal written genres (FICTION, LETTERS) and a specialized one for 9/11.
Revised: The premise data is collected from maximally broad range of genre of American English such as written non-ﬁction genres (SLATE, OUP, GOVERNMENT, VERBATIM, TRAVEL), spoken genres (TELEPHONE, FACETO-FACE), less formal written genres (FICTION, LETTERS) and a specialized one for 9/11.

Original: Half of these selected genres appear in training set while the rest are not, creating in-domain (matched) and cross-domain (mismatched) development/test sets.
Revised: Half of these selected genres appear in training set while the rest are not, creating in-domain (matched) and cross-domain (mismatched) development/test sets.

Original: We use the same data split as provided by
Revised: We use the same data split as provided by

Original: Williams et al.
Revised: Williams et al.

Original: Since test set labels are not provided, the test performance is obtained through submission on Kaggle.com 1 .
Revised: Since test set labels are not provided, the test performance is obtained through submission on Kaggle.com 1 .

Original: Each team is limited to two submissions per day.
Revised: Each team is limited to two submissions per day.

Original: Quora question pair Quora question pair dataset contains over 400k real world question pair selected from Quora.com.
Revised: Quora question pair Quora question pair dataset contains over 400k real world question pair selected from Quora.com.

Original: A binary annotation which stands for match (duplicate) or not match (not duplicate) is provided for each question pair.
Revised: A binary annotation which stands for match (duplicate) or not match (not duplicate) is provided for each question pair.

Original: In our case, duplicate question pair can be interpreted as entailment relation and not duplicate as neutral .
Revised: In our case, duplicate question pair can be interpreted as entailment relation and not duplicate as neutral .

Original: We use the same split ratio as mentioned in (Wang et al., 2017).
Revised: We use the same split ratio as mentioned in (Wang et al., 2017).

Original: We also study the human performance for both SNLI and MultiNLI.

Original: In the dev&test set on SNLI and matched&mismatched development set on MultiNLI, each sentences pair is provided with a set of “annotator labels”.

Original: The “annotator labels” contains a list of ﬁve labels annotated by ﬁve different annotators.

Original: The ﬁnal “gold label” is set to certain label if it has equal to or greater than three votes.

Original: Otherwise, the label is set to “-”, since there is no agreement.

Original: If we look from another perspective, the ratio that “annotator label” fails to match “gold label” is the human performance.

Original: For example, if three annotators vote A and the other two annotators vote B , then following the crowd-sourcing guideline, the “gold label” would be A ; in this case, the human performance is 60%

Original: ) leaderboard: https://inclass.kaggle.com/c/

Original: multinli-matched-open-evaluation/leaderboard

Original: cross

Original: domain

Original: mismatched

Original: ) leader-

Original: board

Original: : https://

Original: inclass

Original: kaggle

Original: .com

Original: c/

Original: multinli-

Original: mismatched for this particular sample.

Original: In light of this guideline, we calculate the human performance for SNLI dev&test set and MultiNLI development sets.

Original: We don’t take into account samples that labeled with “-”, since we discard them during model testing.

Original: The human performance score is 88.1% for SNLI development set, 87.7% for SNLI test set, 88.5% for MultiNLI matched development set and 89.2% for MultiNLI mismatched development set.

Original: Since we don’t have access to MultiNLI test set labels, we don’t provide the human performance for MultiNLI test set here.

Original: 4. E XPERIMENTS SETTING
Revised: 4. E XPERIMENTS SETTING

Original: We implement our algorithm with Tensorﬂow(Abadi et al., 2016) framework.
Revised: We implement our algorithm with Tensorﬂow(Abadi et al., 2016) framework.

Original: An Adadelta optimizer(Zeiler, 2012) with ρ as 0.95 and (cid:15) as 1e − 8 is used to optimize all the trainable weights.
Revised: An Adadelta optimizer(Zeiler, 2012) with ρ as 0.95 and (cid:15) as 1e − 8 is used to optimize all the trainable weights.

Original: The initial learning rate is set to 0.5 and batch size to 70.
Revised: The initial learning rate is set to 0.5 and batch size to 70.

Original: When the model does not improve best indomain performance for 30,000 steps, an SGD optimizer with learning rate of 3e − 4 is used to help model to ﬁnd a better local optimum.
Revised: When the model does not improve best indomain performance for 30,000 steps, an SGD optimizer with learning rate of 3e − 4 is used to help model to ﬁnd a better local optimum.

Original: Dropout layers are applied before all linear layers and after word-embedding layer.
Revised: Dropout layers are applied before all linear layers and after word-embedding layer.

Original: We use an exponential decayed keep rate during training, where the initial keep rate is 1.0 and the decay rate is 0.977 for every 10,000 step.
Revised: We use an exponential decayed keep rate during training, where the initial keep rate is 1.0 and the decay rate is 0.977 for every 10,000 step.

Original: We initialize our word embeddings with pre-trained 300D GloVe 840B vectors Pennington et al.
Revised: We initialize our word embeddings with pre-trained 300D GloVe 840B vectors Pennington et al.

Original: (2014) while the out-of-vocabulary word are randomly initialized with uniform distribution.
Revised: (2014) while the out-of-vocabulary word are randomly initialized with uniform distribution.

Original: The character embeddings are randomly initialized.
Revised: The character embeddings are randomly initialized.

Original: All weights are constraint by L2 regularization, and the L2 regularization at step t is calculated as follows:
Revised: All weights are constraint by L2 regularization, and the L2 regularization at step t is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: where L 2 FullRatio determines the maximum L2 regularization ratio, and L 2 FullStep determines at which step the maximum L2 regularization ratio would be applied on the L2 regularization. 
Revised: where L 2 FullRatio determines the maximum L2 regularization ratio, and L 2 FullStep determines at which step the maximum L2 regularization ratio would be applied on the L2 regularization. We domain

Revised: [Table]

Original: We choose L 2 FullRatio as 0 .
Revised: choose L 2 FullRatio as 0 .

Original: 9 e − 5 and L 2 FullStep as 100,000.
Revised: 9 e − 5 and L 2 FullStep as 100,000.

Original: The ratio of L2 penalty between the difference of two encoder weights is set to 1 e − 3 .
Revised: The ratio of L2 penalty between the difference of two encoder weights is set to 1 e − 3 .

Original: For a dense block in feature extraction layer, the number of layer n is set to 8 and growth rate is set to 20 .
Revised: For a dense block in feature extraction layer, the number of layer n is set to 8 and growth rate is set to 20 .

Original: The ﬁrst scale down ratio FSDR in feature extraction layer is set to 0 .
Revised: The ﬁrst scale down ratio FSDR in feature extraction layer is set to 0 .

Original: 3 and transitional scale down ratio TSDR is set to 0 .
Revised: 3 and transitional scale down ratio TSDR is set to 0 .

Original: The sequence length is set as a hard cutoff on all experiments: 48 for MultiNLI, 32 for SNLI and 24 for
Revised: The sequence length is set as a hard cutoff on all experiments: 48 for MultiNLI, 32 for SNLI and 24 for

Original: Quora Question Pair Dataset.
Revised: Quora Question Pair Dataset.

Original: During the experiments on MultiNLI, we use 15% of data from SNLI as in Williams et al.
Revised: During the experiments on MultiNLI, we use 15% of data from SNLI as in Williams et al.

Original: We select the parameter by the best run of development accuracy.
Revised: We select the parameter by the best run of development accuracy.

Original: Our ensembling approach considers the majority vote of the predictions given by multiple runs of the same model under different random parameter initialization.
Revised: Our ensembling approach considers the majority vote of the predictions given by multiple runs of the same model under different random parameter initialization.

Original: 4. E XPERIMENT ON M ULTI NLI
Revised: 4. E XPERIMENT ON M ULTI NLI

Original: We compare our result with all other published systems in Table 2.
Revised: We compare our result with all other published systems in Table 2.

Original: Besides ESIM, the state-of-theart model on SNLI, all other models appear at RepEval 2017 workshop.
Revised: Besides ESIM, the state-of-theart model on SNLI, all other models appear at RepEval 2017 workshop.

Original: RepEval 2017 workshop requires all submitted model to be sentence encoding-based model therefore alignment between sentences and memory module are not eligible for competition.
Revised: RepEval 2017 workshop requires all submitted model to be sentence encoding-based model therefore alignment between sentences and memory module are not eligible for competition.

Original: All models except ours share one common feature that they use LSTM as a essential building block as encoder.
Revised: All models except ours share one common feature that they use LSTM as a essential building block as encoder.

Original: Our approach, without using any recurrent structure, achieves the new state-of-the-art performance of 80.0%, exceeding current state-of-the-art performance by more than 5%.
Revised: Our approach, without using any recurrent structure, achieves the new state-of-the-art performance of 80.0%, exceeding current state-of-the-art performance by more than 5%.

Original: Unlike the observation from Nangia et al.
Revised: Unlike the observation from Nangia et al.

Original: (2017), we ﬁnd the out-of-domain test performance is consistently lower than in-domain test performance.
Revised: (2017), we ﬁnd the out-of-domain test performance is consistently lower than in-domain test performance.

Original: Selecting parameters from the best in-domain development accuracy partially contributes to this result.
Revised: Selecting parameters from the best in-domain development accuracy partially contributes to this result.

Original: 4. E XPERIMENT ON SNLI
Revised: 4. E XPERIMENT ON SNLI

Original: In Table 3, we compare our model to other model performance on SNLI.
Revised: In Table 3, we compare our model to other model performance on SNLI.

Original: Experiments (2-7) are sentence encoding based model.
Revised: Experiments (2-7) are sentence encoding based model.

Original: Bowman et al.
Revised: Bowman et al.

Original: (2016) provides a BiLSTM baseline.
Revised: (2016) provides a BiLSTM baseline.

Original: Vendrov et al.
Revised: Vendrov et al.

Original: (2015) adopts two layer GRU encoder with pre-trained ”skip-thoughts” vectors.
Revised: (2015) adopts two layer GRU encoder with pre-trained ”skip-thoughts” vectors.

Original: To capture sentence-level semantics, Mou et al.
Revised: To capture sentence-level semantics, Mou et al.

Original: (2015) use tree-based CNN and Bowman et al.
Revised: (2015) use tree-based CNN and Bowman et al.

Original: (2016) propose a stack-augmented parser-interpreter neural network (SPINN) which incorporates parsing information in a sequential manner.
Revised: (2016) propose a stack-augmented parser-interpreter neural network (SPINN) which incorporates parsing information in a sequential manner.

Original: Liu et al.
Revised: Liu et al.

Original: (2016) uses intra-attention on top of BiLSTM to generate sentence representation, and Munkhdalai & Yu (2016) proposes an memory augmented neural network to encode the sentence.
Revised: (2016) uses intra-attention on top of BiLSTM to generate sentence representation, and Munkhdalai & Yu (2016) proposes an memory augmented neural network to encode the sentence.

Original: The next group of model, experiments (8-18), uses cross sentence feature.
Revised: The next group of model, experiments (8-18), uses cross sentence feature.

Original: Rockt¨aschel et al.
Revised: Rockt¨aschel et al.

Original: (2015) aligns each sentence word-by-word with attention on top of LSTMs.Wang [Table] [Table]
Revised: (2015) aligns each sentence word-by-word with attention on top of LSTMs.

Original: Handcrafted features(

Original: [Table]

Original: al.,

Original: [Table]

Original: [Table]

Original: LSTM encoder(Bowman et al., 2016)

Original: [Table]

Original: pretrained GRU encoders(Vendrov et al., 2015)

Original: [Table]

Original: tree-based CNN encoders(Mou et al.,

Original: [Table]

Original: [Table]

Original: SPINN-PI encoders(Bowman et al., 2016)

Original: [Table]

Original: BiLSTM intra-attention encoders(Liu et al., 2016)

Original: [Table]

Original: NSE encoders(Munkhdalai & Yu, 2016)

Original: [Table]

Original: LSTM with attention(Rockt¨aschel et al., 2015)

Original: [Table]

Original: mLSTM(Wang & Jiang,

Original: [Table]

Original: LSTMN with deep attention fusion(Cheng et al.,

Original: [Table]

Original: decomposable attention model(Parikh et al., 2016)

Original: [Table]

Original: Intra-sentence attention + (11)(Parikh et al., 2016)

Original: [Table]

Original: BiMPM(Wang et al., 2017)

Original: [Table]

Original: NTI-SLSTM-LSTM(Yu & Munkhdalai, 2017)

Original: [Table]

Original: re-read LSTM(Sha et al., 2016)

Original: [Table]

Original: ESIM(Chen et al., 2016)

Original: [Table]

Original: ESIM ensemble with syntactic tree-LSTM(Chen et al., 2016)

Original: [Table]

Original: BiMPM (ensemble)(Wang et al., 2017)

Original: [Table]

Original: Human Performance

Original: [Table]

Original: DIIN

Original: [Table]

Original: ensemble) [Table] & Jiang (2015) enforces cross sentence attention word-by-word matching with the proprosed mLSTM model.
Revised: Wang & Jiang (2015) enforces cross sentence attention word-by-word matching with the proprosed mLSTM model.

Original: Cheng et al.
Revised: Cheng et al.

Original: (2016) proposes long short-term memory-network(LSTMN) with deep attention fusion that links the current word to previous word stored in memory.
Revised: (2016) proposes long short-term memory-network(LSTMN) with deep attention fusion that links the current word to previous word stored in memory.

Original: Parikh et al.
Revised: Parikh et al.

Original: (2016) decomposes the task into sub-problems and conquer them respectively.
Revised: (2016) decomposes the task into sub-problems and conquer them respectively.

Original: Yu & Munkhdalai (2017) proposes neural tree indexer, a full n-ary tree whose subtrees can be overlapped.
Revised: Yu & Munkhdalai (2017) proposes neural tree indexer, a full n-ary tree whose subtrees can be overlapped.

Original: Re-read LSTM proposed by Sha et al.
Revised: Re-read LSTM proposed by Sha et al.

Original: (2016) considers the attention vector of one sentence as the inner-state of
Revised: (2016) considers the attention vector of one sentence as the inner-state of

Original: LSTM for another sentence.
Revised: LSTM for another sentence.

Original: Chen et al.
Revised: Chen et al.

Original: (2016) propose a sequential model that infers locally, and a ensemble with tree-like inference module that further improves performance.
Revised: (2016) propose a sequential model that infers locally, and a ensemble with tree-like inference module that further improves performance.

Original: We show our model,
Revised: We show our model,

Original: DIIN, achieves state-of-the-art performance on the competitive leaderboard.
Revised: DIIN, achieves state-of-the-art performance on the competitive leaderboard.

Revised: [Table]

Revised: [Table]

Original: 4. E XPERIMENT ON Q UORA Q UESTION P AIR DATASET
Revised: 4. E XPERIMENT ON Q UORA Q UESTION P AIR DATASET

Original: In this subsection, we evaluate the effectiveness of our model for paraphrase identiﬁcation as natural language inference task.
Revised: In this subsection, we evaluate the effectiveness of our model for paraphrase identiﬁcation as natural language inference task.

Original: Other than our baselines, we compare with Wang et al.
Revised: Other than our baselines, we compare with Wang et al.

Original: (2017) and
Revised: (2017) and

Original: Tomar et al.
Revised: Tomar et al.

Original: B I MPM models different perspective of matching between sentence pair on both direction, then aggregates matching vector with LSTM.
Revised: B I MPM models different perspective of matching between sentence pair on both direction, then aggregates matching vector with LSTM.

Original: D EC A TT word and D EC A TT char uses[Table] automatically collected in-domain paraphrase data to noisy pretrain n -gram word embedding and ngram subword embedding correspondingly on decomposable attention model proposed by (Parikh et al., 2016).
Revised: D EC A TT word and D EC A TT char uses automatically collected in-domain paraphrase data to noisy pretrain n -gram word embedding and ngram subword embedding correspondingly on decomposable attention model proposed by (Parikh et al., 2016).

Original: In Table 4, our experiment shows DIIN has better performance than all other models and an ensemble score is higher than the former best result for more than 1 percent.
Revised: In Table 4, our experiment shows DIIN has better performance than all other models and an ensemble score is higher than the former best result for more than 1 percent.

Original: 4. A NALYSIS
Revised: 4. A NALYSIS

Original: Ablation Study We conduct a ablation study on our base model to examine the effectiveness of each component.
Revised: Ablation Study We conduct a ablation study on our base model to examine the effectiveness of each component.

Original: We study our model on MultiNLI dataset
Revised: We study our model on MultiNLI dataset

Original: and we use Matched validation score as the standard for model selection.
Revised: and we use Matched validation score as the standard for model selection.

Original: The result is shown in Table 5.
Revised: The result is shown in Table 5.

Revised: We studies how EM feature

Revised: [Table]

Revised: contributes to the system.

Revised: After removing the exact match binary feature, we ﬁnd the performance degrade to 78.2 on matched score on development set and 78.0 on mismatched score.

Original:  In the experiment 2, we remove the convolutional feature extractor and then model is structured as a sentence-encoding based model.
Revised: As observed in reading comprehension task Chen et al. (2017a) , the simple exact match feature does help the model to better understand the sentences. In the experiment 3, we remove the convolutional feature extractor and then model is structured as a sentence-encoding based model.

Original: The sentence representation matrix is max-pooled over time to obtain a feature vector.
Revised: The sentence representation matrix is max-pooled over time to obtain a feature vector.

Original: Once we have the feature vector p for premise and h for hypothesis, we use [ p ; h ; | p − h | ; p ◦ h ] as ﬁnal feature vector to classify the relationship.
Revised: Once we have the feature vector p for premise and h for hypothesis, we use [ p ; h ; | p − h | ; p ◦ h ] as ﬁnal feature vector to classify the relationship.

Original: We obtain 73.2 for matched score and 73.6 on mismatched data.
Revised: We obtain 73.2 for matched score and 73.6 on mismatched data.

Original: The result is competitive among other sentence-encoding based model.
Revised: The result is competitive among other sentence-encoding based model.

Original: We further study how encoding layer contribute in enriching the feature space in interaction tensor.
Revised: We further study how encoding layer contribute in enriching the feature space in interaction tensor.

Original: If we remove encoding layer completely, then we’ll obtain a 73.5 for matched score and 73.2 for mismatched score.
Revised: If we remove encoding layer completely, then we’ll obtain a 73.5 for matched score and 73.2 for mismatched score.

Original: The result demonstrate the feature extraction layer have powerful capability to capture the semantic feature.
Revised: The result demonstrate the feature extraction layer have powerful capability to capture the semantic feature.

Original: In experiment 4, we remove both self-attention and fuse gate, thus retaining only highway network.
Revised: In experiment 5, we remove both self-attention and fuse gate, thus retaining only highway network.

Original: The result improves to 77.7 and 77.3 respectively on matched and mismatched development set.
Revised: The result improves to 77.7 and 77.3 respectively on matched and mismatched development set.

Original: However, in experiment 5, when we only remove fuse gate, to our surprise, the performance degrade to 73.5 for matched score and 73.8 for mismatched.
Revised: However, in experiment 6, when we only remove fuse gate, to our surprise, the performance degrade to 73.5 for matched score and 73.8 for mismatched.

Original: On the other hand, if we use the addition of the representation after highway network and the representation after self-attention as skip connection as in experiment 9, the performance increase to 77.3 and 76.3.
Revised: On the other hand, if we use the addition of the representation after highway network and the representation after self-attention as skip connection as in experiment 9, the performance increase to 77.3 and 76.3.

Original: The comparison indicates self-attention layer makes the training harder to converge while a skip connection could ease the gradient ﬂow for both highway layer and self-attention layer.
Revised: The comparison indicates self-attention layer makes the training harder to converge while a skip connection could ease the gradient ﬂow for both highway layer and self-attention layer.

Original: By comparing the base model and the model the in experiment 6, we show that the fuse gate not only well serves as a skip connection, but also makes good decision upon which information the fuse for both representation.
Revised: By comparing the base model and the model the in experiment 6, we show that the fuse gate not only well serves as a skip connection, but also makes good decision upon which information the fuse for both representation.

Original: To show that dense interaction tensor contains more semantic information, we replace the dense interaction tensor with dot product similarity matrix between the encoded representation of premise and hypothesis.
Revised: To show that dense interaction tensor contains more semantic information, we replace the dense interaction tensor with dot product similarity matrix between the encoded representation of premise and hypothesis.

Original: The result shows that the dot product similarity matrix has an inferior capacity of semantic information.
Revised: The result shows that the dot product similarity matrix has an inferior capacity of semantic information.

Original: Dimensionality and Parameter number study To study the inﬂuence of the model dimension d which is also the channel number of interaction tensor, we design experiments to ﬁnd out whether dimension has inﬂuence on performance.

Original: We also present the parameter count of these models.

Original: The dimensionality is 448 where 300 comes from word embedding, 100 comes from char feature, 47 comes from Part of speech tagging and 1 comes from the binary exact match feature.

Original: Since

Original: Highway network sets the output dimensionality default as that in input, we design a variant to highway network so that different output size could be obtained.

Original: The variant of highway layer is

Original: [Table]

Original: [Table]

Original: designed as follows:

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: o

Original: [Equation] [Equation] where x i is the i -th vector of input matrix x , o i is the i -th vector of output matrix o , W (cid:62) t , W (cid:62) g , W (cid:62) x ∈ R d in × d out and b t , b g , b x ∈ R d out are trainable weights.
Revised: Another dimensionality study is provided in supplementary material.

Original: The result shows that higher dimension number have better performance when the dimension number is lower certain threshold, however, when the number of dimensionality is greater than the threshold, larger number of parameter and higher dimensionality doesn’t contribute to performance.

Original: In the case of SNLI, due to its simplicity in language pattern, 250D would be sufﬁce to obtain a good performance.

Original: On the other hand, it requires 350D to achieve a competitive performance on

Original: MultiNLI.

Original: We fail to reproduce our best performance with the new structure on MultiNLI.

Original: It shows that the additional layer on highway network doesn’t helps convergence.

Original: Error analysis To analyze the model prediction, we use annotated subset of development set provided by Williams et al.
Revised: Error analysis To analyze the model prediction, we use annotated subset of development set provided by Williams et al.

Original: (2017) that consists of 1,000 examples each tagged with zero or more following tags:
Revised: (2017) that consists of 1,000 examples each tagged with zero or more following tags:

Original: • CONDITIONAL: whether the sentence contains a conditional.
Revised: • CONDITIONAL: whether the sentence contains a conditional.

Original: • WORD OVERLAP: whether both sentences share more than 70% of their tokens.
Revised: • WORD OVERLAP: whether both sentences share more than 70% of their tokens.

Original: • NEGATION: whether a negation shows up in either sentence.
Revised: • NEGATION: whether a negation shows up in either sentence.

Original: • ANTO: whether two sentences contain antonym pair.
Revised: • ANTO: whether two sentences contain antonym pair.

Original: [Table]

Original: • LONG SENTENCE: whether premise or hypothesis is longer than 30 or 16 tokens respectively.
Revised: • LONG SENTENCE: whether premise or hypothesis is longer than 30 or 16 tokens respectively.

Original: • TENSE DIFFERENCE: whether any verb in two sentences uses different tense.
Revised: • TENSE DIFFERENCE: whether any verb in two sentences uses different tense.

Original: • ACTIVE/PASSIVE: whether there is an active-to-passive (or vice versa) transformation from the premise to the hypothesis.
Revised: • ACTIVE/PASSIVE: whether there is an active-to-passive (or vice versa) transformation from the premise to the hypothesis.

Original: • PARAPHRASE: whether the two sentences are close paraphrases•
Revised: • PARAPHRASE: whether the two sentences are close paraphrases

Revised: [Table]

Original: QUANTITY/TIME REASONING: whether understanding the pair requires quantity or time reasoning.
Revised: QUANTITY/TIME REASONING: whether understanding the pair requires quantity or time reasoning.

Original: • COREF: Whether the hypothesis contains a pronoun or referring expression that needs to be resolved using the premise.
Revised: • COREF: Whether the hypothesis contains a pronoun or referring expression that needs to be resolved using the premise.

Original: • QUANTIFIER: Whether either sentence contains one of the following quantiﬁer: much, enough, more, most, less, least, no, none, some, any, many, few, several, almost, nearly.
Revised: • QUANTIFIER: Whether either sentence contains one of the following quantiﬁer: much, enough, more, most, less, least, no, none, some, any, many, few, several, almost, nearly.

Original: • MODAL:
Revised: • MODAL:

Original: Whether one of the following modal verbs appears in either sentence: can, could, may, might, must, will, would, should.
Revised: Whether one of the following modal verbs appears in either sentence: can, could, may, might, must, will, would, should.

Original: • BELIEF:
Revised: • BELIEF:

Original: Whether one of the following belief verbs appear in either sentence: know, believe, understand, doubt, think, suppose, recognize, forget, remember, imagine, mean, agree, disagree, deny, promise.
Revised: Whether one of the following belief verbs appear in either sentence: know, believe, understand, doubt, think, suppose, recognize, forget, remember, imagine, mean, agree, disagree, deny, promise.

Original: For more detailed descriptions, please resort to Williams et al.
Revised: For more detailed descriptions, please resort to Williams et al.

Original: The result is shown in Table 7.
Revised: The result is shown in Table 6.

Original: We ﬁnd DIIN is consistently better on sentence pair with WORD OVERLAP, ANTO, LONG
Revised: We ﬁnd DIIN is consistently better on sentence pair with WORD OVERLAP, ANTO, LONG

Original: SENTENCE, PARAPHRASE and BELIEF tags by a large margin.
Revised: SENTENCE, PARAPHRASE and BELIEF tags by a large margin.

Original: During investigation, we hypothesize exact match feature helps the model to better understand paraphrase, therefore we study the result from second ablation ablation study where exact match feature is not used.
Revised: During investigation, we hypothesize exact match feature helps the model to better understand paraphrase, therefore we study the result from second ablation ablation study where exact match feature is not used.

Original: Surprisingly, the model without exact model feature does not work worse on PARAPHRASE, instead, the accuracy on ANTO drops about 10%.
Revised: Surprisingly, the model without exact model feature does not work worse on PARAPHRASE, instead, the accuracy on ANTO drops about 10%.

Original: DIIN is also work well on LONG SENTENCE, partially because the receptive ﬁeld is large enough to cover all tokens.
Revised: DIIN is also work well on LONG SENTENCE, partially because the receptive ﬁeld is large enough to cover all tokens.

Original: Visualization We also visualize the hidden representation from interaction tensor I and the feature map from ﬁrst dense block in Figure 2.
Revised: Visualization We also visualize the hidden representation from interaction tensor I and the feature map from ﬁrst dense block in Figure 2.

Original: We pick a sentence pair whose premise is “ South Carolina has no referendum right, so the Supreme Court canceled the vote and upheld the ban. ”
Revised: We pick a sentence pair whose premise is “ South Carolina has no referendum right, so the Supreme Court canceled the vote and upheld the ban. ”

Original: and hypothesis is “ South Carolina has a referendum right, so the Supreme Court was powerless over the state. ”.
Revised: and hypothesis is “ South Carolina has a referendum right, so the Supreme Court was powerless over the state. ”.

Original: The upper row of ﬁgures are sampled from hidden representation of interaction tensor I .
Revised: The upper row of ﬁgures are sampled from hidden representation of interaction tensor I .

Original: We observe the values of neurons are highly correlated row-wise and column-wise in the interaction tensor I and different channel of hidden representation shows different aspect of interaction.
Revised: We observe the values of neurons are highly correlated row-wise and column-wise in the interaction tensor I and different channel of hidden representation shows different aspect of interaction.

Original: Though in certain channel same words, “ referendum ”, or phrases, “ supreme court ”, cause activation, different word or phrase pair, such as “ ban ” and “ powerless over ”, also cause activation in other activation.
Revised: Though in certain channel same words, “ referendum ”, or phrases, “ supreme court ”, cause activation, different word or phrase pair, such as “ ban ” and “ powerless over ”, also cause activation in other activation.

Original: It shows the model’s strong capacity of understanding text in different perspective.
Revised: It shows the model’s strong capacity of understanding text in different perspective.

Original: The lower row of Figure 2 shows the feature map from ﬁrst dense block.
Revised: The lower row of Figure 2 shows the feature map from ﬁrst dense block.

Original: After being convolved from the interaction tensor and previous feature map, new feature maps shows activation in different position, demonstrating differentsemantic features are found.
Revised: After being convolved from the interaction tensor and previous feature map, new feature maps shows activation in different position, demonstrating different are indicate

Original: The ﬁrst ﬁgure in the lower row has similar pattern as normal attention weight whereas others has no obvious pattern.

Original: Different channels of feature maps indicate different kinds of semantic feature.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We show the interaction tensor (or attention weight) contains semantic information to understand the natural language.
Revised: We show the interaction tensor (or attention weight) contains semantic information to understand the natural language.

Original: We introduce Interactive Inference Network, a novel class of architecture that allows the model to solve NLI or NLI alike tasks via extracting semantic feature from interaction tensor end-to-end.
Revised: We introduce Interactive Inference Network, a novel class of architecture that allows the model to solve NLI or NLI alike tasks via extracting semantic feature from interaction tensor end-to-end.

Original: One instance of such architecture, Densely Interactive Inference Network (DIIN), achieves state-of-the-art performance on multiple datasets.
Revised: One instance of such architecture, Densely Interactive Inference Network (DIIN), achieves state-of-the-art performance on multiple datasets.

Original: By ablating each component in DIIN and changing the dimensionality, we show the effectiveness of each component in DIIN.
Revised: By ablating each component in DIIN and changing the dimensionality, we show the effectiveness of each component in DIIN.

Original: Though we have the initial exploration of natural language inference in interaction space, the full potential is not yet clear.
Revised: Though we have the initial exploration of natural language inference in interaction space, the full potential is not yet clear.

Original: We will keep exploring the potential of interaction space.
Revised: We will keep exploring the potential of interaction space.

Original: Incorporating common-sense knowledge from external resources such as knowledge base to leverage the capacity of the mode is another research goal of ours.
Revised: Incorporating common-sense knowledge from external resources such as knowledge base to leverage the capacity of the mode is another research goal of ours.

Revised: A S UPPLEMENTARY M ATERIAL

Revised: Dimensionality and Parameter number study To study the inﬂuence of the model dimension d which is also the channel number of interaction tensor, we design experiments to ﬁnd out whether dimension has inﬂuence on performance.

Revised: We also present the parameter count of these models.

Revised: The dimensionality is 448 where 300 comes from word embedding, 100 comes from char feature, 47 comes from Part of speech tagging and 1 comes from the binary exact match feature.

Revised: Since

Revised: Highway network sets the output dimensionality default as that in input, we design a variant to highway network so that different output size could be obtained.

Revised: The variant of highway layer is designed as follows:

Revised: [Equation]

Revised: where x i is the i -th vector of input matrix x , o i is the i -th vector of output matrix o , W (cid:62) t , W (cid:62) g ,

Revised: W (cid:62) x ∈ R d in × d out and b t , b g , b x ∈ R d out are trainable weights.

Revised: The result shows that higher dimension number have better performance when the dimension number is lower certain threshold, however, when the number of dimensionality is greater than the threshold, larger number of parameter and higher dimensionality doesn’t contribute to performance.

Revised: In the case of SNLI, due to its simplicity in language pattern, 250D would be sufﬁce to obtain a good performance.

Revised: On the other hand, it requires 350D to achieve a competitive performance on

Revised: MultiNLI.

Revised: We fail to reproduce our best performance with the new structure on MultiNLI.

Revised: It shows that the additional layer on highway network doesn’t helps convergence.

Revised: [Table]