Original: L EARNING E NTAILMENT -B ASED S ENTENCE E MBEDDINGS FROM N ATURAL L ANGUAGE I NFERENCE
Revised: L EARNING E NTAILMENT -B ASED S ENTENCE E MBEDDINGS FROM N ATURAL L ANGUAGE I NFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large datasets on natural language inference are a potentially valuable resource for inducing semantic representations of natural language sentences.
Revised: Large datasets on natural language inference are a potentially valuable resource for inducing semantic representations of natural language sentences.

Original: But in many such models the embeddings computed by the sentence encoder goes through an
Revised: But in many such models the embeddings computed by the sentence encoder goes through an

Original: MLP-based interaction layer before predicting its label, and thus some of the information about textual entailment is encoded in the interpretation of sentence embeddings given by this parameterised MLP.
Revised: MLP-based interaction layer before predicting its label, and thus some of the information about textual entailment is encoded in the interpretation of sentence embeddings given by this parameterised MLP.

Original: In this work we propose a simple interaction layer based on predeﬁned entailment and contradiction scores applied directly to the sentence embeddings.
Revised: In this work we propose a simple interaction layer based on predeﬁned entailment and contradiction scores applied directly to the sentence embeddings.

Original: This parameter-free interaction model achieves results on natural language inference competitive with MLP-based models, demonstrating that the trained sentence embeddings directly represent the information needed for textual entailment, and the inductive bias of this model leads to better generalisation to other related datasets.
Revised: This parameter-free interaction model achieves results on natural language inference competitive with MLP-based models, demonstrating that the trained sentence embeddings directly represent the information needed for textual entailment. The inductive bias of this model leads to better generalisation to other natural language inference datasets, and the resulting sentence embeddings are better for other semantic tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: After the enormous success of learning representations of words, learning representations of sentences has become one of the most important challenges in natural language understanding.
Revised: After the enormous success of learning representations of words, learning representations of sentences has become one of the most important challenges in natural language understanding.

Original: Many models have been proposed for embedding sentences in a vector space, but the methods for interpreting those representations are dominated by two approaches: using the dot product (or cosine) to measure the similarity between two vectors, or training a parameterised function to do the interpretation for speciﬁc tasks (e.g. (Conneau et al., 2017)).
Revised: Many models have been proposed for embedding sentences in a vector space, but the methods for interpreting those representations are dominated by two approaches: using the dot product (or cosine) to measure the similarity between two vectors, or training a parameterised function to do the interpretation for speciﬁc tasks (e.g. (Conneau et al., 2017)).

Original: In this paper we consider a different form of interpretation, based neither on similarity nor on a parameterised function.
Revised: In this paper we consider a different form of interpretation, based neither on similarity nor on a parameterised function.

Original: We investigate embedding sentences into vector representations of information inclusion and contradiction.
Revised: We investigate embedding sentences into vector representations of information inclusion and contradiction.

Original: 1. M OTIVATION
Revised: 1. M OTIVATION

Original: Information inclusion, called entailment, is a fundamental concept in theories of the semantics of natural language.
Revised: Information inclusion, called entailment, is a fundamental concept in theories of the semantics of natural language.

Original: This motivated the proposal of recognising entailment in text as a generic task which captures a very wide range of issues in natural language understanding (Dagan et al., 2005), which has been demonstrated by the conversion of a large number of semantic datasets into a single textual entailment task (Poliak et al., 2018a).
Revised: This motivated the proposal of recognising entailment in text as a generic task which captures a very wide range of issues in natural language understanding (Dagan et al., 2005), which has been demonstrated by the conversion of a large number of semantic datasets into a single textual entailment task (Poliak et al., 2018a).

Original: The fundamental nature of this task has motivated the development of many datasets for textual entailment (Poliak et al., 2018a).
Revised: The fundamental nature of this task has motivated the development of many datasets for textual entailment (Poliak et al., 2018a).

Original: The more recent textual entailment datasets have extended this task to include annotations for contradiction, which are often referred to as natural language inference (NLI) datasets.
Revised: The more recent textual entailment datasets have extended this task to include annotations for contradiction, which are often referred to as natural language inference (NLI) datasets.

Original: Modelling these semantic relationships between sentences is not only a crucial problem in natural language processing, but it also has a wide variety of applications, such as question answering (Harabagiu & Hickl, 2006), producing abstract summaries (Lacatusu et al., 2006; Yan et al., 2011a;b), and machine translation evaluation (Pad´o et al., 2009).
Revised: Modelling these semantic relationships between sentences is not only a crucial problem in natural language processing, but it also has a wide variety of applications, such as question answering (Harabagiu & Hickl, 2006), producing abstract summaries (Lacatusu et al., 2006; Yan et al., 2011a;b), and machine translation evaluation (Pad ´ o et al., 2009).

Original: It is possible to transfer the predictions of an NLI model directly to the task, but it has been shown that NLI models trained on large
Revised: It is possible to transfer the predictions of an NLI model directly to the task, but it has been shown that NLI models trained on large

Original: NLI benchmarks such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018a) do not transfer well to other NLI benchmarks (Talman & Chatzikyriakidis, 2019).
Revised: NLI benchmarks such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018a) do not transfer well to other NLI benchmarks (Talman & Chatzikyriakidis, 2019).

Original: Despite recent efforts to improve generalisation (Belinkov et al., 2019b), this remains an open question.
Revised: Despite recent efforts to improve generalisation (Belinkov et al., 2019b), this remains an open question.

Original: A more ﬂexible and powerful approach is to transfer the knowledge learned about the NLI task to other tasks with representation learning.
Revised: A more ﬂexible and powerful approach is to transfer the knowledge learned about the NLI task to other tasks with representation learning.

Original: For example, Conneau et al.
Revised: For example, Conneau et al.

Original: (2017) train sentence embeddings on the NLI task and demonstrate their usefulness on other tasks.
Revised: (2017) train sentence embeddings on the NLI task and demonstrate their usefulness on other tasks.

Original: But there is still no clear interpretation of these embeddings in terms of entailment, other than via a trained parameterised function.
Revised: But there is still no clear interpretation of these embeddings in terms of entailment, other than via a trained parameterised function.

Original: In this work we propose a novel approach to learning sentence embeddings from NLI data, where the sentence embeddings have a direct interpretation in terms of entailment and contradiction.
Revised: In this work we propose a novel approach to learning sentence embeddings from NLI data, where the sentence embeddings have a direct interpretation in terms of entailment and contradiction.

Original: Analogously to the use of the dot product or cosine to deﬁne similarity over a vector space, we deﬁne entailment and contradiction as two parameter-free operators over pairs of vectors.
Revised: Analogously to the use of the dot product or cosine to deﬁne similarity over a vector space, we deﬁne entailment and contradiction as two parameter-free operators over pairs of vectors.

Original: The entailment operator provides a score of how well one vector includes all the information in the other vector.
Revised: The entailment operator provides a score of how well one vector includes all the information in the other vector.

Original: The contradiction operator provides a score of to what extent two vectors contradict each other.
Revised: The contradiction operator provides a score of to what extent two vectors contradict each other.

Original: We use these operators to train sentence encoders which embed sentences into a vector space which has these interpretations, using data for the Natural Language Inference task.
Revised: We use these operators to train sentence encoders which embed sentences into a vector space which has these interpretations, using data for the Natural Language Inference task.

Original: 1. O VERVIEW
Revised: 1. O VERVIEW

Original: Natural language inference involves predicting whether a premise entails the given hypothesis , whether they contradict each other, or neither (Bowman et al., 2015).
Revised: Natural language inference involves predicting whether a premise entails the given hypothesis , whether they contradict each other, or neither (Bowman et al., 2015).

Original: Due to the availability of large-scale datasets for natural language inference (Bowman et al., 2015; Williams et al., 2018b), neural network models have become popular for textual entailment.
Revised: Due to the availability of large-scale datasets for natural language inference (Bowman et al., 2015; Williams et al., 2018b), neural network models have become popular for textual entailment.

Original: We can characterise these models in terms of three stages.
Revised: We can characterise these models in terms of three stages.

Original: First, the encoding stage computes d -dimensional embeddings of the premise and of the hypothesis.
Revised: First, the encoding stage computes d -dimensional embeddings of the premise and of the hypothesis.

Original: Then the interaction layer models the interactions between the two sentences.
Revised: Then the interaction layer models the interactions between the two sentences.

Original: Finally a classiﬁer layer makes the prediction of entailment, contradiction or neutral.
Revised: Finally a classiﬁer layer makes the prediction of entailment, contradiction or neutral.

Original: While most of the literature has been concerned with improving the encoding stage, in this paper we focus on the interaction layer.
Revised: While most of the literature has been concerned with improving the encoding stage, in this paper we focus on the interaction layer.

Original: For the interaction layer, passing the sentence embeddings plus their heuristic matching features (Mou et al., 2016) through a multi-layer perceptron (MLP) has become the standard practice for textual entailment models (Conneau et al., 2017; Bowman et al., 2016;
Revised: For the interaction layer, passing the sentence embeddings plus their heuristic matching features (Mou et al., 2016) through a multi-layer perceptron (MLP) has become the standard practice for textual entailment models (Conneau et al., 2017; Bowman et al., 2016;

Original: Kim et al., 2019; Pan et al., 2018; Yoon et al., 2018; Kiela et al., 2018; Conneau et al., 2018).
Revised: Kim et al., 2019; Pan et al., 2018; Yoon et al., 2018; Kiela et al., 2018; Conneau et al., 2018b).

Original: The heuristic matching features are component-wise measures of similarity between premise and hypothesis, most commonly implemented as the absolute difference and the component-wise multiplication.
Revised: The heuristic matching features are component-wise measures of similarity between premise and hypothesis, most commonly implemented as the absolute difference and the component-wise multiplication.

Original: However, textual entailment is not only about similarity, since it reﬂects an information inclusion relationship.
Revised: However, textual entailment is not only about similarity, since it reﬂects an information inclusion relationship.

Original: Moreover, most similarity features are symmetric, whereas entailment is an intrinsically asymmetric relation.
Revised: Moreover, most similarity features are symmetric, whereas entailment is an intrinsically asymmetric relation.

Original: It is only after the MLP that the sentence embeddings and similarity-based features get transformed into a representation which can be directly mapped to entailment, contradiction and neutral.
Revised: It is only after the MLP that the sentence embeddings and similarity-based features get transformed into a representation which can be directly mapped to entailment, contradiction and neutral.

Original: Often the MLP plays a signiﬁcant role in the model’s architecture, for example requiring 800k of the total 2.1m model parameters in the state-of-the-art Dynamic Self-Attention model (Yoon et al., 2018).
Revised: Often the MLP plays a signiﬁcant role in the model’s architecture, for example requiring 800k of the total 2.1m model parameters in the state-of-the-art Dynamic Self-Attention model (Yoon et al., 2018).

Original: In this paper, we propose a parameter-free interaction layer consisting of only a few scores, including entailment and contradiction, deﬁned in terms of the entailment-vectors framework of Henderson &
Revised: In this paper, we propose a parameter-free interaction layer consisting of only a few scores, including entailment and contradiction, deﬁned in terms of the entailment-vectors framework of Henderson &

Original: Popa (2016), which are passed to a log-linear classiﬁcation layer.
Revised: Popa (2016), which are passed to a log-linear classiﬁcation layer.

Original: This forces the sentence embeddings to have a direct interpretation in terms of entailment and  contradiction.
Revised: This forces the sentence embeddings to have a direct interpretation in terms of entailment and contradiction, and imposes an inductive bias towards learning entailment and contradiction.

Original: Despite drastically reducing the number of trained parameters, this simple interaction layer is empirically competitive with MLP-based models.
Revised: Despite drastically reducing the number of trained parameters, this simple interaction layer is empirically competitive with MLP-based models.

Original: Because it is parameter-free, all of the information learned about entailment and contradiction must be encoded directly in the sentence embeddings, rather than indirectly via its interpretation by the trained MLP.
Revised: Because it is parameter-free, all of the information learned about entailment and contradiction must be encoded directly in the sentence embeddings, rather than indirectly via its interpretation by the trained MLP.

Original: Thus information about entailment or contradiction can be transferred to other tasks simply by transferring the sentence embeddings.
Revised: Thus information about entailment or contradiction can be transferred to other tasks simply by transferring the sentence embeddings.

Original: An ablation analysis indicates that the entailment and contradiction scores are indeed the main factors in the model’s success.
Revised: An ablation analysis indicates that the entailment and contradiction scores are indeed the main factors in the model’s success.

Original: Moreover, we evaluate our trained modelsin a transfer learning setting by testing them on NLI datasets that have not been used during training, and ﬁnd  better transfer performance for our proposed model.
Revised: When we test our trained models on related datasets that have not been used for training, we ﬁnd that the inductive bias of our proposed model results in better transfer performance. When we test our induced sentence embeddings on other tasks, we ﬁnd better transfer performance on semantic tasks.

Original: In summary, we propose a novel, theoretically grounded, parameter-free model of the interactions between premise and hypothesis for entailment and contradiction.
Revised: In summary, we propose a novel, theoretically grounded, parameter-free model of the interactions between premise and hypothesis for entailment and contradiction.

Original: Evaluation on benchmark natural language inference datasets show this model is competitive with modelling interaction with an MLP.
Revised: Evaluation on benchmark natural language inference datasets show this model is competitive with modelling interaction with an MLP.

Original: The resulting sentence embeddings have a direct entailment-based interpretation, which transfers better  to different NLI datasets.
Revised: The resulting sentence embeddings have a direct entailment-based interpretation, and the model’s inductive bias results in better transfer to different NLI datasets and different semantic tasks.

Original:  M ODELLING E NTAILMENT AND C ONTRADICTION R ELATIONS
Revised:  M ODELLING E NTAILMENT AND C ONTRADICTION R ELATIONS

Original: The task of textual entailment takes premise and hypothesis sentences and classiﬁes their relationship into entailment , contradiction , or neutral .
Revised: The task of textual entailment takes premise and hypothesis sentences and classiﬁes their relationship into entailment , contradiction , or neutral .

Original: To train our sentence embeddings, we adopt the widely used approach of ﬁrst encoding each sentence in an embedding, then modelling the interactions between the premise and hypothesis, followed by a softmax classiﬁer which estimates the probability of each label (Conneau et al., 2018).
Revised: To train our sentence embeddings, we adopt the widely used approach of ﬁrst encoding each sentence in an embedding, then modelling the interactions between the premise and hypothesis, followed by a softmax classiﬁer which estimates the probability of each label (Conneau et al., 2018b).

Original: 2. E NCODING S TAGE
Revised: 2. E NCODING S TAGE

Original: The encoding stage embeds the premise s 1 and hypothesis s 2 into two d -dimensional vector representations p , h = enc( s 1 ,s 2) , where enc is a sentence encoder, and p and h are the premise and hypothesis embeddings respectively.
Revised: The encoding stage embeds the premise s 1 and hypothesis s 2 into two d -dimensional vector representations p , h = enc( s 1 ,s 2) , where enc is a sentence encoder, and p and h are the premise and hypothesis embeddings respectively.

Original: There are two types of encoders; intra-sentence encoders only look at each sentence individually, while inter-sentence encoders consider both sentences.
Revised: There are two types of encoders; intra-sentence encoders only look at each sentence individually, while inter-sentence encoders consider both sentences.

Original: In our ﬁrst set of experiments, we consider models with both single-sentence and inter-sentence encoders, speciﬁcally the two baselines from the GLUE benchmark (Wang et al., 2019).
Revised: In our ﬁrst set of experiments, we consider models with both single-sentence and inter-sentence encoders, speciﬁcally the two baselines from the GLUE benchmark (Wang et al., 2019).

Original: The single-sentence encoder is a bidirectional LSTM (Bi-LSTM) with max-pooling, which Conneau et al.
Revised: The single-sentence encoder is a bidirectional LSTM (Bi-LSTM) with max-pooling, which Conneau et al.

Original: (2017) found to work best among a diverse set of encoders.
Revised: (2017) found to work best among a diverse set of encoders.

Original: The inter-sentence encoder adds attention over the other sentence (Wang et al., 2019).
Revised: The inter-sentence encoder adds attention over the other sentence (Wang et al., 2019).

Original: In our second set of experiments, we consider models with the single-sentence encoder of (Reimers & Gurevych, 2019), based on a pre-trained BERT model
Revised: In our second set of experiments, we consider models with the single-sentence encoder of (Reimers & Gurevych, 2019), based on a pre-trained BERT model

Original: Devlin et al.
Revised: Devlin et al.

Original: 2. MLP I NTERACTION S TAGE
Revised: 2. MLP I NTERACTION S TAGE

Original: Most previous neural network approaches to NLI model the interaction between sentences with a non-linear MLP.
Revised: Most previous neural network approaches to NLI model the interaction between sentences with a non-linear MLP.

Original: The input to this MLP includes the embeddings of the hypothesis h and premise p obtained from the encoding stage.
Revised: The input to this MLP includes the embeddings of the hypothesis h and premise p obtained from the encoding stage.

Original: Mou et al.
Revised: Mou et al.

Original: (2016) proposed adding heuristic matching features in order to capture the similarity of the two sentences, which have since been widely adopted (Conneau et al., 2017;
Revised: (2016) proposed adding heuristic matching features in order to capture the similarity of the two sentences, which have since been widely adopted (Conneau et al., 2017;

Original: Chen et al., 2017; Bowman et al., 2016;
Revised: Chen et al., 2017; Bowman et al., 2016;

Original: Kim et al., 2019; Pan et al., 2018; Yoon et al., 2018; Kiela et al., 2018).
Revised: Kim et al., 2019; Pan et al., 2018; Yoon et al., 2018; Kiela et al., 2018).

Original: These features are most commonly the element-wise product and absolute difference between the two sentence embeddings, giving the input vector m :
Revised: These features are most commonly the element-wise product and absolute difference between the two sentence embeddings, giving the input vector m :

Original: [Equation]
Revised: [Equation]

Original: The MLP then computes a hidden layer, from which the softmax classiﬁer predicts the labels.
Revised: The MLP then computes a hidden layer, from which the softmax classiﬁer predicts the labels.

Original: [Equation]
Revised: [Equation]

Original: where W c ∈ R 3 × n , b c ∈ R 3 , W e ∈ R n × 4 d , b e ∈ R n , and n is the size of the hidden layer.
Revised: where W c ∈ R 3 × n , b c ∈ R 3 , W e ∈ R n × 4 d , b e ∈ R n , and n is the size of the hidden layer.

Original: Because of the high dimensionality d of the sentence embeddings, the number of parameters in this interaction layer ( W e ) can be large.
Revised: Because of the high dimensionality d of the sentence embeddings, the number of parameters in this interaction layer ( W e ) can be large.

Original: Following Wang et al.
Revised: Following Wang et al.

Original: (2019), we set n = 512 .
Revised: (2019), we set n = 512 .

Original: 2. T HE P ROPOSED I NTERACTION S TAGE
Revised: 2. T HE P ROPOSED I NTERACTION S TAGE

Original: We propose an alternative interaction stage with no parameters.
Revised: We propose an alternative interaction stage with no parameters.

Original: The input is the two sentence embeddings, and the output is just 5 scores, which are then input to the softmax classiﬁer.
Revised: The input is the two sentence embeddings, and the output is just 5 scores, which are then input to the softmax classiﬁer.

Original: These predeﬁned scores provide a strong inductive bias to the model, and force the sentence embeddings to have the interpretation required by these scores.
Revised: These predeﬁned scores provide a strong inductive bias to the model, and force the sentence embeddings to have the interpretation required by these scores.

Original: Normalisation In the 5 scores, the sentence embeddings are ﬁrst passed through a sigmoid function, and we want them to use the full output range of this function.
Revised: Normalisation In the 5 scores, the sentence embeddings are ﬁrst passed through a sigmoid function, and we want them to use the full output range of this function.

Original: We therefore apply layer normalisation to the sentence embeddings to make their mean zero and their standard deviation s , where s is a hyperparameter set to 6 in all experiments.
Revised: We therefore apply layer normalisation to the sentence embeddings to make their mean zero and their standard deviation s , where s is a hyperparameter set to 6 in all experiments.

Original: We use y , x to refer to the embeddings p , h respectively after applying normalisation.
Revised: We use y , x to refer to the embeddings p , h respectively after applying normalisation.

Original: Entailment Vectors Framework Henderson & Popa (2016) propose a vector-space framework for entailment which explicitly models the information known about a word by interpreting each dimension of a word embedding as the probability that a certain feature is known to be present.
Revised: Entailment Vectors Framework Henderson & Popa (2016) propose a vector-space framework for entailment which explicitly models the information known about a word by interpreting each dimension of a word embedding as the probability that a certain feature is known to be present.

Original: So zero is interpreted as unknown, and not as necessarily false.
Revised: So zero is interpreted as unknown, and not as necessarily false.

Original: Entailment is when all the features that are known about the entailed vector are also known about the entailing vector.
Revised: Entailment is when all the features that are known about the entailed vector are also known about the entailing vector.

Original: They propose different operators for computing the probability of this entailment.
Revised: They propose different operators for computing the probability of this entailment.

Original: Previous work only applied this framework to lexical entailment (Henderson & Popa, 2016), but we apply it to textual entailment.
Revised: Previous work only applied this framework to lexical entailment (Henderson & Popa, 2016), but we apply it to textual entailment.

Original: Modelling Contradictory Information
Revised: Modelling Contradictory Information

Original: The original entailment vectors framework only models binary entailment.
Revised: The original entailment vectors framework only models binary entailment.

Original: We extend this representation to also model contradiction.
Revised: We extend this representation to also model contradiction.

Original: We split entailment vectors into two blocks, one x 1: d 2 for modelling features f and another x d 2 : d for their negation ¬ f (as suggested by Henderson & Popa (2016) for a different purpose).
Revised: We split entailment vectors into two blocks, one x 1: d 2 for modelling features f and another x d 2 : d for their negation ¬ f (as suggested by Henderson & Popa (2016) for a different purpose).

Original: So if x k , k ∈ { 1 , . . .
Revised: So if x k , k ∈ { 1 , . . .

Original: , d 2 } , represents that something is known to be true, then x k + d 2 represents that it is known to be false, and thus having both these features active is a contradiction.
Revised: , d 2 } , represents that something is known to be true, then x k + d 2 represents that it is known to be false, and thus having both these features active is a contradiction.

Original: Entailment Score:
Revised: Entailment Score:

Original: We compute the entailment score between two sentences using the factorised entailment operator ( y ˜ ⇒ x ) proposed by Henderson & Popa (2016), which measures the probability of entailment in the k-th dimension as S k ( entail | x , y )
Revised: We compute the entailment score between two sentences using the factorised entailment operator ( y ˜ ⇒ x ) proposed by Henderson & Popa (2016), which measures the probability of entailment in the k-th dimension as S k ( entail | x , y )

Original: = 1 − σ ( − y k ) σ ( x k ) , and which requires entailment to be probable in every dimension, giving us:
Revised: = 1 − σ ( − y k ) σ ( x k ) , and which requires entailment to be probable in every dimension, giving us:

Original: [Equation]
Revised: [Equation]

Original: Contradiction Score: We consider two sentences to contradict each other if their respective embeddings have contradictory features in at least one dimension.
Revised: Contradiction Score: We consider two sentences to contradict each other if their respective embeddings have contradictory features in at least one dimension.

Original: Our contradiction score for the k-th dimension ∈ { 1 , . . .
Revised: Our contradiction score for the k-th dimension ∈ { 1 , . . .

Original: , d 2 } is thus:
Revised: , d 2 } is thus:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term models that the feature is known to be true in the hypothesis and known to be false in the premise, the second term the reverse, and the third term both.
Revised: where the ﬁrst term models that the feature is known to be true in the hypothesis and known to be false in the premise, the second term the reverse, and the third term both.

Original: The total score of contradiction is the complement of not having contradiction in any dimension:
Revised: The total score of contradiction is the complement of not having contradiction in any dimension:

Original: [Equation]
Revised: [Equation]

Original: Neutral Score:
Revised: Neutral Score:

Original: We deﬁne a neutral score as the non-negative complement of the contradiction and entailment scores, computed as:[Equation] entail [Equation]
Revised: We deﬁne a neutral score as the non-negative complement of the contradiction and entailment scores, computed as:

Original: contradict

Original: [Equation]
Revised: [Equation]

Original: The ReLU function avoids negative scores, and its nonlinearity makes this score non-redundant in the log-linear softmax classiﬁer.
Revised: The ReLU function avoids negative scores, and its nonlinearity makes this score non-redundant in the log-linear softmax classiﬁer.

Original: Similarity Scores: Since similarity information is known to help textual entailment (Mou et al., 2016), we employ two scores that can be viewed as condensed versions of the heuristic matching features, namely averages of element-wise multiplication and absolute difference, respectively:
Revised: Similarity Scores: Since similarity information is known to help textual entailment (Mou et al., 2016), we employ two scores that can be viewed as condensed versions of the heuristic matching features, namely averages of element-wise multiplication and absolute difference, respectively:

Original: [Equation]
Revised: [Equation]

Original: Initialisation For the success of our models, we found it critical to initialise the output layer in such a way that the scores are associated with their respective class label.
Revised: Initialisation For the success of our models, we found it critical to initialise the output layer in such a way that the scores are associated with their respective class label.

Original: To this end, we add 1 to the random initial weight connecting the entailment score to the entailment class, the contradiction score to the contradiction class, and the neutral score to the neutral class, respectively.
Revised: To this end, we add 1 to the random initial weight connecting the entailment score to the entailment class, the contradiction score to the contradiction class, and the neutral score to the neutral class, respectively.

Original: We observed that without this initialisation the model takes several epochs to train before achieving better than random performance.
Revised: We observed that without this initialisation the model takes several epochs to train before achieving better than random performance.

Original:  E VALUATION ON NLI D ATASETS
Revised:  E VALUATION ON NLI D ATASETS

Original: We ﬁrst evaluate our approach to learning sentence embeddings from NLI data without using any resources external to the NLI datasets themselves.
Revised: We ﬁrst evaluate our approach to learning sentence embeddings from NLI data without using any resources external to the NLI datasets themselves.

Original: We train and test our models on two popular largescale datasets for textual entailment, SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018a).
Revised: We train and test our models on two popular largescale datasets for textual entailment, SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018a).

Original: In addition to evaluating classiﬁcation performance, we conduct an ablation study to determine whether our sentence embeddings and associated scores have the interpretations claimed.
Revised: In addition to evaluating classiﬁcation performance, we conduct an ablation study to determine whether our sentence embeddings and associated scores have the interpretations claimed.

Original: 3. E XPERIMENTAL S ETUP
Revised: 3. E XPERIMENTAL S ETUP

Original: For the LSTM-based sentence encoders, we always use Bi-LSTM state vectors with 512 dimensions.
Revised: For the LSTM-based sentence encoders, we always use Bi-LSTM state vectors with 512 dimensions.

Original: With larger models (2048 dimensions)
Revised: With larger models (2048 dimensions)

Original: the results were only slightly better and the pattern of results did not change, while the computational cost was substantially higher.
Revised: the results were only slightly better and the pattern of results did not change, while the computational cost was substantially higher.

Original: We train these models forepochs, use validation accuracy for early stopping, and use mini-batch SGD with batch size of 64 .
Revised: We train these models forepochs, use validation accuracy for early stopping, and use mini-batch SGD with batch size of 64 .

Original: The learning rate is selected among { 0 . 06 , 0 . 08 , 0 . 1 , 0 . 3 , 0 . 5 } based on the validation scores.
Revised: The learning rate is selected among { 0 . 06 , 0 . 08 , 0 . 1 , 0 . 3 , 0 . 5 } based on the validation scores.

Original: For MNLI, we tuned the models on the mismatched validation set, and report results for both mismatched and matched validation sets, only reporting test set results for the best baseline and our best models.
Revised: For MNLI, we tuned the models on the mismatched validation set, and report results for both mismatched and matched validation sets, only reporting test set results for the best baseline and our best models.

Original: 3. B ASELINES
Revised: 3. B ASELINES

Original: We compared our model with several baselines: 1) p,h : Using only sentence embeddings followed by an MLP; 2) HM :
Revised: We compared our model with several baselines: 1) p,h : Using only sentence embeddings followed by an MLP; 2) HM :

Original: Heuristic-matching features input to an MLP (i.e. the baseline from Conneau et al.
Revised: Heuristic-matching features input to an MLP (i.e. the baseline from Conneau et al.

Original: 3) Random : compute ﬁve random scores followed by a linear softmax classiﬁer.
Revised: 3) Random : compute ﬁve random scores followed by a linear softmax classiﬁer.

Original: For baseline (3), we use untrained nonlinear projections of the embeddings p , h of the form:
Revised: For baseline (3), we use untrained nonlinear projections of the embeddings p , h of the form:

Original: [Equation]
Revised: [Equation]

Original: where the weight matrices W i ∈ R
Revised: where the weight matrices W i ∈ R

Original: d × 2 d , W g ∈ R 5 × d and their corresponding biases are randomly generated using Glorot Initialisation (Glorot & Bengio, 2010).
Revised: d × 2 d , W g ∈ R 5 × d and their corresponding biases are randomly generated using Glorot Initialisation (Glorot & Bengio, 2010).

Original: This baseline evaluates how much performance is being gained by employing our speciﬁc ﬁve scores, with their inductive bias.
Revised: This baseline evaluates how much performance is being gained by employing our speciﬁc ﬁve scores, with their inductive bias.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  R ESULTS
Revised: 3. R ESULTS

Original: Table 1 shows the results.
Revised: Table 1 shows the results.

Original: First, note that random scores already achieve reasonable performance, at about the level of the MLP-based baseline p,
Revised: First, note that random scores already achieve reasonable performance, at about the level of the MLP-based baseline p,

Original: h.
Revised: h.

Original: This suggests that the encoder is so powerful that it can generate embeddings suitable for virtually any kind of interaction model.
Revised: This suggests that the encoder is so powerful that it can generate embeddings suitable for virtually any kind of interaction model.

Original: Nonetheless, the performance of our interaction model is over 4 points better than random scores, indicating that our proposed scores impose an inductive bias which is well-suited for textual entailment.
Revised: Nonetheless, the performance of our interaction model is over 4 points better than random scores, indicating that our proposed scores impose an inductive bias which is well-suited for textual entailment.

Original: For single-sentence encoders, our interaction model performs slightly worse than the MLP-based model.
Revised: For single-sentence encoders, our interaction model performs slightly worse than the MLP-based model.

Original: The fact that our models’ performance is close indicates that our model is capturing nearly the same amount of information about NLI, and it has the advantage that all this learned information is encoded in the sentence embeddings, not in the MLP.
Revised: The fact that our models’ performance is close indicates that our model is capturing nearly the same amount of information about NLI, and it has the advantage that all this learned information is encoded in the sentence embeddings, not in the MLP.

Original: When we move to inter-sentence encoders (+attn), the difference reduces in every case, and is virtually eliminated for SNLI.
Revised: When we move to inter-sentence encoders (+attn), the difference reduces in every case, and is virtually eliminated for SNLI.

Original: Since an inter-sentence encoder is able to model some interaction in the encoder itself, this suggests that the performance difference is in part due to the reduced power of our parameter-free interaction model compared to an MLP.
Revised: Since an inter-sentence encoder is able to model some interaction in the encoder itself, this suggests that the performance difference is in part due to the reduced power of our parameter-free interaction model compared to an MLP.

Original: For example, our interaction model with a single-sentence encoder does not seem powerful enough to model known artefacts in the datasets (Gururangan et al., 2018), which we will discuss further in Section 4.
Revised: For example, our interaction model with a single-sentence encoder does not seem powerful enough to model known artefacts in the datasets (Gururangan et al., 2018), which we will discuss further in Section 4.

Original:  A BLATION S TUDY
Revised: 3. A BLATION S TUDY

Original: To better understand the effect of each of the scores, we performed an ablation study by evaluating models with subsets of our proposed scores, as shown in Table 2.
Revised: To better understand the effect of each of the scores, we performed an ablation study by evaluating models with subsets of our proposed scores, as shown in Table 2.

Original: The two similarity scores alone (S) perform rather poorly, indicating that similarity information alone is not enough to model entailment.
Revised: The two similarity scores alone (S) perform rather poorly, indicating that similarity information alone is not enough to model entailment.

Original: In contrast, using only the three entailment scores (E,C,N) achieves very good performance, yielding within 0.66 points of the full model on MNLI and within 0.33 points of the full model on SNLI.
Revised: In contrast, using only the three entailment scores (E,C,N) achieves very good performance, yielding within 0.66 points of the full model on MNLI and within 0.33 points of the full model on SNLI.

Original: Removing the neutral score had a surprisingly large impact on SNLI, but otherwise the neutral score is less important than the contradiction score, which is less important than the entailment score.
Revised: Removing the neutral score had a surprisingly large impact on SNLI, but otherwise the neutral score is less important than the contradiction score, which is less important than the entailment score.

Original: To further understand the roles of the different scores, consider the trained weights of the ﬁnal classiﬁcation layer for the model with only the entailment, contradiction and neutral scores (E,C,N): 
Revised: To further understand the roles of the different scores, consider the trained weights of the ﬁnal classiﬁcation layer for the model with only the entailment, contradiction and neutral scores (E,C,N): [Equation]

Original: [Table]
Revised: [Table]

Original: where the rows denote the labels and the columns denote the scores.
Revised: where the rows denote the labels and the columns denote the scores.

Original: The very large weights in the ﬁrst and last columns indicate that indeed the entailment score predicts entailment and the contradiction score predicts contradiction.
Revised: The very large weights in the ﬁrst and last columns indicate that indeed the entailment score predicts entailment and the contradiction score predicts contradiction.

Original: The neutral score weights interact with the biases to compensate for when the entailment and contradiction scores are both high.
Revised: The neutral score weights interact with the biases to compensate for when the entailment and contradiction scores are both high.

Original:  E VALUATION OF T RANSFER P ERFORMANCE
Revised:  E VALUATION OF T RANSFER P ERFORMANCE

Original: One motivation for using sentence embeddings which are directly interpretable in terms of entailment and contradiction is that the resulting interaction model imposes a stronger inductive bias towards learning entailment and contradiction.
Revised: One motivation for using sentence embeddings which are directly interpretable in terms of entailment and contradiction is that the resulting interaction model imposes a stronger inductive bias towards learning entailment and contradiction.

Original: Such inductive biases are often unhelpful when evaluating within the domain of a given dataset, particularly because the large NLI datasets, used in the previous section, contain annotation artefacts which allow high performance to be achieved without actually modelling entailment and contradiction (Gururangan et al., 2018).
Revised: Such inductive biases are often unhelpful when evaluating within the domain of a given dataset, particularly because the large NLI datasets, used in the previous section, contain annotation artefacts which allow high performance to be achieved without actually modelling entailment and contradiction (Gururangan et al., 2018).

Original: To evaluate the usefulness of this inductive bias for learning the more general regularities behind textual entailment, we evaluate our proposed model on its ability to generalise to a large number of different NLI datasets which do not share these same annotation biases.
Revised: To evaluate the usefulness of this inductive bias for learning the more general regularities behind textual entailment, we evaluate our proposed model on its ability to generalise to a large number of different NLI datasets which do not share these same annotation biases.

Original: We conduct these transfer experiments using sentence encoders which have been pretrained on very large amounts of unannotated text using a language modelling objective, namely the BERT model (Devlin et al., 2019).
Revised: We conduct these transfer experiments using sentence encoders which have been pretrained on very large amounts of unannotated text using a language modelling objective, namely the BERT model (Devlin et al., 2019).

Original: This pretraining has previously been demonstrated to help generalisation in a large number of tasks (Devlin et al., 2019), including state of the art results in NLI.
Revised: This pretraining has previously been demonstrated to help generalisation in a large number of tasks (Devlin et al., 2019), including state of the art results in NLI.

Original: In this section, we ﬁrst evaluate our approach to learning sentence embeddings from NLI data by ﬁne-tuning a pretrained BERT model, replicating the results from the previous section on the performance of the resulting NLI models 1 .
Revised: In this section, we ﬁrst evaluate our approach to learning sentence embeddings from NLI data by ﬁne-tuning a pretrained BERT model, replicating the results from the previous section on the performance of the resulting NLI models 1 .

Original: We then evaluate the resulting sentence embeddings on their model’s ability to generalise to a large number of other related tasks.
Revised: We then evaluate the resulting sentence embeddings on their model’s ability to generalise to a large number of other related tasks.

Revised: In addition, we evaluate the resulting sentence embeddings on transfer to sentence classiﬁcation tasks.

Original: 4. M ODELS
Revised: 4. M ODELS

Original: The original BERT model of NLI (Devlin et al., 2019) encodes the premise and hypothesis together and uses the embedding of the CLS token at the last layer as input to a classiﬁer.
Revised: The original BERT model of NLI (Devlin et al., 2019) encodes the premise and hypothesis together and uses the embedding of the CLS token at the last layer as input to a classiﬁer.

Original: But this model does not produce individual sentence embeddings for the premise and hypothesis.
Revised: But this model does not produce individual sentence embeddings for the premise and hypothesis.

Original: To get a sentence embedding model based on BERT, we use the Sentence-BERT (SBERT) model proposed by
Revised: To get a sentence embedding model based on BERT, we use the Sentence-BERT (SBERT) model proposed by

Original: Reimers & Gurevych (2019).
Revised: Reimers & Gurevych (2019).

Original: Reimers & Gurevych (2019) encode each sentence individually using a pretrained BERT model, then obtain the ﬁnal ﬁxed-sized embedding by average-pooling over thefull range of experiments in this context because of the high computational cost of training with BERT pretrained models. outputs 
Revised: Reimers & Gurevych (2019) encode each sentence individually using a pretrained BERT model, then obtain the ﬁnal ﬁxed-sized embedding by average-pooling over the outputs at the last layer. Average pooling was found to be better than max pooling or using the CLS token embedding.

Original: For the baseline model (HM+SBERT), after obtaining an embedding for the premise and the hypothesis, we proceed similarly to Section 2.2.
Revised: For the baseline model (HM+SBERT), after obtaining an embedding for the premise and the hypothesis, we proceed similarly to Section 2.2.

Original: First the baseline computes heuristic matching features from the sentence embeddings, but instead of using an MLP classiﬁer, it uses a log-linear classiﬁer in accordance with Reimers & Gurevych (2019) 2 .
Revised: First the baseline computes heuristic matching features from the sentence embeddings, but instead of using an MLP classiﬁer, it uses a log-linear classiﬁer in accordance with Reimers & Gurevych (2019) 2 .

Original: Our model (Ours+SBERT) computes the 5 scores deﬁned above and uses them in a log-linear classiﬁer.
Revised: Our model (Ours+SBERT) computes the 5 scores deﬁned above and uses them in a log-linear classiﬁer.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: We use BERT-Large as the foundation and ﬁnetune it with AdamW (Loshchilov & Hutter, 2019).
Revised: We use BERT-Large as the foundation and ﬁnetune it with AdamW (Loshchilov & Hutter, 2019).

Original: We generally use a universal initial learning rate of 2e-5 for all heuristic matching models.
Revised: We generally use a universal initial learning rate of 2e-5 for all heuristic matching models.

Original: When training our entailment-based model, a signiﬁcantly larger learning rate of 0.006 was necessary for the parameters of the classiﬁer.
Revised: When training our entailment-based model, a signiﬁcantly larger learning rate of 0.006 was necessary for the parameters of the classiﬁer.

Original: On SNLI, we found a learning rate of 2e-5 for training the BERT parameters and not initialising the output layer as described in Section 2.3 to work best.
Revised: On SNLI, we found a learning rate of 2e-5 for training the BERT parameters and not initialising the output layer as described in Section 2.3 to work best.

Original: On MNLI, a learning rate of 2e-6 with initialisation worked best.
Revised: On MNLI, a learning rate of 2e-6 with initialisation worked best.

Original: In all cases, there is a warm-up phase lasting for 30% of the training dataset.
Revised: In all cases, there is a warm-up phase lasting for 30% of the training dataset.

Original: For SNLI, we used a batch size of 8 whereas on MNLI we had to reduce the batch size to 4 because of limited memory.
Revised: For SNLI, we used a batch size of 8 whereas on MNLI we had to reduce the batch size to 4 because of limited memory.

Original: 4. NLI R ESULTS
Revised: 4. NLI R ESULTS

Original: When training and testing on the same NLI corpora, ﬁnetuning a pretrained BERT model gives us the same pattern of results as in Section 3.3, as shown in Table 3.
Revised: When training and testing on the same NLI corpora, ﬁnetuning a pretrained BERT model gives us the same pattern of results as in Section 3.3, as shown in Table 3.

Original: We see a small decrease in accuracies for both SNLI and MNLI for our much simpler interaction model, exacerbated by the fact that there is no interaction in the encoding stage.
Revised: We see a small decrease in accuracies for both SNLI and MNLI for our much simpler interaction model, exacerbated by the fact that there is no interaction in the encoding stage.

Original: We hypothesise that this decrease is largely due to the inability of our simpler model to capture the annotation biases in these datasets.
Revised: We hypothesise that this decrease is largely due to the inability of our simpler model to capture the annotation biases in these datasets.

Original: In the next section, we investigate this issue by testing on datasets which do not share the same annotation biases.
Revised: In the next section, we investigate this issue by testing on datasets which do not share the same annotation biases.

Original: [Table]
Revised: [Table]

Original: 4. T RANSFER P ERFORMANCE
Revised: T RANSFER P ERFORMANCE

Revised: Our model introduces a strong inductive bias for textual entailment that forces almost all the information about entailment to be contained in the sentence embeddings rather than the classiﬁer.

Revised: This is potentially very useful for transfer learning in two ways: transfer of the whole model to

Original: To evaluate how well the baseline and proposed models generalise to solving textual entailment in domains which do not share the same annotation biases as the large NLI training sets used above, we take trained NLI models and test them on a number of different NLI datasets.
Revised: To evaluate how well the baseline and proposed models generalise to solving textual entailment in domains which do not share the same annotation biases as the large NLI training sets used above, we take trained NLI models and test them on a number of different NLI datasets.

Original: Datasets: We consider a total of 11 different NLI datasets.
Revised: Datasets: We consider a total of 11 different NLI datasets.

Original: We use the 10 datasets studied by
Revised: We use the 10 datasets studied by

Original: Poliak et al.
Revised: Poliak et al.

Original: (2018b).
Revised: (2018b).

Original: These datasets include MNLI, SNLI, SciTail (Khot et al., 2018), AddOneRTE (Pavlick & Callison-Burch, 2016), Johns Hopkins Ordinal Commonsense Inference (JOCI)
Revised: These datasets include MNLI, SNLI, SciTail (Khot et al., 2018), AddOneRTE (Pavlick & Callison-Burch, 2016), Johns Hopkins Ordinal Commonsense Inference (JOCI)

Original: (Zhang et al., 2017), Multiple Premise Entailment (MPE) (Lai et al., 2017), Sentences Involving Compositional Knowledge (SICK) (Marelli et al., 2014), and three datasets from White et al.
Revised: (Zhang et al., 2017), Multiple Premise Entailment (MPE) (Lai et al., 2017), Sentences Involving Compositional Knowledge (SICK) (Marelli et al., 2014), and three datasets from White et al.

Original: (2017) which are automatically generated from existing datasets for other NLP tasks including: Semantic ProtoRoles (SPR) (Reisinger et al., 2015), Deﬁnite Pronoun Resolution (DPR) (Rahman & Ng, 2012), and FrameNet Plus (FN+) (Pavlick et al., 2015).
Revised: (2017) which are automatically generated from existing datasets for other NLP tasks including: Semantic ProtoRoles (SPR) (Reisinger et al., 2015), Deﬁnite Pronoun Resolution (DPR) (Rahman & Ng, 2012), and FrameNet Plus (FN+) (Pavlick et al., 2015).

Original: We additionally consider the Recognising Textual
Revised: We additionally consider the Recognising Textual

Original: Entailment dev dataset (RTE) from GLUE benchmark Wang et al.
Revised: Entailment dev dataset (RTE) from GLUE benchmark Wang et al.

Original: (2019), and the Quora Question
Revised: (2019), and the Quora Question

Original: Pairs (QQP) dataset 3 , where the task is to determine whether two given questions are semantically matching (duplicate) or not.
Revised: Pairs (QQP) dataset 3 , where the task is to determine whether two given questions are semantically matching (duplicate) or not.

Original: As in Gong et al.
Revised: As in Gong et al.

Original: (2017), we interpret duplicate question pairs as an entailment relation and neutral otherwise.
Revised: (2017), we interpret duplicate question pairs as an entailment relation and neutral otherwise.

Original: We use the same split ratio mentioned by Wang et al.
Revised: We use the same split ratio mentioned by Wang et al.

Original: Since the datasets considered have different label spaces

Original: [Table] when evaluating on each target dataset [Table] we map the model’s labels to the corresponding target dataset’s space.
Revised: Since the datasets considered have different label spaces, when evaluating on each target dataset, we map the model’s labels to the corresponding target dataset’s space.

Original: We train all models on MNLI and evaluate their performance on other target datasets.
Revised: We train all models on MNLI and evaluate their performance on other target datasets.

Original: MNLI contains three labels, contradiction, neutral [Table] and entailment.
Revised: MNLI contains three labels, contradiction, neutral, and entailment.

Original: Some of the datasets we consider contain only two labels.
Revised: Some of the datasets we consider contain only two labels.

Original: In the case of labels entailed and not-entailed , as in DPR, we map contradiction and neutral to the not-entailed class.
Revised: In the case of labels entailed and not-entailed , as in DPR, we map contradiction and neutral to the not-entailed class.

Original: In the case of labels entailment and neutral , as in SciTail, we map contradiction to neutral.
Revised: In the case of labels entailment and neutral , as in SciTail, we map contradiction to neutral.

Original: [Table]
Revised: [Table]

Original: Results Table 4 shows the transfer performance of the baseline and proposed SBERT models to other NLI datasets.
Revised: Results Table 4 shows the transfer performance of the baseline and proposed SBERT models to other NLI datasets.

Original: As shown in prior work (Belinkov et al., 2019a), the SNLI dataset has very similar annotation biases to the MNLI data which the models were trained on, so we do not expect any improvement in the relative performance of the two models for SNLI.
Revised: As shown in prior work (Belinkov et al., 2019a), the SNLI dataset has very similar annotation biases to the MNLI data which the models were trained on, so we do not expect any improvement in the relative performance of the two models for SNLI.

Original: Also, for the ADD-ONERTE dataset, both models essentially fail to generalise to this data, so the comparison is not very meaningful.
Revised: Also, for the ADD-ONERTE dataset, both models essentially fail to generalise to this data, so the comparison is not very meaningful.

Original: For the remaining datasets, our proposed model performs better than the baseline ondatasets, the same on one, and worse on 3.
Revised: For the remaining datasets, our proposed model performs better than the baseline ondatasets, the same on one, and worse on 3.

Original: On average, this represents a substantial improvement in generalisation by using the proposed inductive bias, especially given that our model started out from a lower accuracy on the in-domain data. 
Revised: On average, this represents a substantial improvement in generalisation by using the proposed inductive bias, especially given that our model started out from a lower accuracy on the in-domain data. 4.4. R ESULTS ON D OWNSTREAM TASKS

Revised: Because our model does not use a powerful MLP classiﬁer on top of the sentence embeddings, it has to encode almost all the semantics about entailment in the sentence embeddings themselves.

Revised: This should make the sentence embeddings more useful when used as features in downstream tasks.

Original: C ONCLUSION
Revised: To test this hypothesis, we evaluate the same SBERT-based sentence encoders from Section 4.4. on downstream tasks from the SentEval framework (Conneau & Kiela, 2018), which has previously been used to evaluate sentence embeddings trained on the textual entailment task (Conneau et al., 2017; Reimers & Gurevych, 2019). We use the default evaluation conﬁguration, where for the single-sentence classiﬁcation tasks a logistic regression classiﬁer is trained on top of the given sentence embedding. We exclude the supervised sentence-pair tasks (STS-B, SICK-E, SICK-R, and MRPC) from our analysis because the SentEval implementation uses heuristic matching features for these experiments and thus is incompatible with our model.

Revised: For related tasks, see the NLI experiments in section 4.4.1.

Revised: We report the downstream performance on supervised tasks in Table 5, and on unsupervised tasks in

Revised: Table 6.

Revised: In 10 out of 12 datasets, the sentence embeddings from our model achieve a substantial improvement over the sentence embeddings from the baseline.

Revised: In the remaining two cases (TREC and

Revised: STS15), the performance is approximately the same.

Revised: These results support the hypothesis that our model produces sentence embeddings that carry more semantic information that are useful features in downstream tasks.

Revised: [Table]

Revised: [Table]

Revised: To get a better understanding of what type of features our sentence embeddings encode

Revised: [Table]

Revised: we evaluate them on the 10 linguistic probing tasks introduced by Conneau et al.

Revised: (2018a).

Revised: These tasks are grouped into three categories.

Revised: The ﬁrst two are 1) whether the embedding encodes surface information like word content (WC) or sentence length (SentLen)

Revised: [Table]

Revised: and 2) whether it encodes syntactic information like word order (BShift) or information about the parse tree structure (TopConst,

Revised: Depth).

Revised: The last category tests for semantic information.

Revised: However, Conneau et al.

Revised: (2018a) note that three of these tasks can arguably be solved mainly by considering surface information at the subword level (which are available to BERT), namely identifying the tense (Tense)

Revised: [Table]

Revised: and identifying the number of the subject (SubjNum) and object (ObjNum).

Revised: The last two tasks require a solid semantic understanding of the encoded sentences, namely whether a random noun was replaced (odd-man-out (OMO)) and whether the coordinate clause was swapped with the main clause (CoordInv)

Revised: [Table]

Revised: The results of our model in comparison with the baseline are shown in Table 7.

Revised: With the exception of BShift, the baseline encodes substantially more surface and syntactic information.

Revised: In contrast, our model is superior at the semantically challenging tasks OMO and CoordInv, further supporting the hypothesis that our sentence embeddings encode more useful semantic features than the baseline.

Revised: [Table]

Original: Both the ablation study and the full  results demonstrate that our proposed entailment and contradiction scores are effective for modelling textual entailment.
Revised: Both the ablation study and the full NLI results demonstrate that our proposed entailment and contradiction scores are effective for modelling textual entailment.

Original: This parameter-free model of the relationship between premise and hypothesis forces training to put all information about entailment and contradiction in the sentence embeddings, rather than in the parameters of an MLP classiﬁer, and gives those sentence embeddings a direct entailment-based interpretation.
Revised: This parameter-free model of the relationship between premise and hypothesis forces training to put all information about entailment and contradiction in the sentence embeddings, rather than in the parameters of an MLP classiﬁer, and gives those sentence embeddings a direct entailment-based interpretation.

Original: We also show that the inductive bias given by forcing the learned sentence embeddings to have this entailment-based interpretation results in better generalisations to other NLI datasets.
Revised: We also show that the inductive bias given by forcing the learned sentence embeddings to have this entailment-based interpretation results in better generalisations to other NLI datasets and more useful sentence embeddings for other semantic tasks.