Original: D ON ’ T S ETTLE FOR A VERAGE , G O FOR THE M AX : F UZZY S ETS AND M AX -P OOLED W ORD V ECTORS
Revised: D ON ’ T S ETTLE FOR A VERAGE , G O FOR THE M AX : F UZZY S ETS AND M AX -P OOLED W ORD V ECTORS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent literature suggests that averaged word vectors followed by simple postprocessing outperform many deep learning methods on semantic textual similarity tasks.
Revised: Recent literature suggests that averaged word vectors followed by simple postprocessing outperform many deep learning methods on semantic textual similarity tasks.

Original: Furthermore, when averaged word vectors are trained supervised on large corpora of paraphrases, they achieve state-of-the-art results on standard STS benchmarks.
Revised: Furthermore, when averaged word vectors are trained supervised on large corpora of paraphrases, they achieve state-of-the-art results on standard STS benchmarks.

Original: Inspired by these insights, we push the limits of word embeddings even further.
Revised: Inspired by these insights, we push the limits of word embeddings even further.

Original: We propose a novel fuzzy bag-of-words (FBoW) representation for text that contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors.
Revised: We propose a novel fuzzy bag-of-words (FBoW) representation for text that contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors.

Original: We show that max-pooled word vectors are only a special case of fuzzy BoW and should be compared via fuzzy Jaccard index rather than cosine similarity.
Revised: We show that max-pooled word vectors are only a special case of fuzzy BoW and should be compared via fuzzy Jaccard index rather than cosine similarity.

Original: Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair.
Revised: Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair.

Original: This method is both efﬁcient and easy to implement, yet outperforms current baselines on STS tasks by a large margin and is even competitive with supervised word vectors trained to directly optimise cosine similarity.
Revised: This method is both efﬁcient and easy to implement, yet outperforms current baselines on STS tasks by a large margin and is even competitive with supervised word vectors trained to directly optimise cosine similarity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural languages are able to encode sentences with similar meanings using very different vocabulary and grammatical constructs, which makes determining the semantic similarity between pieces of text a challenge.
Revised: Natural languages are able to encode sentences with similar meanings using very different vocabulary and grammatical constructs, which makes determining the semantic similarity between pieces of text a challenge.

Original: It is common to cast semantic similarity between sentences as the proximity of their vector representations.
Revised: It is common to cast semantic similarity between sentences as the proximity of their vector representations.

Original: More than half a century since it was ﬁrst proposed, the Bag-of-Words (BoW) representation (Harris, 1954; Salton et al., 1975; Manning et al., 2008) remains a popular baseline across machine learning (ML), natural language processing (NLP), and information retrieval (IR) communities.
Revised: More than half a century since it was ﬁrst proposed, the Bag-of-Words (BoW) representation (Harris, 1954; Salton et al., 1975; Manning et al., 2008) remains a popular baseline across machine learning (ML), natural language processing (NLP), and information retrieval (IR) communities.

Original: In recent years, however, BoW was largely eclipsed by representations learned through neural networks, ranging from shallow (Le & Mikolov, 2014; Hill et al., 2016) to recurrent (Kiros et al., 2015; Conneau et al., 2017; Subramanian et al., 2018a), recursive (Socher et al., 2013;
Revised: In recent years, however, BoW was largely eclipsed by representations learned through neural networks, ranging from shallow (Le & Mikolov, 2014; Hill et al., 2016) to recurrent (Kiros et al., 2015; Conneau et al., 2017; Subramanian et al., 2018a), recursive (Socher et al., 2013;

Original: Tai et al., 2015), convolutional (Kalchbrenner et al., 2014; Kim, 2014), self-attentive (Vaswani et al., 2017; Cer et al., 2018a) and hybrid architectures (Gan et al., 2017; Tang et al., 2017; Zhelezniak et al., 2018).
Revised: Tai et al., 2015), convolutional (Kalchbrenner et al., 2014; Kim, 2014), self-attentive (Vaswani et al., 2017; Cer et al., 2018a) and hybrid architectures (Gan et al., 2017; Tang et al., 2017; Zhelezniak et al., 2018).

Original: Interestingly, Arora et al.
Revised: Interestingly, Arora et al.

Original: (2017) showed that averaged word vectors (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2016; Joulin et al., 2017) weighted with the Smooth Inverse
Revised: (2017) showed that averaged word vectors (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2016; Joulin et al., 2017) weighted with the Smooth Inverse

Original: Frequency (SIF) scheme and followed by a Principal Component Analysis (PCA) post-processing procedure were a formidable baseline for Semantic Textual Similarity (STS) tasks, outperforming deep representations.
Revised: Frequency (SIF) scheme and followed by a Principal Component Analysis (PCA) post-processing procedure were a formidable baseline for Semantic Textual Similarity (STS) tasks, outperforming deep representations.

Original: Furthermore, Wieting et al. (2015; 2016) and Wieting & Gimpel (2018) showed that averaged word vectors trained supervised on large corpora of paraphrases achieve state-of-the-art results, outperforming even the supervised systems trained directly on STS.
Revised: Furthermore, Wieting et al. (2015; 2016) and Wieting & Gimpel (2018) showed that averaged word vectors trained supervised on large corpora of paraphrases achieve state-of-the-art results, outperforming even the supervised systems trained directly on STS.

Original: Inspired by these insights, we push the boundaries of word vectors even further.
Revised: Inspired by these insights, we push the boundaries of word vectors even further.

Original: We propose a novel fuzzy bag-of-words (FBoW) representation for text.
Revised: We propose a novel fuzzy bag-of-words (FBoW) representation for text.

Original: Unlike classical BoW, fuzzy BoW contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors.
Revised: Unlike classical BoW, fuzzy BoW contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors.

Original: Next, we show that max-pooled word vectors are a special case of fuzzy BoW. Max-pooling signiﬁcantly outperforms averaging on standard benchmarks when word vectors are trained unsupervised.
Revised: Next, we show that max-pooled word vectors are a special case of fuzzy BoW. Max-pooling signiﬁcantly outperforms averaging on standard benchmarks when word vectors are trained unsupervised.

Original: Since max-pooled vectors are just a special case of fuzzy BoW, we show that the fuzzy Jaccard index is a more suitable alternative to cosine similarity for comparing these representations.
Revised: Since max-pooled vectors are just a special case of fuzzy BoW, we show that the fuzzy Jaccard index is a more suitable alternative to cosine similarity for comparing these representations.

Original: By contrast, the fuzzy Jaccard index completely fails for averaged word vectors as there is no connection between the two.
Revised: By contrast, the fuzzy Jaccard index completely fails for averaged word vectors as there is no connection between the two.

Original: The max-pooling operation is commonplace throughout NLP and has been successfully used to extract features in supervised systems (Collobert et al., 2011; Kim, 2014; Kenter & de Rijke, 2015;
Revised: The max-pooling operation is commonplace throughout NLP and has been successfully used to extract features in supervised systems (Collobert et al., 2011; Kim, 2014; Kenter & de Rijke, 2015;

Original: De Boom et al., 2016; Conneau et al., 2017; Dubois, 2017;
Revised: De Boom et al., 2016; Conneau et al., 2017; Dubois, 2017;

Original: Shen et al., 2018); however, to the best of our knowledge, the present work is the ﬁrst to study max-pooling of pre-trained word embeddings in isolation and to suggest theoretical underpinnings behind this operation.
Revised: Shen et al., 2018); however, to the best of our knowledge, the present work is the ﬁrst to study max-pooling of pre-trained word embeddings in isolation and to suggest theoretical underpinnings behind this operation.

Original: Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair.
Revised: Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair.

Original: DynaMax outperforms averaged word vector with cosine similarity on every benchmark STS task when word vectors are trained unsupervised.
Revised: DynaMax outperforms averaged word vector with cosine similarity on every benchmark STS task when word vectors are trained unsupervised.

Original: It even performs comparably to Wieting & Gimpel (2018)’s vectors under cosine similarity, which is a striking result as the latter are in fact trained supervised to directly optimise cosine similarity between paraphrases, while our approach is completely unrelated to that objective.
Revised: It even performs comparably to Wieting & Gimpel (2018)’s vectors under cosine similarity, which is a striking result as the latter are in fact trained supervised to directly optimise cosine similarity between paraphrases, while our approach is completely unrelated to that objective.

Original: We believe this makes DynaMax a strong baseline that future algorithms should aim to beat in order to justify more complicated approaches to semantic similarity.
Revised: We believe this makes DynaMax a strong baseline that future algorithms should aim to beat in order to justify more complicated approaches to semantic similarity.

Original: As an additional contribution, we conduct signiﬁcance analysis of our results.
Revised: As an additional contribution, we conduct signiﬁcance analysis of our results.

Original: We found that recent literature on STS tends to apply unspeciﬁed or inappropriate parametric tests, or leave out signiﬁcance analysis altogether in the majority of cases.
Revised: We found that recent literature on STS tends to apply unspeciﬁed or inappropriate parametric tests, or leave out signiﬁcance analysis altogether in the majority of cases.

Original: By contrast, we rely on nonparametric approaches with much milder assumptions on the test statistic; speciﬁcally, we construct bias-corrected and accelerated (BCa) bootstrap conﬁdence intervals (Efron, 1987) for the delta in performance between two systems.
Revised: By contrast, we rely on nonparametric approaches with much milder assumptions on the test statistic; speciﬁcally, we construct bias-corrected and accelerated (BCa) bootstrap conﬁdence intervals (Efron, 1987) for the delta in performance between two systems.

Original: We are not aware of any prior works that apply such methodology to STS benchmarks and hope the community ﬁnds our analysis to be a good starting point for conducting thorough signiﬁcance testing on these types of experiments.
Revised: We are not aware of any prior works that apply such methodology to STS benchmarks and hope the community ﬁnds our analysis to be a good starting point for conducting thorough signiﬁcance testing on these types of experiments.

Original:  S ENTENCES AS F UZZY S ETS
Revised:  S ENTENCES AS F UZZY S ETS

Original: The bag-of-words (BoW) model of representing text remains a popular baseline across ML, NLP, and IR communities.
Revised: The bag-of-words (BoW) model of representing text remains a popular baseline across ML, NLP, and IR communities.

Original: BoW, in fact, is an extension of a simpler set-of-words (SoW) model.
Revised: BoW, in fact, is an extension of a simpler set-of-words (SoW) model.

Original: SoW treats sentences as sets, whereas BoW treats them as multisets (bags) and so additionally captures how many times a word occurs in a sentence.
Revised: SoW treats sentences as sets, whereas BoW treats them as multisets (bags) and so additionally captures how many times a word occurs in a sentence.

Original: Just like with any set, we can immediately compare
Revised: Just like with any set, we can immediately compare

Original: SoW or BoW using set similarity measures (SSMs), such as
Revised: SoW or BoW using set similarity measures (SSMs), such as

Original: [Equation]
Revised: [Equation]

Original: These coefﬁcients usually follow the pattern # { shared elements } # { total elements } .
Revised: These coefﬁcients usually follow the pattern # { shared elements } # { total elements } .

Original: From this deﬁnition, it is clear that sets with no shared elements have a similarity of 0 , which is undesirable in NLP as sentences with completely different words can still share the same meaning.
Revised: From this deﬁnition, it is clear that sets with no shared elements have a similarity of 0 , which is undesirable in NLP as sentences with completely different words can still share the same meaning.

Original: But can we do better?
Revised: But can we do better?

Original: For concreteness, let’s say we want to compare two sentences corresponding to the sets
Revised: For concreteness, let’s say we want to compare two sentences corresponding to the sets

Original: A = { ‘he’ , ‘has’ , ‘a’ , ‘cat’ } and B = { ‘she’ , ‘had’ , ‘one’ , ‘dog’ } .
Revised: A = { ‘he’ , ‘has’ , ‘a’ , ‘cat’ } and B = { ‘she’ , ‘had’ , ‘one’ , ‘dog’ } .

Original: The situation here is that A ∩ B = ∅
Revised: The situation here is that A ∩ B = ∅

Original: and so their similarity according to any SSM is 0 .
Revised: and so their similarity according to any SSM is 0 .

Original: Yet, both A and B describe pet ownership and should be at least somewhat similar.
Revised: Yet, both A and B describe pet ownership and should be at least somewhat similar.

Original: If a set contains the word ‘cat’, it should also contain a bit of ‘pet’, a bit of ‘animal’, also a little bit of ‘tiger’ but perhaps not too much of an ‘airplane’.
Revised: If a set contains the word ‘cat’, it should also contain a bit of ‘pet’, a bit of ‘animal’, also a little bit of ‘tiger’ but perhaps not too much of an ‘airplane’.

Original: If both A and B contained ‘pet’, ‘animal’, etc. to some degree , they would have a non-zero similarity.
Revised: If both A and B contained ‘pet’, ‘animal’, etc. to some degree , they would have a non-zero similarity.

Original: This intuition is the main idea behind fuzzy sets: a fuzzy set includes all words in the vocabulary simultaneously, just with different degrees of membership.
Revised: This intuition is the main idea behind fuzzy sets: a fuzzy set includes all words in the vocabulary simultaneously, just with different degrees of membership.

Original: This generalises classical sets where a word either belongs to a set or it doesn’t.
Revised: This generalises classical sets where a word either belongs to a set or it doesn’t.

Original: We can easily convert a singleton set such as { ‘cat’ } into a fuzzy set using a similarity function sim ( w i , w j ) between words.
Revised: We can easily convert a singleton set such as { ‘cat’ } into a fuzzy set using a similarity function sim ( w i , w j ) between words.

Original: We simply compute the similarities between ‘cat’ and all the words w j in the vocabulary and treat those values as membership degrees.
Revised: We simply compute the similarities between ‘cat’ and all the words w j in the vocabulary and treat those values as membership degrees.

Original: As an example, the set { ‘cat’ } really becomes { ‘cat’ : 1 , ‘pet’ : 0 .
Revised: As an example, the set { ‘cat’ } really becomes { ‘cat’ : 1 , ‘pet’ : 0 .

Original: 9 , ‘animal’ : 0 . 85 , . . .
Revised: 9 , ‘animal’ : 0 . 85 , . . .

Original: , ‘airplane’ : 0 .
Revised: , ‘airplane’ : 0 .

Original: Fuzzifying singleton sets is straightforward, but how do we go about fuzzifying the entire sentence { ‘he’, ‘has’, ‘a’, ‘cat’ } ?
Revised: Fuzzifying singleton sets is straightforward, but how do we go about fuzzifying the entire sentence { ‘he’, ‘has’, ‘a’, ‘cat’ } ?

Original: Just as we use the classical union operation ∪ to build bigger sets from smaller ones, we use the fuzzy union to do the same but for fuzzy sets.
Revised: Just as we use the classical union operation ∪ to build bigger sets from smaller ones, we use the fuzzy union to do the same but for fuzzy sets.

Original: The membership degree of a word in the fuzzy union is determined as the maximum membership degree of that word among each of the fuzzy sets we want to unite.
Revised: The membership degree of a word in the fuzzy union is determined as the maximum membership degree of that word among each of the fuzzy sets we want to unite.

Original: This might sound somewhat arbitrary: after all, why max and not, say, sum or average?
Revised: This might sound somewhat arbitrary: after all, why max and not, say, sum or average?

Original: We explain the rationale in Section 2.1; and in fact, we use the max for the classical union all the time without ever noticing it.
Revised: We explain the rationale in Section 2.1; and in fact, we use the max for the classical union all the time without ever noticing it.

Original: Indeed, { ‘cat’ } ∪ { ‘cat’ } = { ‘cat’ } and not { ‘cat’ : 2 } .
Revised: Indeed, { ‘cat’ } ∪ { ‘cat’ } = { ‘cat’ } and not { ‘cat’ : 2 } .

Original: This is simply because we computed max(1 , 1) = 1 and not sum (1 , 1) = 2 .
Revised: This is simply because we computed max(1 , 1) = 1 and not sum (1 , 1) = 2 .

Original: Similarly { ‘cat’ } ∪ ∅ = { ‘cat’ } since max(1 , 0)
Revised: Similarly { ‘cat’ } ∪ ∅ = { ‘cat’ } since max(1 , 0)

Original: = 1 and not avg (1 , 0)
Revised: = 1 and not avg (1 , 0)

Original: = 1 / 2 .
Revised: = 1 / 2 .

Original: The key insight here is the following.
Revised: The key insight here is the following.

Original: An object that assigns the degrees of membership to words in a fuzzy set is called the membership function.
Revised: An object that assigns the degrees of membership to words in a fuzzy set is called the membership function.

Original: Each word deﬁnes a membership function, and even though ‘cat’ and ‘dog’ are different, they are semantically similar (in terms of cosine similarity between their word vectors, for example) and as such give rise to very similar membership functions.
Revised: Each word deﬁnes a membership function, and even though ‘cat’ and ‘dog’ are different, they are semantically similar (in terms of cosine similarity between their word vectors, for example) and as such give rise to very similar membership functions.

Original: This functional proximity will propagate into the SSMs, thus rendering them a much more realistic model for capturing semantic similarity between sentences.
Revised: This functional proximity will propagate into the SSMs, thus rendering them a much more realistic model for capturing semantic similarity between sentences.

Original: To actually compute the fuzzy SSMs, we need just a few basic tools from fuzzy set theory, all of which we brieﬂy cover in the next section.
Revised: To actually compute the fuzzy SSMs, we need just a few basic tools from fuzzy set theory, all of which we brieﬂy cover in the next section.

Original: 2. F UZZY S ETS : T HE B ARE M INIMUM
Revised: 2. F UZZY S ETS : T HE B ARE M INIMUM

Original: Fuzzy set theory (Zadeh, 1996) is a well-established formalism that extends classical set theory by incorporating the idea that elements can have degrees of membership in a set.
Revised: Fuzzy set theory (Zadeh, 1996) is a well-established formalism that extends classical set theory by incorporating the idea that elements can have degrees of membership in a set.

Original: Constrained by space, we deﬁne the bare minimum needed to compute the fuzzy set similarity measures and refer the reader to Klir et al.
Revised: Constrained by space, we deﬁne the bare minimum needed to compute the fuzzy set similarity measures and refer the reader to Klir et al.

Original: for a much richer introduction.
Revised: for a much richer introduction.

Original: Deﬁnition: A set of all possible terms V = { w 1 , w 2 , . . . , w N } that occur in a certain domain is called a universe.
Revised: Deﬁnition: A set of all possible terms V = { w 1 , w 2 , . . . , w N } that occur in a certain domain is called a universe.

Original: Deﬁnition:
Revised: Deﬁnition:

Original: A function µ : V → L ⊆ R is called a membership function.
Revised: A function µ : V → L ⊆ R is called a membership function.

Original: Deﬁnition: A pair A =
Revised: Deﬁnition: A pair A =

Original: ( V , µ ) is called a fuzzy set.
Revised: ( V , µ ) is called a fuzzy set.

Original: Notice how the above deﬁnition covers all the set-like objects we discussed so far.
Revised: Notice how the above deﬁnition covers all the set-like objects we discussed so far.

Original: If L = { 0 , 1 } , then
Revised: If L = { 0 , 1 } , then

Original: A is simply a classical set and µ is its indicator (characteristic) function.
Revised: A is simply a classical set and µ is its indicator (characteristic) function.

Original: If L = N ≥ 0 (non-negative integers), then A is a multiset (a bag) and µ is called a count (multiplicity) function.
Revised: If L = N ≥ 0 (non-negative integers), then A is a multiset (a bag) and µ is called a count (multiplicity) function.

Original: In literature, A is called a fuzzy set when L = [0 , 1] .
Revised: In literature, A is called a fuzzy set when L = [0 , 1] .

Original: However, we make no restrictions on the range and call A a fuzzy set even when L = R , i.e. all real numbers.
Revised: However, we make no restrictions on the range and call A a fuzzy set even when L = R , i.e. all real numbers.

Original: Deﬁnition: Let A
Revised: Deﬁnition: Let A

Original: =
Revised: =

Original: ( V , µ ) and B =
Revised: ( V , µ ) and B =

Original: ( V , ν ) be two fuzzy sets.
Revised: ( V , ν ) be two fuzzy sets.

Original: The union of A and B is a fuzzy set
Revised: The union of A and B is a fuzzy set

Original: A ∪ B =
Revised: A ∪ B =

Original: ( V , max( µ, ν )) .
Revised: ( V , max( µ, ν )) .

Original: The intersection of A and B is a fuzzy set
Revised: The intersection of A and B is a fuzzy set

Original: A ∩ B = ( V , min( µ, ν )) .
Revised: A ∩ B = ( V , min( µ, ν )) .

Original: Interestingly, there are many other choices for the union and intersection operations in fuzzy set theory.
Revised: Interestingly, there are many other choices for the union and intersection operations in fuzzy set theory.

Original: However, only the max - min pair makes these operations idempotent, i.e. such that A ∪ A =
Revised: However, only the max - min pair makes these operations idempotent, i.e. such that A ∪ A =

Original: A and A ∩
Revised: A and A ∩

Original: A =
Revised: A =

Original: A , just as in the classical set theory.
Revised: A , just as in the classical set theory.

Original: By contrast, it is not hard to verify that neither sum nor average satisfy the necessary axioms to qualify as a fuzzy union or intersection.
Revised: By contrast, it is not hard to verify that neither sum nor average satisfy the necessary axioms to qualify as a fuzzy union or intersection.

Original: Deﬁnition: Let A
Revised: Deﬁnition: Let A

Original: =
Revised: =

Original: ( V , µ ) be a fuzzy set.
Revised: ( V , µ ) be a fuzzy set.

Original: The number | A | = (cid:80) w ∈ V µ ( w ) is called the cardinality of a fuzzy set.
Revised: The number | A | = (cid:80) w ∈ V µ ( w ) is called the cardinality of a fuzzy set.

Original: Fuzzy set theory provides a powerful framework for reasoning about sets with uncertainty, but the speciﬁcation of membership functions depends heavily on the domain.
Revised: Fuzzy set theory provides a powerful framework for reasoning about sets with uncertainty, but the speciﬁcation of membership functions depends heavily on the domain.

Original: In practice these can be designed by experts or learned from data; below we describe a way of generating membership functions for text from word embeddings.
Revised: In practice these can be designed by experts or learned from data; below we describe a way of generating membership functions for text from word embeddings.

Original: 2. F UZZY B AG - OF -W ORDS
Revised: 2. F UZZY B AG - OF -W ORDS

Original: From the algorithmic point of view any bag-of-words is just a row vector.
Revised: From the algorithmic point of view any bag-of-words is just a row vector.

Original: The i -th term in the vocabulary has a corresponding N -dimensional one-hot encoding e ( i ) .
Revised: The i -th term in the vocabulary has a corresponding N -dimensional one-hot encoding e ( i ) .

Original: The vectors e ( i ) are orthonormal and in totality form the standard basis of R N .
Revised: The vectors e ( i ) are orthonormal and in totality form the standard basis of R N .

Original: The BoW vector for a sentence S is simply b S
Revised: The BoW vector for a sentence S is simply b S

Original: = (cid:80)
Revised: = (cid:80)

Original: Ni =1 c i e ( i ) , where c i is the count of the word w i in S .
Revised: Ni =1 c i e ( i ) , where c i is the count of the word w i in S .

Original: The ﬁrst step in creating the fuzzy BoW representation is to convert every term vector e ( i ) into a membership vector µ ( i ) .
Revised: The ﬁrst step in creating the fuzzy BoW representation is to convert every term vector e ( i ) into a membership vector µ ( i ) .

Original: It really is the same as converting a singleton set { w i } into a fuzzy set.
Revised: It really is the same as converting a singleton set { w i } into a fuzzy set.

Original: We call this operation ‘word fuzziﬁcation’, and in the matrix form it is simply written as[Equation] Input
Revised: We call this operation ‘word fuzziﬁcation’, and in the matrix form it is simply written as

Original: R × d X ← STACK ROWS ( x (1) , x (2) . . .
Revised: X ← STACK ROWS ( x (1) , x (2) . . .

Original: , x ( k ) )
Revised: , x ( k ) )

Original: Y ← STACK ROWS ( y (1) , y (2) . . .
Revised: Y ← STACK ROWS ( y (1) , y (2) . . .

Original: , y ( l ) )
Revised: , y ( l ) )

Original: U ← STACK ROWS ( X , Y )
Revised: U ← STACK ROWS ( X , Y )

Original: x ← MAX POOL ELEMENTWISE ( x (1) U T , x (2) U T . . .
Revised: x ← MAX POOL ELEMENTWISE ( x (1) U T , x (2) U T . . .

Original: , x ( k ) U T , z ) y ← MAX POOL ELEMENTWISE ( y (1) U T , y (2) U T . . .
Revised: , x ( k ) U T , z ) y ← MAX POOL ELEMENTWISE ( y (1) U T , y (2) U T . . .

Original: , y ( l ) U T , z ) r ← MIN POOL ELEMENTWISE ( x , y )
Revised: , y ( l ) U T , z ) r ← MIN POOL ELEMENTWISE ( x , y )

Original: q ← MAX POOL ELEMENTWISE ( x , y )
Revised: q ← MAX POOL ELEMENTWISE ( x , y )

Original: DMJ← (cid:80) i =1:( k + l ) r i / (cid:80) i =1:
Revised: DMJ

Original: [Equation]
Revised: [Equation]

Original: k + l ) q i

Original: Here W ∈ R N × d is the word embedding matrix and
Revised: Here W ∈ R N × d is the word embedding matrix and

Original: U ∈ R K × d is the ‘universe’ matrix.
Revised: U ∈ R K × d is the ‘universe’ matrix.

Original: Let us dissect the above expression.
Revised: Let us dissect the above expression.

Original: First, we convert a one-hot vector into a word embedding w ( i ) =
Revised: First, we convert a one-hot vector into a word embedding w ( i ) =

Original: e ( i ) W .
Revised: e ( i ) W .

Original: This is just an embedding lookup and is exactly the same as the embedding layer in neural networks.
Revised: This is just an embedding lookup and is exactly the same as the embedding layer in neural networks.

Original: Next, we compute a vector of similarities µ ( i ) =
Revised: Next, we compute a vector of similarities µ ( i ) =

Original: w ( i ) U T between w ( i ) and all the K vectors in the universe.
Revised: w ( i ) U T between w ( i ) and all the K vectors in the universe.

Original: The most sensible choice for the universe matrix is the word embedding matrix itself, i.e.
Revised: The most sensible choice for the universe matrix is the word embedding matrix itself, i.e.

Original: U = W .
Revised: U = W .

Original: In that case, the membership vector µ ( i ) has the same dimensionality as e ( i ) but contains similarities between the word w i and every word in the vocabulary (including itself).
Revised: In that case, the membership vector µ ( i ) has the same dimensionality as e ( i ) but contains similarities between the word w i and every word in the vocabulary (including itself).

Original: The second step is to combine all µ ( i ) back into a sentence membership vector µ s .
Revised: The second step is to combine all µ ( i ) back into a sentence membership vector µ s .

Original: At this point, it’s very tempting to just sum or average over all µ ( i ) , i.e. compute 1 N (cid:80) Ni
Revised: At this point, it’s very tempting to just sum or average over all µ ( i ) , i.e. compute 1 N (cid:80) Ni

Original: =1 c i µ ( i ) .
Revised: =1 c i µ ( i ) .

Original: But we remember: in fuzzy set theory the union of the membership vectors is realised by the element-wise max-pooling.
Revised: But we remember: in fuzzy set theory the union of the membership vectors is realised by the element-wise max-pooling.

Original: In other words, we don’t take the average but max-pool instead:
Revised: In other words, we don’t take the average but max-pool instead:

Original: [Equation]
Revised: [Equation]

Original: Here the max returns a vector where each dimension contains the maximum value along that dimension across all N input vectors.
Revised: Here the max returns a vector where each dimension contains the maximum value along that dimension across all N input vectors.

Original: In NLP this is also known as max-over-time pooling (Collobert et al., 2011).
Revised: In NLP this is also known as max-over-time pooling (Collobert et al., 2011).

Original: Note that any given sentence S usually contains only a small portion of the total vocabulary and so most word counts
Revised: Note that any given sentence S usually contains only a small portion of the total vocabulary and so most word counts

Original: c
Revised: c

Original: i will be 0 .
Revised: i will be 0 .

Original: If the count c i is 0 , then we have no need for µ ( i ) and can avoid a lot of useless computations, though we must remember to include the zero vector in the max-pooling operation.
Revised: If the count c i is 0 , then we have no need for µ ( i ) and can avoid a lot of useless computations, though we must remember to include the zero vector in the max-pooling operation.

Original: We call the sentence membership vector µ S the fuzzy bag-of-words (FBoW) and the procedure that converts classical BoW b S into fuzzy BoW µ S the ‘sentence fuzziﬁcation’.
Revised: We call the sentence membership vector µ S the fuzzy bag-of-words (FBoW) and the procedure that converts classical BoW b S into fuzzy BoW µ S the ‘sentence fuzziﬁcation’.

Original: 2.2. T HE F UZZY J ACCARD I NDEX
Revised: 2.2. T HE F UZZY J ACCARD I NDEX

Original: Suppose we have two fuzzy BoW µ A and µ B .
Revised: Suppose we have two fuzzy BoW µ A and µ B .

Original: How can we compare them?
Revised: How can we compare them?

Original: Since FBoW are just vectors, we can use the standard cosine similarity cos( µ A , µ B ) .
Revised: Since FBoW are just vectors, we can use the standard cosine similarity cos( µ A , µ B ) .

Original: On the other hand, FBoW are also fuzzy sets and as such can be compared via fuzzy SSMs.
Revised: On the other hand, FBoW are also fuzzy sets and as such can be compared via fuzzy SSMs.

Original: We simply copy the deﬁnitions of fuzzy union, intersection and cardinality from Section 2.1 and write down the fuzzy Jaccard index:
Revised: We simply copy the deﬁnitions of fuzzy union, intersection and cardinality from Section 2.1 and write down the fuzzy Jaccard index:

Original: [Equation]
Revised: [Equation]

Original: Exactly the same can be repeated for other SSMs.
Revised: Exactly the same can be repeated for other SSMs.

Original: In practice we found their performance to be almost equivalent but always better than standard cosine similarity (see Appendix B).
Revised: In practice we found their performance to be almost equivalent but always better than standard cosine similarity (see Appendix B).

Original: 2.2. S MALLER U NIVERSES AND M AX -P OOLED W ORD V ECTORS
Revised: 2.2. S MALLER U NIVERSES AND M AX -P OOLED W ORD V ECTORS

Original: So far we considered the universe and the word embedding matrix to be the same, i.e. U = W .
Revised: So far we considered the universe and the word embedding matrix to be the same, i.e. U = W .

Original: This means any FBoW µ S contains similarities to all the words in the vocabulary and has exactly the same dimensionality as the original BoW b S .
Revised: This means any FBoW µ S contains similarities to all the words in the vocabulary and has exactly the same dimensionality as the original BoW b S .

Original: Unlike BoW, however, FBoW is almost never sparse.
Revised: Unlike BoW, however, FBoW is almost never sparse.

Original: This motivates us to choose the matrix U with fewer rows that W .
Revised: This motivates us to choose the matrix U with fewer rows that W .

Original: For example, the top principal axes of
Revised: For example, the top principal axes of

Original: W could work.
Revised: W could work.

Original: Alternatively, we could cluster W into k clusters and keep the centroids.
Revised: Alternatively, we could cluster W into k clusters and keep the centroids.

Original: Of course, the rows of such U are no longer word vectors but instead some abstract entities.
Revised: Of course, the rows of such U are no longer word vectors but instead some abstract entities.

Original: A more radical but completely non-parametric solution is to choose U
Revised: A more radical but completely non-parametric solution is to choose U

Original: =
Revised: =

Original: I , where I ∈ R
Revised: I , where I ∈ R

Original: d × d is just the identity matrix.
Revised: d × d is just the identity matrix.

Original: Then the word fuzziﬁer reduces to a word embedding lookup:
Revised: Then the word fuzziﬁer reduces to a word embedding lookup:

Original: [Equation]
Revised: [Equation]

Original: The sentence fuzziﬁer then simply max-pools all the word embeddings found in the sentence:
Revised: The sentence fuzziﬁer then simply max-pools all the word embeddings found in the sentence:

Original: [Equation]
Revised: [Equation]

Original: From this we see that max-pooled word vectors are only a special case of fuzzy BoW. Remarkably, when word vectors are trained unsupervised, this simple representation combined with the fuzzy
Revised: From this we see that max-pooled word vectors are only a special case of fuzzy BoW. Remarkably, when word vectors are trained unsupervised, this simple representation combined with the fuzzy

Original: Jaccard index is already a stronger baseline for semantic textual similarity than the averaged word vector with cosine similarity, as we will see in Section 4.
Revised: Jaccard index is already a stronger baseline for semantic textual similarity than the averaged word vector with cosine similarity, as we will see in Section 4.

Original: More importantly, the fuzzy Jaccard index works for max-pooled word vectors but completely fails for averaged word vectors.
Revised: More importantly, the fuzzy Jaccard index works for max-pooled word vectors but completely fails for averaged word vectors.

Original: This empirically validates the connection between fuzzy BoW representations and the max-pooling operation described above.
Revised: This empirically validates the connection between fuzzy BoW representations and the max-pooling operation described above.

Original: 2.2. T HE D YNA M AX A LGORITHM
Revised: 2.2. T HE D YNA M AX A LGORITHM

Original: From the linear-algebraic point of view, fuzzy BoW is really the same as projecting word embeddings on a subspace of R d spanned by the rows of U , followed by max-pooling of the features extracted by this projection.
Revised: From the linear-algebraic point of view, fuzzy BoW is really the same as projecting word embeddings on a subspace of R d spanned by the rows of U , followed by max-pooling of the features extracted by this projection.

Original: A fair question then is the following.
Revised: A fair question then is the following.

Original: If we want to compare two sentences, what subspace should we project on?
Revised: If we want to compare two sentences, what subspace should we project on?

Original: It turns out that if we take word embeddings for the ﬁrst sentence and the second sentence and stack them into matrix U , this seems to be a sufﬁcient space to extract all the features needed for semantic similarity.
Revised: It turns out that if we take word embeddings for the ﬁrst sentence and the second sentence and stack them into matrix U , this seems to be a sufﬁcient space to extract all the features needed for semantic similarity.

Original: We noticed this empirically, and while some other choices of U do give better results, ﬁnding a principled way to construct them remains future work.
Revised: We noticed this empirically, and while some other choices of U do give better results, ﬁnding a principled way to construct them remains future work.

Original: The matrix U is not static any more but instead changes dynamically depending on the sentence pair.
Revised: The matrix U is not static any more but instead changes dynamically depending on the sentence pair.

Original: We call this approach Dynamic Max or DynaMax and provide pseudocode in Algorithm 1.
Revised: We call this approach Dynamic Max or DynaMax and provide pseudocode in Algorithm 1.

Original: 2.2. P RACTICAL C ONSIDERATIONS
Revised: 2.2. P RACTICAL C ONSIDERATIONS

Original: Just as SoW is a special case of BoW, we can build the fuzzy set-of-words (FSoW) where the word counts c
Revised: Just as SoW is a special case of BoW, we can build the fuzzy set-of-words (FSoW) where the word counts c

Original: i are binary.
Revised: i are binary.

Original: The performance of FSoW and FBoW is comparable, with FBoW being marginally better.
Revised: The performance of FSoW and FBoW is comparable, with FBoW being marginally better.

Original: For simplicity, we implement FSoW in Algorithm 1 and in all our experiments.
Revised: For simplicity, we implement FSoW in Algorithm 1 and in all our experiments.

Original: As evident from Equation (1), we use dot product as opposed to (scaled or clipped) cosine similarity for the membership functions.
Revised: As evident from Equation (1), we use dot product as opposed to (scaled or clipped) cosine similarity for the membership functions.

Original: This is a reasonable choice as most unsupervised and some supervised word vectors maximise dot products in their objectives.
Revised: This is a reasonable choice as most unsupervised and some supervised word vectors maximise dot products in their objectives.

Original: For further analysis, see Appendix A.
Revised: For further analysis, see Appendix A.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Any method that casts semantic similarity between sentences as the proximity of their vector representations is related to our work.
Revised: Any method that casts semantic similarity between sentences as the proximity of their vector representations is related to our work.

Original: Among those, the ones that strengthen bag-of-words by incorporating the sense of similarity between individual words are the most relevant.
Revised: Among those, the ones that strengthen bag-of-words by incorporating the sense of similarity between individual words are the most relevant.

Original: The standard Vector Space Model (VSM) basis
Revised: The standard Vector Space Model (VSM) basis

Original: e
Revised: e

Original: ( i ) is orthonormal and so the BoW model treats all words as equally different.
Revised: ( i ) is orthonormal and so the BoW model treats all words as equally different.

Original: Sidorov et al.
Revised: Sidorov et al.

Original: (2014) proposed the ‘soft cosine measure’ to alleviate this issue.
Revised: (2014) proposed the ‘soft cosine measure’ to alleviate this issue.

Original: They build a non-orthogonal basis f
Revised: They build a non-orthogonal basis f

Original: ( i ) where cos( f ( i ) , f ( j ) ) =
Revised: ( i ) where cos( f ( i ) , f ( j ) ) =

Original: sim ( w i , w j ) , i.e. the cosine similarity between the basis vectors is given by similarity between words.
Revised: sim ( w i , w j ) , i.e. the cosine similarity between the basis vectors is given by similarity between words.

Original: Next, they rewrite BoW in
Revised: Next, they rewrite BoW in

Original: [Figure]
Revised: [Figure]

Original: terms of f ( i ) and compute cosine similarity between transformed representations.
Revised: terms of f ( i ) and compute cosine similarity between transformed representations.

Original: However, when cos( f ( i ) , f ( j ) ) = cos( w i , w j ) , where w i , w j are word embeddings, their approach is equivalent to cosine similarity between averaged word embeddings, i.e. the standard baseline.
Revised: However, when cos( f ( i ) , f ( j ) ) = cos( w i , w j ) , where w i , w j are word embeddings, their approach is equivalent to cosine similarity between averaged word embeddings, i.e. the standard baseline.

Original: Kusner et al.
Revised: Kusner et al.

Original: (2015) consider L1-normalised bags-of-words (nBoW) and view them as a probability distributions over words.
Revised: (2015) consider L1-normalised bags-of-words (nBoW) and view them as a probability distributions over words.

Original: They propose the Word Mover’s Distance (WMD) as a special case of the
Revised: They propose the Word Mover’s Distance (WMD) as a special case of the

Original: Earth Mover’s Distance (EMD) between nBoW with the cost matrix given by pairwise Euclidean distances between word embeddings.
Revised: Earth Mover’s Distance (EMD) between nBoW with the cost matrix given by pairwise Euclidean distances between word embeddings.

Original: As such, WMD does not build any new representations but puts a lot of structure into the distance between BoW.
Revised: As such, WMD does not build any new representations but puts a lot of structure into the distance between BoW.

Original: Zhao & Mao (2017) proposed an alternative version of fuzzy BoW that is conceptually similar to ours but executed very differently.
Revised: Zhao & Mao (2017) proposed an alternative version of fuzzy BoW that is conceptually similar to ours but executed very differently.

Original: They use clipped cosine similarity between word embeddings to compute the membership values in the word fuzziﬁcation step.
Revised: They use clipped cosine similarity between word embeddings to compute the membership values in the word fuzziﬁcation step.

Original: We use dot product not only because it is theoretically more general but also because dot product leads to signiﬁcant improvements on the benchmarks.
Revised: We use dot product not only because it is theoretically more general but also because dot product leads to signiﬁcant improvements on the benchmarks.

Original: More importantly, however, their sentence fuzziﬁcation step uses sum to aggregate word membership vectors into a sentence membership vector.
Revised: More importantly, however, their sentence fuzziﬁcation step uses sum to aggregate word membership vectors into a sentence membership vector.

Original: We argue that max-pooling is a better choice because it corresponds to the fuzzy union.
Revised: We argue that max-pooling is a better choice because it corresponds to the fuzzy union.

Original: Had we used the sum, the representation would have really reduced to a (projected) summed word vector.
Revised: Had we used the sum, the representation would have really reduced to a (projected) summed word vector.

Original: Lastly, they use FBoW as features for a supervised model but stop short of considering any fuzzy similarity measures, such as fuzzy Jaccard index.
Revised: Lastly, they use FBoW as features for a supervised model but stop short of considering any fuzzy similarity measures, such as fuzzy Jaccard index.

Original: Jimenez et al.
Revised: Jimenez et al.

Original: (2010; 2012; 2013; 2014; 2015) proposed and developed soft cardinality as a generalisation to the classical set cardinality.
Revised: (2010; 2012; 2013; 2014; 2015) proposed and developed soft cardinality as a generalisation to the classical set cardinality.

Original: In their framework set membership is crisp, just as in classical set theory.
Revised: In their framework set membership is crisp, just as in classical set theory.

Original: However, once the words are in a set, their contribution to the overall cardinality depends on how similar they are to each other.
Revised: However, once the words are in a set, their contribution to the overall cardinality depends on how similar they are to each other.

Original: The intuition is that the set A = { ‘lion’ , ‘tiger’ , ‘leopard’ } should have cardinality much less than 3, because A contains very similar elements.
Revised: The intuition is that the set A = { ‘lion’ , ‘tiger’ , ‘leopard’ } should have cardinality much less than 3, because A contains very similar elements.

Original: Likewise, the set
Revised: Likewise, the set

Original: B = { ‘lion’ , ‘airplane’ , ‘carrot’ } deserves a cardinality closer to 3.
Revised: B = { ‘lion’ , ‘airplane’ , ‘carrot’ } deserves a cardinality closer to 3.

Original: We see that the soft cardinality framework is very different from our approach, as it ‘does not consider uncertainty in the membership of a particular element; only uncertainty as to the contribution of an element to the cardinality of the set’ (Jimenez et al., 2010).
Revised: We see that the soft cardinality framework is very different from our approach, as it ‘does not consider uncertainty in the membership of a particular element; only uncertainty as to the contribution of an element to the cardinality of the set’ (Jimenez et al., 2010).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate the proposed similarity measures we set up a series of experiments on the established
Revised: To evaluate the proposed similarity measures we set up a series of experiments on the established

Original: STS tasks, part of the SemEval shared task series 2012-2016 (Agirre et al., 2012; 2013; 2014; Agirre, 2015; Agirre et al., 2016; Cer et al., 2017).
Revised: STS tasks, part of the SemEval shared task series 2012-2016 (Agirre et al., 2012; 2013; 2014; Agirre, 2015; Agirre et al., 2016; Cer et al., 2017).

Original: The idea behind the STS benchmarks is to measure
Revised: The idea behind the STS benchmarks is to measure

Original: [Figure]
Revised: [Figure]

Original: how well the semantic similarity scores computed by a system (algorithm) correlate with human judgements.
Revised: how well the semantic similarity scores computed by a system (algorithm) correlate with human judgements.

Original: Each year’s STS task itself consists of several subtasks.
Revised: Each year’s STS task itself consists of several subtasks.

Original: By convention, we report the mean Pearson correlation between system and human scores, where the mean is taken across all the subtasks in a given year.
Revised: By convention, we report the mean Pearson correlation between system and human scores, where the mean is taken across all the subtasks in a given year.

Original: Our implementation wraps the SentEval toolkit (Conneau & Kiela, 2018) and is available on GitHub 1 .
Revised: Our implementation wraps the SentEval toolkit (Conneau & Kiela, 2018) and is available on GitHub 1 .

Original: We also rely on the following publicly available word embeddings: GloVe (Pennington et al., 2014) trained on Common Crawl (840B tokens); fastText (Bojanowski et al., 2016) trained on Common
Revised: We also rely on the following publicly available word embeddings: GloVe (Pennington et al., 2014) trained on Common Crawl (840B tokens); fastText (Bojanowski et al., 2016) trained on Common

Original: Crawl (600B tokens); word2vec (Mikolov et al., 2013b;c) trained on Google News, CoNLL (Zeman et al., 2017), and Book Corpus (Zhu et al., 2015); and several types of supervised paraphrastic vectors – PSL (Wieting et al., 2015), PP-XXL
Revised: Crawl (600B tokens); word2vec (Mikolov et al., 2013b;c) trained on Google News, CoNLL (Zeman et al., 2017), and Book Corpus (Zhu et al., 2015); and several types of supervised paraphrastic vectors – PSL (Wieting et al., 2015), PP-XXL

Original: (Wieting et al., 2016), and PNMT (Wieting & Gimpel, 2018).
Revised: (Wieting et al., 2016), and PNMT (Wieting & Gimpel, 2018).

Original: We estimated word frequencies on an English Wikipedia dump dated July 1 st 2017 and calculated word weights using the same approach and parameters as in Arora et al.
Revised: We estimated word frequencies on an English Wikipedia dump dated July 1 st 2017 and calculated word weights using the same approach and parameters as in Arora et al.

Original: Note that these weights can in fact be derived from word vectors and frequencies alone rather than being inferred from the validation set (Ethayarajh, 2018), making our techniques fully unsupervised.
Revised: Note that these weights can in fact be derived from word vectors and frequencies alone rather than being inferred from the validation set (Ethayarajh, 2018), making our techniques fully unsupervised.

Original: Finally, as the
Revised: Finally, as the

Original: STS’13 SMT dataset is no longer publicly available, the mean Pearson correlations reported in our experiments involving this task have been re-calculated accordingly.
Revised: STS’13 SMT dataset is no longer publicly available, the mean Pearson correlations reported in our experiments involving this task have been re-calculated accordingly.

Original: We ﬁrst ran a set of experiments validating the insights and derivations described in Section 2.
Revised: We ﬁrst ran a set of experiments validating the insights and derivations described in Section 2.

Original: These results are presented in Figure 1.
Revised: These results are presented in Figure 1.

Original: The main takeaways are the following:
Revised: The main takeaways are the following:

Original: • Max-pooled word vectors outperform averaged word vectors in most tasks.
Revised: • Max-pooled word vectors outperform averaged word vectors in most tasks.

Original: • Max-pooled vectors with cosine similarity perform worse than max-pooled vectors with fuzzy Jaccard similarity.
Revised: • Max-pooled vectors with cosine similarity perform worse than max-pooled vectors with fuzzy Jaccard similarity.

Original: This supports our derivation of max-pooled vectors as a special case of fuzzy BoW, which thus should be compared via fuzzy set similarity measures and not cosine similarity (which would be an arbitrary choice).
Revised: This supports our derivation of max-pooled vectors as a special case of fuzzy BoW, which thus should be compared via fuzzy set similarity measures and not cosine similarity (which would be an arbitrary choice).

Original: • Averaged vectors with fuzzy Jaccard similarity completely fail.
Revised: • Averaged vectors with fuzzy Jaccard similarity completely fail.

Original: This is because fuzzy set theory tells us that the average is not a valid fuzzy union operation, so a fuzzy set similarity is not appropriate for this representation.
Revised: This is because fuzzy set theory tells us that the average is not a valid fuzzy union operation, so a fuzzy set similarity is not appropriate for this representation.

Original: • DynaMax shows the best performance across all tasks, possibly thanks to its superior ability to extract and max-pool good features from word vectors.
Revised: • DynaMax shows the best performance across all tasks, possibly thanks to its superior ability to extract and max-pool good features from word vectors.

Original: Next we ran experiments against some of the related methods described in Section 3, namely WMD (Kusner et al., 2015) and soft cardinality (Jimenez et al., 2015) with clipped cosine similarity as an afﬁnity function and the softness parameter p = 1 .
Revised: Next we ran experiments against some of the related methods described in Section 3, namely WMD (Kusner et al., 2015) and soft cardinality (Jimenez et al., 2015) with clipped cosine similarity as an afﬁnity function and the softness parameter p = 1 .

Original: From Figure 2, we see that even classical Jaccard index is a reasonable baseline, but fuzzy Jaccard especially in the DynaMax formulation handily outperforms comparable methods.
Revised: From Figure 2, we see that even classical Jaccard index is a reasonable baseline, but fuzzy Jaccard especially in the DynaMax formulation handily outperforms comparable methods.

Original: For context and completeness, we also compare against other popular sentence representations from the literature in Table 1.
Revised: For context and completeness, we also compare against other popular sentence representations from the literature in Table 1.

Original: We include the following methods: BoW with ELMo embeddings
Revised: We include the following methods: BoW with ELMo embeddings

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: GloVe avg-cos
Revised: GloVe avg-cos

Original: [Table]
Revised: [Table]

Original: GloVe DynaMax 
Revised: GloVe DynaMax 58.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: fastText avg-cos
Revised: fastText avg-

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: fastText DynaMax 
Revised: DynaMax 60. [Table] [Table] [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: word2vec
Revised: word2vec

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: word2vec
Revised: word2vec

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: PSL avg

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: PSL
Revised: PSL

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: PP-XXL avg-cos

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: PP

Original: [Table]
Revised: [Table]

Revised: XXL

Original: [Table]
Revised: [Table]

Original: (Peters et al., 2018), Skip-Thought (Kiros et al., 2015), InferSent (Conneau et al., 2017), Universal
Revised: (Peters et al., 2018), Skip-Thought (Kiros et al., 2015), InferSent (Conneau et al., 2017), Universal

Original: Sentence Encoder with DAN and Transformer (Cer et al., 2018b), and STN multitask embeddings (Subramanian et al., 2018b).
Revised: Sentence Encoder with DAN and Transformer (Cer et al., 2018b), and STN multitask embeddings (Subramanian et al., 2018b).

Original: These experiments lead to an interesting observation:
Revised: These experiments lead to an interesting observation:

Original: • PNMT embeddings are the current state-of-the-art on STS tasks.
Revised: • PNMT embeddings are the current state-of-the-art on STS tasks.

Original: PP-XXL and PNMT were trained supervised to directly optimise cosine similarity between average word vectors on very large paraphrastic datasets.
Revised: PP-XXL and PNMT were trained supervised to directly optimise cosine similarity between average word vectors on very large paraphrastic datasets.

Original: By contrast, DynaMax is completely unrelated to the training objective of these vectors, yet has an equivalent performance.
Revised: By contrast, DynaMax is completely unrelated to the training objective of these vectors, yet has an equivalent performance.

Original: Finally, another well-known and high-performing simple baseline was proposed by Arora et al.
Revised: Finally, another well-known and high-performing simple baseline was proposed by Arora et al.

Original: However, as also noted by Mu & Viswanath (2018), this method is still ofﬂine because it computes the sentence embeddings for the entire dataset, then performs PCA and removes the top principal component.
Revised: However, as also noted by Mu & Viswanath (2018), this method is still ofﬂine because it computes the sentence embeddings for the entire dataset, then performs PCA and removes the top principal component.

Original: While their method makes more assumptions than ours, nonetheless we make a head-to-head comparison with them in Table 2 using the same word vectors as in Arora et al.
Revised: While their method makes more assumptions than ours, nonetheless we make a head-to-head comparison with them in Table 2 using the same word vectors as in Arora et al.

Original: (2017), showing that DynaMax is still quite competitive.
Revised: (2017), showing that DynaMax is still quite competitive.

Original: To strengthen our empirical ﬁndings, we provide ablation studies for DynaMax in Appendix C, showing that the different components of the algorithm each contribute to its strong performance.
Revised: To strengthen our empirical ﬁndings, we provide ablation studies for DynaMax in Appendix C, showing that the different components of the algorithm each contribute to its strong performance.

Original: We also conduct signiﬁcance testing in Appendix D by constructing bias-corrected and accelerated (BCa) bootstrap conﬁdence intervals (Efron, 1987) for the delta in performance between two algorithms.
Revised: We also conduct signiﬁcance testing in Appendix D by constructing bias-corrected and accelerated (BCa) bootstrap conﬁdence intervals (Efron, 1987) for the delta in performance between two algorithms.

Original: This constitutes, to the best of our knowledge, the ﬁrst attempt to study statistical signiﬁcance on the
Revised: This constitutes, to the best of our knowledge, the ﬁrst attempt to study statistical signiﬁcance on the

Original: STS benchmarks with this type of non-parametric analysis that respects the statistical peculiarities of these datasets.
Revised: STS benchmarks with this type of non-parametric analysis that respects the statistical peculiarities of these datasets.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we combine word embeddings with classic BoW representations using fuzzy set theory.
Revised: In this work we combine word embeddings with classic BoW representations using fuzzy set theory.

Original: We show that max-pooled word vectors are a special case of FBoW, which implies that they should be compared via the fuzzy Jaccard index rather than the more standard cosine similarity.
Revised: We show that max-pooled word vectors are a special case of FBoW, which implies that they should be compared via the fuzzy Jaccard index rather than the more standard cosine similarity.

Original: We also present a simple and novel algorithm, DynaMax, which corresponds to projecting word vectors onto a subspace dynamically generated by the given sentences before max-pooling over the features.
Revised: We also present a simple and novel algorithm, DynaMax, which corresponds to projecting word vectors onto a subspace dynamically generated by the given sentences before max-pooling over the features.

Original: DynaMax outperforms averaged word vectors compared with cosine similarity on every benchmark STS task when word vectors are trained unsupervised.
Revised: DynaMax outperforms averaged word vectors compared with cosine similarity on every benchmark STS task when word vectors are trained unsupervised.

Original: It even performs comparably to supervised vectors that directly optimise cosine similarity between paraphrases, despite being completely unrelated to that objective.
Revised: It even performs comparably to supervised vectors that directly optimise cosine similarity between paraphrases, despite being completely unrelated to that objective.

Original: Both max-pooled vectors and DynaMax constitute strong baselines for further studies in the area of sentence representations.
Revised: Both max-pooled vectors and DynaMax constitute strong baselines for further studies in the area of sentence representations.

Original: Yet, these methods are not limited to NLP and word embeddings, but can in fact be used in any setting where one needs to compute similarity between sets of elements that have rich vector representations.
Revised: Yet, these methods are not limited to NLP and word embeddings, but can in fact be used in any setting where one needs to compute similarity between sets of elements that have rich vector representations.

Original: We hope to have demonstrated the beneﬁts of experimenting more with similarity metrics based on the building blocks of meaning such as words, rather than complex representations of the ﬁnal objects such as sentences
Revised: We hope to have demonstrated the beneﬁts of experimenting more with similarity metrics based on the building blocks of meaning such as words, rather than complex representations of the ﬁnal objects such as sentences. A CKNOWLEDGMENTS

Revised: We would like to thank John Wieting for sharing with us his latest state-of-the-art ParaNMT embeddings, so that we could include the most up-to-date comparisons in the present work

Original: A N ORMALISED V ECTORS AND [0 , 1] - FUZZY S ETS
Revised: A N ORMALISED V ECTORS AND [0 , 1] - FUZZY S ETS

Original: In the word fuzziﬁcation step the membership values for a word w are obtained through a similarity function sim ( w , u ( j ) ) between the word embedding w and the rows of the universe matrix U ,
Revised: In the word fuzziﬁcation step the membership values for a word w are obtained through a similarity function sim ( w , u ( j ) ) between the word embedding w and the rows of the universe matrix U ,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. µ = [ sim ( w , u (1) ) , sim ( w , u (2) ) , . .
Revised: e. µ = [ sim ( w , u (1) ) , sim ( w , u (2) ) , . .

Original: , sim ( w , u ( K ) )] .
Revised: , sim ( w , u ( K ) )] .

Original: In Section 2.
Revised: In Section 2.

Original: [Table]
Revised: [Table]

Original: sim ( w , u ( j ) ) was the dot product w · u ( j ) and we could simply write µ = wU T .
Revised: sim ( w , u ( j ) ) was the dot product w · u ( j ) and we could simply write µ = wU T .

Original: There are several reasons why we chose a similarity function that takes values in R as opposed to [0 , 1]
Revised: There are several reasons why we chose a similarity function that takes values in R as opposed to [0 , 1]

Original: [Table]
Revised: [Table]

Original: First, we can always map the membership values from R to (0 , 1) and vice versa using, e [Table] g. the logistic function σ ( x ) = 1 1+ e − ax with an appropriate scaling factor a > 0 .
Revised: First, we can always map the membership values from R to (0 , 1) and vice versa using, e.g. the logistic function σ ( x ) = 1 1+ e − ax with an appropriate scaling factor a > 0 .

Original: Intuitively, large negative membership values would imply the element is really not in the set and large positive values mean it is really in the set.
Revised: Intuitively, large negative membership values would imply the element is really not in the set and large positive values mean it is really in the set.

Original: Of course, here both ‘large’ and ‘really’ depend on the scaling factor a .
Revised: Of course, here both ‘large’ and ‘really’ depend on the scaling factor a .

Original: In any case, we see that the choice of R vs. [0 , 1] is not very important mathematically.
Revised: In any case, we see that the choice of R vs. [0 , 1] is not very important mathematically.

Original: Interestingly, since we always max-pool with a zero vector, fuzzy BoW will not contain any negative membership values.
Revised: Interestingly, since we always max-pool with a zero vector, fuzzy BoW will not contain any negative membership values.

Original: This was not our intention, just a by-product of the model.
Revised: This was not our intention, just a by-product of the model.

Original: Secondly, note that the membership function for multisets takes values in N ≥ 0 , i.e. the nonnegative integers.
Revised: Secondly, note that the membership function for multisets takes values in N ≥ 0 , i.e. the nonnegative integers.

Original: These values are already outside [0 , 1] and we see that the standard [0 , 1] -fuzzy sets are incompatible with multisets.
Revised: These values are already outside [0 , 1] and we see that the standard [0 , 1] -fuzzy sets are incompatible with multisets.

Original: On the other hand, a membership function that takes values in R can directly model sets, multisets, fuzzy sets, and fuzzy multisets.
Revised: On the other hand, a membership function that takes values in R can directly model sets, multisets, fuzzy sets, and fuzzy multisets.

Original: For completeness, let us insist on the range [0 , 1] and choose sim ( w , u ( j ) ) to be the clipped cosine similarity max(0 , cos ( w , u ( j ) )) .
Revised: For completeness, let us insist on the range [0 , 1] and choose sim ( w , u ( j ) ) to be the clipped cosine similarity max(0 , cos ( w , u ( j ) )) .

Original: This is in fact equivalent to simply normalising the word vectors.
Revised: This is in fact equivalent to simply normalising the word vectors.

Original: Indeed, the dot product and cosine similarity become the same after normalisation, and max-pooling with the zero vector removes all the negative values, so the resulting representation is guaranteed to be a [0 , 1] -fuzzy set.
Revised: Indeed, the dot product and cosine similarity become the same after normalisation, and max-pooling with the zero vector removes all the negative values, so the resulting representation is guaranteed to be a [0 , 1] -fuzzy set.

Original: Our results for normalised word vectors are presented in Table 3.
Revised: Our results for normalised word vectors are presented in Table 3.

Original: After comparing Tables 1 and 3 we can draw two conclusions.
Revised: After comparing Tables 1 and 3 we can draw two conclusions.

Original: Namely, DynaMax still outperforms avg-cos by a large margin even when word vectors are normalised.
Revised: Namely, DynaMax still outperforms avg-cos by a large margin even when word vectors are normalised.

Original: However, normalisation hurts both approaches and should generally be avoided.
Revised: However, normalisation hurts both approaches and should generally be avoided.

Original: This is not surprising since the length of word vectors is correlated with word importance, so normalisation essentially makes all words equally important (Schakel & Wilson, 2015).
Revised: This is not surprising since the length of word vectors is correlated with word importance, so normalisation essentially makes all words equally important (Schakel & Wilson, 2015).

Original: B C OMPARISON OF F UZZY S ET S IMILARITY M EASURES
Revised: B C OMPARISON OF F UZZY S ET S IMILARITY M EASURES

Original: In Section 2 we mentioned several set similarity measures such as Jaccard (Jaccard, 1901), OtsukaOchiai (Otsuka, 1936; Ochiai, 1957) and Sørensen–Dice (Dice, 1945; Sørensen, 1948) coefﬁcients.
Revised: In Section 2 we mentioned several set similarity measures such as Jaccard (Jaccard, 1901), OtsukaOchiai (Otsuka, 1936; Ochiai, 1957) and Sørensen–Dice (Dice, 1945; Sørensen, 1948) coefﬁcients.

Original: Here in Table 4, we show that fuzzy versions of the above coefﬁcients have almost identical performance, thus conﬁrming that our results are in no way speciﬁc to the Jaccard index.
Revised: Here in Table 4, we show that fuzzy versions of the above coefﬁcients have almost identical performance, thus conﬁrming that our results are in no way speciﬁc to the Jaccard index.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Sim.
Revised: Pool & Sim.

Original: [Table]
Revised: [Table]

Original: C D YNA M AX A BLATION S TUDIES
Revised: C D YNA M AX A BLATION S TUDIES

Original: The DynaMax-Jaccard similarity (Algorithm 1) consists of three components: the dynamic universe, the max-pooling operation, and the fuzzy Jaccard index.
Revised: The DynaMax-Jaccard similarity (Algorithm 1) consists of three components: the dynamic universe, the max-pooling operation, and the fuzzy Jaccard index.

Original: As with any algorithm, it is very important to track the sources of improvements.
Revised: As with any algorithm, it is very important to track the sources of improvements.

Original: Consequently, we perform a series of ablation studies in order to isolate the contribution of each component.
Revised: Consequently, we perform a series of ablation studies in order to isolate the contribution of each component.

Original: For brevity, we focus on fastText because it produced the strongest results for both the DynaMax and the baseline (Figure 1).
Revised: For brevity, we focus on fastText because it produced the strongest results for both the DynaMax and the baseline (Figure 1).

Original: The results of the ablation study are presented in Table 5.
Revised: The results of the ablation study are presented in Table 5.

Original: First, we show that the dynamic universe is superior to other sensible choices, such as the identity and random 300 × 300 projection with components drawn from N (0 , 1) .
Revised: First, we show that the dynamic universe is superior to other sensible choices, such as the identity and random 300 × 300 projection with components drawn from N (0 , 1) .

Original: Next, we show that the fuzzy Jaccard index beats the standard cosine similarity on 4 out 5 benchmarks.
Revised: Next, we show that the fuzzy Jaccard index beats the standard cosine similarity on 4 out 5 benchmarks.

Original: Finally, we ﬁnd that max considerably outperforms other pooling operations such as averaging, sum and min.
Revised: Finally, we ﬁnd that max considerably outperforms other pooling operations such as averaging, sum and min.

Original: We conclude that all three components of
Revised: We conclude that all three components of

Original: DynaMax are very important.
Revised: DynaMax are very important.

Original: It is clear that max-pooling is the top contributing factor, followed by the dynamic universe and the fuzzy Jaccard index, whose contributions are roughly equal.
Revised: It is clear that max-pooling is the top contributing factor, followed by the dynamic universe and the fuzzy Jaccard index, whose contributions are roughly equal.

Original: D S IGNIFICANCE A NALYSIS
Revised: D S IGNIFICANCE A NALYSIS

Original: As discussed in Section 4, the core idea behind the STS benchmarks is to measure how well the semantic similarity scores computed by a system (algorithm) correlate with human judgements.
Revised: As discussed in Section 4, the core idea behind the STS benchmarks is to measure how well the semantic similarity scores computed by a system (algorithm) correlate with human judgements.

Original: In this section we provide detailed results and signiﬁcance analysis for all 24 STS subtasks.
Revised: In this section we provide detailed results and signiﬁcance analysis for all 24 STS subtasks.

Original: Our approach can be formally summarised as follows.
Revised: Our approach can be formally summarised as follows.

Original: We assume that the human scores H , the system scores A and the baseline system scores B jointly come from some trivariate distribution P ( H, A, B ) , which is speciﬁc to each subtask.
Revised: We assume that the human scores H , the system scores A and the baseline system scores B jointly come from some trivariate distribution P ( H, A, B ) , which is speciﬁc to each subtask.

Original: To compare the performance of two systems, we compute the sample
Revised: To compare the performance of two systems, we compute the sample

Original: Pearson correlation coefﬁcients r AH and r BH .
Revised: Pearson correlation coefﬁcients r AH and r BH .

Original: Since these correlations share the variable H , they are themselves dependent.
Revised: Since these correlations share the variable H , they are themselves dependent.

Original: There are several parametric tests for the difference between dependent correlations; however, their appropriateness beyond the assumptions of normality remains an active area of research (Hittner et al., 2003;
Revised: There are several parametric tests for the difference between dependent correlations; however, their appropriateness beyond the assumptions of normality remains an active area of research (Hittner et al., 2003;

Original: Wilcox & Tian, 2008; Wilcox, 2009).
Revised: Wilcox & Tian, 2008; Wilcox, 2009).

Original: The distributions of the human scores in the STS tasks are generally not normal; what’s more, they vary greatly depending on the subtask (some are multimodal, others are skewed, etc.).
Revised: The distributions of the human scores in the STS tasks are generally not normal; what’s more, they vary greatly depending on the subtask (some are multimodal, others are skewed, etc.).

Original: Fortunately, nonparametric resampling-based approaches, such as bootstrap (Efron & Tibshirani, 1994), present an attractive alternative to parametric tests when the distribution of the test statistic is unknown.
Revised: Fortunately, nonparametric resampling-based approaches, such as bootstrap (Efron & Tibshirani, 1994), present an attractive alternative to parametric tests when the distribution of the test statistic is unknown.

Original: In our case, the statistic is simply the difference between two correlations ˆ∆ =
Revised: In our case, the statistic is simply the difference between two correlations ˆ∆ =

Original: r AH − r BH .
Revised: r AH − r BH .

Original: The main idea behind bootstrap is intuitive and elegant: just like a sample is drawn from the population, a large number of ‘bootstrap’ samples can be drawn from the actual sample.
Revised: The main idea behind bootstrap is intuitive and elegant: just like a sample is drawn from the population, a large number of ‘bootstrap’ samples can be drawn from the actual sample.

Original: In our case, the dataset consists of triplets D =
Revised: In our case, the dataset consists of triplets D =

Original: { ( h i , a i , b i ) } Mi =1 .
Revised: { ( h i , a i , b i ) } Mi =1 .

Original: Each bootstrap sample is a result of drawing M data points from D with replacement.
Revised: Each bootstrap sample is a result of drawing M data points from D with replacement.

Original: Finally, we approximate the distribution of ∆ by evaluating it on a large number of bootstrap samples, in our case ten thousand.
Revised: Finally, we approximate the distribution of ∆ by evaluating it on a large number of bootstrap samples, in our case ten thousand.

Original: We use this information to construct bias-corrected and accelerated (BCa) 95% conﬁdence intervals for ∆ .
Revised: We use this information to construct bias-corrected and accelerated (BCa) 95% conﬁdence intervals for ∆ .

Original: BCa (Efron, 1987) is a fairly advanced second-order method that accounts for bias and skewness in the bootstrapped distributions, effects we did observe to a small degree in certain subtasks.
Revised: BCa (Efron, 1987) is a fairly advanced second-order method that accounts for bias and skewness in the bootstrapped distributions, effects we did observe to a small degree in certain subtasks.

Original: Once we have the conﬁdence interval for ∆ , the decision rule is then simple: if zero is inside the interval, then the difference between correlations is not signiﬁcant.
Revised: Once we have the conﬁdence interval for ∆ , the decision rule is then simple: if zero is inside the interval, then the difference between correlations is not signiﬁcant.

Original: Inversely, if zero is outside, we may conclude that the two approaches lead to statistically different results.
Revised: Inversely, if zero is outside, we may conclude that the two approaches lead to statistically different results.

Original: The location of the interval further tells us which one performs better.
Revised: The location of the interval further tells us which one performs better.

Original: The results are presented in Table 6.
Revised: The results are presented in Table 6.

Original: In summary, out of 72 experiments we signiﬁcantly outperform the baseline in 56 (77.8%) and underperform in only one (1.39%), while in the remaining 15 (20.8%) the differences are nonsigniﬁcant.
Revised: In summary, out of 72 experiments we signiﬁcantly outperform the baseline in 56 (77.8%) and underperform in only one (1.39%), while in the remaining 15 (20.8%) the differences are nonsigniﬁcant.

Original: We hope our analysis is useful to the community and will serve as a good starting point for conducting thorough signiﬁcance testing on the current as well as future STS benchmarks.
Revised: We hope our analysis is useful to the community and will serve as a good starting point for conducting thorough signiﬁcance testing on the current as well as future STS benchmarks.

Original: T a b l e 6 :
Revised: T a b l e 6 :

Original: P ea r s on c o rr e l a ti on f o r
Revised: P ea r s on c o rr e l a ti on f o r

Original: D yn a M a x a nd a v e r a g e d w o r d v ec t o r , a nd t h e% c on ﬁ d e n ce i n t e r v a l f o r t
Revised: D yn a M a x a nd a v e r a g e d w o r d v ec t o r , a nd t h e% c on ﬁ d e n ce i n t e r v a l f o r t

Original: h e d i ff e r e n ce b e t w ee n t w o c
Revised: h e d i ff e r e n ce b e t w ee n t w o c

Original: o rr e l a ti on s .
Revised: o rr e l a ti on s .

Original: A cc o r d i ng t
Revised: A cc o r d i ng t

Original: o t h i s a n a l
Revised: o t h i s a n a l

Original: y s i s ,
Revised: y s i s ,

Original: D yn a M a x s i
Revised: D yn a M a x s i

Original: gn
Revised: gn

Original: i ﬁ ca n tl y ou t p e rf
Revised: i ﬁ ca n tl y ou t p e rf

Original: o r m s t
Revised: o r m s t

Original: h e b a s e li n e i n 77 .%
Revised: h e b a s e li n e i n 77 .%

Original: o f t
Revised: o f t

Original: h e e xp e r
Revised: h e e xp e r

Original: i m e n t s a nd s
Revised: i m e n t s a nd s

Original: i
Revised: i

Original: gn i ﬁ ca n tl
Revised: gn i ﬁ ca n tl

Original: y l o
Revised: y l o

Original: s e s i n on l y 1 .%
Revised: s e s i n on l y 1 .%

Original: o f t h e m .
Revised: o f t h e m .

Original: I n t
Revised: I n t

Original: h
Revised: h

Original: e
Revised: e

Original: r
Revised: r

Original: e m a
Revised: e m a

Original: i n i ng 20 .%
Revised: i n i ng 20 .%

Original: o f t
Revised: o f t

Original: h e e xp e r
Revised: h e e xp e r

Original: i m e n t s t h e t w o a pp r
Revised: i m e n t s t h e t w o a pp r

Original: o ac
Revised: o ac

Original: h e s p e rf
Revised: h e s p e rf

Original: o r m s t a ti s ti c
Revised: o r m s t a ti s ti c

Original: a
Revised: a

Original: ll y t
Revised: ll y t

Original: h e s a m e .
Revised: h e s a m e .

Original: G l o
Revised: G l o

Original: V e f a s t T e x t w o r d2v ec
Revised: V e f a s t T e x t w o r d2v ec

Original: D yn a M a x - J
Revised: D yn a M a x - J

Original: A vg .
Revised: A vg .

Original: C o s .
Revised: C o s .

Original: ∆% C
Revised: ∆% C

Original: I
Revised: I

Original: D yn a M a x - J
Revised: D yn a M a x - J

Original: A vg .
Revised: A vg .

Original: C o s .
Revised: C o s .

Original: ∆% C
Revised: ∆% C

Original: I
Revised: I

Original: D yn a M a x - J
Revised: D yn a M a x - J

Original: A vg .
Revised: A vg .

Original: C o s .
Revised: C o s .

Original: ∆% C
Revised: ∆% C

Original: I
Revised: I

Revised: STS

Revised: M S R p a r 49 .42 .

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: M S R v i d 71 .66 .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: S M

Revised: T e u r op a r l

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ] s u r p r i s e .

Original: O n
Revised: O n

Original: W N 52 .47 .
Revised: W N 69 .57 .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 73 ] s u r p

Revised: r

Revised: i s e .

Revised: S M

Revised: T n e w s

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: STS

Revised: F N

Revised: W N 39 .38 .[- 6 .

Revised: [Table]

Revised: 40 ] h ea d li n e s 69 .63 .

Revised: [Table]

Revised: O n

Revised: W N 52 .47 .

Original: STS d e f tf o r u m 43 .30 .
Revised: STS d e f tf o r u m 43 .30 .

Original: 32 ] d e f tn e w s 70 .64 .
Revised: 32 ] d e f tn e w s 70 .64 .

Original: 03 ] h ea d li n e s 64 .58 .
Revised: 03 ] h ea d li n e s 64 .58 .

Original: 32 ] i m a g e s 75 .62 .
Revised: 32 ] i m a g e s 75 .62 .

Original: O n
Revised: O n

Original: W N 63 .57 .
Revised: W N 63 .57 .

Original: 45 ] t w ee t - n e w s 74 .53 .
Revised: 45 ] t w ee t - n e w s 74 .53 .

Original: STS a n s w e
Revised: STS a n s w e

Original: r
Revised: r

Original: sf o r u m s 61 .36 .
Revised: sf o r u m s 61 .36 .

Original: a n s w e
Revised: a n s w e

Original: r
Revised: r

Original: ss t ud e n t s 73 .63 .
Revised: ss t ud e n t s 73 .63 .

Original: 79 ] b e li e f 67 .44 .
Revised: 79 ] b e li e f 67 .44 .

Original: h
Revised: h

Original: ea d li n e s 72 .66 .
Revised: ea d li n e s 72 .66 .

Original: 32 ] i m a g e s 79 .69 .
Revised: 32 ] i m a g e s 79 .69 .

Original: STS a n s w e
Revised: STS a n s w e

Original: r-
Revised: r-

Original: a n s w e r 59 .40 .
Revised: a n s w e r 59 .40 .

Original: h
Revised: h

Original: ea d li n e s 71 .61 .
Revised: ea d li n e s 71 .61 .

Original: 38 ] p l a g i a r i s m 79 .54 .
Revised: 38 ] p l a g i a r i s m 79 .54 .

Original: 09 ] po s t e d iti ng 80 .53 .
Revised: 09 ] po s t e d iti ng 80 .53 .

Original: qu e s ti on - qu e s ti on 63 .47 .
Revised: qu e s ti on - qu e s ti on 63 .47 .