Original: N OVEL R ANKING -B ASED L EXICAL S IMILARITY M EASURE FOR W ORD E MBEDDING
Revised: C OMPARISON OF P ARAGRAM AND G LO V E R ESULTS FOR S IMILARITY B ENCHMARKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Distributional semantics models derive word space from linguistic items in context.
Revised: Distributional Semantics Models(DSM) derive word space from linguistic items in context.

Original: Meaning is obtained by deﬁning a distance measure between vectors corresponding to lexical entities.
Revised: Meaning is obtained by deﬁning a distance measure between vectors corresponding to lexical entities.

Original: Such vectors present several problems.
Revised: Such vectors present several problems.

Revised: This work concentrates on quality of word embeddings, improvement of word embedding vectors, applicability of a novel similarity metric used ‘on top’ of the word embeddings.

Original: In this paper we provide a guideline for post process improvements to the baseline  vectors.
Revised: In this paper we provide comparison between two methods for post process improvements to the baseline DSM vectors.

Original: We focus on reﬁning the similarity aspect, address imperfections of the model by applying the hubness reduction method, implementing relational knowledge into the model, and providing a new ranking similarity deﬁnition  that give maximum weight to the top 1 component value. This feature ranking is similar to the one used in information retrieval.
Revised: The counter-ﬁtting method which enforces antonymy and synonymy constraints into the Paragram vector space representations recently showed improvement in the vectors’ capability for judging semantic similarity. The second method is our novel RESM method applied to GloVe baseline vectors. By applying the hubness reduction method, implementing relational knowledge into the model by retroﬁtting synonyms and providing a new ranking similarity deﬁnition RESM that gives maximum weight to the top vector component values we equal the results for the ESL and TOEFL sets in comparison with our calculations using the Paragram and Paragram + Counter-ﬁtting methods.

Original: All these enrichments outperform current literature results for joint ESL and TOEF sets comparison.
Revised: For SIMLEX-999 gold standard since we cannot use the RESM the results using GloVe and PPDB are signiﬁcantly worse compared to Paragram. Apparently, counter-ﬁtting corrects hubness. The Paragram or our cosine retroﬁtting method are state-of-the-art results for the SIMLEXgold standard. They are 0.2 better for SIMLEX-999 than word2vec with sense de-conﬂation (that was announced to be state-of the-art method for less reliable gold standards).

Original: Since single word embedding is a basic element of any semantic task one can expect a signiﬁcant improvement of results for these tasks. Moreover, our improved method of text processing can be translated to continuous distributed representation of biological sequences for deep proteomics and genomics.
Revised: Apparently relational knowledge and counter-ﬁtting is more important for judging semantic similarity than sense determination for words. It is to be mentioned, though that Paragram hyperparameters are ﬁtted to SIMLEXresults. The lesson is that many corrections to word embeddings are necessary and methods with more parameters and hyperparameters perform better.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Distributional language models are frequently used to measure word similarity in natural language (e.g. Frackowiak et al.
Revised: Distributional language models are frequently used to measure word similarity in natural language.

Original: Recent works usually use the Distributional Hypothesis (Harris (1954)) to generate the language models.

Original: This model often consists of a set of vectors; each vector corresponds to a character string, which represents a word.
Revised: This is a basis for many semantic tasks. The DSM often consists of a set of vectors; each vector corresponds to a character string, which represents a word.

Original: Mikolov et al.
Revised: Mikolov et al.

Original: and Pennington et al.
Revised: and Pennington et al.

Original: (2014) implement word embedding (WE) algorithms.
Revised: (2014) implemented the most commonly used word embedding (WE) algorithms.

Original: Vector components in language models created by these algorithms are latent.
Revised: Vector components in language models created by these algorithms are latent.

Original: Similarity between words is deﬁned as a function of vectors corresponding tothe given words.
Revised: Similarity between words is deﬁned as a function of vectors corresponding to given words.

Original: The cosine measure is the most frequently used similarity function.
Revised: The cosine measure is the most frequently used similarity function, although many other functional forms were attempted.

Original: Santus et al.
Revised: Santus et al.

Original: (2016) highlights the fact that the cosine can be outperformed by ranking based functions. 
Revised: (2016) highlights the fact that the cosine can be outperformed by ranking based functions. As pointed out by many works, e.g. Hill et al.

Original:  Vector space word representations obained from purely distributional information of words in large unlabelled corpora are not enough to best the state-of-the-art results in query answering benchmarks, because they suffer from 4 types of weaknesses:
Revised: , evidence suggests that distributional models are far from perfect. Vector space word representations obtained from purely distributional information of words in large unlabeled corpora are not enough to best the state-of-the-art results in query answering benchmarks, because they suffer from several types of weaknesses:

Original: Inadequate deﬁnition of similarity, 2.
Revised: Inadequate deﬁnition of similarity, 2.

Original: Inability of accounting of senses of words,3.
Revised: Inability of accounting of senses of words,

Original: Appeareance of hubness that distorts distances between vectors, 4.
Revised: Appearance of hubness that distorts distances between vectors, 4.

Original: Inability of distinguishing from antonyms
Revised: Inability of distinguishing from antonyms.

Revised: In case of retroﬁtting distortion vector space - loss of information contained in the original vectors

Original: In this paper we use the existing word embedding model but with several post process enhancement techniques.
Revised: In this paper we use the existing word embedding model but with several post process enhancement techniques.

Original: We address threeout of four of these issues.
Revised: We address three of these problems.

Original: In particular we deﬁne  novel similarity measure, dedicated for language models.
Revised: In particular, we deﬁne a novel similarity measure, dedicated for language models.

Original: The Euclidean distance is based on the locations of points in such a space.

Original: Similarity is a function, which isan monotonically opposite to distance.
Revised: Similarity is a function, which is monotonically opposite to distance.

Original: As the distance between two given entities gets shorter, entities are more similar.
Revised: As the distance between two given entities gets shorter, entities are more similar.

Original: This holds for language models.
Revised: This holds for language models.

Original: Similarity between words is equal to similarity between their corresponding vectors.
Revised: Similarity between words is equal to similarity between their corresponding vectors.

Original: There are various deﬁnitions of distance.
Revised: There are various deﬁnitions of distance.

Original: The most common Euclidean distance is deﬁned as follows:
Revised: The most common Euclidean distance is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Similarity based on the Euclidean deﬁnition is inverse to the distance:
Revised: Similarity based on the Euclidean deﬁnition is inverse to the distance:

Original: [Equation]
Revised: [Equation]

Original: Angular deﬁnition of distance is deﬁned with cosine function:
Revised: Angular deﬁnition of distance is deﬁned with cosine function:

Original: [Equation]
Revised: [Equation]

Original: In this case we could apply the inversion, however it is much more convinient to take the negative of distance. Number 1 in the equation stands for the normalizing, hence the similarity is deﬁned as follows:
Revised: We deﬁne angular similarity as:

Original: [Equation]
Revised: [Equation]

Original: Both Euclideandeﬁnition of distance and Cosine  could be looked at as the analysis of vector components.
Revised: Both Euclidean and Cosine deﬁnitions of distance could be looked at as the analysis of vector components.

Original: Simple operations, like addition and multiplication work really well in low dimensional spaces.
Revised: Simple operations, like addition and multiplication work really well in low dimensional spaces.

Original: We believe, that applying those metrics in spaces of higher order is not ideal, hence we compare cosine similarity to a measure of distance dedicated for high dimensional spaces. 
Revised: We believe, that applying those metrics in spaces of higher order is not ideal, hence we compare cosine similarity to a measure of distance dedicated for high dimensional spaces. In this paper we restrict ourselves to three gold standards: TOEFL, ESL and SIMLEX-999.

Original: R ELATED WORK
Revised: The ﬁrst two are small but reliably annotated (and therefore conﬁdence in their performance can be assumed 100%). Other used benchmarks suffer from several drawbacks. Both WS-353 Finkelstein et al. and MEN Bruni et al.

Revised: (2012) do not measure the ability of models to reﬂect similarity.

Revised: Moreover, as pointed out by Hill et al. (2015), for WS-353, RG Rubenstein & Goodenough (1965) and MEN, state-of-the-art models have reached the average performance of a human annotator on these evaluations.

Revised:  R ELATED WORK

Revised: In information retrieval (IR) document similarity to a question phrase can be expressed using ranking.

Revised: The objective and repeatable comparison of rankings requires a rank similarity measure (e.g. the one introduced by Webber et al.

Original: Santus et al.
Revised: Santus et al.

Original: (2016) introduce the ranking based similarity function called APSyn.
Revised: (2016) introduced another ranking based similarity function called APSyn.

Original: In their experiment APSyn outperforms the cosine similarity, reaching 73% of accuracy in the best cases (an improvement of 27% over cosine) on the ESL dataset, and 70% accuracy (an improvement of 10% over cosine) on the TOEFL dataset. 
Revised: In their experiment APSyn outperformed the cosine similarity, reaching 73% of accuracy in the best cases (an improvement of 27% over cosine) on the ESL dataset, and 70% accuracy (an improvement of 10% over cosine) on the TOEFL dataset. In contrast to our work, they use the Positive Pointwise Mutual Information algorithm to create their language model.

Revised: Despite such large improvement the absolute results were quite poor compared to the state-of-theart.

Original: In contrast toour work, they use the Positive Pointwise Mutual
Revised: Our motivation for the RESM method was the NDCG measure that is considered the most appropriate measure for IR evaluation. Levy et al. (2015) investigated Pennington et al. (2014) proposition of using the context vectors in addition to the word vectors as GloVe’s output.

Original: Information algorithm to create their language model.
Revised: This led a different interpretation of its effect on the cosine similarity function. The similarity terms which can be divided into two groups: second-order includes a symmetric combination of the ﬁrst-order and second order similarities of x and y, normalized by a function of their reﬂective ﬁrst-order similarities. The best Levy et al. (2015) result for SIMLEX-999 is 0.438; for WordSim-353 similarity and WordSim-353 relatedness their result is a few hundredths worse than that of Paragram Wieting et al.

Original: A successful avenue to enhance WE was pointed out by Faruqui et al.
Revised: A successful avenue to enhance word embeddings was pointed out by Faruqui et al.

Original: (2014), using WordNet (Miller & Fellbaum (2007)), and the Paraphrase Database  (Ganitkevitch et al.
Revised: (2014), using WordNet (Miller & Fellbaum (2007)), and the Paraphrase Database 1.0 (Ganitkevitch et al.

Original: to provide synonymy relation information to vector optimization equations.
Revised: to provide synonymy relation information to vector optimization equations.

Original: They call this process retroﬁtting, a pattern we adapt to the angular deﬁnition of distance, which is more suitable to our case.
Revised: They call this process retroﬁtting, a pattern we adapt to the angular deﬁnition of distance, which is more suitable to our case.

Original: We also address hubness reduction.
Revised: We also address hubness reduction.

Original: Hubness is related to the phenomenon of concentration of distances - the fact that points get closer at large vector dimensionalities.
Revised: Hubness is related to the phenomenon of concentration of distances - the fact that points get closer at large vector dimensionalities.

Original: Hubness is very pronounced for vector dimensions of the order of thousands.
Revised: Hubness is very pronounced for vector dimensions of the order of thousands.

Revised: To for reduce hubness Dinu & Baroni (2014) used ranking instead of similarity.

Original:  We apply this method of localized centering for hubness reduction (Feldbauer & Flexer (2016)) for the language models. 
Revised: The method requires the availability of more, unlabeled source space data, in addition to the test instances, and is natural in application to cross-language or cross-modal applications We apply this method of localized centering for hubness reduction (Feldbauer & Flexer (2016)) for the language models. The Paragram vectors were further improved by Mrksic et al. This was achieved by the joint retro -ﬁtting and counter-ﬁtting optimization of the following objective function: [Equation]

Original: {+The values of hyperparameters are the following δ = 1 , γ = 
+}
Revised: {+The values of hyperparameters are the following δ = 1 , γ = 
+}

Original: M ETHOD
Revised: and k 1 = k 2 = k 3 . In the original Paragram + Counter-ﬁtting work δ = 1 corresponds to zero angle between a pair of vectors for synonyms, and γ = 0 corresponds to 90 ◦ angle between a pair of vectors for antonyms, which is not quite consistent with SIMLEX-999 (Hill et al. (2015) average value of cosines for synonym pairs, 0.77, and average value of cosines for antonym pairs, 0.17.

Revised: However, as we veriﬁed, using these theoretically more acceptable values does not improve SIMLEX-999 results.

Revised:  M ETHOD

Original: In our work we deﬁne the language model as a set of word representations.
Revised: In our work we deﬁne the language model as a set of word representations.

Original: Each word is represented by its vector.
Revised: Each word is represented by its vector.

Original: We refer to a vector corresponding to a word w i as v i .
Revised: We refer to a vector corresponding to a word w i as v i .

Original: A complete set of words for a given language is referred to as a vector space model.
Revised: A complete set of words for a given language is referred to as a vector space model(VSM).

Original: We deﬁne similarity between words w i and w j as a function of vectors v i and v j .
Revised: We deﬁne similarity between words w i and w j as a function of vectors v i and v j .

Original: [Equation]
Revised: [Equation]

Original: We present an algorithm for obtaining optimized similarity measures given a vector space model for word embedding.
Revised: We present an algorithm for obtaining optimized similarity measures given a vector space model for word embedding.

Original: The algorithm consists of 6 steps:
Revised: The algorithm consists of 6 steps:

Original: Reﬁne the vector space using the L2 retroﬁt algorithm 2.
Revised: Reﬁne the vector space using the cosine retroﬁt algorithm 2.

Original: Obtain vector space of centroids 3.
Revised: Obtain vector space of centroids 3.

Original: Obtain vectors for a given pair of words and optionally for given context words 4.
Revised: Obtain vectors for a given pair of words and optionally for given context words 4.

Original: Recalculate the vectors using the localized centering method 5.
Revised: Recalculate the vectors using the localized centering method 5.

Original: Calculate ranking of vector components for a given pair of words 6.
Revised: Calculate ranking of vector components for a given pair of words 6.

Original: Use the ranking based similarity function to obtain the similarity between a given pair of words.
Revised: Use the ranking based similarity function to obtain the similarity between a given pair of words.

Revised: The cosine retroﬁt algorithm follows the original Faruqui et al.

Revised: (2014) approach except that we use the cosine distance and rotations of retroﬁtted vectors in space.

Revised: The formulae look different compared to Mrksic et al.

Original:  We use all of the methods in together to achieve signiﬁcant improvement over the baseline method.
Revised: but give equivalent results. We use all of the methods in together to achieve signiﬁcant improvement over the baseline method.

Original: We present details of the algorithm in the following sections.
Revised: We present details of the algorithm in the following sections.

Original: 3. B ASELINE
Revised: 3. B ASELINE

Original: The cosine function provides the baseline similarity measure:
Revised: The cosine function provides the baseline similarity measure:

Original: [Equation]
Revised: [Equation]

Original: The cosine function has been to achieving a reasonable baseline.
Revised: The cosine function achieves a reasonable baseline.

Original: It is superior to the Euclidean similarity measure and is used in various works related to word similarity.
Revised: It is superior to the Euclidean similarity measure and is used in various works related to word similarity.

Original: In our work we use several post-process modiﬁcations to the vector space model; we also redeﬁne the similarity measure.
Revised: In our work we use several post-process modiﬁcations to the vector space model. We also redeﬁne the similarity measure.

Original: 3. I MPLEMENTING RELATIONAL KNOWLEDGE INTO THE VECTOR SPACE MODEL
Revised: 3. I MPLEMENTING RELATIONAL KNOWLEDGE INTO THE VECTOR SPACE MODEL

Original: Let us deﬁne a lexicon of synonyms L .
Revised: Let us deﬁne a lexicon of relations (rel) L .

Original: Each row in the lexicon consists of a word and a set of its synonyms.
Revised: Each row in the lexicon consists of a word and a set of its related words (synonyms and antonyms).

Original: [Equation]
Revised: [Equation]

Original: A basic method of implementing synonym knowledge into the vector space model was previously described in Ganitkevitch et al.
Revised: A basic method of implementing synonym knowledge into the vector space model was previously described in Ganitkevitch et al.

Original: We refer to that method as retroﬁt; it uses the iterational algorithm of moving the vector towards an average vector of its synonyms according to the following formula.
Revised: We refer to that method as retroﬁt. It uses the iterational algorithm of moving the vector towards an average vector of its weighted related words according to the following formula.

Original: [Equation]
Revised: [Equation]

Original: In the original formula Faruqui et al.
Revised: In the original formula Faruqui et al.

Original: , variables α and β allow us to weigh the importance of certain synonyms.
Revised: , variables α and β allow us to weigh the importance of certain relations.

Original: The basic retroﬁt method moves the vector towards its destination (shortens the distance between the average synonym to a given vector) using the Euclidean deﬁnition of distance.
Revised: The basic retroﬁt method moves the vector towards its destination (e.g. shortens the distance between the average synonym to a given vector) using the Euclidean deﬁnition of distance.

Original: This is not consistent with the cosine distance.
Revised: This is not consistent with the cosine distance used for evaluation.

Original: Instead we improve Faruqui et al.
Revised: Instead we improve Faruqui et al.

Original: (2014) idea by performing operations in spherical space by normalizing the vector, thus preserving the angular deﬁnition of distance. 
Revised: (2014) idea by performing operations in spherical space, thus preserving the angular deﬁnition of distance. Given a basic rotation matrix R for a plane ( i, j ) .

Original: This amounts to rotating the vector instead of translating it. We implemented the basic transformation for the rotation; however it proved to be time consuming, which affected our work on the subject.
Revised: [Equation]

Original: Therefor, for simplicity, the averagevector of two normalized vectors is precisely between given vectors in both the Euclidean and angular deﬁnition of distance. This gives the following formula:
Revised: For each plain ( p, q ) we calculate angle θ ( p,q ) ( v i , v j ) between word and its related word. We apply a rotation R p,q ( θ ( p,q ) ( v i , v j ) β j ) to that word. Finally, we take an average of a given word vector and average of rotated, related vector.

Original: [Equation]
Revised: [Equation]

Original: We refer to this method as to L2 retroﬁtting.
Revised: We refer to this method as to generalized retroﬁtting.

Revised: The original formula from Faruqui et al.

Original: 3. L OCALIZED CENTERING
Revised: (2014) was obtained by minimizing the following objective function. [Equation] We change the Euclidean deﬁnition of distance to cosine in the formula to obtain a new objective function. [Equation]

Revised: We take ﬁrst derivative of Ψ (cid:48) to deﬁne a single step change of a vector v i for each vector component v ( i, k ) .

Revised: [Equation]

Revised: This method and obtained results are equivalent to the work of Mrksic et al.

Original: We  address the problem of hubness in high dimensional spaces with the localized centering approach applied to every vector in the space.
Revised: We refer to this method as cosine retroﬁtting. 3. L OCALIZED CENTERING We address the problem of hubness in high dimensional spaces with the localized centering approach applied to every vector in the space.

Original: The centered values of vectors, centroids, are the average vectors of k nearest neighbors of the given vector v i .
Revised: The centered values of vectors, centroids, are the average vectors of k nearest neighbors of the given vector v i .

Original: We apply a cosine distance measure to calculate the nearest neighbors.
Revised: We apply a cosine distance measure to calculate the nearest neighbors.

Original: [Equation]
Revised: [Equation]

Original: In Feldbauer & Flexer (2016), the authors pointed out that skewness of a space has a direct connection to the hubness of vectors.
Revised: In Feldbauer & Flexer (2016), the authors pointed out that skewness of a space has a direct connection to the hubness of vectors.

Original: We follow the pattern presented in that work and recalculate the vectors using the following formula.
Revised: We follow the pattern presented in the Hara et al. and recalculate the vectors using the following formula.

Original: [Equation]
Revised: [Equation]

Original: Parameter γ in the equation is equal to the skewness of the space.
Revised: We use empirical method to ﬁnd values of the hyperparameter gamma .

Original: 3.3. R ANKING BASED SIMILARITY FUNCTION
Revised: R ANKING BASED SIMILARITY FUNCTION

Original: We propose a component ranking function as the similarity measure.
Revised: We propose a component ranking function as the similarity measure.

Original: This idea was originally introduced in Santus et al.
Revised: This idea was originally introduced in Santus et al.

Original: (2016) who proposed the APSyn ranking function.
Revised: (2016) who proposed the APSyn ranking function.

Original: Let us deﬁne the vector v i as a list of its components.
Revised: Let us deﬁne the vector v i as a list of its components.

Original: [Equation]
Revised: [Equation]

Original: We then obtain the ranking r i by sorting the list in descending order (d in the equation denotes type of ordering), denoting each of the components with its rank on the list.
Revised: We then obtain the ranking r i by sorting the list in descending order (d in the equation denotes type of ordering), denoting each of the components with its rank on the list.

Original: [Equation]
Revised: [Equation]

Original: APSyn is calculated on the intersection of the N components with the highest score.
Revised: APSyn is calculated on the intersection of the N components with the highest score.

Original: [Equation]
Revised: [Equation]

Original: APSyn was originally computed on the PPMI language model, which has unique feature of nonnegative vector components.
Revised: APSyn was originally computed on the PPMI language model, which has unique feature of nonnegative vector components.

Original: As this feature is not given for every language model, we take into account negative values of the components.
Revised: As this feature is not given for every language model, we take into account negative values of the components.

Original: We deﬁne the negative ranking by sorting the components in ascending order (a in the equation denotes type of ordering).
Revised: We deﬁne the negative ranking by sorting the components in ascending order (a in the equation denotes type of ordering).

Original: [Equation]
Revised: [Equation]

Original: As we want our ranking based similarity function to preserve some of the cosine properties, we deﬁne score values for each of the components and similarly to the cosine function, multiply the scoresfor each component.
Revised: As we want our ranking based similarity function to preserve some of the cosine properties, we deﬁne score values for each of the components and similarly to the cosine function, multiply the scores

Original:  As the distribution of component values is Gaussian, we use the exponential function.[Equation]
Revised: [Table] for each component. As the distribution of component values is Gaussian, we use the exponential function.

Original: [Equation]
Revised: [Equation]

Original: Parameters k and d correspond respectively to weighting of the score function and the dimensionality of the space.
Revised: Parameters k and d correspond respectively to weighting of the score function and the dimensionality of the space.

Original: With high k values, the highest ranked component will be the most inﬂuential one.
Revised: With high k values, the highest ranked component will be the most inﬂuential one.

Original: The rationale is maximizing information gain.
Revised: The rationale is maximizing information gain.

Original: Our measure is similar to infAP and infNDCG measures used in information retrieval Roberts et al.
Revised: Our measure is similar to infAP and infNDCG measures used in information retrieval Roberts et al.

Original: (2017) that give maximum weight to the  top 1 result.
Revised: (2017) that give maximum weight to the several top results.

Original:  P@10 gives equal weight to the top 10,results.
Revised: In contrast, P@10 assigns equal weight to the top 10 results.

Original: Lower k values increase the impact of lower ranked components at the expense of long tail of ranked components.
Revised: Lower k values increase the impact of lower ranked components at the expense of ‘long tail’ of ranked components.

Original: We use the default k value of 10.
Revised: We use the default k value of 10.

Original: The score function is identical for both ascending and descending rankings.
Revised: The score function is identical for both ascending and descending rankings.

Original: We address the problem of polysemy with a differential analysis process.
Revised: We address the problem of polysemy with a differential analysis process.

Original: Similarity between pair of words is captured by discovering the sense of each word and then comparing two given senses of words.
Revised: Similarity between pair of words is captured by discovering the sense of each word and then comparing two given senses of words.

Original: The sense of words is discovered by analysis of their contexts.
Revised: The sense of words is discovered by analysis of their contexts.

Original: We deﬁne the differential analysis of a component as the sum of all scores for that exact component in each of the context vectors.
Revised: We deﬁne the differential analysis of a component as the sum of all scores for that exact component in each of the context vectors.

Original: [Equation]
Revised: [Equation]

Original: Finally we deﬁne the Ranking based Exponential Similarity Measure (RESM) as follows.
Revised: Finally we deﬁne the Ranking based Exponential Similarity Measure (RESM) as follows.

Original: [Equation]
Revised: [Equation]

Original: The equation is similar to the cosine function.
Revised: The equation is similar to the cosine function.

Original: Both cosine and RESM measures multiply values of each component and sum the obtained results.
Revised: Both cosine and RESM measures multiply values of each component and sum the obtained results.

Original: Contrary to the cosine function, RESM scales with a given context.
Revised: Contrary to the cosine function, RESM scales with a given context.

Original: It should be noted, that we apply differential analysis with a context function h .
Revised: It should be noted, that we apply differential analysis with a context function h .

Original: An equation in this form is dedicated for the  test sets we use in evaluation.
Revised: An equation in this form is dedicated for the ESL and TOEFL test sets.

Original: The ﬁnal value is calculated as a sum of the RESM for both types of ordering.
Revised: The ﬁnal value is calculated as a sum of the RESM for both types of ordering.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: 3. I MPLEMENTATION
Revised: 3. I MPLEMENTATION

Original: The algorithm has been implemented in Python 2.7.
Revised: The algorithm has been implemented in C#.

Original: It is publicly available via the repository along with implementation of the evaluation.
Revised: It is publicly available via the repository along with implementation of the evaluation.

Original:  E VALUATION
Revised:  E VALUATION

Original: We have tested our method against TOEFL and ESL test sets.
Revised: This work compares the state-of-the-art word embedding methods for three most reliable gold standards: TOEFL, ESL and SIMLEX-999.

Original: TOEFL consists of 80 questions, ESL consists of 50 questions.
Revised: TOEFL consists of 80 questions, ESL consists of 50 questions.

Original: Questions in ESL are signiﬁcantly harder.
Revised: The questions in ESL are signiﬁcantly harder.

Original: Both tests consist of questions designed for nonnative speakers of English. Each question in the tests consists of a  question word with a set of four  answers.
Revised: Each question in these tests consists of a 1 anonymized repository [Table] question word with a set of four candidate answers.

Original: It is worth pointing out, that the context given by a set of possible answers often deﬁnes the question.
Revised: It is worth pointing out, that the context given by a set of possible answers often deﬁnes the question (a selection of a sense of a word appearing in the question).

Revised: Answering these tests does not require ﬁnding the most similar word out of the whole dictionary but only from multiple choice candidates; therefore TOEFL and ESL are less demanding than SIMLEX-999.

Original: Example questions in Table 1 highlight the problem.
Revised: A question example in Table 1 highlights the problem.

Original: In the ﬁrst question, all of possible answers are building materials.
Revised: In the ﬁrst question, all of possible answers are building materials.

Original: Wood should be rejected as there is more appopriate answer.
Revised: Wood should be rejected as there is more appropriate answer.

Original: In second question, out of possible answers, only wood is a building material which makes it a good candidate for the correct answer.
Revised: In second question, out of possible answers, only wood is a building material which makes it a good candidate for the correct answer.

Original: This is a basis for applying a differential analysis in the similarity measure.
Revised: This is a basis for applying a differential analysis in the similarity measure.

Original: Table 2.
Revised: Table 2.

Original: illustrates state of the art results for both test sets.
Revised: illustrates state of the art results for both test sets.

Original: The TOEFL test set was introduced in Landauer & Dutnais (1997); the ESL test set was introduced in Turney (2001)
Revised: The TOEFL test set was introduced in Landauer & Dumais (1997); the ESL test set was introduced in Turney (2001).

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: We use the unmodiﬁed vector space model trained on 840 billion words from Common Crawl data with the GloVe algorithm introduced in Pennington et al.
Revised: We use the unmodiﬁed vector space model trained on 840 billion words from Common Crawl data with the GloVe algorithm introduced in Pennington et al.

Original: The model consists of 2.2 million unique vectors; Each vector consists of 300 components.
Revised: The model consists of 2.2 million unique vectors; Each vector consists of 300 components.

Original: The model can be obtained via the GloVe authors website.
Revised: The model can be obtained via the GloVe authors website.

Revised: We also use the GloVe reﬁnement enriched with PPDB 1.0 relations called Paragram Wieting et al.

Revised: (there is an extension of this paper at https://arxiv.org/pdf/1506.03487.pdf with more details included).

Original: We run several experiments, for which settings are as follows:
Revised: We run several experiments, for which settings are as follows:

Original: In the evaluation skewness γ = 9 and k = 10 .
Revised: In the evaluation skewness γ = 9 and k = 10 .

Original: All of the possible answers are taken as context words for the differential analysis.
Revised: All of the possible answers are taken as context words for the differential analysis.

Original: In our runs we use all of the described methods separately and conjunctively.
Revised: In our runs we use all of the described methods separately and conjunctively.

Original: We refer to the methods in the following way.
Revised: We refer to the methods in the following way.

Original: We denote the localized centering method for hubness reduction as HR.
Revised: We denote the localized centering method for hubness reduction as HR.

Original: We use a Paraphrase Database lexicon introduced in Ganitkevitch et al.
Revised: We use a Paraphrase Database lexicon introduced in Ganitkevitch et al.

Original: for the retroﬁtting.
Revised: for the retroﬁtting.

Original: We denote L2 retroﬁtting as RETRO.
Revised: We denote the generalized retroﬁtting as RETRO and the cosine retroﬁtting as cosRETRO.

Original: 4. H EURISTIC IMPROVEMENT
Revised: 4. H EURISTIC IMPROVEMENT

Original: Although the hubness reduction method does not increase the number of correct answers for the
Revised: Although the hubness reduction method does not increase the number of correct answers for the

Original: ESL test set, we noticed that the average rank of the correct answer goes down from 1.32 to 1.24.
Revised: ESL test set, we noticed that the average rank of the correct answer goes down from 1.32 to 1.24.

Original: That is a signiﬁcant improvement.
Revised: That is a signiﬁcant improvement.

Original: To obtain better results we combined the results with and without localized centering.
Revised: To obtain better results we combined the results with and without localized centering.

Original: The heuristical method chooses the answer with the best average rank for both sets.
Revised: The heuristic method chooses the answer with the best average rank for both sets.

Original: By applying that method we obtained two additional correct answers. 
Revised: By applying that method we obtained two additional correct answers. 4. S ENSE RECOGNITION

Revised: Semantic distributional methods operate on words or characters.

Revised: A vector position for a word is an average of positions of the word senses.

Revised: The is a major deﬁciency, particularly if a word with a given sense is statistically rare.

Revised: In Table 4 of Mrksic et al.

Original: We improved theaccuracy results by 8.75% and 24% for TOEFL and ESL test sets respectively.
Revised: (2016) a pair: dumb dense appears amount the highest-error SIMLEX-999 word pairs using Paragram vectors (before counter-ﬁtting). Counter-ﬁtting does not improve this result. In Table 3 we also show the results sense recognition systems. The approach of Pilehvar & Navigli (2015) is purely based on the semantic network of WordNet and does not use any pre-trained word embeddings.

Revised: In their approach, similarly to the VSM representation of a linguistic item, the weight associated with a dimension in a semantic signature denotes the relevance or importance of that dimension for the linguistic item.

Revised: In Pilehvar & Collier this method was improved using the Faruqui et al.

Revised: (2014) like formula, with senses in place of synonyms or antonyms.

Revised: In this DECONF method the word embeddings were used following Mikolov et al.

Revised: (2013).They obtained state-of-the-art results for MEN-3K, and RG-65 (here Spearman correlation 0.896 by far exceeds the inter-annotator

Revised: [Table]

Revised: conﬁdence).

Revised: However, for SIMLEX-999 the multi-sense results are 0.3 worse (Pilehvar & Navigli (2015)) and 0.2 worse in Pilehvar & Collier compared to the Paragram based results.

Revised: We improved the accuracy results by 8.75% and 24% for TOEFL and ESL test sets respectively.

Original: We observe the largest improvement of accuracy by applying the localized centering method for
Revised: We observe the largest improvement of accuracy by applying the localized centering method for

Original: TOEFL test set.
Revised: TOEFL test set.

Original: Testing on the ESL question set seems to give the best results by changing the[Table] similarity measure from cosine to RESM.
Revised: Testing on the ESL question set seems to give the best results by changing the similarity measure from cosine to RESM.

Original: Thus each step of the algorithm improves the results.
Revised: Thus each step of the algorithm improves the results.

Original: The signiﬁcance of the improvement varies based on the test set.
Revised: The signiﬁcance of the improvement varies, depending on the test set.

Original: A complete set of measured accuracies is presented in Table 3.
Revised: A complete set of measured accuracies is presented in Table 3.

Original: The results for the APSyn ranking method are obtained using the Glove vector, not using PPMI as in /citeDBLP:journals/corr/SantusCLLH
Revised: The results for the APSyn ranking method are obtained using the Glove vector, not using PPMI as in Santus et al.

Revised: The performance of Paragram with counter-ﬁtting for SIMLEX-999 is better than inter-annotator agreement, measured by average pairwise Spearman ρ correlation, which varies between 0.614 to 0.792 depending on concept types in SIMLEX-999.

Revised: There are many results that are signiﬁcantly worse than these in Table 3.

Revised: For example, in Kiela et al.

Revised: (2015) the accuracy for the TOEFL standard is 88.75% and for SIMLEX-999 the Spearman ρ is 0.53.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: The procedure we provide in this is capable of achieving the best results in the  word embedding category and (nearly) state-of-the art results for any current metod.
Revised: This work compares the state-of-the-art word embedding methods for three most reliable gold standards: TOEFL, ESL and SIMLEX-999. For TOEFL and ESL the GloVe, PPDB baseline with retroﬁtting, our novel RESM similarity measure and hubness reduction we are able to equal the Paragram results. For SIMLEX-999 Paragram with Counter-ﬁtting results are clearly better than the

Original:  The work of Lu et al.
Revised: Glove based methods using the PPDB 1.0. However, we propose the cosine retroﬁtting that basically achieves the Paragram with Counter-ﬁtting results. The Paragram with Counter-ﬁtting method contains several hyperparameters which is one source of its success. Its effects can be seen in Table 10 at https://arxiv.org/pdf/1506.03487.pdf.

Original: (2011) employed 2 ﬁtting constants (and it is not clear that they were the same for all questions) for answering the TOEFL test where only 50 questions are used.
Revised: The Spearman ρ values for SIMLEX-999 are 0.667 for Paragram300 ﬁtted to WS353, and 0.685 for Paragram300 ﬁtted to SIMLEX-999. The difference is even larger for WS353.

Original: Techniques introduced in the paper are lightweight and easy to implement, yet they provide a signiﬁcant performance boost to  the language model. Recently, a lot of progress was achieved in relating antonyms to synonyms Santus et al.
Revised: Then the Spearman ρ values for WS-353 are 0. 769 for Paragram ﬁtted toWS353, and 0.720 for Paragram300 ﬁtted to SIMLEX-999. Still the best word embedding based methods are not able to achieve the performance of other dedicated methods for TOEFL and

Original: (2016), Nguyen et al. We tried the antonym RETRO method to take into account relational knowledge on antonyms. The method repels two vectors that are an antonym pair. Contrary to
Revised: ESL.

Original: Nguyen et al.
Revised: The work of Lu et al.

Original: (2017), who obtained minimal improvement, in our case accuracy does not improve (for any value of repulsion strength), which could be because of relatively large window (10) in the original Glove work.
Revised: (2011) employed 2 ﬁtting constants (and it is not clear that they were the same for all questions) for answering the TOEFL test where only 50 questions are used.

Original: All works that could distingush antonyms from synonyms using word embedding used much smaller context windows. We plan to extent our Glove based calculations to smaller windows. This needs to be studied more thoroughly.
Revised: Techniques introduced in the paper are lightweight and easy to implement, yet they provide a signiﬁcant performance boost to the language model.

Original: Since the single word embedding is a basic element of any semantic task one can expect a signicant improvement of results for these tasks.
Revised: Since the single word embedding is a basic element of any semantic task one can expect a signiﬁcant improvement of results for these tasks.

Original: In particular, SemEval-2017 International Workshop on Semantic Evaluation run (among others) the following tasks(se2):
Revised: In particular, SemEval-2017 International Workshop on Semantic Evaluation run (among others) the following tasks(se2):

Original: Task 1: Semantic Textual Similarity 2.
Revised: Task 1: Semantic Textual Similarity 2.

Original: Task 2: Multilingual and Cross-lingual Semantic Word Similarity 3.
Revised: Task 2: Multilingual and Cross-lingual Semantic Word Similarity 3.

Original: Task 3: Community Question Answering
Revised: Task 3: Community Question Answering

Original: in the category Semantic comparison for words and texts.
Revised: in the category Semantic comparison for words and texts.

Original: Another immediate application would be information retrieval (IR).
Revised: Another immediate application would be information retrieval (IR).

Original: Expanding queries by adding potentially relevant terms is a common practice in improving relevance in IR systems.
Revised: Expanding queries by adding potentially relevant terms is a common practice in improving relevance in IR systems.

Original: There are many methods of query expansion.
Revised: There are many methods of query expansion.

Original: Relevance feedback takes the documents on top of a ranking list and adds terms appearing in these document to a new query.
Revised: Relevance feedback takes the documents on top of a ranking list and adds terms appearing in these document to a new query.

Original: In this work we use the idea to add synonyms and other similar terms to query terms before the pseudo- relevance feedback.
Revised: In this work we use the idea to add synonyms and other similar terms to query terms before the pseudo- relevance feedback.

Original: This type of expansion can be divided into two categories.
Revised: This type of expansion can be divided into two categories.

Original: The ﬁrst category involves the use of ontologies or lexicons (relational knowledge).
Revised: The ﬁrst category involves the use of ontologies or lexicons (relational knowledge).

Original: The second category is word embedding (WE).
Revised: The second category is word embedding (WE).

Original: Here closed words for expansion have to be very precise, otherwise a query drift may occur, and precision and accuracy of retrieval may deteriorate. 
Revised: Here closed words for expansion have to be very precise, otherwise a query drift may occur, and precision and accuracy of retrieval may deteriorate. There are several avenues to further improve the similarity results.

Original: Moreover, our improved method of text processing can be translated to continuous distributed representation of biological sequences for deep proteomics and genomics.
Revised: Use the multi-language version of the methods (e

Original: Protein sequence is typically notated as a string of letters, listing the amino acids starting at the amino-terminal end through to the carboxyl-terminal end. Either a three letter code or single letter code can be used to represent the 20 naturally occurring amino acids.
Revised: g. Recski et al.

Original: Until recently most methods used n-grams. The immediate application is family classiﬁcation task Asgari & Mofrad (2015).
Revised: Use PPDB 2.0 to design the Paragram vectors Pavlick et al. Apply the multi-sense methods (knowledge graps) with state-of-the-art relational enriched vectors 4.

Original: This work is supported by anonymous grant.
Revised: Recalibrate annotation results using state-of-the-art results.