Just Rank: Rethinking Evaluation with Word and Sentence Similarities (2022)

Abstract: Word and sentence embeddings are useful feature representations in natural language processing. However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. Word and sentence similarity tasks have become the de facto evaluation method. It leads models to overfit to such evaluations, negatively impacting embedding models’ development. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. Further, we propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks. Extensive experiments are conducted based on 60+ models and popular datasets to certify our judgments. Finally, the practical evaluation toolkit is released for future benchmarking purposes.

## introduction distributed representation of words @xcite @xcite and sentences @xcite @xcite have shown to be extremely useful in transfer learning to many nlp tasks. therefore, it plays an essential role in how we evaluate the quality of embedding models. among many evaluation methods, the word and sentence similarity task gradually becomes the de facto intrinsic evaluation method. figure 1 shows examples from word and sentence similarity datasets. in general, the datasets consist of pairs of words (w 1 , w 2 ) (or sentences) and human-annotated similarity scores s h . to evaluate an embedding model φ(•), we first extract embeddings for (w 1 ,w 2 ): (e 1 , e 2 ) = (φ(w 1 ), φ(w 2 )). then, a similarity measure is applied to compute an predicted score s @xmath0, e 2 ), where cosine similarity is adopted as sim unquestionably in the majority of cases. finally, the correlation between s h and s p is computed, and a higher correlation suggests good alignment with human annotations and a better embedding model. many studies, especially those targeting on information retrieval via semantic search and clustering @xcite , have used the similarity task as the only or main evaluate method @xcite @xcite @xcite . we observe a number of issues in word or sentence similarity tasks ranging from dataset collection to the evaluation paradigm, and consider that focusing too much on similarity tasks would negatively impact the development of future embedding models. the significant concerns are summarized as follows, which generally apply to both word and sentence similarity tasks. first, the definition of similarity is too vague. there exist complicated relationships between sampled data pairs, and almost all relations contribute to the similarity score, which is challenging to non-expert annotators. second, the similarity evaluation tasks are not directly relevant to the downstream tasks. we believe it is because of the data discrepancy between them, and the properties evaluated by similarity tasks are not the ones important to downstream applications. third, the evaluation paradigm can be tricked with simple post-processing methods, making it unfair to benchmark different models. inspired by spreading-activation theory @xcite , we propose to evaluate embedding models as a retrieval task, and name it as evalrank to address the above issues. while similarity tasks measure the distance between similarity pairs from all similarity levels, evalrank only considers highly similar pairs from a local perspective. our main contributions can be summarized as follows: ## related work word embedding has been studied extensively, and popular work @xcite @xcite are mainly built on the distributional hypothesis @xcite , where words that appear in the same context tend to share similar meanings. the early work on sentence embedding are either built upon word embedding @xcite @xcite or follow the distributional hypothesis on a sentence level @xcite @xcite . recent development of sentence embedding are incorporating quite different techniques including multi-task learning @xcite , supervised inference data @xcite , contrastive learning @xcite @xcite and pre-trained language models @xcite @xcite . nonetheless, even though different methods choose different evaluation tasks, similarity task is usually the shared task for benchmarking purposes. similarity task is originally proposed to mimic human's perception about the similarity level between word or sentence pairs. the first word similarity dataset was collected in 1965 @xcite , which consists of 65 word pairs with human annotations. it has been a standard evaluation paradigm to use cosine similarity between vectors for computing the correlation with human judges @xcite . many studies raise concerns about such evaluation paradigm. @xcite and @xcite points out some problems with word similarity tasks, including low correlation with downstream tasks and lack of task-specific similarity. reimers et al. (2016 ), eger et al. (2019) and zhelezniak et al. (2019) states current evaluation paradigm for semantic textual similarity (sts) tasks are not ideal. one most recent work @xcite questions about the data collection process of sts datasets and creates a new semantic relatedness dataset (str) by comparative annotations @xcite . there are also other intrinsic evaluation methods for word and sentence embedding evaluation, but eventually did not gain much popularity. word analogy task is first proposed in @xcite to detect linguistic relations between pairs of word vectors. zhu and de melo (2020) recently expanded the analogy concept to sentence level. however, the analogy task is more heuristic and fragile as an evaluation method @xcite . recently, probing tasks have been proposed to measure intriguing properties of sentence embedding models without worrying much about practical applications @xcite @xcite . because of the lack of effective intrinsic evaluation methods, @xcite and @xcite seeks to include more domain-specific tasks for evaluation. ## problems with similarity tasks in this work, we discuss the problems of similarity tasks both on word and sentence levels. they are highly similar from data collection to evaluation paradigm and are troubled by the same problems. ## conclusion in this work, we first discuss the problems with current word and sentence similarity evaluations and proposed evalrank, an effective intrinsic evaluation method for word and sentence embedding models. it shows a higher correlation with downstream tasks. we believe that our evaluation method can have a broader impact in developing future embedding evaluation methods, including but not limited to its multilingual and task-specific extensions. the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 951-962, minneapolis, minnesota. association for computational linguistics. xunjie zhu and gerard de melo. 2020. sentence analogies: linguistic regularities in sentence embeddings. in proceedings of the 28th international conference on computational linguistics, pages 3389-3400, barcelona, spain (online). international committee on computational linguistics. xunjie zhu, tingfeng li, and gerard de melo. 2018. exploring semantic properties of sentence embeddings. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 632-637, melbourne, australia. association for computational linguistics.