W ord N et-based similarity metrics for adjectives (2016)

Abstract: Le and Fokkens (2015) recently showed that taxonomy-based approaches are more reliable than corpus-based approaches in estimating human similarity ratings. On the other hand, distributional models provide much better coverage. The lack of an established similarity metric for adjectives in WordNet is a case in point. I present initial work to establish such a metric, and propose ways to move forward by looking at extensions to WordNet. I show that the shortest path distance between derivationally related forms provides a reliable estimate of adjective similarity. Furthermore, I find that a hybrid method combining this measure with vector-based similarity estimations gives us the best of both worlds: more reliable similarity estimations than vectors alone, but with the same coverage as corpus-based methods.

## introduction in this paper i present new @xcite measures to provide reliable estimates of human word similarity ratings. ever since @xcite published their simlex-999 data set, many people have tried to find a way to determine the similarity of all the word pairs without being affected by the relatedness of the words. recently, @xcite showed that taxonomy-based approaches beat vector-based approaches @xcite in the estimation of the simlex data. this is because corpus-based approaches are more affected by association, while taxonomy-based approaches mainly use vertical relations that are well-suited for determining similarity. however, corpus-based approaches do have a big advantage in their coverage. moreover, le and fokkens left adjectives out of consideration, for lack of a good wordnet-similarity measure. my aim was to fill this lacuna, and also to find a way to mitigate the coverage issue. in section 3, i propose three wordnet-based adjective similarity measures, and evaluate them on the simlex-999 data. 1 section 4 provides a more thorough discussion of our results. at the same time, we should acknowledge that the representation of the adjectives in wordnet could use some attention. section 5 proposes future work, looking at some extensions to wordnet that might improve our proposed measures. section 6 concludes. ## evaluation it is important to note that similarity is a relative measure; we do not learn anything from the fact that the similarity between adjectives x and y is 2.4 unless we also know the similarity between other pairs of adjectives. only then do we learn whether x and y are very similar or not similar at all. in other words, being able to rank adjective pairs in terms of their similarity is more important than having a specific number for each pair. this is why the spearman rank correlation is typically used for evaluation. i follow this standard procedure in our general evaluation. le and fokkens (2015) argue for the use of multiple different evaluation methods, since they may lead to different conclusions about the results. they propose to use ordering accuracy (an evaluation of the relative ordering between all combinations of pairs, following @xcite ), supplemented with tie correction, i.e. giving a partial score to word pairs having the same similarity score. this levels the playing field, as taxonomybased similarity values are more prone to yield ties than corpus-based measures (discrete versus real scores). the intuition behind this proposal is that overall ranking is more important than arbitrary local differences. therefore, we should not punish algorithms as much for getting specific pair orderings 'wrong' when they are too close to call. in the discussion (section 4), i will use le and fokkens' comparison by group, where pairs of pairs of adjectives are grouped by the difference in their similarity scores in the gold standard. this is useful to see how well different models perform at varying levels of granularity. ## current possibilities in this section, i examine distance metrics for adjectives in wordnet. i will first look at two classical measures, hso @xcite and @xcite , and show that they perform reasonably well (although not state-of-theart). next, i propose a method based on derivationally related forms, that are associated with the adjective lemmas. though this approach achieves good results, it does suffer from poor coverage. i will then look at an alternative approach using attributes, but conclude that it is not feasible to incorporate them in our distance metric. finally, to remedy the coverage issue, i propose a hybrid approach using both wordnet and distributional vectors. ## discussion from the spearman correlations alone, it seems that we gain precision by involving derivationally related forms (drf) in the estimation of similarity values. this picture changes when we look at ordering accuracy. i found that the drf-based and vector-based approaches achieve comparable results. for the subset of 88 pairs where both adjectives have drfs, i found a slight advantage for the vector-based method compared to the drf-based method: 70% versus 71%. for the full dataset, this is exactly reversed, with a precision of 71% for the hybrid method and 70% for the vector-based method. that is not to say that both measures encode the same information; indeed we find interesting differences when we compare the pairs on a group-by-group basis. ## future work: extensions to wordnet there are several projects that add new information to the adjective synsets, which can be used to increase coverage. below i discuss potential uses and the current limitations of this information. adjective hierarchy germanet @xcite ) contains a hierarchy for adjectives, structured using hyponymy relations. this means that it is possible to use any of the available wordnet distance metrics directly on the adjective synsets. unfortunately, the mapping between ger-manet and princeton wordnet is still incomplete, and there is no dataset similar to simlex for german to test this idea. ## conclusion we have seen several different wordnet-based measures of adjective similarity: the classical lesk and hso measures, and two new measures based on specific cross-pos links and the shortestpath distance between the nouns they are related to. it turns out that the derivationally related forms-link can be used to get state-of-the-art results on the simlex-999 dataset. if coverage is an issue, then the hybrid method from section 3.4 is a better option than using vectors alone (though not by a large margin). we also noted that, on closer inspection, these measures do not seem to capture the same information. therefore, future research should look at new ways to combine distributional and taxonomy-based measures. another way to improve similarity estimations would be to extend wordnet with new information. for example, the attributes-relation currently seems unusable for any similarity-related work, but may still be useful if more attribute links are added to wordnet. and looking at the literature, there is a lot of promising work being done with other wordnets, leaving us with many interesting avenues to explore the relation between wordnet and lexical similarity.