Rethinking Word Similarity: Semantic Similarity through Classification Confusion (2025)

Abstract: Word similarity has many applications to social science and cultural analytics tasks like measuring meaning change over time and making sense of contested terms. Yet traditional similarity methods based on cosine similarity between word embeddings cannot capture the context-dependent, asymmetrical, polysemous nature of semantic similarity. We propose a new measure of similarity, Word Confusion, that reframes semantic similarity in terms of feature-based classification confusion. Word Confusion is inspired by Tversky (1977)’s suggestion that similarity features be chosen dynamically. Here we train a classifier to map contextual embeddings to word identities and use the classifier confusion (the probability of choosing a confounding word c instead of the correct target word t) as a measure of the similarity of c and t. The set of potential confounding words acts as the chosen features. Our method is comparable to cosine similarity in matching human similarity judgments across several datasets (MEN, WirdSim353, and SimLex), and can measure similarity using predetermined features of interest. We demonstrate our model’s ability to make use of dynamic features by applying it to test a hypothesis about changes in the 18th C. meaning of the French word “révolution” from popular to state action during the French Revolution. We hope this reimagining of semantic similarity will inspire the development of new tools that better capture the multi-faceted and dynamic nature of language, advancing the fields of computational social science and cultural analytics and beyond.

## introduction semantic similarity measures allow computational social scientists, digital humanists, and nlp practitioners to perform fine-grained synchronic and diachronic analysis on word meaning, with important applications to areas like cultural analytics and le-gal and historical document analysis @xcite . the cosine between two embedding vectors is the most commonly used similarity metric for textual analysis across a variety of fields @xcite @xcite . however, it neither accounts for the multi-faceted nature of similarity @xcite @xcite nor does it align exactly with how humans perceive similarity @xcite . cosine similarity is dominated by a small number of rogue dimensions due to the anisotropy of contextual embedding spaces @xcite , underestimates the semantic similarity of high-frequency words @xcite , is a symmetric metric that cannot capture the asymmetry of semantic relationships foot_0 @xcite , and often fails in capturing human interpretation @xcite . here, we propose to think about concept similarity metrics differently. we are inspired by @xcite 's seminal work on similarity, presuming that humans have a rich mental representation of concepts. when faced with a particular task, like similarity assessment, we extract and compile from this rich representation only the relevant features for the required task. this formulation highlights the multi-faceted and context-dependent nature of similarity judgments @xcite . to demonstrate the potential of this new framing, we introduce a proof-of-concept: word confusion, a self-supervised method that defines the semantic similarity between words according to a classifier's confusion between them. in a nutshell, we first train a classifier to map from a word embedding to the word itself, distinguishing it from a set of distractors. at inference time, given a new embedding e for a target word t, the probability the classifier assigns to a confound word c is used as a measure of similarity of words c and t. the set of distractor words used in training act as features, thus, the similarity between words is based on their feature interchangeability. we test word confusion on standard wordsimilarity tasks like sentiment and grammatical gender classification and show that it is comparable to standard cosine similarity and can be more meaningful. we then apply word confusion to real-world data exploration tasks in collaboration with the fourth author, who is a scholar of french literature and history. we use the archive parlementaires from 1789-1793 to study a long debated question in the political history of revolutionary france: how "revolution" went from being seen as a means of popular liberation, to becoming identified with governmental actions that often flouted such personal freedoms. in summary, our paper: our new formulation can learn more complex word identity boundaries than cosine similarity alone; accounts for the asymmetrical nature of semantic similarity; can be easily adapted to desired domains; and provides a more interpretable measure. we hope this new formulation will spark the creation of tools for cultural analytics and computational social science that account for the multifaceted and complex nature of semantic similarity. ## introducing word confusion our method begins by defining a set of words, or features. for example, we might choose the set @xmath0, green, blue} if we wanted to study similarity related to colors. these words will act as features that can be selected by the analyst to focus on a particular dimension or question. our process then has two phases: training and inference. in training (illustrated in part (a) of figure 1 )) we extract from a corpus a set of sentences containing each of these words, such as "the sunset painted the sky a brilliant shade of red" for the word "red". we then use bert to extract the contextual embeddings of these feature-words, and train a classifier to map from an embedding to its corresponding word identity. the classifier's training objective is to correctly classify the embedding to the word that corresponds to it. more formally, given a chosen set of word w and embeddings {e 1 , e 2 ....e i } ∈ e that correspond to word identities {w 1 , w 2 , ..., w i } ∈ w , we train a logistic regression classifier on all pairs of {e i , w i }. at inference (part (b) in figure 1 ), we wish to define the semantic similarity of a word in terms of the classifier's classes, which can be thought of as features. 3 now suppose we would like to compute the similarity of the new word "burgundy" to various colors. we extract the contextual embedding of "burgundy" given the sentence "burgundy is a deep reddish-brown shade inspired by wine", and use the trained classifier to compute the probability that the "burgundy"-embedding corresponds to each class @xmath1, green, blue}. we then use the classifier's confusion matrix to understand which primary colors burgundy is similar to. for example, the similarity of "burgundy" to "red" is the probability our classifier assigns to the class "red". more formally, the probability distribution predicted by the model, ⃗ p @xmath2, is used to quantify the semantic similarity between w j (burgundy) and each w i , ∀w @xmath3, green, blue}). thus: the set of distractor words chosen to train the initial classifier thus act as features that can be selected by the analyst to focus on a particular dimension or question. note that as with the example "burgundy", the input word at inference can be out-of-vocabulary with respect to the classifier, or the target word can be one of the classifier's classes (in which case we ignore the probability it assigns to that word and use the other n -1 features.) (a) training word confusion: the classifier is trained in a self-supervised manner, after selecting the desired features (in this example the classes red, green blue). we extract sentences containing those 3 "feature" words. the input to the classifier is the contextual embedding of the class token, e.g., the bert embedding of the word "red" in the sentence "the sunset painted the sky a brilliant shade of red". the classifier is trained to predict a class ("red") from that contextual embedding. figure 1 : word confusion: we predetermine a set of classes for our classifier, in this case {red, green, blue}. this choice of classes defines the similarity features used to describe the input word. at training, we extract sentences containing the chosen class words {red, green, blue}. we then train the classifier to map from a bert contextual embeddings of these words to right class /feature (color, in this case). at inference, we extract bert's contextual embeddings of a target word, that may be a class (red) or may be a new word ("burgundy"). we then input the embedding to the classifier and use its confusion matrix to understand which primary colors burgundy is similar to. ## theoretical intuitions in this section, we discuss the importance of word identifiability and how it enables the core mechanics of word confusion, and discuss some theoretical differences between word confusion and cosine similarity. ## real-world data word confusion is a new similarity measuring tool that could assist in understanding real-world data and trends. in this section, we focus on two applications of word confusion -its ability to serve as a feature extractor and to detect temporal trends in word meaning. ## related work on cultural change both static and contextualized embedding spaces contain semantically meaning dimensions that align with high-level linguistic and cultural features @xcite . these embeddings have enabled a large number of quantitative analyses of temporal shifts in meaning and links to cultural or social scientific variables. for example early on, using static embeddings, hamilton et al. (2016) measured linguistic drifts in global semantic space as well as cultural shifts in particular local semantic neighborhoods. @xcite demonstrated that changes in word embeddings correlated with demographic and occupation shifts through the 1900s. analyzes of contextualized embeddings have identified semantic axes based on pairs of "seed words" or "poles" (soler and apidianaki, 2020; @xcite . across the temporal dimension, such axes can measure the evolution of gender and class @xcite , internet slang @xcite , and more @xcite @xcite . @xcite proposes a probabilistic measure for lexical similarity. it's also instructive to consider the similarity of our method with tasks like word sense disambiguation (wsd) and named entity recognition (ner). the central idea behind word confusion of mapping from embeddings to categories are also found in ner and wsd. what differs is the dynamic nature of the categories. where ner focuses on pre-defined concept hierarchies and wsd on pre-defined senses per word, word confusion focuses on a coherent but dynamic grouping of words that is interpretable for a given task. ## discussion and conclusion in this paper, we reframe the task of semantic similarity from one of measuring distances to one of classification confusion. this formulation highlights the context-dependency of similarity judgments, meanwhile avoiding the pitfalls of geometric similarity measures @xcite . this new framing of semantic similarity in terms of classification confusion introduces new properties that are inspired by cognitive models of similarity @xcite and accounts for the asymmetric nature of semantic similarity, captures different aspects of both similarity and multi-faceted words and ofter a measure that has interpretability benefits. our proof-of-concept method, word confusion, demonstrates the practical applicability and effectiveness of this reframing. empirical results show that it outperforms cosine similarity on standard datasets. for computational social science or cultural analytics applications, word confusion can serve as a way to learn to represent words using target features (e.g., "school" in terms of {positive, negative}, and can be used to trace the meaning of a word as a function of time (like the word "révolution"). the theoretical underpinnings of word confusion allow it to learn complex word identity boundaries and capture the directional nature of similarity, offering a richer and more flexible framework for understanding word meanings. while we implemented word confusion as a linear classifier, the method naturally extends to capturing non-linear relationships among embedding components by replacing the linear projection with neural networks. investigating whether the error function preserves its useful properties in nonlinear settings remains an open question for future work. while our experiments are preliminary and the space of possible similarity measures is enormous, we hope this reimagining of semantic similarity will inspire the development of new tools that better capture the multi-faceted and dynamic nature of language, advancing the fields of computational social science and cultural analytics and beyond.