Variance Matters: Detecting Semantic Differences without Corpus/Word Alignment (2023)

Abstract: In this paper, we propose methods for discovering semantic differences in words appearing in two corpora. The key idea is to measure the coverage of meanings of a word in a corpus through the norm of its mean word vector, which is equivalent to examining a kind of variance of the word vector distribution. The proposed methods do not require alignments between words and/or corpora for comparison that previous methods do. All they require are to compute variance (or norms of mean word vectors) for each word type. Nevertheless, they rival the best-performing system in the SemEval-2020 Task 1. In addition, they are (i) robust for the skew in corpus sizes; (ii) capable of detecting semantic differences in infrequent words; and (iii) effective in pinpointing word instances that have a meaning missing in one of the two corpora under comparison. We show these advantages for historical corpora and also for native/non-native English corpora.

## introduction in this paper, we propose a method for detecting semantic or usage differences foot_1 in words and also a method for extracting their typical instances in context based on variance of contextualized word vectors, which bring out various applications: e.g., historical linguistics @xcite (e.g., discovering words that have acquired a meaning) and second language acquisition research @xcite (e.g., words and their meanings that non-native speakers do not use as native speakers). the key observation is that the more meanings a word covers in a corpus, the shorter its mean word vector becomes. we show that this property of contextualized word vectors can be quantified as a kind of variance of their distribution and that it is a good indicator of semantic differences. the major approach to semantic difference detection, which is based on non-contextualized word vectors such as word2vec @xcite , has several limitations and thus is not always applicable to any corpora as will be discussed in detail in sect. 5. above all, many non-contextualized word vector-based methods require some sort of correspondence between two corpora for comparison (e.g., word alignment). the task is, however, to find words that do not correspond well in terms of their meanings, and thus it is more natural not to assume any correspondence in advance, as @xcite point out. for example, it is not straightforward at all to align words between native and non-native english corpora. besides, most previous methods are computationally costly and are not suitable for targeting all words in a large corpus. in contrast, the proposed methods do not assume any correspondence between corpora. all they require are to compute the mean of contextualized word vectors and its variance (or its norm) for each word type. they do not require training or hyperparameter search unlike previous methods. nevertheless, they rival the best-performing system in the semeval-2020 task 1 @xcite . they are even effective in corpus pairs whose sizes are considerably different and also in infrequent words. in addition, they are capable of pinpointing word instances that have a meaning missing in one of the two corpora under comparison. for instance, in sect. 3, they reveal that near is one of the most semantically different words between the native and non-native sub-corpora (approximately 10,000 and 100,000 words, respectively) of icnale @xcite ; its most representative instance out of the 11 near occurrences in the native portion is "it has near impossible," which is interpreted as almost; this usage does not appear at all in the 267 instances of near in the non-native sub-corpus. the contributions of this paper are three-fold: (i) we show for the first time that norms of the mean contextualized word vectors are a good indicator of semantic differences; (ii) we show that (i) is mathematically equivalent to examining variance of the word vector distribution; (iii) we actually reveal words that have semantic differences with their representative instances in 1800s/2000s english and also in native/non-native english. ## methods we describe two methods, one for detecting words that have semantic differences in two corpora and one for extracting their representative instances. so far, we have used the term word abstractly to mean both word type and word token. hereafter, for better understanding, we will distinguish between the two; we will use the term word type to refer to word types and the term word instance to refer to word tokens in context, which we assume is a whole sentence. ## discussion the evaluation results in sect. 3 show that the concentration parameter κ or its reciprocal, which is interpreted as a kind of variance, is a good indicator of semantic differences. the proposed detection method, which solely relies on the ratio of κ, achieves the best detection accuracy as shown in subsect. 3.2. besides, the differences shown in interestingly, the results in the semeval-2020 sub-task suggest that semantic changes are reflected more in the concentration (or variance) than the mean direction µ; the methods that exploit the mean direction do not perform as well as the κ-based proposed method. it requires more in- one of the advantages of the proposed detection method is that it can predict whether the meaning of a target word is narrowing or broadening as illustrated in subsect. 2.1. namely, positive and negative values of eq.( 1 ) indicate narrowing and broadening, respectively. although it would be difficult to evaluate this property of the proposed method quantitatively, since there is no publicly available dataset in this regard, qualitatively we have seen actual broadening/narrowing word instances in subsect.3.3 to subsect. 3.5. another advantage is that the proposed methods are computationally efficient. they require no training or fine-tuning unlike the previous approaches as discussed in sect.5. besides, they have almost no hyper-parameters except for the threshold for word frequency (word instances whose frequency is more than this threshold are the target of analy-sis). they solely rely on an off-the-shelf language model (bert in our case), which is a large advantage in terms of implementation and development. fortunately, κ, or norms of mean word vectors are stable with respect to word frequency as shown in appendix d; norms of the mean vector become almost constant with a frequency of five or so. this stability of the norm enables the proposed methods to discover semantic differences in infrequent instances. it should be emphasized that only one word instance would be enough to proof that a word type has a certain meaning, as in the near example in it should also be emphasized that its robustness for the low frequency problem comes from the use of contextualized word vectors via a large language model. even if the source and target corpora are small, the obtained word vectors should be statistically reliable considering the language model is trained on a large corpus. in contrast, the previous methods based on non-contextualized word vectors inevitably suffer from the low frequency problem because non-contextualized word vectors are learned from the input corpora. as we have discussed, the proposed methods are simple and efficient, but at the same time effective in discovering semantic differences. all these nice properties come from the assumption of the von mises-fisher distribution behind word vectors. as we will discuss in sect. 7, although this assumption has its limitations theoretically, it works well practically as we have seen in sect. 3. ## related work linguists (e.g., @xcite ) often use frequency-based methods to discover differences in words between two corpora. because they only consider superficial frequency counts, it requires more sophisticated methods to conduct deeper analyses into semantic differences. the use of non-contextualized word vectors is the major approach to semantic difference detection. for diachronic analysis, @xcite propose setting word vectors obtained from the previous time to initial word vectors of the next. for the same purpose, @xcite ; @xcite propose methods for discovering semantic differences by aligning words in two corpora. these alignment-based methods make a strong assumption that words are linearly aligned between two corpora, which does not necessarily hold in any corpus pair (e.g., native and non-native english). @xcite ; @xcite extend this approach to discover semantic differences across languages while their methods require a word-alignment dictionary. yao et al. ( 2018 ) avoid the problem in word alignment by learning word vectors and alignment simultaneously. their method has sensitive hyper-parameters that need to be tuned, which results in a complex combinatorial optimization problem @xcite . @xcite propose a method for detecting semantic differences by simultaneously optimizing multiple word vectors. while this method does not require linear transformations or extensive hyper-parameter search, it requires a list of target words, which is not realistic in practical uses. @xcite extends @xcite 's method by optimizing multiple context vectors together with multiple word vectors. these non-alignment-based methods, however, still make the assumption that word vectors and/or context vectors are close to each other in two corpora. the task of detecting semantic differences is to find words that are not aligned well in terms of their meanings in two corpora, and thus methods that do not require such assumptions are preferable. @xcite propose a method based on nearest neighbors obtained by non-contextualized word vectors to avoid making such assumptions. for this, it is applicable to any pair of corpora. at the same time, it suffers from the bias in corpus sizes and the low frequency problem. some researchers use contextualized word vectors for semantic difference detection. @xcite ; @xcite ; @xcite ; @xcite automatically group contextualized word vectors to predict word meanings and then compare the results to detect semantic differences. predicting word meanings is itself another difficult task. also, it is costly to train a classifier or to conduct clustering for every single word type found in corpora. recently, @xcite have proposed a method exploiting the variance of word vector distribution. their method models the distribution by the gaussian distribution and uses both the mean vector and the covariance matrix. the use of the gaussian distribution has an advantage in that it is not isotropic unlike the von-mises fisher distribution. on the other hand, their method is computationally costly. besides, it requires that words instances should be similar in number in the two corpora for comparison. to avoid the problem, they use sampling to have word instance sets of a similar size, which is not necessary in our methods. ## conclusions in this paper, we have proposed using the concentration parameter (or variance), which can be calculated from the norm of mean word vectors, to detect semantic differences with their representative instances. the proposed methods do not require assumptions concerning words and corpora for comparison that the previous methods do. the only assumption is that word vectors follow the von mises-fisher distribution. this enables the proposed methods to be applied to corpus pairs such as native and non-native english corpora where the assumptions of the previous methods do not hold. also, they are simple and efficient in that they do not require training nor extensive hyper-parameter search. with these methods, we have actually discovered semantic differences in historical corpora and also in native and non-native english corpora. we have shown that they are effective even for infrequent word instances and also for corpora whose sizes are considerably different.