Analogies minus analogy test: measuring regularities in word embeddings (2020)

Abstract: Vector space models of words have long been claimed to capture linguistic regularities as simple vector translations, but problems have been raised with this claim. We decompose and empirically analyze the classic arithmetic word analogy test, to motivate two new metrics that address the issues with the standard test, and which distinguish between class-wise offset concentration (similar directions between pairs of words drawn from different broad classes, such as France-London, China-Ottawa,...) and pairing consistency (the existence of a regular transformation between correctly-matched pairs such as France:Paris::China:Beijing). We show that, while the standard analogy test is flawed, several popular word embeddings do nevertheless encode linguistic regularities.

## introduction vector semantic models saw a surge in interest after embeddings trained under the word2vec architecture @xcite were shown to encode linguistic regularities @xcite . the demonstration relied on the arithmetic analogy test: relations such as king + womanman ≈ queen were shown to hold for a variety of semantic and grammatical relations. evaluation of word embeddings on analogy tests and training on related loss functions remains current. there is also continued interest in theoretically grounding the success of distributional embeddings on these tests @xcite . there is, however, a substantial literature pointing to problems with word analogies (see section 2), leading to the conclusion that word analogies are fragile in practice, and sometimes going so far as to imply that the positive results were erroneous. these critiques have been ambiguous as to whether the problem is that the embeddings in question do not really encode the relevant linguistic structure, or whether the issue is merely that the arithmetic analogy test as it is standardly defined, is flawed. the current paper confirms that there are serious problems with the standard analogy test, as it confounds three different properties of word vectors while purporting to measure only one: class-wise offset concentration (similar directions between pairs of words drawn from different broad classes, such as france-london, china-ottawa), withinpair similarity between test words (for example, the similarity between paris and france), as well as the pairing consistency the test sets out to measure (the presence of a regular direction encoding relations such as capital-of : france-paris, china-beijing). we give an algebraic decomposition of the standard analogy test that explains previous negative results in terms of within-pair similarity. using new measures, we show that, in practice, offset concentration, rather than true pairing consistency, may account for part of word embeddings' success. nevertheless, we show that several standard word embeddings do show pairing consistency for the relations tested by the bats analogy benchmark @xcite . 1 ## related work the validity of the arithmetic analogy test has been questioned in several papers, starting with @xcite . we detail in section 3 several major issues with the test as raised by @xcite , @xcite , and @xcite . @xcite , @xcite , @xcite and @xcite also raised concerns about the test and its assumptions. more recently, @xcite argued against this test as an inadequate tool for studying bias in word embeddings. @xcite observes that many of the issues have been ignored. some works have proposed other measures of linguistic relations in word embeddings. @xcite , @xcite , and @xcite all examined the similarity of vector offsets in a more direct way than the standard analogy test (see section 3 below). @xcite proposed a method based on predicting the class of the unknown word, and @xcite relaxed the assumptions by allowing probabilistic models to predict the relations. we claim that these works still do not provide a satisfactory measure of how well word vector offsets encode linguistic relations. without such a measure, it is impossible to assess whether the original conclusions are correct. we develop this argument, and then develop a new measure, below. in practice, the test assesses whether b * is the nearest neighbour to b + a * -a, using the cosine similarity (cosine of the angle between x and y): the assumption is that, if the offset o @xmath0, then this common offset direction encodes the same linguistic relation. for example, if paris minus france goes in the same direction as beijing minus china, and as other capital-country pairs, the common offset direction can be seen as encoding the relation capital-of. thus, the idea behind the arithmetic analogy test is that offsets corresponding to the same linguistic relation, if they are the same or roughly the same, should be interchangeable. therefore, france + (beijing -china) should be the same as france + (paris -france) (modulo the vector magnitude, if we are hypothesizing that it is the direction, and not the precise normed vector, that encodes the relation capital-of: hence the use of the cosine similarity). the criticisms of @xcite and the research it inspired have often been characterized as problems with "word analogies." this is ambiguous: it does not distinguish between problems with the method of using the arithmetic test just described for testing the presence of linguistic regularities, and the veracity of the conclusion that there are linguistic regularities coded in word embeddings as vector offsets. in order to resolve this ambiguity, one would need to know for sure, using some better measure, whether linguistic regularities are coded. we propose such a measure beginning in section 4. we first analyze the key problems that have been previously raised. ## measuring regularities the similarity between offsets, sim(o a , o b ), is the quantity of primary interest to assess whether linguistic relations are represented as consistent directions. we seek a measure of whether they are. as noted, @xcite showed that 2017 ) analyzed examples of correct and false analogies and showed them to have comparable offset similarities. this makes sense: a baseline level of offset similarity can be found by looking at unrelated pairs. however, that paper examined only one isolated example. we propose an offset concentration score (ocs) measuring the offset similarity among pairs belonging to the same relation and a pairing consistency score (pcs) measuring the distinctness of the similarities from a baseline. we propose pcs as the correct measure of linguistic regularity, with ocs a source of supplementary information. jing is likely to be predicted, since its cosine similarity to china is high. the problem is worse than the 2d schematic would suggests, since it can occur even when the word vectors are normalized, and when the offsets are quite long. china will be predicted if it is not excluded, since it is also in the neighbourhood. perfectly parallel offsets would imply that france and paris differ on the same dimension as china and beijing-and on no other dimensionand similarly for the other offsets of the same kind (canada-ottawa, . . . ). perfect parallelism is not necessary for sim(o a , o b ) to contribute to success on the analogy test. by offset concentration, we mean the degree to which, for an entire set of word pairs, the offsets are parallel to one another (thus, concentrated in a single direction). once we take into account that offsets need not be perfectly parallel, we must bear in mind that positive cosine similarity between offsets does not imply that the embedding space captures linguistic relations. for example, recalling @xcite , training embeddings to capture distributional regularities may group words into linguistic classeswith, for example, country names such as france, china. . . , occupying a region of the space distinct from that of paris, beijijng. . . , due to distributional commonalities. this by itself could yield offset concentration: offsets would all come from a similar origin, and go to a similar destination. however, sim(france-paris, china-beijing) would not necessarily be larger than sim(france-beijing, china-paris). we illustrate this issue in figure 4 . in such an example, we might be able to assert that the class of capital city names is distinct from the class of country names. but asserting that the relation capital-of is captured is a stronger property. it would require that moving from the france vector in the direction of the common capital-of vector lead to paris and not to beijing, nor to any other capital city-or, at least, that it passes closer to paris. we call this property pairing consistency. ## comparing word embeddings we evaluate the pairing consistency on the bats test for popular word embeddings. we include glove @xcite , a purely distributional model similar to word2vec; two embeddings making use of external semantic knowledge: dict2vec @xcite and conceptnet numberbatch @xcite ; and the static token embeddings of bert @xcite and gpt-2 @xcite . the left subfigure in figure 6 presents the standard 3cosadd analogy test accuracy (input words excluded). within broad relation types, we see similar results across embeddings, with a few exceptions (the static gpt-2 embeddings are poor throughout, but much better than the others on derivational morphology). consistent with previous results using the arithmetic analogy test, derivational morphology and encyclopedic and lexical semantic relations obtain lower scores than inflectional morphology @xcite . the pcs scores are presented at right in figure 6 . the general picture is similar: there are systematic differences between relation types, and inflectional morphology tends to be well coded. however, contrary to the conclusions of the standard analogy test, the pcs reveals that numberbatch codes linguistic relations better than the other embeddings, which is more consistent with the evaluations presented in @xcite . the pcs also shows that derivational morphology occupies an intermediate position between inflectional morphology and semantics: word2vec, glove, and numberbatch in fact code derivational morphology better, in the pcs sense, than encyclopedic semantic relations. the elevated analogy test performance of some embeddings on encyclopedic relations is likely an artefact of the kind of category-level geometric coherence discussed in section 5. as we show in the appendix in ## contributions we have made new arguments against the use of the standard arithmetic analogy test in the evaluation of word embeddings. we show in detail how regularities in vector offsets, which are the object of primary interest for the analogy test, are, on the one hand, washed out by within-pair similarity, and, on the other, insufficient to establish the existence of linguistic regularities. we explain the previously observed phenomenon that arithmetic analogy tests consistently predict the input words. we propose a new measure of the presence of linguistic relations in word embeddings, the pairing consistency score, which measures the degree to which offsets between related pairs are parallel above chance, and the offset concentration score, a complementary measure of the absolute degree of parallelism. we show that a variety of word vectors really do capture linguistic regularities, in spite of distortions introduced by the arithmetic analogy test. we show that these distortions lead to spurious conclusions when comparing performance. arithmetic analogy tests and loss functions based on them are deprecated and should be replaced by direct evaluation of pairing consistency. figure 7 : distribution of the cosine similarity of vector offsets to the mean direction for the default word2vec embeddings for each category, then grouped by broad type. the mean similarity improves the similarity even for anisotropic offsets, as discussed. variance changes seem category dependent.