„Mann“ is to “Donna” as「国王」is to « Reine » Adapting the Analogy Task for Multilingual and Contextual Embeddings (2023)

Abstract: How does the word analogy task fit in the modern NLP landscape? Given the rarity of comparable multilingual benchmarks and the lack of a consensual evaluation protocol for contextual models, this remains an open question. In this paper, we introduce MATS: a multilingual analogy dataset, covering forty analogical relations in six languages, and evaluate human as well as static and contextual embedding performances on the task. We find that not all analogical relations are equally straightforward for humans, static models remain competitive with contextual embeddings, and optimal settings vary across languages and analogical relations. Several key challenges remain, including creating benchmarks that align with human reasoning and understanding what drives differences across methodologies.

## introduction ever since the work of @xcite , analogy solving has been a staple of public outreach in nlp: it has been featured both in science communication foot_0 and in the classroom. 2 this task consists in finding a target word b 2 , given a cue word b 1 it is related to, and another pair of words a 1 and b 2 that express the same relation. for example, we can ask what is the word that relates to "king" in the same manner that "woman" relates to "man": this target ought to be "queen". the introduction of pre-trained contextualized embeddings @xcite opened up a new research area where to expand prior knowledge about static models. this includes the analogy task. suggestions have been put forward as to how to best adapt it: @xcite propose to use a prompt-based method, whereas @xcite and @xcite try to derive static embeddings from bert to fall back on the algorithm of @xcite . however, much work remains to be done to properly contrast and compare the performance of contextual and static embedding models on the analogy task. another observation to be made is that reliable comparisons across languages are rare. on the one hand, datasets for english-such as the gats @xcite and bats @xcite benchmarks-have been adapted or translated to a wide variety of languages. on the other hand, approaches specifically focusing on establishing multilingual comparisons are, to our knowledge, limited to @xcite , @xcite , and @xcite -none of which considers contextual embeddings. how do embeddings-and in particular contextual models-perform on the analogy task beyond english? in the present paper, we argue that a principled approach to comparing embeddings on the analogy task across languages consists in creating resources designed to be directly comparable. the most natural way of achieving this is by relying on manual translations, so as to retain a certain degree of control on the output quality and to produce resources that are maximally comparable. given the weaknesses of gats outlined by @xcite , the more reasonable starting point for these translations would be the bats dataset. these considerations effectively rule out the only similar dataset that we know of, by @xcite , where analogies accept only one valid answer, as in gats. to that end, we introduce mats, a multilingual analogy test set for six languages: dutch, french, german, italian, mandarin, and spanish, derived from the original bats dataset of glad-kova et al., spanning across 40 analogical relations equally partitioned between inflectional, derivational, lexicographic and encyclopedic. using this new benchmark, we observe that different adaptations of the analogy task to mbert contextual embeddings need not yield comparable results: not only do we observe different performances when deriving static embeddings from contextual models and when using prompts, we also see that the exact wording of the prompt significantly impacts the model's behavior. we also share some anecdotal evidence questioning the validity of approaches to this task that assume there is a single gold answertrained linguists attempting to solve this task often provide answers that do not match any of the expected targets, which further validates that singletarget analogy benchmarks are ill-suited. two groups of related analogy datasets are often cited: those adapted from gats (google analogy test set, @xcite and those derived from bats (balanced analogy test set, @xcite . the latter distinguishes itself from the former on two major characteristics: first, it is designed for a balanced assessment of performances on the analogies and covers a larger collection of analogical relations; second, it admits multiple valid answers whenever relevant. these differences aim to mitigate some of the flaws @xcite perceived in gats: the emphasis of this dataset on balance is intended to provide a more accurate picture of a model's capabilities when it comes to word analogy solving, and the inclusion of multiple answers aims to mitigate the impact of spelling variation and dataset limitations. datasets similar to bats exist in japanese and icelandic @xcite , whereas gats has been translated in portuguese, hindi, french, polish, and spanish @xcite @xcite . other independently constructed datasets do exist (e.g., @xcite )crucially, covering all languages of interest to this study: in chinese @xcite @xcite , dutch @xcite , english @xcite , french @xcite , german @xcite , italian @xcite , and spanish @xcite . on the other hand, these resources were created by different research groups and may contain items that are not easily comparable or of lesser quality. 3 similar to our approach, @xcite and @xcite both conduct multilingual comparisons of word embeddings on the analogy task, whereas @xcite study how analogies behave under cross-lingual mappings. all three works rely on gats-style benchmarks (where only one valid target is admissible for each analogy relation); all are more limited in the scope of analogies they cover than bats-style datasets; none study how contextual embeddings fit in this picture. this last point is partly due to the initial conception of the task for static models: plenty of works discuss why static models develop linear analogies @xcite @xcite similar evidence has yet to emerge for contextual models. as such, some studies delineate its relevance to static embeddings (e.g., @xcite , but it has been adapted to contextual models @xcite @xcite . ## the multilingual analogy test set to study how analogy fares in a multilingual context, we introduce a multilingual analogy test set (mats), adapted from bats @xcite for dutch, french, german, italian, mandarin, and spanish. this analogy benchmark is structured in two tiers: individual sub-categories instantiating specific analogical relations (e.g., countrycapital) are grouped into four general categories, namely inflection, derivation, encyclopedia, and lexicography. the former two correspond to morphological relations, such as the relation between two inflected forms of a word or the relation between a verb and the corresponding agent noun. the latter two are more closely aligned to commonsense reasoning and include relations such as synonymy or the relation between the name of a coun-try and that of its capital city. the original resource by @xcite emphasizes balance by ensuring that each of the four super-sections contains exactly 10 sub-sections, and that each of the 10 sub-sections contains exactly 50 instances of the same analogical relation; analogy quadruples are created by exhaustively iterating across pairs of instances. this totals to 98,000 distinct analogy quadruplets to test models on, around five times as many items as what is mentioned in @xcite , and mitigates concerns of class imbalance. direct translations from the original bats were taken as starting points before performing language-specific adaptations (cf. infra); we refer the reader to @xcite for supplementary details. in all languages, unidiomatic direct translations and analogically invalid pairs were removed. multi-word expressions (mwe) were also removed, foot_3 before padding all categories except e03 to 50 pairs following the relation of each category. an overview of the outcome with examples and figures can be found in dutch the encyclopedic section e03 was localized using dutch provincies and their capital cities. french the inflectional section i03 was replaced with gender inflection of adjectives since comparatives are periphrastic constructions (e.g., jolie 'cute', plus jolie 'cuter'). the derivational section d01 was replaced with denominal adjectives using the suffix -el, as the formation of privatives using suffixes is not a productive morphological operation. the encyclopedic section e03 was localized using a random selection of 50 french départements and their capital cities, barring those that would be tokenized as mwe. ## setting baseline expectations we first focus on establishing the difficulty of our analogy benchmark, and how it compares to the english bats. we provide a human baseline and static embedding scores on mats. human performance one aspect rarely addressed in analogy benchmarks is that of how consensual and accurate they are. yet, some analogy relations are fundamentally debatable: for instance, whether "tonne" is to "kilogram" as "flower" is to "petal" depends on one's exact definition of a meronymic relation. 5 as such, the assumptions or intuitions of a given resource's designer may or annotators are provided with three of the four terms and ask them to propose a valid fourth term. we then measure (i) their accuracy on the task (i.e., the proportion of analogy items that were solved by the annotators with a valid fourth term in mats) and (ii) their agreement rate (i.e., the proportion of analogy items where the two annotators produced the same answer). results in this method consists in predicting as a target b * 2 the word w whose embedding w is the most codirectional to the offset-based approximation b 1 + a 2a 1 . the starting point of this approach is the assumption that for any two pairs of words instantiating the same semantic relation a 1 , a 2 and b 1 , b 2 , their corresponding embeddings should be related by means of a stable offset. in other words, we assume that there exists a vector x such that mixture of traditional and simplified characters. a 1 + @xmath0, or equivalently a 2a 1 = b 2b 1 , which we can reformulate to solve for b 2 as b 2 = b 1 +a 2 -a 1 . this method can therefore be seen as a direct assessment of whether analogical relations are encoded as stable offsets in the embedding space. in this work, we specifically rely on the vecto library implementation of 3cosadd. 10 results in figure 1 show that fasttext models perform better than conll-2017 word2vec models, confirming the known trend (e.g., @xcite . the noteworthy low performances on the l category across the board can be imputed to its lesser quality. in particular, fast-text models score much higher for i and d, the two categories with morphological relations, likely thanks to their learning of character n-gram representations rather than word type representationswhich makes fasttext models overall more in line with manual annotations. beyond these general observations, language also impacts the scores we observe. for instance, the high scores observed for english word2vec on the i category are never attested for word2vec models in other languages-which can be pinned on the rather simplistic inflectional system in english. both dutch models along with the conll-2017 french model perform surprisingly poorly. in the case of dutch, this is likely due to training data limitations: zeman et al. (2017) report training dutch models on fewer than 3b words, whereas all other languages were trained on over 5b words. what is novel beyond these replicated findings is the observation that humans also struggle with e and l analogies. this can account in part for the lower performances observed for these categories. this also suggests that more lenient benchmarks like bats, which allow multiple valid answers, are preferable to stricter ones, such as gats. ## analogies and contextual embeddings we now turn to benchmarking a contextual architecture, viz. uncased mbert @xcite . by definition, such architecture computes contextual representations of words: unlike static embeddings, contextual embeddings vary depending on the entire input sequence. the default use-case intended for these models pertains to token-level semantics-whereas analogy benchmarks evaluate word-type-level semantics. one word may have different meanings depending on context-depending on which context we use, results on the task may vary drastically. this complicates the use of these representations for the analogy task, by introducing the need of deriving some form of type-level judgment from token-level representations. static representations from mbert one possible approach to testing a contextual model on the analogy task consists in deriving word type representations from mbert, and proceeding as one would with static embeddings. to determine which word types we need vectors for, we construct reference corpora of 300m sentences per language sampled from oscar @xcite , and retrieve all word types with at least 50 occurrences. 11 all corpora were case-folded and tokenized using spacy. 12 for mandarin, we normalized all characters to their simplified form using opencc. 13 corpora statistics are shown in we experiment with layer pooling and two different means of deriving static word-type vectors. overall accuracy results are displayed in figure 2 ; results per category are available in appendix b, figure 7 . context-sample embeddings almost systematically outperform or equal the singleton approach for all layer groups and languages. mandarin performs surprisingly well, and scores for all languages on the l category are extremely poor. with singleton embeddings, lower layers tend to perform better, which matches with previous studies @xcite , but performances for mandarin are better when considering the embedding layer, whereas all other languages benefit most from pooling across the first four transformer layers. on the other hand, european-language context-sample embeddings yield their highest performances with middle or top layers. we suspect that mandarin has a very regular segmentation for d items, whereas latinalphabet languages may have different segmentations for otherwise regular suffixal construction, and therefore require some computation in order to properly reconstruct formal regularities. scores per category provided in figure 7 , appendix b confirm that much (almost all) of the performance attested 14 we choose 10 contexts in order to strike a reasonable balance between diversity of contexts and computational costs. prompt-based approaches contextualized embeddings can also be tested by converting the task to a prompt format. we draw inspiration from the methodology of @xcite , but frame our analogies as an unmasking task. we fill a three-slot template t that contains a mask with three given analogy cues a 1 , b 1 , and a 2 , and perform unmasking given the resulting sequence t (a 1 , b 1 , a 2 ). we measure a model's zero-shot accuracy by considering whether the unmasked word-pieces match with any of the listed valid targets' word-pieces. all relevant templates are listed in given the relative novelty of prompt-based approaches, we explore whether results are reliable across small changes of the prompts, such as the presence of quotation marks around analogy terms. results in figure 3 show that, besides english, performances are often lower than what we observed previously, and especially low on the i category. prompts only outperform static vectors on the l category, which we established to be less reliable. using quotes alleviates this trend, with a more pronounced effect on i and d. the higher english bats scores are likely due to the large proportion of english training samples in mbert. we also test how behavior changes across semantically equivalent templates, using four alternative german templates, along with the effects of enquoting analogy terms. these templates are listed in ## conclusions in this paper, we have presented a multilingual analogy test set, a resource five times larger than prior comparable datasets, with which we have looked at the analogy task in a multilingual context and studied how it fits in the modern nlp landscape. the dataset allows for a comparable multilingual evaluation of embedding models across a wide range of semantic analogy relations. manual evaluation showed that the quality of mats data in specific languages is comparable to the original english bats. we saw that not all analogy types are equally straightforward not only to computational models but also to humans, and that behavior on the task depends on the language, the embedding model, and the methodology involved. this also entails that static model behavior is not a reliable indicator of what contextual models might yield. we have been able to establish some trends across most of the methodological approaches we adopted here. in particular, from this work, we can outline three major conclusions. first, that not all categories are equally straightforward for humans (section 4); this also explains why lower performances are attested on semantic analogies across most of our experiments. second, that static models remain competitive with multilingual embedding models such as mbert (sections 4 and 5)-which replicates the conclusions of @xcite . third, that equally valid prompts can yield vastly differing results (section 5)-or more broadly, that different methodologies for adapting the analogy task to contextual embeddings can yield conflicting results. these conclusions also entail some practical guidelines for future work. in particular, there is a need to factor in human uncertainty as to what the correct target is; moreover, when adopting a prompt-based approach, testing a diverse array of prompts is necessary to properly establish how volatile a model's behavior is and how much variance in performance we should expect. as such, a number of key challenges remain in the field of analogy solving, such as devising benchmarks that more closely match human intuitions or providing an explanatory framework for the discrepancies observed across prompts and methodologies. there are other aspects we have left open, such as whether the analogy task is suitable for lexical semantic evaluation (cf. appendix c). we look forward to conducting future work in these directions, as well as expanding our observations to other architectures and methodologies. singleton embeddings, and optimal layer groups vary across languages and categories.