Scientific and Creative Analogies in Pretrained Language Models (2022)

Abstract: This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.

## introduction analogy-making is a cornerstone of human intelligence @xcite , allowing us to acquire new knowledge and creatively explore new concepts. according to @xcite 's structure-mapping theory, analogical reasoning is different from surface similarity. instead, the attributes of a familiar concept (the source domain) are mapped to something less familiar (the target domain) if their relational structures are similar enough. for example, while not directly similar in their attributes, the underlying relational structure of the solar system matches that of an atom. the relationship between two source domain attributes (e.g. sun and planet) helps us to understand that between their target domain counterparts (nucleus and electron). within natural language processing (nlp), the word analogy task @xcite , has been widely used to demonstrate analogical reasoning capabilities of pretrained word embedding models. the task involves solving analogies of the form a:b :: c:d (i.e., a is to b as c is to d) by exploiting (local) linear properties of word vectors (vector offsets). subsequently, word analogy became one of the standardized tasks for intrinsic evaluation of word embedding quality. however, @xcite showed that the vector offset method was not sufficient for most types of analogical relations, and @xcite pointed out shortcuts that the models were taking. existing word analogy datasets focus on a limited set of analogical relations, include words that are semantically similar and do not require the model to relate distinct concepts via systematic comparison of their relational structures, all of which is necessary for humanlike analogy making. in parallel, the field has seen the development of large-scale pretrained sentence encoders, whose analogical reasoning capabilities have not yet been fully tested. to address these issues, we devise and release a new dataset -the scientific and creative analogy dataset (scan) -comprising holistic analogies between concepts from semantically distant domains. it draws on metaphorical and scientific analogies. resolving these analogies requires the models to identify systematic ontological correspondences between two distinct semantic domains, such as in the solar system -atom example. our contributions are threefold: 1) we present the scan analogy evaluation task and dataset, which we make publicly available to the research community; 2) we systematically evaluate current state-of-the-art lms on the established bats dataset @xcite , which consists of a large number of traditional word analogies, as well as the novel scan dataset. we show that in the latter case the models exhibit severe limitations in understanding analogies; 3) we show that a high performance on bats is not indicative of how well the models solve the complex scan analogies, supporting our hypothesis that bats does not require full analogical reasoning. transformer language models such as bert @xcite have pushed the state-of-theart on a number of nlp tasks. but since 3cosadd cannot easily be applied to them, due to their word embeddings not being fixed but dynamically calculated, their analogical capabilities have not been investigated much. while some headway has been made in that regard @xcite zhao (2020), zhu and de melo (2020), ushio et al. (2021) ), the focus has been on transferring the traditional word analogy datasets to the sentence level. this does, however, also transfer their limitations. in general, the low performance of models in @xcite and @xcite suggest that completing word analogies is challenging for state-ofthe-art lms, even when structure mapping across distinct domains is not explicitly tested. ## scan dataset our dataset contains 449 analogy instances, clustered into 65 full concept mappings. a source concept is mapped to a target concept along with a number of related attributes. scan offers richer and more holistic analogies than traditional word analogy datasets. taking a statistical view, the chances of the words in the source and target domains co-occurring are much lower than in bats. for example, countries and their capitals, animals and the sounds they make, and most grammatical analogy types in bats are quite likely to occur in the training corpus together. however, the same cannot be said for most scan analogies, meaning that true analogical transfer needs to occur. additionally, the in-domain words in scan are semantically more dissimilar. for example, in the argument domain, debater and topic are fundamentally different concepts. in bats, on the other hand, every domain member is another instance of the same concept, e.g. france and germany instances of a country. lastly, the analogical relationships between domains in scan are more abstract than those in bats. to successfully extract the same relationship from solar system -atom and planet -electron, more abstraction and inference over the relational structure of the domains is needed than in bats. in bats, the relationships between, e.g. france -paris and germany -berlin, are straightforward and do not require much abstraction. overall, scan offers more human-like analogies by employing more diverse in-domain words, more abstract mapping relations and by avoiding obvious co-occurrences. due to its full-concept mappings, scan is not confined to the word analogy task. by holistically mapping entire source domains to a new target domain we want to encourage a broader range of analogy representations. ## models we probe the analogical capabilities of several widely used language models: gpt-2, bert and multilingual bert (m-bert). we use glove as a baseline, given it has been shown to outperform language models on some relation types in previous analogy tasks @xcite . gpt-2 @xcite can be viewed as a "true" lm since it is trained to predict the next word in a sequence, and can be used for language generation. it is a transformer-based model with 48 layers and 1542m parameters, trained on a custom dataset, webtext, created only from outbound links from reddit to improve text quality. due to its predictive nature, gpt-2 is one-directional, i.e. only the context on the left-hand side influences the prediction of the next word. bert @xcite is a bidirectional language representation model. it is trained with two objectives: masked-token prediction and nextsentence prediction. we use bert-base with 12 layers in our experiments (110m parameters). since bert is bidirectional, it can incorporate information from both sides of the masked token. m-bert is a bert model, trained on a wikipedia dump of 100 languages. the model performs best on high-resource languages such as english, french and chinese, since lower-resource languages are underrepresented in the training data. we test whether m-bert's pre-training on a wide range of languages, and thus a wide range of culture-specific analogies, might enhance the model's general analogy understanding. ## experiments setup we use pretrained model instances of gpt-2, bert base and multilingual bert foot_0 . as bert and gpt-2 are trained on full sentences, we insert the word analogy quadruple into a placeholder sentence. we use "if a is like b, then c is like d.", which was selected from a set of candidates as it performed best on the development set. similarly to ettinger (2020), who probed bert with a number of cloze and negation tasks, the models need to predict the last token of the sentence. we force the models to predict word d by either masking it for the two bert models, or by cutting the sentence off before it for gpt-2. we report the mean reciprocal rank (mrr) of the first token of the target word (or that of one of the alternative answers) in only the top 10 predicted tokens to reduce compute. if the label is not in the top 10 tokens, its rr is 0. model performance in terms of accuracy, recall@10 and recall@5 is reported in the appendix. we use an nvidia 16gb gpu. scan vs. bats to evaluate how well the models can solve different types of analogy, we test them on bats in addition to scan. we do not fine-tune the models. bats consists of 98000 examples of balanced relations. there are four main relations -inflectional and derivational morphology, and lexicographic and encyclopedic semantics -each of which consists of ten subcategories. for some examples, multiple correct answers are listed. zero-shot vs. one-shot previous work on analogical reasoning in gpt-3 @xcite has shown that when the model is given a full example of a word analogy in addition to the incomplete one, the performance on the incomplete analogy substantially increases. we see this as a form of one-shot vs. zero-shot testing and also test the models this way, investigating whether this has an impact on the lms' bats scan science meta. glove .099 .022 .099 .006 gpt-2 .098 .057 .073 .054 bert .207 .044 .092 .034 m-bert .205 .041 .088 .031 training set effects lastly, we further investigate the difference between the types of word analogies in bats and those in scan. we split the bats dataset into a train, validation and test set (70/15/15 ratio), ensuring that each word pair appears in only one of them. we fine-tune the lms on the training set and take each model's version with the best score on the bats validation set. we train (∼ 4h) all models with the adamw @xcite optimizer, a learning rate of 5e -5 and a batch size of 16 for 4 epochs (based on manual tuning). if the model has learned about general analogy-making it must understand new analogical relations "on-the-fly" and improve not only on the bats test set but also on scan. we expect there to be strong improvements on bats compared to its untrained counterpart, but little to none on scan, showing them to be inherently different. this is not performed on glove, since 3cosadd outputting an embedding is not part of its original training setup. ## results & discussion zero vs. one-shot training set effects after training on bats, one could expect that if the models learn about analogical reasoning in general, they would also naturally do better on the scan dataset with more complex analogies. however, our results in error analysis while glove scores consistently on all relation types in bats, this is not the case for the other models. on scan, none of the models predict the mappings of all attributes of a concept (or even most of them) correctly. while the models are able to solve some individual mappings, the fact that they cannot apply this to all aspects of the concept indicates that none of them are really able to grasp the workings of analogy. in cases where analogies remain entirely unsolved, it is likely that the required domain knowledge is lacking. ## conclusion analogical reasoning remains a challenging task even when state-of-the-art transformer lms are used. we have shown that, even with models such as bert and gpt-2, there is large room for improvement on automated reasoning and understanding of realistic analogies. we have introduced a new dataset, scan, that is different from existing word analogy datasets in that it is composed of whole concept mappings across semantically dissimilar domains, demonstrating that popular lms are unable to fully understand these analogies. we further tested whether a full example of the task can help the models, finding that this is not helpful in our setup. lastly, our results indicate that the scan analogies are substantially different from those of traditional word analogy datasets. improving on them is a line of research we wish to investigate further in the future. we make scan and the related code publicly available. foot_1 8 limitations our experimental design focuses on evaluating the models' analogical capabilities in a generative setting. we see value in this, as analogical reasoning is inherently a generative cognitive function. the bert models are, however, not trained to perform left-to-right generation and, furthermore, rely on wordpiece vocabulary for tokenization. the evaluation of its predictions in the analogy task is, therefore, less straightforward and not exactly comparable to other models. we adapt the task for bert models by letting them only predict the first token of the missing answer. comparing only the first token leaves some variability, however, when matching the prediction and the right answer. we expect this effect to be limited in english due to its sparse morphology. furthermore, the metaphorical analogies come from english literature and cultural background. it would be interesting to compare these with analogies from other languages and cultures to investigate whether the language models' lack of understanding is due to encoding of language-specific properties, missing domain knowledge or the general analogical mapping abilities. lastly, some metaphors in scan exhibit antiquated gender roles, e.g. the metaphor "government:household :: governor:father". while these relationships are culturally often still relevant for metaphor understanding, the underlying implied gender roles need to be treated carefully and issues of their encoding by neural models investigated further.