BERT is to NLP what A lex N et is to CV : Can Pre-Trained Language Models Identify Analogies? (2021)

Abstract: Analogies play a central role in human commonsense reasoning. The ability to recognize analogies such as “eye is to seeing what ear is to hearing”, sometimes referred to as analogical proportions, shape how we structure knowledge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the language model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using benchmarks obtained from educational settings, as well as more commonly used datasets. We find that off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations, and results are highly sensitive to model architecture and hyperparameters. Overall the best results were obtained with GPT-2 and RoBERTa, while configurations using BERT were not able to outperform word embedding models. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations.

## introduction one of the most widely discussed properties of word embeddings has been their surprising ability to model certain types of relational similarities in terms of word vector differences @xcite while the title is probably self-explanatory, this is a small note explaining it. bert is to nlp what alexnet is to cv is making an analogy on what the bert and alexnet models represented for natural language processing (nlp) and computer vision (cv), respectively. they both brought a paradigm shift in how research was undertaken in their corresponding disciplines and this is what the analogy refers to. an example of such a problem is shown in given the central role of analogy in human cognition, it is nonetheless important to understand the extent to which nlp models are able to solve these more abstract analogy problems. besides its value as an intrinsic benchmark for lexical semantics, the ability to recognize analogies is indeed important in the contexts of human creativity @xcite , innovation @xcite , computational creativity @xcite and education @xcite . analogies are also a prerequisite to build ai systems for the legal domain @xcite and are used in machine learning @xcite @xcite and for ontology alignment @xcite , among others. within nlp, however, the task of recognizing analogies has received relatively little attention. to solve such problems, @xcite proposed latent relational analysis (lra), which was essentially designed as a relational counterpart to latent semantic analysis @xcite . somewhat surprisingly, perhaps, despite the substantial progress that word embeddings and language models (lms) have enabled in nlp, lra still represents the current state-of-the-art in solving abstract word analogy problems. when going beyond a purely unsupervised setting, however, gpt-3 was recently found to obtain slightly better results @xcite . the aim of this paper is to analyze the ability of pre-trained lms to recognize analogies. our focus is on the zero-shot setting, where lms are used without fine-tuning. to predict whether two word pairs (a, b) and (c, d) are likely to be analogical, we need a prompt, i.e. a template that is used to construct the input to the lm, and a scoring function. we extensively analyze the impact of both of these choices, as well as the differences between different lms. when the prompt and scoring function are carefully calibrated, we find that gpt-2 can outperform lra, standard word embeddings as well as the published results for gpt-3 in the zero-shot setting. however, we also find that these results are highly sensitive to the choice of the prompt, as well as two hyperparameters in our scoring function, with the optimal choices not being consistent across different datasets. moreover, using bert leads to considerably weaker results, underperforming even standard word embeddings in all of the considered configurations. these findings suggest that while transformer-based lms learn relational knowledge to a meaningful extent, more work is needed to understand how such knowledge is encoded, and how it can be exploited. ## word analogies in this section, we describe the word analogy formulation that is used for our experiments (section 3.1). subsequently, we provide an overview of the datasets used in our experiments (section 3.2). ## methodology in this section, we explain our strategy for using pretrained lms to solve analogy problems without fine-tuning. first, in section 4.1 we explain how each relation pair is converted into a natural sentence to be fed into the lm. in section 4.2, we then discuss a number of scoring functions that can be used to select the most plausible answer candidate. finally, we take advantage of the fact that analogical proportion is invariant to particular permutations, which allows for a natural extension of the proposed scoring functions (section 4.3). figure 1 shows a high-level overview of our methodology. ## evaluation in this section, we evaluate language models on the five analogy datasets presented in section 3. ## analysis we now take a closer look into our results to investigate parameter sensitivity, the correlation between model performance and human difficulty levels, and possible dataset artifacts. the following analysis focuses on s mppl as it achieved the best results among the lm based scoring functions. parameter sensitivity we found that optimal values of the parameters α and β are highly dependent on the dataset, while other parameters such as the template type t vary across lms. on the other hand, as shown in figure 3 , the optimal permutations of the templates are relatively consistent, with the original ordering a : b :: c : d typically achieving the best results. the results degrade most for permutations that mix the two word pairs (e.g. a : c :: b : d). in the appendix we include an ablation study for the sensitivity and relevance of other parameters and design choices. ## conclusion in this paper, we have presented an extensive analysis of the ability of language models to identify analogies. to this end, we first compiled datasets with psychometric analogy problems from educational resources, covering a wide range of difficulty levels and topics. we also recast two standard benchmarks, the google and bats analogy datasets, into the same style of problems. then, we proposed standard techniques to apply language models to the unsupervised task of solving these analogy problems. our empirical results shed light on the strengths and limitations of various models. to directly answer the question posed in the title, our conclusion is that language models can identify analogies to a certain extent, but not all language models are able to achieve a meaningful improvement over word embeddings (whose limitations in analogy tasks are well documented). on the other hand, when carefully tuned, some language models are able to achieve state-of-the-art results. we emphasize that results are highly sensitive to the chosen hyperparameters (which define the scoring function and the prompt among others). further research could focus on the selection of these optimal hyperparameters, including automatizing the search or generation of prompts, along the lines of bouraoui et al. (2020) and shin et al. (2020), respectively. finally, clearly lms might still be able to learn to solve analogy tasks when given appropriate training data, which is an aspect that we leave for future work.