Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance (2023)

Abstract: While analogies are a common way to evaluate word embeddings in NLP, it is also of interest to investigate whether or not analogical reasoning is a task in itself that can be learned. In this paper, we test several ways to learn basic analogical reasoning, specifically focusing on analogies that are more typical of what is used to evaluate analogical reasoning in humans than those in commonly used NLP benchmarks. Our experiments find that models are able to learn analogical reasoning, even with a small amount of data. We additionally compare our models to a dataset with a human baseline, and find that after training models approach human performance.

## introduction solving proportional analogies a : b :: c : d (ex. paris:france::tokyo:japan) @xcite with embeddings has become an iconic way to demonstrate that they encode semantic information regarding the relationships between word pairs. proportional analogies have been formalized with word embeddings by means of a simple arithmetic equation (often referred to as the vector offset method) ba + @xmath0, d), or other slightly more complicated similarity measures @xcite @xcite . this task has developed into many datasets such as the google analogy test set and the bigger analogy test set (bats) @xcite . examples of relation types included in these datasets are morphological in nature (entertain : entertainer :: examine : examiner), based on encyclopedic knowledge (p aris : f rance :: lima : p eru), or lexicographic relations, such as hypernyms and hyponyms (candy : lollipop :: color : white). notably, the relations between entities that form each analogy are explicitly verbalized for each example, and are grouped into collections of many pairs with equivalent relations. while the ability of word embeddings to solve these sorts of analogies is interesting, the analogies contained in these datasets are different from what is typically used to test analogical reasoning in humans. many would be trivially easy to solve, as is the case with morphological relations and a lot of encyclopedic knowledge @xcite . even the task of completing an analogy by filling in a correct d term is perhaps not well suited to humans. @xcite noted that analogy questions are often given to people as multiple choice questions, as people would likely fill in d with a variety of words where a single correct answer is not always the case. it has also been pointed out that model performance on analogy tests tend to rely on how semantically related all the entities in the analogies are to each other, suggesting these tests might in fact not be measuring analogical reasoning @xcite . when using the vector offset method, the a, b, and c terms are generally excluded from candidates for d. if they are not excluded b or c is often the closest neighbor to the estimate of d and will be selected @xcite . analogical reasoning has the potential to extend beyond just solving proportional analogies. analogies and analogical reasoning have value regarding scientific discovery, problem solving, human creative endeavors such as metaphor generation, as well as in the field of education @xcite @xcite @xcite . in this work, we do not focus on the semantic/morphological analogies in datasets such as bats, but on more "complex" analogies that are closer to what is used to test analogical reasoning in humans, what @xcite referred to as "psychometric analogy tests". most of the data we use has been developed precisely to test humans. furthermore, many of these analogies will have no verbalized relation between entities available, and there are likely no ways to group instances by type of relation without over-generalizing the relation. unlike most previous studies, we want to go beyond just exploring what pretrained language models know about analogical reasoning. our main contributions are as follows: we propose a training objective that allows language models to learn analogies. we find that learning is effective even with a fairly small amount of data, and approaches human performance on an unseen test set constructed for testing human analogical reasoning. lastly, we find that fine-tuning models with this training objective does not deteriorate performance on external, but related tasks. ## related work while a lot of work tends to focus on performance on datasets such as bats, there exist some datasets which were developed to test humans. @xcite explored the ability of word embeddings from various transformer models as well as some pretrained word embeddings (such as glove) to solve a variety of analogies originally designed to test humans in addition to the bats and google analogy test set. they tested three different scoring methods that aggregated across all valid analogy permutations. their experiments found that the analogies designed for humans were much harder for all word embeddings to solve, with the better performing models' accuracy being less than 60% on datasets designed for humans, while the highest accuracy attained on google and bats was 96.6% and 81.2% respectively. @xcite introduced a dataset composed of scientific and metaphorical analogies, called scan, also testing performance on transformer based embeddings by using a natural language prompt, and filling in an ending [mask] token. they experimented with a zero and oneshot learning setting, and additionally fine-tuned models on the bats dataset. overall they found that performance on the scan was low, and that models which had been fine-tuned on bats lead to a decreased performance on the scan dataset, which the authors attribute to the types of analogies being inherently different and that datasets such as the bats did not well represent how humans utilize analogies. in this work, we test a novel training objective to explore whether analogical reasoning in itself is a task that can be learned, and later test to unseen analogies with non-equivalent relation types. there have been a few works in addition to @xcite that attempted at actively learning analogies or relations between words @xcite using language models. @xcite and @xcite , handle the analogy problem in the more common way of predicting a word to complete the analogy. our training focuses more on similarity in relations between entities, as opposed to similarity between entities in the analogy themselves, and uses this objective to fine-tune a pretrained bertbase model to solve analogies as a classification problem. @xcite proposed relbert, whose motivation and training scheme is most similar to our set up in that they are focused on relations and relational similarity, even if their training task is not formulated as an analogy problem specifically. however their methods involve prompting and their training data was much larger (semeval 2012 task 2) and involved only lexical relations. our work involves an arguably simpler training scheme, with a much smaller training dataset that includes a wider variety of analogical relations. additionally, there has been a lot of work in knowledge graph representation to learn embeddings for relations between entities, some of these which explicitly utilize analogical structure to learn embeddings @xcite . however, this area of research is out of scope as we are focusing specifically on learning to represent knowledge and relations in contextual language models. ## methods all our experiments used a pretrained bert-base uncased model (110m parameters) foot_0 @xcite unless stated otherwise. models are trained on a binary classification task-when given an analogy, they must label it as either positive or negative. we evaluate our models in three ways: 1) we evaluate the models' ability to learn analogy classification using two variants of a classification task, the binary classification task used during training and a ranking task, where we present them with both a positive analogy and its negative counterpart (how these are created is described in the datasets section below), much like a multiple choice question with two possible answers. whichever pair the model scores as more likely an analogy is the one the model "chooses". we chose to include this task as this is how it was formulated for the human baseline we use to compare our models, and is closer to how analogy tasks are often presented to humans @xcite . unlike the binary classification task, which will rely on a cutoff chosen to differentiate positive from negative, the decision made using ranking is relative to what a specific analogy is being compared to. 2) we compare the models' ability to solve analogies to a human baseline on an unseen dataset (also using ranking as described in the previous paragraph). ## conclusion, limitations and future work in this paper, we aimed to move from testing relatively simple analogical relations in pretrained language models to testing the ability to learn more complex relations used for testing human analogical reasoning, with a tailored training objective. we found that overall, analogies are something that can be learned. we reach an accuracy of 0.69 up from 0.53 while being 0.15 below the human upper bound, on an unseen test set constructed to test human analogical reasoning. lastly, we find that fine-tuning models with certain training objectives generally does not deteriorate their performance on external, but related tasks. in fact, on some tasks we observed improved accuracy. our experiments involve several limitations. for one, the dataset is small, making claims that analogical reasoning is something that for sure can or cannot be learned with language models is not possible. another important consideration is that analogies are permutable @xcite . given an analogy (1) a : b :: c : d, the following analogies also hold: (2) b : a :: d : c, (3) c : d :: a : b, (4) d : c :: b : a, (5) a : c :: b : d, (6) c : a :: d : b, (7) b : d :: a : c, (8) d : b :: c : a. our a-b models account for 1-4, while our a-c models account for 5-8. these permutations are not without criticism -specifically (5) a : c :: b : d and all its derivatives (6-8) @xcite . consider the analogy (electron : nucleus :: planet : sun). in our a-b models, we are making the assumption cos((a-b),(cd)). in natural language, the relation on either side could be verbalized as revolves around. in the measure cos((a-c),(b-d)), there is no verbalizer that can describe both the a,c and b,d equivalently. the question is if that corresponds to no vector transformation that is equivalent between the two pairs. lastly, @xcite mentioned some of the metaphorical analogies contained antiquated gender roles, which could be potentially harmful. one direction for future work is to address limitations of word embeddings or their representation power, such as the anisotropy that exists among contextual word embeddings @xcite . perhaps a distance measure between two entities that is not subtraction would be a better way to represent their relation. additionally, since prior work has suggested that analogies where the entities are close to each other in space are generally easier to solve with the vector offset method, perhaps focusing on incorporating knowledge regarding semantic distance between entities during training would be helpful @xcite . we would also like to explore whether augmenting lm training with analogy learning for other common nlp benchmarks affects performance on these benchmarks.