ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models (2023)

Abstract: Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts. In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity – (i) word, (ii) word vs. sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using thirteen datasets and three different distance measures, we evaluate the abilities of eight LLMs in identifying analogical pairs in the semantic vector space. Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy.

## introducing analogial -a benchmark for analogy the ability of humans to perceive a situation in one context as similar to that in a different context is known as analogy-making. it is considered to be a central component of human cognition and learning. analogy-making has received attention from a broad audience, including cognitive scientists @xcite , linguists @xcite , and educators @xcite during the last several decades. current neural network-based word embeddings are primarily influenced by the distributional hypothesis "you shall know a word by the company it keeps" @xcite . however, only a few investigations have been done on the capabilities of llms to draw analogies between long text @xcite . for example -embeddings of sentences 'i can speak two languages.' and 'i am bilingual.' should be closeby in vector space and 'i like chocolate.' and 'i do not like chocolate.' should not be close-by. performance evaluations of modern llm are driven mainly by extrinsic measures based on benchmarks such as glue @xcite , and superglue @xcite . we take this opportunity to introduce a new benchmark to intrinsically evaluate llms using analogies consisting of long text (sentences, paragraphs). we hypothesize that an llm should be able to organize the semantic vector space so that analogical lexical pairs are closer to each other (see figure 1 ). in this paper, we introduce analogical -a benchmark based on an analogy taxonomy consisting of six levels of analogy complexity -(i) word level, (ii) word vs. sentence level, (iii) syntactic level, (iv) negation level, (v) semantic (entailment) level and (vi) metaphor level. we proxy analogy complexity with the length of lexical items compared. we derive five and identify eight datasets at each level of the analogy taxonomy. euclidean distance and cosine similarity are the de facto standards for capturing analogy in the nlp community. we show that, in contrast, such measures suffer from several correlations and indirect dependencies among the vector dimensions. finally, we argue and empirically report that mahalanobis distance @xcite better captures the semantic equivalence in high dimensional vector spaces. ## related work in this section, we elaborate on previous work on analogy identification, the background of encoderbased language models and distance measures used in analogy-based comparisons in nlp. there have been previous work on analogy identification by turney ( 2008 current neural network-based lms play a pivotal role in the present-day nlp landscape by performing exceptionally well in numerous nlp tasks such as machine translation @xcite , classification @xcite , and sentiment analysis @xcite . these lms are trained on large, heterogeneous text corpora resulting in pretrained lms that are then used on downstream tasks via supervised fine-tuning. this work uses the pretrained lms in a zero-shot setting for embedding creation. previous research in nlp has used cosine distance/ similarity, euclidean distance and mahalanobis distance as popular distance measures to quantify the semantic similarity between text @xcite @xcite . even though mahalanobis distance has been popularly used to measure the distance between a sample and a distribution, it has been increasingly used to measure the distance between two samples in a dataset @xcite . this work extends these distance measures to measure the analogy between two lexical items. ## analogical -six levels of analogy analogical is a comprehensive benchmark focusing on six distinct categories of analogies organized within a taxonomy. these categories are determined based on the level of complexity they pose for current llms. even though current language models perform exceptionally well on tasks that involve pattern recognizing the underlying text distribution and learning to infer correlations, they struggle with complex and intricate tasks such as basic symbol manipulation @xcite , compositionality @xcite , and appropriating commonsense knowledge @xcite . in higher levels of this taxonomy, the lms are required to identify analogies between long and more abstract texts and, when doing so, have to face the complexities highlighted above. in the next section, we formally introduce the analogy taxonomy and the datasets representing each level in the taxonomy. analogies are often expressed as an explicit or implicit relational similarity, involving two main lexical items. in this work, these two lexical items vary from single words to word phrases or sentences. more formally, we denote analogy as x ∶∶ y , where x and y are the two lexical items and analogy is a symmetric relation. the taxonomy of analogy is divided into six levels (see figure 2 ) where complexity is increased from bottom to top. in this section, we identify and introduce different datasets corresponding to each level of complexity in the analogy taxonomy that can be used to evaluate the performances of several sota language models. ## large language models to evaluate analogical modern llms are built upon the transformer architecture @xcite . the llms we use in this study fall into two classes based on their training objective. masked language models (mlms) are trained to predict randomly masked tokens (random words replaced by a [mask] token) based on all the other words present in a sequence in a bidirectional manner. mlms use the encoder portion of the transformer architecture. encoder-decoder language models(edlms) build upon the entire encoder-decoder architecture of transformers and are trained by predicting the original sequence of text given a corrupted version of the text sequence. in the current empirical study, we examine the performance of eight popular pretrained language models on identifying analogies introduced in the analogy taxonomy without fine-tuning (zero-shot setting). we choose six mlm-based llms, namely (i) bert @xcite , (ii) roberta @xcite , (iii) albert @xcite , (iv) linkbert @xcite , (v) spanbert @xcite , and (vi) xlnet @xcite , t5 @xcite , an encoder-decoder-based model, and electra @xcite an llm with two transformers, one as a generator and the other as a discriminator. we include further details of these llms in appendix c). ## distance measures and their importance previous work @xcite used static word embeddings with vector offset approaches (such as 3cosmul, 3cosadd) to identify word analogies. in this work, we use the distance between the lexical items in a high-dimensional vector space to identify the analogy between two lexical items. we identify three distance measures, namely, cosine distance (cd), euclidean distance(ed), and mahalanobis distance(md). next, we briefly explain md. cd and ed are explained in the appendix. ## experiment settings we have set up comprehensive experiments across eight llms, thirteen datasets, and three distance measures adding up to 312 (8×13×3) experiments. we analyze the performance of llms across the analogy taxonomy by comparing the normalized distance measures. we present the complete results table for all the experiments in appendix a). the embedding (representation) of each lexical item in an analogical pair (word embedding, sentence embedding) is extracted from eight lms (in this work, we use the simplest representation, which is the [cls] token representation). the distance measures between these two representations are then calculated using ed, cd, and md. for each dataset containing analogical pairs, these distance measures are calculated, and the mean of all the data points of a dataset is considered the representative distance for that dataset (these distances are min-max normalized). given the analogy taxonomy (figure 2 ), except for the negation dataset at level 4, all the other datasets are positive analogies, meaning, that the two lexical items of a data point are considered analogical to each other. therefore the mean distance values of these datasets should indicate such similarity (low cosine, euclidean, and mahalanobis distances). for the negation dataset, the two lexical items in a data point should not be analogical to each other. therefore, the representative distance measures should be large. we discuss the implementation details in appendix d. ## conclusion & future avenues this work introduces analogical, a benchmark for llms based on a taxonomy of six levels of analogies. through comprehensive experiments, we show that llms increasingly struggle to identify analogies when the complexity of analogies increase (going up the analogy taxonomy). the datasets derived for level three are crude at this time. in the future, we will incorporate more challenging and comprehensive datasets to this level. we also will move on from this empirical study to investigate why some llms perform well at specific levels and not others.