Learning Trajectories of Figurative Language for Pre-Trained Language Models (2025)

Abstract: Figurative language and figures of speech, such as metaphors and hyperboles, are used every day in written and oral communication among human beings. Nonetheless, this imaginative use of words in a non literal way requires a solid understanding of semantics and a deep real-world knowledge. In the longstanding debate about whether Neural Language Models (NLMs) really have a full understanding of text, analysing how they can recognise figurative language can provide some intuition of their functioning, their capabilities and their limits. Therefore, in this paper, we exploit probing tasks to study how several NLMs of different sizes recognise four different figures of speech: hyperboles, metaphors, oxymorons and pleonasms. We analyse whether this information is learned and how it is acquired during the training of the model, describing its learning trajectory. Moreover, we analyse which layers have a better comprehension of figurative language and the influence of pre-training data. Datasets and code are available at https://github.com/nicolarici/learning-trajectories .

## introduction investigating the capabilities of neural language models (nlms) is a well established research area in natural language processing (nlp) @xcite . major studies have been conducted on how these models understand grammar @xcite , how attention weights encode dependency relations @xcite and on world knowledge contained in such models @xcite . this is often done by probing tasks, i.e. by training a classifier to verify if the embedded representation of words (or sentences) produced by the models contain some specific information, such as the part-of-speech or whether a word describes the subject of the sentence @xcite . however, analysing semantic-related aspects is quite more complex @xcite . in fact, a word can have different meanings and nuances depending on the context in which it appears. moreover, it can be used literally, metaphorically, or even ironically. with respect to grammar and syntax, into which words have definite and recognisable characteristics which can be easily annotated, these nuances increase the difficulty of creating datasets and designing simple tasks for this kind of evaluation. in this paper, we tackle the problem on how nlms deal with semantics by analysing figurative language and, in particular, figures of speech. understanding figurative language is a challenging task in the overall context of text comprehension @xcite . as an example, consider the sentence "the real madrid players go 3,000 kilometers per hour". in order to understand that it contains a hyperbole, a lot of implicit semantic and real-world knowledge is needed: that real madrid players are humans, that kilometers per hour is a unit of measure of speed, and that humans do not reach that speed. since figures of speech rely on using words in a nuanced, rhetorical and imaginative way, investigating them can provide interesting insight into the semantic knowledge of nlms, their inner workings and some of their limits. from a practical perspective, understanding how this knowledge is acquired during training (in terms of quantity and types of data required) could provide some insights for optimizing the training process. we structure the recognition of the presence of a figure of speech (in particular, we analyse metaphors, oxymorons, hyperboles and pleonasms for the english language) in a sentence as a probing task, and we evaluate it through the state-of-the-art method of minimum description length (mdl) by @xcite . with this configuration, we can also study the performance of the different layers across the nlms architecture. most importantly, we are not only interested in the capabilities of the fully trained models. following the works in @xcite and @xcite , we also perform this analysis through the entire training process of an nlm, analysing how this semanticbased knowledge is acquired and how it evolves during training. by this procedure, we can also analyse how different data (a small general purpose wikipedia dump, the literary corpus project gutenberg and the extensive the pile dataset @xcite ) influence the learning of figurative language information. in summary, through this work we aim to address these research questions: q1. how do nlms learn figurative language? is there a common pattern or different figures of speech are learned in different ways? q2. do larger models have a better understanding of figurative language? q3. which layers are mostly associated with figurative language-related knowledge? q4. how is such knowledge acquired during training? how is it influenced by different data? ## related work alongside the success and diffusion of neural language models, one of the top research priorities became their explanation and interpretation @xcite . in this line of work, many approaches have been proposed, such as studying the behaviour of self-attention mechanisms @xcite ; assessing whether some forms of real-world knowledge are learned by the models @xcite , or analysing lexical and grammatical capabilities through probing tasks @xcite . a probe is a simple neural network model that uses the embedded representation of words or sentences generated by a pre-trained language model and that it is trained to solve a specific supervised task @xcite . this technique has been exploited to study syntactic properties @xcite including dependency parsing @xcite and temporal relations @xcite . differently from these works, our focus is not on syntax or grammar, but on semantics and, in particular, on figurative language. recent methods such as sparse autoencoders @xcite and sparse probing @xcite work in an unsupervised framework focusing on identifying the function of individual neurons (for instance, whether a neuron activates in the presence of words beginning with the letter w @xcite ). this is substantially different from our study, which analyses whether the whole word representation encodes a high level concept in a supervised manner. probing for figurative language has been studied by some works focusing on concepts like compositionality @xcite and idiomaticity @xcite . regarding figures of speech, which are the focus of our study, @xcite analyse how metaphors are recognised by several nlms across different datasets and languages. similarly, the work in @xcite analyses hyperboles on three different transformer-based encoder models. unlike these works, we do not focus on a single detection task; instead, we expand the scope to multiple figures of speech, including also oxymorons and pleonasms. moreover, they focus only on the fully trained model. instead, we analyse how these capabilities are learned by nlms during their training. understanding how and when a property is learned by an nlm is a relatively new and under investigated research field. @xcite investigated how linguistic properties are encoded in the hidden representations of a lstm language model. they discovered that syntactic features (such as the part-of-speech) are learned early in the training, whereas the learning of topicrelated information is acquired later. similar results were obtained by @xcite for the al-bert model @xcite . @xcite conduct a similar analysis of roberta @xcite learning trajectories focusing more on factual and common sense knowledge, finding that this type of information is learned more in depth as the training progresses. to the best of our knowledge, our work is the first study on how figurative language is learned during the model training. evaluating the effect of the pre-training on the behaviour of nlms is another underexplored field. primarily, the works by @xcite and @xcite evaluate how different sources (books, code repositories, web pages) influence the performance on several nlp downstream tasks. differently from these works, we analyse how different datasets change the learning trajectory of specific semantic knowledge through probing tasks. ## case studies and datasets we analyse four types of figures of speech: hyperbole, metaphor, oxymoron and pleonasm. in the following, we explain them in detail and introduce the datasets used for our analysis. all datasets are in english, publicly available and released freely for research purposes. hyperbole a hyperbole is an exaggeration to amplify or reduce the representation of the connotations of what is being communicated over a qualitative or quantitative scale @xcite . as a dataset, we used the one introduced by @xcite in the slightly modified version released in @xcite . it consists of 1396 sentences collected from the web and annotated by several human annotators. the dataset is divided equally between hyperbolic and literal sentences. metaphor following strapparava (2018), a metaphor can be defined as the replacement of one word by another one whose literal sense bears a resemblance to the literal sense of the word replaced. unlike a comparison, where affinities and divergences between the compared entities are explicitly shown, in a metaphor the two entities are merged into one. a well-known benchmark of manually annotated metaphors is the language computer corporation (lcc) by @xcite . more specifically, we adopt the binary version of lcc released in @xcite . from the 40k sentences contained in binary lcc, we randomly selected 1600 of them almost equally distributed (785 literals and 815 metaphors) to perform a better comparison with the other smaller datasets. oxymoron an oxymoron is a juxtaposition of two semantically opposite terms; the terms may be morphologically connected (as in happily unhappy) or not (as in screaming silence) @xcite . for our analysis, we created a new dataset starting from the 287 oxymorons collected and annotated by @xcite . while their dataset contains only positive instances (i.e. sentences containing an oxymoron), we generated the negative ones ourselves. the generation process exploits the gpt-4 model through the official openai api. for each oxymoron, we generated a sentence which contains the same words that compose the oxymoron but used in a literal meaning. then, the generated sentences were validated by the authors. more details about the prompt and the generation process are given in appendix a.1. the final dataset contains 1564 examples evenly distributed between positive and negative. pleonasm a pleonasm is an overabundant expression formed by the addition of one or more words that are not necessary from a grammatical or conceptual point of view. it is often used to provide the sentence with emphasis, confidence or verbosity @xcite . for our analysis, we exploited a dataset based on the semantic pleonasm corpus benchmark (spc) by @xcite . this dataset contains 3019 sentences with different pleonasms made by a pair of consecutive words. each instance has been labelled by human annotators that evaluate whether one of the two words is redundant, both words are redundant or none of them are. to adapt this dataset to our binary task, we discard the sentences labelled as both. our final dataset contains 3002 sentences, of which 1720 with a pleonasm and 1282 without. ## methodology in this section, we describe the method we use to analyse the learning trajectories of nlms applied to figurative language. ## experimental evaluation for our experimental evaluation, we considered several nlms (for the english language) and their available checkpoints, based on the gpt-neox architecture @xcite . we evaluate each model in terms of compression through the mdl method over all the available checkpoints. from a theoretical point of view, as described in section 4, mdl is the state-of-theart method for evaluating probing tasks, given its robustness versus randomness and control tasks. moreover, from a practical point of view, using accuracy leads to a definitely less clear evaluation. for instance, hyperbole accuracy ranges from 70% to 80% for almost the entire trajectory, with no real distinctions among models despite their much different size and number of parameters. instead, compressions lead to more stable and understandable results. for the engaged reader, results in terms of accuracy are reported in appendix c. we considered the pythia benchmark suite (biderman et al., 2023), i.e. gpt-neox models @xcite , a variant of gpt very similar to @xcite and gpt-j @xcite . all these models were trained over the pile dataset @xcite , for a total of almost 300b tokens. for each model, a checkpoint is saved after 0 (i.e. with the model weights randomly initialised), 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000 and every subsequent 1000 training steps. each step consists of 2m tokens. the models considered differ in terms of number of layers, heads, embedding dimension and therefore for their overall number of parameters. we selected the models with 14m , 31m , 70m , 160m , 410m , 1.4b and 6.9b parameters. 3 5.1 results for the last layer (q1, q2) in this section, we conduct a detailed comparison of the performance of gpt models, providing answers to q1 and q2. results are available in figure 1 , which shows the learning trajectories of the last layer for each model. considering the values of compression, on the y-axis of each plot, we can see notable differences among the figures of speech considered. those which are mostly recognisable by all the gpt models are oxymoron, which reaches a compression higher than 4.5 with the 1.4b model, and hyperbole, which reaches a compression higher than 2 with the 410m and 1.4b models. some difficulties can be observed for metaphor. even if the three biggest models (410m , 1.4b and 6.9b) reach a compression of about 1.6, this value is notably lower than the ones obtained for oxymoron and hyperbole. for pleonasm, all models obtain similar performance and do not improve with respect to the randomly initialised model, which obtains a compression of 1.46. in order to improve performance for metaphor and pleonasm, we have tried tuning the hyperparameters of the neural classifier, checking 100 further configurations, obtaining no improvement. for metaphor, even doubling and quadrupling the training data did not lead to improvements. this suggests that recognising metaphor and pleonasm is particularly difficult for the gpt models we considered. for hyperbole, oxymoron and metaphor, we can see that, on average, bigger models obtain a better performance. taking hyperbole as an example, 31m obtains a maximum value of 1.73 whereas 160m obtains 1.85 and 410m obtains 2.04. notable exceptions are 1.4b and 6.9b for hyperbole (which do not perform better than 410m ) and 410m , 1.4b and 6.9b for metaphor, which reach similar compression. the learning trajectories of hyperbole, oxymoron and metaphor share some similarities. as can be seen in figure 1 , most of the models start increasing their compression value in the initial phases of the training (before 10b tokens) and they do not have remarkable improvements afterwards. this suggests that, in order to acquire the capability of recognising these figures of in some cases, we can see a worsening of compression during the training process. whereas the 14m model for hyperbole obtains only a slight worsening (from 1.65 to 1.58), this behaviour is more evident for metaphor and oxymoron. for metaphor, the 160m model after receiving 180b tokens in training rapidly decreases its performance, passing from 1.58 to 1.49 of compression. this behaviour is even more drastic for oxymoron: all models apart from 410m and 1.4b progressively worsen their performance in the latter stages of training. an interesting aspect that can be noted is that smaller models worsen earlier: 14m starts decreasing its compression after about 100b tokens, 31m after 142b, 70m after 167b and 160m after 199b. we do not observe this behaviour for 410m and 1.4b. finally, for oxymoron the 6.9b model exhibits a peculiar behaviour: similarly to smaller models, its performance does not increase much over training. however, as the larger models, its compression is stable and does not worsen after a certain iteration. we conducted further experiments to analyse whether the low compression was due to the probe dimensions. more specifically, we performed a random search of the probe hyperparameters (number of hidden layers from 0 to 4 and number of neurons from n/2 to n/32, where n is the embedding size). however, we did not see any improvement. thus, we claim this is a characteristic of the model. we have also conducted another experiment for all case studies on 410m , considering as the probe input only the span of the sentence containing the figure of speech. the complete results are reported in appendix c. for metaphor and pleonasm we did not observe a visible difference in performance; on the contrary, for oxymoron and hyperbole, a sharp deterioration in performance is observed, achieving, respectively, 1.64 (vs 1.98) and 1.85 (vs 4.09). this could be caused by the poor quality of the annotated spans for some datasets, or by the loss of contextual information that nlms distribute over all tokens. finally, we have performed the same experiments over a bert @xcite and a roberta @xcite models. the results, available in appendix c, are very similar to those obtained for the gpt-neox models, and they confirm the generalizability of our results. ## conclusions and future work considering seven different gpt-neox models, we investigated whether and how nlms are able to recognise four types of figurative language (hyperboles, metaphors, oxymorons and pleonasms) describing their learning trajectories. our results show that the most recognisable figures of speech are hyperboles and oxymorons, whereas more difficulties can be found for metaphors and especially for pleonasms. analysing the learning trajectories, we discovered that such capabilities are acquired in the early stages of the training and usually do not improve in the latter stages (q1). on the contrary, we observed that for oxymoron the final layer of several models decreases its results, coherently with @xcite . @xcite and @xcite . apart from the 6.9b model, bigger models tend to better understand the figures of speech we analysed (q2). analysing the different layers of each model, we have seen that best performing ones are usually in the central part or in the second half of the model, consistently with the literature @xcite . by comparing models trained on different datasets, we saw that using a literary corpus does not provide advantages in recognising figures of speech, with respect to a model trained on generic content. moreover, a larger general dataset drastically improves the results, coherently with @xcite and @xcite . as future work, we aim to expand our analysis considering more figures of speech, more datasets and their impact, considering also other language and multilingual models. for instance, it would be interesting to identify whether figures of speech knowledge is language specific or shared by more languages @xcite . finally, we will focus on llms and black box models.