Can Large Language Models Interpret Noun-Noun Compounds? A Linguistically-Motivated Study on Lexicalized and Novel Compounds (2024)

Abstract: Noun-noun compounds interpretation is the task where a model is given one of such constructions, and it is asked to provide a paraphrase, making the semantic relation between the nouns explicit, as in carrot cake is “a cake made of carrots.” Such a task requires the ability to understand the implicit structured representation of the compound meaning. In this paper, we test to what extent the recent Large Language Models can interpret the semantic relation between the constituents of lexicalized English compounds and whether they can abstract from such semantic knowledge to predict the semantic relation between the constituents of similar but novel compounds by relying on analogical comparisons (e.g., carrot dessert). We test both Surprisal metrics and prompt-based methods to see whether i.) they can correctly predict the relation between constituents, and ii.) the semantic representation of the relation is robust to paraphrasing. Using a dataset of lexicalized and annotated noun-noun compounds, we find that LLMs can infer some semantic relations better than others (with a preference for compounds involving concrete concepts). When challenged to perform abstractions and transfer their interpretations to semantically similar but novel compounds, LLMs show serious limitations.

## introduction noun-noun compounds represent an important challenge for all the applications related to natural language understanding, given the implicit semantic relation assumed between the two components, namely: head and modifier @xcite . their correct interpretation is an essential step for several natural language processing applications such as question answering, machine trans-1 data and code available at: https://osf.io/67k9u/ ?@xmath0, and information extraction. for example, if a question answering system is asked something about birthday cake, it must understand that the user is talking about a cake made for birthdays; while if it is asked about a carrot cake, it must understand that the query refers to a cake made with carrots (not for carrots). the capacity to grasp the semantic connection underlying the pairing of two terms in a compound represents a form of abstraction inherent to human cognition, applicable to concrete and abstract concepts alike (concrete such as carrot cakes and abstract such as bank loans). this skill is often wielded even for never-encountered-before compounds @xcite . previous research stressed the role of structured world knowledge in the interpretation of compounds @xcite , which includes the knowledge of the constituent entities and their potential relations. moreover, people are able to interpret novel compounds by abstracting from knowledge based on past experiences with similar conceptual combinations @xcite @xcite and to extend them by relying on analogical comparisons @xcite . can the modern large language models (llms) do the same? the main goal of our study is to propose a more refined methodology to understand when and how llms are capable of performing abstractions that humans routinely do, namely: understanding the semantic relation existing between the two components of a lexicalized compound and then extending such relation to novel compounds that are constructed in such a way to maintain the semantics of the original components. to do so, we manually manipulated existing compounds by replacing one of the two terms (head or modifier) with their hypernym, namely a word denoting a superordinate concept @xcite . this allowed us to gener-compound coarse-grained @xcite fine-grained @xcite hatcher-bourque @xcite paraphrase @xcite ate novel compounds such as birthday dessert and event cake, based on the lexicalized birthday cake. to test the llms' ability to understand the semantics of lexicalized and novel compounds, we assess whether surprisal, a metric directly based on the log probabilities of the llms, is able to differentiate between the possible interpretations of a compound. we hypothesize that llms may be accurate in recognizing the correct semantic relation holding between the two components of a lexicalized compound. moreover, if we were to observe any differences in the performance across different types of compounds, we would argue that such differences may be (at least partially) explained by the concreteness of the compound, in line with previous psychological findings showing that concrete concepts are processed more easily than abstract ones @xcite . as a complement to surprisal analyses, we performed a metalinguistic prompt asking to identify the correct interpretation of a compound from a list of options. we relied on llms trained with instruction tuning, a method that has recently been proposed to enhance the generalization capability of llms, and assessed the performance of some of the most popular architectures on this task. contributions our contributions can be summarized as follows: ## related work the problem of the interpretation of compounds has generally been addressed via two different tasks: the first one is the classification in a limited inventory of ontological/semantic relations holding between the two nouns @xcite , and the second one is the free generation of a paraphrase describing the same relations @xcite @xcite . with the introduction of transformer-based language models, several studies have proposed to investigate their internal representations to understand how the constituent meanings are composed @xcite @xcite , and if and to what extent they are able to generalize to interpret unseen compounds @xcite . @xcite proposed a few-shot model based on gpt-3 @xcite to tackle interpretation, and they were able to achieve almost perfect performance on a semeval noun compounds benchmark by @xcite . however, by measuring the n-gram overlap between the generated paraphrases and the c4 corpus @xcite , they found that gpt-3 might just be parroting word sequences seen in the training data, and the strategy turned out to be less effective with rare or novel compounds. is the knowledge encoded in recent llmsincluding instruction-tuned ones-sufficient to interpret the relation between constituent nouns and to generalize the interpretations to novel compounds? language models retain a non-trivial amount of knowledge about the world, and this is reflected in the log probability scores that they assign to real-world situations and events described by natural language sentences @xcite ; moreover, the recent progress on instruction tuning led to even better alignment with conceptual representations in the human brain @xcite . therefore, our investigation will focus on three of the most popular llms (llama-2, falcon, and mistral), both in their base and in their instruct version, to see if instruction tuning leads to performance improvements also in the interpretation of compounds. ## general discussion this paper evaluated recent llms on their ability to interpret noun-noun compounds and, specifically, to correctly identify the semantic relation underlying existing and novel compounds. for the interpretation of existing compounds (lncs), we released a dataset that assembles several linguistic and conceptual features associated with each compound, extracted from previous resources or added by the authors (concreteness, semantic type, semantic relation from different classifications) together with a limited set of paraphrases generated from pepper (2022)'s classification. llms accuracy was tested on both the surprisal scores and metalinguistic knowledge extracted by prompting strategies. in both settings, the models showed different performance levels in the identification of different semantic relations. some relations like production are easy to recognize; that is, its paraphrase is the most expected (considering surprisal scores) and more frequently identified in a metalinguistic prompt task. moreover, compounds characterized by higher concreteness were interpreted more accurately overall, as hypothesized. this effect may be explained by the so-called concreteness effect @xcite , which suggests that concrete concepts are processed faster and more easily than abstract ones. previous studies reported that llms generate compound definitions that highly resemble human-generated paraphrases, reaching an almost perfect performance. however, the analyses presented here reveal that they are not as perfect when asked to identify the correct paraphrase, given alternatives. our outcomes confirm what was observed by @xcite : llms' performance can largely be attributed to parroting definitions or parts of definitions extracted from the training corpora. however, it is unclear to what extent llms extract the relational linguistic patterns they learn from corpora and use them to hypothesize about the most likely relationship underpinning a noun compound. in other words, while the models can somehow interpret the semantic relation underlying compounding, there is still a question far from being completely answered: what linguistic properties make compounds more or less difficult to interpret by llms? for this reason, we believe that more effort should be made in designing a comprehensive dataset of noun-noun compounds annotated with different factors influencing the plausibility of the noun compounds. the second experiment represents the first attempt to model novel compounds to understand llms' abilities to abstract and transfer knowl-edge. according to previous studies, the interpretation of a novel combination relies on previous language experience @xcite @xcite . that is, people are able to interpret novel compounds by abstracting from their knowledge of past experiences with similar conceptual combinations, which provide an analogical basis for the production and interpretation of novel compounds @xcite 7 . we manipulated a subset of lexicalized compounds by replacing the modifier or the head word with a hypernym and observed how much harder it is for the llms to interpret the generated compounds. as expected, language models are challenged by this task, but we observe that they still look for a suboptimal solution. for instance, they choose the purpose relation, which has a more general paraphrase (indended for) than other relations (such as loca-tion or container). we believe that this task could provide a window into a specific aspect of the creative abilities of llms. in conclusion, the present study illustrates that there are still questions unanswered regarding how llms interpret compounding. future works will focus on expanding both the lnc and the nnc datasets, including more linguistic features and evaluating the acceptability of selected paraphrases with human judgments.