From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds? (2023)

Abstract: Noun compound interpretation is the task of expressing a noun compound (e.g. chocolate bunny) in a free-text paraphrase that makes the relationship between the constituent nouns explicit (e.g. bunny-shaped chocolate). We propose modifications to the data and evaluation setup of the standard task (Hendrickx et al., 2013), and show that GPT-3 solves it almost perfectly. We then investigate the task of noun compound conceptualization, i.e. paraphrasing a novel or rare noun compound. E.g., chocolate crocodile is a crocodile-shaped chocolate. This task requires creativity, commonsense, and the ability to generalize knowledge about similar concepts. While GPT-3’s performance is not perfect, it is better than that of humans—likely thanks to its access to vast amounts of knowledge, and because conceptual processing is effortful for people (Connell and Lynott, 2012). Finally, we estimate the extent to which GPT-3 is reasoning about the world vs. parroting its training data. We find that the outputs from GPT-3 often have significant overlap with a large web corpus, but that the parroting strategy is less beneficial for novel noun compounds.

## introduction noun compounds (ncs) are prevalent in english, but most individual ncs are infrequent @xcite ). yet, it is possible to derive the meaning of most ncs from the meanings of their constituent nouns. the task of noun compound interpretation (nci) addresses this by explicitly uncovering the implicit semantic relation between the constituent nouns. we focus on the paraphrasing variant @xcite , where the goal is to generate multiple paraphrases that explicitly express the semantic relation between the constituents. for example (figure 1 ), a chocolate bunny is a "chocolate shaped like a bunny". earlier methods for nci represented ncs as a function their constituents' representations (e.g. van de @xcite . in recent years, pre-trained language models (plms) caused a paradigm shift in nlp. such models are based on the transformer architecture @xcite , which by design computes a word representation as a function of the representation of its context. further, plms are pre-trained on vast amounts of text, which equips them with broad semantic knowledge @xcite . such knowledge may facilitate interpreting unseen ncs based on observed ncs that are semantically similar. indeed, @xcite showed that a masked language model is useful for this task, and @xcite demonstrated the utility of generative language models on this task. we formalize the experiments presented in @xcite and evaluate generative models on nci. we manually analyze and correct many problems with the standard semeval 2013 task 4 dataset @xcite , and release a cleaned version of the dataset. following the criticism in @xcite on the task's dedicated evaluation metrics, we propose a more complete set of evaluation metrics including both automatic metrics and human evaluation. our experiments show that a few-shot model based on gpt-3 @xcite achieves near-perfect performance on the nci test set. the impressive performance may be due to a combination of factors. first, it tends to memorize texts seen during pre-training @xcite , likely including partial or complete definitions of common ncs. second, it has learned vast commonsense and world knowledge from its pre-training corpus, which-together with its ability to generalizemay be useful for interpreting less frequent ncs. to test the extent that gpt-3 reasons about its knowledge as opposed to memorizes definitions, we propose a second task: noun compound conceptualization (ncc). the setup is identical to nci, but the ncs are rare or novel (e.g., chocolate crocodile in fig. 1 ), requiring a model to come up with a plausible interpretation based on its existing knowledge. we construct a test set for this task based on data from @xcite . the results show that gpt-3 outperforms humans on ncc, presumably thanks to its fast access to a huge "knowledge base", and compared to the relative human slowness on this task @xcite . yet, compared to its performance on nci, gpt-3's performance on ncc shows a significant drop. we thus quantify the extent that gpt-3 copies from its pre-training corpus when generating paraphrases for either nci or ncc. we find that the generated paraphrases have significant overlap with a large web-based corpus, but that as expected, the copying strategy is less beneficial for ncc than for nci. we anticipate that the cleaned dataset and proposed evaluation setup will be adopted by the research community for nci, and hope to see further research on ncc. foot_0 2 background ## noun compound interpretation we first evaluate plms' ability to interpret existing noun compounds. we focus on the free-text paraphrasing version of nci, as exemplified in ## noun compound conceptualization gpt-3's impressive success at interpreting existing noun compounds is related to plms' ability to associate nouns with their hypernyms @xcite and to generate accurate definitions for terms @xcite . such models are trained on vast amounts of texts, including said definitions, and the target nc itself occurring alongside contexts that indicate its meaning. humans are different in their ability to interpret ncs. we can often rely on a single context, or no context at all, to have at least an educated guess at the meaning of a new nc. we are capable of representing new concepts by "mentally manipulating old ones" @xcite , e.g. coming up with a plausible interpretation for chocolate crocodile based on similar concepts such as chocolate bunny. prior work on nci simulated this by training a model to jointly predict a paraphrase as well as answer questions such as "what can chocolate be shaped like?" @xcite . we are interested in learning whether plms already do this implicitly, or more broadly, to what extent can plms interpret new noun compounds? inspired by studies in cognitive science about "conceptual combination" @xcite , we define the task of noun compound conceptualization (ncc). ncc has the same setup as nci ( §3), but the inputs are rare or novel noun compounds. the task thus requires some level of creativity and the ability to make sense of the world. we first describe the creation of the ncc test set (sec 4.1). we evaluate the best model from sec 3.2 on the new test set, and present the results in sec 4.2. ## does gpt-3 parrot its training data? while gpt-3 performs fairly well on ncc, looking at failure cases brings up interesting observations. for example, one of its responses for chocolate crocodile was "a large, aggressive freshwater reptile native to africa". this response seems to have ignored the chocolate part of the nc entirely, and opted to provide an answer to "what is a crocodile?". much like a student who doesn't know the answer to a question so instead regurgitates everything they memorized about the topic in hopes that it will include the correct answer. 3 to quantify the extent to which gpt-3 may be parroting its training corpus, we look at n-gram overlap between gpt-3's generated paraphrases and the large-scale web-based corpus c4 (raffel et al., 2020). 4 figure 2 displays the percents of n-grams among the generated paraphrases (for @xmath0, 4, 5}) that occur in the c4 corpus 0, 1-5, or 5+ times, for each of the nci and ncc test sets. the results are presented separately for paraphrases deemed correct and incorrect by human evaluators. we learn several things from figure 2 . first, the generated paraphrases often had significant overlap with the corpus (34-94%). as expected, trigrams are copied more than 4-grams, which are copied more than 5-grams, as those tend to be rarer. second, for the nci test set, for each n, we see that n-grams from the correct paraphrases are copied from the web more often than n-grams from the incorrect paraphrases. the trend is reversed for ncc, where incorrect paraphrases are copied from the web more often than correct ones. naturally, the copying strategy is less useful for ncc, which requires reasoning about new concepts. when gpt-3 generates correct paraphrases for ncc, their ngrams tend to not appear in the web at all. we reach a similar conclusion by looking at the percent of n-grams in correct vs. incorrect paraphrases that are copied from the web. the vast majority of n-grams copied from the web (97%) for the nci test set were correct, as opposed to only 80% for ncc. ## conclusion we evaluated plms on their ability to paraphrase existing and novel noun compounds. for interpre-c4 @xcite is a colossal, cleaned version of common crawl, thus it is the closest to gpt-3's training corpus. tation of existing ncs (nci), we released a cleaned version of the semeval 2013 dataset, with manual correction and automatic augmentation of paraphrases, and proposed additional evaluation metrics to overcome limitations described in prior work. gpt-3 achieved near perfect performance on this new test set. we then investigated the task of noun compound conceptualization (ncc). ncc evaluates the capacity of plms to interpret the meaning of new ncs. we showed that gpt-3 still performs reasonably well, but its success can largely be attributed to copying definitions or parts of definitions from its training corpus.