Visually Grounded Interpretation of Noun-Noun Compounds in E nglish (2022)

Abstract: Noun-noun compounds (NNCs) occur frequently in the English language. Accurate NNC interpretation, i.e. determining the implicit relationship between the constituents of a NNC, is crucial for the advancement of many natural language processing tasks. Until now, computational NNC interpretation has been limited to approaches involving linguistic representations only. However, much research suggests that grounding linguistic representations in vision or other modalities can increase performance on this and other tasks. Our work is a novel comparison of linguistic and visuo-linguistic representations for the task of NNC interpretation. We frame NNC interpretation as a relation classification task, evaluating on a large, relationally-annotated NNC dataset. We combine distributional word vectors with image vectors to investigate how visual information can help improve NNC interpretation systems. We find that adding visual vectors increases classification performance on our dataset in many cases.

## introduction conceptual combination is the cognitive process that allows us to combine two mental concepts into one, for example by juxtaposing or otherwise merging two concepts. for instance, a house located on a beach might typically be called a 'beach house'. noun-noun compounds (nncs) are the linguistic phenomenon in which two nouns are joined to form one single, syntactically inseparable unit. the process of combining nouns into new nominal units is both highly prevalent and infinitely productive in a language like english @xcite , and also exists in various forms in many other languages, including but not limited to german, norwegian, hindi, tamil, japanese, chinese, bulgarian, and turkish @xcite . in english, the head of the nnc is usually the rightmost word, and determines the semantic category of the compound. the leftmost word in english nncs is referred to as the modifier. although nncs are a common occurrence, the highly productive nature of compounding @xcite means that individual nncs tend to have relatively low frequency counts @xcite . compositional models have therefore been of particular interest to researchers working on computational nnc representations (e.g. @xcite . due to of the high prevalence and complex nature of english nncs, the ability to interpret compounds would greatly improve several important natural language processing tasks, such as machine translation @xcite ), text summarization (e.g. @xcite , question answering (e.g. @xcite , and natural language inference (e.g. @xcite . in this paper, we frame compound interpretation as a classification problem. the goal is to identify the semantic relationship between the nominal elements of a compound. we explicitly compare the contribution of linguistic and multimodal (visuolinguistic) representations to this task. 1 in part, the motivation for this is theoretical, as a computational account of linguistic meaning has to address the link between symbolic and non-symbolic information @xcite . a further motivation is the empirical observation that grounding representations in vision gives rise to richer meaning representations @xcite . composition in the visual modality has also been shown to be possible for certain nncs @xcite . a final motivation comes from find-ings in cognitive science suggesting that visually grounded word representations yield results closer to human performance on some nnc processing tasks @xcite . our goal is to assess to what extent visual grounding helps to accurately identify the semantic relationship between nnc constituents. for example, figure 1 displays images of the constituents of 'beach house' as well as the compound itself. does the relationship between the constituents in the nnc become easy to predict once such visual information is incorporated, in addition to the textual representation of the constituents? ## background early approaches to the automatic interpretation of noun compounds included rule-based approaches @xcite or semi-automatic approaches requiring some user interaction @xcite . other work utilized frequency statistics of nnc constituents to build probabilistic models for nnc interpretation @xcite . @xcite leveraged wordnet @xcite similarities in supervised training approaches. some approaches to nnc interpretation deal with identifying an appropriate paraphrase for a compound which explicitly states the relation between the compound's constituents. several paraphrasing-based approaches have viewed the task of freely paraphrasing noun compounds as a goal in itself @xcite @xcite , whereas others have used paraphrases as inputs to a model, representing nncs in some way through their paraphrases. other approaches to nnc relation classification tend to be centered around classifying nncs based on a pre-defined set of compound relations using various representations of the compounds themselves as input. both compositional and distributional representations have been tested. dima (2016 ) performed the first experiments on compositional representations of english nncs, using compositional models such as the fulladd model @xcite and the matrix model @xcite ). dima's results, which were tested on datasets by @xcite and ó séaghdha (2008), reached a similar performance to the results obtained by the creators of said datasets, respectively. yet, dima's work utilized simpler methods and did not include lexical and relational information, as opposed to tratz and hovy and ó séaghdha. visuo-linguistic representations for nnc interpretation have received far less attention. @xcite created the first computational model of visuo-linguistic conceptual combination, reporting positive results on several nnc processing tasks. @xcite found that certain compounds can be composed in the visual domain by simple addition of image feature vectors. however, none of these studies have touched upon nnc interpretation using visuo-linguistic data, an area that remains unexplored, to our knowledge. the present work focuses on the interpretation of nncs that possess at least some degree of compositionality. this is justified on the grounds that novel compounds, which are very common @xcite , must be interpreted compositionally on first encounter. we employ one compositional model, called the full additive model @xcite , as well as simple vector concatenation, in our experiments to construct compound vectors from individual constituent vectors. we do this for linguistic and visual vectors separately, and then combine the two modalities using vector concatenation. the following section will describe how we obtain our visual and linguistic vectors as well as introduce the noun compound dataset that we use. ## data to perform our experiments, we use two main sources of data: a relationally-labeled nnc interpretation dataset for training and testing @xcite , and imagenet @xcite to extract visual feature embeddings. the following subsections will describe these datasets in more detail. split train val test coarse random 13835 928 3701 lexical (full) 4650 1593 766 lexical (mod) 9555 5316 3593 lexical (head) 9048 5516 3900 fine random 13968 934 3725 lexical (full) 4614 1574 843 lexical (mod) 9511 5270 3846 lexical (head) 8938 5640 4049 to be able to test how compound interpretation models perform when dealing with unseen constituents, the @xcite dataset is split in various ways to ensure no previously seen constituents are available in the validation and testing phase. different lexical splits ensure that the test and validation dataset contain no constituents previously seen in the training data -the lexical mod split ensures no previously seen modifiers (e.g. 'beach' in 'beach house'), the lexical head split ensures no previously seen heads (e.g. 'house'), and the lexical full split ensures no previously seen constituents at all. the random split does not take into account whether constituents are found in the training data or not. before performing our experiments, we do some filtering on the data in which we remove the fine-grained classes personal noun, per-sonal title, and lexicalized. our reason for removing the personal noun and per-sonal title classes is that there is some doubt as to whether proper names and titles possess the same semantic characteristics as common nouns @xcite . several works on nnc interpretation remove proper nouns from their data (e.g. @xcite . others, like @xcite , choose to keep these categories but still acknowledge that their presence in the dataset is questionable. we remove the lexicalized class because our work is mainly centered around how to interpret compounds that have a certain degree of compositionality, seeing as novel compounds, which likely make up the majority of compound types in most corpora, will need to be interpreted compositionally. ## methods we frame the nnc interpretation task as a classification problem, experimenting with passing linguistic and visuo-linguistic vectors as inputs to an svm classifier. our experimental process can be described in three steps: to obtain linguistic vectors and visual vectors, we utilize pre-trained word2vec @xcite and resnet @xcite ) models, respectively. our models, as well as our experimental setups and baselines, will be described in this section. ## results and evaluation we evaluate our setup on the tratz (2011) dataset and report f1 scores for all dataset configurations. split mc-o mc-m mc-h coarse random 7.5 40.0 59.3 lexical (full) 6.7 -lexical (mod) 7.8 -58.8 lexical (head) 7.0 38.6 -fine random 5.3 34.5 54.1 lexical (full) 5.6 -lexical (mod) 6.3 -52.6 lexical (head) 5.2 34.8 - as has been shown in previous research on this dataset, the most challenging dataset split is the full lexical split, where no constituents in the validation and test data are previously seen in the training data. as expected, the fine-grained setting is generally more challenging than the coarse-grained one. as we can see from comparing tables 4a and 4b, the results in the former table are more ambiguous, meaning that we cannot conclude that one input type (linguistic or visuo-linguistic vectors) is better than another. in one might be inclined to question why our full-add experiments on this dataset perform worse than very similar experiments done by e.g. @xcite and @xcite . this is likely due to the fact that @xcite and @xcite trained their own word embeddings specifically for this task, meaning that they were able to obtain distributed embeddings for more of the compounds in the @xcite dataset than what we had available through our pre-trained model, and as a consequence had more training data for the fulladd model. as our goal with this work is not to compare composition functions for linguistic vectors, we saw training our own embeddings as being superfluous for this study. overall, we see that, in our experiments with concatenated compound vectors, adding visual information helps increase the scores in all cases, and in some cases the increases are statistically significant. ## conclusion and future work in this paper, we have presented nnc interpretation experiments on the tratz (2011) dataset, comparing linguistic and visuo-linguistic inputs to an svm classifier. we have found that, in our best-performing case, concatenating visual feature vectors with linguistic vectors (word embeddings) helps increase f1 scores on the @xcite dataset in almost all experimental settings. our findings indicate that utilizing visual information for this nnc relation classification task might indeed be a promising endeavor. future work should aim to further refine our approach by for example using more sophisticated methods for selecting images to represent words, exploring ways to represent abstract or nonimageable words, and finding better ways to visually ground polysemous words. in this regard, recent multimodal encoders pretrained on visual and linguistic data (e.g. @xcite , are a promising way forward. another possible angle for future work could be to consider nnc interpretation in visual and linguistic contexts. in the future, we would also be eager to explore visual grounding in other aspects of computational nnc related tasks, such as nnc generation. additionally, our approach should be tested on different datasets and in different circumstances, for example in a task that determines the probability of compound categories rather than fixed classes. one final potential angle for future work could be to look further into the task of visual composition. a first step could be to more closely examine the effects of using the fulladd function with image vectors. to conclude, our results are in line with previous work from both cognitive science and computational linguistics suggesting that more psychologically plausible models of nnc processing should incorporate grounding.