Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks (2020)

Abstract: Semantic models derived from visual information have helped to overcome some of the limitations of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when compared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representation. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge.

## introduction many approaches to representing the meaning of imageable, concrete concepts (e.g. frog, apple, car, guitar) have been developed in the fields of cognitive science, computational linguistics and computer vision. most explicitly, property listing studies have been used in cognitive psychology and cognitive neuroscience to characterise word meaning in terms of discrete semantic properties @xcite @xcite . in property listing studies, human participants enumerate as many features as they can for each concept word, and these responses are then aggregated and normalised to a set of verbal semantic descriptors that correspond to elements of concept meaning (e.g. does-croak for frog). this gives a representation of each concept as a sparse vector which encodes the semantic properties that occur for that concept. the resulting properties can then been applied to research investigating the organisation of semantic processing across the cortex, and to studies of the speed or ease of semantic processing for different concepts and different types of concept knowledge @xcite @xcite . a desirable trait of semantic property norms is their interpretability, since this interpretability facilitates the design of cognitive experiments on conceptual semantics @xcite . this interpretability has also allowed researchers in nlp interested in distributional lexical semantics to gain better insights into the kinds of information that dense vector space models attain from pretraining. even though many state-of-the-art vector space models of word meaning perform well when evaluated on both intrinsic and downstream tasks @xcite @xcite , researchers have demonstrated that such models often fail to fully encode certain facets of conceptual meaning @xcite . for example, taxonomic properties (i.e. properties describing the object category, such as is-an-amphibian for frog) and properties reflecting encyclopedic information or information about object function tend to be well-represented in the vector space, but properties that correspond to other kinds of attributes, such as colour, form, and modes of motion, tend to be poorly encoded @xcite . these insights have motivated the development of multimodal semantic representations, which learn from multiple sources of information to ground these representations in the real world -an approach most successfully demonstrated by models that combine data from text and images @xcite @xcite . generally, when image data is incorporated into the construction of word embedding spaces, vectors are derived from the penultimate layer of a deep convolutional neural network trained on an image classification task. although not usually explicitly stated, the rationale for using the penultimate layer of the network is that this layer should be the layer that is maximally relevant to the predicted label for the objects, whilst also being the layer that is least influenced by the low-level visual noise due to the details of the sampled training images. however, it is possible that meaningful object knowledge is present in lower layers of the network even if such information is not directly relevant to the object labelling task that the computer vision model is trained on. for example, in human property listing data, the property is-green is reliably given for frog. in a computer vision model trained to discriminate between object classes, however, the greenness of frogs may not be strongly relevant to discriminating between images of frogs and images of other amphibians or reptiles (which also tend to be green) and so this information -highly relevant to human semantic representations -may not be strongly represented in the penultimate layer. in this work, we demonstrate that whilst using the penultimate layer of a computer vision model is an effective way to capture concept semantics, this approach is not optimal with respect to the goal of producing representations that encode information about cognitively-relevant semantic attributes of concepts. using human property norm data, we demonstrate that certain features and feature types are more decodable at particular layers of the network than others. to produce representations that can make use of all the available semantics-relevant information from the network, we merge the feature map information by constructing meta-embeddings using information from all convolutional layers. we demonstrate that these ensembles of distributional models produce more complete representations of conceptual meaning, when evaluated against human conceptual knowledge. to our knowledge, ours is the first work to consider and evaluate the use of all layers of computer vision models for constructing semantic models, and is the first to consider a range of recent computer architectures in building ensembled and multi-modal representations. finally, we demonstrate how these meta-embeddings can be used in a zero-shot property mapping task, which allows us to automate the generation of interpretable semantic properties for unseen concepts. we make our code, embeddings, and analysis pipleine openly available foot_0 . ## related work whilst word embedding vectors derived from text data have a long history of proven utility on a wide range of downstream tasks, they have been shown to struggle with encoding certain types of semanticallyrelevant information. directly analysing the relationship between explicit property knowledge found in property norm data and the information in pretrained distributional models shows that particular properties relating to more sensory or perpetual information about object semantics can be poorly captured, compared with more associative and encyclopedic knowledge @xcite @xcite . @xcite ), using probing classifiers, showed that many of these properties may not be decodable at all from text-based embedding spaces. findings such as these have motivated researchers to incorporate multimodal information into representations of word meaning @xcite . meta-embeddings have emerged as a useful method for combining information from different word embeddings models @xcite . different embeddings may be trained on various corpora of text, with different sizes, vocabularies, learning methods or model architecture. meta-embeddings can then be created that combine the complementary information from all sources @xcite . in the context of language modelling, different layers of a pre-trained language model may be sensitive to different kinds of linguistic information, and effectively combining embedding information across layers has been shown to improve performance on different tasks sensitive to different kinds of information (such as pos-tagging and word sense disambiguation) @xcite . a number of successful methods have emerged for combining word embedding spaces from different sources. coates and @xcite demonstrate that combining vectors using element-wise addition can be just as effective as concatenating, given that the embeddings are orthogonal. @xcite propose a number of autoencoder type networks to combine one or more vector space models. finally, neill and @xcite give a comprehensive set of empirical results for a number of models and loss functions for learning complex meta-embeddings, demonstrating that loss functions that focus on vector direction such as cosine or kl-divergence based losses give the best performance on intrinsic benchmarks. in distributional semantics, powerful approaches have been developed for building functions that can map from one semantic space to another @xcite , compelling researchers to construct cross-modal mappings from dense distributional models with much larger vocabularies onto property norm data @xcite . for example, @xcite constructed distributed semantic representations for each property dimension used in a large set of property norms, whilst @xcite showed that deep neural networks provide the best performance for zero-shot mapping between semantic spaces. motivating our work on ensembling over layers of deep convolutional neural network vision models, it has been shown that different layers of such networks learn features that reflect different kinds of visual properties. the lower layers of vision networks tend to capture visually basic features such as gabor filters and colour gradients, which are then combined at later layers to construct task-specific high-level visual features that are relevant to object classification @xcite . ## approach the most prominent method for building image-based semantic models involves using pretrained deep convolutional neural networks (dcnns) to extract visual information from image data by retrieving the output vectors from the penultimate layer of the network. in this work, we utilise dcnns trained for the imagenet lsvrc competition @xcite , which aim to predict the correct object in an image from a set of 1000 possible labels. in general, these networks use convolutional layers to extract visual information and build increasingly complex features, before one or many fully-connected layers are used to compute the probability distribution over the classes. distributional semantic spaces can then be constructed from the penultimate fully-connected layer. here, we instead focus our analysis on representational spaces generated at all convolutional layers of the network. our goal is to demonstrate that the types of semantic attributes encoded in these feature maps depend on their depth in the network, and thus using only the penultimate layer may be suboptimal for representing concept meaning. ## analysis to measure how well each layer of a dcnn encodes salient properties of human conceptual knowledge, we train supervised models to predict the presence of a property for a given concept. we note that while a supervised classifier's ability to identify the presence of a property indicates that the property is encoded in the representations, the converse is not always true @xcite . ## improving distributional semantic models with visual meta-embeddings we have seen that particular layers of dcnns best capture different types of semantic information. here we investigate whether we can use this insight to obtain improved image-based semantic embedding spaces and thus build more faithful representations of conceptual meaning. (in these experiments, we focus on resnet, since it gave the best performance on the decoding task; for results with all four models, see the supplementary ## experiments to evaluate these two meta-embedding models, we repeat the decoding experiment with the svd and a 1ton meta-embeddings both of size 300 built using the feature maps from all of the resnet convolutional layers. we compare the meta-embeddings with embeddings constructed from the penultimate layer of resnet (i.e. the traditional approach) to see how well they decode each property type. ## conclusion we have demonstrated the potential of utilizing interpretable semantic primitives derived from human property norm data as a tool for investigating the information captured in the latent representations of deep convolutional neural networks. we reveal that, whilst the widely accepted approach for extracting visual semantic representations, using the penultimate layer of dcnns, yields strong representations of conceptual meaning, they overlook key information generated by the neural network. instead, we develop meta-embeddings that encompass all the salient feature information encoded in the representations produced at all layers of several dcnns. these new vector space models are not only closer representations of human conceptual knowledge, but also can be used to build multimodal semantic models that improve performance on a zero-shot cross-modal mapping task and give better fit to human semantic similarity benchmarks. furthermore, the field of meta-embeddings is rich in potential methods for combining vector space models from different semantic domains, while our research offers empirical evidence that supports our method for constructing meta-embeddings to improve image-based and multi-modal distributional semantic models. concept: guitar. next, we chose an image of the concept guitar (displayed in figure 5 ), which is another one of the concepts in our lexicon, though is not directly taken from the training data. we see that the model detects some visual properties in the lower layers, which it assumes is related to another high-level concept, an animal. we can see this from the top prediction being is-an-animal and other features related to animals such as has-a-tail and has-fur-hair. it is not surprising that there is a high degree of association between certain properties in the norming study, since many related to particular taxonomies, in this case, is-an-animal. nevertheless, as we move through the network the trajectory of the prediction quickly becomes more related to a guitar, though is-an-animal is still predicted in the top ten features. as we can see, these models generalise quite well to other images and can still decode complex features related to conceptual categories. concept: fruit. next, we chose an image of the concept fruit (displayed in figure 6 ), which is neither taken from the training data or the lexicon, but instead consists of many concepts from the data such as apple, banana and kiwi. here we want to analyse how our cross-modal model copes with multiple instances of the concepts. here, we see that the model can quickly detect visual properties such as is-yellow, is-red and is-small, though other high-level properties emerge such as is-eaten-edible and is-a-fruit. again, is-an-animal emerges as a property, which may be due to bias in the model towards high occurring properties. concept: wampimuk. finally, we chose an imagined concept, known as a wampimuk (displayed in figure 7 ). a wampimuk is a fictitious concept proposed by @xcite , to convey how context can shape our perception of a concept, even if we have never heard of it before. humans are capable of building complete semantic representations for concepts, even when the information is fragmented @xcite . hence, a sentence like "we found a cute, hairy wampimuk sleeping behind the tree" can communicate a lot of information about what a wampimuk might be, in this case, a small furry animal. the authors create a potential image of such an animal that does not exist, yet we can extract properties about the concept just as well. hence, we also examine the convolutional layers of the network when given such a creature, to determine whether reasonable semantic properties can be captured by our cross-modal model. we see that the network produces features that the cross-modal map detects as salient aspects of the image such as is-small and has-fur-hair. furthermore, the model can detect conceptual knowledge related to this imaginary creature based on the context of this information. for example, in the lower layers, has-a-tail is predicted by the cross-modal map, even though there is no evidence of this in the picture, yet it would make sense for a small creature. as we move to the final layer of the network, we can even see complex taxonomies emerge, such as is-a-mammal that is quite plausible. (a) predictions from resnet convolution layer 3 (b) predictions from resnet convolution layer 15 (c) predictions from resnet convolution layer 27 (d) predictions from resnet convolution layer 35 figure 7: top property predictions based on the image of a wampimuk with features extracted from resnet. each colour represents a feature category: taxonomic , visual perceptual , functional and other perceptual .