Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case (2020)

Abstract: Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed. While the power of visual-semantic embeddings comes from the distillation and enrichment of information through machine learning, their inner workings are poorly understood and there is a shortage of analysis tools. To address this problem, we generalize the notion ofprobing tasks to the visual-semantic case. To this end, we (i) discuss the formalization of probing tasks for embeddings of image-caption pairs, (ii) define three concrete probing tasks within our general framework, (iii) train classifiers to probe for those properties, and (iv) compare various state-of-the-art embeddings under the lens of the proposed probing tasks. Our experiments reveal an up to 16% increase in accuracy on visual-semantic embeddings compared to the corresponding unimodal embeddings, which suggest that the text and image dimensions represented in the former do complement each other.

## introduction semantic analysis aims to infer meaning from data. in a mathematical sense, the analysis relates objects in a syntactic domain to objects in a semantic domain. in natural language processing, semantic embeddings such as word2vec and, more recently, bert and gpt-2, have had a revolutionary impact on semantic analysis. the embeddings map words to real-valued vectors which reveal semantic aspects, for example, if words are related in meaning or belong to the same topic. creating such an embedding means to enrich as well as filter out information. unavoidably, some (usually surface and syntactic) information will be lost in the process of projecting words and their contexts onto a representation that focuses on meaning. hence, there is a trade-off: the better we capture the semantics, the more surface and syntactic information becomes blurred. it depends on the downstream task what is the right balance between abstraction and detail. in multimodal semantic analysis, the syntactic domain is a cartesian product of two or more domains, such as a video with audio tracks. this type of semantic analysis is increasingly applicable, showcasing convincing applications and results. multimodal learning models such as devise @xcite demonstrate in particular that zero-shot learning can be signficantly improved by engaging multiple modalities. there are many models based on machine learning techniques that jointly process the input modalities @xcite . the combination of modalities that has hitherto received the greatest interest is the pairing of images and text. semantic embeddings of such data are commonly called visual-semantic embeddings. throughout the rest of this paper, we focus on visualsemantic embeddings, and base the empirical part of our work on the dataset common objects in context (ms-coco), which consists of images with captions @xcite . a tiny bat is held by someone with a camera. a man gently attempts to feed a baby bird. a man in shorts is swinging a bat. a man is swinging a club with both hands. when semantic analysis is applied to the text component of an image-caption pair, the visual information can resolve semantic uncertainties such as in the phrase "a man with a bat in his hands". figure 1 shows two ms-coco images, foot_0 each image accompanied by two of its associated captions in the dataset. while humans would probably glean the correct interpretation of 'bat' and 'club' from the text alone (but not of 'bird'), the visual-semantic information is much less ambiguous. the vector diagram to the right of the images illustrates how one might imagine the vectors of a word embedding such as word2vec to be affected by moving to the multimodal embedding, including visual information. imagine the vectors bat , bird , and club to be those of the pure word embedding. that is, for simplicity we assume that the words are embedded as in the original word2vec embedding, without taking the context provided by the sentence into account. in particular, the two occurrences of 'bat' in the captions are represented by one and the same vector bat , and similarly for the two occurrences of 'club'. now, also incorporating the information in the corresponding image may affect the vectors. the principle is shown in red for the combination of 'bat' and 'bird' with the left image,and in blue for the combination of 'bat' and 'club' with the right image. on the left, bat becomes chiroptera (i.e., the vector now represents the mammal of the order chiroptera) and bird becomes bird-like chiroptera , intuitively representing a hybrid between birds and chiropteras. in the right, both bat and club are turned into a vector club/bat representing bats in the sense of clubs. note that, while the information becomes semantically more accurate, other aspects are lost, e.g. whether the word 'bat' or 'club' was used, and probably also the fact that the second caption on the left actually mentioned a bird. interpretable or explainable machine learning investigates ways to provide explanations for the behaviour and properties of systems based on machine learning @xcite to make them more transparent. to date, it is remains unclear what properties are encoded in semantic or visual-semantic embeddings. ignorance about what is actually captured in an automatically learned semantic representation may lead to serious consequences of various kinds such as propagating discrimination bias @xcite @xcite , or causing safety hazards in robotics by inducing unexpected robotic actions that put humans at risk @xcite . this paper is intended to provide a basis for better explainability of systems relying on visual-semantic embeddings. for this, we transfer the idea of linguistic probing tasks to this realm. the aim of probing @xcite @xcite is to reveal what information an embedding actually encodes. probing tasks should be agnostic to the specifics of encoder architectures, so that they can be used to compare across different methods. @xcite define a (linguistic) probing task to be a classification task that categorizes sentences according to specific linguistic properties, such as sentence length. from the performance of the classifier, it should be possible to draw conclusions about the probed embedding; if the classifier succeeds it indicates that the semantic embedding captures interpretable information regarding the aspect under consideration (although the converse is not necessarily true). @xcite argue that the performance of a probe alone is not sufficient, and introduce so-called control tasks to improve interpretability of probing tasks. a control task reveals whether high accuracy of a probing task really indicates that semantic representations encode a linguistic property, or whether the probing task itself learns this property. in particular, a probing task is complemented with a control task that associates random outputs to the properties under consideration (for example, pos tags). thus, a control task with low accuracy indicates that a corresponding probing task with high accuracy does indeed encodes the probed property. we propose probing tasks for visual-semantic embeddings (in other words, images with captions). in particular, we are interested in tasks that shed light on whether and how a given embedding makes use of the image information in relation to linguistic phenomena such as synonyms and polysemy. the paper is structured as follows. section 2 motivates our approach and relates it to existing work. section 3 provides a systematic discussion and formalisation of probing tasks for visual-semantic embeddings. with this, we hope to map out which properties probing tasks of various types can be used to investigate. section 4 introduces three concrete probing tasks that illustrate our approach, and which are used in our actual experiments reported on in section 5. the code is publicly available. 2 the conclusion in section 6 summarizes our findings and lists future challenges for multimodal probing. ## motivation and related work the probing tasks proposed by @xcite probe sentence embeddings and are categorized according to the type of linguistic properties they capture: surface,-syntactic,-and semantic information. we now give a brief account of these categories, which are outlined in type of information probed for: surface syntactic semantic sentence length bigram shift tense word content tree depth number of subjects top constituent number of objects semantic incongruence coordination inversion surface information comprises probing for sentence length (number of words) and the word content (whether the sentence contains a given word). the probing tasks in the syntactic category ask to detect bigram shift, tree depth and top constituent, revealing whether the embedding makes certain syntactic information accessible. bigram shift tries to predict whether two adjacent words have been swapped (that is, encoding the syntactic order of words). tree depth asks to determine the depth of the syntactic tree of the sentence, and the top constituent task asks the classifier to determine the sequence of the top constituents directly below the sentence (s) node. the probing tasks that probe for semantic properties are tense, subject and object number, semantic odd man out, and coordination inversion. the tense task consists in finding the tense of the main verb, whereas the subject and object number tasks ask to predict the grammatical number of subjects and objects of the main verb, respectively. the task semantic odd man out is about predicting whether a sentence has been modified or not (i.e., a random noun or verb was replaced with another noun or verb). coordination inversion probes for the information whether two coordinate clauses in a sentence have been switched. for example, "they might be only memories, but i can feel each one" and "i can still feel each one, but they might be only memories" @xcite . these probing tasks were defined for unimodal embeddings of natural language. machine learning that utilize multimodal embeddings is a lively field @xcite @xcite , but little is known about what properties these multimodal embeddings actually capture. work, such as by @xcite , aiming to analyze embeddings according to the composition of their encoded concepts is rare. in this paper, we focus on probing visual-semantic embeddings. the establishment of probing tasks is one way to gain systematic knowledge about what embeddings actually capture. another complementary way is to build taxonomies of multimodal machine learning techniques and multimodal embeddings. such taxonomies are proposed, for example, by @xcite and @xcite . both groups of authors categorize embeddings according to different but partly overlapping criteria. the taxonomy by @xcite classifies approaches according to five categories of criteria: (a) representation -how complementary and redundant information is represented, (b) translation -how data is mapped between modalities, (c) alignment -whether and how elements in the different modalities are aligned, (d) fusion -how information coming from different modalities is integrated, and (e) co-learning -in which ways the learning exploits multimodality. the taxonomy by @xcite for (learning) multimodal representations distinguishes between (f) concept representations -embeddings that use low-level representations of concepts, (g) projections -embeddings that represent concepts using only one of the modalities, and (h) compositional representations -approaches that fuse or jointly embed the different modalities. multimodal probing tasks can support the location of a given method in a taxonomy without requiring intimate knowledge of its inner workings: probing which information is accessible by a network trained on the resulting embeddings provides insight into what information is present and how it is represented. some major difficulties of multimodal processing tasks and representations are discussed (from the perspective of multimodal grounding) by @xcite . their discussion illustrates the usefulness of multimodal probing in general, and of visual-semantic probing in particular: 3 combining complementary pieces of information different modalities contribute to the information content of multimodal input in complementary ways. for example, highly relevant visual properties, like the fact that birds have wings and violins are brown, are not usually mentioned in text as they are the default. conversely, taxonomic and functional relations between concepts are poorly represented in images. probing tasks that check whether, e.g., the word brown relates to images of violins would allow to draw conclusions about how successfully these dimensions are combined in the embedding. ## systematic probing for properties of visual-semantic embeddings in this section we develop a general view of visual-semantic probing tasks, and lift the ideas of @xcite to the multimodal realm. consider a property π that a given embedding e may or may not have. in the visual-semantic case, such a property may be "the embedding associates visual properties with the nouns in the text component" or "the embedding encodes the number of objects in the image". a probing task is defined to be a machine learning task -usually a classification task -that is designed in such a way that a model can be trained on e, and the achieved performance allows to draw conclusions regarding the extent to which e possesses property π. we are specifically interested in developing probing tasks for visual-semantic embeddings e, where π is a property that reflects aspects of the multimodal nature of e. ultimately, the goal is to come up with tasks that probe how the embedding maps the individual modalities into a common space. while we are not quite there yet, below we provide a general discussion of what to look for, and how such tasks may be categorized. probing tasks that meet the following requirements seem to be especially valuable: we propose that the probing tasks are organized according to how they make use of the information in the sample data to map out embedding characteristics. for the visual-semantic case, at an abstract level, each probing task either probes the embedding of the original text-image pair (t, i), or it is based on turning (t, i) into (t , i ) in a well-specified manner, such that by comparing the performance of a classifier on emb(t , i ) and emb(t, i), one can draw conclusions about the embedding. depending on how t and i are obtained, different types of probing tasks arise. ## concrete probing tasks this section illustrates the abstract principles introduced in section 3 through a set of concrete probing tasks. these tasks will be experimentally tested in section 5 and will, in future work, be extended with tasks of the types proposed in section 3 to highlight complementary aspects of the semantic embeddings. ## experiments this section describes our experiments with direct probing (see section 4.1) and semantic congruence probing (see section 4.2). see also appendix 5 for further details on reproducibility. ## conclusions and future work probing semantic embeddings with neural-network based classifiers is like looking into a black box with a lens that is itself a black box. valuable information can still be derived, but experiments that take this approach must be made with care, and the results analysed with caution. one approach to mitigate such opacity is proposed by @xcite , namely that the probing task is complemented with a control task to alleviate a possible misinterpretation of what semantic representations actually encode. in the multimodal setting, it is helpful to use probing tasks (as well as complementing control tasks) that are simple, well-defined, and easily implemented on standard data sets. the importance of a task being well-defined is illustrated, albeit in a negative way, by the numobjects task: since there are countless equally valid ways to semantically decompose an image, it can simultaneously be true that an image shows dozens of sheep and that it shows a single herd. the flaw is arguably not as much in the task itself, as in the combination of task and data set. we may, for example, expect that the numobjects task comes to its right in situations where logical units of counting are understood in advance, e.g., in the case of camera footage tracking traffic congestion, where a natural unit would be the number of vehicles. an interesting finding from our initial experiments was the importance of linguistic compared to visual information for complexity estimation and semantic incongruity detection. as our next steps, we would like to repeat the experiments presented here on multimodal data sets other than the original ms-coco dataset, in particular the flickr @xcite and foil-coco @xcite datasets. the semantic congruity task can be implemented in numerous different ways, which makes it a particularly versatile tool for understanding language grounding. continued work will explore how visual-semantic embeddings respond to different types of linguistic manipulation of both image captions, as well as direct image manipulation.