Probing Contextual Language Models for Common Ground with Visual Representations (2021)

Abstract: The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations. In this work, we consider a new question: to what extent contextual representations of concrete nouns are aligned with corresponding visual representations? We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Moreover, they are effective in retrieving specific instances of image patches; textual context plays an important role in this process. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans. We hope our analyses inspire future research in understanding and improving the visual capabilities of language models.

## introduction contextual language models trained on text-only corpora are prevalent in recent natural language processing (nlp) literature @xcite @xcite . understanding what their representations encode has been the goal of a number of recent studies @xcite . yet, much is left to be understood about whether-or to what extent-these models can encode visual information. we study this problem in the context of language grounding @xcite @xcite @xcite , empirically investigating whether text-only representations can naturally be connected to the visual domain, without explicit visual supervision in pre-training. figure 1 : we introduce a probing mechanism that learns a mapping from contextual language representations to visual features. for a number of contextual language models, we evaluate how useful their representations are for retrieving matching image patches. we argue that context plays a significant role in this investigation. in language, the ability to form context-dependent representations has shown to be crucial in designing pre-trained language models @xcite . this is even more important for studying grounding since many visual properties depend strongly on context @xcite . for instance, a "flying bat" shares very few visual similarities with a "baseball bat"; likewise, a "dog sleeping" looks different from a "dog running". while alignments between language representations and visual attributes have attracted past interest @xcite @xcite @xcite , the role of context has been previously overlooked, leaving many open questions about what visual information contextual language representations encode. in this work, we introduce a method for empirically probing contextual language representations and their relation to the visual domain. in general, probing examines properties for which the models are not designed to predict, but can be encoded in their representations @xcite figure 2 : examples of retrieved image patches from text-only representations using our probe. all shown images are retrieved from ms-coco @xcite , using representations from bert base. importantly, these object categories (e.g. kite) are previously unseen by our probe. on the bottom rows, we show examples of the influence of context in retrieval: while all retrieved image patches belong to the correct object category, cat, more descriptive contexts allow more accurate retrieval at the instance level. 2020). here, our probe is a lightweight model trained to map language representations of concrete objects to corresponding visual representations. the probe (illustrated in figure 1 ) measures whether language representations can be used to give higher scores to matching visual representations compared to mismatched ones. textual and visual representations are collected from image captioning data, where we find pairs of concrete words (e.g. cat or kite) and their corresponding image patches. the probe is trained using a contrastive loss @xcite ) that gauges the mutual information between the language and visual representations. given text-only representations of an unseen object category, the trained probe is evaluated by retrieving corresponding image patches for categories it has never seen during training. qualitative examples can be found in figure 2 . we examine representations from a number of contextual language models including bert, roberta, albert and t5 @xcite @xcite . for all of them, we find that interesting mappings can be learned from language to visual representations, as illustrated in figure 2 . in particular, using its top-5 predictions, bert representations retrieve the correctly paired visual instance 36% of the time, strongly outperforming non-contextual language models (e.g., glove @xcite ). moreover, for all examined models, image patches of the correct object category are retrieved with a recall of 84-90%. our experiments are backed by a control task where visual representations are intentionally mismatched with their textual counterparts. retrieval performance drops substantially in these settings, attesting the selectivity of our probe. moreover, we measure the impact of context on retrieval at the instance level. contextual models substantially outperform non-contextual embeddings, but this difference disappears as context is gradually hidden from contextual models. when the context includes adjectives directly associated with the noun being inspected, we find significantly better instance retrieval performance. finally, we investigate a number of grounded language models-such as lxmert and vilbert @xcite @xcite )that see visual data in training, finding them to slightly outperform text-only models. contrasting the learned mappings with human judgment, the examined visually grounded language models significantly underperform human subjects, exposing much room for future improvement. ## related work what is encoded in language representations? understanding what information nlp models encode has attracted great interest in recent years @xcite . from factual @xcite @xcite to linguistic @xcite @xcite and commonsense @xcite knowledge, a wide set of properties have been previously analysed. we refer to @xcite and @xcite for a more comprehensive literature review. a common approach, often used for inspecting contextual models, is probing @xcite @xcite . in short, it consists of using supervised models to predict properties not directly inferred by the models. probing is typically used in settings were discrete, linguistic annotations such as parts of speech are available. our approach differs from previous work in both scope and methodology, using a probe to measure similarities with continuous, visual representations. closer to our goal of better understanding grounding is the work of @xcite , that design probes for examining multi-modal models. in contrast, our work examines text-only models and does not rely on their ability to process images. language grounding. a widely investigated research direction aims to connect natural language to the physical world @xcite @xcite @xcite @xcite . this is typically done through training and evaluating models in tasks and datasets where both images and text are used, such as visual question answering @xcite . a number of previous work have investigated mappings between language and visual representations or mappings from both to a shared space. @xcite investigate semantic similarities between words and images through a joint latent space, finding a positive correlation with human rated similarities. similarly, silberer and lapata (2014) builds multi-modal representations by using stacked autoencoders. @xcite and @xcite show that a shared latent space allows for zero-shot learning, demonstrating some generalization to previously unseen objects. @xcite construct grounded word representations by exposing them to aligned visual features at training time. @xcite investigate how well word representations can predict perceptual and conceptual features, showing that a number of such features are not adequately predicted. collell talleda et al. zero-shot detection. recent work attempts to build object detectors that generalize to unseen object categories, by conditioning the predictions on word embeddings of the class @xcite , visual attributes @xcite @xcite ) or text descriptions @xcite . in our work, we use language representations of words in context (captions) as inputs. more fundamentally, although our experiments on unseen object categories can be used for zero-shot detection, we differ from previous work in motivation, which translates to further experimental differences. given our goal to analyse already trained models (as opposed to learning a generalizable object detector), we train nothing apart from a lightweight probe in our analyses. ## probing contextual representations our main goal is to characterize the relation between contextual language representations and the visual domain. we first describe how language and visual representations of concrete concepts can be collected from image captioning datasets ( §3.1). next, we design a probe that examines the relation between these representations, learning a mapping from language to visual representations ( §3.2). an overview is illustrated in figure 3 . ## results and discussion at a high level, our experiments show that i) language representations are strong signals for choosing between different visual features both at the instance and category levels; ii) context is largely helpful for instance retrieval; iii) infonce works better than other studied losses, and some consistency is found across datasets; iv) visually grounded models outperform text-only models; v) all models lag greatly behind human performance. we provide further details in §5.1-5.3. ## conclusion understanding the similarities between language and visual representations has important implications on the models, training paradigms and benchmarks we design. we introduced a method for empirically measuring the relation between contextual language representations and corresponding visual features. we found contextual language models to be useful-while far from human subjects-in discerning between different visual representations. moreover, we explored how these results are influenced by context, loss functions, datasets and explicit grounding during training. altogether, we hope that our new methodological and practical insights foster further research in both understanding the natural connections between language and visual representations and designing more effective models at the intersection the two modalities.