Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation (2021)

Abstract: This study carries out a systematic intrinsic evaluation of the semantic representations learned by state-of-the-art pre-trained multimodal Transformers. These representations are claimed to be task-agnostic and shown to help on many downstream language-and-vision tasks. However, the extent to which they align with human semantic intuitions remains unclear. We experiment with various models and obtain static word representations from the contextualized ones they learn. We then evaluate them against the semantic judgments provided by human speakers. In line with previous evidence, we observe a generalized advantage of multimodal representations over language- only ones on concrete word pairs, but not on abstract ones. On the one hand, this confirms the effectiveness of these models to align language and vision, which results in better semantic representations for concepts that are grounded in images. On the other hand, models are shown to follow different representation learning patterns, which sheds some light on how and when they perform multimodal integration.

## introduction increasing evidence indicates that the meaning of words is multimodal: human concepts are grounded in our senses @xcite , and the sensory-motor experiences humans have with the world play an important role in determining word meaning @xcite . since (at least) the first operationalizations of the distributional hypothesis, however, standard nlp approaches to derive meaning representations of words have solely relied on information extracted from large text corpora, based on the generalized assumption that the meaning of a word can be inferred from the effects it has on its linguistic context @xcite . language-only semantic representations, from pioneering 'count' vectors @xcite @xcite to either static @xcite or contextualized @xcite neural network-based embeddings, have proven extremely effective in many linguistic tasks and applications, for which they constantly increased state-of-the-art performance. however, they naturally have no connection with the real-world referents they denote @xcite . as such, they suffer from the symbol grounding problem @xcite , which in turn limits their cognitive plausibility @xcite . to overcome this limitation, several methods have been proposed to equip language-only representations with information from concurrent modalities, particularly vision. until not long ago, the standard approach aimed to leverage the complementary information conveyed by language and vision-for example, that bananas are yellow @xcite and rich in potassium (language)-by building richer multimodal representations @xcite . overall, these representations have proved advantageous over purely textual ones in a wide range of tasks and evaluations, including the approximation of human semantic similarity/ relatedness judgments provided by benchmarks like simlex999 @xcite or men @xcite . this was taken as evidence that leveraging multimodal information leads to more human-like, full-fledged semantic representations of words @xcite . more recently, the advent of transformer-based pre-trained models such as bert @xcite has favored the development of a plethora of multimodal models @xcite @xcite @xcite aimed to solve downstream language and vision tasks such as visual question answering @xcite and visual dialogue @xcite . similarly to the revolution brought about by transformer-based language-only models to nlp (see @xcite , these systems have rewritten the recent history of research on language and vision by setting new state-of-the-art results on most of the tasks. moreover, similarly to their language-only counterparts, these systems have been claimed to produce all-purpose, 'taskagnostic' representations ready-made for any task. while there has been quite a lot of interest in understanding the inner mechanisms of bert-like models (see the interpretability line of research referred to as bertology; @xcite and the nature of their representations @xcite , comparably less attention has been paid to analyzing the multimodal equivalents of these models. in particular, no work has explicitly investigated how the representations learned by these models compare to those by their language-only counterparts, which were recently shown to outperform standard static representations in approximating people's semantic intuitions @xcite . in this work, we therefore focus on the representations learned by state-of-the-art multimodal pre-trained models, and explore whether, and to what extent, leveraging visual information makes them closer to human representations than those produced by bert. following the approach proposed by @xcite , we derive static representations from the contextualized ones produced by these transformer-based models. we then analyze the quality of such representations by means of the standard intrinsic evaluation based on correlation with human similarity judgments. we evaluate lxmert @xcite , uniter @xcite , vilbert @xcite , @xcite , and vokenization @xcite ) on five human judgment benchmarks foot_0 and show that: (1) in line with previous work, multimodal models outperform purely textual ones in the representation of concrete, but not abstract, words; (2) representations by vokenization stand out as the overall bestperforming multimodal ones; and (3) multimodal models differ with respect to how and when they integrate information from language and vision, as revealed by their learning patterns across layers. ## data we aim to evaluate how the similarities between the representations learned by pre-trained multimodal transformers align with the similarity judgments by human speakers, and how these representations compare to those by textual transformers such as bert. to do so, we need data that (1) is multimodal, that is, where some text (language) is paired with a corresponding image (vision), and (2) includes most of the words making up the word pairs for which human semantic judgments are available. in what follows, we describe the semantic benchmarks used for evaluation and the construction of our multimodal dataset. ## experiments in our experiments, we build representations for each of the words included in our semantic benchmarks by means of various language-only and 3 the average frequency of our vocabulary words is 171 (min. 1, max. 8440). 61 words (3%) have frequency 1. multimodal models (section 4.1). in all cases, representations are extracted from the samples included in our dataset. in the language-only models, representations are built based on only the sentence; in the multimodal models, based on the sentence and its corresponding image (or, for vokenization, just the sentence but with visual supervision in pre-training, as explained later). since representations by most of the tested models are contextualized, we make them static by means of an aggregation method (section 4.2). in the evaluation (section 4.3), we test the ability of these representations to approximate human semantic judgments. ## results in language-only models we notice that bert evaluated on our dataset (hence, just bert) systematically outperforms glove. this is in line with @xcite , and replicates their findings that, as compared to standard static embeddings, averaging over contextualized representations by transformer-based models is a valuable method for obtaining semantic representations that are more aligned to those of humans. it is interesting to note, moreover, that the results we obtain with bert actually outperform the best results reported by @xcite using the same model on 1m wikipedia contexts (bert-1m-wiki). this is intriguing since it suggests that building representations using a dataset of visually grounded language, as we do, is not detrimental to the representational power of the resulting embeddings. since this comparison is partially unfair due to the different methods employed in selecting language contexts, we also obtain results on a subset of wikipedia that we extract using the method described for vico (see section 3.2), foot_8 and which is directly comparable to our dataset. as can be seen, representations built on this subset of wikipedia (bert-wiki ours) turn out to perform better than those by bert for wordsim353, simlex999, and simverb3500 (the least concrete benchmarks-see multimodal models turning to multimodal models, we observe that they outperform bert on two benchmarks, rg65 and men. though vokenization is found to be the best-performing architecture on both of them, all multimodal models surpass bert on men (see rightmost panel of figure 3 ; dark blue bars). in contrast, no multimodal model outperforms or is on par with bert on the other three benchmarks (figure 3 shows the results on wordsim353 and simlex999). this indicates that multimodal models have an advantage on benchmarks containing more concrete word pairs (recall that men and rg65 are the overall most concrete benchmarks; see concreteness our results show a generalized advantage of multimodal models on more concrete benchmarks. this seems to indicate that visual information is beneficial for representing concrete words. however, it might still be that models are just better at representing the specific words contained in these benchmarks. ≥4 out of 5 in @xcite . for each model, we consider the results by the layer which is best-performing on the whole benchmark. ## analysis we perform analyses aimed at shedding light on commonalities and differences between the vari-ous models. in particular, we explore how model performance evolves through layers (section 6.1), and how various models compare to humans at the level of specific word pairs (section 6.2). ## conclusion language is grounded in the world. thus, a priori, representations extracted from multimodal data should better account for the meaning of words. we investigated the representations obtained by transformer-based pre-trained multimodal models -which are claimed to be general-purpose semantic representations-and performed a systematic intrinsic evaluation of how the semantic spaces learned by these models correlate with human semantic intuitions. though with some limitations (see @xcite , this evaluation is simple and interpretable, and provides a more direct way to assess the representational power of these models compared to evaluations based on task performance @xcite . moreover, it allows to probe these models on a purely semantic level, which can help answer important theoretical questions regarding how they build and represent word meanings, and how these mechanisms compare to previous methods (see @xcite , for a similar discussion). we proposed an experimental setup that makes evaluation of various models comparable while maximizing coverage of human judgments data. all the multimodal models we tested-lxmert, uniter, vilbert, visualbert, and vokenization-show higher correlations with human judgments than language-only bert for more concrete words. these results confirm the effectiveness of transformer-based models in aligning language and vision. among these, vokenization exhibits the most robust results overall. this suggests that the token-level approach to visual supervision used by this model in pre-training may lead to more fine-grained alignment between modalities. in contrast, the sentence-level regime of the other models may contribute to more uncertainty and less well defined multimodal word representations. further work is needed to better understand the relation between these different methods.