Enriching Language Models with Visually-grounded Word Vectors and the L ancaster Sensorimotor Norms (2021)

Abstract: Language models are trained only on text despite the fact that humans learn their first language in a highly interactive and multimodal environment where the first set of learned words are largely concrete, denoting physical entities and embodied states. To enrich language models with some of this missing experience, we leverage two sources of information: (1) the Lancaster Sensorimotor norms, which provide ratings (means and standard deviations) for over 40,000 English words along several dimensions of embodiment, and which capture the extent to which something is experienced across 11 different sensory modalities, and (2) vectors from coefficients of binary classifiers trained on images for the BERT vocabulary. We pre-trained the ELECTRA model and fine-tuned the RoBERTa model with these two sources of information then evaluate using the established GLUE benchmark and the Visual Dialog benchmark. We find that enriching language models with the Lancaster norms and image vectors improves results in both tasks, with some implications for robust language models that capture holistic linguistic meaning in a language learning context.

## introduction children learn their first spoken language in a highly interactive setting where generally the first words children learn are concrete words that denote physical objects, which is an important developmental step in child first language acquisition @xcite @xcite . this is partly because handling the symbol grounding problem-the ablity to connect symbolic knowledge of language with representations of the physical world @xcite )-must take place before children learn more abstract concepts later in their cognitive development @xcite . importantly, the physical world is not just the visual world; children learn that words ground into proprioperceptive states (e.g., a hand grasp around an object has specific muscle activations tied to the word grab), interoceptive states (i.e., affect and valence), as well as all other sensory modalities (e.g., the word stinky grounds into olfactory, the word loud grounds into auditory). these claims are evidenced in a large body of child development and cognitive science literature. @xcite , for example, identified that babies' experience of the world is profoundly multimodal: babies live in a physical world full of rich regularities that organize perception, action and thought; babies learn in a social world to learn a shared linguistic communicative system that is symbolic. furthermore, a growing body of literature from linguistics and computational linguistics makes a strong case that the process of language learning (indeed, general human cognition) is embodied, interactive, and enacted; i.e., movement in the world is required @xcite @xcite @xcite @xcite ; see also the prior work in developmental robotics research; e.g., @xcite , chapter 7. 1 taken together, it is clear that aspects of the physical world are necessary for holistic knowledge of semantic meaning, which has implications for how language is modeled computationally. in particular, what does this mean for language models that are trained purely on text (likely largely written by adults), such as bert @xcite or gpt-3? these models have clearly led to important advances for natural language processing tasks and applications, but it is also clear that language models trained only on text are missing critical semantic information @xcite . in this paper, we contribute to a growing body of recent work that attempts to addresses these limitations by (1) leveraging multimodal and sensorimotor knowledge of the lancaster sensorimotor norms @xcite and (2) using vectorized representations of images by treating both (1) and (2) as embeddings of language models for glue and visual dialog benchmarks. in the following section, we explain related work-a growing body of literature that is adding multimodal information to language models, then we explain our two embeddings that we will use. we explore how these embeddings can be used to enrich the elec-tra language model's pre-training and fine-tuning, and evaluate on the glue benchmark (experiment 1, section 4), and how they can be used to replace input embeddings for a pre-trained roberta model for the visual dialog task (experiment 2, section 5). our experiments shed light on how useful multimodal information can be in a task that is text-only (experiment 1) and a task that is multimodal (experiment 2). our results show that our parsimonious method to unifying vision (and sensorimotor knowledge) in existing language models shows improvements in multimodal benchmarks with accessible hardware (i.e., a single gpu) as a step towards models that can be trained in settings similar to that of child language learners. ## related work language models are trained on text. @xcite took up the question do words inherit sensorimotor activation from purely linguistic context? and showed that experience is necessary for reactivating experiential traces, but this reactivation is not a necessary condition for understanding the corresponding aspects of word meaning. we take this to mean that humans are very adept at learning new concepts from language exposure alone (i.e., abstract concepts); e.g., someone who has never seen a zebra before, but hears them described as "horses with vertical black and white stripes" can compose a connotation of what zebra denotes without direct visual exposure. however, this only works if an agent that has learned the language has the knowledge of horses, black, white, stripes, and vertical concepts-i.e., via direct experience, not just through linguistic exposure or encyclopedic definitions. these claims are further backed up by neuroscience research that showed that neural assemblies encode concrete content words (i.e., words that denote visual objects) and verbs (i.e., words that denote actions) are learned and represented in different brain regions @xcite . @xcite provides a recent primer and overview of research that has attempted to uncover strengths and weaknesses of bert and related language models (so-called bertology). while our work does fit into that growing body of literature, our criticisms on current language models specifically lies in the fact that they are only trained on easy-to-obtain text. this criticism is born out in @xcite which showed that bert can guess affordances and properties of objects because that information can be found in text (e.g., a typical chair has the affordance of being sittable, and a property of having legs), but has no notion of how objects are related semantically to each other, and @xcite further showed that real-world perceptual properties are likely to be assumed instead of inferred. furthermore, bender and koller (2020) make a strong case that bert learns form instead of meaning, and while the fact that bert performs so well on many tasks is difficult to dispute, models trained on text are missing semantic information crucial for holistic language understanding. since before bert which has proven powerful in many language processing tasks, efforts have been made to encode multimodal (i.e., more than just text as a learning modality) information into embeddings and language models @xcite @xcite and recent, continued efforts towards bridging grounded visual representations to distributional representations of word meanings give credence to the claim that text-only models like bert are missing crucial semantic information because enriching bert with visual information improves performance in several known tasks @xcite @xcite . these models usually treat language and vision as separate pipelines; our method directly endows the language model with visual and sensorimotor knowledge. ## data in this section, we motivate and introduce of multimodal information we will use in our experiments. the lancaster sensorimotor norms the lancaster sensorimotor norms @xcite provide ratings (means and standard deviations) for 40,000 english words along dimensions of embodiment which capture the extent to which a concept is experienced across 11 different sensory modalities, and measures derived from those categories, listed below (each has an example word that rates highly for that modalitiy): the last three can be seen as aggregates from the 11 modalities; they also have .action values representing the extent to which a concept is experienced as an action (as opposed to .perceptual), and .sensorimotor values representing the extent a concept is experience as sensorimotor. as these norms were derived from surveys given to adults, these norms represent the degree to which the survey participants assigned those words to those categories. though this does not represent a neurophysiological grounding of words to those modalities learned through interaction and embodiment, this serves as a useful approximation. the final set is a vocabulary of 39,707 words (after removing rows which had null values), each represented as a vector of length 39 (i.e., 11 mean, 11 stdev columns; max-strength, minkowski, and exclusivity columns for different ways of aggregating the modalities). we normalize each value in the vector independently to a value between 0-1 by dividing each value over its max value. we call this the lancaster vectors. we performed t-sne on the lancaster vectors (mapping to 2 dimensions) to determine if clus-ters would reveal any intuitions about the kinds of semantic relatedness that the words might have with each other. some clusters emerged such as foods (presumably because they have similar gustatory ratings), leg-movement verbs (e.g., walk, jump, sit), colors with eye-related words (e.g., purple, green, blue, dark, see, eyes), soft things (e.g., hug, tummy, pillow, clothes), audio-related words (e.g., talk, story, sound, music, lie, say), among others. words-as-classifiers image vectors the words-as-classifiers (wac) approach to grounded semantics is quite simple: train a binary classifier for each word in a corpus where the features to that classifier are derived from images @xcite . each classifier is given positive and negative examples of visual denotations of each word by the images and learns a "fitness" score by the classifier. for example, the red classifier is given images of objects that are referred to as red in a corpus, and randomly assigned negative examples of things that are not referred to as red, as depicted in figure 1 . we follow @xcite and use google image search to find images using the bert vocabulary, resulting in 27,152 words and corresponding images (some words did not result in images, and we did not download images for filler words). for each word, we perform an image search and download the top 100 images. we then follow @xcite and process each image by passing them through the recent clip @xcite convolutional neural network (trained on imagenet, using clip's vit-b/32 model), yielding a vector of size 512 for each image. we use the 100 images as positive examples for each term in our vocabulary and randomly select three negative examples for each positive example. we then use a logistic regression classifier (@xmath0.25, max @xmath1, one for each word, trained on the images for each word. after training, we then follow @xcite and extract the coefficients to arrive at a vector of size 513 (all coefficients plus the bias term) which we use in our evaluations below. we call these the wac vectors. the wac model is useful because, as explained in @xcite , the classifiers can actually identify objects (something that language models cannot do on their own), the coefficients represent a computed word intension, new words in a vocabulary can easily be added without retraining all other classifiers including adjectives like red which are often missing from pre-trained object classifiers, and the classifiers are effectively learned with only a few examples, making it effective for fast learning of concrete, grounded concepts. however, the wac model suffers from two assumptions: first, that all words have concrete, visual denotations even though many abstract words like utopia clearly do not, and that all words are independent of each other in terms of linguistic context. we hypothesize in both experiments below that these coefficients used as vectorized embeddings will be useful to a text-only language model because they add necessary visual information; the language model complements wac by using linguistic context (i.e., text) for training, overcoming wac's assumptions. in this experiment, and crucially for our ongoing work that aligns with child-inspired language acquisition, we use electra @xcite ) as a language model because it has been shown to be trainable with smaller amounts of data than other language models, yet yield respectable results and can be trained using a single gpu. task & procedure @xcite introduced the glue benchmark which consists of nine english sentence understanding tasks covering several domains (e.g., movie reviews and online question answering). we opt for this benchmark because of its coverage over several domains and to show that adding multimodal knowledge improves tasks that are based on text. 2 our aim is to achieve improved results over the text-only baseline with a specified number of training steps using the openwebtext data for training. 3 we report results on the development set, as done in @xcite . we only report the results for the mrpc (a paraphrase task that uses accuracy and f1 metrics), cola (a grammatical acceptibility task; uses matthew's correlation), and wnli (ambiguity resolution; uses an accuracy metric) tasks because they are sufficient to illustrate the utility of our method when applied to electra. to give electra knowledge about additional modalities from the lancaster and wac vectors, we tie the vectors to the the weights of the generator and discriminator of electra depicted in figure 2 , and vary whether the embeddings are frozen or not during pre-training, then train for 100,000 steps. 4 we then fine-tune the resulting electra model on the glue tasks using the multimodal vectors following standard fine-tuning protocols; that is, we add a linear layer with a softmax to the pre-trained model and use the adam solver with a learning rate of 2e-5 for 3 epochs. as the wac vectors were larger than electra's expected embedding size of 128, we applied umap to reduce the dimensionality to 128; similarly for the wac and lancaster concatenated embeddings. for cases where there was no vector for a word (e.g., the [unmapped] words or words outside of the vocabulary of the lancaster vectors), we simply used zero vectors. for lancaster vectors, we set the electra embedding size to 39. we explored freezing the embeddings; our hypothesis is that not freezing the embeddings will lead to better results because the training regime can overpower the embeddings, but retain the multimodal knowledge. for a broader comparison, we also compared to glove @xcite and several ablations where we concatenate multimodal vectors with the glove vectors (we used the evaluation script for glove provided by @xcite ). we also use the same training and evaluation regime for the wac and lancaster vectors, and a concatenation of the two, on their own treating them as word-level embeddings similar to glove. ## experiment 2: replacing roberta embeddings, fine-tuninng on visual dialog the evaluation in experiment 1 was made up of text-based tasks. in this experiment, we use an evaluation that requires knowledge of the visual world by evaluating the lancaster and wac vectors on the visual dialog task @xcite , termed visdial. moreover, experiment 1 used pre-training on a subset of the data for only 100,000 steps. in this experiment, we evaluate using a fully pre-trained roberta model by replacing its embeddings with the wac and lancaster vectors. task following @xcite , given an image, dialogue history consisting of questionanswer pairs, and a follow-up question about the image, the task of visdial is to predict a free-form natural language answer to the question. the visdial dataset introduced in @xcite also includes evaluation metrics and human-annotated answers to the natural language queries about the image. five human annotators identified which responses out of 100 candidates could be considered correct. this allows multiple answers to be correct (e.g., yes and yeah are semantically identical). ## conclusion the main contribution of this paper is to explore using the lancaster sensorimotor norms and the words-as-classifiers model as vectorized knowledge from the physical world on the glue and visual dialog tasks. lancaster norms performed well on their own in one glue task compared to other word embeddings like glove, and coupled with the wac vectors as the embedding in an electra model, they performed respectably on the glue task. the wac vectors, when used as embeddings in the roberta model performed well on the visual dialog task, particularly when the vocabulary was more restricted to the age of acquisition vocabulary. crucially, this work differs from other visually grounded models because the grounded knowledge is part of the language model itself (i.e., the embeddings) rather than computed in parallel and added for a task-specific purpose. moreover, standard language models cannot actually identify denotations when they are present; i.e., electra and roberta are not actually capable of determining if an object is red or soft from observing that object-a basic ability for a language learning child-simply because those models cannot observe the world outside of text, though the purpose of the wac (and models like vilbert) model is to do just that: identify denotations; by coupling wac with electra and roberta, both models can make use of that capability. this work is critical in our ongoing efforts towards a model that learns language in a co-located setting in an embodied platform. in particular, our knowledge from this paper informs us that the electra model with embeddings tied to wac classifier weights is a good candidate for live interaction of a robot that is learning words from a human collaborator because the electra-wac model can function with small amounts of data and the embedding layer can successfully be tied to weights of the wac classifiers. we leave implementation and evaluation of this model on a robotic platform for future work.