Grounded and well-rounded: a methodological approach to the study of cross-modal and cross-lingual grounding (2023)

Abstract: Grounding has been argued to be a crucial component towards the development of more complete and truly semantically competent artificial intelligence systems. Literature has divided into two camps: While some argue that grounding allows for qualitatively different generalizations, others believe it can be compensated by mono-modal data quantity. Limited empirical evidence has emerged for or against either position, which we argue is due to the methodological challenges that come with studying grounding and its effects on NLP systems. In this paper, we establish a methodological framework for studying what the effects are—if any—of providing models with richer input sources than text-only. The crux of it lies in the construction of comparable samples of populations of models trained on different input modalities, so that we can tease apart the qualitative effects of different input sources from quantifiable model performances. Experiments using this framework reveal qualitative differences in model behavior between cross-modally grounded, cross-lingually grounded, and ungrounded models, which we measure both at a global dataset level as well as for specific word representations, depending on how concrete their semantics is.

## introduction some researchers have explicitly argued that achieving natural, human-like linguistic behavior requires a richer input scheme than pure text, and specifically that perceptual input is necessary. this has been one of the arguments of the well-known position paper on the inherent limitations of language modeling @xcite . luckily, recent developments have prepared the ground for empirical computational testing of this line of argumentation: pretrained transformer-based models have been adapted to handle different modalities, ranging from natural language text to images, programming code, and to video game controls (e.g., @xcite @xcite . these approaches to multimodality are often practically motivated: the initial research question of @xcite is specifically whether a single model can handle multiple types of inputs, that of @xcite is specifically whether pretrained language models could be co-opted for human-robot instructions. this research field is vibrant, and novel insights have been plenty: to date, researchers have put forth theoretical arguments that we expect qualitative differences among models due to grounding, and have demonstrated practical use-cases where handling multiple types of inputs is desirable and feasible. there is however a dearth of empirical studies bridging the two: a dearth of studies assessing whether, when it comes to language modeling, grounding systematically yields different textual predictions. in this paper, we therefore take one step back and focus on what evidence there might be that grounding leads to an observable qualitative difference in model behavior, beyond the trivial quantitative expectation that richer input might lead to better downstream performance. this sort of endeavor is difficult to set up, due to interrelated confounding factors one is confronted with. firstly, data for different modalities tend to not be easily comparable-images are after all radically different from text. as such it is difficult to assess whether models trained on different modalities are of similar quality-and therefore useful points of comparison are hard to come by. secondly, there is somewhat of a lack when it comes to defining what reasonable expectations with respect to grounding ought to be and how to measure them. however, none of these hurdles are insurmountable, and in the present paper we propose a methodological framework for studying the effects of grounding. this framework is built upon a distinction between two notions of grounding (section 3.1): one weaker and one stronger, depending on whether we can disentangle input representation from task processing. on a more practical level, we focus on multimodal and multilingual data, and define a procedure to construct samples of models so that they are broadly comparable. we apply this approach to cross-lingually grounded @xcite , cross-modally grounded, and ungrounded models so as to better characterize how these different groups of models compare to and contrast with each other. our experiments foot_0 demonstrate that access to richer and more diverse inputs impacts model behavior, even when factoring in model performance-which we observe both on a global, dataset-wide level and more narrowly with respect to a lexicon of concrete and abstract words. ## related work the idea that language data alone do not suffice to build full-fledged semantic representations has been discussed by numerous philosophers, with various areas of focus. one controversial landmark work is @xcite , which introduces a thought experiment based on translation to argue that systems that only ever deal with symbols cannot gain understanding of their surroundings. @xcite discusses the need of perceptual input; @xcite provides a thought experiment for monolingual situations. this line of thought has led to more modern theory-focused approaches, i.e., nlp scientists trying to repurpose the cognitive science concept of grounding to make sense of the behavior of neural language models. one particular discourseprovoking piece is that of @xcite , which questions whether modern language models can be expected to display some form of understanding despite lacking grounding. it is worth echoing some remarks of @xcite , who outline that, while cognitive science defines grounding as establishing a common ground for communication, the nlp community has adopted a specific, more restricted definition of grounding as linking concepts from different sources such as text (single and multiple languages), images, video and speech, and so on. the opposite trend, viz. researchers interested in perceptual information in human development using neural networks as models, also exist: see for instance the work of @xcite ; @xcite . a related trend in computational semantics relates specific aspects of meaning to situated information @xcite . we also refer the reader to the survey of @xcite on recent visually grounded models of spoken language from the nlp and cognitive science communities. the works cited above focus primarily on theoretical aspects of grounding; other works take a more practical angle. these latter fall into two broad categories: (i) works that probe for specific aspects of grounding, e.g. patel and pavlick (2022); tenney et al. (2019); hwang et al. (2021) ; and (ii) works tackling engineering challenges and opportunities that come with systems handling multiple channels of inputs (e.g., @xcite @xcite @xcite . in recent years, most research in this area has converged on transformerbased systems which align inputs from different channels into a shared semantic space to enable multimodal interaction. most research on grounding involves signal from different modalities, most commonly text and images. cross-lingual grounding, on the other hand, is still an under-researched area, even though already @xcite remarked on the usefulness of bilingual lexicons to study fine-grained semantic differences. the idea that cross-lingual studies could provide insights into semantics eventually informed practical applications, e.g. @xcite used translations as a source of semantic information that can elucidate the relations between words that are hard to specify based on a monolingual corpus. this idea also underlies works that use multilingual corpora to automatically construct language-specific wordnets (e.g., @xcite . more recently, @xcite demonstrates how a model trained on a massively multilingual corpus can give rise to an 'interlingua' space that enables translation between language pairs that are not in the training corpus, linking cross-lingual grounding to another core concept long studied in machine translation @xcite . there is nonetheless an important difference between cross-modal and cross-lingual grounding: 2 while the former is generally understood as linking symbols to non-symbolic external data such as perceptual data, @xcite proposal slightly alters the definition of grounding as possessing representations that can "resolve language-internal ambiguities" ( §1). these two positions are not as incompatible as one might initially think: for instance @xcite does concede that symbolic meaning can be derived from " @xcite in a first language and in real world experience and knowledge" ( §2.2). in that respect, tiedemann's view can be understood as positing that the availability of a first language is already sufficient for grounding, regardless of real world experience and knowledge. ## difference in behavior the first question we address is as follows: are we justified in expecting that different input sources should lead to qualitatively different predictions, after having controlled for model accuracy? re-mark that in principle, we have no reason to expect qualitatively different predictions from models that reach equivalent accuracy: all of these models are trained to predict the same outputs and are confronted to equally ambiguous inputs. to answer this, we construct three samples of models to compare: (i) a sample where we compare p, c, and t single-task models; (ii) a sample where we compare single-task p models to multi-task p∨c, p∨t, and p∨c∨t models; and (iii) a sample comprised of single-task p models and multi-modal p∧c, p∧t, and p∧c∧t models. we then compute agreement rates for every pair of models m 1 , m 2 in each sample-i.e., how often they make the same prediction, given the same target prefix. for multi-task models, we use the p features to derive predictions, both for constructing the sample and computing agreement. the distributions of agreement scores are shown in figure 2 . we observe that most comparisons involving models of the same setups tend to yield higher agreement scores than most comparisons of models from different setups. this leads to a very salient contrast for single-task models in figure 2a : pairs of models of the same setup yield higher agreements than pairs of different setups (mann whitney u, @xmath0, common language effect size @xmath1.88). this also holds for multi-modal models in figure 2b (@xmath2, @xmath3.62) and even multi-task models in figure 2c (@xmath4, @xmath5.52) though the effect is much weaker and mostly noticeable through third quartiles. for single-task models, we also see that comparing two models trained on text yields higher scores than comparing text and video models. c / t c / p p / t c / c p / p t / t task pair 0.60 0.65 0.70 0.75 0.80 agreement single-task models same tasks different tasks (a) single-task sample p c / p t p / p t p / p c t p c / p c t p / p c p c / p c p / p p t / p t p c t / p t p c t / p c t task pair 0.60 0.65 0.70 0.75 0.80 agreement multi-modal models same tasks different tasks (b) multi-modal sample p / p t p / p c t p / p p / p c p c t / p t p t / p t p c / p t p c t / p c t p c / p c t p c / p c task pair 0.60 0.65 0.70 0.75 0.80 agreement multi-task models same tasks different tasks (c) multi-task sample to sum up, even after controlling for accuracy, we can highlight a qualitative difference in behavior: models of the same setup make more similar predictions than models of different setups. the contrast is starkest with single-task models; and while this effect tends to be less pronounced in multi-modal models and very subtle in multi-task models, we can still observe it. it should not be surprising that models in the multi-task sample barely display a difference: recall that these models all use the same inputs (p features) on top of being conditioned on the same target prefix. that we can register a difference at all after controlling for accuracy suggests that the use of different input sources impacts the learned parameters. this effect cannot be subsumed to a mere difference in inputs that translates into different outputs; rather, it has to be construed as guiding models towards different behaviors. referring back to our theoretical framework in section 3.1, this is evidence in favor of the stronger notion of grounding that we outlined. in multi-modal models, given their implementa-tion as transformer decoders, we can further highlight the importance of the different input sources in model behavior by focusing on source-attention patterns. we compute the attention paid to features from p, c, and t, and contrast whether similar attention patterns entail similar behaviors. while attention-based overviews are known to have limitations @xcite @xcite , they do offer a practical insight into the models' computation. in practice, for every multi-modal model and every item in our test set, we can compute the average attention per feature type in p, c, t. as such, every multi-modal model has a corresponding 3d vector describing its general attention pattern. these are displayed in figure 3a . given that the attention-pattern vectors' components sum to one, they furthermore lie on the positive quadrant of the ℓ 1 unit-sphere; i.e., the equilateral triangle with summits (1,0,0), (0,1,0), (0,0,1). a visualization of the corresponding vector population is shown in figure 3b : each point corresponds to a all multi-modal models exhibit different behaviors: models trained on p and c tend to attend to paraphrasing features rather than captioning features; models trained on p and t tend to display more balanced patterns; attention patterns for models trained with all three input spaces p, t and c vary more along how much they attend c and t features rather than how much they attend to p features. remark that some models trained with c features may entirely ignore captioning features, whereas the same never holds for paraphrase and translation input features. this likely highlights that the nlg task at hand tends to be more straightforward using cross-lingual, rather than cross-modal, features. beyond these general observations, we focus on whether we can establish a correlation between distance in attention patterns and pairwise agreement scores. we expect an anti-correlation, since a lesser distance between attention patterns ought to correspond to a greater degree of agreement. we consider three measures of vector dissimilarity: cosine distance, viz. 1cos(a 1 , b 2 ), as well as ℓ 1 and ℓ 2 distances. corresponding results are shown in ## concreteness we have established that there is some evidence in favor of a stronger notion of grounding, that is to say, that we expect the functions described by grounded models to differ across setups. we now turn to whether we can connect these qualitative functional differences with other prior observations. namely, we focus on concreteness @xcite , and study how it might affect model parameters (viz., the input embedding layer) as well as model activations (viz., the last hidden state before vocabulary prediction). 8 we have replicated these findings with kl-divergences, however other statistical indicators need not align with this behavior. in particular, differences in terms of entropy are rarely statistically significant, tentatively suggesting that grounding does not reduce uncertainty. in detail, we only observe a significant difference for single-task models (kruskal-wallis h-test: @xmath6.011), which is owed to c models having lower entropy than p or t models (mann-whitney u-tests: p vs c @xmath7.04, @xmath8.63; c vs t, @xmath9.0031, @xmath10.31. this could be a side effect of selecting checkpoints at different number of epochs: if some models have been optimized for longer, they may have converged onto more peaked distributions. we leave a proper evaluation of this aspect for future work. we use concreteness ratings of words from @xcite . these ratings were gathered through crowd sourcing where annotators were instructed to rate how concrete the meaning of each word is on the scale of 1-5. higher ratings characterize concrete words, which refer to a perceptible entity and can be experienced directly through the senses (experience-based). conversely, words with low ratings are abstract, referring to concepts that cannot be directly experienced; instead, their meaning has to be derived through other words (language-based). for simplicity, we binarize this dataset by taking the 1000 lowest rated words to construct a set of abstract words, and similarly the 1000 highest rated to construct a concrete set. we compare the embeddings of concrete and abstract words learned by models trained on the different setups that we outlined. we consider two scenarios: in the first, we assess whether concrete and abstract word sets cluster into two (and only two) distinct groups of embeddings: more precisely, we evaluate the compactness of each group through their silhouette scores. in the second, we relax the assumption that embeddings need form only two clusters: instead, we perform clustering using affinity propagation @xcite , a clustering method that does not rely on a predefined number of clusters, and evaluate the purity of the formed clusters. 9 the multimodal models exhibit a somewhat different behavior. all models have similar silhouette scores and purity scores when using the embed- ## conclusion in the present work, we have proposed to study cross-lingual and cross-modal grounding by constructing samples of comparable models. our experiments have revealed how populations of models using different types of input sources make different textual predictions. at a global dataset level, we have observed that models tend to agree less when they come from different setups than when they share the same training conditions (section 4). adopting a more linguistically motivated approach, we have also discussed how the representations of concrete and abstract words are impacted by grounding (section 5): in particular, we can highlight that our captioning-like task was the most impactful when it came to cleanly delin-14 p vs p∧c∧t @xmath11.3 • 10 -3 ; p∧c vs p∧c∧t @xmath12.5 • 10 -2 ; p∧c∧t vs p∧t @xmath13.6 • 10 -3 eating abstract and concrete words. the methodology we outlined in section 3.4 allowed us to select groups of models so as to guarantee they are of similar accuracy, thereby teasing apart variation due to performance from that due to the type of input sources a model receive. this methodology can easily be adapted to any other situation where one wishes to control for a specific variable. in our specific case, it allows us to smooth over the necessary differences between text and video features and provide a more principled outlook on grounding. we believe the adoption of methodologies such as the one proposed here to be invaluable to studies on grounding: they allow for systematic empirical comparisons which are often missing, as most studies focus on foundational models or theoretical arguments. beyond these methodological contributions, the core finding our experiments suggest is that the effects of grounding on neural network models cannot be subsumed to simply learning adequate representations: rather, we observe results consistent with a stronger notion of grounding, where we cannot disentangle the effects of learning to represent inputs of different modalities or languages from that of learning how to solve the task at hand. a second conclusion that emerges from our experiments is that the effects of cross-lingual grounding are empirically distinct from that of crossmodal grounding. cross-lingually grounded models were found in several instances to be more closely in line with models trained on a paraphraselike task. a better characterization of the difference between cross-lingual and cross-modal grounding is one direction we hope to address in future work.