On the Effects of Video Grounding on Language Models (2022)

Abstract: Transformer-based models trained on text and vision modalities try to improve the performance on multimodal downstream tasks or tackle the problem Transformer-based models trained on text and vision modalities try to improve the performance on multimodal downstream tasks or tackle the problem of lack of grounding, e.g., addressing issues like models’ insufficient commonsense knowledge. While it is more straightforward to evaluate the effects of such models on multimodal tasks, such as visual question answering or image captioning, it is not as well-understood how these tasks affect the model itself, and its internal linguistic representations. In this work, we experiment with language models grounded in videos and measure the models’ performance on predicting masked words chosen based on their imageability . The results show that the smaller model benefits from video grounding in predicting highly imageable words, while the results for the larger model seem harder to interpret.of lack of grounding, e.g., addressing issues like models’ insufficient commonsense knowledge. While it is more straightforward to evaluate the effects of such models on multimodal tasks, such as visual question answering or image captioning, it is not as well-understood how these tasks affect the model itself, and its internal linguistic representations. In this work, we experiment with language models grounded in videos and measure the models’ performance on predicting masked words chosen based on their imageability. The results show that the smaller model benefits from video grounding in predicting highly imageable words, while the results for the larger model seem harder to interpret.

## introduction a traditional language model is only exposed to textual data. while ample information exists in the form of text, some text-external knowledge might be missing, such as commonsense knowledge about the physical world, how objects look like, relate to each other, and how we interact with them. there is an abundance of work on trying to expose language models to other information sources and modalities, or in other words, grounding them; however, it is not clear how that would affect a language model in general. one promising modality to ground language models in is vision. previous work has studied the grounding of language models in visual input and how this affects their performance on downstream multimodal tasks, such as visual question answering and image retrieval @xcite @xcite , and on models' "understanding" of the world and their grasp of commonsense knowledge @xcite @xcite . our aim with this work is to see whether grounding in videos affects the performance of transformer-based language models on masked lnaguage modeling. masked language modeling is the task of predicting one or more masked tokens, given other tokens in the sentence. evaluating a model's performance on such cloze-test-style fillin-the-blank tasks is simple to implement and does not require expensive annotated data. still, it can provide us with helpful intuition about how models work. this method also makes it easy to compare language models grounded in different modalities without further fine-tuning them. we choose to experiment with videos rather than images because they contain more information about the physical world, and may be more useful for the development of spatial, temporal and causal reasoning. videos are also less studied in the literature. the masked words that we want the model to predict are chosen based on their imageability. imageability is a well-established notion from the field of psychology, defined as "the ease with which a word gives rise to a sensory mental image" @xcite . for instance, words like "to prance" and "oven" are considered highly imageable, while words like "to consider" and "problem" are not. imageability is highly correlated with concreteness, but the class of imageable words also includes abstract words, e.g. emotion words such as "anger". at the same time, this class does not include less experienced, yet concrete, words such as "armadillo" @xcite . we use a dataset consisting of 2,645 words annotated with their imageability scores @xcite to experiment with different types of models and investigate whether there is a performance difference between grounded and not grounded language models when predicting low-imageability versus high-imageability words. the words in our dataset are labeled with their parts-of-speech, which we will use in our experiments and analysis of the results. we continue the paper with explaining the models' architecture in §2 and the data sets used in §3. the experimental settings and the results are described in §4, where we also analyze the results and try to interpret them. in §5 we briefly discuss some related work. ## model we mainly follow the data preprocessing steps, the architecture, and the training regime of videobert model @xcite . we experiment with a pretrained bert-base model @xcite and distilbert @xcite . bert is essentially a transformer-based model @xcite pretrained with masked language modeling and next sentence prediction objectives, and distil-bert is the distilled version of the bert model, which has half the number of layers as bert-base. both language models are pretrained on the same data. as for the video features, we use the i3d model pre-trained on the kinetics dataset @xcite to encode video clips that are sampled at 20 fps and are 1.5 seconds long. we then apply hierarchical k-means clustering to the video features, setting the number of hierarchy levels to 4 and the number of clusters per level k to 12, which results in 12 4 = 20,736 clusters. henceforth, we use the closest cluster centroids as video tokens instead of continuous video features. as the output of the i3d model is of size 600, we use a fully connected layer to map to the embedding size of the respective model. we further train the pre-trained language models with a masked language modeling training objective with a masking probability of 0.15 for each modality. the embeddings of the word tokens (w i ) and the video tokens (v j ) are concatenated with a new special token [>] as the text-video separator. this results in an input i of the form the [cls] and [sep] tokens are the models' special tokens for classification and separation of sentences, respectively. the embedding weights and video features are frozen during training. the input i is then fed to the model to get the output o, which is mapped to the vocabulary space by means of a projection layer consisting of two fully connected layers (f c) and layer normalization: all the new layers and embedding weights are initialized randomly from a uniform distribution @xcite . the final objective is to maximize the log-likelihood l @xmath0; θ), where l is the masked token, and the xs are the input tokens, text or video, without the lth token. special tokens are never masked. we train two models with almost the same architecture, as described above, once only with textual input, and once with text and video input. the only difference in the structures is that the text model lacks the projection layer, which makes comparison between the models possible. the random seed is the same for both models all the time and changes by epoch. the models are trained using the adam optimizer with a learning rate of 10 -5 and batch size of 2 10 . we stop the training when the model's loss and accuracy plateaus on the validation set. the implementations and the pretrained weights of the hugging face transformer library @xcite are used in these experiments. ## data to get imageability scores for nouns and verbs, we use bird's dataset, in which words with different parts-of-speech, are rated with imageability scores from 100 to 700. for training and testing the models, howto100m @xcite dataset is used, which is a collection of 1.2m narrated english youtube videos from various categories. we randomly choose 55k videos from the dataset and split these into 45k videos for training, 5k for development, and 5k for testing, or in other words 4.7m samples for training and ∼500k for the other sets. to get some idea on how the howto100m data looks, we measure the mean imageability score on a random set of 300k tokens from the dataset, which was 454 on type level, and 366 on token level, which shows a high frequency of low imageability words in the dataset. there are a total of 892 verb types and 1,304 noun types in the bird dataset. we split the words in the bird dataset into low imageability (≤ 300) and high imageability (≥ 500) ones. this results in 114 low imageability and 511 high imageability types, or 67k and 92k tokens, respectively. the type-token ratio for low imageability words is 17 × 10 -4 , while being 55 × 10 -4 for highly imageable ones. ## results and analysis the model is fed with those sentences from the howto100m dataset that contains at least one word from bird's dataset. for each sample, we only mask one noun or verb at a time to make the analysis simple. the experiments are done on two models and in three different settings: the same settings are repeated for both distilbert and bert-base models. on the other hand, for bert, numbers are harder to interpret. we still see some increase for high imageability words, and more for verbs compared to nouns, but we see more or less the same amount of increase for low imageability words. it is hard to say why this is happening only for the bert model, but one reason might be that the model receives more learning signals during training when the sequences are longer (tv), hence the higher number of masked tokens. removing the video input from the input (tv-t) hurts the high imageability words the most, which shows the dependence of the model on the video signal. these results are not consistent with the distilbert model. one should bear in mind that the relative increase in accuracy for high imageability words, e.g., between t-t and tv-tv, is higher than for low imageability ones, as the accuracy for low imageability words is always considerably higher than that of the high imageability ones. for example, an increase of 1.4% in high imageability verb prediction accuracy in the distilbert tv-tv model is a 7.4% relative increase, while 1.0% for bert tv-tv low imageability verbs is only a 2.5%. one should also consider the fact that low imageability words have a much higher frequency in the data, which means the model has seen them more often. while the average imageability score in the bird dataset is around 460, the average token-based im-ageability score is around 360 for howto100m and some other datasets, including violin @xcite , and tvqa subtitles @xcite . the differences between different models' performances are not large, however, considering the size of the test set, they are quite significant. additionally, a bootstrap test always shows a p-value of smaller than 3.9e -5, which indicates a very high significance for all the results. we ran the bootstrap test as described in @xcite : a sample x (i) of the same size as the test set is drawn with replacement for @xmath1, and p-value is calculated as s/b, where s is the number of times where δ(x (i) ) > 2δ(x) holds. δ is the performance difference of systems a and b, and x is the original test set. comparing distilbert t-t and tv-tv shows that the words that benefit from the video signal are predominantly highly imageable ones, e.g., add, cook, plant, hair, bottom, turn, pour, house, remove, and ground, while low imageability words, such as see, want, go, way, take, and like, see a reduction in prediction accuracy. go is a special verb in the sense that it typically appears as an auxiliary verb to indicate the future tense, which is low in imageability. when removing the video signal in bert (tv-t), high imageability words see a reduction in accuracy, while it is the opposite for the low imageability ones. interestingly, the top 30 words that benefit the most from the video signal in distilbert (tv-tv) have a 63% overlap with the ones that see the most reduction when removing the signal in bert (tv-t). bert (t-t) is already good at predicting the words (high or low in imageability), and does not benefit from the video signal as distilbert. however, training it on video signals apparently makes it more dependent on them for predicting high imageability words, so that removing the signal hurts the performance. ## related work recent work on visual grounding has explored the effects of joint modeling of paired textual and visual modalities, with a focus on neural models based on the transformer architecture @xcite @xcite @xcite . there is also some work that goes deeper into the problem, such as @xcite , who studies the effects of visual grounding on text processing abilities of a language model using transferred and associative grounding, and how they improve text-only baselines, such as commonsense-related downstream tasks. another work is hendricks and nematzadeh (2021), who study how text-image pre-trained transformer models perform in situations that require "noun or verb understanding". according to them, such models perform poorly when evaluated on verbs compared to other parts of speech. @xcite experiment with an interactive simulated kitchen environment and conclude that certain machine learning models predict verbs less accurately than nouns, given a scene. they are motivated by work in psychology showing that predicting actions (verbs) is much harder than predicting objects (nouns) for people, given a video scene and the linguistic context of the word @xcite . in this work, we mainly followed videobert @xcite , but there are other methods of integrating text and video as well. one other work is hero @xcite , which does not use discretized video features, but continuous features with a regression loss. one other interesting work is clipbert @xcite , which tries to utilize sparse sampling to use fewer video frames to improve the text-video downstream tasks. there are also some work on joint representation of text and video, such as actbert (zhu and yang, 2020) and mil-nce @xcite . ## conclusion and future work although it is hard to draw strong conclusions based on these results, it might be that smaller models benefit more from video grounding than larger ones in the task of masked token prediction. the results are in line with the recent work on image grounding @xcite , which suggests that the visual input might not be exploited by the model to the fullest. while the results are not strongly indicative, these models are relatively small and training data size is also minimal. the data size is chosen based on the results in @xcite , who show that this much data should be enough to see some improvement. increasing the data and model size could be a direction for future work. another interesting research question that was not addressed in this paper is whether we really need to ground in videos for the model to gain the relevant knowledge, or can get the same results by using images or sampled key frame(s).