Sequential Compositional Generalization in Multimodal Models (2024)

Abstract: The rise of large-scale multimodal models has paved the pathway for groundbreaking advances in generative modeling and reasoning, unlocking transformative applications in a variety of complex tasks. However, a pressing question that remains is their genuine capability for stronger forms of generalization, which has been largely underexplored in the multimodal setting. Our study aims to address this by examining sequential compositional generalization using CompAct (Compositional Activities), a carefully constructed, perceptually grounded dataset set within a rich backdrop of egocentric kitchen activity videos. Each instance in our dataset is represented with a combination of raw video footage, naturally occurring sound, and crowd-sourced step-by-step descriptions. More importantly, our setup ensures that the individual concepts are consistently distributed across training and evaluation sets, while their compositions are novel in the evaluation set. We conduct a comprehensive assessment of several unimodal and multimodal models. Our findings reveal that bi-modal and tri-modal models exhibit a clear edge over their text-only counterparts. This highlights the importance of multimodality while charting a trajectory for future research in this domain.

## introduction humans possess a remarkable ability to rapidly understand new concepts by leveraging and combining prior knowledge. this compositional generalization allows for an understanding of complex inputs as a function of their constituent parts. for instance, having grasped the meanings of "dax" and "walk twice" humans can effortlessly understand "dax twice" (lake and @xcite . however, even as neural networks trained on increasingly larger datasets achieve impressive results across a wide range of tasks, their ability to compositionally generalize remains limited. recently, the research community has demonstrated growing interest in evaluating models under different distributions, such as temporal shifts @xcite , or unseen compositions (lake and @xcite @xcite . within the domain of multimodal learning, prior investigations into compositionality have primarily delved into visual grounding @xcite , downstream multimodal tasks like image captioning @xcite and visual question answering @xcite , or vocabulary acquisition from videos @xcite or with interactive agents @xcite . addressing the challenge of compositional generalization in the context of multimodal models is increasingly important with the recent advances in large multimodal foundation models, such as @xcite , flamingo @xcite , and idefics @xcite . experimenting with closed-source or proprietary models introduces challenges, including reproducibility issues, associated costs, and limited transparency regarding their development and training methodologies @xcite . this inspires us to investigate the potential of open-source large multimodal foundation models -idefics in particular, for multimodal sequential compositional generalization, which we define as the model's capability to understand and generate predictions about novel compositions of primitive elements derived from sequential multimodal inputs -for instance, video data wherein actions unfold in a discernible order. consider the process of cooking onions: one typically needs to peel and slice an onion before frying it in a pan. our central inquiry revolves around the proficiency of models in comprehending such sequential and compositional activities. in this study, we introduce compact (compositional activities) to investigate multimodal sequential compositional generalization, a uniquely constructed compositional dataset curated from the epic kitchens-100 dataset @xcite . the ek-100 dataset encompasses 100 hours of egocentric video footage from 45 distinct kitchens, documenting people performing routine household tasks. each video contains three streams of information: visual data in the videos; audio data involving non-narrative audio elements -such as the sounds associated with chopping an onion; and textual data in the form of short, crowd-sourced descriptions of the depicted activities, like "slice the carrot", "pick up the milk", or "wash the plate". from these descriptions, individual verb and object concepts such as slice, pick up, wash, and carrot, milk, plate are extracted. the compositional splits are devised based on the verb and object concepts gleaned from the video descriptions, resulting in training and evaluation sets showcasing similar distributions of atomic concepts but featuring varied combinations therein. consequently, models should compositionally generalize from the training data. aligning with the "dax twice" principle from lake and @xcite , if a model has been large-scale pretrained models are prompted for a task in a zero-shot setting, given a support set of task demonstrations. in our study, we conduct a comprehensive evaluation of publicly available models, encompassing encoder-only pretrained models such as im-agebind @xcite and merlot reserve @xcite in addition to (multimodal) large language models (llms) like llama2 @xcite and idefics @xcite . these models exhibit versatility in processing various combinations of input streams, ranging from language-only to combinations like video + language, video + audio, and even video + language + audio. our key experimental finding indicates the formidable challenge that all of these models face in mastering compositional generalization. yet, it becomes abundantly clear that the utilization of multimodal input sources yields discernible advantages, suggesting a promising direction for refining future models. 2022). as previously noted, each video in the ek-100 features first-person perspectives of unscripted kitchen activities occurring within natural household environments. a video is composed of a sequence of shorter clips, represented as @xmath0, . . . , v k ), each of which is accompanied by manually annotated english narrations, denoted by x 1 , . . . , x k , describing the activities within. additionally, these clips are integrated with audio tracks, a 1 , . . . , a k , which contain the sounds of occurring actions. we define an instance in the dataset as a combination of video -audio -narration: (v, a, x) each instance consists of a window of 4 clips, with the initial 3 clips serving as context and the last one designated for prediction. given this dataset, our primary focus is to facilitate researchers in exploring how multimodal models compositionally generalize to unseen combinations of concepts. we meticulously curate the compact dataset to ensure a specific property: the individual concepts are consistently distributed across training and evaluation sets, while their compositions are novel in the evaluation set. this design mandates that a model should exhibit systematic generalization when interpreting the evaluation set. to illustrate, refer to the example shown in fig. 1 . during training, the model comes across nouns such as celery, garbage bin, fridge, onion, and verbs including take, wash, close, put down. in our evaluation set, we seek instances where an object-verb composition has not been previously encountered during training; for example, the unique pairing of the cut with the celery. ## multimodal sequential compositional generalization anticipating what comes next is a fundamental aspect of human cognition @xcite . from a cognitive perspective, it also serves as an engaging training paradigm @xcite . in multimodal sequential compositional generalization, we seek to understand the extent to which multimodal foundation models are capable of understanding what comes next in activity sequences. we propose two tasks to measure multimodal sequential compositional generalization in the com-pact dataset: (i) next utterance prediction, and (ii) atom classification. ## models in our experiments, we benchmark a variety of neural network models on the proposed next utterance prediction and atom classification tasks, including several text-only (unimodal) and multimodal models for better understanding the importance of different modalities in compositional generalization. ## experimental setup evaluation metrics: we use unigram bleu @xcite , exact match (em), categorical accuracy (ca) and bertscore @xcite metrics. the reported values represent the mean and standard deviation across 3 separate runs. in llama2 and idefics, we use nucleus sampling instead of separate runs. for em, we calculate the accuracy between the generated text sequence and the ground truth. ca uses the verb and noun categories in ek-100 and calculates the accuracy based on category match between the prediction and ground truth, e.g. the verbs slice, dice, and chop fall into the same verb category cut, and the nouns cheddar, paneer and parmesan are grouped into the same noun category cheese. therefore, the slice paneer prediction is deemed accurate if the ground truth is dice parmesan. ## conclusion in this paper, we investigate linguistic compositionality and systematic generalization in a grounded setting for multimodal sequential compositional generalization. we show how a multimodal dataset can be utilized as a challenging test bed for this purpose. we design next utterance prediction and atom classification tasks adopting a methodical approach in generating the training, validation and test sets for our compositional splits. we experiment with several baseline models and investigate models' ability to generalize to novel compositions and show how multimodal data can contribute towards solving systematic generalization problem and highlight major challenges. we hope our work will stimulate further research in these directions. 5599 despite the promising results, there are a few limitations of our work. in our work, we introduce a novel dataset called compact carefully curated from the ek-100 dataset @xcite , which involves videos of daily kitchen activities, to dissect the impact of visual and auditory signals on linguistic compositionality. hence, our conclusions may hinge upon certain domain-specific variables. it could be interesting to conduct future studies in an open-domain setting which might unravel additional insights e.g. @xcite . we investigate several different multimodal models for both the next utterance prediction and atom classification tasks. however, it is important to note that for multimodal learning how to integrate different modalities is considered as an open research problem. in the literature, different strategies for multimodal data fusion have been proposed. our experimental analysis could be further extended by considering some models that fuse the modalities in a way different than ours. more interestingly, from a systematic generalization point of view, an analysis could be carried out to explore the most effective fusion scheme. finally, we acknowledge the textual utterances that we use in our work are inherently simplistic and do not capture all of the complexities in natural languages. consequently, extending this work to a more natural source of language data that mirrors those complexities could be quite interesting direction for future research.