Can We Use Small Models to Investigate Multimodal Fusion Methods? (2022)

Abstract: Many successful methods for fusing language with information from the visual modality have recently been proposed and the topic of multimodal training is ever evolving. However, it is still largely not known what makes different vision-and-language models successful. Investigations into this are made difficult by the large sizes of the models used, requiring large training datasets and causing long train and compute times. Therefore, we propose the idea of studying multimodal fusion methods in a smaller setting with small models and datasets. In this setting, we can experiment with different approaches for fusing multimodal information with language in a controlled fashion, while allowing for fast experimentation. We illustrate this idea with the math arithmetics sandbox. This is a setting in which we fuse language with information from the math modality and strive to replicate some fusion methods from the vision-and-language domain. We find that some results for fusion methods from the larger domain translate to the math arithmetics sandbox, indicating a promising future avenue for multimodal model prototyping.

## introduction having models learn language from text alone has been criticised based on several aspects, from fundamental arguments about how language works @xcite ) to findings on certain information lacking in text @xcite . consequently, there is much interest in creating models that learn from more than text, i.e. "multimodal models". many different multimodal models that fuse different types of information with text have been developed, ranging from vision-and-language models (vl models) @xcite to language models fused with knowledge graphs @xcite . in this work, we mainly focus on the vision-and-language domain, while our results may generalize to other multimodal domains as well. a standard approach for fusing information from images with language is firstly to use a backbone, a large neural network that has been trained to create good representations of images. one such model is resnet trained on visual genome @xcite . the training data in this standard approach typically consists of image samples paired with linguistic information, such as ms coco @xcite . multimodal fusion is then achieved by generating representations of image samples using the backbone and feeding those together with paired linguistic samples to a large language model pre-trained on text. the language model is then expected to learn to leverage the visual information for its linguistic usage if trained on a sufficient amount of image-text pairs. we illustrate this approach in figure 1 . significant success on several multimodal visionand-language tasks has been achieved with this ap-proach @xcite @xcite . however, it has seldom been investigated from a methodological perspective, and existing methodological investigations indicate that performance on different vision-and-language tasks may not originate from what we would expect. for example, differences in performance for different vl models have previously been ascribed to different model architectures, while @xcite recently showed that training data and embedding layers matter more than e.g. if the model is dual-or single-stream. hessel and lee (2020) also showed that performance improvements for some vl models on image-text classification tasks mainly originate from unimodal signals, and not cross-modal interactions. additionally, @xcite found that vl models are not necessarily symmetrical in their cross-modal interactions. additionally, the standard approach for fusing information from images with language does not necessarily encourage experiments with different methods for multimodal fusion, mainly since the associated compute power is substantial, as a consequence of large models and data sizes. for example, the size of the faster r-cnn visual features used by @xcite is approximately 1.6 tb, significantly larger than e.g. the 20.3 gb for english wikipedia foot_0 . in this work, we hypothesize that methods for multimodal fusion can be developed in a smaller domain for more efficient investigations. we experiment with a sandboxed experimental setting for investigating different multimodal fusion methods. it is based on synthetic multimodal data, in a very constrained math-and-language domain consisting of simple arithmetic math statements, as illustrated in figure 1 . using this setup, we are free to investigate different multimodal fusion methods like the one seen in figure 1 in silico, using models that are easier to work with, with faster training times and full control of the data since we can synthetically generate it. the sandbox is described in section 2 and we release the corresponding code to enable other researchers to build on it. 2 we reason that results in the math-and-language domain could generalize to more complex domains, such as the vision-and-language domain (section 2) and provide empirical support for this with a few experiments (section 3). to summarize, our contributions are two, 1) proposal of idea in using smaller domains for easier investigations of vision-and-language fusion methods, and 2) demonstration of idea in the math arithmetics sandbox set in the math-and-language domain. we couple this demonstration with validating experiments that compare against recent results in the vision-and-language domain. ## the math arithmetics sandbox similarly to how images are described with pixel values over some grid, equations can be described with numerical values over potential positions in an equation. also similarly to how we use image specific models to generate representations of images for language fusion, we can use math specific models to generate representations of equations for language fusion, illustrated in figure 1 . in the math arithmetics sandbox we limit ourselves to two-digit numbers and equations describing sums of these numbers. an example of a data sample in the math modality of the sandbox is then 54 + 21 = 75, with the corresponding string "fifty-four plus twenty-one is equal to seventy-five". we can work with a total of 10 4 = 10, 000 different such math-language pairs in our sandbox. this number of possible samples is much smaller than e.g. the corresponding number for image data, for which the size of the support is (v 3 ) 32×32 for 32 × 32-pixel images with v possible values for each rgb channel, with underlying probability distributions of the pixel values. an example of a potential task in the math arithmetics sandbox is to predict the continuation of the string "fifty-four plus twenty-one is equal to" given the math information 54 + 21 = 75. this task can be compared to the task of visual question answering in the vision-and-language domain, for which a question could be "what is the color of the fruit?" provided with an image of a yellow lemon. the underlying format of the two tasks is essentially the same, a text prompt is provided together with information from another modality that encodes the information necessary to answer the prompt. we hypothesize that results from experiments in the math arithmetics sandbox can give intuitions about the vision-and-language domain, since the difference between the math arithmetics sandbox and more complex multimodal domains mainly lies in the size of the support, while the underlying for-mat for the multimodal tasks are similar. thus, it is interesting to investigate whether we can acquire valuable knowledge in the math arithmetics sandbox and exploit it in more complex multimodal domains, such as the vision-and-language domain. ## experiments to investigate whether insights gained in the math arithmetics sandbox are comparable to other multimodal domains, we perform a set of experiments. ## conclusions we propose the idea of studying vision-andlanguage fusion methods from a smaller domain. we reason that it would be easier and less resource demanding to develop performant multimodal fusion methods if we could investigate them using small models and domains. we exemplify this with the math arithmetics sandbox and get promising results. however, more experiments on other small domains are necessary to evaluate the potential of our proposed idea. it would also be beneficial to investigate small domains that are more similar to e.g. the vision-and-language domain. of these in the model m l in the math-and-language domain.