A Small but Informed and Diverse Model: The Case of the Multimodal G uess W hat!? Guessing Game (2022)

Abstract: Pre-trained Vision and Language Transformers achieve high performance on downstream tasks due to their ability to transfer representational knowledge accumulated during pretraining on substantial amounts of data. In this paper, we ask whether it is possible to compete with such models using features based on transferred (pre-trained, frozen) representations combined with a lightweight architecture. We take a multimodal guessing task as our testbed, GuessWhat?!. An ensemble of our lightweight model matches the performance of the finetuned pre-trained transformer (LXMERT). An uncertainty analysis of our ensemble shows that the lightweight transferred representations close the data uncertainty gap with LXMERT, while retaining model diversity leading to ensemble boost. We further demonstrate that LXMERT’s performance gain is due solely to its extra V&L pretraining rather than because of architectural improvements. These results argue for flexible integration of multiple features and lightweight models as a viable alternative to large, cumbersome, pre-trained models.

## introduction current multimodal models often make use of a large pre-trained transformer architecture component, which is then fine-tuned for the final task. this setup can lead to high performance, due to the immense amount of data embodied in the pretrained component, together with its large capacity in terms of parameters. however, these models are also extremely costly to train; even fine-tuning requires non-negligible resources. here we wonder whether the need for pre-training data, and for large and computationally costly neural networks could be mitigated by feeding the models with richer candidate representations, and by using an ensemble of lightweight models. is it a living thing? yes is it black and white? yes is it in the background? yes are its ears up in the air? yes the guesser receives the image and dialogue as input, and has to pick the correct target (in green) from the list of candidates. we consider different ways of representing the candidates, e.g. using category information, visual features, and/or spatial position. the motivation for this paper is to disentangle the contributions to good model performance on grounded multimodal tasks: is it due to better architecture, e.g. transformers for text, vs lstms? or to exposure to large amounts of multimodal data during pretraining? or learning good representations for the task during fine-tuning? as the task for this case study we take a multimodal referential game, guess what?! (gw) @xcite , specifically the final guessing task. here the aim is to guess which object in the image is the correct target, based on the dialogue history (the series of questions) and the image. the model receives as input an image and a sequence of question-answer pairs, which are together passed to the multimodal encoder. the final hidden layer of the encoder is then the guesser's input representation. the guesser classifier then generates a representation for each of the candidate objects in the image, and selects the target based on the similarity between the candidate representations and the input (image and dialogue) encoding. the creation of the guesser representation is the focus of this paper. we examine both the effect of input encoding (lxmert vs v-lstm) and the effect of using different features to represent the candidate targets. our results show that these two factors interact: a lightweight input encoding can be compensated for by richer candidate targets; conversely, a heavy encoder can work with impoverished features. when introducing the gw task and the baseline models, @xcite run a comparative evaluation of the guesser model; the visual embedding was shown to be not informative in selecting the target object and better results were obtained by using just the embedding learned from the category label and spatial coordinates. all the following work on the gw task retained the baseline candidate representations. we put the attention on this evalution again situating it in the new context of the pre-trained large encoders we are now familiar with. our motivation behind re-evaluating the features used by the guesser is the observation that in gw, guessing the right target could require different sorts of information about the candidates. for instance, in the game illustrated in figure 1 , the guesser requires candidate representations that encode the ontological information that dogs are living things whereas pillows are not, in order to make sense of the dialogue. the candidate representations also have to distinguish the target dog from the other dogs: here, visual features encoding the colour could differentiate the black and white dogs from the other two dogs. finally, spatial information is essential to locate the target in the background. note that if the choice set contained other distractors, other features might have been needed to identify the target. we hypothesize (and our results confirm) that the combination of visual features and spatial location information on the level of individual candidates, plus some kind of semantic information about candidate categories (e.g. task-specific embeddings or general pre-trained embeddings) will lead to the best guesser. along with improving the candidate representations, we also investigate the effect of ensembling multiple models. as well as potentially bringing a boost in performance, ensembling also allows us to inspect the uncertainty of the models under a bayesian interpretation of deep ensembles as bayesian model averaging @xcite @xcite . hence, after comparing models based on their task-success, we use data and model uncertainty as a post-hoc analysis to get an in-depth comparison of models' behaviour. this allows us to better understand the effect of richer features: for v-lstm guessers, they provide key information, while for pre-trained lxmert they seem to be redundant with the input encoding. however, for lxmert without pretraining, the guesser is not as able to integrate the information from the input encoding and candidate feature representations. to recap, we investigate whether providing informative candidate representations (which are, themselves, gleaned from pre-trained models) to the guesser model can make the task more feasible when using lightweight input (dialogue + image) encoders, i.e. v-lstms vs lxmert. lxmert has the advantage of significant pretraining on a large corpus of v&l data, as well as orders of magnitude more parameters. hence, we compare v-lstm ensembles against lxmert, both alone and ensembled. we also compared the lxmert architecture trained from scratch on the gw task, to understand the relative contributions of pretraining vs architecture. we compare the models both in terms of task-accuracy as well as doing an uncertainty analysis. our results show that • an ensemble of lightweight models with good candidate representations can match the performance of a single lxmert model; most recently, @xcite propose a new transformer-based architecture for gw, while @xcite evaluate the impact of vilbert as encoder and design a state-estimator for the guesser that let it accumulate belief state incrementally. while these models perform well, they are significantly larger and more complex, and do not permit the targeted study done in this paper. ## guesser model in this section we describe the guesser model. we use the same guesser architecture introduced in de @xcite which has been employed in virtually all follow-up work on gw. the guesser receives as input a 512d vector, encoding the grounded dialogue, and a vector representation of each candidate. this vector representation is the result of feeding the concatenated features for each candidate through a two layer mlp with relu activations, resulting in a 512d vector. the guesser then computes a dot product between the vector representing the grounded dialogue and each candidate representation (processed by the mlp described above). the resulting scores are combined into a softmax layer, resulting in a probability distribution over the candidates. note that the mlps share parameters between candidates. in our experiments, we use as encoder lxmert or v-lstm, and study the impact of using an ensemble of encoders together with the enrichment of candidate embeddings. ## measuring uncertainties the uncertainty of a model, parameterised as θ, is commonly measured by the entropy of the predictive distribution p θ (y|x), averaged over a test set. for each example x, a confident model will put most probability mass on a single choice y, leading to low entropy, while an uncertain model will spread its bets, leading to higher entropy. within an ensemble, the ensemble total uncertainty is the entropy of its predictive distribution, which combines the distributions of the n ensem-ble components: the difference between total uncertainty and data uncertainty is model uncertainty, which measures the extent to which the models disagree (i.e., the extent to which the ensemble's predictive distribution does not match the average ensemble component). model uncertainty is always non-negative. in this paper we compare different models, differing in their choice representations, as ensembles. within the gw guesser, the choice representation should be considered part of the input x. inadequate choice representation will thus lead to high data uncertainty, since the representation is not sufficient to make confident decisions. since the humans playing the original gw game, generating the test and training data, guessed correctly, overly high "data uncertainty" values point to problems with data representations, rather than inherently ambiguous data. formally, within a bayesian framework, it is the mutual information between y and the ensemble parameters θ estimated from data d, derived from the difference between entropy and crossentropy: h[p(y, |x, d)] = e θ|d h[p(y|x, θ)] -m i[y, θ|x, d], where the left hand term is total uncertainty (marginalising over θ) and the first term on the right is data uncertainty. we note here that, while 'data uncertainty' has been identified with 'aleatoric uncertainty' @xcite @xcite , namely the true uncertainty of the example in the world @xcite , this doesn't hold inasmuch as the representation of the data is a modelling decision (see also hüllermeier and waegeman (2021), sec 2.3). comparing different data representations doesn't change the true aleatoric uncertainty, which is an lower bound on data uncertainty. whether model uncertainty should also be minimised is a different question. in theory, if all ensemble components have found the global optimum, model uncertainty will be zero. in practice, not being able to find the global optimum, we use ensembles to approximate a distribution over good local optima. ensemble 'boost' (the improvement in performance over the component average) also requires model diversity. it is thus more useful to have a collection of strong but different opinions (low data, high model uncertainty) than homogeneous equivocal opinions (high data, low model uncertainty). ## experiments prior work has shown that lxmert outperforms v-lstm when using the standard candidate representation, which uses only spatial and category embeddings: lxmert accuracy is 69.2% while v-lstm reaches just 64.5% @xcite . below we ask whether v-lstm accuracy can reach lxmert's if provided with different candidate representations. improved candidate representations should make it easier for the fine-tuned model to learn to match the correct target with the image and dialogue encoding, by facilitating the match between features of the candidates and the features discussed in the dialogue. secondly we experiment with ensembling our models. we find that ensembling the v-lstm models leads to a larger boost in accuracy, while ensembling the lxmert models helps less. this is a very convenient result, since training ensembles of lightweight v-lstms is fast and computationally cheap, compared to fine-tuning even a single lxmert. analysing ensemble uncertainty confirms that v-lstm encodings allow the guesser to make better use of improved candidate representations, while they have less of an effect on lxmert. furthermore, we see that an ensemble of v-lstm guessers contains sufficient diversity, in terms of model uncertainty, to make ensembling worth it. @xcite . the worst combination, visual+spatial, is ten percentage points worse than the best combination, glove+visual+spatial (55.19% vs. 66.72%). models with three or four types of candidate representations outperform models with only two types; however there is not a benefit of combining all four types over only three. category/type information is crucial for success: the model with only token-level information, visual+spatial, clearly underperforms all the others. the category representations, namely cat and glove, lead to similar results when combined with other representations, and do not benefit from being combined together (unlike the token representations). glove representations do seem to be slightly more beneficial than cat representations, indicating that the additional world knowledge that they contain can be useful. (see figure 2 for an example where glove representations allow the model to guess correctly.) ensembling the results of ensembling five versions of each v-lstm guesser are reported in next, we check whether lxmert could also benefit from ensembling and from the richer candidate representations. we evaluate both pre-trained lxmert and lxmert-scratch trained only on gw. when ensembling pre-trained lxmert, the only source of randomness is in the order of the training data, while for lxmert-scratch, the random initializations also differ between ensemble components. for both lxmert and lxmert scratch, training an ensemble for gw requires 21hrs × 5 = 4.3 days, 27 times more compute than the v-lstm ensemble. as shown in together these results indicate that lxmert's improved performance hinges on being able to use the information seen during pretraining, rather than architectural improvements. furthermore, for lxmert, adding candidate representations (like glove and visual) that are extracted from pretrained models, and thus incorporate similar kinds of pretraining knowledge to the lxmert encoder, does not help over the weaker candidate representations (cat+spatial). we presume this is due to the pre-trained lxmert model already having learned the relevant ontological information, as well as the ability to localise visual information, and thus not needing it to be provided explicitly. this hypothesis is strengthened by the weaker performance of lxmert-scratch, which has not been able to learn the relevant features from additional pretraining data. ## conclusion in this paper, we re-evaluated the need for a deep pre-trained multimodal encoder on a testbed multimodal guessing task (gw). we demonstrated that a lightweight v-lstm model was able to achieve matching performance, given useful features and the reduction in uncertainty enabled by ensembling. we show that for guesswhat?!, the candidate representations that lead the v-lstm ensemble to reach higher accuracy are those encoding ontological (glove), visual and spatial information. the pre-trained model does not profit from either of the richer representation, or the ensemble. the uncertainty analysis of the ensemble models shows that while with poor candidate representations v-lstm models are highly uncertain, richer candidate representations let these models behave more similarly to the pre-trained lxmert in terms of both data and model uncertainty. the richer candidate representations effectively transfer information from other corpora (glove) or visual recognition models (visual). these features do not match exactly what lxmert sees during pretraining, but given that lxmert does not benefit from them, they do not seem to add crucial information for lxmert. conversely, v-lstm benefits from these 'cheap' features to the extent of matching deep contextual model performance, indicating a continuing role for these types of representations in grounded language tasks. hence, we conclude that the good performance obtained by the guesser when based on the pretrained multimodal transformer is not due to its architecture or to the representation learned during fine-tuning, but rather to the exposure to a large amount of multi-modal data during pre-training. our results may help mitigate environmental issues given by the training of large models. it remains to be seen whether these results hold for other tasks and other unimodal and multimodal models.