On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022)

Abstract: Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning. More general text generation however remains elusive. We take a step back and ask: How do these models work for more complex generative tasks, i.e. conditioning on both text and images? Are multimodal models simply visually adapted language models, or do they combine they reason jointly over modalities?We investigate these questions in the context of self-rationalization (jointly generating task labels/answers and free-text explanations) of three tasks: (i) visual question answering in VQA-X, (ii) visual commonsense reasoning in VCR, and (iii) visual-textual entailment in E-SNLI-VE. We show that recent unimodal advances, CLIP image representations and scaling of language models, do not consistently improveself-rationalization in multimodal tasks. We find that no single model type works universally best across tasks, datasets, and finetuning data sizes. Our findings motivate the need for novel general backbones that move text generation from images and text beyond image captioning.

## introduction the pretrain-finetune paradigm has changed the field of nlp. inspired by its success, there has been an explosion of interest in multimodal pretraining @xcite @xcite @xcite @xcite . to enable text generation from images, captioning is often included as one of the pretraining tasks @xcite @xcite . captioning is also the only generative task used to evaluate and compare * work undertaken while shruti palaskar and ana marasović were at the allen institute for ai. we study these questions through the lens of a newly emerging and important, but challenging task of self-rationalization @xcite : jointly generating both the task label/answer and a free-text explanation for the prediction. standard tasks for studying multimodal self-rationalization present different levels of difficulty. explaining vqa is similar to captioning since corresponding explanations describe visible information in the image that is relevant to the context (fig. 1 , left ). on the other hand, vcr instances require higherorder reasoning about unstated information such as commonsense (fig. 1 , right). since vcr answers are full sentences, self-rationalization consists of two generative sub-tasks. we evaluate the following models: (i) a joint vision-language model, vlp @xcite , (ii) a pretrained language model, t5 @xcite , that we visually adapt only through finetuning, and (iii) vl-t5/vl-bart @xcite , a combination of the previous two approaches. namely, vl-t5/vl-bart are developed from t5 and bart @xcite ) by doing a second round of pretraining, using multimodal datasets and objectives. we finetune all models for selfrationalization in two data settings: (i) using entire finetuning datasets (high-resource setting), and (ii) using 30% of the data (low-resource setting). we first present an analysis of the factors influencing performance of the visually adapted t5: the choice of image representation ( §4.1) and t5's model size ( §4.2). we demonstrate that recent advances in representing images, namely clip features, can be easily leveraged to get more accurate visually adapted t5 models. however, these improvements are not realized for a more complex sub-task of explanation generation. moreover, unlike for text generation conditioned only on text (including self-rationalization of tasks with textual inputs; marasović et al. 2022) we do not see clear performance improvements from scaling visually adapted plms. finally, the main comparison of the three model types described above ( §4.3) shows that no model type works universally the best across tasks and data regimes. these results demonstrate that outside of image captioning, there is no clear choice of model backbone or training regime in image conditioned text generation. we aim to motivate research on multimodal model comparisons across generative tasks and experimental setups, to help realize benefits of different model families ( in ## experimental setup in this section, we describe tasks, datasets, and evaluation setup for self-rationalization of vision-andlanguage tasks introduced in prior work @xcite . ## results to study whether there is a base model family that is more suitable for text generation conditioned on images and text, we compare: (i) a joint visionand-language model, vlp ( §2.1), (ii) a visually adapted plm, va-t5 ( §2.2), and (iii) vl-t5 / vl- bart, a combination of the previous two model families ( §2.3), for self-rationalization of the three tasks in figure 1 and before presenting the outcomes of this comparison in §4.3, we study the impact of the choice of image features and model size on va-t5's performance. ## limitations while we examine multiple methods that were available to us while we were conducting this research, it is inevitable that new multimodal models will be released, leaving the question of whether those models are already superior for multimodal self-rationalizing open. for example, the recently proposed model ofa @xcite could be particularly suitable. it is available in large sizes (33m, 93m, 182m, 472m, 930m), trained with a filtered version of the large-scale text corpus pile (140gb; @xcite as well as with a variety of multimodal datasets and objectives including image captioning and grounded captioning. besides that, all of the models we examined have been trained on data in the english language and using clean, high quality images from similar data sources (ms coco, flickr). inherent biases stemming from this source of data would need to be studied in future work towards scaling this work to multiple languages and other image sources (for e.g. noisy, dense context, adversarial images). our main measure of explanation quality is plausibility that does not answer whether these plausible explanations are useful in real-world applications of vqa and nli to actual stakeholders. another limitation are our computational resources. with access to even more compute, we would be able to examine at a larger scale such as t5 11b or more.