Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Tasks (2023)

Abstract: Large language models have demonstrated robust performance on various language tasks using zero-shot or few-shot learning paradigms. While being actively researched, multimodal models that can additionally handle images as input have yet to catch up in size and generality with language-only models. In this work, we ask whether language-only models can be utilised for tasks that require visual input – but also, as we argue, often require a strong reasoning component. Similar to some recent related work, we make visual information accessible to the language model using separate verbalisation models. Specifically, we investigate the performance of open-source, open-access language models against GPT-3 on five vision-language tasks when given textually-encoded visual information. Our results suggest that language models are effective for solving vision-language tasks even with limited samples. This approach also enhances the interpretability of a model’s output by providing a means of tracing the output back through the verbalised image content.

## introduction in recent years, large language models have gained significant attention in the natural language processing (nlp) community due to their impressive performance on various tasks such as machine translation, text generation, and language modelling @xcite . these models, which are trained on massive amounts of data, have been shown to capture complex linguistic patterns and generate coherent text @xcite . some of the most popular models are trained by openai, a research organization that has released several models, including gpt @xcite , gpt-2 @xcite , and @xcite . in addition to gpt models, there are also many open-source or open-access large language models that researchers and organizations around the world have developed, such as bloom @xcite , gpt-j @xcite , opt @xcite , flan-t5 @xcite . recent work by @xcite provided an in-depth analysis of many large language models (llm) across 42 core scenarios. 1 all scenarios are language tasks that are evaluated with multiple metrics by prompting the language models with few-shots from the selected datasets, also known as in-context learning. currently, there are no comparable models directly suitable for tasks that require visual information as part of the context, even though such multimodal tasks have similar practical relevance. pre-trained vision-language models @xcite @xcite have shown great promise by learning joint representation of images and text documents, but so far they have not been optimised for prompting on vision-language tasks but rather using the learned joint representations for fine-tuning on downstream tasks. moreover, as we note below, many multimodal tasks appear to rely on reasoning capabilities, which larger language models have been shown to perform well on @xcite . hence, in this work, we attempt to utilise such models to do in-context learning on multimodal data, achieving this by encoding the visual information in language. while there has been some recent work going in this direction (see discussion below), it falls short in terms of evaluating the performance of large language models across multiple dimensions, applying them to a diverse range of vision-language tasks, and comparing the performance of gpt models with open-source or open-access models. as such, there is a need for further research in this area to fully understand the capabilities and limitations of these models for vision-language tasks. ## text-visual in-context learning in this section, we describe the proposed methodology of applying in-context learning to visionlanguage tasks. in-context learning essentially works by prompting a pre-trained language model with the task and expecting it to generate text that solves a particular task. it is performed by giving a few-shots of the respective task at inference time without requiring updating the model weights and expecting the model to generate text corresponding to the expected output. formally, given a query input text x and a set of candidate answers @xmath0...y m }, which can be class labels for a particular task or free text, a pre-trained model m outputs a candidate answer with the maximum score conditioned on the task description t , n in-context sample pairs @xmath1, y 1 )...(x n , y n )}. the likelihood of the candidate answer y j can be represented by a scor-ing function f with the language model m @xcite : the final predicted candidate answer of the model (ŷ) can be formulated as: our proposed methodology for "text-visual incontext learning" is shown in figure 1 . first, all images from all evaluated datasets have been passed through multiple pre-trained image models to obtain the textual description of the visual content, which we refer to as image-as-textrepresentation throughout the paper. the image-astext-representation is essentially a textual description of the visual content that captures important visual aspects. the prompt text comprises the task description, in-context sample pairs, and the input of the evaluation sample. given such a prompt text, the language model generates a sequence of text tokens as an output. we evaluate the proposed methodology on various vision-language datasets that include either classification or question answering tasks. thus, the task description is different between these two categories. the task description for the classification tasks is further replaced with the task-specific phrase that describes the downstream task and provides the task-specific class labels. more details on the exact prompt text for each dataset are provided in appendix a. next, we describe the methods for extracting image-as-text-representation, selecting in-context samples, and aggregating answer predictions in cases where the language model is prompted multiple times with various in-context samples for the evaluation sample. ## experimental setup in this section, we describe the details of the building blocks of the methodology, the evaluated datasets, large language models, and methods for obtaining a textual description of images. ## results and analysis in this section, we discuss the obtained results from the experiments such as the impact of in- ## conclusion in conclusion, our study has demonstrated the suitability and effectiveness of using large language models via prompting on vision-language tasks. our approach relies on verbalising the visual content employing image captioning and classification models and prompts the language models along with the textual content. we have also shown that the choice of in-context samples and the method of verbalising the visual content impact the results. our experimental evaluation suggests that this approach can achieve impressive results by presenting only a few samples from a dataset compared to models that are fine-tuned on entire train splits of the evaluated datasets. furthermore, our study has also highlighted the importance of considering the choice of language models when applying them to such downstream tasks. we have demonstrated that different models perform better on various tasks, with gpt-3 achieving the highest overall performance across three datasets and open-source models t0pp and flan-t5 achieving the best overall performance among them. even though the performance of these models may not have been the best across all evaluated tasks, they still have the potential to be used in such cases and even achieve comparable results. for instance, t0pp yielded the best performance on the mvsa dataset, even in a zero-shot setting. thus, the choice of language models is crucial for achieving optimal results in vision-language tasks.