Z - L a VI : Zero-Shot Language Solver Fueled by Visual Imagination (2022)

Abstract: Large-scale pretrained language models have made significant advances in solving downstream language understanding tasks. However, they generally suffer from reporting bias, the phenomenon describing the lack of explicit commonsense knowledge in written text, e.g., ”an orange is orange”. To overcome this limitation, we develop a novel approach, Z-LaVI, to endow language models with visual imagination capabilities. Specifically, we leverage two complementary types of ”imaginations”: (i) recalling existing images through retrieval and (ii) synthesizing nonexistent images via text-to-image generation. Jointly exploiting the language inputs and the imagination, a pretrained vision-language model (e.g., CLIP) eventually composes a zero-shot solution to the original language tasks. Notably, fueling language models with imagination can effectively leverage visual knowledge to solve plain language tasks. In consequence, Z-LaVI consistently improves the zero-shot performance of existing language models across a diverse set of language tasks.

## introduction large-scale pretrained language models (plms) have achieved great success on various natural language understanding (nlu) tasks and even exhibit impressive zero-shot capabilities without taskspecific fine-tunings @xcite . and recent research suggests that such ability improves by further scaling up the model size (e.g., to hundreds of billions of parameters) and the amount of textual pretraining data (to tbs of raw texts) @xcite @xcite . however, zero-shot language learners solely trained on texts inevitably suffer from human reporting bias. for example, people tend not to write common or apparent things @xcite , and the frequency of a certain textual statement does not always correspond to their relative * work was done during the internship at tencent ai lab. this example is {news type}. a news image of {news type}. imagine textual visual figure 1: our system endows language models with two complementary types of visual imagination capabilities: recalling existing images (through retrieval) and synthesizing nonexistent images (via image-to-text generation). they effectively alleviate the reporting bias issue and improves the zero-shot performance for solving plain language tasks. we experiment with three types of tasks: (a) word sense disambiguation, (b) science question answering, and (c) topic classification. in this paper, we focus on incorporating vision knowledge to facilitate the solution of plain lan-guage understanding tasks. cognitive science has demonstrated that the human vision system is crucial to supplement, interact, and influence the language system @xcite . for example, there exists a fast mapping between vision and language in the human language learning process @xcite . inspired by this, we propose a visual imagination framework, z-lavi, to endow any plms (e.g., gpt, bert, bart, etc.) with visual imagination capabilities. specifically, we apply two different types of "visual imaginations" to the input texts. given input text, the first approach recalls existing images (e.g., through search engines), and the second one synthesizes nonexistent images via text-to-image generation models (e.g., dall-e @xcite ). these two strategies mimic different types of human mental behaviors, i.e., recalling past memories and creative mental image construction. interestingly, we find that these two mechanisms are highly complementary to each other. our proposed visual imagination module tends to rely more on recalling when input texts are short because their corresponding objects or scenes generally exist and are easy to find. however, when input texts are long and complex, the module is more inclined to create new images. we develop a unified framework (figure 1 ) that exploits both types of imaginations along with the original textual inputs to compose zero-shot solutions to a broad set of downstream language tasks. note that our work differs from existing multi-modal tasks such as vqa @xcite or visual dialog, @xcite , which have both textual and visual inputs. instead, we use visual imagination as machinery to facilitate the (zero-shot) solution of pure language tasks. we show that on a diverse set of language understanding tasks, z-lavi consistently improves the performance of existing language models of different sizes and architectures. in particular, our z-lavi with sbert can achieve a zero-shot f1 score of 87.5% on the wsd task without finetuning, even outperforming bert-large, which is fine-tuned with three examples per sense, by 2.3%. z-lavi also beats all existing zero-shot models on four science qa tasks and two topic classification tasks by a large margin. our analysis demonstrates that z-lavi can complement language models and significantly alleviate plms zero-shot prediction errors by adaptively executing two visual imagination mechanisms -recall and synthesis. ## conclusion in this paper, we propose a novel approach, z-lavi, to alleviate the reporting bias problem of pretrained language models and enhance their zero-shot inference ability. we develop two complementary visual imagination mechanisms, i.e., recall that aims to retrieve existing objects or scenes and syn-thesis that generates nonexistent ones. experiments on a wide range of language tasks show that our approach can significantly outperform existing zero-shot language models, pointing towards a promising direction to solve an unseen language task with visual imagination.