VLIS : Unimodal Language Models Guide Multimodal Language Generation (2023)

Abstract: Multimodal language generation, which leverages the synergy of language and vision, is a rapidly expanding field. However, existing vision-language models face challenges in tasks that require complex linguistic understanding. To address this issue, we introduce Visual-Language models as Importance Sampling weights (VLIS), a novel framework that combines the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without further training. It extracts pointwise mutual information of each image and text from a visual-language model and uses the value as an importance sampling weight to adjust the token likelihood from a text-only model. VLIS improves vision-language models on diverse tasks, including commonsense understanding (WHOOPS, OK-VQA, and ScienceQA) and complex text generation (Concadia, Image Paragraph Captioning, and ROCStories). Our results suggest that VLIS represents a promising new direction for multimodal language generation.

## introduction visual language models (vlms) extend unimodal text-only language models by conditioning their outputs on image context. recent vlms @xcite @xcite can perform diverse multimodal tasks from commonsense vqas @xcite to incontext learning @xcite @xcite . moreover, instruction tuning with visual inputs @xcite @xcite has improved the vlms' responsiveness to an even more extensive variety of tasks @xcite . however, most vlms only partially inherit the linguistic understanding capability of the unimodal models @xcite . we here illustrate two intriguing failure cases of the recent vlms, figure 1 : top: vlis correctly recognizes named entities, unlike the vlms. bottom: vlis is not deceived by the distractor images. note that the images show a seagull and a monkey, not an ostrich and a chimpanzee. vlis inherits this favorable linguistic capability from a text-only language model @xcite , and use vlms as a guide for visual alignment. the examples are truncated for visualization purposes: we provide the full-text in appendix a.2. using both a strong image captioning model (blip-2 @xcite ) and an instruction-tuned model (llava @xcite ). firstly, vlms avoid specifying named entities. the upper examples of figure 1 show the vlm failing to describe a public figure (diego maradona) or movie character (don corleone). the problem is not the lack of knowledge: after applying our zero-shot method (vlis), the vlm tells the names. we further investigate this phenomenon in the landmark recognition experiment in appendix a.1. secondly, vlms rely on the image context, even when they should not. the lower examples of the same figure show the vlm being misled by image context to deny commonsense knowledge. the questions are not unanswerable: the text-only language model without the image context answers both correctly. we provide more samples on visual distraction in appendix a.2. hence, the linguistic capabilities of the vlms are not optimal yet. on the other hand, the unimodal text-only language models themselves @xcite show reliable linguistic understanding and known for their knowledge understanding @xcite and complex reasoning capabilities @xcite . hence, it becomes reasonable to delegate the burden of language modeling to the text-only models. to this end, we propose visual-language models as importance sampling weights ( vlis) as a plug-and-play method to enhance the unreliable linguistic understanding of the vlms. when generating each text token, vlis follows the token likelihoods of the unimodal text-only language model. furthermore, vlis multiplies importance sampling @xcite weights derived from a vlm to provide the visual alignment signals. to isolate the visual conditioning capability of the vlms from their language modeling preference, we incorporate the exponentiated pointwise mutual information (pmi) @xcite of the image context and the current text token as the weights. as a result, vlis can maintain the favorable language modeling capability of the text-only model and control the visual conditioning strength simultaneously. we evaluate vlis on two vlm backbones to test whether vlis is effective both when the language modeling capability of the vlm is weaker than that of the text-only model (blip-2 @xcite ) and when the vlm is expected to model language well owing to the visual instruction tuning process (llava @xcite ). our ex-periments consist of various tasks that require both reliable language modeling and strong visual conditioning, including weirdness identification (whoops @xcite ) and commonsense vqa (ok-vqa @xcite , scienceqa @xcite ), extended image captioning (concadia @xcite and image paragraph captioning @xcite ), and open-ended generation (rocstories @xcite ). compared to the dataset-specific state-of-the-art baselines and the base vlms, vlis improves linguistic capabilities such as responsiveness to prompts while maintaining visual conditioning according to a comprehensive set of evaluation metrics. ## vlms as importance sampling weights we propose visual-language models as importance sampling weights (vlis) to harmonize the visual conditioning capability of the vlms with the linguistic fluency of the text-only language models. we provide the intuition behind our approach in §2.1, describe our token-level visual alignment scores in §2.2, and combine the said scores with the text-only model via importance sampling in §2.3. ## experiments: describing facts we verify that vlis can alleviate the factual inaccuracy concern raised in figure 1 with various multimodal benchmarks: weirdness identification §3.1, commonsense understanding §3.2, and scientific reasoning §3.3. vlis consistently outperforms the backbone vlm and shows comparable factual correctness to the strong baselines. experimental setups. we explore two experimental setups. our experiments on the whoops dataset incorporate llava @xcite and lynx @xcite as the vlms and vicuna 7b @xcite as the text-only model. in the vqa experiments, we use blip-2 opt 2.7b @xcite pipe represents further pipelining with gpt3 and 0shot denotes a zero-shot method. the best numbers are bolded and the second best ones are underlined. ## experiments: text generation in addition to factual knowledge, text-only language models manifest two critical capabilities: following prompt instructions and generating fluent and diverse text. we demonstrate that vlis extends these qualities to the visual domain with contextualized captioning ( §4.1), paragraph captioning ( §4.2), and visual story generation ( §4.3). metrics. both captioning benchmarks use automatic text metrics, including cider @xcite , meteor @xcite , and bleu-4 @xcite . in the openended generation problem of visual storytelling, we use various fluency metrics (2-gram repetition, diversity, coherence, mauve @xcite ) and a visual strength metric (clipscore @xcite ). refer to @xcite for details on the fluency metrics. ## qualitative results commonsense understanding. figure 4 illustrates zero-shot results in the ok-vqa dataset @xcite . in (a) and (b), the baselines including the base vlm and naïve ensemble fail to understand the intention of the question (kind of dog and native to north america). while the text-only model understands the question better and suggests plausible answer candidates (pug and wolf ), it has no access to the visual inputs and ultimately outputs an incorrect answer. on the other hand, vlis sensibly combines commonsense reasoning and visual context. results for images (c) and (d) depict the failure cases. in (c), vlis follows the reasoning process of the text-only language model to deduce that the answer should be a type of material. however, as the vlm focuses on the frontal object (umbrella), vlis wrongly concludes the answer is the material of that object (paper, which is coincidentally a flammable material as well). in (d), the textonly model produces an incoherent output (ocean). vlis inherits this misinterpretation and likewise generates an incorrect answer (water). in conclusion, vlis induces coordination of the vlm's visual specificity and the text-only model's commonsense understanding but carries on the modeling insufficiency of the individual modalities. open-ended generation. lastly, we demonstrate the open-ended generation capability of vlis in figure 5. here, vlis should condition its output on the diverse text prompt and the image. unlike the base vlm, it clings tighter to the prompt and produces realistic self-introduction (hey, it's me), personal journal (today i went), and romantic messages (here is a romantic message. answer:). also, vlis plays pun on the word apple (see apple laptop in the image and apple of my eye). refer to appendix g for more baseline samples. better aligned with our approach are decodingoriented methods for image-conditioned text gen-eration. zerocap @xcite uses the gradient signal from a pretrained image-text alignment scorer (clip @xcite ) to update the language model's memory. magic @xcite ) also utilizes clip. unlike zerocap and magic, vlis utilizes autoregressive vlms @xcite , rather than clip. language model decoding. language model decoding is the process of generating text from a pretrained language model. traditional decoding methods use greedy decoding and beam search to find the most likely sequence of words. the truncated sampling algorithms such as top k sampling @xcite @xcite , nucleus sampling @xcite , and typical p sampling @xcite have been proposed to avoid text degeneracy. recent deterministic algorithms, such as contrastive decoding @xcite and contrastive search @xcite , provide a better trade-off between text fluency and model likelihood. neurologic @xcite and neurologic a*esque decoding @xcite control the language models to include given words in their outputs. as shown in the experiments, vlis can be used jointly with any decoding method, including beam search and contrastive search. ## conclusion we propose vlis, a novel framework to alleviate the language modeling burden of visual-language models (vlms). vlis combines the linguistic understanding capability of the text-only language models with the visual conditioning strength of the vlms by importance sampling. to isolate the vlms' visual conditioning power, vlis uses pointwise mutual information to suppress their text-only marginal distribution. our framework enhances the base vlm in commonsense reasoning (whoops (bitton-guetta et al., 2023), ok-vqa @xcite , and scienceqa @xcite ) and complicated text generation (concadia @xcite , image paragraph captioning @xcite , and rocstories @xcite ) problems. in the future, vlis can be extended to incorporate other modalities for which the paired multimodal data is even scarcer. we hope that vlis sparks an interest in better utilization of off-the-shelf multimodal pretrained models.