Learning the Visualness of Text Using Large Vision-Language Models (2023)

Abstract: Visual text evokes an image in a person’s mind, while non-visual text fails to do so. A method to automatically detect visualness in text will enable text-to-image retrieval and generation models to augment text with relevant images. This is particularly challenging with long-form text as text-to-image generation and retrieval models are often triggered for text that is designed to be explicitly visual in nature, whereas long-form text could contain many non-visual sentences. To this end, we curate a dataset of 3,620 English sentences and their visualness scores provided by multiple human annotators. We also propose a fine-tuning strategy that adapts large vision-language models like CLIP by modifying the model’s contrastive learning objective to map text identified as non-visual to a common NULL image while matching visual text to their corresponding images in the document. We evaluate the proposed approach on its ability to (i) classify visual and non-visual text accurately, and (ii) attend over words that are identified as visual in psycholinguistic studies. Empirical evaluation indicates that our approach performs better than several heuristics and baseline models for the proposed task. Furthermore, to highlight the importance of modeling the visualness of text, we conduct qualitative analyses of text-to-image generation systems like DALL-E.

## introduction people typically communicate knowledge and information textually, but most prefer to consume visually rich content. text-to-image generation/retrieval models could augment text with appropriate images, aiding the creation of appealing and easy-to-understand documents. models like dall-e @xcite and stable diffusion @xcite work phenomenally well for input text that is carefully constructed to elicit images. however, they cannot handle longform text with a mix of sentences that may or may in case of your failure to answer, judgment will be taken against you by default for the relief demanded in the complaint. human-annotated visualness @xmath0.44 human-annotated visualness @xmath1.67 clip model fine-tuned for identifying visual text text input "visual text" "non-visual text" text input text-to-image (e.g., dall-e) our contributions figure 1: overview of the sentence visualness identification task, along with a motivating downstream application (passive generation of relevant images). prior approaches for quantifying the visualness of text operate on a word or phrase level @xcite and leverage lexicons that contain humanassigned world-level imageability scores @xcite . however, besides being limited in their coverage, our experiments also show that word or phrase-level visualness cannot be aggregated to quantify sentence-level visualness. to this end, in this work, we curate a corpus of 3,260 sentences in english paired with their human ratings for visualness, as well as a noisy but large corpus of 48,077 automatic alignments between text and visual assets in long-form documents. the textual part of the resulting alignment pairs can be used as examples of visual and non-visual sentences. we propose a strategy to fine-tune visionlanguage models like clip, allowing classification inferences over text-only inputs. our objective also ensures that the learned embeddings remain usable for downstream text-to-image retrieval. we compare the performance of our proposed approach against several heuristic and model-based baselines. our extensive evaluation suggests that our fine-tuning strategy leads to the most accurate visual and non-visual text classifier. finally, we conduct several analyses to glean insights into the model's learned attention mechanism, text-toimage retrieval abilities, and downstream text-toimage generation capabilities. 1 in sum, our key contributions are: ## text imageability dataset (timed) our proposed fine-tuning approach follows multistage training of a large vision-language model clip @xcite . in the first stage, we conduct large-scale fine-tuning, followed by finetuning on a relatively smaller annotated corpus in the second stage. we first discuss the curation of a large-scale corpus that comprises automaticallyassigned and distant labels and then describe the curation of the human-labeled corpus of visual & non-visual sentences. ## tip-clip for scoring text visualness background: the clip model @xcite jointly trains image and text encoders to predict the correct pairing between images and textual descriptions. in a batch size of n images and n texts (n 2 possible image-text pairings), the objective function ensures that the cosine similarity between the embeddings of correct image-text pairs is maximized while the cosine similarity between the (n 2 -n ) incorrect image-text pairs is minimized. the encoders are trained over a large multimodal dataset of ∼ 400 million image-text pairs. updated training objective: when predicting text visualness, the goal is to assign a higher score to text that is visual (evokes a concrete image for the person reading it) and a lower score for non-visual the common loon, minnesota's state bird, usually nests on islands or on shore lines of our northern lakes colliers international makes no guarantees, representations or warranties of any kind, expressed or implied, regarding the info... there' s only one way to prove them wrong. text encoder image encoder t1 .t1 t2 t n t n-1 i1 i1 .t2 i1 .t n-1 i1 .t n i1 .t1 inull .t2 inull .t n-1 inull .t n inull .t1 i n-1 .t2 i n-1 .t n-1 i n-1 .t n i n-1 .t1 inull .t2 inull .t n-1 inull .t n inull inull visual text visual text non-visual text non-visual text inull i n-1 null image null image text (text that does not evoke an image). in line with the original training objective, we further train the clip model to match text that is identified as visual with the corresponding image. we adapt the clip training to match text that is identified as non-visual with a single null image (see fig. 2 ). matching visual text with the corresponding image while non-visual text to a null image not only encourages the model to distinguish between visual and non-visual text, but also allows it to anchor non-visual text in the common null image that can be used during inference without having access to a potentially paired image. formally, the adapted training objective is given as, v and v are the set of examples in the current batch that belong to non-visual and visual categories, respectively. finally, i e null denotes the embedding of the null image. during inference, we compute the cosine similarity between the representation of a given text with the representation of the null image; non-visual texts will have a high similarity with the null image. conversely, the visualness score s of any text with embedding t e can be obtained using for the null image, we create an rgb image of size (224, 224, 3) in which each pixel value is chosen randomly (see figure 2 ). however, experiments with different types of null images indicate that the choice of null image does not affect the model's performance; see appendix a.1. an alternative formulation for adapting the clip training objective could have been to match visual text with a single image while matching non-visual text with a single null image. however, this formulation of the training objective is similar to binary classification and does not enforce a contrastive objective for the positive examples. matching visual text with its corresponding image instead of a common image for all visual text affords text embeddings that can be used for downstream tasks like text-to-image retrieval; we provide empirical evidence for worse text-to-image retrieval performance with the alternative formulation in results. for the first stage of training, we fine-tune the clip model (vit/b-32) on the proposed objective (see eq. 1) using the 48,077 examples with automatic labels. this training is done on tesla t4 gpus, for 5 epochs, and a learning rate initialized at 5 × 10 -5 and optimized using adam optimizer @xcite . following this, for the second stage, we further fine-tune the same model for 2 epochs using the same objective and hyper-parameters, but this time using the train set of human-annotated timed. 4 the hyper-parameters are selected by performing a grid search while observing performance on the validation set of timed. based on the performance on the validation set of timed, we set the threshold of s (eq. 2) to be 0.79 to categorize text as visual or non-visual. we refer to the model trained using our fine-tuning strategy as tip-clip -text imageability predictor clip, and report performance on the test set of timed. ## results and analyses evaluation on held-out test set of timed: we first evaluate the baselines and our approach on the test set of the human-annotated timed, computing macro-averaged f 1 , precision, recall scores, and classification accuracy. correlation of attention weights with mrc imageability scores: attention mechanisms could be taken as proxies for explainability @xcite . since the finetuned bert, pre-trained clip, and our tip-clip are attention-based models, we compute the correlation between average word-level attention scores (obtained from the last layer) on a given dataset with the imageability scores assigned by humans in the mrc lexicon. we compute these values for two datasets-the mscoco dataset @xcite and the test set of timed. we only consider words that occur more than once in the specific corpus. the trends are consistent across both datasets. the relative ordering of models in terms of the correlation of their attention scores with mrc imageability scores follows the same order as their performance on the test set of timed. however, all correlation scores are in the low range, indicating a non-trivial relationship between sentence-and word-level imageability. the same trends hold for propagated visualness scores; see app. a.4. we also analyze the reason behind higher correlation scores on mscoco with respect to the timed corpus in appendix a.4. ## conclusion and future work we propose the task of predicting the visualness of text and curate a human-annotated dataset of sentence-level visualness scores. additionally, we propose a two-stage fine-tuning objective for the task that involves training on a distantly supervised corpus followed by a smaller human-annotated corpus. comparisons with several baselines demonstrate the effectiveness of our approach in distinguishing visual and non-visual text. we analyze the attention weights and downstream text-to-image retrieval capabilities of the model. qualitative analysis of attention weights over textual input reinforces that our model attends to visual words to a greater extent. in closing, we show qualitative examples of how predicting text visualness can make text-toimage generation more effective. in the future, we will study alternate objectives for learning text visualness while ensuring that the learned representations are transferable to related downstream tasks. we are also interested in using measures relating to the quality of the images generated from text-to-image generation systems to decipher signals about the visualness of input text, enabling the creation of auto-labeled examples. as the aggregation of word-level visualness scores leads to poor predictability of sentence-level visualness, future work could aim to understand what linguistic factors (like compositionality) precipitate sentence-level visualness.