Composition and Deformance: Measuring Imageability with a Text-to-Image Model (2023)

Abstract: Although psycholinguists and psychologists have long studied the tendency of linguistic strings to evoke mental images in hearers or readers, most computational studies have applied this concept of imageability only to isolated words. Using recent developments in text-to-image generation models, such as DALLE mini, we propose computational methods that use generated images to measure the imageability of both single English words and connected text. We sample text prompts for image generation from three corpora: human-generated image captions, news article sentences, and poem lines. We subject these prompts to different deformances to examine the model’s ability to detect changes in imageability caused by compositional change. We find high correlation between the proposed computational measures of imageability and human judgments of individual words. We also find the proposed measures more consistently respond to changes in compositionality than baseline approaches. We discuss possible effects of model training and implications for the study of compositionality in text-to-image models.

## introduction did you ever read one of her poems backward, because the plunge from the front overturned you? -emily dickinson @xcite imageability is the capacity of a linguistic string to elicit imagery. humans can identify highly imageable words, such as "banana", "beach", "sunset", and words with low imageability, such as "criterion", "actuality", "gratitude"; however, it's difficult to measure imageability computationally. psycholinguists and psychologists have conducted interviews with humans and released databases of the human imageability ratings, such as the medical research council (mrc) psycholinguistic database, to help researchers in their fields, as well as other fields such as linguistics and computer science, to measure these intangible attributes of verbal content @xcite . however, conducting these interviews is costly and laborious. volunteers had to rate hundreds and thousands of words, thus expanding these psycholinguistics databases to the size of modern natural language processing (nlp) corpora such as corpus of contemporary american english (coca), which has more than 60k lemmas with word frequency and part of speech tags, is unrealistic. furthermore, these ratings are only on isolated words. to calculate a sentence's imageability, many applications have simply added the scores of its component words. other work uses each word's concreteness level, which research has found highly correlated with imageability @xcite @xcite . these methods, while they are able to roughly measure imageability, dismiss a fundamental property of a sentence: compositionality. compositionality depends on word order as well as word choice. sentences with the same component words but with different word order vary not only in their syntax and semantics, but also their intensity and construction of the imagery, e.g. the famous example: "the dog bit the man" vs. "the man bit the dog". previous bag-ofword approaches such as @xcite would consider a sentence and its backward version as having the same imageability, but to human readers, the level of imageability has significantly altered. in this paper, we investigate a new computational approach to measure imageability using text-toimage models such as dall•e mini. we propose two methods to measure the imageability level of both individual words and connected text by generating images with a text-to-image model. we test our methods with both isolated words from the mrc database and connected text from poems, image captions, and news articles, and compare our result with previous bag-of-words methods such as @xcite and @xcite . we propose, firstly, measuring the average clip score provided by dall•e mini and, secondly, calculating the average pairwise cosine similarity between embeddings computed by a pretrained resnet-18 model. we find that these methods are more highly correlated to human imageability judgments of individual words than other automatic techniques proposed by @xcite . we further demonstrate the robustness of our proposed methods by subjecting connected text to various deformances. as suggested by the epigraph from emily dickinson, literary scholars design transformations of the original text to elicit more or less intense reactions from human readers @xcite and help them calibrate their interpretations of literary works. this approach is similar to how contrastive training might be used for models such as word2vec or bert. we compare our computational imageability measurements with human judgment collected from amazon mechanical turk (amt) and found that, on mrc isolated words, our methods have reasonable correlations with human judgment, but on connected text, the correlations vary among different types of connected text. ## related work paivio introduced the idea of imageability and defined imageability as "the ease/difficulty with which words arose a sensory experience" @xcite . although imageability is associated with many modalities, some researchers have found that visual modality is its most prominent modality @xcite . imageability is also highly correlated with concreteness @xcite @xcite , and concreteness has also been found to be most related to visual modality @xcite . however, some researchers have found their relation to be more complex: words with high imageability and concreteness "evoke sensations connected to the perception of the objects they denote", words with high imageability and low concreteness "evoke sensations connected to affective arousal" @xcite . an example for the latter is "anger", which is highly imageable since most of us have the experience of being angry, but "anger" itself is an abstract word. since imageability and concreteness are highly correlated, in this paper, we will compare some works using concreteness to measure word imageability, but we agree that these two attributes should ideally be disentangled @xcite . the imageability rating of an isolated word in psycholinguistics research is usually derived from interviewing human subjects: asking them how imageable a word or a concept is on a 7-point likert scale @xcite @xcite . due to the cost of this procedure, the most popular dataset, mrc psycholinguistics database, combines three different sources and, even so, has a limited vocabulary size of 9240. others have attempted to expand mrc imageability ratings using synonyms and hyponyms iden-tified in wordnet @xcite ; @xcite made a small expansion of 3000 words but only on disyllabic words. concreteness ratings in the mrc database were obtained in the same way as imageability ratings, although recently @xcite , using amazon mechanical turk, was able to expand the vocabulary to 37,058 words. a concrete idea is assumed to have more shared representation than an abstract idea. @xcite estimate the concreteness of a word in a database by measuring how clustered a word's associated images according to the image embeddings provided by resnet-18. @xcite estimate imageability using more explicit visual features, such as color distributions, local and global gradient descriptions, and high-level features, such as image theme, content, and composition. however, this approach is supervised and requires a large amount of data to train. the above works are all on isolated words. our paper aims to measure a sentence's imageability beyond bag-of-words methods, where the latter is insensitive to compositional change and imagery loss/gain. we will compare both to the work of @xcite , who measure imageability with bag-of-words models, and of @xcite , who estimate word concreteness in an unsupervised manner. we will demonstrate our methods' advantage via correlation with singleword human judgments from mrc ( the poetry dataset consists of 355 english poems written by different types of poets: imagists, contemporary poets, contemporary amateur poets, and 19th-century poets. they were collected by @xcite from different poetry websites and publications: des imagistes (1914), some imagist poets (1915), contemporary american poetry @xcite , amateur writing (website), and famous poets and poems (website). in their paper, they use various linguistic and psycholinguistic attributes as features for identifying different poets and poem types. in this paper, however, we do not focus on poem-level classification. the dataset was provided by the authors of @xcite for research purposes. conceptual 12m (cc12m) foot_1 @xcite is a dataset of 12 million image-caption pairs, designed for vision-and-language pre-training. we randomly sampled 5000 captions from the dataset. in the 12m captions, real names are replaced @xmath0, and some captions contain hashtags. we only use captions with @xmath1. cornell newsroom dataset foot_2 @xcite is a summarization dataset of 1.3 million articles from 38 major english-language news publications. we extract sentences using nltk "sent_tokenize", then randomly sample 5000 sentences of 10-30 words from the training set original news articles. ## human judgment we recruit workers on amazon mechanical turk (amt) to rate the imageability of randomly sampled mrc words, poem lines, captions, and news sentences. workers were informed that they would be participating in a psycholinguistics and natural language processing research study before they accepted the task. for mrc vocabulary, we sample 400 words in total, and for each connected text corpus, we sample 120 sentences for each deformance. we recruit 300 workers in total: the first 100 workers rated 4 mrc words and 6 poem lines each, and the second 100 workers rated 6 captions each, and the rest of the workers rated 6 news sentences. each worker is only allowed to participate in the entire research once. in total, we recruit 300 workers, and workers are paid $0.50 for answering 6 or 10 questions. every participating worker has hit approval rates for all requesters' hits greater than 95% and number of hits approved greater than 100, and we require their location to be in the us or canada. for poems, we mistakenly use two lines of deformed text as one single prompt for workers to rate, and we counter that mistake by averaging the aveclip , imgsim from the two lines. for connected text, we test our methods on three different corpora: poem lines, captions, and news sentences. unlike isolated words, sentences' meaning is compositional and depends on word choice and word order. a good sentence imageability method, therefore, should detect the change in imageability caused by compositional change. the biggest downfall of previous bag-of-words methods is that when a sentence is subject to a deformance such as permutation, imageability is unchanged, which is contradictory to human expectation. with a text-to-image model, our methods are able to take the entire sentence as one entity, preserving its composition. we test our methods against a noisy amt human judgment ( the language of these three different corpora is very different. overall, image captions have the highest average imageability rating as well as brysbaert et al. concreteness rating, with poems being the second most imageable, and news sentences being the second most concrete. since image captions' language is usually concise, and it possibly has higher noun density, it's reasonable to see overall the highest impact from deformances under aveclip and imgsim. all three corpora experience negative impact from permutation under both aveclip and imgsim, which we assume to be the strongest deformance since it completely randomizes the word order of a sentence. when aveclip and imgsim have opposite signs, we notice that a higher absolute value from one measurement also tends to result in a lower absolute value of the other measurement if their signs are different, thus we use fig 4 to explore the percent change distribution between two measurements. in fig 4 , we look at the original lines with the highest and lowest 10% aveclip and imgsim and inspect the percent change between them and their different deformances (fig 4 ): the lines with the highest scores consistently decrease their scores after deformances, and vice versa. while the lines with top scores follow our intuition, the increase in lines with low scores reverses the mean: given the lowest score is 0, there isn't much room for the imageability score to fall further. the performance difference between different connected text also makes us wonder if the training data of the text-to-image model has an effect on the performance. the performance on captions is more contrastive in future work should consider further ways of measuring imageability computationally. as more text-to-image models become available and hopefully more transparent with their training process, we hope researchers will be able to compare different models' performance.