Describing Sets of Images with Textual- PCA (2022)

Abstract: We seek to semantically describe a set of images, capturing both the attributes of single images and the variations within the set. Our procedure is analogous to Principle Component Analysis, in which the role of projection vectors is replaced with generated phrases. First, a centroid phrase that has the largest average semantic similarity to the images in the set is generated, where both the computation of the similarity and the generation are based on pretrained vision-language models. Then, the phrase that generates the highest variation among the similarity scores is generated, using the same models. The next phrase maximizes the variance subject to being orthogonal, in the latent space, to the highest-variance phrase, and the process continues. Our experiments show that our method is able to convincingly capture the essence of image sets and describe the individual elements in a semantically meaningful way within the context of the entire set. Our code is available at: https://github.com/OdedH/textual-pca .

## introduction given a set of images with a common theme, it seems to be extremely easy for humans to identify and describe the common theme. while computer algorithms can identify in-set and out-of-set images using anomaly detection methods @xcite , describing the common theme seems more challenging. captioning methods @xcite @xcite are extremely effective in describing single images. however, one cannot directly employ such a method to the mean image representation, in hope of describing a set of images. since image captioning engines are trained to be specific and not to provide general terms, the resulting captions would not be generic enough. for example, images of people are described by image captioning methods as "man", "woman", "child", etc., and not by generic terms, such as "person". to create a representation of an image set, one has to employ higher-level themes. unable to do so, image captioning methods output non-grammatical phrases, which include, for example, phrases such as "cat dog" for sets that contain both pet species. our first contribution is to retool the blip @xcite image captioning tool to perform the task of image-set captioning. this is done through modifying the autoregressive process of blip without retraining the underlying network. wordnet @xcite is used to manipulate the likelihood of words by reducing the likelihood of hyponyms (specific terms) and increasing the likelihood of shared hypernyms (more generic terms). once a common theme is generated for the image set, we seek to identify the directions of variation within the set. this way, we can position the set elements in the context of the entire set. for example, staying with the example of images of persons, the images can vary by pose, age, hair style, facial expression, etc. motivated by the pca method, we seek to find the phrase whose visual-language similarity to the images of the set has the highest variance. then, once again following pca, we recover a phrase that maximizes the variance among all phrases that are orthogonal to the first phrase in the textual embedding space. our second contribution is, therefore, the ability to extract different phrases that capture directions of semantic variability in the image set. this is also done by retooling blip. in this case, the likelihood of the next token combines three terms: (i) the likelihood assigned by blip, (ii) a term that maximizes the variance of the similarity between the resulting phrase and the images of the input set, and (iii) an orthogonality term that distances the generated phrase from the previously found phrases. lsun -bridge celeba average sentence: image of a car parked principal sentence: ['suv', 'front', 'luxury', 'silver', 'black', 'red', 'used motor'] average sentence: image of a bridge view principal sentence: ['river', 'suspension'] average sentence: image of a adult person principal sentence: ['woman young', 'actress hair', 'man smiling', 'woman blond', 'woman actress', 'woman hair', 'woman posing'] suitable for describing sets of images in an intuitive way that combines the general theme with the modes of variation. consider, for example, fig. 1 . the average phrase clearly depicts that celeba is a dataset featuring images of adult persons or that lsun-bridge contains images of bridges. from the principal phrases we can learn about the traits of the dataset. for example, hair varies considerably in celeba, and the lsun-bridge images differ mainly in the type of the bridge (suspension) or whether it crosses a river. we evaluate our method on multiple datasets, including existing image datasets, such as celeba, lsun, and imagenet, and on sets of images obtained by applying clustering methods to large image collections. we also show that the obtained projections are more informative than the baselines in predicting attributes in celeba. finally, in lineup-type experiments, we show that the projections we create provide enough information for users to identify the images out of the set. ## related work some of the earliest deep learning attempts in image captioning relied on rnns with attention @xcite @xcite , while more recent approaches apply spatial reasoning via graphs @xcite , adapt to multi-image processing @xcite , and handle multi-modal input @xcite . annotations by humans are used to train most of the current captioning methods. as human references cannot account for every possible scene, other approaches rely more on web-scale unsupervised image-text datasets @xcite @xcite . in these approaches, smaller datasets annotated by humans are used as final fine-tuning. captions based on human annotations can enhance correspondence with human annotators. there is, however, a tendency for them to be repetitive and not particularly informative. recently, several methods that employ web-scale training data directly have been suggested. the first method, zerocap, employs clip @xcite , a prominent web-scale image-text matching model, to guide a pre-trained language model, gpt-2, to caption images @xcite without performing any training ("zero-shot"). magic @xcite , is another zero-shot method for image captioning that skews the next-token distribution of a gpt-2 language model to match a given image, based on the clip score. unlike zerocap, no gradient updates are applied. blip @xcite applies conventional training (not zero-shot) and jointly learns an image-text metric with a contrastive loss and a caption decoder head. also relevant to our work are causality frameworks that study concept discovery, such as tcav @xcite and cace @xcite . by intervening on labeled concepts, these approaches study causal effects. however, they require labeled data. although expertise can be acquired to identify the underlying structure of a problem, the process is still contaminated by human bias. in contrast to these approaches, ours does not require supervision. the discovery of visual concepts is also a subject of research. an intuitive approach involves clustering according to segmented regions @xcite . a shapley theorem-inspired method is described in subsequent work @xcite al., { } ... , , , suv red luxury car parked } } figure 2 : an overview of our approach to generate principal phrases by modifying blip's caption head. the first step is to create an average phrase, 'car parked' that captures the common features of the set of images i. then, during the auto-regression process, we consider two types of operators: (i) v for maximizing the variance over the images, and (ii) o for generating phrases that are orthogonal to the previous phrases (e.g., 'suv', 'red'). the result is a novel phrase 'luxury' that describes the car's pricing. ## method our approach is analogous to principal component analysis (pca), in which the projection directions are replaced by generated phrases, fig. 2 . pca identifies the vectors that best fit the data, in terms of euclidean distance between each data point and its reconstructed vector. it can be equivalently defined as finding directions that maximize the variance of the projected data. the principal vectors are, therefore, directions in which data varies, making them informative. this makes them highly useful for data analyses and dimensionality reduction. these vectors are extracted as mutually orthogonal vectors, in order to capture different directions, and for the projections to span the entire data set. while pca can be applied to embedded image vectors, we find that it lacks semantic meaning. in this work, we propose textual-pca. formally, our goal is to create a average phrase s 0 and a sequence of principal phrases s @xmath0, . . . s l ⟩, where l is the number of principal phrases. the phrases are aimed to be a concise set that describes a set of images @xmath1, . . . , x n }, where n is the number of images. the phrase s 0 captures common traits in i, and the rest capture different modes of variability. in the following sections we discuss how we find fluent principal phrases. a principal phrase is created by finding a textual direction that captures semantic variance within an image set. throughout our process, we employ the multi-modal blip model @xcite , which provides textual and image encoders (e t and e i , respectively), an image-text metric (blip m ), and a captioning head (blip c ). given the set of images i, we start the process with the generation of the average phrase s 0 . this phrase captures the common attributes of all images in the set. as the first step of generating the average phrase, we average the image representations, @xmath2. we then use x to initialize the auto-regressive blip's captioning head, ## results in all experiments, we set the hyperparameter λ @xmath3. these values were determined early on in the development process and kept unmodified for obtaining all results. evaluation is done on datasets covering a variety of objects and settings (i.e., named datasets). celeba @xcite is a large-scale dataset of faces. lsun @xcite contains ten different scene categories, from which we use the images of bridges, churches, and kitchens. the stanford cars dataset contains different types of vehicles. the first 20 categories of imagenet @xcite were also used. we also employ sets that were obtained by hierarchically clustering coco @xcite with an agglomerative algorithm in the clip embedding space. we cluster until we obtain 80 clusters. this number was selected since there is a small hinge around 80 in the graph depicting the number of clusters vs. the clustering error. second, with 80 clusters, large enough clusters are starting to form. fig. 3 demonstrates the level of specificity obtained for three levels of clustering. with 4 clusters, one can see a cluster of animals; with 80 clusters, a cluster of cats exists, and with 800 clusters, a subset of the cats with less variation is obtained. notably, to describe datasets with varying themes, clustering as a first step is a practical technique. for computational efficiency, from large datasets, such as celeba or lsun and some coco clusters, we sampled 500 images (each) as the working set. baselines our baselines consider sets of vectors in the embedding space of blip and then transcribe these into phrases. the sets of vectors are obtained by one of two methods, which are applied to the set of all embedding vectors extracted, using the image encoder, for a given set of images. the first method is pca, which uses singular value decomposition, and the second method is cosine k-means clustering, which was selected since figure 4 : four image sets: coco-horses, coco-baseball, and coco-street (names were assigned by us) obtained by clustering coco, a large visual dataset, and lsun-church. for each image set we present the average phrase and principal phrases generated by our method (underlined). for comparison, we present four baselines: zerocap+pca, zerocap+kmeans, magic+pca, and magic+kmeans, obtained in blip space. note that k-means and pca are related with relaxation assumptions @xcite . turning these sets of principle vectors into phrases is done through either the zerocap method @xcite or using magic @xcite . since our method operates in the blip encoding space, we created a version of both zerocap and magic that are blip-based. the results of the unmodified baselines when using clip embedding space are also presented. we also attempted to use blip's captioning head to generate text from the principal vectors. however, despite considerable effort, the generation collapsed into the model's default caption. this is most likely due to the sensitivity of the captioning head to distribution changes, i.e., the distribution of pca vectors differs considerably from the encoding of single images that blip was trained on. we further evaluate a naive baseline of most frequent words. we used blip captioning head to generate captions for the images in the set. then, we created phrases based on the most frequent words. ## discussion and limitations we identify a few ways in which our results could be extended. first, similarly to other generative tasks, evaluation of the results is not straightforward. since describing an image set in a way that encompasses both the common theme and the modes of variation is a novel task, there is no established methodology for this evaluation. we believe that the lineup experiments presented demonstrate that humans are able to relate the obtained projections to the images. given more resources to train human annotators, it would be interesting to obtain human phrases for both the theme and the variation and evaluate the degree to which these would match the method's results. second, our method could help create more informative image captions by including the information of the radar plots we present within the generated text. this way, a rich and descriptive captioning, which addresses the common variations from in-set images with a common theme, would be created. finally, there is no reason not to apply our method to sets of phrases or paragraphs, by replacing clip with a summarization engine, such as those based on transformers @xcite . distancing ourselves even further from the current work, we note that with the advent of powerful image generation engines @xcite the role that images and text play in our work could be reversed. images can visually capture the common theme of a set of phrases or paragraphs as well as their modes of variation. ## conclusions dimensionality reduction methods capture the most significant information of the input vectors using a smaller set of variables. however, these modes are free from semantic constraints and often mix multiple attributes, due to correlations that exist in the data. in this work, we follow in the footsteps of pca, perhaps the most widely used dimensionality reduction method, and propose a method for extracting orthogonal semantic directions that describe a set of images in the latent space of a vision-language model. first, the "centroid" phrase, which describes the main theme of the set is extracted. then, the directions with the highest variability in the vision-language similarity to the images of the set are extracted. our solution combines the blip image captioning model with information derived from the wordnet graph. an extensive set of experiments demonstrates that the obtained list of semanticallyorthogonal phrases accurately describes the set of images given as input.