D omain- S pecific L exical G rounding in N oisy V isual- T extual D ocuments (2020)

Abstract: Images can give us insights into the contextual meanings of words, but current image-text grounding approaches require detailed annotations. Such granular annotation is rare, expensive, and unavailable in most domain-specific contexts. In contrast, unlabeled multi-image, multi-sentence documents are abundant. Can lexical grounding be learned from such documents, even though they have significant lexical and visual overlap? Working with a case study dataset of real estate listings, we demonstrate the challenge of distinguishing highly correlated grounded terms, such as “kitchen” and “bedroom”, and introduce metrics to assess this document similarity. We present a simple unsupervised clustering-based method that increases precision and recall beyond object detection and image tagging baselines when evaluated on labeled subsets of the dataset. The proposed method is particularly effective for local contextual meanings of a word, for example associating “granite” with countertops in the real estate dataset and with rocky landscapes in a Wikipedia dataset.

## introduction multimodal data consisting of text and images is not only ubiquitous but increasingly diverse: libraries are digitizing visual-textual collections (british library labs, 2016; the smithsonian, 2020); news organizations release over 1m images per year to accompany news articles (the associated press, 2020); and social media messages are rarely sent without visual accompaniment. in this work, we focus on one such specialized, multimodal domain: new york city real estate listings from the website streeteasy. to effectively index image-text datasets for search, retrieval, and other tasks, we need algorithms that learn connections between modalities, doing so from data that is naturally abundant. in documents that contain multiple images and sentences, there may be no explicit annotations for image-sentence associations or bounding box-word associations. as a result, existing image captioning/tagging methods are difficult to adapt to unlabeled multi-image, multi-sentence documents. indeed, most prior image captioning work has focused on rare and expensive single-image, singlecaption collections such as mscoco, which focuses on literal, context-free descriptions for 80 object types @xcite . similarly, off-the-shelf object detectors may not account for contextual factors: to an imagenet classifier, "pool" refers to a pool table @xcite . in the specialized real estate context, "pool" commonly refers to a swimming pool. ## task and models we consider a direct image-text grounding task: for each word type, we aim to retrieve images mostassociated with that word. models are evaluated by their capacity to compute word-image similarities that align with human judgment. entsharp. for each image in a document we iteratively infer a probability distribution over the words present in the document. during training, these distributions are encouraged to have low entropy. the output is an embedding of each word into image space: the model computes word-image similarities in this joint space. this can be thought of as a soft clustering, such that each word type is equivalent to a cluster but only certain clusters are available to certain images. this approach could also be situated within the framework of multipleinstance learning @xcite . each image i starts with a fixed feature vector @xmath0. let i be the set of these image embeddings. for each word w we initialize a cluster centroid @xmath1. let 1 i,w be 1 if image i co-occurs with word w in any document and 0 otherwise. each image i is assumed to have a membership distribution p i over words, where p i is initially uniform over co-occurring words. at each iteration, cluster centroids are updated to the weighted average of co-occurring images' embeddings: w := @xmath2. each image's distribution over clusters is updated by taking a softmax of the cosine similarity between pairs of image and word embeddings, first multiplying similarities by a sharpness coefficient foot_2 equal to the iteration number, and finally masking for cooccurrence: after training, we calculate the cosine similarity between image embeddings and the learned wordcluster embedding. untrained entsharp baseline. we consider a simple averaging baseline, corresponding to the cluster center initializations of entsharp: each word embedding is set to the mean of the features for all its co-occurring images. object detection baselines. we can use ima-genet to identify objects, but most words in the full vocabulary are not in the imagenet labels. we implement two object detection baselines that map images to object names and then match object names to words in documents @xcite . for each image, we first get the image's top class predictions from densenet169 @xcite pretrained on the imagenet classification task @xcite . these predictions are for a whole image and are restricted to the 1000 ima-genet labels. we bridge the gap between imagenet labels and the vocabulary by then creating an image vector by averaging the word vectors corresponding to these predictions. finally, for each word in the full vocabulary, we rank images by the cosine similarity between the word's vector and these image vectors. words are represented in one baseline by word2vec embeddings @xcite and in the other by the output of roberta @xcite when fed a single token as input. image tagging baselines. inspired by @xcite , we implement softmax and multinomial regression models. the former, softmax regression, takes image features and predicts a distribution over the words in the vocabulary with a softmax loss. it computes the word type indicator vector for each document, i.e., 1 if word w was in the document else 0, and then ℓ 1 normalizes. multinomial regression computes the word type indicator vector, and-instead of normalizing-computes the logistic sigmoid loss treating the labels as 0/1 indicators. this is equivalent to training a separate logistic regression for each word type to predict the presence/absence of a word type in each document, given the image features. both models finally use the predicted conditional distributions to produce a ranking of images for each word. experimental details for entsharp. for each image, features are extracted from the final preclassification layer of densenet169 pre-trained on imagenet @xcite and then randomly projected from 1,664 dimensions to 256. 5 we use a vocabulary of 7,971 words that occur at least ten times across this corpus and wikipedia (to eliminate misspellings). we run entsharp for 100 iterations. 6 setups for baselines are comparable, and more details are available in appendix c. 5 random projection is a time and memory optimization. the baseline approaches have access to full feature vectors. 6 the average runtime is 198 ± 3.6 minutes on an intel xeon gold 6134 (3.20ghz) cpu with 512 gb ram. results. as shown in qualitatively (figure 3 ), we see that entsharp associates "bedroom" with empty rooms containing a door and a window while the word2vec baseline associates the word with rooms that contain a bed or a sofa. similarly, "outdoor" manifests in streeteasy as building exteriors, but the roberta baseline returns images of bike rooms, presumably because bicycles are usually seen outdoors. in streeteasy the word "pool" more frequently refers to swimming pools rather than the billiards tables seen in imagenet. the baseline is not technically wrong in this case (indeed, we marked pool tables as correct), but it misses the more common contextual meaning of the word in the local collection. finally, none of the baselines are able to handle"fitness". wikipedia experiments. we also ran entsharp on a multimodal wikipedia dataset @xcite . figure 1 shows that the algorithm often grounds words differently in wikipedia's much broader range of images than it does in the streeteasy dataset. similarly, top ranked images in wikipedia for "fitness" included marathon runners rather than the streeteasy dataset's exercise rooms. ## discussion we present entsharp, a simple clustering-based algorithm for learning image groundings for words. it is motivated by the unlabeled multimodal data that exists in abundance rather than relying on expensive custom datasets. by encouraging words to compete to claim responsibility for images, we "sharpen" the resulting image/word associations. the method is effective at finding contextual lexical groundings of words in unlabeled multi-image, multi-sentence documents even in the presence of high cross-document similarity. one area for future work would be to better identify and model words that either don't have a visual grounding or whose identified visual grounding doesn't align with human expectation. for example, the word "gristedes" (the name of a super- evaluation word image top densenet169 predictions "kitchen" 'dishwasher', 'microwave', 'refrigerator' "bedroom" 'sliding door', 'wardrobe', 'window shade' "outdoor" 'mountain bike', 'bicycle-built-for-two' "pool" 'pool table', 'fountain', 'tub' "washer" 'washer', 'microwave', 'reflex camera' "fitness" 'shoe shop', 'dumbbell', 'barbell'