SANDI : Story-and-Images Alignment (2021)

Abstract: The Internet contains a multitude of social media posts and other of stories where text is interspersed with images. In these contexts, images are not simply used for general illustration, but are judiciously placed in certain spots of a story for multimodal descriptions and narration. In this work we analyze the problem of text-image alignment, and present SANDI, a methodology for automatically selecting images from an image collection and aligning them with text paragraphs of a story. SANDI combines visual tags, user-provided tags and background knowledge, and uses an Integer Linear Program to compute alignments that are semantically meaningful. Experiments show that SANDI can select and align images with texts with high quality of semantic fit.

## introduction it is well-known (and supported by studies @xcite ; @xcite ) that the most powerful messages are delivered with a combination of words and pictures. on the internet, such multimodal content is abundant in the form of news articles, social media posts, and personal blog posts where authors enrich their stories with carefully chosen and placed images. as an example, consider a vacation report, to be posted on a blog site or online community. the backbone of the travel report is a textual narration, but the user typically places illustrative images in appropriate spots, carefully selected from her photo collection from this trip. these images can either show specific objective highlights such as waterfalls, mountain hikes or animal encounters, or may serve to depict the thematic mood of the trip, e.g., by showing nice sunsets. another example is brochures for various organizations. here, the text describes the mission, achievements and ongoing projects, and is accompanied with judiciously selected and placed photos of buildings, people, products and images depicting the subjects of interest, e.g., galaxies or telescopes for research in astrophysics. the generation of such multimodal stories requires substantial human judgement and reasoning, and is thus time-consuming and labor-intensive. in particular, the effort on the human side includes selecting the right images from a pool of storyspecific photos (e.g., the traveler's own photos) and possibly also from a broader pool for visual illustration (e.g., from pinterest). even if the set of photos were exactly given, there is still considerable effort to place them within appropriate paragraphs, paying attention to the semantic coherence between surrounding text and image. in this paper, we set out to automate this human task, formalizing it as a story-and-images (sandi) alignment problem. problem statement. given a story-like text document and a set of images, the problem is to automatically decide where individual images are placed in the text. figure 1 depicts this task. the problem comes in different variants: either all images in the given set need to be placed, or a subset of given cardinality must be selected and aligned with text paragraphs. formally, given n paragraphs and @xmath0, assign @xmath1, such that each paragraph has at most one image. ## model for story-images alignment our story-image alignment model constitutes an integer linear program (ilp) which jointly optimizes the placement of selected images within a story. the main ingredient for this alignment is the pairwise similarity between images and units of text. we consider a paragraph as a text unit. ## quality measures in this section we define metrics for automatic evaluation of the text-image alignment problem. the two tasks involved -image selection and image placement -call for separate evaluation metrics as discussed below. ## experiments and results we evaluate the two flavors of sandi -complete alignment and selective alignment -based on the quality measures from section 5. ## conclusion in this paper we introduced the problem of storyimages alignment -selecting and placing a set of representative images within a story. we analyzed features towards meaningful alignments from realworld multimodal datasets -lonely planet and asia exchange blogs -and defined various evaluation measures. we presented sandi, a methodology for automating such alignments by a constrained optimization problem maximizing semantic coherence between text-image pairs jointly for the entire story. evaluations show that sandi pro-duces semantically meaningful alignments. nevertheless, some follow-up questions arise. additional features. our feature space covers most natural aspects. in addition, gps locations where available may provide cues for geographic named entities, while timestamps may capture temporal aspects of a storyline. abstract and metaphoric relations. we do not address stylistic elements like metaphors and sarcasm in text, which would entail more challenging alignments. for example, the text "the news was a dagger to his heart" should not be paired with a picture of a dagger. although user provided tags may provide some cues towards such abstract relationships, a deeper understanding of semantic coherence is desired. the proposed text-image alignment system is available at https://sandi.mpi-inf.mpg.de , and a video of the demonstration can be viewed at https://youtu.be/k5gu2pnxdnu .