Control Image Captioning Spatially and Temporally (2021)

Abstract: Generating image captions with user intention is an emerging need. The recently published Localized Narratives dataset takes mouse traces as another input to the image captioning task, which is an intuitive and efficient way for a user to control what to describe in the image. However, how to effectively employ traces to improve generation quality and controllability is still under exploration. This paper aims to solve this problem by proposing a novel model called LoopCAG, which connects Contrastive constraints and Attention Guidance in a Loop manner, engaged explicit spatial and temporal constraints to the generating process. Precisely, each generated sentence is temporally aligned to the corresponding trace sequence through a contrastive learning strategy. Besides, each generated text token is supervised to attend to the correct visual objects under heuristic spatial attention guidance. Comprehensive experimental results demonstrate that our LoopCAG model learns better correspondence among the three modalities (vision, language, and traces) and achieves SOTA performance on trace-controlled image captioning task. Moreover, the controllability and explainability of LoopCAG are validated by analyzing spatial and temporal sensitivity during the generation process.

## introduction image captioning is a fundamental task to examine whether an intelligent system can understand the visual world by letting the system describe it with natural language. generating a reasonable caption requires the model to link linguistic tokens to objects, relationships, scenes of the visual world in the input image. thus, a great captioning model will help us better understand what characteristics promise a good joint visual-linguistic representation. ## conclusion in this paper, we focus on the controlled image captioning task and find mouse traces provide an intuitive and efficient way for a user to control the description. we propose a novel caption generation model with contrastive constraints and attention guidance called loopcag to control the captioning process spatially and temporally. the experimental results demonstrate the our model's effectiveness, and our work will inspire more future research on vision-linguistic understanding and generation. 2025 original image image with trace ground truth captioning baseline captioning in this image i can see a person wearing white shirt, blue tie, blue blazer, skirt and black shoes is standing and holding a black colored bag in his hand. in the background i can see the white colored wall. in this picture we can see a man standing and holding a mobile in his hand, in the background we can find a wall. +trace captioning loopcag captioning in the middle of the image a man is standing and smiling and he is holding a tennis racket. behind him there is a cloth on the red color wall. bottom left side of the room there are two shoes. in this image i can see a person wearing blue coat, black pant and black shoe is standing and holding a black colored bag in his hand. in the background i can observe the white colored wall.