C ap O n I mage: Context-driven Dense-Captioning on Image (2022)

Abstract: Existing image captioning systems are dedicated to generating narrative captions for images, which are spatially detached from theimage in presentation. However, texts can also be used as decorations on the image to highlight the key points and increase theattractiveness of images. In this work, we introduce a new taskcalled captioning on image (CapOnImage), which aims to generatedense captions at different locations of the image based on contextual information. To fully exploit the surrounding visual context togenerate the most suitable caption for each location, we propose amulti-modal pre-training model with multi-level pre-training tasksthat progressively learn the correspondence between texts and image locations from easy to difficult. Since the model may generateredundant captions for nearby locations, we further enhance thelocation embedding with neighbor locations as context. For thisnew task, we also introduce a large-scale benchmark called CapOnImage2M, which contains 2.1 million product images, each with anaverage of 4.8 spatially localized captions. Compared with other image captioning model variants, our model achieves the best resultsin both captioning accuracy and diversity aspects.

## introduction building upon the advances in computer vision and natural language processing areas, the new research direction called vision-and-language has attracted more and more attentions, which pushes to tackle new problems that need to bridge the two areas to advance the concept comprehension and reasoning capabilities. the image captioning task, as one of the most classic vision-and-language tasks, aims to generate natural language descriptions for images (vinyals figure 1 : illustration of descriptive texts on image scenarios. the captions and images in existing tasks (e.g visual genome (a) @xcite ) are spatially detached, without any mutual association. in contrast, our caponimage task aims to generate descriptive text on the image, which has strong necessity and broad application prospects in some scenarios @xcite and (c)). @xcite @xcite . however, the captions and the images in this task are spatially detached in presentation, without any association between each other (figure 1(a) ). in fact, there are many scenarios where the image and text are tightly associated. for example, the product images on e-commercial website (figure 1(b) ) usually contain descriptive texts, explaining multiple perspectives of the product (e.g., product characteristics, selling points, etc) , which makes the image more informative and attractive. on the social media platform, users usually upload daily pictures with descriptive texts as decorations (figure 1(c) ). therefore, it is significant to explore captioning on the image, which requires to consider not only the visual description, but also the text description placement on the image. besides, to generate the informative captions in these scenarios, additional textual knowledge is usually needed, such as the product informa-tion and the background story about the image etc. therefore, in this work, we introduce a new task called caponimage, which aims to generate dense captions at different locations of the image based on contextual information. the caponimage task involves two steps, where the model needs to first predict a reasonable and aesthetic text layout @xcite @xcite , and then generates a phrase or sentence for each text box. in this work, we simplify this task to generate captions for a provided list of text box locations, thereby removing the requirement for layout prediction. therefore, the main focus of this work is to generate captions that are most suitable for the corresponding image locations. the caponimage task involves two new challenges: (i) better understanding of context: the captions at different locations can be greatly diverse. as shown in figure 1 (b), the texts around the product are descriptive captions describing the product features, while those at the bottom introduce the selling points. therefore, the model needs to fully exploit the visual context around the text box to determine what caption is suitable to generate here. since our task aims to generate captions on image, location context is vital for our task which is also validated on to solve the aforementioned challenges, we propose a multi-modal pre-training and fine-tuning framework which contains multi-level pre-training tasks to effectively exploit multi-modal contextual information. first, to better exploit context, we design multi-level pre-training tasks to help the model "feel" the context. it explicitly equip the model with the ability to distinguish which captions are appropriate for the current location and image while which are not. besides, inspired by the learning progression of easy to complex biological vision systems, we further propose a progressive training strategy which learns multi-level pre-training tasks from easy to hard. second, to solve the problem of caption redundancy, we introduce a neighborenhanced location encoding module, which utilizes the surrounding text box locations as context, so that our model can "see" the adjacent context. we show the captioning diversity results with different ranges of adjacent text boxes involved in the location encoding module, and demonstrate the importance of such neighbor context. in order to evaluate our model and benchmark progress in the caponimage task, we introduce the caponimage2m dataset. it contains 2.1 million product images crawled from an e-commercial website, and each image contains multiple spatially localized captions describing the associated product. we automatically acquire the text contents and their spatial locations from the image via ocr @xcite , and finally collect 4.8 captions for each image on average. we also crawl the product title and attributes as additional context information for caption generation. with the empirical analysis on caponimage2m dataset, we show that the visual context, location information and the additional product information are beneficial for the caption generation, and our model can generate corresponding types of captions at different spatial locations (figure 3 ). furthermore, we demonstrate that our proposed neighbor-enhanced location encoding module and multi-level pre-training tasks significantly improve the captioning accuracy and diversity. the main contributions of this work are as follows: ## related work in recent years, significant progress has been made in the image captioning task @xcite @xcite @xcite , which aims to describe the image content in one natural sentence. with the advances in visual understanding abilities, researchers are not satisfied with generating dull and less informative captions and extend the traditional image captioning task along two directions. the first direction is called dense captioning @xcite @xcite @xcite , which targets to describe detailed visual content with a set of sentences. johnson et al. @xcite propose a fully convolutional localization network to unify the object detection @xcite @xcite and image captioning in one framework to predict a set of descriptions across object regions. krishna et al. @xcite ) migrate it to the video, which aims to predict sequential event proposals and generate description for each clip. in these works, the dense captions deliver more details of visual content than traditional single sentence. the second direction is called text-aware image captioning @xcite @xcite , where the model generates captions not only according to the image, but also utilizes additional textual information as context. besides , sidorov et al. @xcite and @xcite propose to generate image captions with scene texts, which exploit ocr tokens as the textual context. although impressive progresses have been made along the two directions, they still remain separate. the proposed caponimage task can be considered as a combination of the two directions, where the model needs to first predict the text layout (spatial locations on the image) and then generate caption for each location conditioned on both the image and textual information. there are several key distinctions between our proposed caponimage task and dense captioning: 1) dense image captioning aims to generate captions for subregions within an image, and there is no length limitation of captions. however, our task is to generate text and affix it to specific regions within an image, and the text length should be controlled according to the region size. 2) the visual content is the only input of dense captioning. while for our task, there are three inputs: visual content, additional textual information, and the specified location to affix. all of them will impact the generated text content. 3) the generated text is a plain description of the visual content for dense captioning. but for our task, the generated text is on the ## caponimage2m dataset in this section, we introduce our proposed capon-image2m dataset, which is the benchmark for the caponimage task. we first present an overview of the dataset collection and statistics, and then compare it with other related image captioning datasets. a more detailed datasheet describing the motivation, composition, and recommended uses of our caponimage2m dataset following @xcite can be found in the appendix a. ## model in this section, we introduce our caponimage method based on the pre-training and fine-tuning framework as illustrated in figure 2 . first, we introduce the multi-modal representation of visual, location coordinates, and product information of the given input images. then, a progressive training strategy with multi-level pre-training tasks is proposed to enhance the correspondence learning between textbox locations and captions for caption generation with a multi-layer transformer. ## experiments we carry out experiments to evaluate the ability of models for captioning on image given a provided text layout on the caponimage2m dataset. we evaluate the caption generation qualities from multiple aspects, including the accuracy measurement against the references, and the diversity measurement within an image. since the correct caption for each textbox location is not unique, we further evaluate the fitness of generated captions to the corresponding textbox locations with respect to the caption length and type. besides, we also conduct human evaluations. more results can be found in supplementary materials. ## conclusion in this work, we propose a new vision-andlanguage task called caponimage, which aims to generate dense captions at different locations on an image with visual and textual context. we propose a multi-modal pre-training and fine-tuning model with multi-level pre-training tasks from easy to difficult for the correspondence learning between image location and text, and enhance the current location embedding with neighboring locations to reduce captioning redundancy. experimental results shown that our model can generate controllable length and type of captions at different image locations. in the future work, we will explore to generate dense captions with self-predicted text layout and combine the layout generation with caption generation in one joint framework to benefit from each other.