Query-based Image Captioning from Multi-context 360 cdegree Images (2023)

Abstract: A 360-degree image captures the entire scene without the limitations of a camera’s field of view, which makes it difficult to describe all the contexts in a single caption. We propose a novel task called Query-based Image Captioning (QuIC) for 360-degree images, where a query (words or short phrases) specifies the context to describe. This task is more challenging than the conventional image captioning task, which describes salient objects in images, as it requires fine-grained scene understanding to select the contents consistent with user’s intent based on the query. We construct a dataset for the new task that comprises 3,940 360-degree images and 18,459 pairs of queries and captions annotated manually. Experiments demonstrate that fine-tuning image captioning models further on our dataset can generate more diverse and controllable captions from multiple contexts of 360-degree images.

## introduction image captioning is a task of describing the context of an image in natural language @xcite @xcite . existing image captioning datasets, such as microsoft coco captions @xcite and flickr30k @xcite , are based on the conventional images with limited camera field-ofview. for such images, the contexts of the images are selective and even simplified because people tend to choose what to photograph and crop out. such images often include a clear context, such as "a bird flying over a body of water" or "a cat sitting on a mat." image captions in the existing dataset often follow such salient context, ignoring minor but non-negligible contexts of the image. this becomes particularly problematic in the case of 360-degree images captured by omnidirectional cameras, as they capture the entire visible context of the scenes indiscriminately and preserve rich minor details. ## quic-360 • dataset we introduce the quic-360 • dataset, which contains 3,940 panoramic images annotated with query-focused 18,459 captions. this is the first dataset that uses linguistic information to control the scene description for 360-degree images. we first describe the collection of 360-degree images from the web and present how to annotate these images. we also present statistics of our quic-360 • dataset. ## experiment we validate that the model learning with the quic-360 • dataset surely enhances the controllability of scene captioning with baseline models. we use 3,000/400/400 images and 9,438/1,251/6,000 corresponding query-caption pairs in the train/valid/test split. ## conclusion we have proposed a query-based image captioning for 360 • images and constructed the quic-360 • dataset for our new task, containing humanannotated captions focused on queries. in quic, models leverage textual queries to describe images with multiple visual contexts from 360-degree images. experimental results show that further finetuning existing image captioning models with our dataset can improve the controllability and diver-sity of captions on 360-degree images. in addition, models trained on our dataset are robust to both unknown queries and out-of-domain images to some extent. further investigation also reveals that clipbased metrics are unsuitable for the evaluation of our task, highlighting the target difference between quic and the conventional image captioning task.