“Let’s not Quote out of Context”: Unified Vision-Language Pretraining for Context Assisted Image Captioning (2023)

Abstract: Well-formed context aware image captions and tags in enterprise content such as marketing material are critical to ensure their brand presence and content recall. Manual creation and updates to ensure the same is non trivial given the scale and the tedium towards this task. We propose a new unified Vision-Language (VL) model based on the One For All (OFA) model, with a focus on context-assisted image captioning where the caption is generated based on both the image and its context. Our approach aims to overcome the context-independent (image and text are treated independently) nature of the existing approaches. We exploit context by pretraining our model with datasets of three tasks- news image captioning where the news article is the context, contextual visual entailment, and keyword extraction from the context. The second pretraining task is a new VL task, and we construct and release two datasets for the task with 1.1M and 2.2K data instances. Our system achieves state-of-the-art results with an improvement of up to 8.34 CIDEr score on the benchmark news image captioning datasets. To the best of our knowledge, ours is the first effort at incorporating contextual information in pretraining the models for the VL tasks.

## introduction large enterprises have several teams to create their content for the purpose of marketing, campaigning, or even maintaining a brand presence. multimodal assets, particularly images are an integral part of this. the scale and the speed at which one needs to create and update content, especially to ensure personalization requires several resources, which in turn acts as a hindrance to the success of the enterprise. opportunistic updates are critical for success in the current competitive marketing and advertising scenario. ensuring that every multimodal asset associated with any piece of enterprise content has an appropriate caption and tag is not possible given the scale. while apparently a nuanced aspect of any image, the caption serves as the key information carrier of what the image is all about, in turn ensuring the right recall (ability to find) for the content that contains this image. if an image has a well-formed caption, that captures the context accurately -it also makes the document accessible. making enterprise content accessible is an important metric for large organizations as they strive to be inclusive. the scale and the quick turn-around time demanded for the content creation cycle results in the lack of correct tagging and captions of images (multimodal assets), in turn an eventual lost of revenue and a target customer base. we propose a method towards automated context-aware captioning of images targeted to reduce the tedium and this critical gap in the enterprise authoring and content creation process. large-scale pretraining of language models @xcite ) has witnessed great success in many downstream nlp tasks. this success has inspired multi modal pretraining for image-text, image-only, and video-text tasks. currently, building unified models that jointly learn multiple vision-language tasks is gaining a lot of attention and has shown promising results on many vl tasks @xcite @xcite . the existing unified vision-language models focus on tasks like image captioning @xcite , visual question answering @xcite , visual entailment @xcite , and image-text retrieval @xcite that consider the image as a standalone entity. however, images are typically accompanied by text that adds additional meanings which are not utilized in these tasks as shown in figure 1 . also, the same image can mean different things in different contexts. for example, a picture of a football player being emotional can mean they are celebrating a goal or are disappointed with their shot, depending on the context. hence it is essential to consider the context of the image for understanding it completely. traditional image captioning models do not use contextual information. in news image captioning @xcite , the generated caption contains information extracted from both the news article and the image. the news image captioning task is a special subtask of context assisted image captioning task that uses the news article as the contextual information about the image. in our work, the task names-news image captioning and context assisted image captioning are hence used interchangeably. existing pretrained vl models lack the ability to use contextual information as the pretraining tasks do not contain long text associated with image-text pairs. we propose a new unified vl model based on the one for all (ofa) model, with a focus on using the contextual information associated with the image for real-world problems like news image captioning. as there are no existing vl classification task that uses contextual information, we introduce a new vl task called 'contextual visual entailment'. visual entailment @xcite is a refined image-text matching task that checks for the entailment of the caption with the premise image. visual entailment deals with only the descriptive characteristics of the image. in our contextual visual entailment task, both the image and the context of the image are treated as the premise, and the en-tailment of the caption is predicted with respect to both. our contributions are: news image captioning deals with the generation of captions for news images. the news articles contain the context of the image and they are taken into account during the caption generation process. biten et al. (2019) propose an encoder-decoder model with attention over both image and news article encodings to generate news image captions. it generates captions with placeholders for named entities and fills those placeholders by choosing named entities from the news article. chen and zhuge (2019) model it as a query-based summarization problem where the news image acts as the query and the news article is the source text to be summarized. tran et al. (2020) use transformers with separate encoders for extracting image features, object features and faces present in the image. the decoder receives the input from all three encoders to generate caption. liu et al. (2021) use a visual selective layer that learns to align the image features with the text in the news article to generate captions. yang et al. (2021a) discuss the journalistic guidelines followed while writing news image captions in journals and incorporate them in the generation process. zhang et al. (2022) use prompt tuning to finetune pretrained models for news image captioning. unified vision-language (vl) modeling is a new paradigm that involves creating a unified framework for multiple vision-language tasks, allowing models to be trained on a range of datasets constructed for a range of tasks. vilbert @xcite extends the bert @xcite architecture to work with visual inputs. @xcite propose a multi-task training approach with 12 vl datasets on 4 broad tasks. vl-t5 @xcite combines multiple vl tasks as text generation tasks using pretrained models for image features. unit @xcite unifies cross-modal tasks by using a modality specific encoder and a shared decoder. ufo @xcite proposes to use the same transformer architecture as the encoder for both image and text in vl tasks. unitab @xcite supports vl tasks with bounding boxes by encoding the text and box output sequences to shared token sequences. ofa @xcite abstracts all vl tasks into sequence-to-sequence problems. existing unified vl models do not consider the context of the image in their pretraining tasks. our unified model is pretrained with tasks that include contextual information and hence it achieves state-of-the-art results on news image captioning datasets. ## dataset we pretrain our model on a large pretraining dataset and evaluate its performance on benchmark datasets for news image captioning. ## our model unified vision-language (vl) modeling has shown great promise in multiple vl tasks. hence, we use a unified model for all three tasks-context assisted image captioning, contextual visual entailment, and keyword extraction. figure 2 shows an overview of our unified vl pretraining strategy. we use the ofa large @xcite architecture. ofa is a task and modality agnostic model that unifies all vision-language, vision-only, and language-only tasks using a sequence-to-sequence learning framework. we use @xcite and vqgan @xcite to obtain visual tokens for the given image. the text (context and caption) is tokenized by byte-pair encoding (bpe). a single unified vocabulary is used for both visual and linguistic tokens. transformers are used as encoders and decoders and all vision-language tasks are abstracted to seq-to-seq conversion tasks with specific instructions created for each task, similar to the ofa pretraining. the pretraining of our model involves three tasks-news image captioning, contextual visual entailment, and keyword generation. for news image captioning, we convert the image, caption, and context into a sequence of input tokens and generate the caption as a sequence of tokens conditioned on these input tokens. for keyword generation, the news article is tokenized as the input sequence and the keywords are generated by the model as the output sequence. contextual visual entailment is a classification task, so the input sequence to the model is the image, caption, and context tokens and the model is trained to generate 'yes' or 'no' as the output indicating if the caption entails the image and context or not respectively. the model is trained to reduce the cross entropy loss. for the input sequence x consisting of visual and text tokens and output y, the loss is given as: ## experiments the experimental details for pretraining and finetuning for context assisted image captioning are discussed here. ## results and analysis our model achieves state-of-the-art results on both goodnews and nytimes800k datasets. the ofa model finetuned on benchmark datasets also shows good performance. this shows the ability of ofa to adapt to new tasks and the correctness of our instructions for finetuning. however, it can be seen that due to the lack context information in the pretraining tasks used in ofa, the model doesn't produce substantially better results compared to the current sota models. our model pretrained on the three tasks shows a 5.14 cider score improvement over the ofa model on the goodnews dataset which is an 8.34 cider score improvement over the current sota model. the model also achieves a sota result of 66.41 cider score on the nytimes800k dataset. the average length of news articles in good-news and nytimes800k dataset are 451 and 974 words respectively. the larger article length in nytimes800k dataset is the reason for the cider scores being closer to the current sota as the context length in our experiments is restricted to 512 tokens. we also obtain comparable performance in precision and recall of named entity generation despite not feeding the named entities directly to the model like in the previous works. the blip-2 model has shown great promise in zero-shot image caption generation. we use blip-2 to generate descriptive captions and feed those captions as input to the gpt-3 model along with the context to generate the final context assisted caption. the blip-2 + gpt-3 model generates fluent captions but it does not contain the relevant information based on the image features as indicated by the poor performance on evaluation metrics in our pretraining tasks indirectly direct the model to capture named entity information from the context. however, earlier works on news image captioning show that extracting named entities and feeding them directly to the model can help it generate better captions with correct named entities. our pretrained model showed a slight decrease in performance when named entity information is presented to it in the prompt. this is because the named entity tokens take up valuable space in the 512 tokens allowed for the context, leading to information loss gained from the context. also, since only 64 tokens are allowed for named entities, not all the important entities in the news article are presented in the prompt and it disadvantages the model in both ways. we also pretrained the ofa model with a subset of the three pretraining tasks to identify the importance of each task in pretraining. the models pretrained with a combination of two tasks and with only the captioning task performed poorly compared to the model pretrained on the three tasks. this shows the importance of training with all three tasks. between the two two-task pretrained models, the model that used contextual visual entail task performed better, indicating the usefulness of the task we introduced. ## summary and conclusion in this work, we proposed a new unified vl model that uses contextual information of images that has not been utilized in pretraining before. we introduce a new vl classification task called contextual visual entailment and pretrain a model with three subtasks that uses long text along with image and caption. our model achieves new state-ofthe-results on benchmark datasets for news image captioning and highlights the importance of using contextual information in pretraining. in the future, we aim to deploy our model to allow context-aware caption generation which could be used in enterprise authoring and many other content creation processes. train val test pretraining keyword extraction 259902 10000 10000 contextual visual entailment 1005925 55884 55884 news image captioning 1117697 40000 40000 benchmark goodnews 424000 18000 23000 nytimes800k 763000 8000 22000