Towards Multimodal Vision-Language Models Generating Non-Generic Text (2021)

Abstract: Vision-language models can assess visual context in an image and generate descriptive text. While the generated text may be accurate and syntactically correct, it is often overly general. To address this, recent work has used optical character recognition to supplement visual information with text extracted from an image. In this work, we contend that vision-language models can benefit from information that can be extracted from an image, but are not used by current models. We modify previous multimodal frameworks to accept relevant information from any number of auxiliary classifiers. In particular, we focus on person names as an additional set of tokens and create a novel image-caption dataset to facilitate captioning with person names. The dataset, Politicians and Athletes in Captions (PAC), consists of captioned images of well-known people in context. By fine-tuning pretrained models with this dataset, we demonstrate a model that can naturally integrate facial recognition tokens into generated text by training on limited data. For the PAC dataset, we provide a discussion on collection and baseline benchmark scores.

## introduction vision-language models combine deep learning techniques from computer vision and natural language processing to assimilate visual and textual understanding. such models demonstrate visual and linguistic knowledge by performing tasks such as vision question answering (vqa) and image captioning. there are many applications of these tasks, including aiding the visually impaired by providing scene information and screen reading @xcite . to perform a vision-language task, a model needs to understand visual context and natural language, and operate in a shared embedding space between the two. approaches in the literature have improved performance by pre-training models for both visual context and language understanding @xcite @xcite @xcite . these models have yielded accurate and semantically appropriate vqas or captions. however, the text generated from these models are general and overlook content that allow for richer text generation with improved contextualization. for example, they ignore clearly visible text or the presence of well-known individuals. to improve specificity in generated text, recent work has used optical character recognition @xcite to incorporate text that appears in images @xcite @xcite @xcite @xcite @xcite . in many cases, this significantly enhances the usefulness of the generated text @xcite . such frameworks include ocr as an additional input modality. this results in three modalities for vqa (image, question, and ocr) and two modalities for image captioning (image and ocr). while using ocr allows enhancement of some generated text, specific information that exists in human-level description may also come from additional sources. without proper nouns or other specific vocabulary, the generated text is at the risk of being awkwardly general, demonstrating a lack of shared knowledge that is expected in society. for example in figure 1 , arguably the most relevant content in the image is the presence of a wellknown political figure. consequently, a reasonable description of the image should include the name of the well-known figure, which is 'bernie sanders' is in this case, instead of generic "a man". this is notably absent in the caption from the previous model. in this work, we propose the special token approach, a novel method for integrating tokens from several upstream vision classifiers into image captions. 2 we generalize the ocr input modality to accept additional helpful outputs from any number of auxiliary classifiers (section 3.2). we use a rich feature representation for upstream tokens that allows the captioning model to learn to differentiate tokens from different classifiers (section 3.3). this method potentially allows a model to leverage easily available sophisticated libraries to recognize faces, scene-text, cityscapes, animal species, etc. we refer to all tokens from upstream sources, including ocr tokens, as special tokens. in this work, we focus on using person names and scenetext as example special tokens. to facilitate using person names in image captions, we create a novel image-caption dataset, politicians and athletes in captions (pac), which includes person names in captions in addition to relevant scene-text found on signs, labels, or other entities in the image. pac has 1,572 images and three captions per image. a discussion on the dataset is provided in section 4. by training on pac in addition to other imagecaption datasets, we create a model that can naturally integrate person names into captions. the same model still performs well on previous image captioning benchmarks. evaluation of the methods is available in section 5. in summary, this paper makes three primary contributions. the special tokens framework is proposed as a method to incorporate tokens from several external sources into generated text. the pac image-captioning dataset is collected and baseline results are presented. lastly, this paper demonstrates the first model in the literature that integrates both facial recognition and ocr into image captioning. ## related work the ubiquitous encoder-decoder architecture divides the image captioning task into two parts. the encoder acts as feature extractor and the decoder handles word generation. early deep learning models for image captioning used cnn encoders for feature extraction from the input image as a whole @xcite @xcite . current models rely on attention @xcite to generate high-quality image captions. the seminal image captioning model, show, attend and tell @xcite , applied attention mechanism on input visual features and the previously generated word (during inference) at each time step for textual caption word generation. the majority of current state-of-the-art methods for image captioning and visual question answering benefit from the bottom-up and top-down attention mechanism @xcite . bottom-up attention, a hard attention mechanism, leverages an object detector, faster r-cnn @xcite to detect the most important regions in the image. top-down attention, a soft attention mechanism, performs modulation over the set of input visual features from object detection regions. following the adoption of bottom-up attention for ocr features @xcite , we use the same mechanism to learn to include features obtained from facial recognition. rather than faster r-cnn, we use rfbnet @xcite for facial region detection. for facial feature extraction, we use arcface @xcite pre-trained on megaface dataset @xcite . several techniques have been proposed to handle ocr tokens in vision-language tasks. the m4c algorithm uses an indiscriminate attention layer followed by a dynamic pointer network @xcite . the ss-baseline model uses individual attention blocks for each input modality followed by a single fusion encoding layer @xcite . several approaches have been proposed to better handle spatial information about ocr tokens @xcite @xcite @xcite . the mmr method utilizes spacial information about objects and scene-text via a graph structure @xcite . textocr was introduced as an end-to-end method for identifying ocr tokens @xcite . tap was introduced as a method to integrate ocr tokens into pre-training. more similar to our work, zhao et al. use an upstream classifier as input to a captioning model. they introduce a multi-gated decoder for handling input from external classifiers @xcite . in contrast, we use general ocr and facial recognition classifiers rather than a web entity recognizer as an upstream classifier. our approach is different from zhao et al. in that we use bottom-up and top-down attention rather than a standalone cnn for object detection, use a common embedding space rather than a gated decoder for handling multi-modal inputs, and use rich representations (see section 3.3) rather than only textual information for handling tokens from upstream classifiers. ms-coco @xcite is a large dataset for common objects in context used for image captioning. similar to ms-coco, flickr30k @xcite is another common dataset used for image captioning. google's conceptual captions @xcite is a vast dataset used for pretraining multitasking vision-language models and fine-tuning them on other vision-language down stream tasks @xcite . the captions in these datasets are generic. to facilitate use of optical character recognition in the vision-language domain, several datasets have been released, including st-vqa @xcite for scene text visual question answering and textcaps @xcite for image captioning with reading comprehension. along with the introduction of textcaps dataset, the m4c model @xcite originally used for visual question answering was adopted for image captioning. we modify the m4c model so that it includes bottomup facial recognition features. ## special tokens we use the term special token as a placeholder for extracted relevant information that is identified in an image by upstream sources. tokens from upstream classifiers are special in that they often are named entities, offering unique descriptors for generic objects. for example in figure 1 , 'bernie sanders' is not a new object, but rather a special descriptor for an already recognized generic object (i.e. man). likewise, 'this week' is not a generic temporal entity. instead, it can be used to give more detail about a generic object: a screen that says 'this week', referring to a tv show or event called 'this week'. we call our corresponding method for integrating special tokens into image captions the special token approach. in our approach, there are two modalities that hold information about an image. the first modality corresponds to generic visual features (yellow box in figure 2 ) which are responsible for informing the model of general context (all vision-language models have a visual modality). the second modality, special tokens (red box in figure 2 ), is responsible for informing the model of specific terms that are relevant to the image. the embeddings for the first modality are calculated from visual features from an object detector. the embeddings from the special token modality are calculated from visual feature vectors (faster-rcnn and a bounding box), textual features (fasttext @xcite and pyramidal histogram of characters (phoc) @xcite ), and a source feature (one-hot encoding) as shown in figure 3 . additionally, special tokens are made available for direct copy into generated text which allows for zero-shot inclusion of words not seen prior. this structure has been successful on ocr vision-language datasets. the key hypothesis of this paper is that a model can learn to differentiate tokens from separate upstream classifiers. subsequently, the model can learn to use each token type appropriately in generated text. for example, a caption for the image in figure 1 should neither say "a screen that says bernie sanders" nor should it say " 'this week' standing in front of a screen." as mentioned in section 1, this work demonstrates using two types of special tokens, ocr tokens and facial recognition tokens. we focus our experimentation on learning to integrate facial recognition tokens by training on the pac dataset. however, any set of words that can be identified by some classification or recognition module can conceivably be a set of special tokens. we leave integration of more upstream vision classifiers for future work. ## pac dataset with this paper we create the politicians and athletes in captions (pac) dataset. pac is imagecaption dataset consisting of images of well-known individuals in context. pac includes 1,572 images and three captions per image. samples from pac can be seen in figure 4 and additional samples can be found in the supplementary materials. we create pac with the goal of studying the use of non-generic vocabulary in image captioning. the non-generic terms emphasized in pac are person names and ocr tokens. the pac dataset offers several technical challenges: 1) correctly identifying people in a variety of settings, 2) reasoning about the effect of the presence of the individual. if a known person is in a scene, the description of the scene is often based on the known person, and 3) natural integration of a name into a generated caption. ## experiments in our experiments, we test the special token approach by training on pac and textcaps. we present baseline results on pac. additionally, we present a visualization for the special token embedding space.