Multi-Modal Image Captioning for the Visually Impaired (2021)

Abstract: One of the ways blind people understand their surroundings is by clicking images and relying on descriptions generated by image-captioning systems. Current work on captioning images for the visually impaired do not use the textual data present in the image when generating captions. This problem is critical as many visual scenes contain text, and 21% of the questions asked by blind people about the images they click pertain to the text present in them. In this work, we propose altering AoANet, a state-of-the-art image-captioning system, to leverage text detected in the image as an input feature. In addition, we use a pointer-generator network to copy detected text to the caption when tokens need to be reproduced accurately. Our model outperforms AoANet on the benchmark dataset VizWiz, giving a 35% and 16.2% performance improvement on CIDEr and SPICE scores, respectively.

## introduction image captioning as a service has helped people with visual impairments to learn about images they take and to make sense of images they encounter in digital environments. applications such as (tap-tapsee, 2012) allow the visually impaired to take photos of their surroundings and upload them to get descriptions of the photos. such applications leverage a human-in-the-loop approach to generate descriptions. in order to bypass the dependency on a human, there is a need to automate the image captioning process. unfortunately, the current state-of-the-art (sota) image captioning models are built using large, publicly available, crowdsourced datasets which have been collected and created in a contrived setting. thus, these models perform poorly on images clicked by blind people * equal contribution largely because the images clicked by blind people differ dramatically from the images present in the datasets. to encourage solving this problem, @xcite released the vizwiz dataset, a dataset comprising of images taken by the blind. current work on captioning images for the blind do not use the text detected in the image when generating captions (figures 1a and 1b show two images from the vizwiz dataset that contain text). the problem is critical as many visual scenes contain text and up to 21% of the questions asked by blind people about the images clicked by them pertain to the text present in them. this makes it more important to improvise the models to focus on objects as well as the text in the images. with the availability of large labelled corpora, image captioning and reading scene text (ocr) have seen a steady increase in performance. however, traditional image captioning models focus only on the visual objects when generating captions and fail to recognize and reason about the text in the scene. this calls for incorporating ocr tokens into the caption generation process. the task is challenging since unlike conventional vocabulary tokens which depend on the text before them and therefore can be inferred, ocr tokens often cannot be predicted from the context and therefore represent independent entities. predicting a token from vocabulary and selecting an ocr token from the scene are two rather different tasks which have to be seamlessly combined to tackle this task. in this work, we build a model to caption images for the blind by leveraging the text detected in the images in addition to visual features. we alter aoanet, a sota image captioning model to consume embeddings of tokens detected in the image using optical character recognition (ocr). in many cases, ocr tokens such as entity names or dates need to be reproduced exactly as they are in the caption. to aid this copying process, we employ a pointer-generator mechanism. our contributions are 1) we build an image captioning model for the blind that specifically leverages text detected in the image. 2) we use a pointer-generator mechanism when generating captions to copy the detected text when needed. automated image captioning has seen a significant amount of recent work. the task is typically handled using an encoder-decoder framework; imagerelated features are fed to the encoder and the decoder generates the caption @xcite @xcite . language modeling based approaches have also been explored for image captioning @xcite . apart from the architecture, image captioning approaches are also diverse in terms of the features used. visual-based image captioning models exploit features generated from images. multimodal image captioning approaches exploit other modes of features in addition to image-based features such as candidate captions and text detected in images @xcite . the task we address deals with captioning images specifically for the blind. this is different from traditional image captioning due to the authenticity of the dataset compared to popular, synthetic ones such as ms-coco @xcite and flickr30k @xcite . the task is relatively less explored. previous works have solved the problem using human-in-the-loop approaches @xcite @xcite as well as automated ones (microsoft; facebook). a particular challenge in this area has been the lack of an authentic dataset of photos taken by the blind. to address the issue, @xcite created vizwiz-captions, a dataset that consists of descriptions of images taken by people who are blind. in addition, they analyzed how the sota image captioning algorithms performed on this dataset. concurrent to our work, @xcite 2020 ) created a multi-modal transformer that consumes resnext based visual features, object detection-based textual features and ocr-based textual features. our work differs from this approach in the following ways: we use aoanet as our captioning model and do not account for rotation invariance during ocr detection. we use bert to generate embeddings of the ocr tokens instead of fasttext. since we use bottom-up image feature vectors extracted using a pre-trained faster-rcnn, we do not use object detection-based textual features. similarly, since the faster-rcnn is initialized with resnet-101 pre-trained for classification, we do not explicitly use classification-based features such as those generated by resnext. we explored copy mechanism in our work to aid copying over ocr tokens from the image to the caption. copy mechanism has been typically employed in textual sequence-to-sequence learning for tasks such as summarization @xcite . it has also been used in image captioning to aid learning novel objects @xcite . also, @xcite introduced an m4c model that recognizes text, relates it to its visual context, and decides what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities such as objects. ## dataset the vizwiz captions dataset @xcite consists of over 39, 000 images originating from people who are blind that are each paired with five captions. the dataset consists of 23, 431 training images, 7, 750 validation images and 8, 000 test images. the average length of a caption in the train set and the validation set was 11. we refer readers to the vizwiz dataset browser @xcite as well as the original paper by @xcite for more details about the dataset. ## approach we employ aoanet as our baseline model. aoanet extends the conventional attention mechanism to account for the relevance of the attention results with respect to the query. an attention mod-ule f att (q, k, v ) operates on queries q, keys k and values v . it measures the similarities between q and k and using the similarity scores to compute a weighted average over v . aoa module then adds another attention by applying the attention gate to the information vector to obtain the attended information î. the aoa module can thus be formulated as: the aoa module is applied to both the encoder and decoder. the model is trained by minimizing the cross-entropy loss: ## experiments in our experiments, we alter aoanet as per the approaches described in section 4 and compare these with the baseline model. aoanet-e refers to aoanet altered as per the approach described in section 4.1. to observe the impact of the number of ocr words added to the extended vocabulary, we train two extended variants: (1) e5: only ocr words with frequency greater than or equal to 5. (2) e2: only ocr words that occur with frequency greater than or equal to 2. aoanet-p refers to aoanet altered as per the approach described in section 4.2. the extended vocabulary consists of ocr words that occur with frequency greater than or equal to 2. we use the code foot_1 released by the authors of aoanet to train the model. we cloned the repository and made changes to extend the feature set and the vocabulary using ocr tokens as well as to incorporate the copy mechanism during decoding foot_2 . we train our models on a google cloud vm instance with 1 tesla k80 gpu. like the original work, we use a faster-rcnn @xcite model pre-trained on imagenet @xcite and visual genome @xcite to extract bottom-up feature vectors of images. the ocr token embeddings are extracted using a pre-trained base, uncased bert model. the aoanet models are trained using the adam optimizer and a learning rate of 2e -5 annealed by 0.8 every 3 epochs as recommended in @xcite . the baseline aoanet is trained for 10 epochs while aoanet-e and aoanet-p are trained for 15 epochs. ## results we show quantitative metrics for each of the models that we experimented with in we compare different models and find that merely extending the vocabulary helps to improve model performance on the dataset. we see that the aoanet-e5 matches the validation scores for aoanet but we see an improvement in the cider score. moreover, we see a massive improvement in validation and test cider scores for aoanet-e2. similarly, we see a gain in the other metrics too. this goes to show that the bert embeddings generated for each ocr token for the images do provide an important context to the task of generating captions. moreover, we see the aoanet-p scores, where we use pointer-generator to copy ocr tokens after extending the vocabulary also perform better than our baseline aoanet model. this goes to show that an ocr copy mechanism is an essential task in generating image captions. intuitively, it makes sense because we would expect to humans to use these words while generating lengthy captions ourselves. we feel that top-k sampling is a worthwhile direction of thought especially when we would like some variety in the captions. beam-search is prone to preferring shorter captions, as the probability values for longer captions accumulates smaller values as discussed by @xcite . ## error analysis although there have been concerns about the robustness of the gcp api towards noise @xcite , we focused our attention on the model's captioning performance and on the pointergenerator mechanism. we agree that the api's performance might hinder the quality of the captions generated but we expected it to not have a large enough impact. we first look at how the extended variants compare with the baseline. we observe that adding text-based features to the feature set imparts useful information to the model. in 2a, aoanet perceives the card as a box of food. addition of text features enables aoanet-e5 to perceive it as a box with black text. while not entirely correct, it is an improvement over the baseline. the alteration also encourages it to be more specific. when the model is unable to find the token that entails specificity, it resorts to producing unk. extending the vocabulary to accommodate more ocr words helps address this problem. in image 2b, baseline aoanet is unable to recognize that the bottle is a supplements bottle. aoanet-e5 attempts to be specific but since 'dietary' and 'supplement' are not present in the extended vocabulary, it outputs unk. aoanet-e2 outputs a much better caption. we see a similar pattern in 2c. we now look at how the pointer variant performs compared to the baseline and the extended variant. incorporating copy mechanism helps the pointer variant in copying over ocr tokens to the caption. aoanet-p is able to copy over 'oats' and 'almonds' in 2d and the token 'rewards' in 2e. but the model is prone to copying tokens multiple times as seen in images 2b and 2f. this is referred to as repetition which is a common problem in sequenceto-sequence models @xcite as well as in pointer generator networks. coverage mechanism @xcite is used to handle this and we wish to explore this in the future. ## conclusion in this work, we propose a pointer-generator based image captioning model that deals specifically with images taken by people with visual disabilities. our alteration of aoanet shows significant improvement on the vizwiz dataset compared to the baseline. as stated in section 7, we would like to explore coverage mechanism in the future. @xcite recently discussed their winning entry to the vizwiz grand challenge. in addition, @xcite introduced a model that has shown to gain significant performance improvement by using ocr tokens. we intend to compare our model with these and improve our work based on the observations made.