Image Caption Generation Framework for A ssamese News using Attention Mechanism (2021)

Abstract: Automatic caption generation is an artificial intelligence problem that falls at the intersection of computer vision and natural language processing. Although significant works have been reported in image captioning, the contribution is limited to English and few major languages with sufficient resources. But, no work on image captioning has been reported in a resource-constrained language like Assamese. With this inspiration, we propose an encoder-decoder based framework for image caption generation in the Assamese news domain. The VGG-16 pre-trained model at the encoder side and LSTM with an attention mechanism are employed at the decoder side to generate the Assamese caption. We train the proposed model on the dataset built in-house consisting of 10,000 images with a single caption for each image. We describe our experimental methodology, quantitative and qualitative results which validate the effectiveness of our model for caption generation. The proposed model shows a BLEU score of 12.1 outperforming the baseline model.

## introduction image caption generation is a new and exciting topic in artificial intelligence that has sparked much interest and has been studied extensively in recent years. to interpret the visual contents, computer vision and natural language processing are necessary. as a result, both semantic and linguistic information about the image is required. expressing the semantic content like human and grammatically correct is a challenging task. caption generation has a wide range of potential real-world applications. it is helpful for visually impaired people to understand the content of the image. it can also be employed in self-driving automobiles and image search engines. this puts a halt to a slew of important real-world applications, prompting researchers to develop a better model for generating captions in the same way that humans do. journalists can use news image captioning to describe the contents of the news as well as multimedia analytics. image captioning requires recognizing the important objects, attributes, and relationships in an image to generate syntactic and semantically correct description. earlier caption generation works are based on template and information retrieval. the template-based approach generates the captions by extracting the actions, objects and other attributes in an image and filling them into a pre-defined template. in comparison, the information retrievalbased approach needs a large image database that extracts the visually similar image and generates an image caption by using the caption of the retrieved image. nowadays, most models are based on deep learning architecture @xcite . the study found that the caption generated using the deep learning approach is more expressive and fluent than the traditional caption generation approaches. several significant image caption generation works are proposed in english using deep learning approach. according to our study, image caption generation in assamese language is still at infancy stage. the assamese language belongs to the indo-european language family and it is spoken mainly in the state of assam in india by approximately 15 million people. in this paper, we propose a caption generation model based on encoder-decoder architecture on news images. at the encoder side, a vgg-16 pre-trained model is used to represent the visual features of an image and to generate the assamese caption, an lstm layer with an attention mechanism is employed at the decoder side. a large set of images and good-quality captions are required for a caption generation system. existing english datasets are flickr8k @xcite , ms-coco @xcite , lifelog @xcite , visual genome @xcite , multi30k @xcite etc. however, there is currently no comparable annotated dataset accessible in assamese. dataset scarcity is a major challenge, particularly for a morphologically rich @xcite language like assamese. therefore, we present an annotated dataset for caption generation in the news domain and forward it for future research. first, we collect the news articles consisting of both images and text from the local assamese newspapers. we pre-process the data as an initial step after the collection. after that, each news image is manually annotated with one description. furthermore, we evaluate the model using both quantitative and qualitative parameters. the proposed system is one of the earliest reported assamese news image caption generation model and the experiment results of the developed model are promising. the objectives of this paper are: fang et al. (2015) suggested a caption generation system on the ms-coco dataset, which received a bleu-4 score of 29.1. a visual recurrent representation model was suggested by chen and lawrence zitnick (2015) for image caption generation on the ms-coco, flickr 8k and 30k and pascal 1k datasets. to describe and visualize the image caption, a bi-directional mapping between images and sentence-based descriptions was carried out. karpathy and fei-fei (2015) also described the image region using a multimodal recurrent neural network on flickr8k, flickr30k and ms-coco datasets. a chinese image caption generation model was introduced by peng and li (2016) on flickr30k and ms coco dataset. they demonstrated that a character-level strategy is more effective than a word-level one. soh (2016) reported a top-down caption generation strategy employing cnn-lstm architecture on the ms coco dataset with a 3.3 bleu score. miyazaki and shimizu (2016) proposed a deep recurrent network based image caption generation model on the cross-lingual domain. the yj captions 26k dataset, a japanese version of the ms-coco dataset, was built for this purpose. amritkar and jabade (2018) reported an image caption generation with cnn and rnn architecture on flickr8k and ms coco datasets. an attention-based remote sensing image captioning system was reported by @xcite on their own built remote sensing caption generation dataset. they employed both hard and soft attention mechanisms to train the model. they found that the hard attention mechanism performed better than the soft attention mechanism. @xcite used attention-based architecture to report a hindi caption generation approach. they manually translated the ms coco dataset into hindi for the dataset. @xcite developed a semantic image attention model to concentrate on the linguistically significant image object. @xcite proposed an encoder-decoderbased news image caption generating architecture on the bbc news data. the model takes an image from the news related to news documents as input and outputs an appropriate image caption. rahman et al. (2019) introduced chittron, a bangla image captioning model. a total of 16,000 images was collected and has been manually annotated by two native bangla speakers. next, a vgg-16 image embedding model integrated with a stack lstm layer is used to train the model. the proposed model has gained a bleu score of 2.5. again kamal et al. (2020) used deep learning techniques to create an automated bangla caption generation system called textmage on the banglalekhaimagecaptions dataset. the textmage model could understand the visual scenes that belong to the bangladeshi geographical context. the proposed model is trained on the banglalekhaimagecaptions dataset consisting of 9,154 images along with two descriptions for each image. the use of visual genome image captioning in a multimodal machine translation challenge was reported by meetei et al. (2019b). the generation and evaluation of hindi image caption on the visual genome dataset were carried out by singh et al. (2021a). additionally, attention based image and video caption generation framework were carried out by singh et al. (2021c), singh et al. ## an assamese multimodal dataset several benchmark datasets for caption generation are flickr8k, flickr30k and mscoco, available in english, but none are accessible for resourceconstrained languages, including assamese. so, we built an assamese multimodal dataset from the news domain. we carried out the following dataset preparation steps: ## system architecture the first stage of an image caption generation model is image feature extraction, and the second part is image description generation. a convolutional neural network is used to extract the image features at the encoder side and lstm layers are used to train the language model for image description at the decoder side. this paper describes the development of an encoder-decoder based image caption generation framework using cnn-lstm architecture with attention mechanism in the assamese language news domain. the proposed model consists of three phases: before feeding the text input to the neural network, it is important to pre-process the text data and transform it into a numerical form. for input text representation, a word embedding layer is used. it provides a dense representation of the input text and then passes it to the next lstm layer. আ ঃৰা য় সৗ য িতেযািগতাত ড বাইৰ ৰাণী িখতােপেৰ জিলিকল অসম স ান সাগিৰকা (in international beauty contest, assam girl sagarika wins the glory of queen of dubai.) caption image কা জৰঙাত পিহলা নেৱ ৰৰ পৰা পয টকৰ দশ নৰ সু িবধােথ এিলেফ ছাফাৰী মু কিল কিৰব (the elephant safari will be opened in kaziranga from november 1st for the convenience of tourists) (a) (b) figure 2: an example of assamese caption dataset ## experimental results and discussion the quantitative and qualitative analysis of the proposed system is covered in this section. ## conclusion and future work in this paper, we report a cnn-lstm based framework with an attention mechanism for assamese caption generation on the multimodal news dataset. the attention mechanism decides where to pay attention in order to generate a meaningful caption. we also report the findings of the investigation of caption generation on the assamese language on low resource setting. to assess model performance, we used both qualitative and quantitative approaches. it is observed that the proposed framework outperforms the baseline model. in future, various architectures such as resnet with mbert or indic bert may be explored for any significant improvement of the system results further. we intend to expand the dataset in the future with a more diverse and wide collection of images of various domain-specific events, each with several appropriate descriptions, in order to build a human-like caption.