A Scaled Encoder Decoder Network for Image Captioning in H indi (2021)

Abstract: Image captioning is a prominent research area in computer vision and natural language processing, which automatically generates natural language descriptions for images. Most of the existing works have focused on developing models for image captioning in the English language. The current paper introduces a novel deep learning architecture based on encoder-decoder with an attention mechanism for image captioning in the Hindi language. For encoder, decoder, and attention, several deep learning-based architectures have been explored. Hindi, the fourth-most spoken language globally, is widely spoken in India and South Asia and is one of India’s official languages. The proposed encoder-decoder architecture utilizes scaling in convolution neural networks to achieve better accuracy than state-of-the-art image captioning methods in Hindi. The proposed method’s performance is compared with state-of-the-art methods in terms of BLEU scores and manual evaluation (in terms of adequacy and fluency). The obtained results demonstrate the efficacy of the proposed method.

## introduction caption generation from images is a complex job as it necessitates object recognition and articulating the object's relationship in natural language. caption generation is challenging in comparison to object recognition and image classification, which have been the primary research focus in computer vision. nowadays, deep learning-based architecture has emerged as a result of recent developments in machine translation. recent advances in language modeling, object recognition, and image classification opened up new possibilities. a generated image caption can assist visually challenged individuals to perceive the web content @xcite . the architecture based on encoderdecoder has been widely employed to solve the image captioning problem @xcite @xcite . in the literature, two different approaches have been used for caption generation ; the top-down approach @xcite ) @xcite , @xcite , and the bottom-up approach @xcite @xcite . in this paper, we built a model of caption generation from images in the hindi language, which is spoken throughout india, south asia, and other parts of the world as well. it is one of the world's ancient languages and the third most spoken language globally. it originated from the sanskrit language @xcite . in the literature, there are just a few works on hindi image captioning @xcite @xcite . the first work was carried out in @xcite , resnet 101 @xcite , and gru @xcite is employed in the architecture. in @xcite , authors had employed various attention models. in this paper, the authors had explored several architectures with various attention. authors of @xcite proposed a architecture using transformer. the transformer is employed here as a decoder. this work also utilizes deep-learning based architectures for generating captions of images in the hindi language. the key contributions of this work are as follows: in the past, two approaches for image captioning have been used; the first is the top-down approach @xcite ) @xcite , and second approach is an older approach i.e. bottom up approach @xcite ) @xcite ) @xcite . in the first approach, the input image is turned into words, but in the second approach, words define the many features of an image; words are joined to form an image caption. the architecture's parameters are learned in top-down methodology, which comprises end-to-end learning for caption generation. the language model combines the object characteristics, vocabulary, visual description, and sentences, etc., in the bottom-up approach. association of the appropriate sentence to an input image is explored in @xcite ; this sentence is considered as the input image's caption. in (elliott and keller, 2013), a template-based method was utilized, it uses visual dependency modeling to record the links among objects. for image captioning nowadays, the top-down method is very popular. authors of @xcite had developed the captioning architecture using the multimodal rnn to generate the caption. a probability distribution model is employed to generate the word based on prior words and an image. the probability distribution is used to generate the image caption. it is analogous to the approach of machine translation employing encoder-decoder architecture. in @xcite , authors have used a generative model using an rnn trained to optimize the likelihood of the target sentence given an input image. authors of @xcite have proposed an image captioning model by using a combination of cnn and rnn over image region utilizing the alignment model. they used bidirectional rnn for language modeling and a structured, objective function aligning two modalities through a multimodal embedding. a language pre-training model unified version is investigated in @xcite . a meshed memory transformer @xcite is utilized for image's feature extraction and language modeling; it learns a multi-level relationship between previous information and regions of the image. authors of @xcite have proposed an image captioning model using generative adversarial networks using retrieval and ensemble based approaches. the method given by @xcite has an image captioning structure using variational generative adversarial network and variational autoencoder; the approach generates an image caption based on an image summary. the authors also used part-of-speech as a description that assists in generating the description of the image. most of the relevant works for image captioning in the literature are published particularly for the english. only a limited number of attempts have been made for image caption generation in the hindi language. in @xcite , the first attempt for image captioning in the hindi language is made. a transformer-based image captioning model has been proposed in @xcite . in @xcite , authors have investigated a variety of architectures with various attention methods for caption generation from images. ## proposed methodology we employed the encoder-decoder framework with attention for image captioning in the proposed framework (as shown in fig 1 ). ## experimental setup this section covers the methods employed to create the dataset and evaluate the proposed methodology. ## results and discussions a comprehensive overview of obtained results and generated captions are discussed in this section. ## conclusion and future work we present a novel approach for caption generation from images in hindi that employs an encoderdecoder model based on efficientnet and gru, as well as attention techniques. we use effcient-net as an encoder because its efficacy outperforms state-of-the-art cnns for image classification and feature extraction. we use a gated recurrent unit as a decoder for language modeling as it is less computationally expensive and it achieves state-ofthe-art efficacy for language modeling. further, the use of bahdanau attention makes the system robust. aside from that, we undertake an ablation analysis to find the ideal architecture. the proposed methodology could be expanded for image-to-paragraph generation and dense image captioning.