CNN -Transformer based Encoder-Decoder Model for N epali Image Captioning (2022)

Abstract: Many image captioning tasks have been carried out in recent years, the majority of the work being for the English language. A few research works have also been carried out for Hindi and Bengali languages in the domain. Unfortunately, not much research emphasis seems to be given to the Nepali language in this direction. Furthermore, the datasets are also not publicly available in the Nepali language. The aim of this research is to prepare a dataset with Nepali captions and develop a deep learning model based on the Convolutional Neural Network (CNN) and Transformer combined model to automatically generate image captions in the Nepali language. The dataset for this work is prepared by applying different data preprocessing techniques on the Flickr8k dataset. The preprocessed data is then passed to the CNN-Transformer model to generate image captions. ResNet-101 and EfficientNetB0 are the two pre-trained CNN models employed for this work. We have achieved some promising results which can be further improved in the future.

## introduction image caption generation is one of the computer vision and natural language processing (nlp) tasks which generates the description of an image. it can be used to aid visually impaired people for describing scenarios, for image indexing, creating image-based search engines, annotating news images, and many more. automatically captioning an image is one of the main goals of scene understanding in computer vision. the caption generation model should handle the challenges of identifying the objects in an image as well as capturing and expressing their description in the natural language @xcite . it is a complex process compared to the existing object detection and classification tasks. in this regard, computer vision techniques are used to understand the contents and extract features from the image whereas the natural language processing techniques generate words or descriptions from the extracted features in the right order @xcite . recent advancements in nlp and computer vision have paved the way to perform various tasks using the native language. image captioning for the nepali language is one of the least researched topics and very limited literature are available owing to the language complexity and unavailability of datasets. the only work carried out in this domain was proposed by @xcite which employs a traditional cnn-rnn encoderdecoder architecture. hindi, marathi, and konkani are some of the other languages that share similar grammatical structures as the nepali language as all of these languages use the devanagari script. additionally, bengali language also belongs to the indic language groups along with other languages mentioned above. hence, this work is an attempt to develop image captioning for the nepali language with reference to the existing works. some of the existing works in this field include hindi image captioning techniques shown by @xcite and bengali image captioning proposed by @xcite . in this research, we propose a cnn-transformerbased nepali image captioning model. the reason behind opting the transformer network is its wide applicability in today's natural language processing domain as well as its performance in image captioning tasks for other languages. it is computationally faster than rnn as it supports parallel processing. furthermore, transformer networks have not been implemented for nepali image captioning so far and hence, this work to the best of our knowledge is the first to implement it. the main focus of this study is to create nepali datasets and develop a cnn-transformer model to generate nepali captions. the remainder of this paper is arranged in the following order: section 2 discusses some of the related works, section 3 shows the methodology used, and section 4 consists of results and discussions of the work followed by a conclusion and future directions. ## related works various works have been carried out in the image captioning domain, especially in resourceful languages such as english. a significant number of research works can also be seen in hindi and bengali which are closely related to the nepali language. @xcite proposed the only work in nepali that utilizes the two encoderdecoder models with and without visual attention inspired by @xcite . the models use resnet-50 as encoder and lstm/gru as decoder respectively trained on ms coco datasets after translating and preprocessing. mishra et al. (2021) proposed a transformed-based encoder-decoder architecture for image captioning for the hindi language where resnet-101 is used as an encoder and transformer as a decoder. they have outlined the problems with rnn architecture and proposed a transformer model with a stacked attention mechanism to decode the image feature vectors into sentences. the authors have calculated bleu1, bleu2, bleu3, and bleu4 scores with values of 62.9, 43.3, 29.1, and 19.0 respectively. similarly, palash et al. (2021), shah et al. (2021), and ami et al. (2020) have proposed image captioning in the bengali language using cnn and transformer networks. palash et al. (2021) have used resnet-101 for extracting image features whereas shah et al. (2021) and ami et al. (2020) opted for two pretrained cnn models: inceptionv3 and xception. palash et al. (2021) used banglalekha datasets for training and testing the model. similarly, ami et al. (2020) have used flickr8k datasets after preprocessing and the work is extended by shah et al. (2021) using banglalekha datasets. bleu1, bleu2, bleu3, bleu4, and meteor scores obtained in palash et al. ( 2021 ) work are 0.694, 0.580, 0.505, 2.22e-308, and 0.337 respectively which is better compared to @xcite . moreover, @xcite proposed a new model for remote sensing image captioning tasks for the english language. the transformer-based decoder was used to generate captions from the image features. the semantic and spatial features were extracted from the image using cnn. to capture a deeper relationship between image features and text descriptions, the semantic features were added to the decoder's every single sub-layer. the datasets used for this research were obtained from the sydney dataset, remote sensing image caption dataset (rsicd), and ucm dataset. the above literature suggests that different works have been carried out in the image captioning domain in several languages and most of them have concluded that transformer networks perform better than traditional cnn-rnn architecture. in this regard, the nepali image captioning system using transformers is yet to be explored and the nepali caption datasets are also not publicly available. based on these facts and literature, a cnn-transformer model is attempted to implement in this research. ## methodology the methodology employed for this research involves experimentation of data on the model architecture. the datasets for nepali image captions are not available publicly, hence two procedures are followed for developing the nepali image captioning system. the first step involves dataset preparation, followed by model architecture design and implementation. ## results and discussions the outcome of this work comprises the nepali image captions dataset and experimental results of the model performance. a dataset of over parameters value image size (299,299) max. vocab size 15000 sequence length 20 embedding size 512 batch size 128 optimizer adam loss function categorical-crossentropy ## limitations the proposed models have not acquired sufficient accuracy and are not able to generate the desired captions for all input images. moreover, the datasets are also limited and only two pre-trained cnn models are considered for this work. similarly, the hyperparameters may not be optimally tuned due to limited experimentations. such limitations can affect the model's efficiency. we consider addressing them in future. ## conclusion and future works in this research work, a cnn-transformer-based nepali image captioning system is implemented. at first, the flickr8k datasets are translated and preprocessed to create a nepali captions dataset. the datasets are then fed to two models: model a and model b with the same model parameters. the outcome of this experiment shows promising results. model b performed slightly better than model a on 40,455 nepali image-caption pairs. moreover, these models are able to generate captions from the given input image. on the other hand, this work has some limitations as well which can be addressed in the future. the experimentation on larger datasets and fine-tuning of the hyperparameters can be performed in future which are expected lead to better results. ms coco and flickr30k datasets can be used after preprocessing for this purpose. similarly, other cnn models such as inceptionv3, xception, efficientnetb7 etc. can be explored for image captioning tasks. furthermore, this research work can be extended to video captioning for the nepali language as well.