Image Caption Generation for Low-Resource A ssamese Language (2022)

Abstract: Image captioning is a prominent Artificial Intelligence (AI) research area that deals with visual recognition and a linguistic description of the image. It is an interdisciplinary field concerning how computers can see and understand digital images

& videos, and describe them in a language known to humans. Constructing a meaningful sentence needs both structural and semantic information of the language. This paper highlights the contribution of image caption generation for the Assamese language. The unavailability of an image caption generation system for the Assamese language is an open problem for AI-NLP researchers, and it’s just an early stage of the research. To achieve our defined objective, we have used the encoder-decoder framework, which combines the Convolutional Neural Networks and the Recurrent Neural Networks. The experiment has been tested on Flickr30k and Coco Captions dataset, which have been originally present in the English language. We have translated these datasets into Assamese language using the state-of-the-art Machine Translation (MT) system for our designed work.

## introduction over 24 million native speakers speak the assamese language in the north-eastern part of india. it is an eastern indo-aryan (indic) language which is the official language of india's assam state. assamese is an indigenous indo-aryan language which has been influenced in vocabulary, phonetics, and structure by the region's close association with tibeto-burman dialects. its grammar is notable for its highly inflected forms, different pronouns, plural noun markers, and honorific and nonhonorific constructions. the assamese script is very close to the bengali script. assamese, like english, is written from left to right. the assamese literary tradition can be traced back to the 13th century. in the 16th century, prose texts, most notably buranjis (historical works), began to appear. the assamese alphabet (assamese, oxomiya bornomala) is shown in fig. 1 which is the bengali-assamese script used in the assamese language. other north-eastern languages that are using the script include bodo (now devanagari), khasi (now roman), mising (now roman), jaintia (now roman), and others. the kamarupi script was used to create it. since fifth century umachal/nagajari-khanikargaon rock inscriptions written in an eastern variant of the gupta script, the script has evolved continuously, with significant influences from the siddha script in the 7th century @xcite . the current format is identical to the bengali alphabet with the exception of two letters, (ro) and (vo); and the letter (khya) has progressed into an independent consonant with its own phonetic quality, however in the bengali alphabet it is a conjunct of two letters. attempting to make computers mimic humans' ability to interpret the visual world is one of the long goal of artificial intelligence researchers. even though significant advancement have been made in numerous computer vision tasks, for example, attribute classification @xcite , object identification @xcite , action classification @xcite , scene recognition @xcite , and image classification @xcite , the field of natural language processing have seen recent huge advances with the addition of transformer architectures. moreover, allowing a computer to automatically describe an image in human language is a comparatively new task. image caption generation is the process of describing the visual information of an image based on the objects and actions depicted in the image using a machine's visual perception and a language model. the study of how computers can apprehend digital images and videos as well as describe them in a language that humans can understand is an interdisciplinary field. recent advancements in the field of natural language generation (nlg) have helped the advancements of a plethora of fields like machine translation, text summarization, answer generation, and image captioning @xcite . the inclusion of several pre-trained transformer-based language models such as bert @xcite have taken the natural language processing (nlp) to new heights. the interpretation of an image is highly dependent on acquired image features. in the prior studies, there are two approaches that have been taken into consideration to accomplish this task @xcite : one that uses a statistical probability language model to generate handcrafted features, and another one uses the neural net-work models based on encoder-decoder language model to extract the deep features. in this paper, we propose an encoderdecoder framework for creating image captions in assamese. the proposed model is based on a separate language model and a visual understanding machine. the rest of this paper has placed out as follows: section 2 presents the works that has common factors with our work. the data used in our experiment has been discussed in 3 section. 4 section contains the procedures we used to prepare our system. the results, advantages and drawbacks have been discussed in 5 section. finally, in section 6, the paper is concluded with a discussion on future work. ## related works most of the image caption generation systems comprised of rudimentary vision-based signifiers and language models which have been used during the early stage of the research. these systems mainly includes rule-based and hand-coded approaches. these systems only worked on a limited set of images. in recent time, the image captioning systems produced significantly improved results, following the same deep learning-based architecture as machine translation, as deep learning meth-ods enhanced. these works was using the same encoder-decoder framework and framed image captioning as a text-to-image translation. cnn was used to encode images and rnn was used to decode the images into sentences in these systems. @xcite used cnn as an encoder to encode images and rnn-lstm to decode image features into text, where image captioning is defined as predicting the probability of a sentence based on the input image feature. the most simple lstm-based captioning architecture is based on a single-layer lstm. during training, input words are taken from the ground-truth sentence, while during inference, input words are those generated at the previous step. @xcite provide both image and text features to the sequential language model at each time step, rather than inputting image features to the system at the start. the encoder-decoder framework's next advanced version is an attention guided framework. xu et al. @xcite proposed the first attention mechanism in image caption generator. the encoder-decoder framework is more focused on the salient region of an image while generating an image description. it is a method that allows you to weight different areas of an image differently. it can, for example, add more weights to an image's important region. the attention model developed by them involved assigning weights to a random portion of an image. as a result, some critical aspect of an image was overlooked in order to generate a caption. to address this limitation, you et al. @xcite developed a semantic attention model that focuses on linguistically significant objects or action in the image. in the preceding attention mechanism, the model forces visual attention to be active for every word, even those that do not explain visual information. stop words such as 'the', 'of', and so on do not explain the image object. to address this issue, lu et al. @xcite developed an adaptive attention mechanism, which automatically determines whether to rely on the visual signal or the language model. whenever the adaptive attention model starts paying attention to a visual signal, it will automatically decide which part of the image to focus on. vaswani et al @xcite described the fully-attentive paradigm which has changed the way people think about language generation completely. the transformer model was fully embraced as the defacto architecture for several language understanding tasks, as well as the groundwork for other advancements in nlp, such as bert and gpt. the transformer architecture has been used for image captioning because it can be viewed as a sequence-to-sequence problem. a masked self-attention operation is applied to words in the standard transformer decoder, followed by a cross-attention operation. words serve as queries, and the final encoder layer's outputs serve as key / value, as well as an ultimate feedforward network. improvement of language generation and visual feature encoding have also been proposed. the north-east of india is one of the country's most linguistically and culturally diverse regions. every state has its own culture, language, and customs. languages serve as a link between people and aid in the formation of bonds. the languages are mostly divided into three groups: indo-aryan, sino-tibetan, and austro-asiatic. assamese, bengali, english, hindi, manipuri, and nepali are the most widely spoken languages in the northeast. as a result, the northeast is also known as india's multilingual and multicultural region. for natural languages, a large number of different nlp applications is being developed in india, as well as across the world. as saiful islam et al. @xcite described , there are only a few nlp applications for ne languages have been developed in india. natural language processing in assamese is being worked on in a number of different ways. assamese is a computationally underdeveloped language, and nlp study is still in its early stages. works have mainly been carried out in the fields of machine translation as we can see in the work of english to assamese using statistical machine translation(smt) @xcite . laskar et al worked on multi-modal translation using both textual and visual features (laskar et al., the 34th conference on computational linguistics and speech processing (rocling 2022) taipei, taiwan, november 21-22, 2022. the association for computational linguistics and chinese language processing 2019). other works have been done in the field of automatic speech recognition by agarwalla et al @xcite and supervised named entity recognition in assamese language @xcite . in this piece of writing, we are suggesting an encoderdecoder framework for writing assamese image captions. the suggested model is based on a unique language model and a machine that can understand visuals. ## dataset a number of datasets are available for highresource languages such as english, hindi, etc., to carry out the experiments in image caption generation. for low-resource languages, the unavailability of sufficient data is the prime challenge faced by the researcher. in the low-resource assamese language, there is no data available for the task of image caption generation. in this proposed work, we have generated the data using the translator cognitive service provided by microsoft azure. herein, we have used the flickr30k and ms coco dataset @xcite which are initially available in english. moreover, we have used the machine translation systems (translator cognitive service provided by microsoft azure) to translate the captions of these datasets into the assamese language. the generated dataset is pictorially depicted in fig. 2 . the designed datasets differ in several ways, including the number of images, the number of captions per image, the format of the captions, and the size of the images. ## framework the architecture of the proposed model is shown in fig. 3 which primarily relies on the encoder-decoder mechanism. in the proposed model, image features are encoded using a convolutional neural network, and the image captions (word sequences) are encoded using a recurrent neural network. later, the encoded image is passed to a text feature decoder which predicts the caption word by word. as it generated each word of the caption, the model used attention to focus on the most important part of the image. ## results the results evaluation of the proposed approach for assamese language is quantitatively and qualitatively challenging task. the system generated results values are remarkable and it set benchmark for other existing systems for the task of caption generation in assamese language. there are a variety of evaluation metrics used in image captioning tasks that can be found in the literature. the bleu score @xcite is the most commonly used metric. in addition to this, the rouge score @xcite is also one of the popular metric to computes the performance of the image caption generation system. these metrics works by comparing a system generated captions with a set of reference summaries. the test sets of flickr30k and coco 2017 datasets contain 2000 and 5000 test images, to evaluate the proposed model's performance. for the test dataset, bleu has been recorded. we also experimented with our combined dataset, which contains 150k images in the training set and 7000 test images in the test set. we keep track of both our bleu and rouge and presented the scores in the majority of the images in this dataset, feature's human subjects with captions that are nearly identical. as a result of being trained on a large number of similar human subjects, the model during testing is unable to distinguish and describe non-human subjects. machine translation of english captions to assamese language has some limitation to translate compound sentences. the combined system was supposed to give a better accuracy in terms of bleu and rouge score. but, they surprisingly gave a bit worse result as both the datasets contained different type of images so combining both the dataset was not good for our system. although, it leaves us a space for future to work better how to get better results after the combination of two datasets. as shown in the some drawbacks can be solved by the use of multi head attention which can help to solve this problem by focusing on more than one region. also, transformers have a long way to go in the field of image captioning as the data is not always processed in the same order by transformers, and the attention mechanism provides context for any position in the input sequence. ## conclusions and future work in this paper, we have proposed an encoderdecoder framework for creating image captions in assamese. we looked at the assamese alphabet and its presence in the devanagari script, drawing parallels to the bengali language. the current model is based on a separate language model and a visual understanding machine. our primary focus was on translating english sentences, but in the long run, we are motivated to create a gold dataset for assamese image captions. with the sudden rise in the use of transformer-based frameworks in machine translation and other nlp tasks, image captioning using attention-based transformers could be a good experiment to investigate in the future. our main motivation was to create a benchmark model for image captioning in a lowresource language like assamese for the first time. we hope that other machine learning researchers will work in this area to develop better models and improve the system's functionality and accuracy which may benefit many coming researchers. in the future, we'll explore the possibility of using a combination of different encoder and decoder architectures to enhance the result even further. we'll also experiment with different sampling techniques to see if we can eliminate the bias toward certain phrases. the gain of this experiment can be added to the research on assamese image captioning and treated like a baseline model for further studies and future research.