Violet: A Vision-Language Model for A rabic Image Captioning with Gemini Decoder (2023)

Abstract: Although image captioning has a vast array of applications, it has not reached its full potential in languages other than English. Arabic, for instance, although the native language of more than 400 million people, remains largely underrepresented in this area. This is due to the lack of labeled data and powerful Arabic generative models. We alleviate this issue by presenting a novel vision-language model dedicated to Arabic, dubbed Violet. Our model is based on a vision encoder and a Gemini text decoder that maintains generation fluency while allowing fusion between the vision and language components. To train our model, we introduce a new method for automatically acquiring data from available English datasets. We also manually prepare a new dataset for evaluation. Violet performs sizeably better than our baselines on all of our evaluation datasets. For example, it reaches a CIDEr score of 61.2 on our manually annotated dataset and achieves an improvement of 13 points on Flickr8k.

## introduction captioning images involves describing the visual elements of a picture using natural language. this requires a system that combines the strengths of two models: one that can represent the visual elements of an image, and another that can translate this representation into natural language. the latter employs a language model to produce fluent (i.e., grammatically accurate) and adequate (i.e., capturing sufficient semantic information) descriptions. in recent years, research on vision language models (vlms) and their applications has boomed @xcite @xcite . owing to the rapid advancements in large language models (llms), the performance of vlms has improved dramatically. more concretely, vlms have progressed from merely providing descriptions that vaguely resemble a given image @xcite to accurately describing complex visual cues within the image. the pretraining-then-finetuning paradigm also plays a significant role in achieving such impressive results, as it allows models to first grasp general language structures and then specialize in the specific task of image captioning @xcite . progress in vlms, however, has been witnessed thus far primarily on english @xcite . this leaves behind a large number of other languages for which no sufficient image captioning data or language models exist. arabic is a case in point where image captioning lags far behind @xcite . similar to other low-resource languages, progress in arabic image captioning has been hampered by the lack of publicly available datasets and limited efforts in creating any such data. manual creation of image datasets, after all, requires a huge amount of time and labor. again, the unavailability of powerful arabic language models that understands the struc-1 ture of the language and can capture its rich morphology has also caused a delay in the development of vlms. given the rapid progress in vision language technologies and their wide applications in society, limited progress in this area can have negative consequences for the arabic-speaking world. to address this important issue, we introduce a novel arabic image captioning model dubbed violet. our new model is comprised of two main components: a vision encoder and a text decoder. for the vision encoder, we employ an object detector network based on @xcite to extract visual features that are then passed to a compact transformer encoder. at the decoder side, we leverage the recently developed generative pretrained model jasmine @xcite . taking inspiration from @xcite , we split our text decoder into two halves: the first half functions as a text decoder, whereas the second incorporates cross-attention layers, effectively serving as a fusion decoder. given the dual nature of our decoder, we refer to it as gemini. drawing parallels with visualgpt @xcite and the meshed transformer @xcite , we also adopt a meshed connection between the transformer vision encoder and the text decoder to foster enhanced communication between the encoder and decoder layers. the other major challenge we face in our work is the unavailability of native arabic captioning data. we alleviate this challenge by introducing a method for automatically acquiring captions that is based on first employing a powerful machine translation model followed by a quality assurance mechanism for removing poor captions. for evaluation, in addition to reporting on arabic translated dataset, we task five human annotators to manually caption an image dataset. compared to previous works and baselines, our novel model excels in captioning images in fluent arabic. figure 1 offers four examples of fluent arabic captions generated by our novel model. figure 2 shows a comparison of our model performance with prior research on flickr8k in cider score. in summary, our contributions are as follows: early methods for image captioning involve either retrieving descriptions @xcite or using template filling combined with manually designed natural language generation techniques @xcite . however, modern image captioning primarily relies on deep learning models. in early work, image captioning is framed as an imageto-sequence task using encoder-decoder models, with convolutional neural networks (cnns) as encoders and recurrent neural networks (rnns) as decoders while incorporating attention mechanisms @xcite @xcite . soon after, using a transformer architecture of a vision encoder with a text decoder became the defacto direction towards solving the problem of image captioning @xcite . some approaches use a detection model to extract visual features and then pass it to a transformer text decoder as in oscar @xcite , while others like coca @xcite train a transformer vision encoder with a text decoder from scratch on a large-scale dataset. more recently, there has been a shift towards using pre-trained llms and vision models. generative image-to-text transformer (git) @xcite is a decoder-only transformer that utilizes a clip @xcite visual encoder to incorporate both visual and textual inputs. another method to consider is visualgpt @xcite ., we use an object detection network to extract k object features (k equal 50 in our case) from an image. after projecting to a lower dimension, the features are fed to an l-layer (three-layer in our architecture) transformer encoder. meshed connection is employed between the encoder and decoder layers, where each encoder layer contributes to the cross-attention output. our text decoder is split into two halves, the first half is the standard frozen pretrained text decoder layers, while the second half has cross-attention layers inserted after each self-attention layer. we call this design a gemini decoder. we employ a gating mechanism through πt and πm that controls the flow of information from the vision and language sides. the final input to the feed forward network in each cross-attention layer is the weighted sum of each encoder-decoder attention controlled by the α parameters. ## experiments we analyze the performance of three variations of our architecture: (i) using the normal decoder with cross-attention in each layer, (ii) using gemini decoder without freezing the text part, and (iii) using gemini decoder while freezing the text part. as a baseline, we train a visualgpt model @xcite on the english mscoco training set then translate output into arabic using nllb. our trained visualgpt achieves a 117.8 cider score on the english mscoco validation set. we conduct our experiments on three datasets, as follows: ## conclusion in this paper, we introduced violet, an arabic image captioning model leveraging the pretrained text decoder jasmine. our results demonstrated the efficacy of our gemini decoder in enhancing performance while simultaneously reducing the number of model parameters and computations. we also presented a new method that is effective for acquiring arabic captioning data from available english data. in addition, we manually annotated a new dataset for evaluating arabic image captioning models. our model outperforms all of our baselines and promises to enable benchmarking in this area. we will release our model and datasets to advance arabic vison-language research.