Image2tweet: Datasets in H indi and E nglish for Generating Tweets from Images (2021)

Abstract: Image Captioning as a task that has seen major updates over time. In recent methods, visual-linguistic grounding of the image-text pair is leveraged. This includes either generating the textual description of the objects and entities present within the image in constrained manner, or generating detailed description of these entities as a paragraph. But there is still a long way to go towards being able to generate text that is not only semantically richer, but also contains real world knowledge in it. This is the motivation behind exploring image2tweet generation through the lens of existing image-captioning approaches. At the same time, there is little research in image captioning in Indian languages like Hindi. In this paper, we release Hindi and English datasets for the task of tweet generation given an image. The aim is to generate a specialized text like a tweet, that is not a direct result of visual-linguistic grounding that is usually leveraged in similar tasks, but conveys a message that factors-in not only the visual content of the image, but also additional real world contextual information associated with the event described within the image as closely as possible. Further, We provide baseline DL models on our data and invite researchers to build more sophisticated systems for the problem.

## introduction generating a textual description of an image is called image captioning. it can be an easy process for most adults, but for a machine to generate a rich and vivid description is a difficult task. image captioning requires to recognize the important objects, their attributes and their relationships in an image. it also needs to generate syntactically and semantically correct sentences. this task involves the knowledge of both computer vision and natural language processing. image captioning has been a very popular research area since the last decade. even before the boom of neural network based techniques people tried various hand crafted features such as local binary patterns (lbp) @xcite , scale-invariant feature transform (sift) @xcite , the histogram of oriented gradients (hog) @xcite along with classical ml methods like svm for image captioning. on the other hand, while using neural network based techniques, features are learned automatically from training data and they can handle a large and diverse set of images @xcite @xcite . moreover, the availability of large and new datasets has made the learning-based image captioning an interesting research area. the popular datasets for english image captioning are -flickr30k dataset @xcite , ms coco @xcite , and google conceptual caption dataset @xcite . however, there is almost no research of image captioning in hindi and/or indian languages. image captioning is important for many reasons. for example, they can be used for automatic image indexing. image indexing is important for content-based image retrieval (cbir) and therefore, it can be applied to many areas, including biomedicine, commerce, the military, education, digital libraries, and web searching. image2tweet takes one step ahead of regular image captioning task. it involves generating captions that are not only semantically rich but also contain some real world knowledge @xcite . the task is that given an image, the machine has to generate a tweet from it. an example is provided in figure 1 . generating this level of detailed tweets requires person identification (sachin tendulkar), object detection (bjp logo) etc. in this paper, we describe the image2tweet task and release a new dataset for the task and also release a novel hindi dataset to ignite the image captioning research for indian languages. ## related work there are quite a few popular image captioning datasets. flickr30k @xcite consists of 30k images and each image has 5 captions. coco @xcite dataset consists of 330k images and each image has 5 captions. google conceptual caption (sharma et al., 2018) has approximately 3.3 million images and each image has only one caption. however, such datasets use commonly found images over the web and couple the images with alttext descriptions. most of the descriptions use proper nouns (such as characters, places, locations, organizations, etc.). such proper nouns pose some problems because a image captioning model is difficult to learn such fine-grained proper noun inference from the input image pixels. at the same time, there is very little research done on hindi image captioning. to the best of our knowledge, ours is the first dataset to generate tweet from images and to release a hindi dataset. deep learning methods are the most popular to solve the image captioning task. jiang et al. (2018) proposed novel recurrent fusion network (rfnet), which exploits complementary information from multiple encoders to tackle image captioning. xu et al. (2015) propose an encoder-decoder method which incorporate spatial attention mechanism to help the model to determine which regions to focus in an image. yang et al. (2016) propose a framework called reviewnet. zhou et al. (2020) proposed a unified vision-language pre-training for image captioning which can be easily fine tuned. similar to caption generator meme generation has also been a eye-catching task for researchers. the task is to generate memes based on the image. unlike captioning, here in meme generation it has to generate text for multiple persons, if multiple persons are involved in meme image. kurochkin (2020) released a dataset consisting of 650k meme instances. they applied gpt-2(radford et al., 2019) model for meme generation and observed that machine generated meme text's are not that engaging as human generated. ## task description image captioning for english is well studied paradigm and researchers have tried various methods like hand crafted features @xcite @xcite along with classical ml methods like svm. during the last decade numerous of big datasets have been released and quite a few efforts can be noticed but there is a still shortage of works in indic languages. image2tweet is a shared task where we move a step forward from image captioning. the task is to generate a tweet like a human/news reporter given an image. we release datasets for two languages -english and hindi. figures 2 and 3 show an instance from the english and hindi data respectively. ## dataset the data consists of image-tweet pairs. we provide 2 dataset -english and hindi. the hindi data is collected by crawling tweets from two well known hindi newspapers -dainik bhaskar and dainik jagran. the english data is crawled from the twitter handle of times of india. we use twitter api foot_0 to crawl the tweets. we collect total 70k tweets for english image2tweet and 51k for hindi image2tweet. ## baseline we develop our baseline using bert @xcite and vgg-19 @xcite . the bert model is pre-trained on whole english wikipedia and brown corpus for next sentence prediction objective. we design is a two branch model (refer figure 8 ). while training, the image embedding obtained from vgg-19 is passed to one branch and the text is passed to the other branch. in the first branch the image embedding is passed to dense layer. in the second branch the text is sent to bert tokenizer and its output passed to the pre-trained bert. then the output from the last layer of bert is passed to an lstm layer which is given as an input to max pooling and to and average polling. the output vectors of max pooling and average pooling are concatenated. after this, we concatenate the outputs from both the branches, and give the concatenated vector as input to an lstm followed by a dense layer. the output vector of this dense layer is used to generate the words in tweet. for training we use adam optimiser, and train it with a mini-batch size of 32. the learning rate is set to 1e -5. the max caption length is set to 34. while testing, we pass the image vector and the sequence of words generated so far and will predict the next word. likewise we go on until the end token appears. we use greedy search method to generate the whole tweet. the baseline code is available at https:// github.com/git-rishabh-jha/image2tweet. ## results ## conclusion in this paper we define the task image2tweet and release datasets in hindi and english for the task. the english and hindi datasets consists of 70k and 51k image-tweet pairs respectively. we cluster the similar tweets in our dataset for better evaluation of the system generated tweets. generated tweets are evaluated using cider. further, we provide vgg19 + bert based baseline systems for our data. image2tweet is more difficult than traditional image captioning and we believe it needs further research attention. future work includes collecting data for more languages, building more complex systems for the task etc.