VICTR : Visual Information Captured Text Representation for Text-to-Vision Multimodal Tasks (2020)

Abstract: Text-to-image multimodal tasks, generating/retrieving an image from a given text description, are extremely challenging tasks since raw text descriptions cover quite limited information in order to fully describe visually realistic images. We propose a new visual contextual text representation for text-to-image multimodal tasks, VICTR, which captures rich visual semantic information of objects from the text input. First, we use the text description as initial input and conduct dependency parsing to extract the syntactic structure and analyse the semantic aspect, including object quantities, to extract the scene graph. Then, we train the extracted objects, attributes, and relations in the scene graph and the corresponding geometric relation information using Graph Convolutional Networks, and it generates text representation which integrates textual and visual semantic information. The text representation is aggregated with word-level and sentence-level embedding to generate both visual contextual word and sentence representation. For the evaluation, we attached VICTR to the state-of-the-art models in text-to-image generation.VICTR is easily added to existing models and improves across both quantitative and qualitative aspects.

## introduction over the past decade, deep learning has achieved remarkable success in multimodal research problems, such as visual question answering, visual dialog, and image captioning. however, it is quite challenging to solve the text-to-vision multimodal tasks that deal with a text input and produce a visual output, such as text to image generation, text to video generation, or text to image retrieval. generally, natural language (incl. text) is a more natural medium for a human to describe the image that they want to generate or retrieve. however, raw text includes only limited information to fully describe and represent an image. for example, text to image generation tasks aim to generate photo-realistic images according to the given text descriptions. the current state-of-the-art (sota) text-to-image generation models @xcite @xcite mainly focus on generating the high resolution images by applying generative adversarial networks (gan) @xcite and rather neglect understanding input text descriptions. the most common text encoding approach in those sota models applies recurrent neural networks (rnn) to extract global sentence-level and word-level embedding, and the last hidden state of rnn cells is used as a direct input to the gan image generation model. however, rnnbased text encoding approaches are not capable of fully representing the rich visual semantics of input text descriptions in order to describe or generate photo-realistic visual output (e.g. image). for example, from the given sample text caption, a man on a skateboard with a brown dog outside, the last hidden state of rnn cells holds semantics and order of words and stores words' importance in the sentence. however, those information do not include most of the required information to describe/generate the image; for example, what objects are in the image (aspect of objects)? where are those objects (position of objects)? how to represent the relations between objects (relation between objects)? or for example, what objects are in the image (aspect of objects); where are those objects (position of objects); how to represent the relations between objects (relation between objects)? the question is: what would be the best approach to extract the rich visual semantics of input text descriptions in order to describe/generate an image? in this paper, we introduce a new visual contextual text representation (victr: visual information captured text representation), which represents the input text description with its visual semantics, as shown in figure 1 . the proposed model, victr, can be applied to diverse text-to-image multimodal tasks. victr has five different modules: 1) text to scene graph parsing, 2) scene graph embedding, 3) positional graph embedding, 4) visual semantic embedding, 5) visual contextual text representation. first, we extract scene graphs from input text descriptions in order to define what objects, attributes and relations should be in the image. the scene graph is initially proposed by @xcite in order to represent the objects, its attributes and their relations in the image. inspired by this, we generate scene graphs from the raw text description by using dependency parsing and transformer-based object-attribute-relation classification. then, we train the extracted object, attribute and relation nodes via graph convolutional networks (gcn) in order to generate the visual contextual text representation, a word-level representation which incorporates textual syntactic and visual semantic information. finally, it aggregates with word-level and sentence-level embedding respectively in order to generate a visual contextual word representation and visual contextual sentence representation. for evaluating quantitative and qualitative aspects of our proposed model, we attach victr to the sota models in text-to-image generation. thorough experiments on the coco benchmark dataset demonstrate the superiority of victr with respect to both semantic consistency and visual reality. the main contribution is summarised in the end of the sec. 2.3 ## related works and contributions we explore research trends in diverse text-to-vision multimodal tasks, which use text information as an only input to produce a visual output, including text to image generation and text to video generation. ## methodology as shown in figure 1 , the proposed visual contextual text representation, victr (visual information captured text representation), mainly focuses on capturing and representing the visual semantic information (i.e. location or aspect of the object in the image, positional relation between objects) from raw text descriptions. this is crucial for text-to-image generation tasks. in summary, the architecture of the proposed victr is composed into five modules: 1) text to scene graph parsing, 2) scene graph embedding, 3) positional graph embedding, 4) visual semantic embedding, 5) visual contextual text representation. note that the first module text to scene graph can be considered as a pre-processing step to generate visual information embedded text representation for the rest of the architecture. ## evaluation setup baselines three text-to-image generation sota models, stackgan @xcite , attngan @xcite , and dm-gan @xcite were selected as baselines, which all used text representation as the only source for image generation. we replaced the original text representation with our proposed victr and compared the generated images. dataset we evaluated the model performance on coco2014 @xcite ( http://cocodataset.org/ ), which is the most common benchmark and contains photo-realistic images with diverse objects and relations. detailed dataset statistics are shown in ## conclusion in this paper, we proposed a new visual contextual text representation for text-to-image multimodal tasks, called victr, which extracts rich visual semantic information from input text descriptions. we have shown improvement across both quantitative and qualitative aspects when applying victr to diverse sota models in text-to-image generation. we also present an analysis showing the ability of victr to automatically organise different aspects of objects and learn the relations between them. the human evaluation results show that victr produces images that are highly aligned with text captions and very realistic. it is hoped that victr provides the insight into future integration of text handling in text-tovision multimodal tasks.