When an Image Tells a Story: The Role of Visual and Semantic Information for Generating Paragraph Descriptions (2020)

Abstract: Generating multi-sentence image descriptions is a challenging task, which requires a good model to produce coherent and accurate paragraphs, describing salient objects in the image. We argue that multiple sources of information are beneficial when describing visual scenes with long sequences. These include (i) perceptual information and (ii) semantic (language) information about how to describe what is in the image. We also compare the effects of using two different pooling mechanisms on either a single modality or their combination. We demonstrate that the model which utilises both visual and language inputs can be used to generate accurate and diverse paragraphs when combined with a particular pooling mechanism. The results of our automatic and human evaluation show that learning to embed semantic information along with visual stimuli into the paragraph generation model is not trivial, raising a variety of proposals for future experiments.

## introduction the quality of automatically generated image captions @xcite has been continuously improving as evaluated by a variety of metrics. these improvements include use of neural networks @xcite , attention mechanisms @xcite and more fine-grained image features @xcite . more recently, a novel openended task of image paragraph generation has been proposed by @xcite . this task requires the generation of multi-sentence image descriptions, which are highly informative, thus, include descriptions of a large variety of image objects, and attributes, which makes them different from standard single sentence captions. in particular, a good paragraph generation model has to produce descrip-tive, detailed and coherent text passages, depicting salient parts in an image. when humans describe images, especially over longer discourses, they take take into account (at least) two sources of information that interact with each other: (i) perceptual information as expressed by visual features and (ii) cognitive reasoning that determines the communicative intent of the text and the use of language @xcite . perceptual information mainly determines what to refer to while the reasoning mechanisms tell us how and when to refer to it. both mechanisms interact: that a particular object is described at a particular point of discourse and with particular words depends not only on its perceptual salience but also whether that object should be referred to at that point of the story that the text is narrating which is its discourse salience. compare for example: "two cows are standing in the field", "there are trees in the field" and "a few of them are close to the trees". the selection and the order of the relevant features are described by a cognitive mechanism of attention and memory @xcite . in this paper, we investigate the interplay between visual and textual information (reflecting background knowledge about the world and communicative intent) and their ability to generate natural linguistic discourses spanning over several sentences. our primary research question is as follows: does using both visual and linguistic information improve accuracy and diversity of generated paragraphs? we experiment with several types of inputs to the paragraph generator: visual, language or both. we also investigate the effects of different kinds of information fusion between visual and textual information using either attention or maxpooling. we demonstrate that multimodal input paired with attention on these modalities benefits model's ability to generate more diverse and accu-rate paragraphs. we evaluate the accuracy and diversity of our paragraphs with both automatic metrics and human judgements. we also argue that, as some previous work shows @xcite , n-grambased metrics might be unreliable for quality evaluation of generated texts. the generated paragraph can be accurate as of the image, but because it does not match the ground truth, this would score low based on the automatic evaluation. to provide a different view on paragraph evaluation, we asked humans to judge the subset of generated paragraphs across several criteria, more specifically described in section 3.4 and appendix a. in language and vision literature, "diversity" of image descriptions has been mostly defined in terms of lexical diversity, word choice and n-gram based metrics @xcite @xcite . in these papers, the focus is on generating a diverse set of independent, one-sentence captions, with each describing image as a whole. each of these captions might refer to identical objects due to the nature of the task ("describe an image with a single sentence"). then, diversity is measured in terms of how different object descriptions are from one caption to another (e.g. a man can be described as a "person" or "human" in two different captions). however, as argued above, a good image paragraph model must also introduce diversity at the sentence level, describing different scene objects throughout the paragraph. here, we define paragraph diversity with two essential conditions. first, a generative model must demonstrate the ability to use relevant words to describe objects without unnecessary repetitions (word-level diversity). secondly, it must produce a set of sentences with relevant mentions of a variety of image objects in an appropriate order (sentence-level diversity). producing structured and ordered sets of sentences (e.g. coherent paragraphs) has been a topic of research in nlg community for a long time with both formal theories of coherence @xcite and traditional rule-based model implementations @xcite . the coherence of generated text depends on several nlg sub-tasks: content determination (selection), the task of deciding which parts of the source information should be included in the output description, and text structuring (micro-planning), the task of ordering selected information @xcite . we believe that the hierarchical structure of our models reflects the nature of these tasks. first, the model attends to the image objects and defines both their salience and order of mention and then it starts to realise them linguistically, first as paragraph visual-textual topics and then as individual sentences within paragraphs. the focus of our work is not to improve on the results of @xcite but to investigate the effects of different multi-modal fusion on the accuracy or the diversity of paragraph descriptions. ## related work neural image paragraph captioning the task of generating image paragraphs has been introduced in @xcite along with the dataset of image-paragraph pairs. the authors hierarchically construct their model: sentence rnn is conditioned on visual features to output sentence topics. then, each of these topics is used by another rnn to generate actual sentences. our models are based on this hierarchical model. however, we substantially change its structure and also remove the end of paragraph prediction. @xcite also use the hierarchical network, but with an adversarial discriminator, that forces model to generate realistic paragraphs with smooth transitions between sentences. @xcite also address cross-sentence topic consistency by modelling the global coherence vector, conditioned on all sentence topics. different from these approaches, melas-kyriazi et al. (2018) employ self-critical training technique @xcite to directly optimise a target evaluation metric for image paragraph generation. lastly, @xcite use convolutional auto-encoder for topic modelling based on region-level image features. they demonstrate that extracted topics are more representative and contain information relevant to sentence generation. we also model topic representations, but we use additional semantic representations of image objects as part of the input to our topic generator. @xcite has proposed a non-neural approach to generate texts describing images. however, this approach depends on multi-ple components: visual scene parsing, generative grammar for learning from training descriptions, and an algorithm, which analyses scene graphs and extracts semantic trees to learn about dependencies across sentences. ## conclusion in this paper, we addressed the problem of generating both accurate and diverse image paragraphs. we demonstrated that utilising both visual and linguistic information might benefit the quality of generated texts depending on the pooling mechanism that is used. we showed that intrinsic evaluation metrics are insufficient for evaluation of paragraphs as they focus on lexical choice and do not capture human level of judgement: lng+att is judged as the best model in human evaluation, while it is not among the leaders according to the automatic evaluation. we believe that our work is a good starting point for further investigation of the ways multiple sources of information about the world can be merged for learning generation of high-quality multi-sentence stories, describing real-world visual scenes. in our future work we also intend to test how our models can generate task-dependent paragraphs. for this task we will use the dataset of image description sequences @xcite which consists of paragraphs collected in a taskbased setting to train our models. in contrast, in the stanford dataset humans were not given a specific task when describing images. we believe that generation from more context-dependent and structured descriptions can open up new perspectives for the research on image paragraphs.