Do Decoding Algorithms Capture Discourse Structure in Multi-Modal Tasks? A Case Study of Image Paragraph Generation (2022)

Abstract: This paper describes insights into how different inference algorithms structure discourse in image paragraphs. We train a multi-modal transformer and compare 11 variations of decoding algorithms. We propose to evaluate image paragraphs not only with standard automatic metrics, but also with a more extensive, “under the hood” analysis of the discourse formed by sentences. Our results show that while decoding algorithms can be unfaithful to the reference texts, they still generate grounded descriptions, but they also lack understanding of the discourse structure and differ from humans in terms of attentional structure over images.

## introduction what are the properties of the well-generated text? this question has been in the centre of many debates in the natural language generation community @xcite . while human evaluation has always been the gold standard in the quality assessment of generated texts, the field is often reluctant to run such evaluation due to the lack of standardisation in evaluation reports and generally high cost @xcite . therefore, a number of simpler and cheaper automatic metrics were introduced, specifically in the field of machine translation, although their validity has been questioned @xcite . as computer vision and nlp started to merge, automatic metrics became an important part of the evaluation process of image descriptions. in general, image descriptions are evaluated with means of bleu @xcite , meteor @xcite , rouge @xcite , cider @xcite and wmd @xcite . however, @xcite and @xcite have demonstrated that such metrics only weakly correlate with human judgements in the context of image description generation task. the discrepancy between human and automatic evaluation is deeply rooted in the differences between the fields of machine translation which originally introduced aforementioned metrics and image captioning, which adopted them. in principle, text-only evaluation is highly constrained: the key requirement for high-quality translation is the perseverance of semantics between two parallel texts. in comparison, evaluation of texts generated in multi-modal tasks is influenced by many factors as the generated texts might mention a different set of objects, attributes and relations which are not described in reference texts. such generations would cause low values from reference-based metrics, although they could be completely plausible and truthful to the image. as such, the tasks of machine translation and image captioning are inherently dissimilar in terms of evaluation. to mitigate this problem, metrics that directly compare texts against image objects have been proposed @xcite @xcite . they are typically better than bleu in that they assign a more accurate score to image-correct descriptions. a relatively recent trend has been to develop a set of metrics that would evaluate goal-oriented captions, produced with specific communicative intent @xcite or for a specific group of users @xcite , for example, if an image of a snowdrop is described as "the spring flower". a notable feature of the aforementioned metrics is their sole focus on evaluation of image captions. different from captions, multi-sentence image descriptions impose additional challenges for generation systems including understanding of the textual discourse in the multi-modal context. analysis of discourse has been in the focus of both text-only @xcite and language-and-vision tasks @xcite . however, given a huge interest in generation of longer image descriptions, e.g. image paragraphs @xcite @xcite , recipes @xcite and stories @xcite , we believe it is important to gain a deeper insight into how humans and models structure and realise discourse in such descriptions. in this paper, we understand discourse as a match between linearisation of the semantic knowledge (e.g., a fit of non-linear concepts into linguistic linear order) and underlying planning @xcite . we build on previous intuitions about evaluation in nlg and look under the hood of how different decoding algorithms build discourse in image paragraphs. we compare a number of decoding strategies for correspondence with how humans distribute and describe objects in longer texts. the main purpose of this study is to gain insights into whether decoding strategies generate texts similar to humans and whether these texts exhibit the corresponding discourse structure. there is a limitation on what and how things can be communicated and decoding algorithms have a direct control over it. the choice of decoding algorithm also has an effect on how information is expressed in the communicative channel @xcite and how successful its reconstruction by the perceiver will be @xcite . our results shed more light on the differences between decoding algorithms in terms of (i) the discourse structure, (ii) faithfulness to the reference texts, (ii) groundedness into the image and (iv) attentional structure. ## on the importance of decoding it is impossible to neglect the impact of the choice of the decoding on the structure of the generated texts foot_0 . discourse in multi-modal descriptions can be affected by many factors, including scene structure @xcite , the desire to have more accurate or more diverse texts @xcite and aspects of the task @xcite . other constraints include adherence to a specific topic as in poetry generation by controlling for content and form @xcite and incorporating pragmatic reasoning when describing images with text @xcite or optimising model's predictions for a specific metric @xcite @xcite in the spirit of reinforcement learning. notably, balakrishnan ## task and model as our modelling task, we choose the task of image paragraph generation and the tell-me-more corpus described in @xcite . in this task, a human is given an image and five (5) text fields. the describer writes sentences about the image so that they help a potential listener to identify it within a set. the describer is also asked to write sentences in a sequence, keeping in mind that after each sentence the listener needs more information to identify the image, e.g. thus, tell-me-more. @xcite show that collected multi-sentence descriptions have a fixed intentional structure, in the sense of @xcite , but attention structure demonstrates a different behaviour as supported by the analysis in @xcite . as our model, we use the architecture of the object relation transformer proposed by @xcite foot_1 . this is a two-stream multi-modal transformer, which consists of three self-attention blocks, operating on the image, text and across modalities. each block has the standard parts of the transformer @xcite : multi-head self-attention followed by a feed-forward network, residual connection and layer normalisation. on the vision side, the model takes the set of pre-extracted visual features of detected objects, which we receive by using the object detector released by @xcite foot_2 and pre-trained on visual genome @xcite . specifically, every object o j in the the set of detected image objects @xmath0, . . . , o |o| ) has a visual feature v @xmath1, where |o| = 36 and @xmath2. in addition, we store other outputs of the object detector, including object labels, attributes and confidence scores. they will be used in later stages to link paragraphs with objects in the image. the benefit of the object relation transformer is its ability to encode complex geometric relations between bounding boxes. thus, we also extract the set of geometric features @xmath3, y, w, h}, which are fused with visual features inside the model foot_3 . on the textual side, the model generates a paragraph word by word in auto-regressive fashion. specifically, it takes the current token w j and constructs its representation based on previously generated tokens w 1 , . . . , w j-1 . all the future tokens in the paragraph w j+1 , . . . , w |w| are replaced with the mask token, framing the task as the classic next word prediction task. the generation starts with the start token and ends when either the maximum length of the paragraph l is reached or when the end token is generated. as the last step, representation from two self-attention blocks are processed by the cross-attention which outputs the probability of all tokens from the vocabulary v. in terms of model's parameters, we keep all of them untouched, thus they correspond to the original set of parameters described in @xcite . we train the model on the full tell-memore dataset, consisting of 3590 image-paragraph pairs in the train set and 410 pairs in both validation and test sets. the analysis in this paper is performed on the test set only. ## decoding algorithms given the model vocabulary v and l as the maximum length of the generated sequence, the space of possible sequences has |v| l members, thus, becoming intractable. rather than traversing through such space, a number of different decoding methods are used to find the most likely sequence. the most straightforward heuristics is to take the most probable word w at timestamp j until either the maximum length of the generated sequence w is reached (@xmath4. we employ standard greedy search: a more popular and standardized approach is to use beam search, a version of the breadth-first search, that tracks multiple candidate sequences @xmath5, . . . , w k ) and chooses the one with the highest cumulative probability score, frequently computed as summation of word scores in each sequence. typically, the most probable sequence is picked as the final one, but other sequences can also be considered. the search starts with the word sequence w 1 = {start} and continues until the length of every predicted sequence reaches the maximum length l or all of them are completed with the end token: in beam search, the parameter k denotes the number of desired sequence candidates and b stands for the set of sequences currently under generation. beam search is computationally more expensive, but it is also more efficient in finding the optimal sequence due to more sophisticated exploration of the word space. however, bigger k often leads to "safe" and generic texts and candidate generations themselves can resemble each other a lot, lacking diversity @xcite or becoming repetitive @xcite . the problems of beam search have been addressed by many different approaches, mostly focused on increasing intra-set diversity of generated sequences @xcite . in one of such approaches, @xcite propose to extend beam search by incorporating a dissimilarity term in the objective function. specifically, diverse beam search splits beam sets into g groups w 1 , . . . , w g and at each word generation timestamp j for every sequence in the current group w g @xmath6, it encourages diversity with sequences from previous groups w h, @xmath7: a known issues with the top-k sampling algorithm is that it is hard to find the optimal value for the parameter k since setting it too low could remove highly probable words or, on the contrary, keep the less probable words if it is too high. instead of relying on pre-defined number of tokens, nucleus sampling @xcite takes words from the subset of the vocabulary in which the defined probability mass is concentrated: the main advantage of nucleus sampling is its ability to track the shape of the probability distribution, allowing for dynamic control of the number of candidates at each timestamp. a different, but related method to introduce controlled randomness is to use temperature scaling. the diversity is achieved by controlling the peaks in the distribution and dividing it by the parameter τ : for our experiments with decoding algorithms, we set the following set of parameters. we set the beam size @xmath8. @xcite argue that setting setting @xmath9, therefore, we set @xmath10.5. for top-k sampling, we try multiple values for k, aiming to investigate the impact of this parameter on generation. specifically, we generate texts with k being the value from the following set: {25, 50, 75, 100}. for nucleus sampling, we set p to one of the following values: {25, 50, 95}. we also run pure sampling with @xmath11.5. our parameters for different inference algorithms are chosen based on experiences from the corresponding research that introduces these algorithms. they also reflect our goal of evaluating how results generated by different searches can be affected by a single hyperparameter. ## linking in the context of the image paragraph generation task, discourse structure in texts is affected by both ## automatic evaluation the reason for a high performance of greedy search could be its ability to generate the "safest" words combined with the simplicity of the images and the lack of surprisal in them. for example, images in the dataset correspond to standard room types and thus contain standard objects. this is a blessing if we care about model's ability to generalise over the house environments, but also a curse since occasionally a highly salient and surprising object might appear in the images and searches will not be able to describe it. we believe that our results show the inadequacy of automatic metrics in measuring the sensitivity of inference algorithms ref there is a very large well made bed on the left. this looks like a hotel room. two lamps are on a table on the right side. a window that you can barely see out of can be seen if you look straight ahead. there is a picture on the wall to the left and another on the right side wall. g this is a living room with a white couch and love seat. there is a white couch with a white and white pillow. there is a coffee table in front of the couch. there is a glass coffee table in the middle of the room. there is a picture hanging on the wall. b2 this is a living room. there is a white couch. there is a white couch. there is a white coffee table. there is a couch. s25 this is a tv in unk. the couches match the couches and bedspread is white. there are two green unk and there is a sofa table with pillows. there are two different decorative beige feet couch. s50 a large room with a sofa for a coffee table. a four drawer rectangular coffee table sits full of glass roses loveseat. a fireplace is on the left hand wall. a large telephone is near the couch. there are two brown chairs sitting at the far end of the couch. s75 its a formal living room. a glass sofa and ottoman. a glass table is behind the couch. the tv is behind the sofa. a patterned sofa is on the right side of the one sofa. s100 living room with a black couch with white print pillows underneath. coffee table piece of 2 candles hang on the back wall. green coffe coffee table lobby is sitting in the middle of the couch by chair and books. decorative and an condition painted on the floor in color. light brown seat at the back end of the room. st50 this is a living room. the couches are white. there is a white couch. there is a coffee table. there is a white couch. ## human evaluation to support our hypothesis that automatic metrics are not enough to measure fine-grained differences between various decoding algorithms, we conduct a human evaluation on amazon mechanical turk. we randomly sample 10% of images from the test set, which equals 41 items. for each of these images, we take generated texts from the top-6 decodings based on the cider score. we get 287 different image-text pairs to evaluate. during the evaluation, we provide workers with an image and its description and ask them to answer 3 (three) dif-ferent questions, aiming to evaluate (i) relevance: does the text describe relevant and essential objects, (ii) correctness: does the text describe objects correctly (e.g., using correct words), (iii) composition: do object descriptions naturally follow each other. the example item for human evaluation is shown in appendix a. each judgement is a score on a scale between 1 and 5, where 1 is the lowest rank. we collect three different judgements per item and average them. we pay 0.17 us dollars for a single assignment and restrict the location of the workers to the us, the uk, canada, ireland or australia. we also ran our experiments with master workers only (25 different human participants). we follow ref g b2 s25 s50 s75 s100 st50 n25 n50 n95 db2 s1 2.9 0.9 0.3 1.0 1.1 1.1 1.0 0.9 1.1 1.1 1.1 0.9 s2 1.6 1.7 1.5 1.7 1.7 1.8 1.8 1.7 1.8 1.8 1.7 1.8 s3 1.4 1.6 1.4 1.7 1.8 1.8 1.8 1.6 1.8 1.8 1.8 1.6 s4 1.3 1.6 1.4 1.8 1.8 1.8 1.9 1.6 1.8 1.9 1.8 1.6 s5 1.2 1.7 1.4 1.8 1.8 1.8 1.7 1.6 1.7 1.8 1.7 1. 7 the correlation scores are presented in ## grounded evaluation the results demonstrate that half and more of the generated noun phrases can be linked with objects in the image. in general, sampling algorithms generate fewer number of grounded noun phrases, possibly due to the increased randomness. greedy search, beam and diverse beam generate the highest number of noun phrases which are truthful to the image. we believe that while reference-correctness of generated texts can get worse, inference algorithms are still able to generate alternative descriptions of images which can be grounded. however, the structure of discourse reflected on the surface level and the level of grounding might not necessarily correspond to the one observed in references. in the next experiment, we look at the problem under the angle of attentional structure and examine spatial arrangement of linked objects and how these arrangements differ between decoding algorithms. ## attentional structure of discourse figure 1 demonstrates a number of the attention heatmaps across areas in the image for different sentences. at first glance, different inference algorithms look at similar locations in the image and also focus on parts which are attended by humans. however, there are relatively more areas described in the first sentence of the references, while a much smaller and fewer areas are described in generated texts. this could be directly related to the fewer number of objects and under-generation discussed previously. the second and third sentences describe specific areas of the image in all cases, mostly central ones. interestingly, greedy and diverse beam have highly similar attention across the image. in sentence 4, human attention disperses over the full scene, while it is unclear whether the same pattern happens in generated texts. this could signal a possible topic shift, happening in later parts of the paragraph and inability of searches to capture that. to understand the differences on the level of sentences better, we measure the correlation between flattened heatmaps pixel by pixel. we use pearson product-moment correlation coefficient which can be applied to images across the channels. the results are shown in figure 2 . as we can see, in sentence 4 attentional structure on the image differs between searches and references, supporting the idea of topic shift. sampling methods have the lowest correlation with the references, while nucleus with @xmath14. note that the correlation in the first sentence is lower than in the second and the third one for most of the searches. this could be related to the importance of the first sentence and a big-ger number of noun phrases in it, which are not generated during the decoding stage. ## conclusion in this paper we described our analysis of how decoding strategies structure discourse in multimodal longer image descriptions. we performed evaluation using intuitions from different evaluation perspectives: automatic, surface-based (nongrounded), image-based (grounded) and attentionbased. the results suggest that for the task of image paragraph generation decoding algorithms diverge from humans in generating specific type of discourse. although they might generate referenceincorrect but image-correct descriptions, it is unclear what kind of discourse is generated in the end. in general, algorithms which are less random construct discourse similar to the one in human references, while sampling-based methods generate a different type of discourse, which is hard to control for. we plan to use the insights described in this paper and build a metric that would evaluate the structure of longer image paragraphs, reflected in both object and relation descriptions as this is currently a much needed evaluation measure.