Building Joint Relationship Attention Network for Image-Text Generation (2022)

Abstract: Attention based methods for image-text generation often focus on visual features individually, while ignoring relationship information among image features that provides important guidance for generating sentences. To alleviate this issue, in this work we propose the Joint Relationship Attention Network (JRAN) that novelly explores the relationships among the features. Specifically, different from the previous relationship based approaches that only explore the single relationship in the image, our JRAN can effectively learn two relationships, the visual relationships among region features and the visual-semantic relationships between region features and semantic features, and further make a dynamic trade-off between them during outputting the relationship representation. Moreover, we devise a new relationship based attention, which can adaptively focus on the output relationship representation when predicting different words. Extensive experiments on large-scale MSCOCO and small-scale Flickr30k datasets show that JRAN achieves state-of-the-art performance. More remarkably, JRAN achieves new 28.3% and 58.2% performance in terms of BLEU4 and CIDEr metric on Flickr30k dataset.

## introduction image-text generation (i.e., image captioning) is a typical cross-modal task that connects natural language processing (nlp) and computer vision (cv) @xcite . its core goal is to automatically predict a meaningful and grammatically correct sentence, which can accurately describe the main content of images. practical applications for this task mainly include injecting visual intelligence into the chatbots, searching semantic image, and helping people with visual impairments to understand the visual world. however, image-text generation is still a challenging task. the main @xcite that mainly uses visual features to generate sentence. (b) presents a more accurate sentence generated by our jran method which learns two relationship, i.e., visual relationship between region features and visualsemantic relationship between region features and semantic features (i.e., background and environment). for difficulty 1), encouraging by the method @xcite , the noise can be injected into rnn hidden states to predict the mean and standard deviation, and manipulate the rnn transition states. in this way, the network robustness can be significantly enhanced and the issue can be well solved. however for difficulty 2), although some related visual attention based methods @xcite @xcite achieve remarkable progress, they usually focus on the image visual features while ignoring the relationships between them. this makes the model often difficult to generate an accurate or appropriate description that can correctly describe the relationships among objects in the image. for example, as illustrated in figure 1 , only using the detected visual features in the image, sc (base) method @xcite predicts a description "three boys are playing near the ocean.", where the verb phrase "playing near" indicates the relationship between the objects "boy" and "ocean". obviously, this description cannot accurately reflect the main scene of the image. contrarily, our jran accurately describes the main content of image by effectively learn the visual relationship between object region features and the visual-semantic relationship between region features and semantic features, generating a more relevant sentence "three children are flying a box kite on the beach." the word "flying" appropriately describes the relationship between the two region features "children" and "kite", and the word "on" accurately represents the relationship between region information "children" and semantic information "beach" (i.e., background/surrounding). thus, learning the relationship between image features is of crucial importance for generating accurate sentence description for image. based on the above observations, different from previous relationship based approaches @xcite @xcite (see figure 2 (top)) that only explore the single feature relationship in the image, we present a new joint relationship attention network (jran) that novelly learns the joint relationship between region features and semantic features in figure 2 (bottom). specifically, we first utilize the object detector faster r-cnn and cnn to extract region features and complementary semantic features from the input image. then, jran builds two types of relationship, i.e., visual relationship and visualsemantic relationship. the former is solely based on the detected region features in the image, and encodes the visual relationship between region features. instead, the latter takes the region features and semantic features of the image into account to fully explore the visual-semantic relationship between them. as shown in figure 3 , to obtain the image representation containing joint relationship features, we devise a core competition module called joint relationship learning network, which can effectively learn the visual relationship and visual-semantic relationship while dynamically balancing the different contributions between them. after that, we introduce the relationship based attention module, which can adaptively focus on the obtained most relevant joint relationship features during generating words. the whole framework of jran could be jointly learnt and optimized in an end-to-end way. our jran method achieves significant improvement compared with the related relationship based methods. more remarkably, it can be integrated into a better baseline model to achieve better performance. our contributions mainly include: firstly, we propose a novel image-text generation network that utilizes the complementary region and semantic features in the image for enriching feature representations. secondly, we devise a joint relationship learning network, which can fully learn both visual relationship and visual-semantic relationship in the image, and further balance their contributions during predicting different words. finally, exhaustive experiments indicate that our jran is not only effective on large-scale but also achieves superior performance on small-scale datasets. ## our approach the main purpose of this work is to explore the relationship between different features in the image, so as to generate a sentence description containing accurate interaction information for the input image. the overall architecture of the proposed jran is shown in figure 3 . the critical elements in the architecture are described in detail as follows.