Structural and Functional Decomposition for Personality Image Captioning in a Communication Game (2020)

Abstract: Personality image captioning (PIC) aims to describe an image with a natural language caption given a personality trait. In this work, we introduce a novel formulation for PIC based on a communication game between a speaker and a listener. The speaker attempts to generate natural language captions while the listener encourages the generated captions to contain discriminative information about the input images and personality traits. In this way, we expect that the generated captions can be improved to naturally represent the images and express the traits. In addition, we propose to adapt the language model GPT2 to perform caption generation for PIC. This enables the speaker and listener to benefit from the language encoding capacity of GPT2. Our experiments show that the proposed model achieves the state-of-the-art performance for PIC.

## introduction to effectively communicate with human, an important step involves image captioning (ic) that requires systems to describe images using natural language captions. image captioning (ic) has been studied extensively, featuring deep learning models (i.e., the encoder-decoder architectures) as the dominant approach @xcite @xcite . despite its popularity, the current work on ic has mainly considered the factual setting for ic where the generated captions should faithfully present the visual content of images. a major limitation for this factual ic task concerns its failure to incorporate human factors (i.e., personalities or traits) into the caption generation process. as such, ones might prefer to produce engaging captions where his/her personality traits are explicitly expressed and the visual concepts in the images are not necessarily covered in their full details. consequently, in this work, we seek to fill in this gap for ic by exploring personality image captioning (pic) where the models need to further consider a personality/trait in the captioning process. in particular, we leverage personality-captions (pc) @xcite , the first dataset for pic, to evaluate the models in this work. which characteristics should a caption have to adequately describe an image in pic? motivated by the functional and structural decomposition for language learning @xcite @xcite , we argue that an effective caption for pic should posses two important properties. on the one hand, the captions in pic should follow the natural language structures to induce effective communication with human (i.e., the structural view or naturalness of the captions). on the other hand, for the functional view, the generated captions from a model should involve sufficient information to enable another system or human to uniquely identify the input images and traits. in this paper, we propose to achieve these two goals by recasting pic as a multi-agent communication framework that involves a speaker and a listener @xcite . the speaker attempts to generate a natural language caption for a given image and trait (i.e., for the structural property) while the listener seeks to identify the input images and personality traits based on the generated caption from the speaker (i.e., for the functional property). by training this framework, we expect that the generated captions of the speaker can be regularized to naturally convey the information in the images and express the provided personality traits at the same time. to our knowledge, this is the first work to solve pic via a multi-agent communication framework. a bottleneck in the training of the speakerlistener framework concerns the ability to model the language effectively for the captions in pic. in particular, the speaker would benefit from a highquality language model that can produce natural captions for pic while the listener would make better predictions for the image and trait identification if it can effectively encode the generated captions. although ones can attempt to learn those language modeling abilities directly from the provided captions of the pic datasets, this approach cannot exploit the enormous amount of the external text to boost the performance for pic. in this work, we propose to employ the pretrained language model gpt2 @xcite as a language prior for both the speaker and listener in the multi-agent communication framework. as gpt2 has been trained on a large amount of unlabeled text, we expect that its incorporation can significantly improve the language modeling/encoding for the speaker and listener. to our knowledge, this is also the first work to consider pre-trained language models for pic. finally, we conduct extensive experiments on the pc dataset to demonstrate the benefits of the proposed framework, leading to the state-of-the-art performance for this dataset. ## model given an image i and a personality trait t (i.e., a word), the goal of pic is to generate an engaging caption ĉ = ŵ1 , ŵ2 , . . . , ŵ n (i.e., of n words). in the supervised learning setting, there is a ground-truth caption c for each pair (i, t ): @xmath0, w 2 , . . . , w n (i.e., of n words). to encode i, we first feed it into the resnext convnet model @xcite to obtain a feature map of size 7⇥7⇥2048. this can be viewed as a matrix v of size 49⇥2048, where each row encodes the visual content for a cell of the uniform image grid. v is called the representation of i in the following. also, we use t to refer to the personality trait or its embedding vector interchangeably in this work (these vectors are randomly initialized and updated during training). ## related work the main approach for ic so far involves deep learning models where several datasets have been created @xcite and different variants of the encoder-decoder architectures have been proposed @xcite @xcite . pic is a way to encourage more engaging captions for which several features are considered, i.e., location and age @xcite , reader's active vocabulary @xcite , humour @xcite , sentiment @xcite , dialog/conversation @xcite , and caption styles @xcite . the closest work to ours is @xcite that examines a different feature of diverse personality traits. our work also bears some similarity with the previous ic models that attempts to improve the ability to discriminate images for the generated captions @xcite @xcite . however, these ic models do not capture personality traits for pic as we do. we also note the stylized ic model in @xcite that applies a style classification loss. however, this work does not consider the speaker-listener framework with reinforce training as gpt-speaker. above all, none of these works has exploited pre-trained language models (i.e., gpt2) for pic. ## conclusions we formulate pic as a communication framework between a speaker and a listener. a novel training mechanism for this framework is introduced, exploiting the rewards in reinforce to encourage the generated captions to be natural and informative about the input images and traits. we also introduce the pre-trained language model gpt2 into the model to benefit from its language modeling/encoding capacity. the experiments demonstrate the effectiveness of the proposed model for pic.