Most of the existing works on image description focus on generating expressive descriptions. The only few works that are dedicated to generating stylish (e.g., romantic, lyric, etc.) descriptions suffer from limited style variation and content digression. To address these limitations, we propose a controllable stylish image description generation model. It can learn to generate stylish image descriptions that are more related to image content and can be trained with the arbitrary monolingual corpus without collecting new paired image and stylish descriptions. Moreover, it enables users to generate various stylish descriptions by plugging in style-specific parameters to include new styles into the existing model. We achieve this capability via a novel layer normalization layer design, which we will refer to as the Domain Layer Norm (DLN). Extensive experimental validation and user study on various stylish image description generation tasks are conducted to show the competitive advantages of the proposed model. The image description generation (IDG) problem concerns about generating a natural language description that transcribes an input image. Over the years, tremendous effort has been dedicated to developing models that are descriptive. However, little effort is dedicated to generating descriptions that are stylish (e.g. romantic, lyric, etc). Even for the handful of stylish IDG models that exist, they only have a loose control over the style. Ideally, a stylish IDG model should allow users to flexibly control over the generated descriptions as shown in Fig 1. Such a model would be useful for increasing user engagement in applications requiring human interaction such as chatbot and social media sharing. A naive approach to tackle the stylish IDG problem is to collect new corpora of paired images and descriptions for training. However, this is expensive. For each style that we wish to generate, we have to ask human annotators to write the romantic descriptions for each image in the training dataset. In this paper, we propose a controllable stylish IDG model. Our model is jointly trained with a paired unstylish * Equal contribution Copyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. A cat view himself as the king of the room, waiting for the servant to serve the food and water. He said: Food! Romance: A cat laying in a luggage bag on a bed with his beloved owner every day. She just wake up, waiting for the kiss from owner. Humorous: A cat laying in a luggage bag on a bed, thinking about to spend his life in that bag. But his owner is going to kick him out. Lyrics: Funny cat scenes, on the front of my screen. Hey, my video on my table you know, you must know. A cat laying in a luggage bag on a bed image description corpus (source domain) and a monolingual corpus of the specific style (target domain). In this setting, our model can learn to generate various styles without collecting new paired data in the target domain. Our main contribution is to show that the layer normalization can be used to disentangle language styles from the content of source and target domains via a small tweak. This design enables us to use the shared content to generate descriptions that are more relevant to the image as well as control the style by plugging in a set of style-specific parameters. We refer this mechanism as Domain Layer Normalization (DLN) since we treat each style as the target domain in the domain transfer setting. We conduct an extensive experimental evaluation to validate the proposed approach using both subjective and objective performance metrics. We evaluate our model on four different styles, including fairy tale, romance, humor, and country song lyrics style (lyrics). Experiment results show that our model generates stylish descriptions that are more preferred by human subjects. It also outperforms prior works on the objective performance metrics. (Liu, Breuel, and Kautz 2017) to encourage the same image in two domains to be mapped into common latent code. While our method is similar to these works in high level, the discrete property of language required new model design. Language style transfer. Supervised learning can be used to generate various linguistic attribute (e.g., different sentiments and different degrees of descriptiveness), but it requires a significant amount of labeled data. Many recent works assume there exist a share content space and a latent style vector between two non-parallel corpora for unsupervised language style transfer. Shen et al. (Shen et al. 2017) propose an encoder-decoder structure with adversarial training to learning this space. Following the same line, Melnyk et al. (Melnyk et al. 2017 ) introduce content preservation loss and classification loss to improve the transfer performance. Fu et al. (Fu et al. 2018) propose to use a multidecoder for different styles and a discriminator to learn a shared content code. Zhang et al. (Ye Zhang 2018) also use similar structure by using shared and private encoderdecoder. In a recent work, Prabhumoye et al. (Shrimai Prabhumoye 2018) introduce to ground the sentence in translation model, then apply adversarial training to get the desired style. What differs us from prior works is that we require generated stylish descriptions to match the visual content. Moreover, the style transferred in our work is more abstract instead of explicit styles such as sentiment, gender, or authorship in previous works. Image description generation. Several works have been proposed to generate image descriptions by using paired image description data (Vinyals et al. 2015; Krause et al. 2017; Liang et al. 2017) . To increase the naturalness and diversity of generated descriptions, ) apply adversarial training approach to train an evaluator to score the quality of generated descriptions. Chen et al. (Chen et al. 2017) propose an adversarial training procedure to adapt image captioning style using unpaired images and captions. A new objective is proposed in (Dai and Lin 2017) to enhance the distinctiveness of generated captions. On the other hand, there exist a few works proposed to enhance the attractiveness and style of the generated descriptions. Zhu et al. align the book and the corresponding movie release to a story-like description of the visual content. However, this method does not preserve the visual content. Matthews et al. (Mathews, Xie, and He 2016) propose the switch RNN to generate caption with positive and negative sentiments, which requires word level supervision and might not be able to scale. Recently, Gan et al. (Gan et al. 2017b) investigate to generate tag-dependent caption by extending the weight matrix of LSTM to consider tag information. The following work StyleNet (Gan et al. 2017a ) explores to decomposes LSTM matrix to incorporate the style information. One key difference is that we leverage an arbitrary stylish monolingual corpus that is not paired with any image dataset as target corpus instead of using paired images with stylish ground truth. The most similar to our work is (Mathews, Xie, and He 2018), the major differences are that we do not exploit the language features such as POS tag of corpus and we do not pre-process the target corpus to make it similar to the source one. Our approach is end to end with minimal pre-process of target corpus. The goal of stylish Image Description Generation (IDG) is to generate a natural language description d T in space D T given an image I in the image space I. The style of the description is implicitly captured in the description space D T , where we use subscript T to emphasize the target style. There exist two settings for learning a stylish IDG model. In supervised stylish IDG, we are given a training dataset T ) is a pair of image and its target stylish description sampled from the joint distribution p(I, D T ). The goal is to learn the conditional distribution p(D T |I) using D so that we can generate stylish image descriptions for an input image. Unsupervised stylish IDG. In unsupervised stylish IDG, we are given two training datasets where S is referred to as the source domain which is typically unstylish. , where the corresponding images are not available. Hence, the learning task is considered as unsupervised. The goal of unsupervised stylish IDG is to learn the conditional distribution p(D T |I) using D S and D T . Unsupervised stylish IDG is an ill-posed problem since it is about learning the conditional distribution p(D T |I) without using samples from the joint distribution p(I, D T ). Therefore, learning an unsupervised stylish IDG function is difficult without leveraging some useful assumptions. However, under the unsupervised setting, training data collection is greatly simplified: one could pair a general image description dataset (e.g., the MS-COCO dataset (Lin et al. 2014) ) with an existing corpus of the target style (e.g., some romantic novels) for learning. A solution to the unsupervised problem could enable many stylish image description generation applications. Assumptions. To deal with the ill-posed unsupervised stylish IDG problem, we make several assumptions illustrated in Figure 2 . We first assume that there exists a latent space Z providing a common ground to effectively map to and from the image space I, the source description space D S , and the target stylish description space D T . From latent space to description space, we assume that there exists a source description generation function G S (z) ∈ D S and a target stylish description generation function G T (z) ∈ D T . From non-latent space to latent pace, we assume that there exist an image encoder E I (I) ∈ Z and a target description encoder E T (d T ) ∈ Z. Our goal is to learn the generation functions (G T and G S ) and the encoding functions (E I and E T ) from the unsupervised stylish IDG training data D S and D T . Note that this is a challenging learning task if G T and G S is completely independent of each other. Hence, we assume that G T and G S share the ability to describe the same factual content but with different styles. Once these functions are learned, we can simply first encode the image I to a latent code using E I and then using G T to generate a stylish image description. In other words, the stylish image description is given by G T (E I (I)). We model the conditional distribution as p(D T |I) = δ(G T (E I (I))), where δ is the delta function. Inspired by the success of deep learning, we model both of the generation and encoding functions using deep networks. Specifically, we model E I using a deep convolutional neural network (CNN) (Krizhevsky, Sutskever, and Hinton 2012) and model E T , G T , and G S using recurrent neural network as illustrated in Figure 3 . We also use Skip-Thought Vectors (STV) to model E T . For G T and G S , we use Layer Normalized Long Short Term Memory unit (LN-LSTM) as their recurrent module (Ba, Kiros, and Hinton 2016; Hochreiter and Schmidhuber 1997). Training sketch. With the source domain dataset D S , we can train z S = E I (I) and d S = G S (z S ) jointly by solving the supervised IDG learning task, where z S is the learned latent representation in the source domain. On the other hand, with the target domain dataset D T , we can train jointly by solving an unsupervised description reconstruction learning task, where z T is the learned latent representation in the target domain. To ensure that the latent space is shared (i.e., z T ∈ Z and z S ∈ Z), we further assume that the generation functions G S and G T share most of their parameters. Domain Layer Norm. Specifically, we assume G S and G T share all the parameters except those in their layer norm parameters (Ba, Kiros, and Hinton 2016) . In other words, the domain description generators (G S and G T ) only defer in the layer norm parameters. We refer this weight-sharing scheme as the Domain Layer Norm (DLN) scheme. The intuition behind DLN is to encourage the shared weight to capture the factual content between two domains while the differences (i.e., styles) are captured in layer norm parameters. This design helps G T generate descriptions that are related to the image content even without the supervision of the corresponding images in training. Training E I and G S via Supervised IDG. The goal of supervised image description generation is to learn p(D S |I) by using D S . The G S consists of an embedding matrix θ W that maps input text x k to a vector e k , an LN-LSTM module, and an output matrix θ V that maps hidden state to predicted tokenŷ. Formally, where h k is the hidden feature in the LN-LSTM, k ∈ {−1 . . . m − 1} is time step of description with length m, and 1{} denotes the operator for one-hot encoding. To train the network, we minimize the sum of cross-entropy of correct words as follows, where x k is the k th word in the ground truth sentence. Training E T and G T via Stylish Image Description Reconstruction. The G T contains the LN-LSTM module, the same output matrix and embedding matrix used in G S . Formally, where d T is the target style image description. To train the network, we minimize the reconstruction error as follows, where d k T is the k th word in the target style image description. Relating G S and G T via Domain Layer Norm. We relate G S and G T by sharing all weights except layer norm parameters in the LN-LSTM. Details inside the LN-LSTM are shown in Fig 4, where the layer norm operation (LN) is applied to each gate of LSTM. Take the input gate as an example:î whereî k and i k are the normalized and unnormalized input gates, θ ie , θ ih are two projection matrices that map the embedding vector and the previous hidden state into the same Figure 2 : We make several assumptions to deal with the challenging unsupervised stylish image description generation problem. We first assume there exists a shared latent space Z so that a latent code z ∈ Z can be mapped to the source description space D S and the target stylish description space D T via G S and G T . We also assume there exists a stylish image description embedding function E T that can map a stylish description to a latent code. Finally, we assume there exists an image embedding function E I that can map an image to a latent code. Once these functions are learned from data, we can generate a stylish image description for an image by applying E I and G T sequentially. But at last the time came for them to go back to England, The king, with his wise men and brave knights, set sail early in the day dimension. The LN operation converts any input a to a normalized outputâ as follows, where a i denotes the i th entry in the vector a, p h is the dimention of the input a, µ and σ are the mean and standard deviation of the input a, g and b are scaling and shifting vectors (i.e., layer norm parameters) learned from the data. We train the whole network by jointly minimizing the supervised IDG loss L S and the unsupervised image description reconstruction loss L T subject to the architectural constraint set to G S and G T as below, where λ is a hyperparameter. Extension to New Target Styles. Given a model with parameters θ V , θ W , θ E I , and θ G S , pre-trained on a pair of the source and one target domain, we aim to adapt it to a new target domain (i.e, style) by enlarging θ V and θ W to θ V and θ W to accommodate new vocabulary and finetuning the remaining parameters to θ E I , θ E T , θ G S and θ G T . Hence, we define a new loss function as: where λ 1 and λ 2 are hyperparameters. The regularization ) cannot be applied. It is also not suitable to calculate these metrics between stylish sentences and the unstylished ground truth because the goal of stylish description generation is to change the word usage while preserve certain semantic relevance between the stylish description and images. We propose content similarity to evaluate the semantic relevance between generated stylish sentences and the unstylished ground truth. To calculate content similarity, we define C S as the set of nouns in the ground truth (source domain), and C S as the union between C S and synonyms for each noun in C S , for the model may describe the same object with different words (e.g., cup and mug). Similar logic is applied to C T and C T in the generated description (target domain). We calculate: We take the f-score of the p and r as the content similarity score. The overall content similarity score is averaged over the testing data. This is because we assume stylish descriptions should at least contain objects which appear in the image. We also report SPICE (Anderson et al. 2016) score, which calculate the f-score of semantic tuples between untylished ground truth and the generated stylish descriptions. The final score is average over all testing data. Metrics of stylishness. We use transfer accuracy to evaluate the stylishness of our generated description. The transfer accuracy is widely used in language style transfer task (Shen et al. 2017; Melnyk et al. 2017; Fu et al. 2018) . It measures how often do descriptions have labels of target style on test dataset based on a pre-trained style classifier. We follow the definition of transfer accuracy in (Fu et al. 2018) , which is where s is the output probability score of the classifier. We define R T = Nvt Nvs as our transfer accuracy, which is the fraction of number of testing N vs data in source domain and number of testing data that correctly transfer description with target style N vt . The final score is average over all testing data. Human evaluation. The difficulty in generating stylish sentence in unpaired setting is to remain semantic relevance. Therefore, we conduct a human study on Amazon Mechanical Turk (AMT) independently for each methods to judge the semantic relevance between image and description. For each model, we randomly sample 100 images then generate stylish descriptions for each style. Two workers are asked to vote the semantic relevance with following prompt: Given an image and a paragraph from the book (Our stylish corpus), how well does the paragraph content relate to objects in the image. Workers are forced to vote from unrelated to related. The criteria for eligible workers are having at least 100 successful HITs with 70% acceptance rate. The total number of HIT is 2400. For each HIT, the order of options is randomized. Workers are forced to vote and all responses are counted without aggregation. The result of the first experiment is summarized in comparison. It is worth noting that the perfect transfer accuracy may not be the best since the model could greedily generate the vocabulary used in the target domain and digress from the image content. Therefore, an ideal stylish description is the one with the high content similarity score and an acceptable transfer accuracy. Our DLN consistently outperforms other baselines in term of all semantic related metrics with a marginal drop of transfer accuracy on most datasets. All baselines are better than Random, which suggests all baselines can generate semantic-related description to certain degree. We observe NST has large n p and n r in fairy tale. We think this is because NST tends to generate long sentences. For each style (Fairy, Humor, Romance, and Lyrics), the average sentence length of NST is (119, 109, 103, 84) while that of DLN is (38, 54, 41, 97). Therefore, it is possible that NST generates more nouns in the unstylish ground truth. We also report the performance of DLN and DLN-RNN on unstylish description generation task in Table 2 . We calculate the BLEU-4, METEOR and CIDEr scores between generated sentences and unstylished ground truth. Combined with the result of stylish description generation in Table 1, we can conclude that the proposed domain layer norm can benefit the unpaired image to stylish description as we have a better model in conventional image to text generation. The result of human study is shown in Fig 5, we report the best of our model in Table 1 shows that the description generated by DLN is related to images. Note that the goal of generated stylish description is not to match every factual aspect of images, it should better be judged whether the description is related to the image if the image appears in the target corpus. Multi-style. We progressively expand DLN to include three target domains (fairy, romance, lyrics) to demonstrate the flexibility of our model. In other words, we follow Eq 17 to train source and fairy tale style then include romance and lyrics style, which is denoted as DLN-Multi. To generate the description, we use the same target decoder with a different style-specific embedding matrix, layer norm parameters, and output matrix. We conduct another human study by asking five workers to determine the best description given following priorities: content, style, and naturalness. This prompt forces workers to choose the better one if the two options are equally related to images. We sample 100 images for each and use the same criteria to select workers. The result is presented in Table 3 , which shows the performance of DLNMulti is competitive to DLN. DLN-Multi thus gives users the capability to include new style into the existing model, which is a novel feature not reported in other baselines. Discussion: transfer accuracy and domain shift. We observe a drop in transfer accuracy on the source to humor transfer in DLN, and we believe this is related to the scale of domain shift. To quantify this, we analyze the percentage of shared noun between the source (V src = 6.2k) and target domain, which are (50%, 68%, 74%, 60%) for lyrics, romance humor and fairy tale. For the transfer from the source to humor domain, the shared nouns account for over 70% nouns in the source domain, which means the domain shift between the source and humor is smaller than others. This makes it more difficult for the classifier to distinguish two domains. Therefore, the transfer accuracy of the source to humor is lower. We note Random get lowest transfer accuracy in lyrics style and we believe this is because sampling word from the vocabulary of lyrics alone cannot have sentences with new line symbol (i.e. ;), which is an important Table 3 : Result of DLN and DLN-Multi. CS, S, T and P are content similarity, SPICE, tansfer accuracy and human preference score. Overall, the performance of DLN-Multi is competitive to DLN in all metrics. feature for being classified as stylish. We propose a novel unsupervised stylish IDG model via domain layer norm with the capability to progressively include new styles. Experiment results show that our stylish IDG results are more preferred by human subjects. We plan to invesitgate the intermediate style generated by interpolation of domain layer norm parameter and address the fluency of generated sentences in the future.  The content of this supplementary material is summarized as below: • Data statistic • Implementation details of baselines and DLN • Interface of human evaluation • More qualitative examples For romance and humor data, we randomly sampled 50000 passages from BookCorpus . For fairy tale, we crawled from the website 1 and also sample 50000 passages from it. For the country song lyrics, we use the data released by Kaggle 2 and use all country song lyrics as our corpus. The Fig 7 shows the word usage of each corpus. • Neural Story Teller : NST contains two separate modules: image to caption in the source domain and stylish story decoder in the target domain. The first module is used to extract the source domain textual representation of the image. They use pre-trained image caption alignment (Kiros, Salakhutdinov, and Zemel 2014) on MS-COCO to extract the top N neighbor captions for an image. The textual representation of the image is calculated by averaging the skip-thought encoded vector of top N captions. The second module is used to reconstruct target domain textual representation to original stylish story passage, where a decoder is trained to reconstruct the textual representation of stylish story passage to the original stylish story passage. The textual representation here is also the skip-thought encoded vector. To generate the description for given image, it first subtracts the mean of all skip-thought encoded MS-COCO caption and added the mean of skip-thought encoded stylish story passage to transform the source textual representation to the target one; then, feed it into the second module to generate the stylish story. • StyleNet (Gan et al. 2017a): We re-implement StyleNet as one of our baseline. The StyleNet is based on factorized LSTM, which factorized the weights mapping inputs to hidden representation to θW = U SV , where U ∈ R pe×pm , S ∈ R pm×pn and V ∈ R pn×p h , where pe,p h are dimension of embedding and hidden size, and pm,pn are the dimension of the matrix. It contains a language model trained on target corpus and a source image to caption decoder where each factorized LSTM is associated with a style matrix Ss and St. During inference, it replace the source style matrix with target style matrix and generate descriptions. NST. We follow the original setting and implementation 3 and train the decoder on our corpus till converge. We set the maximum length of text data equals to 100. DLN. We use the same text length as used in NST. We use most common 10000 vocabularies in the source domain. For target domain, we use 10000 vocabularies for lyrics and 15000 vocabularies for romance, humor and fairy tale corpora. 1 http://www.loyalbooks.com 2 https://www.kaggle.com/gyani95/380000-lyrics-frommetrolyrics 3 https://github.com/ryankiros/neural-storyteller We use skips-thought vector released by Tensorflow (Abadi et al. 2016) as our ET . We follow the original NST implementation to stack uni-skip and bi-skip skips-thought vector to get 4800 dimension feature for text. We use pre-split training set in VGPara (Krause et al. 2017) . We use ResNet50 from Keras 4 as our EI . The dimension of our latent space is 620, which is the same as our word embedding dimension. In implementation, we fix EI and ET and append projection matrix θP I and θP T as the last layers. During training, we only update projection matrix. We initialize all weight matrix by uniform initialization. The number of hidden units used in LN-LSTM is 1000, and we optimize our model by Adam optimizer (Kingma and Ba 2015) with start learning rate as 0.001 and decayed factor as 0.5 every 80. We follow NST to use gradient clipping = 5 in DLN. The training epoch is 100 with batch size as 64. We choose λ = 0.5 for DLN training. DLN-Multi The vocabulary size of each domain is 5500. We choose λ1 = 0.2 and λ2 = 0.1. Other settings and hyperparamters are the same as DLN except the regularization term We implement the θ W by θ W = θW ||θ∆W , where || is the concatenation operation of matrix and θ∆W is the new vocabulary used in the new style. In the subtraction, we only subtract the θW part in θ W to match the dimension of matrix. Similar logic can be applied to θ V . We use the projection weight θP I learned during pre-training and the weight during the training of DLN-Mutli as our θ E I and θE I . StyleNet To train StyleNet, we use the same hidden unit of LSTM as DLN and follow the iterative training method reported in the original paper except that we also update the share weight when training language model on target corpus, which we found this modification has better convergence in our task as shown in Fig 8. We also apply our decay learning setting, which we found has faster convergence. During inference, we follow StyleNet and NST by using beam search with a beam width of 5 and the unknown token for all methods. Pretrained classifier in transfer accuracy. For the classifier used in evaluation, we use convolutional network proposed in (Kim 2014). We train the classifier to achieve over 99% accuracy to distinguish DS and DT . We performed two human evaluation tasks using the Amazon Mechanical Turk 5 platform. The first was a relevance task, asking how well does descriptions relate to the image content on a four level scale. We provide screen-shots of the instructions given to workers in Fig 9. The second study aims to compare the attractiveness of descriptions generated by the DLN mode and DLN-Multi. Fig 9 is the screen-shots given to works for this experiment. To ensure reliable results and avoid workers who choose randomly, only workers with more than 70% accuracy and 100 successful HITs previously are allowed to attend the study. In the second human study, we also provide a dummy text as trap option to monitor the labelling quality. The result shows almost no trap options are chosen, indicating experiment results to be reliable. We demonstrate more qualitative examples of stylish image description generated by DLN in Fig 11 . This is exactly what he was playing on his tennis court. He felt an overwhelming compassion for him, and when you hit him. Weakly related Please evaluate the relevance between the image and paragraph below You are given an image and a paragraph from novels. Please evaluate how well does the paragraph content relate to objects that appeare in the image. For example, if there are houses and dogs in the image, words like dog, house, or other semantic related nouns(Ex. animal, cat, building, apartment, home) are considered relating to the image content. Meaning of each options are explained below. Related: Most nouns in the paragraph are related to the image content. Partly related: About half nouns in the paragraph are related to image content. Weakly related: A few nouns in the paragraph are related to image content Unrelated: None of nouns in the paragraph are related to image content. Figure 9: A screen-shot of human study on semantic relevance of generated paragraphs. Fairy tale Image to be described: Sed ut perspiciatis unde omnis iste The clock tower made very much like Mary, and hardly had taken so much of it that there would be soon after all the fairies began to be most anxious for all that. There's a church at the market, which, from the house, the rooms of the cities; and the other time, two and people, every day is done;. Paragraphs below should describe the given image with a specific style (e.g., tell a romantic story based on wedding ceremony picture). Please choose the most attractive paragraph based on the following PRIORITIES: 1. Content: How well does the paragraph describe objects appeared in the image. For example, if the dog and house appear in the image, choose the paragraph which contains more words like dog, house, or other semantic related words(Ex. animal, cat, building, apartment) 2. Style: How well does the paragraph match the specific style. 3. Naturalness: How similar the paragraph is to human language. More fluent and more grammatically correct paragraph is better.
REF propose to extract and control the style of the image caption through domain layer normalization.
Unsupervised Stylish Image Description Generation via Domain Layer Norm
ArXiv
ArXiv
Computer Science