The current popular language models learned from largescale data sets still cannot well represent abstract attributes. Table 1 shows the top 10 nearest words of abstract attributes acquired using a typical language model (LM) learned from large-scale textual data [1] . We can see a textual LM often fails to represent these abstract attributes. In contrast, human can illustrate abstract concepts at early age visually. For example, Figure 1 shows various children's drawings about the concept "family", by using sun and smiling faces to represent happiness, and height to distinguish parents and children.
We aim to teach computer to automatically illustrate tex- tual attributes by generating portraits. Portrait is a special genre in painting, where the goal is to present not only the outward appearance of a specific human subject, but also their inner significance inspired by admiration or affection for the subject. A good portrait needs to be realistic, and thus it's important to take a photo of the subject as input. Some recent attempts [2, 3, 4, 5] try to automate the conversion from photo to portrait. Most of these methods are only based on visual style transfer [2, 3, 6] . However, art works reflect not only the artist's hard work and dexterous technique but also often carry their personal emotions and memories on the subjects, due to the intimate relationship between the artist and the subject, either bound together before or during the painting. As Aristotle stated, "The aim of Art is to present not the outward appearance of things, but their inner significance; for this, not the external manner and detail, constitutes true reality." [7] . From a good portrait, we can often reveal the story of the subject's life, such as occupation, hobby, personality, mood, or a special occasion which may even involve the artist, from a certain facial expression, hairstyle, or the artist's clever use of colors and lines. ject with 11 text attributes, as shown in Table 2 , including mood, facial expression, gender and hairstyle. We design a new portrait generation framework called MUSE, which takes these inspirational textual attributes in addition to face regions as input for portrait generation. We first feed the extracted face from the input photo into the encoder part of a UNet [8] , a convolutional network architecture that has show promising results on image generation [9, 10] . Then we directly feed the textual attributes into an attribute encoder. Finally the textual attribute embeddings are integrated with the hidden representation of the input photo as an input to the attribute-based decoder to generate the portrait. Figure 2 illustrates an example where the facial expression and hair style are automatically changed based on a set of attributes expressed in text.
Moreover, existing evaluation metrics such as Inception Score (IS) [11] only check the overlapped visual content between the machine-generated and human-generated portraits. But this is not how human assessors approach and appreciate a painting. We design a novel metric to evaluate how many text attributes human assessors can reconstruct from the system generated portraits. Experiments show that our method outperforms state-of-the-art on all measures, and portrait generation is an effective way to acquire and illustrate textual attributes.
In summary, the main contributions of this paper are as follows:
• We propose the first inspiration-to-portrait generation framework MUSE that takes text description of attributes into account to generate portraits aligned with its background story such as underlying emotions.
• We develop a novel neural network architecture incorporating textual attributes for portrait generation. Rather than using a binary sequence of facial attributes as input, we apply attribute embeddings, which are initialized from portrait data and optimized during training. Instead of preserving the attributes of input photos, we design a novel discriminator to encourage diversity and realism.
• We create a large portrait generation data set containing 3,928 photo-portrait pairs with manually annotated attributes as a new benchmark, along with 51,939 portraits without annotations, and will share the resources with the community for further exploration.
• We propose a novel evaluation metric based on human attribute reconstruction to better assess the quality of generated portraits. The generator G is an attribute-aware UNet that incorporate attribute embeddings in hidden representation. The discriminator D is a stack of convolutional layers followed by fully connected layers. By using the adversarial loss function, La, and the attribute classification loss functions, Lc, the discriminator D can not only recognize the realness of the generated portraitŷ but also classifyŷ into different class of each attribute.
MUSE takes two sources of input to generate portraits: (1) a photo x of the subject; (2) a set of textual attributes in form of type-value pairs
MUSE contains a generator G and a corresponding discriminator D for adversarial training. To demonstrate the importance and effectiveness of textual attributes, we modify a state-of-the-art image generator, UNet [8, 10] , to incorporate attribute embeddings v into generation. The discriminator D takes the system generated portraitŷ together with its corresponding human generated portrait y to evaluate the generator G. The overall architecture of MUSE is depicted in Figure 2 .
We propose 11 textual attributes to represent the inspirations, as shown in Table 2 . We select these attributes from various online resources including the LitCharts Library, ClarkandMiller.com, and ManyThings.org.
We assign each type-value pair
Given a set of n attribute type-value pairs
as input, we concatenate embeddings for n pairs to get the input attribute embedding
The attribute values in Table 2 are correlated. For instance, we find that the clothing under rainy weather is more likely to be blazer or coat. To capture such interdependency between multiple attribute values, we initialize v i by using domain-specific attribute embeddings trained from portrait data, where we use skip-gram methods in Word2Vec [12] and consider attribute values corresponding to the same portrait as a bag of words in one context window.
UNet [8, 10] is a high-quality image-to-image generative model. Given an input photo x, UNet first encodes x into a hidden representation h ∈ R d h = G enc (x) using multi-layer Convolutional Neural Networks (CNNs), which is further decoded into an output imageŷ = G dec (h) using a stack of transposed convolutional layers.
Our generator G employs the UNet architecture. The input photo x is first encoded into the hidden representation h. Then we aggregate the hidden representation h and attribute embeddings v of the expected portrait as
σ is an activation function and we use ReLU [13] . We decode the portrait asŷ = G dec (h a ). For simplicity, we use G a enc (x, v) to represent h a .
We apply adversarial training simultaneously for the generator and discriminator (G, D) to learn the mapping from the input photos X and input attribute embeddings V to the output portraits Y . Given training samples
where the input photo x (i) ∈ X, input attribute embeddings v (i) ∈ V and the human generated portrait y (i) ∈ Y , we denote the data distribution as x ∼ p data(x) , v ∼ p attr and y ∼ p data(y) . While G tries to generate realistic portraitsŷ similar to the portraits y in Y domain, D tries to distinguish betweenŷ and y.
We compute the adversarial loss following GAN [14] , with input photo x, attribute embeddings v, and y as corresponding human generated portraits rescaled to the same size as the outputs of G. Specifically, the adversarial losses of the generator G i and discriminator D are formulated as follows:
In addition to the adversarial loss L G a , we further use L1 distance to force the generator not only to generate realistic portraits to fool the discriminator but also get close to the human generated portrait. The L1 loss can be obtained as follows:
While the adversarial learning is employed on the system generated portraitŷ to ensure its visual reality,ŷ is also expected to correctly contain the desired attributes v. Hence, an attribute classifier F is used to constrain the system generated portraitŷ with v. Letṽ denote the one-hot vectors of v, the loss of the attribute classification L c can be obtained as follows:
where ρ is the summation of binary cross-entropy losses of all attributes as follows:
Overall, by combining the adversarial loss and the attribute classification loss, the final objective functions of the generator G and the discriminator D are as follows:
where λ 1 , λ 2 , λ 3 are hyper-parameters that control the relative importance of the losses.
In the training phase, we use photo-attribute-portrait training samples {(x (i) , v (i) , y (i) )} N i=1 to train our model.
The generator G takes x and v as input and generate the portraitsŷ. The discriminator D learns to distinguish real and generated samples by taking the generated portraits from G and the human generated portraits y, respectively. In the testing phase, the photo-attribute pairs are fed into G to generate portraitsŷ. [23] (26,351 portraits) without attribute annotations, and included them in our released benchmark, for future work on learning visual features for various genres and potentially with less human supervision. Baselines. The first baseline is a simple implementation of an image-to-image style transfer model. The input photos are transferred into portrait style by using a UNet-based generator and a discriminator is used to play against the generator. We also compare with two state-of-the-art image generation methods, StarGAN [24] and AttGAN [9] . Experiment Setting. In our experiments, we set the epoch number to 600 and the batch size to 32. The images are augmented by rotating, normalizing, and flipping before training to encourage the generalization of the model. The learning rate is 0.0002 and the Adam optimizer (β 1 = 0.5, β 2 = 0.999) is used. The slope of leaky ReLU is 0.2 and dropout rate is 0.5.
A good portrait must attract our eyes in some way: its subject matter, it's use of color, an interesting juxtaposition of objects, its realistic appearance, or any number of other factors. We evaluate the general visual quality of a portrait using the standard metric Inception Score (IS) [11] based on ImageNet [25] predefined classes. Suppose V g is the collection of generated portraits from the last layer of the generator G M from the generation stack. We feed all generated portraits v ∈ V g into Inceptive-v3 networks [26] to obtain their conditional probability distributions p(s|v) over 1000 ImageNet [25] classes, where each class is denoted by s. Class distribution is then marginalized by assuming uni- form distribution of v ∈ V g , i.e.
IS score is the average KL-divergence between p v = p(·|v) and marginal distribution q(·),
A higher IS indicates the generated images are more realistic in the sense that their conditional distributions concentrate on a small subset of classes. Although IS is widely used, it does not compare the generated results with real samples [27] . Fréchet Inception Distance (FID) [27] is another popular metric for conditioned image generation, which measures the Fréchet distance between the generated image distribution and the real image (gold standard) distribution. Lower FID is better, indicating the generated results and target samples are more similar.
As shown in Table 3 , the proposed MUSE outperforms the baseline with the IS score increased by 6% and the FID score decreased by 11%, respectively. By taking attribute embeddings as additional input, the model can build the alignment between the input photo and target portrait more easily, making the hidden representation more meaningful to produce more reasonable composition. For example, the attribute value blond hair can provide clearer guidance of generating a particular hair color rather than some unpredictable behavior.
We propose a new metric to check if the generated portraits reflect the input attribute values, and further explore how attribute semantics are learned and grounded into generated portraits. Subject Attribute Reconstruction Accuracy. Since most of ImageNet classes are objects and the number of classes that appear in portraits is quite limited, the IS metric cannot evaluate the subtle details of human subjects in portraits. We use our attribute classifier F trained on the portrait data set to classify each generated portrait. We compute the Fscore of the estimated attributes against the textual attributes extracted from the human generated portrait. Table 4 shows the attribute reconstruction accuracy. We can see that our approach successfully illustrates 78% attributes. Single Attribute Coherence We first consider a simpler scenario where we consider changing only one attribute.
We train a separate model for each attribute, for which we constrain the attribute embeddings v to contain only the specific attribute (e.g. hair) during training. Figures 3  to 6 show example outputs of the proposed MUSE compared with StarGAN [24] and AttGAN [9] by modifying hair, face expression, mood and gender respectively. We can see our models have successfully changed these attributes and learned to generate according to attribute values rather than performing some random behav-ior. On the contrary, the StarGAN mistakenly interprets the attributes and AttGAN does not learn from the attributes at all, due to the small amount of training data and lacking of the mechanisms for cross-domain (photo to portrait) image transformation.
Multiple Attributes Here we consider a more complex scenario that we change all attributes for portrait generation. Figure 7 shows examples using a combination of 11 attributes listed in Table 2 . We can see MUSE can capture the explicit attribute values such as blond and black hair, and capture the abstract concepts such as stormy vs sunny and cold vs hot by adjusting the background darkness levels.
Grounding of Attributes We further examine how the model ground attributes into generated portraits, by taking an demonstrative example of the hair attribute as shown in Figure 8 . From these results we observe that MUSE can rely on either face outlines or facial features such as eyes or mouths to estimate the relative position of components including hair and facial area. In the fourth column we design a blank face with hair outline. Although MUSE cannot perfectly color the hair area, we do observe the model tends to color along the lines. We also show in the last column that MUSE works in a similar way for faces in profile.
Inter-dependency of Attributes As discussed in section 2.1, the attribute values in Table 2 are correlated.
Here we show some examples of two related attributes, weather and mood, in Figure 9 . For example, the happy portrait in the same row is associated with a brighter background implying a sunny weather. We also use the pretrained classifier F to quantitatively evaluate the interdependency between weather and mood. Given the happy portraits, the predicted probabilities to be sunny and stormy are 38.28% and 0.22%, respectively. Similarly, given the sunny portraits, the predicted probabilities to be happy and sad are 39.20% and 0.31%, respectively. Affordances of Attributes Not all the compositions of attributes are semantically valid. For example, it is not possible to have "smile" face expression and "sad" mood in the same portrait. Although it is challenging to inform a model with such affordances, we can use visual signal to improve the generalization capabilities of the model since invalid combinations will not appear in visual domain. Table 5 shows the attribute reconstruction accuracy, IS score and the FID score of the generated "smile sad" and "smile happy" portraits. Using a valid attributes combination, the model can generate a portrait with better quality and better reconstruction accuracy.
Good art works should reflect not only explicit information of the subject but also implicit attributes such as personality, occasion, occupation, nationality, hobby. How- 
Generative Adversarial Networks (GAN) [14] achieve great success in generating realistic images without much control [28] . Image-to-image generation [29, 3] takes an image as input condition, and transfers it into another image in a different domain, including satellite-to-map generation, sketches-to-shoes generation, night-to-day generation and Landscape-to-Photo generation.
Another line of work takes natural language as input condition and generates images accordingly. [30] first introduces a conditional DC-GAN architecture which achieves positive results for generating low-resolution images (64 × 64), but it is not equally successful in higher resolution image generation. To address this problem, Zhang et al. [31] propose Stacked Generative Adversarial Networks (Stack-GAN) that first generate low resolution images and make refinements thereafter. [32] further proposes hierarchicallynested losses that refines images in multiple steps. [33] uses dual inference over conditional and unconditional latent variables for disentanglement of content and style. Rather than starting from low-resolution image generation, [34] first constructs semantic layouts from text and generates images based on them. Despite promising results, the above methods only incorporate sentence-level features without considering fine-grained attributes and thus yield unsatisfactory results when the input sentences are complex.
Xu et al. [35] proposes Attentional GAN (AttnGAN) to enable the generative networks to be trained on words of higher relevance, and develops an inter-modal attention mechanism to compute the similarity between the generated image and the relevant text description. Recent methods [36, 37] incorporate attention mechanism to improve semantic consistency. They rely on general caption-type instructions to generate images of flowers [38] , birds [39] or common objects [40] . However, such instructions usually lack of identity information, which makes these models impossible to generate images of a specific person, flower or bird. In contrast, we propose to take both photo and textual attributes as input. It is worth noting that although AttGAN also leverages facial attributes when generating images, our model is different from the following aspects: 1) our goal is to generate creative and abstract portraits instead of generating photo-realistic images. 2) we apply more complex and abstract attributes to guide the portrait generation. For each attribute type, we include more values and some of them have shared semantics. Also our attributes are carefully designed for portraits and are more abstract such as mood and weather. In addition, we use trainable attribute embeddings to better represent the inter-dependency between multiple attribute values. [41] describes an interesting user study to evaluate the results of generating animations from screenplays where users are asked to evaluate, on a five-point Likert scale [42] , if the video shown was a reasonable animation for the text, how much of the text information was depicted in the video and how much of the information in the video was present in the text. Our attribute reconstruction metric aims at a similar goal, but we compute the scores based on pre-defined attribute categories and thus our metric is more objective.
We have developed a novel method, MUSE, which can precisely illustrate textual attributes using portrait generation. In the future we plan to extend MUSE to unstructured text descriptions and apply open-domain attribute extraction techniques to extract as input, and extend it to cover a wider range of entity types and attribute types.