Paper:
Image-to-image (I2I) translation aims to translate images from one domain to another. To tackle the multi-modal version of I2I translation, where input and output domains have a one-to-many relation, an extra latent input is provided to the generator to specify a particular output. Recent works propose involved training objectives to learn a latent embedding, jointly with the generator, that models the distribution of possible outputs. Alternatively, we study a simple, yet powerful pre-training strategy for multi-modal I2I translation. We first pre-train an encoder, using a proxy task, to encode the style of an image, such as color and texture, into a low-dimensional latent style vector. Then we train a generator to transform an input image along with a style-code to the output domain. Our generator achieves stateof-the-art results on several benchmarks with a training objective that includes just a GAN loss and a reconstruction loss, which simplifies and speeds up the training significantly compared to competing approaches. We further study the contribution of different loss terms to learning the task of multi-modal I2I translation, and finally we show that the learned style embedding is not dependent on the target domain and generalizes well to other domains.

Review:
In this paper, the authors tackle the problem of multi-modal image-to-image translation by pre-training a style-based encoder. The style-based encoder is trained with a triplet loss that encourages similarity between images with similar styles and dissimilarity between images with different styles. The output of the encoder is a style embedding that helps differentiates different modes of image synthesis.  When training the generator for image synthesis, the input combines an image in the source and a style embedding, and the loss is essentially the sum of image conditional GAN loss and perceptual loss. Additionally, the authors propose a mapping function to sample styles from a unit Gaussian distribution.
I think the idea of pre-training a style-based encoder is straightforward. I am mainly concerned about the performance of the presented approach. First, there are no many visual comparisons in the paper. The only visual comparison is in Figure 8, but results are only limited to faces. The visual results in Figure 5 do not look appealing to me. The change in the style mainly comes from the global change in color: no much change in the texture or local color. The "night2day" results look poor to me.  I am concerned about the diversity of the styles learned in the model.
On the other hand, I am convinced that the proposed model is better than BicycleGAN, and the approach is somehow novel. The user study in Table 5 suggests that the proposed method is somehow better than BicyleGAN in visual quality on one task. My overall rating is borderline.
Minor comments:
- In the first sentence of Section 3.2, I do not think "one-to-one correspondence" is the right description. The encoder is not expected to be invertible.
- In Equation (3), "e_i" is a little bit misleading. It does not mean the i-th element in {"e_j"}. You may want to replace "e_i" with "s_i" to avoid confusion. 
- The explanation of ours v1, v2, v3, v4 is not clear. It is also difficult to find its definition.