L OG H YPERBOLIC C OSINE L OSS I MPROVES V ARIATIONAL A UTO -E NCODER
Anonymous authors
Paper under double-blind review
A BSTRACT
In Variational Auto-Encoder (VAE), the default choice of reconstruction loss function between the decoded sample and the input is the squared L 2 .
We propose to replace it with the log hyperbolic cosine (log-cosh) loss, which behaves as Lat small values and as L 1 at large values, and differentiable everywhere.
Compared with L 2 , the log-cosh loss improves the reconstruction without damaging the latent space optimization, thus automatically keeping a balance between the reconstruction and the generation.
Extensive experiments on MNIST and CelebA datasets show that the log-cosh reconstruction loss signiﬁcantly improves the performance of VAE and its variants in output quality, measured by sharpness and
FID score.
In addition, the gradient of the log-cosh is a simple tanh function, which makes the implementation of gradient descent as simple as adding one sentence in coding.
I NTRODUCTION
Unsupervised generative modeling aims to represent probability distributions over input data.
The explicit approach usually employs certain parametric model to estimate and approximate the unknown data distribution p data , but the effect is sensitive to the model selection, for which expert knowledge is often needed.
In a different vein, implicit generative models aim to generate samples from the data distribution without estimating the distribution itself.
This greatly weaken the dependency on the model selection, and ﬁnds numerous applications such as simulating possible futures in reinforcement learning Finn & Levine (2017), predicting the next frame of a video sequence Lotter et al.
and image super-resolution Ledig et al.
Prominent examples of implicit generative models include Boltzmann machines Aarts & Korst (1988), Generative Adversarial Network (GAN) Goodfellow et al.
and
Variational Auto-Encoder (VAE) Kingma & Welling (2013).
Among these methods, VAE enjoys an efﬁcient and stable training process, and the encoder component of VAE gives dimension reduction and feature learning as a by-product.
Meanwhile, it is also observed that images generated by VAE are often blurry.
One potential reason is that the model fails to represent high-dimensional probability distributions accurately, which is the target issue we aim to address in this paper.
1. V ARIATION A UTO -E NCODERS
We now give a very brief account of VAE.
In VAE, there is an encoder that converts the real input data x to a latent code z ∼ q φ ( z | x ) , and a decoder that reconstructs an image ˆ x ∼ p θ ( x | z ) from the latent code z , where q and p are encoder and decoder, with parameter φ and θ respectively.
VAE has two objectives: one is to match the decoded samples ˆ x to the input x , and the other is to maintain the posterior q φ ( z | x ) of the hidden code vector z to a given prior distribution p ( z ) .
Combining these two, VAE uses the following loss function for datapoint x .
[Equation]
Here the ﬁrst term is the reconstruction loss, which reduces to the squared L 2 loss (cid:107) x − ˆ x (cid:107) 22 if we assume that the decoder predicts a Gaussian distribution at each pixel i.e. p θ ( x | z ) =
N (ˆ x, σ 2 I ) .
The second term of the objective matches the distribution of latent code to a prior p ( z ) , which by default is a standard normal distribution.
1. R ECONSTRUCTION LOSS
To improve accuracy of representing probability distributions, many variants of VAE have been proposed.
Much attention has been paid to improving the design of the second term, for example, changing the prior on the latent code (e.g. S -VAE Davidson et al.
(2018), VAE with householder ﬂow Tomczak & Welling (2016)), and changing the loss between the latent distribution and the prior (e.g. AAE Makhzani et al.
, WAE Tolstikhin et al.
An alternative approach for improvement is through the reconstruction loss, i.e. the ﬁrst term in
Eq.
(1), which measures the error between the decoded sample and the input data points.
Although the squared L 2 loss is widely used as reconstruction loss in VAE setting, it has long been observed that this loss function results in problems such as blurry images Mathieu et al. (2015).
Several new metrics are proposed.
Ridgeway et al.
(2015) applied the structural-similarity (SSIM) index Wang et al.
as a reconstruction metric of an auto-encoder.
Larsen et al.
(2016) employs a GAN discriminator to compute the reconstruction loss in VAE.
Dosovitskiy & Brox (2016) combined a GAN discriminator with the perceptual loss which computes distances between image features extracted by deep neural networks.
These experiments demonstrate the potential of improving VAE by re-designing the reconstruction loss.
However, disadvantages are also evident:
The ﬁrst one
Ridgeway et al.
(2015) only demonstrated results on gray-scale images, which are very blurry.
The last two Larsen et al.
; Dosovitskiy & Brox (2016) reported that their produce images suffer from severe high-frequency artifacts, possibly due to feature loss as discussed in Mahendran &
Vedaldi (2015).
In addition, as other GAN-based methods, introducing a discriminator makes the training harder and instable.
In this paper, we propose an element-wise reconstruction loss based on a simple log-cosh function
[Equation]
where a ∈ R + is a parameter and log is the natural logarithm.
The plots of f ( t ; a ) in interval [ − 1 , 1] for different a are shown in Fig.
The function is close to L 1 for large | t | and close to L 2 for small | t | , thereby combining the smoothness advantage of L 2 , and robustness and image
sharpness advantage of L 1 .
In addition, the derivative of the function is simply the sigmoid function, rendering a very efﬁcient training and very simple implementation.
In Sec.
4 we show empirically that even if the model trained with the log-cosh reconstruction loss achieved a larger error measured by L 2 loss, it signiﬁcantly outperforms the model trained with squared L 2 loss.
The result also indicates that squared L 2 loss can be ineffective in evaluating high-dimensional data like images, which will be discussed in more details later.
[Figure]
L IMITATIONS OF TRADITIONAL LOSS
Given two vectors x and ˆ x , the squared L 2 loss and the L 1 loss are deﬁned as
[Equation]
[Equation]
respectively, where x i stands for the i -th entry of x .
The loss functions also extend to matrices by treating them as long vectors obtained from concatenating the rows.
When applied in the VAE setting, x and ˆ x are the input of encoder and output of decoder, respectively.
As mentioned in the last section, the squared L 2 loss widely used in VAE comes naturally from multivariate Gaussian distribution p θ ( x | z ) ∝ exp( − (cid:107) x − ˆ x (cid:107) 22 ) .
Other than the aforementioned issue of causing blurry images, the squared L 2 loss is also well known to be sensitive to large noise, and it has long been argued that in image space, it is ineffective in evaluating visual quality Wang &
Bovik (2009) and makes the training to easily get stuck in local minima Zhao et al.
If we choose a decoder with zero-mean Laplace distribution p θ ( x | z ) ∝ exp( − (cid:107) x − ˆ x (cid:107) 1 ) , we will obtain a loss function in terms of the L 1 norm.
The L 1 loss is used as image to image loss in some variants of VAE and GAN, such as AEGAN Rosca et al.
and
cycle GAN Zhu et al.
The gradient of L 1 loss is ± 1 at differential points.
The L 1 loss is generally robust to noise and, when used as a regularizer, encourages sparsity in solution.
It is found to be quite useful in computer vision tasks such as deblurring Cai et al.
; Freeman et al.
; Xu & Jia (2010), since it appears to enable the solver to escape from local minima, although the exact reason for this is still elusive.
In addition, Zhao et al.
(2017) shows that L 1 can outperform L 2 when directly used as a loss function for neural networks in image restoration applications such as denoising Jain & Seung (2009), deblurring Xu et al.
(2014), demosaicking Wang (2014), and super-resolution Dong et al.
However, an obvious disadvantage of the L 1 loss is that it is not differentiable if any entry x i − ˆ x i = 0 , causing oscillation between ± 1 during the training process.
In this paper, we will show that our proposed log-cosh loss takes the advantage of L 1 while overcoming the problem of non-differentiability.
Experiments show that the log-cosh loss outperforms both L 1 and L 2 loss.
L OG - COSH LOSS 3. P ROPERTIES OF LOG - COSH LOSS
Based on the log-cosh function shown in Eq.
(2), we propose to use the log-cosh loss as the reconstruction error, taking advantages of both squared L 2 loss and L 1 loss while overcome their limitations.
The log-cosh loss is deﬁned as
[Equation]
where a ∈ R + is a hyper-parameter.
We will ﬁrst show that the log-cosh function behaves like Laround origin and like L 1 at other points.
Proposition 1.
The log-cosh function f ( t ; a ) for a > 0 approximates | t | − 1 a log 2 when t → ∞ , and approximates 0 .
5 at 2 when t → 0 .
More precisely, f
[Equation]
a
[Equation] [Equation]
when |
[Equation]
|
[Equation]
when |
[Equation]
|
[Equation] [Equation]
Proof.
When | t | → ∞ , we have f ( t ; a ) =
a log(( e a | t | + e − a | t | ) / 2) → 1 a log( e a | t | / 2) =
| t |− 1 a log 2 .
When | t | → 0 , the Taylor expansion gives f ( t ; a ) = 0 .
5 at 2 + O ( t 4 ) .
For general t , one has bounds | t
[Equation]
The log-cosh loss can be viewed as a smoothed version of L 1 loss differentiable everywhere.
The gradient of f ( t ; a ) is
[Equation]
where tanh( t ) =
( e t − e − t ) /
( e t + e − t ) is the hyperbolic tangent function and σ ( t )
= 1 / (1 + e − t ) is the sigmoid function, a commonly used activation function in deep networks.
3. M OTIVATION OF APPLYING LOG - COSH LOSS TO VAE
The log-cosh loss applied to VAE can be understood in the following way, which was also how we obtained the method.
In general we hope to balance the reconstruction accuracy and latent space optimization in VAE.
If the squared L 2 loss is used for the reconstruction, then the reconstruction accuracy deteriorates in the region of small reconstruction error, as L 2 penalizes these small errors too lightly, making the objection function Eq.(1) to be dominated by the second term (the KL divergence term).
To solve this problem, simply increasing the relative weight of the reconstruction loss does not work well, since it harms the optimization in the latent space when the reconstruction error is large.
Putting both large-error and small-error regions into consideration, a feasible solution is to increase the weight of reconstruction loss when it is small, but at the same time to clip the gradient of the loss with an upper bound so that the reconstruction penalty does not always increase linearly with the error.
A smooth approximation of this is the tanh function, whose integral turns out to be the log-cosh function.
The resulting log-cosh loss becomes more sensitive to small errors because it behaves as a 0 .
5 a -scaled squared L 2 loss around the origin.
At the same time, it avoids penalizing too much when error is large when it behaves likes the L 1 loss.
Therefore, we are motivated to implement the log-cosh loss to improve the reconstruction accuracy of VAE without damaging its latent space optimization.
The implementation of the log-cosh loss in VAE is quite simple, as the gradient is explicitly available as shown in Eq.
In tensorﬂow the implementation is as simple as replacing the reconstruction loss in the back-propagation with
[Equation]
where x and ˆ x are the input of encoder and output of decoder, respectively.
This form also avoids overﬂow or underﬂow during the computation, as it directly combines the gradient of the log-cosh loss (w.r.t decoder samples) and the gradient of the samples (w.r.t parameters).
E XPERIMENTS
We empirically compare the log-cosh loss to the squared L 2 loss in this section, leaving the comparison to L 1 to Appendix B.
All the objective functions studied are explicitly given in Appendix A.1.
The experiments are run on two datasets: (1) MNIST LeCun et al.
(1998), which consists of 70k images of handwritten digits with size 28 × 28 , and (2) CelebA Liu et al.
(2015), which contains roughly 203k celebrity images cropped and resized to 64 × 64 as many previous works.
We quantify the performance of VAE using two measures.
The ﬁrst one is sharpness, for which every image is convolved with the Laplace ﬁlter (cid:16) 0 1 0 1 − 4 1 0 1 0 (cid:17) , and then the variance of the activations is computed as the sharpness score.
The second one is Fr ´ echet Inception Distance (FID)
Heusel et al.
(2017), for which the coding layer of a classiﬁcation model trained by us is used to extract features and
the
Fr ´ echet distance Fr ´ echet (1957) is computed between distributions of the layer obtained from two groups of samples.
FID captures the similarity of generated images to real ones in terms of visionrelevant features.
We demonstrate that the log-cosh loss performs well on different architectures, including VAE Kingma & Welling (2013) and Wasserstein Auto-Encoders (WAE)
Tolstikhin et al.
(2018), with multi-layered perceptrons (MLP) networks and convolutional (Conv) networks.
4. E XPERIMENTAL SETUP
In all the experiments we use an Euclidian latent spaces Z = R d and Gaussian prior distributions p θ ( z ) =
N ( Z ; 0 , I d ) .
Following Tolstikhin et al.
(2018), we use d = 8 for MNIST and d = 64 for
CelebA. For MNIST, we report results for MLP architecture and Conv architecture.
For CelebA, we only use convolutional networks, considering that VAE implemented with MLP always performs badly on complex images like human faces.
In all models, we use the sigmoid function at the output layer of the decoder.
We test the quality of reconstructed samples and generated samples.
The reconstructed samples are the outputs of the decoder given vectors z encoded from input image samples.
The generated samples are obtained by decoding random noise vectors z sampled from standard Gaussian.
The sharpness we obtain is averaged across 10 3 randomly selected images, and the FID is computed
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
based on 10 4 randomly generated samples.
For further details on experiment setup, please refer to
Appendix A.
4. R ESULTS
Numerical results on MNIST dataset are summarized in Table.
1, from which we can see that the logcosh loss improves the sharpness up to 245% and the FID up to 78% .
In all models, the log-cosh loss consistently outperforms the squared L 2 loss.
Larger sharpness stands for images of higher quality while smaller FID means that the distribution of generated samples is closer to the distribution of real dataset in terms of visual-relevant features.
One can also observe that WAE improves sample quality compared with plain VAE, but the log-cosh loss can further improve WAE.
We show generated samples in Fig.
As can be seen, many samples generated by traditional methods are quite blurry; some blurry examples are highlighted using colored boxes.
After replacing the squared L 2 loss with our log-cosh function, we get sharper examples with a clearer background.
The reconstructed samples are shown in Fig.
3, which further conﬁrms the beneﬁt of the log-cosh loss.
Our method is able to reconstruct sharp samples accurately.
We use boxes to highlight some corresponding samples that are either wrongly reconstructed or extremely blurry when the squared
L 2 loss function is used.
We also conduct interpolation experiments, i.e., linearly interpolate between latent vectors and decode samples from these vectors.
The interpolation results are reported in Appendix C, and suggest that our method enforces well-behaved manifolds in the latent space.
Results on CelebA dataset are summarized in Table.
In VAE, the log-cosh loss outperforms the squared L 2 loss by a factor of ≈ 30% in both sharpness and FID.
In WAE, the sharpness does not improve much, while the FID score is signiﬁcantly improved by 30% .
This indicates that our method better approximates the real dataset distribution.
Generated samples shown in Fig.
4 further conﬁrms this: Some samples of traditional methods are very dark and blurry, and some suffer from serious artifacts as marked by boxes.
In comparison, our method generates many fewer such examples.
[Figure]
Algorithm
Sharpness
[Figure]
Genarated
[Figure] [Figure]
WAE
[Figure]
Conv
[Figure]
Squared L
[Figure]
In Fig.
5 we show the reconstruction results
[Figure]
and we ﬁnd our method to reconstruct input images more accurately.
For example, the images reconstructed by traditional method and marked by boxes have either a wrong facial expression or a wrong gender, while our method reconstructs these features accurately.
Real data WAE (Conv)
S qua r ed
L
Log - c o s h
VAE (Conv)
Real data WAE (Conv)
S qua r ed
L
Log - c o s h
VAE (Conv)
Real data WAE (Conv)
S qua r ed
L
Log - c o s h
VAE (Conv)
4. D ISCUSSION
In this section, we present a more in-depth analysis of the results shown in Sec.
4.3. I S L 2 A GOOD DISTANCE MEASURE FOR SIMILAR PICTURES ?
We study the titled question in the context of VAE by examining the L 2 and log-cosh loss functions.
Take VAE (MLP) as an example, and use the squared L 2 and the log-cosh as loss functions.
We keep track of the quality of the parametrized networks N iL 2 and N ilog - cosh after each mini-batch i during the training.
Here the quality is measured by the squared L 2 loss when running the networks on the testing set, and this quality comparison is recorded in Fig. 6(a).
We can see that both training processes converge, and the L 2 loss of the N iL 2 networks is consistently smaller than that of the
N ilog - cosh networks.
However, as we have seen from both the sharpness/FID scores and visual comparison in Sec. 4.2, the N ilog - cosh networks actually generate better ﬁgures than the N iL 2 networks.
We also showcase one speciﬁc example in the ﬁgure, where two pictures of digit 3 are reconstructed.
The upper one is clearly better, but it has a larger L 2 error.
This suggests that the L 2 loss function is not a good measure for similar pictures in the context of VAE.
4.3. B ALANCE BETWEEN RECONSTRUCTION AND LATENT SPACE
In this section we explain that the log-cosh loss contributes to ﬁnding a better balance between the reconstruction and the latent space optimization, thus improving the performance of VAE.
Recall that VAE simultaneously minimizes two losses: the reconstruction loss which measures the error between input of encoder and output of decoder, and the latent space loss which matches the distribution of latent vectors to a given prior distribution.
The two losses are conﬂicting each other, and decreasing one often causes the other to increase.
The trade-off between reconstruction and latent space is crucial because the goal is to accurately reconstruct the data points and at the same time to obtain a compact manifold on latent space for generation.
We show in Sec.
3 that the log-cosh loss behaves like a 0 .
5 a -scaled squared L 2 loss at small errors, and like a L 1 loss at large errors.
When the reconstruction error is small, a large a makes it visible in the objective function and in turn further decreases the reconstruction error.
When the error is large, the gradient of log-cosh loss is bounded by L 1 , which avoids harming latent space much.
We
[Figure]
show in Fig.
6(b) that the latent space error of model trained with the log-cosh loss ( a = 100 ) does not increase much compared with the one trained with L 2 .
To further investigate the relationship between reconstruction and latent space, we train models with increasing parameter a and draw the reconstructed digits and FID of generation in Fig. 6(c), which shows that our method improves reconstruction and generation simultaneously.
This conﬁrms that the log-cosh loss improves reconstruction without breaking its balance with latent space optimization.
In contrast, without our log-cosh loss, the λ -scaled squared L 2 does improve the reconstruction sometimes with increasing λ .
However, the scaled L 2 is quite sensitive to outliers, which harms optimization of latent space especially when the reconstruction error is large, thus severely decreasing generation quality as shown in Fig. 6(c).
4.3. M ORE APPLICATIONS AND FUTURE WORK
The advantages of the log-cosh loss should make it useful in more applications.
We provide an example of text removal in Appendix D, where the log-cosh loss consistently outperforms the Land L 1 loss.
It is worth trying to apply the log-cosh loss to various image restoration tasks where
L 1 can outperform L 2 , as demonstrated by Zhao et al.
More tasks worth exploring include denoising Jain & Seung (2009), deblurring Xu et al.
(2014), demosaicking Wang (2014), superresolution Dong et al.
(2014), etc.
C ONCLUSION
In this paper, we propose to use the log-cosh loss as the reconstruction error of VAE.
1 We provide theoretical justiﬁcations, and conduct extensive experiments.
Empirical results demonstrate that compared with traditional L 2 loss, the log-cosh loss improves the reconstruction without damaging the latent space optimization, thus automatically keeping a balance between the reconstruction and the generation.
As a result, our method signiﬁcantly improves the performance of VAE on image data in terms of sharpness and FID score.
We also show the usefulness of the log-cosh loss for text removal, and call for more systematic studies of the loss function on other tasks in vision and general machine learning problems.
A F URTHER DETAILS ON EXPERIMENTS A. O BJECTIVE FUNCTIONS
In all implementations
[Equation]
we just change the reconstruction loss term while keep the loss term of latent space unchanged.
We list all objective functions here
[Equation]
including VAE and WAE with the squared
L 2 , L 1 and log-cosh loss as the reconstruction loss
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
where λ > 0 is a hyperparameter which by default equals to 1 in the context of VAE, the L log - cosh is the log-cosh loss deﬁned in Eq.
(5), D KL is the KL-divergence and D MMD is a divergence called the maximum mean discrepancy Tolstikhin et al.
To be accurately, the loss function of WAE should be given w.r.t batches of x and now the second term is D MMD ( (cid:82) x
q φ ( z | x ) p ( x ) dx (cid:107) p ( z )) .
A. MNIST
In all experiments on MNIST, we use mini-batches of size 100 and train the model for 40 epochs.
The activation function is Rectiﬁed Linear Unit (ReLU).
All models are trained by Adam with the learning rate η = 10 − 3 .
The log-cosh reconstruction loss Eq.(2) is implemented with parameter a = 100 .
VAE (MLP) and WAE (MLP) use the relatively simple neural networks of multi-layered perceptrons (MLP).
The encoder architecture is: x ∈ R 28 × 28 → FC 500 → FC 500 → FC 8 , and the decoder architecture is: z ∈ R 8 → FC 500 → FC 500 → FC 28 × 28 .
In the VAE (Conv) and WAE (Conv), we use convolutional architectures with 4 × 4 convolutional ﬁlters and batch normalization Ioffe & Szegedy (2015).
The encoder architecture is: x ∈ R 28 × 28 →
Conv 128 → Conv 256 → Conv 512 → Conv 1024 → FC 8 , and the decoder architecture is: z ∈
R 8 → FC 7 × 7 × 1024 → Deconv 512 →
Deconv 256 → Deconv 1 .
Here the Conv k stands for convolution with k ﬁlters, the Deconv k stands for deconvolution with k ﬁlters, and the FC k stands for fully connected layer with k neurons.
A. C ELEB A
In all experiments on CelebA, we use mini-batch size 100, ReLU activation, and Adam training algorithm as the MINST experiment.
For the learning rate, we follow WAE Tolstikhin et al.
(2018) by initializing it to η = 10 − 3 and decreasing it by factor of 2 after 30 epochs, and further by factor of 5 after ﬁrst 50 epochs.
The reported VAE (Conv) was trained for 68 epochs and WAE (Conv) for 55 epochs.
We use a = 100 in VAE (Conv) and a = 2 . 5 in WAE (Conv).
We use convolutional architectures with 4 × 4 convolutional ﬁlters and batch normalization Ioffe & Szegedy (2015).
The encoder architecture is: x ∈ R 64 × 64 × 3 → Conv 128 → Conv 256 →
Conv 512 → Conv 1024 → FC 64 , and the decoder architecture is: z ∈ R 64 → FC 8 × 8 × 1024 →
Deconv 512 → Deconv
→
Deconv 128 → Deconv 3 .
B E XPERIMENTAL RESULTS OF L 1 LOSS
In this section, we test the L 1 loss on MNIST and CelebA dataset, and compare the numerical results with the L 2 loss and the log-cosh loss reported in Sec. 4.
Samples of model trained with
[Table] [Table]
L4 .
× 10 − 3 (+29%) 3 .
× 10 − 3 (+30%) 39 (-15%)
WAE (Conv)
Squared L6 .
7 × 10 −5 .
8 × 10 −
Log-cosh 7 .
× 10 − 3 (+4%) 5 .
7 × 10 − 3 (-2%) 21 (-30%)
L6 .
5 × 10 − 3 (
[Table]
1 × 10 − 3 (-12%) 33 (+10%)
VAE (MLP)
VAE (Conv)
WAE (Conv) WAE (MLP)
R e c on s t r u c t ed
G ene r a t ed
[Figure] [Figure]
[Figure] [Figure]
[Figure]
on s t r u c t ed
G ene r a t ed
[Figure]
L 1 reconstruction loss are shown in Fig 7 and Fig. 8, and the numerical results are summarized in
Table 3 and Table 4, where the improvement percentages are computed w.r.t the squared L 2 loss.
One can see that on the MNIST dataset, the log-cosh loss performs much better than the L 2 loss, as expected.
It is also observed that the L 1 loss also performs very well, and sometimes even slightly better than the log-cosh loss.
This is because 90% pixels of MNIST data are very close to binary value { 0 , 1 } and the outputs of our networks are in the interval (0 , 1) , always falling into one side of the ground truth value (0 or 1).
Thus the typical issue of the L 1 loss oscillating between nondifferential point does not occur often here, and thus the advantage of log-cosh being smooth around 0 does not exhibit beneﬁts than L 1 .
[Figure]
R e c on s t r u c t ed
G ene r a t ed
[Figure]
R e c on s t r u c t ed
G ene r a t ed
[Figure] [Figure]
VAE ( M L P )
VAE ( C on v )
W AE ( C on v )
Squared L
[Figure]
W AE ( M L P )
Squared L
[Figure]
cosh
VAE ( M L P )
VAE ( C on v )
W AE (
[Figure]
v )
[Figure]
W AE ( M L P )
In comparison, for colored images or black-white images with gray pixels, the ground truth in general is somewhere between 0 and 1, and an output of a network may fall left or right of this value, suffering from the oscillation of derivatives of L 1 .
Indeed, we can see that on CelebA, the L 1 loss performs much worse than the log-cosh loss in both scores (Table 4) and the visualization (Fig.
C I NTERPOLATION EXPERIMENTS ON MNIST
We randomly take two samples ( x 1 , x 2 ) from real dataset and feed them to the encoder to get the latent vectors ( z 1 , z 2 ) .
Then we linearly interpolate between z 1 and z 2 in the latent space and use them to decode images.
The decoded images are shown in Fig 9.
The log-cosh loss enforces well behaved manifold in latent space, which is obviously better than squared L 2 loss.
[Figure]
D A PPLYING LOG - COSH TO IMAGE RESTORATION TASKS
It is promising to apply the log-cosh loss to various image restoration applications such as denoising, deblurring, demosaicking, and super-resolution.
Here we provide an example of text removal, and show that the log-cosh loss consistently outperforms the L 2 and L 1 loss.
We use a recently proposed denoising framework called noise2noise Lehtinen et al.
(2018), which learns to restore good images by only going through bad ones.
The network architecture is based on the idea of convolutional auto-encoder, consisting of convolutional and deconvolutional layers.
The training dataset consists of 291 images widely used (e.g. in Schulter et al.
; Kim et al.
Among these,images of the training set come from the Berkeley Segmentation Dataset Martin et al.
and the rest 91 images come from Yang et al.
Following Lehtinen et al.
(2018), we add texts to the images and let the model learn to remove these texts.
After training, we test the trained model on the Urban 100 dataset Huang et al.
(2015) which consists of 100 high resolution images.
We use the SSIM index Wang et al.
(2004) which is often used as a measure to evaluate the similarity between the restored images and the target images in the denoising tasks.
We compare the squared
L 2 loss and the L 1 loss to the log-cosh loss, and the results are shown in Fig.
We can see that the log-cosh loss consistently outperforms both the squared L 2 loss and the L 1 loss.
Actually, the log-cosh loss fails to outperform the L 1 loss in only 1 out of the 100 tested images, illustrated by the red point in Fig.
We list some restored images in Fig.
11 to show how the log-cosh loss stands out.
The L 2 loss may lead to a severe color shift, because the L 2 loss encourages the outputs to shift toward the mean color of the noisy images, which is locally distorted by texts.
The L 1 loss encourages to take the median and does not suffer from color shift, but severe artifacts can appear, as highlighted by red boxes in
Fig.
The log-cosh loss releases the problems by averaging observations around the median.
It behaves like the L 1 at the beginning of training when the error is large, thus avoiding the color shift.
In addition, when the error is small, The log-cosh loss behaves like the L 2 and encourages to smooth the color by taking the mean, thus reducing the artifacts introduced by the L 1 loss.
As highlighted by red boxes, in the results of the log-cosh loss, the artifacts are removed or sometimes still exist but their color is much closer to the background.
Clean targets
Input
L L
Log-cosh Clean targets
Input
L L
Log-cosh 1-SSIM 0.
0.083 1-SSIM 0.
1-SSIM 0.
0.028 1-SSIM 0.
1-SSIM 0.
0.135 1-SSIM 0.
1-SSIM 0.
0.055 1-SSIM 0.
Clean targets
Input
L L
Log-cosh 1-SSIM 0.
1-SSIM 0. 1-SSIM 0.
1-SSIM 0.
Clean targets
Input
L L
Log-cosh 1-SSIM 0.
1-SSIM 0.
1-SSIM 0. 1-SSIM 0.