Original: G ENERATIVE A DVERSARIAL N ETWORKS FOR E XTREME L EARNED I MAGE C OMPRESSION
Revised: G ENERATIVE A DVERSARIAL N ETWORKS FOR E XTREME L EARNED I MAGE C OMPRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a framework for extreme learned image compression based on Generative Adversarial Networks (GANs), obtaining visually pleasing images at signiﬁcantly lower bitrates than previous methods.
Revised: We propose a framework for extreme learned image compression based on Generative Adversarial Networks (GANs), obtaining visually pleasing images at signiﬁcantly lower bitrates than previous methods.

Original: This is made possible through our GAN formulation of learned compression combined with a generator/decoder which operates on the full-resolution image and is trained in combination with a multi-scale discriminator.
Revised: This is made possible through our GAN formulation of learned compression combined with a generator/decoder which operates on the full-resolution image and is trained in combination with a multi-scale discriminator.

Original: Additionally, if a semantic label map of the original image is available, our method can fully synthesize unimportant regions in the decoded image such as streets and trees from the label map, therefore only requiring the storage of the preserved region and the semantic label map.
Revised: Additionally, if a semantic label map of the original image is available, our method can fully synthesize unimportant regions in the decoded image such as streets and trees from the label map, therefore only requiring the storage of the preserved region and the semantic label map.

Original: A user study conﬁrms that for low bitrates, our approach is preferred to state-of-the-art methods, even when they use more than double the bits.
Revised: A user study conﬁrms that for low bitrates, our approach is preferred to state-of-the-art methods, even when they use more than double the bits.

Original: [Figure]
Revised: [Figure]

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image compression systems based on deep neural networks (DNNs), or deep compression systems for short, have become an active area of research recently.
Revised: Image compression systems based on deep neural networks (DNNs), or deep compression systems for short, have become an active area of research recently.

Original: These systems (e.g. (Theis et al., 2017;
Revised: These systems (e.g. (Theis et al., 2017;

Original: Ball´e et al., 2016b; Rippel & Bourdev, 2017; Ball´e et al., 2018; Mentzer et al., 2018)) are often competitive with modern engineered codecs such as WebP (WebP), JPEG2000 (Taubman & Marcellin, 2001) and even BPG(Bellard) (the state-of-the-art engineered codec).
Revised: Ball´e et al., 2016b; Rippel & Bourdev, 2017; Ball´e et al., 2018; Mentzer et al., 2018)) are often competitive with modern engineered codecs such as WebP (WebP), JPEG2000 (Taubman & Marcellin, 2001) and even BPG(Bellard) (the state-of-the-art engineered codec).

Original: Besides achieving competitive compression rates on natural images, they can be easily adapted to speciﬁc target domains such as stereo or medical images, and promise efﬁcient processing and indexing directly from compressed representations (Torfason et al., 2018).
Revised: Besides achieving competitive compression rates on natural images, they can be easily adapted to speciﬁc target domains such as stereo or medical images, and promise efﬁcient processing and indexing directly from compressed representations (Torfason et al., 2018).

Original: However, deep compression systems are typically optimized for traditional distortion metrics such as peak signal-to-noise ratio (PSNR) or multi-scale structural similarity (MS-SSIM)
Revised: However, deep compression systems are typically optimized for traditional distortion metrics such as peak signal-to-noise ratio (PSNR) or multi-scale structural similarity (MS-SSIM)

Original: (Wang et al., 2003).
Revised: (Wang et al., 2003).

Original: For very low bitrates (below 0.1 bits per pixel (bpp)), where preserving the full image content becomes impossible, these distortion metrics lose signiﬁcance as they favor pixel-wise preservation of local (high-entropy) structure over preserving texture and global structure.
Revised: For very low bitrates (below 0.1 bits per pixel (bpp)), where preserving the full image content becomes impossible, these distortion metrics lose signiﬁcance as they favor pixel-wise preservation of local (high-entropy) structure over preserving texture and global structure.

Original: To further advance deep image compression it is therefore of great importance to develop new training objectives beyond PSNR and MS-SSIM.
Revised: To further advance deep image compression it is therefore of great importance to develop new training objectives beyond PSNR and MS-SSIM.

Original: A promising candidate towards this goal are adversarial losses (Goodfellow et al., 2014) which were shown recently to capture global semantic information and local texture, yielding powerful generators that produce visually appealing high-resolution images from semantic label maps (Isola et al., 2017; Wang et al., 2018).
Revised: A promising candidate towards this goal are adversarial losses (Goodfellow et al., 2014) which were shown recently to capture global semantic information and local texture, yielding powerful generators that produce visually appealing high-resolution images from semantic label maps (Isola et al., 2017; Wang et al., 2018).

Original: In this paper, we propose and study a generative adversarial network (GAN)-based framework for extreme image compression, targeting bitrates below 0.1 bpp.
Revised: In this paper, we propose and study a generative adversarial network (GAN)-based framework for extreme image compression, targeting bitrates below 0.1 bpp.

Original: We rely on a principled GAN formulation for deep image compression that allows for different degrees of content generation.
Revised: We rely on a principled GAN formulation for deep image compression that allows for different degrees of content generation.

Original: In contrast to prior works on deep image compression which applied adversarial losses to image patches for artifact suppression (Rippel & Bourdev, 2017; Galteri et al., 2017), generation of texture details (Ledig et al., 2017), or representation learning for thumbnail images (Santurkar et al., 2017), our generator/decoder operates on the full-resolution image and is trained with a multi-scale discriminator (Wang et al., 2018).
Revised: In contrast to prior works on deep image compression which applied adversarial losses to image patches for artifact suppression (Rippel & Bourdev, 2017; Galteri et al., 2017), generation of texture details (Ledig et al., 2017), or representation learning for thumbnail images (Santurkar et al., 2017), our generator/decoder operates on the full-resolution image and is trained with a multi-scale discriminator (Wang et al., 2018).

Original: We consider two modes of operation (corresponding to unconditional and conditional GANs (Goodfellow et al., 2014; Mirza & Osindero, 2014)), namely
Revised: We consider two modes of operation (corresponding to unconditional and conditional GANs (Goodfellow et al., 2014; Mirza & Osindero, 2014)), namely

Original: • generative compression (GC) , preserving the overall image content while generating structure of different scales such as leaves of trees or windows in the facade of buildings
Revised: • generative compression (GC) , preserving the overall image content while generating structure of different scales such as leaves of trees or windows in the facade of buildings

Original: and • selective generative compression (SC) , completely generating parts of the image from a semantic label map while preserving user-deﬁned regions with a high degree of detail
Revised: and • selective generative compression (SC) , completely generating parts of the image from a semantic label map while preserving user-deﬁned regions with a high degree of detail

Original: We emphasize that GC does not require semantic label maps (neither for training, nor for deployment).
Revised: We emphasize that GC does not require semantic label maps (neither for training, nor for deployment).

Original: A typical use case for GC are bandwidth constrained scenarios, where one wants to preserve the full image as well as possible, while falling back to synthesized content instead of blocky/blurry blobs for regions for which not sufﬁcient bits are available to store the original pixels.
Revised: A typical use case for GC are bandwidth constrained scenarios, where one wants to preserve the full image as well as possible, while falling back to synthesized content instead of blocky/blurry blobs for regions for which not sufﬁcient bits are available to store the original pixels.

Original: SC could be applied in a video call scenario where one wants to fully preserve people in the video stream, but a visually pleasing synthesized background serves the purpose as well as the true background.
Revised: SC could be applied in a video call scenario where one wants to fully preserve people in the video stream, but a visually pleasing synthesized background serves the purpose as well as the true background.

Original: In the GC operation mode the image is transformed into a bitstream and encoded using arithmetic coding.
Revised: In the GC operation mode the image is transformed into a bitstream and encoded using arithmetic coding.

Original: SC requires a semantic/instance label map of the original image which can be obtained using off-the-shelf semantic/instance segmentation networks, e.g., PSPNet (Zhao et al., 2017) and
Revised: SC requires a semantic/instance label map of the original image which can be obtained using off-the-shelf semantic/instance segmentation networks, e.g., PSPNet (Zhao et al., 2017) and

Original: Mask R-CNN (He et al., 2017), and which is stored as a vector graphic.
Revised: Mask R-CNN (He et al., 2017), and which is stored as a vector graphic.

Original: This amounts to a small, image dimension-independent overhead in terms of coding cost.
Revised: This amounts to a small, image dimension-independent overhead in terms of coding cost.

Original: On the other hand, the size of the compressed image is reduced proportionally to the area which is generated from the semantic label map, typically leading to a signiﬁcant overall reduction in storage cost.
Revised: On the other hand, the size of the compressed image is reduced proportionally to the area which is generated from the semantic label map, typically leading to a signiﬁcant overall reduction in storage cost.

Original: For GC, a comprehensive user study shows that our compression system yields visually considerably more appealing results than BPG (Bellard) (the current state-of-the-art engineered compression algorithm) and the recently proposed autoencoder-based deep compression (AEDC) system (Mentzer et al., 2018).
Revised: For GC, a comprehensive user study shows that our compression system yields visually considerably more appealing results than BPG (Bellard) (the current state-of-the-art engineered compression algorithm) and the recently proposed autoencoder-based deep compression (AEDC) system (Mentzer et al., 2018).

Original: In particular, our GC models trained for compression of general natural images are preferred to BPG when BPG uses up to 95% and 124% more bits than those produced by our models on the Kodak (Kodak) and RAISE1K (Dang-Nguyen et al., 2015) data set, respectively.
Revised: In particular, our GC models trained for compression of general natural images are preferred to BPG when BPG uses up to 95% and 124% more bits than those produced by our models on the Kodak (Kodak) and RAISE1K (Dang-Nguyen et al., 2015) data set, respectively.

Original: When constraining the target domain to the street scene images of the Cityscapes data set (Cordts et al., 2016), the reconstructions of our GC models are preferred to BPG even when the latter uses up to 181% more bits.
Revised: When constraining the target domain to the street scene images of the Cityscapes data set (Cordts et al., 2016), the reconstructions of our GC models are preferred to BPG even when the latter uses up to 181% more bits.

Original: To the best of our knowledge, these are the ﬁrst results showing that a deep compression method outperforms BPG on the Kodak data set in a user study—and by large margins.
Revised: To the best of our knowledge, these are the ﬁrst results showing that a deep compression method outperforms BPG on the Kodak data set in a user study—and by large margins.

Original: In the SC operation mode, our system seamlessly combines preserved image content with synthesized content, even for regions that cross multiple object boundaries, while faithfully preserving the image semantics.
Revised: In the SC operation mode, our system seamlessly combines preserved image content with synthesized content, even for regions that cross multiple object boundaries, while faithfully preserving the image semantics.

Original: By partially generating image content we achieve bitrate reductions of over 50% without notably degrading image quality.
Revised: By partially generating image content we achieve bitrate reductions of over 50% without notably degrading image quality.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Deep image compression has recently emerged as an active area of research.
Revised: Deep image compression has recently emerged as an active area of research.

Original: The most popular DNN architectures for this task are to date auto-encoders (Theis et al., 2017; Ball´e et al., 2016b;
Revised: The most popular DNN architectures for this task are to date auto-encoders (Theis et al., 2017; Ball´e et al., 2016b;

Original: Agustsson et al., 2017; Li et al., 2017; Torfason et al., 2018; Minnen et al., 2018) and recurrent neural networks (RNNs) (Toderici et al., 2015; 2016).
Revised: Agustsson et al., 2017; Li et al., 2017; Torfason et al., 2018; Minnen et al., 2018) and recurrent neural networks (RNNs) (Toderici et al., 2015; 2016).

Original: These DNNs transform the input image into a bit-stream, which is in turn losslessly compressed using entropy coding methods such as Huffman coding or arithmetic coding.
Revised: These DNNs transform the input image into a bit-stream, which is in turn losslessly compressed using entropy coding methods such as Huffman coding or arithmetic coding.

Original: To reduce coding rates, many deep compression systems rely on context models to capture the distribution of the bit stream (Ball´e et al., 2016b; Toderici et al., 2016;
Revised: To reduce coding rates, many deep compression systems rely on context models to capture the distribution of the bit stream (Ball´e et al., 2016b; Toderici et al., 2016;

Original: Li et al., 2017;
Revised: Li et al., 2017;

Original: Rippel & Bourdev, 2017; Mentzer et al., 2018).
Revised: Rippel & Bourdev, 2017; Mentzer et al., 2018).

Original: Common loss functions to measure the distortion between the original and decompressed images are the mean-squared error (MSE) (Theis et al., 2017; Ball´e et al., 2016b;
Revised: Common loss functions to measure the distortion between the original and decompressed images are the mean-squared error (MSE) (Theis et al., 2017; Ball´e et al., 2016b;

Original: Agustsson et al., 2017; Li et al., 2017; Ball´e et al., 2018; Torfason et al., 2018), or perceptual metrics such as MS-SSIM (Toderici et al., 2016; Rippel & Bourdev, 2017;
Revised: Agustsson et al., 2017; Li et al., 2017; Ball´e et al., 2018; Torfason et al., 2018), or perceptual metrics such as MS-SSIM (Toderici et al., 2016; Rippel & Bourdev, 2017;

Original: Ball´e et al., 2018; Mentzer et al., 2018).
Revised: Ball´e et al., 2018; Mentzer et al., 2018).

Original: Some authors rely on advanced techniques including multiscale decompositions (Rippel & Bourdev, 2017), progressive encoding/decoding strategies (Toderici et al., 2015; 2016), and generalized divisive normalization (GDN) layers (Ball´e et al., 2016b;a).
Revised: Some authors rely on advanced techniques including multiscale decompositions (Rippel & Bourdev, 2017), progressive encoding/decoding strategies (Toderici et al., 2015; 2016), and generalized divisive normalization (GDN) layers (Ball´e et al., 2016b;a).

Original: Generative adversarial networks (GANs) (Goodfellow et al., 2014) have emerged as a popular technique for learning generative models for intractable distributions in an unsupervised manner.
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) have emerged as a popular technique for learning generative models for intractable distributions in an unsupervised manner.

Original: Despite stability issues (Salimans et al., 2016; Arjovsky & Bottou, 2017; Arjovsky et al., 2017; Mao et al., 2017), they were shown to be capable of generating more realistic and sharper images than prior approaches and to scale to resolutions of 1024 × 1024 px (Zhang et al., 2017; Karras et al., 2017) for some datasets.
Revised: Despite stability issues (Salimans et al., 2016; Arjovsky & Bottou, 2017; Arjovsky et al., 2017; Mao et al., 2017), they were shown to be capable of generating more realistic and sharper images than prior approaches and to scale to resolutions of 1024 × 1024 px (Zhang et al., 2017; Karras et al., 2017) for some datasets.

Original: Another direction that has shown great progress are conditional GANs (Goodfellow et al., 2014; Mirza & Osindero, 2014), obtaining impressive results for image-to-image translation (Isola et al., 2017;
Revised: Another direction that has shown great progress are conditional GANs (Goodfellow et al., 2014; Mirza & Osindero, 2014), obtaining impressive results for image-to-image translation (Isola et al., 2017;

Original: Wang et al., 2018; Zhu et al., 2017; Liu et al., 2017) on various datasets (e.g. maps to satellite images), reaching resolutions as high as 1024 × 2048 px (Wang et al., 2018).
Revised: Wang et al., 2018; Zhu et al., 2017; Liu et al., 2017) on various datasets (e.g. maps to satellite images), reaching resolutions as high as 1024 × 2048 px (Wang et al., 2018).

Original: Arguably the most closely related work to ours is (Rippel & Bourdev, 2017), which uses an adversarial loss term to train a deep compression system.
Revised: Arguably the most closely related work to ours is (Rippel & Bourdev, 2017), which uses an adversarial loss term to train a deep compression system.

Original: However, this loss term is applied to small image patches and its purpose is to suppress artifacts rather than to generate image content.
Revised: However, this loss term is applied to small image patches and its purpose is to suppress artifacts rather than to generate image content.

Original: Furthermore, it uses a non-standard GAN formulation that does not (to the best of our knowledge) have an interpretation in terms of divergences between probability distributions, as in (Goodfellow et al., 2014;
Revised: Furthermore, it uses a non-standard GAN formulation that does not (to the best of our knowledge) have an interpretation in terms of divergences between probability distributions, as in (Goodfellow et al., 2014;

Original: Nowozin et al., 2016).
Revised: Nowozin et al., 2016).

Original: We refer to Sec. 6.1 and Appendix A for a more detailed discussion.
Revised: We refer to Sec. 6.1 and Appendix A for a more detailed discussion.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: (2017) use a GAN framework to learn a generative model over thumbnail images, which is then used as a decoder for thumbnail image compression.
Revised: (2017) use a GAN framework to learn a generative model over thumbnail images, which is then used as a decoder for thumbnail image compression.

Original: Other works use adversarial training for compression artifact removal (for engineered codecs) (Galteri et al., 2017) and single image super-resolution (Ledig et al., 2017).
Revised: Other works use adversarial training for compression artifact removal (for engineered codecs) (Galteri et al., 2017) and single image super-resolution (Ledig et al., 2017).

Original: Finally, related to our SC mode, spatially allocating bitrate based on saliency of image content has a long history in the context of engineered compression algorithms, see, e.g.,, (Stella & Lisin, 2009; Guo & Zhang, 2010; Gupta et al., 2013).
Revised: Finally, related to our SC mode, spatially allocating bitrate based on saliency of image content has a long history in the context of engineered compression algorithms, see, e.g.,, (Stella & Lisin, 2009; Guo & Zhang, 2010; Gupta et al., 2013).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Generative Adversarial Networks:
Revised: Generative Adversarial Networks:

Original: Given a data set X , Generative Adversarial Networks (GANs) can learn to approximate its (unknown) distribution p x through a generator G ( z ) that tries to map samples z from a ﬁxed prior distribution p z to the distribution p x .
Revised: Given a data set X , Generative Adversarial Networks (GANs) can learn to approximate its (unknown) distribution p x through a generator G ( z ) that tries to map samples z from a ﬁxed prior distribution p z to the distribution p x .

Original: The generator G is trained in parallel with a discriminator D by searching (using stochastic gradient descent (SGD)) for a saddle point of a mini-max objective
Revised: The generator G is trained in parallel with a discriminator D by searching (using stochastic gradient descent (SGD)) for a saddle point of a mini-max objective

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where G and D are DNNs and f and g are scalar functions.
Revised: where G and D are DNNs and f and g are scalar functions.

Original: The original paper (Goodfellow et al., 2014) uses the “Vanilla GAN” objective with f ( y )
Revised: The original paper (Goodfellow et al., 2014) uses the “Vanilla GAN” objective with f ( y )

Original: =
Revised: =

Original: log( y ) and g ( y )
Revised: log( y ) and g ( y )

Original: = log(

Revised: = log(

Original: − y ) .
Revised: − y ) .

Original: This corresponds to G minimizing the Jensen-Shannon (JS) Divergence between the (empirical) distribution of x and G ( z ) .
Revised: This corresponds to G minimizing the Jensen-Shannon (JS) Divergence between the (empirical) distribution of x and G ( z ) .

Original: The JS Divergence is a member of a more generic family of f -divergences, and
Revised: The JS Divergence is a member of a more generic family of f -divergences, and

Original: Nowozin et al.
Revised: Nowozin et al.

Original: (2016) show that for suitable choices of f and g , all such divergences can be minimized with (1).
Revised: (2016) show that for suitable choices of f and g , all such divergences can be minimized with (1).

Original: In particular, if one uses f ( y ) = ( y − 1) 2 and g ( y ) =
Revised: In particular, if one uses f ( y ) = ( y − 1) 2 and g ( y ) =

Original: y 2 , one obtains the LeastSquares GAN (Mao et al., 2017) (which corresponds to the Pearson χ 2 divergence), which we adopt in this paper.
Revised: y 2 , one obtains the LeastSquares GAN (Mao et al., 2017) (which corresponds to the Pearson χ 2 divergence), which we adopt in this paper.

Original: We refer to the divergence minimized over G as
Revised: We refer to the divergence minimized over G as

Original: [Equation]
Revised: [Equation]

Original: Conditional Generative Adversarial Networks:
Revised: Conditional Generative Adversarial Networks:

Original: For conditional GANs (cGANs)
Revised: For conditional GANs (cGANs)

Original: (Goodfellow et al., 2014; Mirza & Osindero, 2014), each data point x is associated with additional information s , where ( x , s ) have an unknown joint distribution p x , s .
Revised: (Goodfellow et al., 2014; Mirza & Osindero, 2014), each data point x is associated with additional information s , where ( x , s ) have an unknown joint distribution p x , s .

Original: We now assume that s is given and that we want to use the GAN to model the conditional distribution p x | s .
Revised: We now assume that s is given and that we want to use the GAN to model the conditional distribution p x | s .

Original: In this case, both the generator
Revised: In this case, both the generator

Original: G ( z , s ) and discriminator D ( z , s ) have access to the side information s , leading to the divergence
Revised: G ( z , s ) and discriminator D ( z , s ) have access to the side information s , leading to the divergence

Original: [Equation]
Revised: [Equation]

Original: Deep Image Compression: To compress an image x ∈ X , we follow the formulation of (Agustsson et al., 2017; Mentzer et al., 2018) where one learns an encoder E , a decoder G , and a ﬁnite quantizer q .
Revised: Deep Image Compression: To compress an image x ∈ X , we follow the formulation of (Agustsson et al., 2017; Mentzer et al., 2018) where one learns an encoder E , a decoder G , and a ﬁnite quantizer q .

Original: The encoder E maps the image to a latent feature map w , whose values are then quantized to L levels { c 1 , . .
Revised: The encoder E maps the image to a latent feature map w , whose values are then quantized to L levels { c 1 , . .

Original: c L } ⊂ R to obtain a representation ˆ w = q ( E ( x ))
Revised: c L } ⊂ R to obtain a representation ˆ w = q ( E ( x ))

Original: that can be encoded to a bitstream.
Revised: that can be encoded to a bitstream.

Original: The decoder then tries to recover the image by forming a reconstruction ˆ x = G ( ˆ w ) .
Revised: The decoder then tries to recover the image by forming a reconstruction ˆ x = G ( ˆ w ) .

Original: To be able to backpropagate through the non-differentiable q , one can use a differentiable relaxation of q , as in (Mentzer et al., 2018).
Revised: To be able to backpropagate through the non-differentiable q , one can use a differentiable relaxation of q , as in (Mentzer et al., 2018).

Original: The average number of bits needed to encode ˆ w is measured by the entropy H ( ˆ w ) , which can be modeled with a prior (Agustsson et al., 2017) or a conditional probability model (Mentzer et al., 2018).
Revised: The average number of bits needed to encode ˆ w is measured by the entropy H ( ˆ w ) , which can be modeled with a prior (Agustsson et al., 2017) or a conditional probability model (Mentzer et al., 2018).

Original: The trade-off between reconstruction quality and bitrate to be optimized is then
Revised: The trade-off between reconstruction quality and bitrate to be optimized is then

Original: [Equation]
Revised: [Equation]

Original: where d is a loss that measures how perceptually similar ˆ x is to x .
Revised: where d is a loss that measures how perceptually similar ˆ x is to x .

Original: Given a differentiable estimator of the entropy H ( ˆ w ) , the weight β controls the bitrate of the model (large β pushes the bitrate down).
Revised: Given a differentiable estimator of the entropy H ( ˆ w ) , the weight β controls the bitrate of the model (large β pushes the bitrate down).

Original: However, since the number of dimensions dim ( ˆ w ) and the number of levels L are ﬁnite, the entropy is bounded by (see, e.g., (Cover & Thomas, 2012))
Revised: However, since the number of dimensions dim ( ˆ w ) and the number of levels L are ﬁnite, the entropy is bounded by (see, e.g., (Cover & Thomas, 2012))

Original: [Equation]
Revised: [Equation]

Original: It is therefore also valid to set β = 0 and control the maximum bitrate through the bound (5) (i.e., adjusting L and/or dim ( ˆ w ) through the architecture of E ).
Revised: It is therefore also valid to set β = 0 and control the maximum bitrate through the bound (5) (i.e., adjusting L and/or dim ( ˆ w ) through the architecture of E ).

Original: While potentially leading to suboptimal bitrates, this avoids to model the entropy explicitly as a loss term.
Revised: While potentially leading to suboptimal bitrates, this avoids to model the entropy explicitly as a loss term.

Original:  GAN S FOR EXTREME IMAGE COMPRESSION 4. G ENERATIVE C OMPRESSION
Revised:  GAN S FOR EXTREME IMAGE COMPRESSION 4. G ENERATIVE C OMPRESSION

Original: ˆ w w x
Revised: ˆ w w x

Original: [Equation]
Revised: [Equation]

Original: w w x
Revised: w w x

Original: [Equation]
Revised: [Equation]

Original: x ˆ x
Revised: x ˆ x

Original: E
Revised: E

Original: G
Revised: G

Original: D
Revised: D

Original: G E
Revised: G E

Original: D q q
Revised: D q q

Original: The proposed GAN framework for extreme image compression can be viewed as a combination of (conditional) GANs and learned compression.
Revised: The proposed GAN framework for extreme image compression can be viewed as a combination of (conditional) GANs and learned compression.

Original: With an encoder E and quantizer q , we encode the image x to a compressed representation ˆ w = q ( E ( x )) .
Revised: With an encoder E and quantizer q , we encode the image x to a compressed representation ˆ w = q ( E ( x )) .

Original: This representation is optionally concatenated with noise v drawn from a ﬁxed prior p v , to form the latent vector z .
Revised: This representation is optionally concatenated with noise v drawn from a ﬁxed prior p v , to form the latent vector z .

Original: The decoder/generator G then tries to generate an image ˆ x = G ( z ) that is consistent with the image distribution p x while also recovering the speciﬁc encoded image x to a certain degree (see inset Fig.).
Revised: The decoder/generator G then tries to generate an image ˆ x = G ( z ) that is consistent with the image distribution p x while also recovering the speciﬁc encoded image x to a certain degree (see inset Fig.).

Original: Using z =
Revised: Using z =

Original: [ ˆ w , v ] , this can be expressed by our saddle-point objective for (unconditional) generative compression,
Revised: [ ˆ w , v ] , this can be expressed by our saddle-point objective for (unconditional) generative compression,

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 balances the distortion term against the GAN loss and entropy terms.
Revised: where λ > 0 balances the distortion term against the GAN loss and entropy terms.

Original: Using this formulation, we need to encode a real image, ˆ w = E ( x ) , to be able to sample from p ˆ w .
Revised: Using this formulation, we need to encode a real image, ˆ w = E ( x ) , to be able to sample from p ˆ w .

Original: However, this is not a limitation as our goal is to compress real images and not to generate completely new ones.
Revised: However, this is not a limitation as our goal is to compress real images and not to generate completely new ones.

Original: Since the last two terms of (6) do not depend on the discriminator D , they do not affect its optimization directly.
Revised: Since the last two terms of (6) do not depend on the discriminator D , they do not affect its optimization directly.

Original: This means that the discriminator still computes the same f divergence L GAN as in (2), so we can write (6) as
Revised: This means that the discriminator still computes the same f divergence L GAN as in (2), so we can write (6) as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We note that equation (6) has completely different dynamics than a normal GAN, because the latent space z contains ˆ w , which stores information about a real image x .
Revised: We note that equation (6) has completely different dynamics than a normal GAN, because the latent space z contains ˆ w , which stores information about a real image x .

Original: A crucial ingredient is the bitrate limitation on H ( ˆ w ) .
Revised: A crucial ingredient is the bitrate limitation on H ( ˆ w ) .

Original: If we allow ˆ w to contain arbitrarily many bits by setting β = 0 and letting L and dim( ˆ w ) be large enough, E and G could learn to near-losslessly recover x from
Revised: If we allow ˆ w to contain arbitrarily many bits by setting β = 0 and letting L and dim( ˆ w ) be large enough, E and G could learn to near-losslessly recover x from

Original: G ( z ) =
Revised: G ( z ) =

Original: G ( q ( E ( x ))) , such that the distortion term would vanish.
Revised: G ( q ( E ( x ))) , such that the distortion term would vanish.

Original: In this case, the divergence between p x and p G ( z ) would also vanish and the GAN loss would have no effect.
Revised: In this case, the divergence between p x and p G ( z ) would also vanish and the GAN loss would have no effect.

Original: By constraining the entropy of ˆ w , E and G will never be able to make d fully vanish.
Revised: By constraining the entropy of ˆ w , E and G will never be able to make d fully vanish.

Original: In this case,
Revised: In this case,

Original: E, G need to balance the GAN objective L GAN and the distortion term λ E
Revised: E, G need to balance the GAN objective L GAN and the distortion term λ E

Original: [ d ( x , G ( z ))] , which leads to G ( z ) on one hand looking “realistic”, and on the other hand preserving the original image.
Revised: [ d ( x , G ( z ))] , which leads to G ( z ) on one hand looking “realistic”, and on the other hand preserving the original image.

Original: For example, if there is a tree for which E cannot afford to store the exact texture (and make d small) G can synthesize it to satisfy L GAN , instead of showing a blurry green blob.
Revised: For example, if there is a tree for which E cannot afford to store the exact texture (and make d small) G can synthesize it to satisfy L GAN , instead of showing a blurry green blob.

Original: In the extreme case where the bitrate becomes zero (i.e., H ( ˆ w ) → 0 , e.g., by setting β = ∞ or dim ( ˆ w ) = 0 ), ˆ w becomes deterministic.
Revised: In the extreme case where the bitrate becomes zero (i.e., H ( ˆ w ) → 0 , e.g., by setting β = ∞ or dim ( ˆ w ) = 0 ), ˆ w becomes deterministic.

Original: In this setting, z is random and independent of x (through the v component) and the objective reduces to a standard GAN plus the distortion term, which acts as a regularizer.
Revised: In this setting, z is random and independent of x (through the v component) and the objective reduces to a standard GAN plus the distortion term, which acts as a regularizer.

Original: We refer to the setting in (6) as generative compression (GC), where E, G balance reconstruction and generation automatically over the image.
Revised: We refer to the setting in (6) as generative compression (GC), where E, G balance reconstruction and generation automatically over the image.

Original: As for the conditional GANs described in Sec. 3, we can easily extend GC to a conditional case.
Revised: As for the conditional GANs described in Sec. 3, we can easily extend GC to a conditional case.

Original: Here, we also consider this setting, where the additional information s for an image x is a semantic label map of the scene, but with a twist: Instead of feeding the semantics to E, G and D , we only give them to the discriminator D during training.
Revised: Here, we also consider this setting, where the additional information s for an image x is a semantic label map of the scene, but with a twist: Instead of feeding the semantics to E, G and D , we only give them to the discriminator D during training.

Original: This means that no semantics are needed to encode or decode images with the trained models (since
Revised: This means that no semantics are needed to encode or decode images with the trained models (since

Original: E, G do not depend on s ).
Revised: E, G do not depend on s ).

Original: We refer to this setting as GC ( D + ) .
Revised: We refer to this setting as GC ( D + ) .

Original: 4. S ELECTIVE G ENERATIVE C OMPRESSION
Revised: 4. S ELECTIVE G ENERATIVE C OMPRESSION

Original: For GC and its conditional variant described in the previous section, E, G automatically navigate the trade-off between generation and preservation over the entire image, without any guidance.
Revised: For GC and its conditional variant described in the previous section, E, G automatically navigate the trade-off between generation and preservation over the entire image, without any guidance.

Original: Here, we consider a different setting, where we guide the network in terms of which regions should be preserved and which regions should be synthesized.
Revised: Here, we consider a different setting, where we guide the network in terms of which regions should be preserved and which regions should be synthesized.

Original: We refer to this setting as selective generative compression (SC) (an overview of the network structure is given in Fig.
Revised: We refer to this setting as selective generative compression (SC) (an overview of the network structure is given in Fig.

Original: 8 in Appendix C).
Revised: 8 in Appendix C).

Original: For simplicity, we consider a binary setting, where we construct a single-channel binary heatmap m of the same spatial dimensions as ˆ w .
Revised: For simplicity, we consider a binary setting, where we construct a single-channel binary heatmap m of the same spatial dimensions as ˆ w .

Original: Regions of zeros correspond to regions that should be fully synthesized, whereas regions of ones correspond to regions that should be preserved.
Revised: Regions of zeros correspond to regions that should be fully synthesized, whereas regions of ones correspond to regions that should be preserved.

Original: However, since our task is compression, we constrain the fully synthesized regions to have the same semantics s as the original image x .
Revised: However, since our task is compression, we constrain the fully synthesized regions to have the same semantics s as the original image x .

Original: We assume the semantics s are separately stored, and thus feed them through a feature extractor F before feeding them to the generator G .
Revised: We assume the semantics s are separately stored, and thus feed them through a feature extractor F before feeding them to the generator G .

Original: To guide the network with the semantics, we mask the (pixel-wise) distortion d , such that it is only computed over the region to be preserved.
Revised: To guide the network with the semantics, we mask the (pixel-wise) distortion d , such that it is only computed over the region to be preserved.

Original: Additionally, we zero out the compressed representation ˆ w in the regions that should be synthesized.
Revised: Additionally, we zero out the compressed representation ˆ w in the regions that should be synthesized.

Original: Provided that the heatmap m is also stored, we then only encode the entries of ˆ w corresponding to the preserved regions, greatly reducing the bitrate needed to store it.
Revised: Provided that the heatmap m is also stored, we then only encode the entries of ˆ w corresponding to the preserved regions, greatly reducing the bitrate needed to store it.

Original: At bitrates where ˆ w is normally much larger than the storage cost for s and m (about 2kB per image when encoded as a vector graphic), this approach can result in large bitrate savings.
Revised: At bitrates where ˆ w is normally much larger than the storage cost for s and m (about 2kB per image when encoded as a vector graphic), this approach can result in large bitrate savings.

Original: We consider two different training modes: Random instance (RI) which randomly selects 25% of the instances in the semantic label map and preserves these, and random box (RB) which picks an image location uniformly at random and preserves a box of random dimensions.
Revised: We consider two different training modes: Random instance (RI) which randomly selects 25% of the instances in the semantic label map and preserves these, and random box (RB) which picks an image location uniformly at random and preserves a box of random dimensions.

Original: While the RI mode is appropriate for most use cases, the RB can create more challenging situations for the generator as it needs to integrate the preserved box seamlessly into the generated content.
Revised: While the RI mode is appropriate for most use cases, the RB can create more challenging situations for the generator as it needs to integrate the preserved box seamlessly into the generated content.

Original:  E XPERIMENTS 5. A RCHITECTURE , L OSSES , AND H YPERPARAMETERS
Revised:  E XPERIMENTS 5. A RCHITECTURE , L OSSES , AND H YPERPARAMETERS

Original: The architecture for our encoder E and generator G is based on the global generator network proposed in (Wang et al., 2018), which in turn is based on the architecture of (Johnson et al., 2016).
Revised: The architecture for our encoder E and generator G is based on the global generator network proposed in (Wang et al., 2018), which in turn is based on the architecture of (Johnson et al., 2016).

Original: We present details in Appendix C.
Revised: We present details in Appendix C.

Original: For the entropy term βH ( ˆ w ) , we adopt the simpliﬁed approach described in Sec. 3, where we set β = 0 , use L = 5 centers C
Revised: For the entropy term βH ( ˆ w ) , we adopt the simpliﬁed approach described in Sec. 3, where we set β = 0 , use L = 5 centers C

Original: = {− 2 , 1 , 0 , 1 , 2 } , and control the bitrate through the upper bound
Revised: = {− 2 , 1 , 0 , 1 , 2 } , and control the bitrate through the upper bound

Original: H ( ˆ w ) ≤ dim ( ˆ w ) log 2 ( L ) .
Revised: H ( ˆ w ) ≤ dim ( ˆ w ) log 2 ( L ) .

Original: For example, for GC, with C = 2 channels, we obtain 0 .
Revised: For example, for GC, with C = 2 channels, we obtain 0 .

Original: 0181 bpp.
Revised: 0181 bpp.

Original: We note that this is an upper bound; the actual entropy of H ( ˆ w ) is generally smaller, since the learned distribution will neither be uniform nor i.i.d, which would be required for the bound to hold with equality.
Revised: We note that this is an upper bound; the actual entropy of H ( ˆ w ) is generally smaller, since the learned distribution will neither be uniform nor i.i.d, which would be required for the bound to hold with equality.

Original: When encoding the channels of ˆ w to a bit-stream, we use an arithmetic encoder where frequencies are stored for each channel separately and then encode them in a static (non-adaptive)
Revised: When encoding the channels of ˆ w to a bit-stream, we use an arithmetic encoder where frequencies are stored for each channel separately and then encode them in a static (non-adaptive)

Original: [Figure]
Revised: [Figure]

Original: manner, similar to Agustsson et al.
Revised: manner, similar to Agustsson et al.

Original: In our experiments, this leads to 8 .
Revised: In our experiments, this leads to 8 .

Original: 8% smaller bitrates compared to the upper bound.
Revised: 8% smaller bitrates compared to the upper bound.

Original: By using a context model and adaptive arithmetic encoding, we could reduce the bitrate further, either in a post processing step (as in (Rippel & Bourdev, 2017; Ball ´ e et al., 2016b)), or jointly during training (as in (Mentzer et al., 2018; Minnen et al., 2018))—which led to ≈ 10% savings in these prior works.
Revised: By using a context model and adaptive arithmetic encoding, we could reduce the bitrate further, either in a post processing step (as in (Rippel & Bourdev, 2017; Ball ´ e et al., 2016b)), or jointly during training (as in (Mentzer et al., 2018; Minnen et al., 2018))—which led to ≈ 10% savings in these prior works.

Original: For the distortion term we adopt d ( x , ˆ x ) =
Revised: For the distortion term we adopt d ( x , ˆ x ) =

Original: MSE ( x , ˆ x ) with coefﬁcient λ = 10 .
Revised: MSE ( x , ˆ x ) with coefﬁcient λ = 10 .

Original: Furthermore, we adopt the feature matching and VGG perceptual losses, L FM and L VGG , as proposed in (Wang et al., 2018) with the same weights, which improved the quality for images synthesized from semantic label maps.
Revised: Furthermore, we adopt the feature matching and VGG perceptual losses, L FM and L VGG , as proposed in (Wang et al., 2018) with the same weights, which improved the quality for images synthesized from semantic label maps.

Original: These losses can be viewed as a part of d ( x , ˆ x ) .
Revised: These losses can be viewed as a part of d ( x , ˆ x ) .

Original: However, we do not mask them in SC, since they also help to stabilize the GAN in this operation mode (as in (Wang et al., 2018)).
Revised: However, we do not mask them in SC, since they also help to stabilize the GAN in this operation mode (as in (Wang et al., 2018)).

Original: We refer to Appendix D for training details.
Revised: We refer to Appendix D for training details.

Original: 5. E VALUATION
Revised: 5. E VALUATION

Original: Data sets: We train GC models (without semantic label maps) for compression of diverse natural images using 188k images from the Open Images data set (Krasin et al., 2017) and evaluate them on the widely used Kodak image compression data set (Kodak) as well as 20 randomly selected images from the RAISE1K data set (Dang-Nguyen et al., 2015).
Revised: Data sets: We train GC models (without semantic label maps) for compression of diverse natural images using 188k images from the Open Images data set (Krasin et al., 2017) and evaluate them on the widely used Kodak image compression data set (Kodak) as well as 20 randomly selected images from the RAISE1K data set (Dang-Nguyen et al., 2015).

Original: To investigate the beneﬁts of having a somewhat constrained application domain and semantic information at training time, we also train
Revised: To investigate the beneﬁts of having a somewhat constrained application domain and semantic information at training time, we also train

Original: GC models with semantic label maps on the Cityscapes data set (Cordts et al., 2016), using 20 randomly selected images from the validation set for evaluation.
Revised: GC models with semantic label maps on the Cityscapes data set (Cordts et al., 2016), using 20 randomly selected images from the validation set for evaluation.

Original: To evaluate the proposed SC method (which requires semantic label maps for training and deployment) we again rely on the Cityscapes data set.
Revised: To evaluate the proposed SC method (which requires semantic label maps for training and deployment) we again rely on the Cityscapes data set.

Original: Cityscapes was previously used to generate images form semantic label maps using GANs (Isola et al., 2017; Zhu et al., 2017).
Revised: Cityscapes was previously used to generate images form semantic label maps using GANs (Isola et al., 2017; Zhu et al., 2017).

Original: Baselines: We compare our method to the HEVC-based image compression algorithm BPG (Bellard) (in the 4:2:2 chroma format) and to the AEDC network from (Mentzer et al., 2018).
Revised: Baselines: We compare our method to the HEVC-based image compression algorithm BPG (Bellard) (in the 4:2:2 chroma format) and to the AEDC network from (Mentzer et al., 2018).

Original: BPG is the current state-of-the-art engineered image compression codec and outperforms other recent codecs such as JPEG2000 and WebP on different data sets in terms of PSNR (see, e.g. (Ball´e et al., 2018)).
Revised: BPG is the current state-of-the-art engineered image compression codec and outperforms other recent codecs such as JPEG2000 and WebP on different data sets in terms of PSNR (see, e.g. (Ball´e et al., 2018)).

Original: We train the AEDC network (with bottleneck depth C = 4 ) on Cityscapes exactly following the procedure in (Mentzer et al., 2018) except that we use early stopping to prevent overﬁtting (note that Cityscapes is much smaller than the ImageNet dataset used in (Mentzer et al., 2018)).
Revised: We train the AEDC network (with bottleneck depth C = 4 ) on Cityscapes exactly following the procedure in (Mentzer et al., 2018) except that we use early stopping to prevent overﬁtting (note that Cityscapes is much smaller than the ImageNet dataset used in (Mentzer et al., 2018)).

Original: The so-obtained model has a bitrate of 0.07 bpp and gets a slightly better MS-SSIM than BPG at the same bpp on the validation set.
Revised: The so-obtained model has a bitrate of 0.07 bpp and gets a slightly better MS-SSIM than BPG at the same bpp on the validation set.

Original: To investigate the effect of the GAN term in our total loss, we train a baseline model with an MSE loss only (with the same architecture as GC and the same training parameters, see Sec.
Revised: To investigate the effect of the GAN term in our total loss, we train a baseline model with an MSE loss only (with the same architecture as GC and the same training parameters, see Sec.

Original: D in the Appendix), referred to as “MSE baseline”.
Revised: D in the Appendix), referred to as “MSE baseline”.

Original: [Figure]
Revised: [Figure]

Original: User study:
Revised: User study:

Original: In the extreme compression regime realized by our GC models, where texture and sometimes even more abstract image content is synthesized, common reconstruction quality measures such as PSNR and MS-SSIM arguably lose signiﬁcance as they penalize changes in local structure rather than assessing preservation of the global image content (this also becomes apparent by comparing reconstructions produced by our GC model with those obtained by the MSE baseline and BPG, see Fig. 2).
Revised: In the extreme compression regime realized by our GC models, where texture and sometimes even more abstract image content is synthesized, common reconstruction quality measures such as PSNR and MS-SSIM arguably lose signiﬁcance as they penalize changes in local structure rather than assessing preservation of the global image content (this also becomes apparent by comparing reconstructions produced by our GC model with those obtained by the MSE baseline and BPG, see Fig. 2).

Original: Indeed, measuring PSNR between synthesized and real texture patches essentially quantiﬁes the variance of the texture rather than the visual quality of the synthesized texture.
Revised: Indeed, measuring PSNR between synthesized and real texture patches essentially quantiﬁes the variance of the texture rather than the visual quality of the synthesized texture.

Original: To quantitatively evaluate the perceptual quality of our GC models in comparison with BPG and
Revised: To quantitatively evaluate the perceptual quality of our GC models in comparison with BPG and

Original: AEDC (for Cityscapes) we therefore conduct a user study using Amazon Mechanical Turk (AMT).
Revised: AEDC (for Cityscapes) we therefore conduct a user study using Amazon Mechanical Turk (AMT).

Original: We consider two GC models with C = 4 , 8 trained on Open Images, three GC ( D + ) models with
Revised: We consider two GC models with C = 4 , 8 trained on Open Images, three GC ( D + ) models with

Original: C = 2 , 4 , 8 trained on Cityscapes, and BPG at rates ranging from 0 .
Revised: C = 2 , 4 , 8 trained on Cityscapes, and BPG at rates ranging from 0 .

Original: 045 to 0 .
Revised: 045 to 0 .

Original: 12 bpp.
Revised: 12 bpp.

Original: Questionnaires are composed by combining the reconstructions produced by the selected GC model for all testing images with the corresponding reconstruction produced by the competing baseline model side-byside (presenting the reconstructions in random order).
Revised: Questionnaires are composed by combining the reconstructions produced by the selected GC model for all testing images with the corresponding reconstruction produced by the competing baseline model side-byside (presenting the reconstructions in random order).

Original: The original image is shown along with the reconstructions, and the pairwise comparisons are interleaved with 3 probing comparisons of an additional uncompressed image from the respective testing set with an obviously JPEG-compressed version of that image.
Revised: The original image is shown along with the reconstructions, and the pairwise comparisons are interleaved with 3 probing comparisons of an additional uncompressed image from the respective testing set with an obviously JPEG-compressed version of that image.

Original: 20 randomly selected unique users are asked to indicate their preference for each pair of reconstructions in the questionnaire, resulting in a total of 480 ratings per pairing of methods for Kodak, and 400 ratings for RAISE1K and Cityscapes.
Revised: 20 randomly selected unique users are asked to indicate their preference for each pair of reconstructions in the questionnaire, resulting in a total of 480 ratings per pairing of methods for Kodak, and 400 ratings for RAISE1K and Cityscapes.

Original: For each pairing of methods, we report the mean preference score as well as the standard error (SE) of the per-user mean preference percentages.
Revised: For each pairing of methods, we report the mean preference score as well as the standard error (SE) of the per-user mean preference percentages.

Original: Only users correctly identifying the original image in all probing comparisons are taken into account for the mean preference percentage computation.
Revised: Only users correctly identifying the original image in all probing comparisons are taken into account for the mean preference percentage computation.

Original: To facilitate comparisons for future works, we will release all images used in the user studies.
Revised: To facilitate comparisons for future works, we will release all images used in the user studies.

Original: Semantic quality of SC models: The issues with PSNR and MS-SSIM for evaluating the quality of generated content described in the previous paragraph become even more severe for SC models as a large fraction of the image content is generated from a semantic label map.
Revised: Semantic quality of SC models: The issues with PSNR and MS-SSIM for evaluating the quality of generated content described in the previous paragraph become even more severe for SC models as a large fraction of the image content is generated from a semantic label map.

Original: Following image translation works Isola et al.
Revised: Following image translation works Isola et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: , we therefore measure the capacity of our
Revised: , we therefore measure the capacity of our

Original: SC models to preserve the image semantics in the synthesized regions and plausibly blend them with the preserved regions—the objective SC models are actually trained for.
Revised: SC models to preserve the image semantics in the synthesized regions and plausibly blend them with the preserved regions—the objective SC models are actually trained for.

Original: Speciﬁcally, we use
Revised: Speciﬁcally, we use

Original: PSPNet (Zhao et al., 2016) and compute the mean intersection-over-union (IoU) between the label map obtained for the decompressed validation images and the ground truth label map.
Revised: PSPNet (Zhao et al., 2016) and compute the mean intersection-over-union (IoU) between the label map obtained for the decompressed validation images and the ground truth label map.

Original: For reference we also report this metric for baselines that do not use semantic label maps for training and/or deployment.
Revised: For reference we also report this metric for baselines that do not use semantic label maps for training and/or deployment.

Original:  R ESULTS 6. G ENERATIVE COMPRESSION
Revised:  R ESULTS 6. G ENERATIVE COMPRESSION

Original: Fig.
Revised: Fig.

Original: 4 shows the mean preference percentage obtained by our GC models compared to BPG at different rates, on the Kodak and the RAISE1K data set.
Revised: 4 shows the mean preference percentage obtained by our GC models compared to BPG at different rates, on the Kodak and the RAISE1K data set.

Original: In addition, we report the mean preference percentage for GC models compared to BPG and AEDC on Cityscapes.
Revised: In addition, we report the mean preference percentage for GC models compared to BPG and AEDC on Cityscapes.

Original: Example validation images for side-by-side comparison of our method with BPG for images from the Kodak, RAISE1K, and
Revised: Example validation images for side-by-side comparison of our method with BPG for images from the Kodak, RAISE1K, and

Original: Cityscapes data set can be found in Figs.
Revised: Cityscapes data set can be found in Figs.

Original: 1, 3, and 2, respectively.
Revised: 1, 3, and 2, respectively.

Original: Furthermore, we perform extensive visual comparisons of all our methods and the baselines, presented in Appendix F.
Revised: Furthermore, we perform extensive visual comparisons of all our methods and the baselines, presented in Appendix F.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Our
Revised: Our

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Users Preferring Our GC to BPG on Kodak
Revised: Users Preferring Our GC to BPG on Kodak

Original: [Table]
Revised: [Table]

Original: Our GC
Revised: Our GC

Original: BPG on RAISE1K
Revised: BPG on RAISE1K

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and
Revised: and

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C =4 ( 0.033bpp )
Revised: C =4 ( 0.033bpp )

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: GC C =4 preferred
Revised: GC C =4 preferred

Original: GC C =8 preferred
Revised: GC C =8 preferred

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Users Preferring Our GC to BPG on RAISE1K
Revised: Users Preferring Our GC to BPG on RAISE1K

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table]
Revised: [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: GC C =4 preferred
Revised: GC C =4 preferred

Original: GC C
Revised: GC C

Original: [Table]
Revised: [Table]

Original: 8 preferred
Revised: 8 preferred

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: Our GC
Revised: Our GC

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 
Revised:

Original: Our GC models with C = 4 are preferred to BPG even when images produced by BPG use 95% and 124% more bits than those produced by our models for Kodak and RAISE1K, respectively.
Revised: Our GC models with C = 4 are preferred to BPG even when images produced by BPG use 95% and 124% more bits than those produced by our models for Kodak and RAISE1K, respectively.

Original: Notably this is achieved even though there is a distribution shift between the training and testing set (recall that these GC models are trained on Open Images).
Revised: Notably this is achieved even though there is a distribution shift between the training and testing set (recall that these GC models are trained on Open Images).

Original: The gains of domain-speciﬁcity and semantic label maps (for training) becomes apparent from the results on Cityscapes: Our GC models with
Revised: The gains of domain-speciﬁcity and semantic label maps (for training) becomes apparent from the results on Cityscapes: Our GC models with

Original: C = 2 are preferred to BPG even when the latter uses 181% more bits.
Revised: C = 2 are preferred to BPG even when the latter uses 181% more bits.

Original: For C = 4 the gains on
Revised: For C = 4 the gains on

Original: Cityscapes are comparable to those obtained for GC on RAISE1K. For all three data sets, BPG requires between 21 and 49% more bits than our GC models with C = 8 .
Revised: Cityscapes are comparable to those obtained for GC on RAISE1K. For all three data sets, BPG requires between 21 and 49% more bits than our GC models with C = 8 .

Original: Discussion: The GC models produce images with much ﬁner detail than BPG, which suffers from smoothed patches and blocking artifacts.
Revised: Discussion: The GC models produce images with much ﬁner detail than BPG, which suffers from smoothed patches and blocking artifacts.

Original: In particular, the GC models convincingly reconstruct texture in natural objects such as trees, water, and sky, and is most challenged with scenes involving humans.
Revised: In particular, the GC models convincingly reconstruct texture in natural objects such as trees, water, and sky, and is most challenged with scenes involving humans.

Original: AEDC and the MSE baseline both produce blurry images.
Revised: AEDC and the MSE baseline both produce blurry images.

Original: We see that the gains of our models are maximal at extreme bitrates, with BPG needing 95–181% more bits for the C
Revised: We see that the gains of our models are maximal at extreme bitrates, with BPG needing 95–181% more bits for the C

Original: = 2 , 4 models on the three datasets.
Revised: = 2 , 4 models on the three datasets.

Original: For C = 8 gains are smaller but still very large (BPG needing 21–49% more bits).
Revised: For C = 8 gains are smaller but still very large (BPG needing 21–49% more bits).

Original: This is expected, since as the bitrate increases the classical compression measures (PSNR/MS-SSIM) become more meaningful—and our system does not employ the full complexity of current state-of-the-art systems, as discussed next.
Revised: This is expected, since as the bitrate increases the classical compression measures (PSNR/MS-SSIM) become more meaningful—and our system does not employ the full complexity of current state-of-the-art systems, as discussed next.

Original: State-of-the-art on Kodak: We give an overview of relevant recent learned compression methods and their differences to our GC method and BPG in Table 1 in the Appendix.
Revised: State-of-the-art on Kodak: We give an overview of relevant recent learned compression methods and their differences to our GC method and BPG in Table 1 in the Appendix.

Original: Rippel & Bourdev (2017) also used GANs (albeit a different formulation) and were state-of-the-art in MS-SSIM in 2017, while the concurrent work of Minnen et al.
Revised: Rippel & Bourdev (2017) also used GANs (albeit a different formulation) and were state-of-the-art in MS-SSIM in 2017, while the concurrent work of Minnen et al.

Original: (2018) is the current state-of-the-art in image compression in terms of classical metrics (PSNR and MS-SSIM) when measured on the Kodak dataset (Kodak).
Revised: (2018) is the current state-of-the-art in image compression in terms of classical metrics (PSNR and MS-SSIM) when measured on the Kodak dataset (Kodak).

Original: Notably, all methods except ours (BPG, Rippel et al., and Minnen et al.) employ adaptive arithmetic coding using context models for improved compression performance.
Revised: Notably, all methods except ours (BPG, Rippel et al., and Minnen et al.) employ adaptive arithmetic coding using context models for improved compression performance.

Original: Such models could also be implemented for our system, and have led to additional savings of 10% in
Revised: Such models could also be implemented for our system, and have led to additional savings of 10% in

Original: Mentzer et al.
Revised: Mentzer et al.

Original: Since Rippel et al.
Revised: Since Rippel et al.

Original: and Minnen et al.
Revised: and Minnen et al.

Original: have only released a selection of their decoded images (for 3 and 4, respectively, out of the 24 Kodak images), and at signiﬁcantly higher bitrates, a comparison with a user study is not meaningful.
Revised: have only released a selection of their decoded images (for 3 and 4, respectively, out of the 24 Kodak images), and at signiﬁcantly higher bitrates, a comparison with a user study is not meaningful.

Original: Instead, we try to qualitatively put our results into context with theirs.
Revised: Instead, we try to qualitatively put our results into context with theirs.

Original: In Figs.
Revised: In Figs.

Original: 12–14 in the Appendix, we compare qualitatively to Rippel & Bourdev (2017).
Revised: 12–14 in the Appendix, we compare qualitatively to Rippel & Bourdev (2017).

Original: We can observe that even though Rippel & Bourdev (2017) use 29–179% more bits, our models produce images of comparable or better quality.
Revised: We can observe that even though Rippel & Bourdev (2017) use 29–179% more bits, our models produce images of comparable or better quality.

Original: [Figure]
Revised: [Figure]

Original: In Figs.
Revised: In Figs.

Original: 15–18, we show a qualitative comparison of our results to the images provided by the concurrent work of Minnen et al.
Revised: 15–18, we show a qualitative comparison of our results to the images provided by the concurrent work of Minnen et al.

Original: (2018), as well as to BPG (Bellard) on those images.
Revised: (2018), as well as to BPG (Bellard) on those images.

Original: First, we see that BPG is still visually competitive with the current state-of-the-art, which is consistent with moderate 8 .
Revised: First, we see that BPG is still visually competitive with the current state-of-the-art, which is consistent with moderate 8 .

Original: 41% bitrate savings being reported by Minnen et al.
Revised: 41% bitrate savings being reported by Minnen et al.

Original: in terms of PSNR.
Revised: in terms of PSNR.

Original: Second, even though we use much fewer bits compared to the example images available from Minnen et al.
Revised: Second, even though we use much fewer bits compared to the example images available from Minnen et al.

Original: (2018), for some of them (Figs. 15 and 16) our method can still produce images of comparable visual quality.
Revised: (2018), for some of them (Figs. 15 and 16) our method can still produce images of comparable visual quality.

Original: Given the dramatic bitrate savings we achieve according to the user study (BPG needing 21–181% more bits), and the competitiveness of BPG to the most recent state-of-the-art (Minnen et al., 2018), we conclude that our proposed system presents a signiﬁcant step forward for visually pleasing compression at extreme bitrates.
Revised: Given the dramatic bitrate savings we achieve according to the user study (BPG needing 21–181% more bits), and the competitiveness of BPG to the most recent state-of-the-art (Minnen et al., 2018), we conclude that our proposed system presents a signiﬁcant step forward for visually pleasing compression at extreme bitrates.

Original: Sampling the compressed representations: In Fig.
Revised: Sampling the compressed representations: In Fig.

Original: 5 we explore the representation learned by our
Revised: 5 we explore the representation learned by our

Original: GC models (with C = 4 ), by sampling the (discrete) latent space of ˆ w .
Revised: GC models (with C = 4 ), by sampling the (discrete) latent space of ˆ w .

Original: When we sample uniformly, and decode with our GC model into images, we obtain a “soup of image patches” which reﬂects the domain the models were trained on (e.g. street sign and building patches on Cityscapes).
Revised: When we sample uniformly, and decode with our GC model into images, we obtain a “soup of image patches” which reﬂects the domain the models were trained on (e.g. street sign and building patches on Cityscapes).

Original: Note that we should not expect these outputs to look like normal images, since nothing forces the encoder output ˆ w to be uniformly distributed over the discrete latent space.
Revised: Note that we should not expect these outputs to look like normal images, since nothing forces the encoder output ˆ w to be uniformly distributed over the discrete latent space.

Original: However, given the low dimensionality of ˆ w ( 32 × 64 × 4 for 512 × 1024 px Cityscape images), it would be interesting to try to learn the true distribution.
Revised: However, given the low dimensionality of ˆ w ( 32 × 64 × 4 for 512 × 1024 px Cityscape images), it would be interesting to try to learn the true distribution.

Original: To this end, we perform a simple experiment and train an improved Wasserstein GAN (WGAN-GP) (Gulrajani et al., 2017) on ˆ w extracted from Cityscapes, using default parameters and a ResNet architecture.
Revised: To this end, we perform a simple experiment and train an improved Wasserstein GAN (WGAN-GP) (Gulrajani et al., 2017) on ˆ w extracted from Cityscapes, using default parameters and a ResNet architecture.

Original: By feeding our GC model with samples from the WGAN-GP generator, we easily obtain a powerful generative model, which generates sharp 1024 × 512 px images from scratch .
Revised: By feeding our GC model with samples from the WGAN-GP generator, we easily obtain a powerful generative model, which generates sharp 1024 × 512 px images from scratch .

Original: We think this could be a promising direction for building high-resolution generative models.
Revised: We think this could be a promising direction for building high-resolution generative models.

Original: In Figs.
Revised: In Figs.

Original: 19–21 in the Appendix, we show more samples, and samples obtained by feeding the MSE baseline with uniform and learned code samples.
Revised: 19–21 in the Appendix, we show more samples, and samples obtained by feeding the MSE baseline with uniform and learned code samples.

Original: The latter yields noisier “patch soups” and much blurrier image samples than our GC network.
Revised: The latter yields noisier “patch soups” and much blurrier image samples than our GC network.

Original: [Figure]
Revised: [Figure]

Original: 6. S ELECTIVE GENERATIVE COMPRESSION
Revised: 6. S ELECTIVE GENERATIVE COMPRESSION

Original: Fig.
Revised: Fig.

Original: 6 shows the mean IoU on the Cityscapes validation set as a function of bpp for SC networks with C = 2 , 4 , 8 , along with the values obtained for the baselines.
Revised: 6 shows the mean IoU on the Cityscapes validation set as a function of bpp for SC networks with C = 2 , 4 , 8 , along with the values obtained for the baselines.

Original: Additionally, we plot mean IoU for GC with semantic label maps fed to the discriminator ( D + ), and the MSE baseline.
Revised: Additionally, we plot mean IoU for GC with semantic label maps fed to the discriminator ( D + ), and the MSE baseline.

Original: [Figure]
Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: 7 we present example Cityscapes validation images produced by the SC network trained in the RI mode with C = 8 , where different semantic classes are preserved.
Revised: 7 we present example Cityscapes validation images produced by the SC network trained in the RI mode with C = 8 , where different semantic classes are preserved.

Original: More visual results for the
Revised: More visual results for the

Original: SC networks trained on Cityscapes can be found in Appendix F.7, including results obtained for the
Revised: SC networks trained on Cityscapes can be found in Appendix F.7, including results obtained for the

Original: RB operation mode and by using semantic label maps estimated from the input image via PSPNet (Zhao et al., 2017).
Revised: RB operation mode and by using semantic label maps estimated from the input image via PSPNet (Zhao et al., 2017).

Original: Discussion: The quantitative evaluation of the semantic preservation capacity (Fig. 6) reveals that the SC networks preserve the semantics somewhat better than pix2pixHD, indicating that
Revised: Discussion: The quantitative evaluation of the semantic preservation capacity (Fig. 6) reveals that the SC networks preserve the semantics somewhat better than pix2pixHD, indicating that

Original: the
Revised: the

Original: SC networks faithfully generate texture from the label maps and plausibly combine generated with preserved image content.
Revised: SC networks faithfully generate texture from the label maps and plausibly combine generated with preserved image content.

Original: The mIoU of BPG, AEDC, and the MSE baseline is considerably lower than that obtained by our SC and GC models, which can arguably be attributed to blurring and blocking artifacts.
Revised: The mIoU of BPG, AEDC, and the MSE baseline is considerably lower than that obtained by our SC and GC models, which can arguably be attributed to blurring and blocking artifacts.

Original: However, it is not surprising as these baseline methods do not use label maps during training and prediction.
Revised: However, it is not surprising as these baseline methods do not use label maps during training and prediction.

Original: In the SC operation mode, our networks manage to seamlessly merge preserved and generated image content both when preserving object instances and boxes crossing object boundaries (see Appendix F.7).
Revised: In the SC operation mode, our networks manage to seamlessly merge preserved and generated image content both when preserving object instances and boxes crossing object boundaries (see Appendix F.7).

Original: Further, our networks lead to reductions in bpp of 50% and more compared to the same networks without synthesis, while leaving the visual quality essentially unimpaired, when objects with repetitive structure are synthesized (such as trees, streets, and sky).
Revised: Further, our networks lead to reductions in bpp of 50% and more compared to the same networks without synthesis, while leaving the visual quality essentially unimpaired, when objects with repetitive structure are synthesized (such as trees, streets, and sky).

Original: In some cases, the visual quality is even better than that of BPG at the same bitrate.
Revised: In some cases, the visual quality is even better than that of BPG at the same bitrate.

Original: The visual quality of more complex synthesized objects (e.g. buildings, people) is worse.
Revised: The visual quality of more complex synthesized objects (e.g. buildings, people) is worse.

Original: However, this is a limitation of current GAN technology rather than our approach.
Revised: However, this is a limitation of current GAN technology rather than our approach.

Original: As the visual quality of GANs improves further, SC networks will as well.
Revised: As the visual quality of GANs improves further, SC networks will as well.

Original: Notably, the SC networks can generate entire images from the semantic label map only.
Revised: Notably, the SC networks can generate entire images from the semantic label map only.

Original: Finally, the semantic label map, which requires 0.036 bpp on average for the downscaled 1024 × 512 px Cityscapes images, represents a relatively large overhead compared to the storage cost of the preserved image parts.
Revised: Finally, the semantic label map, which requires 0.036 bpp on average for the downscaled 1024 × 512 px Cityscapes images, represents a relatively large overhead compared to the storage cost of the preserved image parts.

Original: This cost vanishes as the image size increases, since the semantic mask can be stored as an image dimension-independent vector graphic.
Revised: This cost vanishes as the image size increases, since the semantic mask can be stored as an image dimension-independent vector graphic.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed and evaluated a GAN-based framework for learned compression that signiﬁcantly outperforms prior works for low bitrates in terms of visual quality, for compression of natural images.
Revised: We proposed and evaluated a GAN-based framework for learned compression that signiﬁcantly outperforms prior works for low bitrates in terms of visual quality, for compression of natural images.

Original: Furthermore, we demonstrated that constraining the application domain to street scene images leads to additional storage savings, and we explored combining synthesized with preserved image content with the potential to achieve even larger savings.
Revised: Furthermore, we demonstrated that constraining the application domain to street scene images leads to additional storage savings, and we explored combining synthesized with preserved image content with the potential to achieve even larger savings.

Original: Interesting directions for future work are to develop a mechanism for controlling spatial allocation of bits for GC (e.g. to achieve better preservation of faces; possibly using semantic label maps), and to combine SC with saliency information to determine what regions to preserve.
Revised: Interesting directions for future work are to develop a mechanism for controlling spatial allocation of bits for GC (e.g. to achieve better preservation of faces; possibly using semantic label maps), and to combine SC with saliency information to determine what regions to preserve.

Original: In addition, the sampling experiments presented in Sec.
Revised: In addition, the sampling experiments presented in Sec.

Original: 6.1 indicate that combining our GC compression approach with GANs to (unconditionally) generate compressed representations is a promising avenue to learn high-resolution generative models.
Revised: 6.1 indicate that combining our GC compression approach with GANs to (unconditionally) generate compressed representations is a promising avenue to learn high-resolution generative models.

Original: A C OMPARISON WITH S TATE - OF - THE - ART BPG
Revised: A C OMPARISON WITH S TATE - OF - THE - ART BPG

Original: Rippel et al.
Revised: Rippel et al.

Original: Minnen et al.
Revised: Minnen et al.

Original: Ours (GC) (2017)
Revised: Ours (GC) (2017)

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Savings to BPG in PSNR
Revised: Savings to BPG in PSNR

Original: [Table]
Revised: [Table]

Original: Savings to BPG in User Study
Revised: Savings to BPG in User Study

Original: [Table]
Revised: [Table]

Original: B C OMPRESSION D ETAILS
Revised: B C OMPRESSION D ETAILS

Original: When encoding the channels of ˆ w to a bit-stream, we use an arithmetic encoder where frequencies are stored for each channel separately and then encode them in a static (non-adaptive) manner, similar to Agustsson et al.
Revised: When encoding the channels of ˆ w to a bit-stream, we use an arithmetic encoder where frequencies are stored for each channel separately and then encode them in a static (non-adaptive) manner, similar to Agustsson et al.

Original: In our experiments, this leads to 8 .
Revised: In our experiments, this leads to 8 .

Original: 8% smaller bitrates compared to the upper bound.
Revised: 8% smaller bitrates compared to the upper bound.

Original: We compress the semantic label map for SC by quantizing the coordinates in the vector graphic to the image grid and encoding coordinates relative to preceding coordinates when traversing object boundaries (rather than relative to the image frame).
Revised: We compress the semantic label map for SC by quantizing the coordinates in the vector graphic to the image grid and encoding coordinates relative to preceding coordinates when traversing object boundaries (rather than relative to the image frame).

Original: The so-obtained bitstream is then compressed using arithmetic coding.
Revised: The so-obtained bitstream is then compressed using arithmetic coding.

Original: To ensure fair comparison, we do not count header sizes for any of the baseline methods throughout.
Revised: To ensure fair comparison, we do not count header sizes for any of the baseline methods throughout.

Original: C A RCHITECTURE D ETAILS
Revised: C A RCHITECTURE D ETAILS

Original: For the GC, the encoder E convolutionally processes the image x and optionally the label map s , with spatial dimension W × H , into a feature map of size W / 16 × H / 16 × 960 (with 6 layers, of which four have 2 -strided convolutions), which is then projected down to C channels (where C ∈ { 2 , 4 , 8 } is much smaller than 960 ).
Revised: For the GC, the encoder E convolutionally processes the image x and optionally the label map s , with spatial dimension W × H , into a feature map of size W / 16 × H / 16 × 960 (with 6 layers, of which four have 2 -strided convolutions), which is then projected down to C channels (where C ∈ { 2 , 4 , 8 } is much smaller than 960 ).

Original: This results in a feature map w of dimension W / 16 × H / 16 × C , which is quantized over L centers to obtain the discrete ˆ w .
Revised: This results in a feature map w of dimension W / 16 × H / 16 × C , which is quantized over L centers to obtain the discrete ˆ w .

Original: The generator G projects ˆ w up to 960 channels, processes these with 9 residual units (He et al., 2016) at dimension W / 16 × H / 16 × 960 , and then mirrors E by convolutionally processing the features back to spatial dimensions W × H (with transposed convolutions instead of strided ones).
Revised: The generator G projects ˆ w up to 960 channels, processes these with 9 residual units (He et al., 2016) at dimension W / 16 × H / 16 × 960 , and then mirrors E by convolutionally processing the features back to spatial dimensions W × H (with transposed convolutions instead of strided ones).

Original: Similar to E , the feature extractor F for SC processes the semantic map s down to the spatial dimension of ˆ w , which is then concatenated to ˆ w for generation.
Revised: Similar to E , the feature extractor F for SC processes the semantic map s down to the spatial dimension of ˆ w , which is then concatenated to ˆ w for generation.

Original: In this case, we consider slightly higher bitrates and downscale by 8 × instead of 16 × in the encoder E , such that dim ( ˆ w ) =
Revised: In this case, we consider slightly higher bitrates and downscale by 8 × instead of 16 × in the encoder E , such that dim ( ˆ w ) =

Original: W / 8 × H / 8 × C .
Revised: W / 8 × H / 8 × C .

Original: The generator then ﬁrst processes ˆ w down to W / 16 × H / 16 × 960 and then proceeds as for
Revised: The generator then ﬁrst processes ˆ w down to W / 16 × H / 16 × 960 and then proceeds as for

Original: GC.
Revised: GC.

Original: For both GC and SC, we use the multi-scale architecture of (Wang et al., 2018) for the discriminator
Revised: For both GC and SC, we use the multi-scale architecture of (Wang et al., 2018) for the discriminator

Original: D , which measures the divergence between p x and p G ( z ) both locally and globally.
Revised: D , which measures the divergence between p x and p G ( z ) both locally and globally.

Original: We adopt the notation from (Wang et al., 2018) to describe our encoder and generator/decoder architectures and additionally use q to denote the quantization layer (see Sec.
Revised: We adopt the notation from (Wang et al., 2018) to describe our encoder and generator/decoder architectures and additionally use q to denote the quantization layer (see Sec.

Original: 3 for details).
Revised: 3 for details).

Original: The output of q is encoded and stored.
Revised: The output of q is encoded and stored.

Original: • Encoder GC: c7s1-60,d120,d240,d480,d960,c3s1- C ,q
Revised: • Encoder GC: c7s1-60,d120,d240,d480,d960,c3s1- C ,q

Original: • Encoders SC: – Semantic label map encoder: c7s1-60,d120,d240,d480,d – Image encoder:
Revised: • Encoders SC: – Semantic label map encoder: c7s1-60,d120,d240,d480,d – Image encoder:

Original: c7s1-60,d120,d240,d480,c3s1- C ,q,c3s1-480,d
Revised: c7s1-60,d120,d240,d480,c3s1- C ,q,c3s1-480,d

Original: The outputs of the semantic label map encoder and the image encoder are concatenated and fed to the generator/decoder
Revised: The outputs of the semantic label map encoder and the image encoder are concatenated and fed to the generator/decoder

Original: • Generator/decoder: c3s1-960,R960,R960,R960,R960,R960,R960,R960, R960,R960,u480,u240,u120,u60,c7s1-
Revised: • Generator/decoder: c3s1-960,R960,R960,R960,R960,R960,R960,R960, R960,R960,u480,u240,u120,u60,c7s1-

Original: [Figure]
Revised: [Figure]

Original: D T RAINING D ETAILS
Revised: D T RAINING D ETAILS

Original: We employ the ADAM optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 0002 and set the mini-batch size to 1 .
Revised: We employ the ADAM optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 0002 and set the mini-batch size to 1 .

Original: Our networks are trained for 150000 iterations on Cityscapes and foriterations on Open Images.
Revised: Our networks are trained for 150000 iterations on Cityscapes and foriterations on Open Images.

Original: For normalization we used instance normalization (Ulyanov et al., 2016), except in the second half of the Open Images training, we train the generator/decoder with ﬁxed batch statistics (as implemented in the test mode of batch normalization (Ioffe & Szegedy, 2015)), since we found this reduced artifacts and color shift.
Revised: For normalization we used instance normalization (Ulyanov et al., 2016), except in the second half of the Open Images training, we train the generator/decoder with ﬁxed batch statistics (as implemented in the test mode of batch normalization (Ioffe & Szegedy, 2015)), since we found this reduced artifacts and color shift.

Original: E D ATASET AND P REPROCESSING D ETAILS
Revised: E D ATASET AND P REPROCESSING D ETAILS

Original: To train GC models (which do not require semantic label maps, neither during training nor for deployment) for compression of diverse natural images, we use 200k images sampled randomly from the Open Images data set (Krasin et al., 2017) (9M images).
Revised: To train GC models (which do not require semantic label maps, neither during training nor for deployment) for compression of diverse natural images, we use 200k images sampled randomly from the Open Images data set (Krasin et al., 2017) (9M images).

Original: The training images are rescaled so that the longer side has length 768 px, and images for which rescaling does not result in at least 1 .
Revised: The training images are rescaled so that the longer side has length 768 px, and images for which rescaling does not result in at least 1 .

Original: 25 × downscaling as well as high saturation images (average S > 0 .
Revised: 25 × downscaling as well as high saturation images (average S > 0 .

Original: 9 or V > 0 . 

Revised: 9 or V > 0 .

Original: in HSV color space) are discarded (resulting in an effective training set size of 188k).
Revised: in HSV color space) are discarded (resulting in an effective training set size of 188k).

Original: We evaluate these models on the Kodak image compression dataset (Kodak) (24 images, 768 × 512 px), which has a long tradition in the image compression literature and is still the most frequently used dataset for comparisons of learned image compression methods.
Revised: We evaluate these models on the Kodak image compression dataset (Kodak) (24 images, 768 × 512 px), which has a long tradition in the image compression literature and is still the most frequently used dataset for comparisons of learned image compression methods.

Original: Additionally, we evaluate our GC models on 20 randomly selected images from the RAISE1K data set (Dang-Nguyen et al., 2015), a real-world image dataset consisting of 8156 high-resolution RAW images (we rescale the images such that the longer side has length 768 px).
Revised: Additionally, we evaluate our GC models on 20 randomly selected images from the RAISE1K data set (Dang-Nguyen et al., 2015), a real-world image dataset consisting of 8156 high-resolution RAW images (we rescale the images such that the longer side has length 768 px).

Original: To investigate the beneﬁts of having a somewhat constrained application domain and semantic labels at training time, we also train GC models with semantic label maps on the Cityscapes data set (Cordts et al., 2016) (2975 training and 500 validation images, 34 classes, 2048 × 1024 px resolution) consisting of street scene images and evaluate it on 20 randomly selected validation images (without semantic labels).
Revised: To investigate the beneﬁts of having a somewhat constrained application domain and semantic labels at training time, we also train GC models with semantic label maps on the Cityscapes data set (Cordts et al., 2016) (2975 training and 500 validation images, 34 classes, 2048 × 1024 px resolution) consisting of street scene images and evaluate it on 20 randomly selected validation images (without semantic labels).

Original: Both training and validation images are rescaled to 1024 × 512 px resolution.
Revised: Both training and validation images are rescaled to 1024 × 512 px resolution.

Original: To evaluate the proposed SC method (which requires semantic label maps for training and deployment) we again rely on the Cityscapes data set.
Revised: To evaluate the proposed SC method (which requires semantic label maps for training and deployment) we again rely on the Cityscapes data set.

Original: Cityscapes was previously used to generate images form semantic label maps using GANs (Isola et al., 2017; Zhu et al., 2017).
Revised: Cityscapes was previously used to generate images form semantic label maps using GANs (Isola et al., 2017; Zhu et al., 2017).

Original: The preprocessing for
Revised: The preprocessing for

Original: SC is the same as for GC.
Revised: SC is the same as for GC.

Original: F V ISUALS
Revised: F V ISUALS

Original: In the following Sections, F.1, F.2, F.3, we show the ﬁrst ﬁve images of each of the three datasets we used for the user study, next to the outputs of BPG at similar bitrates.
Revised: In the following Sections, F.1, F.2, F.3, we show the ﬁrst ﬁve images of each of the three datasets we used for the user study, next to the outputs of BPG at similar bitrates.

Original: Secs.
Revised: Secs.

Original: F.4 and F.5 provide visual comparisons of our GC models with Rippel & Bourdev (2017) and
Revised: F.4 and F.5 provide visual comparisons of our GC models with Rippel & Bourdev (2017) and

Original: Minnen et al.
Revised: Minnen et al.

Original: (2018), respectively, on a subset of images form the Kodak data set.
Revised: (2018), respectively, on a subset of images form the Kodak data set.

Original: In Section F.6, we show visualizations of the latent representation of our GC models.
Revised: In Section F.6, we show visualizations of the latent representation of our GC models.

Original: Finally, Section F.7 presents additional visual results for SC.
Revised: Finally, Section F.7 presents additional visual results for SC.

Original: F. G ENERATIVE C OMPRESSION ON K ODAK
Revised: F. G ENERATIVE C OMPRESSION ON K ODAK

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: F. G ENERATIVE C OMPRESSION ON C ITYSCAPES
Revised: F. G ENERATIVE C OMPRESSION ON C ITYSCAPES

Original: [Figure]
Revised: [Figure]

Original: bpp
Revised: bpp

Original: [Figure]
Revised: [Figure]

Original: bpp
Revised: bpp

Original: [Figure]
Revised: [Figure]

Original: bpp
Revised: bpp

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C OMPARISON TO
Revised: C OMPARISON TO

Original: [Figure]
Revised: [Figure]

Original: IPPEL
Revised: IPPEL

Original: [Figure]
Revised: [Figure]

Original: OURDEV
Revised: OURDEV

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: INNEN
Revised: INNEN

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Original
Revised: Original

Original: Ours, 0 . 03418 bpp
Revised: Ours, 0 . 03418 bpp

Original: Minnen et al., 0 .
Revised: Minnen et al., 0 .

Original: 123 bpp, 259% larger ,
Revised: 123 bpp, 259% larger ,

Original: BPG, 0 .
Revised: BPG, 0 .

Original: 119 bpp
Revised: 119 bpp

Original: C it y s ca p e s
Revised: C it y s ca p e s

Original: O p e n
Revised: O p e n

Original: I m a g e s
Revised: I m a g e s

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: F. A DDITIONAL RESULTS
Revised: F. A DDITIONAL RESULTS

Original: We collect here additional results for the discussion with the reviewers, so that they are easily found.
Revised: We collect here additional results for the discussion with the reviewers, so that they are easily found.

Original: We will integrate these results into the paper.
Revised: We will integrate these results into the paper.

Revised: In Table 2 we compute the PSNR on the Cityscapes test set, when varying the entropy constraint (i.e. changing C ), and the two extremes (a) when MSE is only optimized and (b) when the GAN loss is only optimized.

Revised: The ﬁrst three rows shows as the entropy constraint is relaxed, the network can more easily optimize the distortion term leading to a higher PSNR.

Revised: The fourth row shows that when optimizing for MSE only (see Fig.7 for a qualitative example) we obtain superior PSNR (but at the expense of visual quality with blurry images).

Revised: The last rows shows that when turning off distortion losses, the network does optimize reconstruction at all.

Original: [Table]
Revised: Additionally, for λ = 10 we observe that the GAN ”collapses” and outputs repetitive textures (see Fig. 25) , suggesting the distortion losses are crucial for stability of training. In Figures 26&27 we show the loss curves when training our GC, C = 8 model on OpenImages(Krasin et al., 2017). We note that the loss ﬂuctuates heavily across iterations due to the small batch size (one), but the smoothed losses are stable.

Revised: For all our experiments, both on Cityscapes and

Revised: OpenImages, we kept the weights of the losses and ratio between discriminator/generator iterations constant and at point did our (GC and SC) models collapse during training for either dataset.

Revised: [Table]

Revised: Our GC, C = 2 , H ( ˆ w ) < 0 .

Revised: 018 bpp

Revised: [Table]

Revised: Our GC, C = 4 , H ( ˆ w ) < 0 .

Revised: [Table]

Revised: Our GC, C = 8 , H ( ˆ w ) < 0 .

Revised: [Table]

Revised: MSE bl.

Revised: C = 4 , H ( ˆ w ) < 0 .

Revised: [Table]

Revised: GC, λ = 0 , C = 8 , H ( ˆ w ) < 0 .

Revised: [Table]

Revised: the effect of the GAN loss, the distortion losses and the entropy constraint on the PSNR of the trained model on the Cityscapes dataset.

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: .0k

Revised: 120.0k 140.0k

Revised: 160.0k 180.0k

Original: TensorBoard
Revised: TensorBoard

Original: SCALARS IMAGES
Revised: SCALARS IMAGES

Original: INACTIVE D_fake
Revised: INACTIVE D_fake

Original: D_
Revised: D_

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Filter tags (regular expressions supported)
Revised: Filter tags (regular expressions supported)

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 20.00k 40.00k 60.00k
Revised: 20.00k 40.00k 60.00k

Original: [Table]
Revised: [Table]

Original: .00k
Revised: .00k

Original: [Table]
Revised: [Table]

Original: 0k 140.0k 160.
Revised: 0k 140.0k 160.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] .0k
Revised: [Figure]

Original: 120.0k 140.0k

Original: 160.0k 180.0k