Original: NECST: N EURAL J OINT S OURCE -C HANNEL C ODING
Revised: NECST: N EURAL J OINT S OURCE -C HANNEL C ODING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes.
Revised: For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes.

Original: However, this decomposition can fall short in the ﬁnite bit-length regime, as it requires non-trivial tuning of hand-crafted codes and assumes inﬁnite computational power for decoding.
Revised: However, this decomposition can fall short in the ﬁnite bit-length regime, as it requires non-trivial tuning of hand-crafted codes and assumes inﬁnite computational power for decoding.

Original: In this work, we propose Neural Error Correcting and Source Trimming (NECST) codes to jointly learn the encoding and decoding processes in an end-to-end fashion.
Revised: In this work, we propose Neural Error Correcting and Source Trimming (NECST) codes to jointly learn the encoding and decoding processes in an end-to-end fashion.

Original: By adding noise into the latent codes to simulate the channel during training, we learn to both compress and error-correct given a ﬁxed bit-length and computational budget.
Revised: By adding noise into the latent codes to simulate the channel during training, we learn to both compress and error-correct given a ﬁxed bit-length and computational budget.

Original: We obtain codes that are not only competitive against several capacity-approaching channel codes, but also learn useful robust representations of the data for downstream tasks such as classiﬁcation.
Revised: We obtain codes that are not only competitive against several capacity-approaching channel codes, but also learn useful robust representations of the data for downstream tasks such as classiﬁcation.

Original: Finally, we learn an extremely fast neural decoder, yielding almost an order of magnitude in speedup compared to standard decoding methods based on iterative belief propagation.
Revised: Finally, we learn an extremely fast neural decoder, yielding almost an order of magnitude in speedup compared to standard decoding methods based on iterative belief propagation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider the problem of encoding images as bit-strings so that they can be reliably transmitted across a noisy communication channel.
Revised: We consider the problem of encoding images as bit-strings so that they can be reliably transmitted across a noisy communication channel.

Original: Classical results from (Shannon (1948)) show that for a memoryless communication channel, as the image size goes to inﬁnity, it is optimal to separately: 1) compress the images as much as possible to remove redundant information (source coding) and 2) use an error-correcting code to re-introduce redundancy, which allows for reconstruction in the presence of noise (channel coding).
Revised: Classical results from (Shannon (1948)) show that for a memoryless communication channel, as the image size goes to inﬁnity, it is optimal to separately: 1) compress the images as much as possible to remove redundant information (source coding) and 2) use an error-correcting code to re-introduce redundancy, which allows for reconstruction in the presence of noise (channel coding).

Original: The separation theorem has been studied in a wide variety of contexts and has also enjoyed great success in the development of new coding algorithms.
Revised: This separation theorem has been studied in a wide variety of contexts and has also enjoyed great success in the development of new coding algorithms.

Original: However, this elegant decomposition suffers from two critical limitations in the ﬁnite bit-length regime.
Revised: However, this elegant decomposition suffers from two critical limitations in the ﬁnite bit-length regime.

Original: First, without an inﬁnite number of bits for transmission, the overall distortion (reconstruction quality) is a function of both source and channel coding errors.
Revised: First, without an inﬁnite number of bits for transmission, the overall distortion (reconstruction quality) is a function of both source and channel coding errors.

Original: Thus optimizing for both: (1) the number of bits to allocate to each process and (2) the code designs themselves is an extremely difﬁcult task.
Revised: Thus optimizing for both: (1) the number of bits to allocate to each process and (2) the code designs themselves is an extremely difﬁcult task.

Original: Second, maximum-likelihood decoding is in general NP-hard
Revised: Second, maximum-likelihood decoding is in general NP-hard

Original: (Berlekamp et al.
Revised: (Berlekamp et al.

Original: Thus obtaining the accuracy that is possible in theory is contingent on the availability of inﬁnite computational power for decoding, which is impractical for real-world systems.
Revised: Thus obtaining the accuracy that is possible in theory is contingent on the availability of inﬁnite computational power for decoding, which is impractical for real-world systems.

Original: Though several lines of work have explored various relaxations of this problem to achieve tractability (Koetter &
Revised: Though several lines of work have explored various relaxations of this problem to achieve tractability (Koetter &

Original: Vontobel (2003), Feldman et al.
Revised: Vontobel (2003), Feldman et al.

Original: (2005), Vontobel & Koetter (2007)), many decoding systems rely on heuristics and early stopping of iterative approaches for scalability, which can be highly suboptimal.
Revised: (2005), Vontobel & Koetter (2007)), many decoding systems rely on heuristics and early stopping of iterative approaches for scalability, which can be highly suboptimal.

Original: To address these challenges we propose Neural Error Correcting and Source Trimming (NECST) codes, a deep learning framework for jointly learning to compress and error-correct an input image given a ﬁxed bit-length budget.
Revised: To address these challenges we propose Neural Error Correcting and Source Trimming (NECST) codes, a deep learning framework for jointly learning to compress and error-correct an input image given a ﬁxed bit-length budget.

Original: Three key steps are required.
Revised: Three key steps are required.

Original: First, we use neural networks to encode each image into a suitable bit-string representation, sidestepping the need to rely on hand-designed coding schemes that require additional tuning for good performance.
Revised: First, we use neural networks to encode each image into a suitable bit-string representation, sidestepping the need to rely on hand-designed coding schemes that require additional tuning for good performance.

Original: Second, we simulate a discrete channel within the model and inject noise directly into the latent codes to enforce robustness.
Revised: Second, we simulate a discrete channel within the model and inject noise directly into the latent codes to enforce robustness.

Original: Third, we amortize the decoding process such that after training, we can scale our model to massively large datasets.
Revised: Third, we amortize the decoding process such that after training, we can scale our model to massively large datasets.

Original: These components pose signiﬁcant optimization challenges due to the inherent non-differentiability of discrete latent random variables.
Revised: These components pose signiﬁcant optimization challenges due to the inherent non-differentiability of discrete latent random variables.

Original: We overcome this issue by leveraging recent advances in unbiased low-variance gradient estimation for variational learning of discrete latent variable models, and train the model using a variational lower bound on the mutual information between the images and their binary representations to obtain good, robust codes.
Revised: We overcome this issue by leveraging recent advances in unbiased low-variance gradient estimation for variational learning of discrete latent variable models, and train the model using a variational lower bound on the mutual information between the images and their binary representations to obtain good, robust codes.

Original: At its core, NECST can also be seen as an implicit deep generative model of the data derived from the perspective of the joint-source channel coding problem.
Revised: At its core, NECST can also be seen as an implicit deep generative model of the data derived from the perspective of the joint-source channel coding problem.

Original: In experiments, we test NECST on several grayscale and RGB image datasets, obtaining improvements over industry-standard compression (e.g, JPEG) and error-correcting codes (e.g., low density parity check codes).
Revised: In experiments, we test NECST on several grayscale and RGB image datasets, obtaining improvements over industry-standard compression (e.g, JPEG) and error-correcting codes (e.g., low density parity check codes).

Original: We also learn an extremely fast neural decoder, yielding almost an order of magnitude (two orders for magnitude for GPU) in speedup compared to standard decoding methods based on iterative belief propagation.
Revised: We also learn an extremely fast neural decoder, yielding almost an order of magnitude (two orders for magnitude for GPU) in speedup compared to standard decoding methods based on iterative belief propagation.

Original: Additionally, we show that in the process we learn discrete representations of the input images that are useful for downstream tasks such as classiﬁcation.
Revised: Additionally, we show that in the process we learn discrete representations of the input images that are useful for downstream tasks such as classiﬁcation.

Original:  S OURCE AND CHANNEL CODING 2. P RELIMINARIES
Revised:  S OURCE AND CHANNEL CODING 2. P RELIMINARIES

Original: We consider the problem of reliably communicating data across a noisy channel using a system that can detect and correct errors.
Revised: We consider the problem of reliably communicating data across a noisy channel using a system that can detect and correct errors.

Original: Let X be a space of possible inputs (e.g., images), and p data ( x ) be a source distribution deﬁned over x ∈ X .
Revised: Let X be a space of possible inputs (e.g., images), and p data ( x ) be a source distribution deﬁned over x ∈ X .

Original: The goal of the communication system is to encode samples x ∼ p data ( x ) as messages in a codeword space (cid:98) Y = { 0 , 1 } m .
Revised: The goal of the communication system is to encode samples x ∼ p data ( x ) as messages in a codeword space (cid:98) Y = { 0 , 1 } m .

Original: The messages are transmitted over a noisy communication channel, where they are corrupted to become noisy codewords in Y .
Revised: The messages are transmitted over a noisy communication channel, where they are corrupted to become noisy codewords in Y .

Original: Finally, a decoder produces a reconstruction ˆ x ∈ X of the original input x from a received noisy code y .
Revised: Finally, a decoder produces a reconstruction ˆ x ∈ X of the original input x from a received noisy code y .

Original: The goal is to minimize the overall distortion (reconstruction error) (cid:107) x − ˆ x (cid:107) in (cid:96) 1 or (cid:96) 2 norm while keeping the message length m as small as possible ( rate ).
Revised: The goal is to minimize the overall distortion (reconstruction error) (cid:107) x − ˆ x (cid:107) in (cid:96) 1 or (cid:96) 2 norm while keeping the message length m as small as possible ( rate ).

Original: In the absence of channel noise, low reconstruction errors can be achieved using a compression scheme to encode inputs x ∼ p data ( x ) as succinctly as possible (e.g., JPEG).
Revised: In the absence of channel noise, low reconstruction errors can be achieved using a compression scheme to encode inputs x ∼ p data ( x ) as succinctly as possible (e.g., JPEG).

Original: In the presence of channel noise, longer messages are typically needed to redundantly encode the information and recover from errors (e.g., parity check bits)
Revised: In the presence of channel noise, longer messages are typically needed to redundantly encode the information and recover from errors (e.g., parity check bits)

Original: . 2. T HE J OINT S OURCE -C HANNEL C ODING P ROBLEM
Revised: . 2. T HE J OINT S OURCE -C HANNEL C ODING P ROBLEM

Original: Based on fundamental results due to Shannon (Shannon, 1948), existing systems adhere to the following pipeline.
Revised: Based on fundamental results due to Shannon (Shannon, 1948), existing systems adhere to the following pipeline.

Original: A source encoder compresses the source image into a bit-string with the minimum number of bits possible.
Revised: A source encoder compresses the source image into a bit-string with the minimum number of bits possible.

Original: A channel encoder re-introduces redundancies, part of which were lost during source coding to prepare the codeword ˆ y for transmission.
Revised: A channel encoder re-introduces redundancies, part of which were lost during source coding to prepare the codeword ˆ y for transmission.

Original: The decoder then leverages the channel coding scheme to infer the original signal ˆ y , producing an approximate reconstruction ˆ x .
Revised: The decoder then leverages the channel coding scheme to infer the original signal ˆ y , producing an approximate reconstruction ˆ x .

Original: In the separation theorem, Shannon proved that the above approach is optimal in the limit of inﬁnitely long messages.
Revised: In the separation theorem, Shannon proved that the above approach is optimal in the limit of inﬁnitely long messages.

Original: That is, we can minimize distortion by optimizing the source and channel coding processes independently.
Revised: That is, we can minimize distortion by optimizing the source and channel coding processes independently.

Original: However, given a ﬁnite bit-length budget, the relationship between rate and distortion becomes more complex.
Revised: However, given a ﬁnite bit-length budget, the relationship between rate and distortion becomes more complex.

Original: The more bits that we reserve for compression, the fewer bits we have remaining to construct the best error-correcting code and vice versa.
Revised: The more bits that we reserve for compression, the fewer bits we have remaining to construct the best error-correcting code and vice versa.

Original: Balancing the two sources of distortion through the optimal bit allocation, in addition to designing the best source and channel codes, makes this joint source-channel coding (JSCC) problem challenging.
Revised: Balancing the two sources of distortion through the optimal bit allocation, in addition to designing the best source and channel codes, makes this joint source-channel coding (JSCC) problem challenging.

Original: In addition to these design challenges, real-world communication systems face computational and memory constraints that hinder the straightforward application of Shannon’s results.
Revised: In addition to these design challenges, real-world communication systems face computational and memory constraints that hinder the straightforward application of Shannon’s results.

Original: Speciﬁcally, practical decoding algorithms rely on approximations that yield suboptimal reconstruction accuracies.
Revised: Speciﬁcally, practical decoding algorithms rely on approximations that yield suboptimal reconstruction accuracies.

Original: The information theory community has studied this problem extensively, and proposed different bounds for ﬁnite bit-length JSCC in a wide variety of contexts (Pilc (1967), Csiszar (1982),
Revised: The information theory community has studied this problem extensively, and proposed different bounds for ﬁnite bit-length JSCC in a wide variety of contexts (Pilc (1967), Csiszar (1982),

Original: Kostina & Verd´u (2013)).
Revised: Kostina & Verd´u (2013)).

Original: Rather than relying on hand-crafted coding schemes that may require additional tuning in practice, we propose to learn the appropriate bit allocations and coding mechanisms using a ﬂexible deep learning framework.
Revised: Rather than relying on hand-crafted coding schemes that may require additional tuning in practice, we propose to learn the appropriate bit allocations and coding mechanisms using a ﬂexible deep learning framework.

Original:  N EURAL S OURCE AND C HANNEL CODES
Revised:  N EURAL S OURCE AND C HANNEL CODES

Original: Given the above considerations, we explore a learning-based approach to the JSCC problem.
Revised: Given the above considerations, we explore a learning-based approach to the JSCC problem.

Original: To do so, we consider a ﬂexible class of codes parameterized by a neural network and jointly optimize for the encoding and decoding procedure.
Revised: To do so, we consider a ﬂexible class of codes parameterized by a neural network and jointly optimize for the encoding and decoding procedure.

Original: This approach is inspired by recent successes in training (discrete) latent variable generative models, such as VAEs (Neal (1992), Rolfe (2016),
Revised: This approach is inspired by recent successes in training (discrete) latent variable generative models, such as VAEs (Neal (1992), Rolfe (2016),

Original: van den Oord et al.
Revised: van den Oord et al.

Original: We explore the connection to different types of autoencoders in section 4.1.
Revised: We explore the connection to different types of autoencoders in section 4.1.

Original: 3. C ODING P ROCESS
Revised: 3. C ODING P ROCESS

Original: Let X, ˆ Y , Y, ˆ X be random variables denoting the inputs, codewords, noisy codewords, and reconstructions respectively.
Revised: Let X, ˆ Y , Y, ˆ X be random variables denoting the inputs, codewords, noisy codewords, and reconstructions respectively.

Original: We model their joint distribution p ( x, ˆ y, y, ˆ x ) using the following graphical model X → ˆ Y → Y → ˆ X as:
Revised: We model their joint distribution p ( x, ˆ y, y, ˆ x ) using the following graphical model X → ˆ Y → Y → ˆ X as:

Original: [Equation]
Revised: [Equation]

Original: In equation 1, p data ( x ) denotes the distribution over inputs X .
Revised: In equation 1, p data ( x ) denotes the distribution over inputs X .

Original: It is not known explicitly in practice, and only accessible through samples.
Revised: It is not known explicitly in practice, and only accessible through samples.

Original: p channel ( y | (cid:98) y ; (cid:15) ) is the channel model, where we speciﬁcally focus on the binary symmetric channel (BSC) case.
Revised: p channel ( y | (cid:98) y ; (cid:15) ) is the channel model, where we speciﬁcally focus on the binary symmetric channel (BSC) case.

Original: The BSC independently ﬂips each bit in the codeword with probability (cid:15) (e.g., 0 → 1 ).
Revised: The BSC independently ﬂips each bit in the codeword with probability (cid:15) (e.g., 0 → 1 ).

Original: Therefore, ˆ Y takes values in ˆ Y
Revised: Therefore, ˆ Y takes values in ˆ Y

Original: =
Revised: =

Original: Y = { 0 , 1 } m and p channel 
Revised: Y = { 0 , 1 } m and p channel ( y |

Original: [Equation]
Revised: [Equation]

Original: y |
Revised: y ;

Original: [Equation]
Revised: [Equation]

Original: where ⊕ denotes addition modulo 2 (i.e., an eXclusive OR).
Revised: where ⊕ denotes addition modulo 2 (i.e., an eXclusive OR).

Original: A stochastic encoder q encoder (ˆ y | x ; φ ) generates a codeword ˆ y given an input x .
Revised: A stochastic encoder q encoder (ˆ y | x ; φ ) generates a codeword ˆ y given an input x .

Original: Speciﬁcally, we model each bit ˆ y i in the code with an independent Bernoulli random vector.
Revised: Speciﬁcally, we model each bit ˆ y i in the code with an independent Bernoulli random vector.

Original: We model the parameters of this Bernoulli with a neural network f φ ( · ) (an MLP or a CNN) parameterized by φ :
Revised: We model the parameters of this Bernoulli with a neural network f φ ( · ) (an MLP or a CNN) parameterized by φ :

Original: [Equation]
Revised: [Equation]

Original: i )))
Revised: i )))

Original: (1 − ˆ y i ) where σ ( z ) denotes the sigmoid function.
Revised: (1 − ˆ y i ) where σ ( z ) denotes the sigmoid function.

Original: Similarly, we posit a probabilistic decoder p decoder (ˆ x | y ; θ ) parameterized by an MLP/CNN that, given y , generates a decoded image ˆ x .
Revised: Similarly, we posit a probabilistic decoder p decoder (ˆ x | y ; θ ) parameterized by an MLP/CNN that, given y , generates a decoded image ˆ x .

Original: We model each pixel as a factorized Gaussian with a ﬁxed, isotropic covariance to yield: ˆ x | y ∼ N ( f θ ( y ) , σ 2 I ) , where f θ ( · ) denotes the decoder network.
Revised: We model each pixel as a factorized Gaussian with a ﬁxed, isotropic covariance to yield: ˆ x | y ∼ N ( f θ ( y ) , σ 2 I ) , where f θ ( · ) denotes the decoder network.

Original:  E ND - TO -E ND V ARIATIONAL L EARNING OF N EURAL C ODES
Revised:  E ND - TO -E ND V ARIATIONAL L EARNING OF N EURAL C ODES

Original: In order to learn an effective coding scheme, wewould like to maximize the mutual information between the input X and the corresponding noisy codeword Y (Barber & Agakov (2006)).
Revised: In order to learn an effective coding scheme, we maximize the mutual information between the input X and the corresponding noisy codeword Y (Barber & Agakov (2006)).

Original: That is, the code ˆ y should be robust to partial corruption; even its noisy instantiation y should preserve as much information about the original input x as possible (MacKay (2003)).
Revised: That is, the code ˆ y should be robust to partial corruption; even its noisy instantiation y should preserve as much information about the original input x as possible (MacKay (2003)).

Original: First, we note that we can analytically compute the encoding distribution q noisy enc ( y | x ; (cid:15), φ ) after it has been perturbed by the channel by marginalizing over ˆ y :
Revised: First, we note that we can analytically compute the encoding distribution q noisy enc ( y | x ; (cid:15), φ ) after it has been perturbed by the channel by marginalizing over ˆ y :

Original: [Equation]
Revised: [Equation]

Original: which yields:
Revised: which yields:

Original: [Equation]
Revised: [Equation]

Original: It is relatively straightforward to handle other types of channel noise; we provide additional examples of the binary erasure channel (BEC) and deletion channel in the Appendix.
Revised: It is relatively straightforward to handle other types of channel noise; we provide additional examples of the binary erasure channel (BEC) and deletion channel in the Appendix.

Original: We get the following optimization problem:
Revised: We get the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where p ( x | y ; (cid:15), φ ) is the true (and intractable) posterior from Eq.
Revised: where p ( x | y ; (cid:15), φ ) is the true (and intractable) posterior from Eq.

Original: 1 and p decoder ( x | y ; θ ) is an amortized variational approximation.
Revised: 1 and p decoder ( x | y ; θ ) is an amortized variational approximation.

Original: The true posterior p ( x | y ; (cid:15), φ ) — the posterior probability over possible inputs x given the received noisy codeword y — is the best possible decoder.
Revised: The true posterior p ( x | y ; (cid:15), φ ) — the posterior probability over possible inputs x given the received noisy codeword y — is the best possible decoder.

Original: However, it is also often intractable to evaluate and optimize.
Revised: However, it is also often intractable to evaluate and optimize.

Original: We therefore use a tractable variational approximation p decoder (ˆ x | y ; θ ) .
Revised: We therefore use a tractable variational approximation p decoder (ˆ x | y ; θ ) .

Original: Crucially, this variational approximation is amortized (Kingma & Welling (2013)), and is the inference distribution that will actually be used for decoding at test time.
Revised: Crucially, this variational approximation is amortized (Kingma & Welling (2013)), and is the inference distribution that will actually be used for decoding at test time.

Original: Because of amortization, decoding is guaranteed to be efﬁcient , in contrast with existing error correcting codes, which typically involve NP-hard MPE inference queries.
Revised: Because of amortization, decoding is guaranteed to be efﬁcient , in contrast with existing error correcting codes, which typically involve NP-hard MPE inference queries.

Original: Given any encoder ( φ ), we can ﬁnd the best amortized variational approximation by maximizing the lower bound (3) as a function of θ .
Revised: Given any encoder ( φ ), we can ﬁnd the best amortized variational approximation by maximizing the lower bound (3) as a function of θ .

Original: Therefore, the NECST training objective is given by:
Revised: Therefore, the NECST training objective is given by:

Original: [Equation]
Revised: [Equation]

Original: In practice, we approximate the expectation of the data distribution p data ( x ) with a ﬁnite dataset D :
Revised: In practice, we approximate the expectation of the data distribution p data ( x ) with a ﬁnite dataset D :

Original: [Equation]
Revised: [Equation]

Original: Thus we can jointly learn the encoding and decoding scheme by optimizing the parameters φ and θ .
Revised: Thus we can jointly learn the encoding and decoding scheme by optimizing the parameters φ and θ .

Original: In this way, the encoder ( φ ) is “aware” of the computational limitations of the decoder ( θ ), and is optimized accordingly.
Revised: In this way, the encoder ( φ ) is “aware” of the computational limitations of the decoder ( θ ), and is optimized accordingly.

Original: However, a main learning challenge is that we are not able to backpropagate directly through the discrete latent variable y ; we elaborate upon this point further in Section 5.1.
Revised: However, a main learning challenge is that we are not able to backpropagate directly through the discrete latent variable y ; we elaborate upon this point further in Section 5.1.

Original: The solutionobtained will depend on the number of available bits m , the noise level (cid:15) , and the structure of the input data x .
Revised: The solution will depend on the number of available bits m , the noise level (cid:15) , and the structure of the input data x .

Original: To gain some intuition, consider the case m = 0 .
Revised: For intuition, consider the case m = 0 .

Original: In this case, no information can be transmitted, and the best decoder will ﬁt a single Gaussian to the data.
Revised: In this case, no information can be transmitted, and the best decoder will ﬁt a single Gaussian to the data.

Original: When m = 1 and (cid:15) = 

Revised: When m = 1 and (cid:15) =

Original: (noiseless case), the model will learn a mixture of 2 m = 2 Gaussians.
Revised: (noiseless case), the model will learn a mixture of 2 m = 2 Gaussians.

Original: However, if m = 1 and (cid:15) = 0 .
Revised: However, if m = 1 and (cid:15) = 0 .

Original: 5 , again no information can be transmitted, and the best decoder will ﬁt a single Gaussian to the data.
Revised: 5 , again no information can be transmitted, and the best decoder will ﬁt a single Gaussian to the data.

Original: Adding noise forces the model to decide how it should effectively partition the data such that (1) things within the same cluster are similar; and (2) the clusters are ”well-separated” (adding redundancy so that even when a few bits are ﬂipped they still reconstruct to similar things instead of ”crossing over” to a different cluster that will be decoded to something different).
Revised: Adding noise forces the model to decide how it should effectively partition the data such that (1) similar items are grouped together in the same cluster; and (2) the clusters are ”well-separated” (even when a few bits are ﬂipped, they do not ”cross over” to a different cluster that will be decoded incorrectly).

Original: Thus, from the perspective of unsupervised learning, NECST attempts to learn robust binary representations of the data.
Revised: Thus, from the perspective of unsupervised learning, NECST attempts to learn robust binary representations of the data.

Original: 4. NECST AS A G ENERATIVE M ODEL
Revised: 4. NECST AS A G ENERATIVE M ODEL

Original: The objective function in equation (4) closely resembles those commonly used in generative modeling frameworks; in fact, NECST can also be viewed as a generative model.
Revised: The objective function in equation (4) closely resembles those commonly used in generative modeling frameworks; in fact, NECST can also be viewed as a generative model.

Original: In its simplest form, our model with a noiseless channel (i.e., (cid:15) = 0 ), deterministic encoder, and deterministic decoder is identical to a traditional autoencoder (Bengio et al.
Revised: In its simplest form, our model with a noiseless channel (i.e., (cid:15) = 0 ), deterministic encoder, and deterministic decoder is identical to a traditional autoencoder (Bengio et al.

Original: Once channel noise is present ( (cid:15) > 0 ) and the encoder/decoder become probabilistic, NECST begins to more closely resemble other variational autoencoding frameworks.
Revised: Once channel noise is present ( (cid:15) > 0 ) and the encoder/decoder become probabilistic, NECST begins to more closely resemble other variational autoencoding frameworks.

Original: Speciﬁcally, NECST is similar to Denoising Autoencoders (DAE) (Vincent et al.
Revised: Speciﬁcally, NECST is similar to Denoising Autoencoders (DAE) (Vincent et al.

Original: (2008)), except that it is explicitly trained for robustness to partial destruction of the latent space , as opposed to the input space.
Revised: (2008)), except that it is explicitly trained for robustness to partial destruction of the latent space , as opposed to the input space.

Original: NECST is also a variant of the VAE, with two nuanced distinctions.
Revised: NECST is also a variant of the VAE, with two nuanced distinctions.

Original: While both models learn a joint distribution over the observations and latent codes, the VAE: (1) optimizes a variational lower bound to the marginal log-likelihood p ( X ) as opposed to the mutual information
Revised: While both models learn a joint distribution over the observations and latent codes, the VAE: (1) optimizes a variational lower bound to the marginal log-likelihood p ( X ) as opposed to the mutual information

Original: I ( X, Y ) , and (2) explicitly posits a prior distribution p ( Y ) over the latent variables.
Revised: I ( X, Y ) , and (2) explicitly posits a prior distribution p ( Y ) over the latent variables.

Revised: Their close relationship is evidenced by a line of work on rate-distortion optimization in the context of VAEs (Ball´e et al.

Revised: (2016), Alemi et al.

Revised: (2017)), as well as other information-theoretic interpretations of the VAE’s information preference (Hinton & Van Camp (1993), Honkela & Valpola (2004), Chen et al.

Original: We also note that existing autoencoders are aimed at compression, while for sufﬁciently large 
Revised: We also note that existing autoencoders are aimed at compression, while for sufﬁciently large m

Original: m NECST will attempt to learn lossless compression with added redundancy for error correction.
Revised: NECST will attempt to learn lossless compression with added redundancy for error correction.

Original: Finally, NECST can be seen as a discrete version of the Uncertainty Autoencoder (UAE) (Grover & Ermon (2018)).
Revised: Finally, NECST can be seen as a discrete version of the Uncertainty Autoencoder (UAE) (Grover & Ermon (2018)).

Original: The two models share identical objective functions with two notable differences:
Revised: The two models share identical objective functions with two notable differences:

Original: For NECST, (1) the latent codes Y are discrete random variables, and (2) the noise model is no longer continuous.
Revised: For NECST, (1) the latent codes Y are discrete random variables, and (2) the noise model is no longer continuous.

Original: The special properties of the continuous UAE carry over directly to its discrete counterpart.
Revised: The special properties of the continuous UAE carry over directly to its discrete counterpart.

Original: Grover & Ermon (2018) proved that under certain conditions, the UAE speciﬁes an implicit generative model (Diggle & Gratton (1984), Mohamed & Lakshminarayanan (2016)).
Revised: Grover & Ermon (2018) proved that under certain conditions, the UAE speciﬁes an implicit generative model (Diggle & Gratton (1984), Mohamed & Lakshminarayanan (2016)).

Original: We restate their major theorem and extend their results here.
Revised: We restate their major theorem and extend their results here.

Original: Starting from any data point x (0) ∼ X , we deﬁne a Markov chain over X × Y with the following transitions:
Revised: Starting from any data point x (0) ∼ X , we deﬁne a Markov chain over X × Y with the following transitions:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: For any ﬁxed value of φ and (cid:15) > 0 , suppose that the NECST objective is globally maximized for some choice of θ ∗ so that inequality (3) is tight.
Revised: For any ﬁxed value of φ and (cid:15) > 0 , suppose that the NECST objective is globally maximized for some choice of θ ∗ so that inequality (3) is tight.

Original: Then the Markov Chain (5) with parameters φ and θ ∗ is ergodic and its stationary distribution is given by p data ( x ) q noisy enc ( y | x ; (cid:15), φ ) .
Revised: Then the Markov Chain (5) with parameters φ and θ ∗ is ergodic and its stationary distribution is given by p data ( x ) q noisy enc ( y | x ; (cid:15), φ ) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.1.
Revised: See Appendix A.1.

Original: Hence NECST has an intractable likelihood but can generate samples from p data by running the chain above.
Revised: Hence NECST has an intractable likelihood but can generate samples from p data by running the chain above.

Original: Samples initialized from both random noise and test data can be found in Appendix D.5.
Revised: Samples initialized from both random noise and test data can be found in Appendix D.5.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We ﬁrst review the optimization challenges of training discrete latent variable models and elaborate on our training procedure.
Revised: We ﬁrst review the optimization challenges of training discrete latent variable models and elaborate on our training procedure.

Original: Then to validate our work, we ﬁrst assess NECST’s compression and error correction capabilities against a combination of two widely-used compression (JPEG, VAE) and channel coding (ideal code, LDPC
Revised: Then to validate our work, we ﬁrst assess NECST’s compression and error correction capabilities against a combination of two widely-used compression (JPEG, VAE) and channel coding (ideal code, LDPC

Original: (Gallager (1962))) algorithms.
Revised: (Gallager (1962))) algorithms.

Original: We experiment on randomly generated length-100 bitstrings, MNIST (LeCun (1998)), Omniglot
Revised: We experiment on randomly generated length-100 bitstrings, MNIST (LeCun (1998)), Omniglot

Original: (Lake et al. (2015)), Street View
Revised: (Lake et al. (2015)), Street View

Original: Housing Numbers (SVHN) (Netzer et al. (2011)), and CelebA (Liu et al.
Revised: Housing Numbers (SVHN) (Netzer et al. (2011)), and CelebA (Liu et al.

Original: datasets to account for different image sizes and colors.
Revised: datasets to account for different image sizes and colors.

Original: Next, we test the decoding speed ofthe NECST’s neural decoder and ﬁnd that it performs upwards of an order of magnitude faster than standard decoding algorithms based on iterative belief propagation (and two orders of magnitude on GPU).
Revised: Next, we test the decoding speed of NECST’s neural decoder and ﬁnd that it performs upwards of an order of magnitude faster than standard decoding algorithms based on iterative belief propagation (and two orders of magnitude on GPU).

Original: Finally, we assess the quality of the latent codes after training, and examine interpolations in latent space as well as how well the learned features perform for downstream classiﬁcation tasks.
Revised: Finally, we assess the quality of the latent codes after training, and examine interpolations in latent space as well as how well the learned features perform for downstream classiﬁcation tasks.

Original: 5. O PTIMIZATION P ROCEDURE
Revised: 5. O PTIMIZATION P ROCEDURE

Original: Recall the NECST objective in equation 4.
Revised: Recall the NECST objective in equation 4.

Original: While obtaining Monte Carlo-based gradient estimates with respect to θ is easy, gradient estimation with respect to φ is challenging because these parameters specify the Bernoulli random variable q noisy enc ( y | x ; (cid:15), φ ) .
Revised: While obtaining Monte Carlo-based gradient estimates with respect to θ is easy, gradient estimation with respect to φ is challenging because these parameters specify the Bernoulli random variable q noisy enc ( y | x ; (cid:15), φ ) .

Original: The commonly used reparameterization trick cannot be applied in this setting, as the discrete stochastic unit in the computation graph renders the overall network non-differentiable (Schulman et al.
Revised: The commonly used reparameterization trick cannot be applied in this setting, as the discrete stochastic unit in the computation graph renders the overall network non-differentiable (Schulman et al.

Original: A simple alternative is to use the score function estimator in place of the gradient, as deﬁned in the REINFORCE algorithm (Williams (1992)).
Revised: A simple alternative is to use the score function estimator in place of the gradient, as deﬁned in the REINFORCE algorithm (Williams (1992)).

Original: However, this estimator suffers from high variance, and several others have explored different formulations and control variates to mitigate this issue (Wang et al. (2013), Gu et al.
Revised: However, this estimator suffers from high variance, and several others have explored different formulations and control variates to mitigate this issue (Wang et al. (2013), Gu et al.

Original: (2015), Ruiz et al.
Revised: (2015), Ruiz et al.

Original: , Tucker et al.
Revised: , Tucker et al.

Original: (2017), Grathwohl et al.
Revised: (2017), Grathwohl et al.

Original: Others have proposed a continuous relaxation of the discrete random variables, as in the
Revised: Others have proposed a continuous relaxation of the discrete random variables, as in the

Original: Gumbel-softmax (Jang et al.
Revised: Gumbel-softmax (Jang et al.

Original: and Concrete (Maddison et al. (2016)) distributions.
Revised: and Concrete (Maddison et al. (2016)) distributions.

Original: To preserve the hard “discreteness” of the latent codes, we used VIMCO (Mnih & Rezende (2016)), a multi-sample variational lower bound objective for obtaining low-variance gradients.
Revised: To preserve the hard “discreteness” of the latent codes, we used VIMCO (Mnih & Rezende (2016)), a multi-sample variational lower bound objective for obtaining low-variance gradients.

Original: VIMCO constructs leave-one-out control variates using its samples, as opposed to the single-sample objective
Revised: VIMCO constructs leave-one-out control variates using its samples, as opposed to the single-sample objective

Original: NVIL (Mnih & Gregor (2014)) which requires learning additional baselines during training.
Revised: NVIL (Mnih & Gregor (2014)) which requires learning additional baselines during training.

Original: Thus, we used the 5-sample VIMCO objective in subsequent experiments for the optimization procedure, leading us to our ﬁnal multi-sample ( K = 5 ) objective:
Revised: Thus, we used the 5-sample VIMCO objective in subsequent experiments for the optimization procedure, leading us to our ﬁnal multi-sample ( K = 5 ) objective:

Original: [Equation]
Revised: [Equation]

Original: 5. F IXED DISTORTION : JPEG + I DEAL CHANNEL CODE
Revised: 5. F IXED DISTORTION : JPEG + I DEAL CHANNEL CODE

Original: In this experiment, we compare the performances of: (1) NECST and (2) JPEG + ideal channel code in terms of compression.
Revised: In this experiment, we compare the performances of: (1) NECST and (2) JPEG + ideal channel code in terms of compression.

Original: Speciﬁcally, we ﬁx the number of bits m used by NECST to source and channel code and obtain the corresponding distortion levels (reconstruction errors) at various noise levels (cid:15) .
Revised: Speciﬁcally, we ﬁx the number of bits m used by NECST to source and channel code and obtain the corresponding distortion levels (reconstruction errors) at various noise levels (cid:15) .

Original: For ﬁxed m , distortion will increase with (cid:15) .
Revised: For ﬁxed m , distortion will increase with (cid:15) .

Original: Then, for each noise level we estimate the number of bits an alternate system using JPEG and an ideal channel code — the best that is theoretically possible — would have to use to match NECST’s distortion levels.
Revised: Then, for each noise level we estimate the number of bits an alternate system using JPEG and an ideal channel code — the best that is theoretically possible — would have to use to match NECST’s distortion levels.

Original: Assuming an ideal channel code implies that all messages will be transmitted perfectly across the noisy channel; thus the resulting distortion will only be a function of the compression mechanism.
Revised: Assuming an ideal channel code implies that all messages will be transmitted perfectly across the noisy channel; thus the resulting distortion will only be a function of the compression mechanism.

Original: We note that some approximations commonly used in the information theory community are used to obtain the analytic expressions used for this estimate.
Revised: We note that some approximations commonly used in the information theory community are used to obtain the analytic expressions used for this estimate.

Original: Full details and additional results may be found in Appendix D.1.
Revised: Full details and additional results may be found in Appendix D.1.

Original: We ﬁnd that with the exception of random data (shown in Appendix), NECST excels at compression; Figure 1 shows that in order to achieve the same level of distortion as NECST, the JPEG-ideal channel code system requires a much larger number bits across all noise levels .
Revised: We ﬁnd that with the exception of random data (Appendix D.1), NECST excels at compression; Figure 1 shows that in order to achieve the same level of distortion as NECST, the JPEG-ideal channel code system requires a much larger number bits across all noise levels .

Original: In particular,  note that
Revised: In particular, we note that

Original: NECST is slightly more effective than JPEG at pure compression ( (cid:15) = 0 ), and becomes signiﬁcantly better at higher noise levels (e.g., for (cid:15) = 0 . 4 , NECST requires 20 times less bits).
Revised: NECST is slightly more effective than JPEG at pure compression ( (cid:15) = 0 ), and becomes signiﬁcantly better at higher noise levels (e.g., for (cid:15) = 0 . 4 , NECST requires 20 × less bits).

Original: The negative result forthe random bitstrings is to be expected because the data has no  statistical dependencies that the model can leverage.
Revised: The negative result for random bits is to be expected because the data has no exploitable statistical dependencies.

Original: [Figure]
Revised: [Figure]

Original: 5. F IXED R ATE : SOURCE CODING + LDPC
Revised: 5. F IXED R ATE : SOURCE CODING + LDPC

Original: Next, we compare the performances of: (1) NECST and (2) discrete VAE + LDPC in terms of distortion.
Revised: Next, we compare the performances of: (1) NECST and (2) discrete VAE + LDPC in terms of distortion.

Original: Speciﬁcally, we ﬁx the bit-length budget for both systems and evaluate whether learning compression and error-correction jointly (NECST) improves reconstruction errors (distortion) for the same rate.
Revised: Speciﬁcally, we ﬁx the bit-length budget for both systems and evaluate whether learning compression and error-correction jointly (NECST) improves reconstruction errors (distortion) for the same rate.

Original: We do not report a comparison against JPEG + LDPC as we were often unable to obtain valid JPEG ﬁles to compute distortions after imperfect LDPC decoding.
Revised: We do not report a comparison against JPEG + LDPC as we were often unable to obtain valid JPEG ﬁles to compute distortions after imperfect LDPC decoding.

Original: However, as we have seen in the JPEG + ideal channel code experiment, we can reasonably assume that the JPEG-LDPC system would have displayed worse performance.
Revised: However, as we have seen in the JPEG + ideal channel code experiment, we can reasonably assume that the JPEG-LDPC system would have displayed worse performance.

Original: We experiment with a discrete VAE with a uniform prior over the latent codes for encoding.
Revised: We experiment with a discrete VAE with a uniform prior over the latent codes for encoding.

Original: To ﬁx the total number of bits that are transmitted through the channel, we double the length of the VAE encodings with an LDPC channel code to match NECST’s latent code dimension m .
Revised: To ﬁx the total number of bits that are transmitted through the channel, we double the length of the VAE encodings with an LDPC channel code to match NECST’s latent code dimension m .

Original: We note from
Revised: We note from

Original: Figure 2a that although the resulting distortion is similar, NECST outperforms the VAE across all noise levels for image datasets.
Revised: Figure 2a that although the resulting distortion is similar, NECST outperforms the VAE across all noise levels for image datasets.

Original: These results suggest that NECST’s learned coding scheme performs at least as well as LDPCs, an industrial-strength, widely deployed class of error correcting codes.
Revised: These results suggest that NECST’s learned coding scheme performs at least as well as LDPCs, an industrial-strength, widely deployed class of error correcting codes.

Original: 5. D ECODING T IME
Revised: 5. D ECODING T IME

Original: Another advantage of NECST over traditional channel codes is that after training, the amortized decoder can very efﬁciently map the transmitted code into its best reconstruction at test time.
Revised: Another advantage of NECST over traditional channel codes is that after training, the amortized decoder can very efﬁciently map the transmitted code into its best reconstruction at test time.

Original: Other sparse graph-based coding schemes such as LDPC, however, are more time-consuming because decoding involves an NP-hard optimization problem typically approximated with multiple iterations of belief propagation.
Revised: Other sparse graph-based coding schemes such as LDPC, however, are more time-consuming because decoding involves an NP-hard optimization problem typically approximated with multiple iterations of belief propagation.

Original: To compare the speed of our neural decoder to LDPC’s belief propagation, we ﬁxed the number of bits that were transmitted across the channel and averaged the total amount of time taken for decoding across ten runs.
Revised: To compare the speed of our neural decoder to LDPC’s belief propagation, we ﬁxed the number of bits that were transmitted across the channel and averaged the total amount of time taken for decoding across ten runs.

Original: The results are shown below in Figure 2b for the binarized MNIST and
Revised: The results are shown below in Figure 2b for the binarized MNIST and

Original: Omniglot datasets, and refer the reader to Appendix D.7 for the simulated random data.
Revised: Omniglot datasets, and refer the reader to Appendix D.7 for the simulated random data.

Original: On CPU,
Revised: On CPU,

Original: NECST’s decoder averages an order of magnitude faster than the LDPC decoder running foriterations (default setting), which is based on an optimized C implementation.
Revised: NECST’s decoder averages an order of magnitude faster than the LDPC decoder running foriterations, which is based on an optimized C implementation.

Original: On GPU, NECST displays two orders of magnitude in speedup .
Revised: On GPU, NECST displays two orders of magnitude in speedup .

Original: [Figure]
Revised: [Figure]

Original:  NECST FOR R OBUST R EPRESENTATION L EARNING
Revised:  NECST FOR R OBUST R EPRESENTATION L EARNING

Original: In addition to its ability to source and channel code effectively, NECST also serves as an implicit generative model that yields robust and interpretable latent representations and realistic samples from the underlying data distribution.
Revised: In addition to its ability to source and channel code effectively, NECST also serves as an implicit generative model that yields robust and interpretable latent representations and realistic samples from the underlying data distribution.

Original: Speciﬁcally, in light of Theorem 1, we can think of q encoder ( (cid:98) y | x ; φ ) as mapping images x to latent representations (cid:98) y .
Revised: Speciﬁcally, in light of Theorem 1, we can think of q encoder ( (cid:98) y | x ; φ ) as mapping images x to latent representations (cid:98) y .

Original: 6. I NTERPOLATION IN LATENT SPACE
Revised: 6. I NTERPOLATION IN LATENT SPACE

Original: To assess whether the model has: (1) injected redundancies into the learned codes and (2) learned interesting features, we interpolate between different data points in latent space and qualitatively observe whether the model captures semantically meaningful variations in the data.
Revised: To assess whether the model has: (1) injected redundancies into the learned codes and (2) learned interesting features, we interpolate between different data points in latent space and qualitatively observe whether the model captures semantically meaningful variations in the data.

Original: We select two test points to be the start and end, sequentially ﬂip one bit at a time in the latent code, and pass the altered code through the decoder to observe how the reconstruction changes.
Revised: We select two test points to be the start and end, sequentially ﬂip one bit at a time in the latent code, and pass the altered code through the decoder to observe how the reconstruction changes.

Original: In Figure 3, we show two illustrative examples from the MNIST digits.
Revised: In Figure 3, we show two illustrative examples from the MNIST digits.

Original: From the starting digit, each bit-ﬂip slowly alters characteristic features such as rotation, thickness, and stroke style until the digit is reconstructed to something else entirely.
Revised: From the starting digit, each bit-ﬂip slowly alters characteristic features such as rotation, thickness, and stroke style until the digit is reconstructed to something else entirely.

Original: We note that becausethe NECST is trained to encode redundancies, we do not observe a drastic change per ﬂip.
Revised: We note that because NECST is trained to encode redundancies, we do not observe a drastic change per ﬂip.

Original: Rather, it takes a signiﬁcant level of corruption for the decoder to interpret the original digit as another.
Revised: Rather, it takes a signiﬁcant level of corruption for the decoder to interpret the original digit as another.

Original: Also, due to the i.i.d.
Revised: Also, due to the i.i.d.

Original: nature of the channel noise model, we observe that the sequence at which the bits are ﬂipped do not have a signiﬁcant effect on the reconstructions.
Revised: nature of the channel noise model, we observe that the sequence at which the bits are ﬂipped do not have a signiﬁcant effect on the reconstructions.

Original: Additional interpolations may be found in Appedix D.4.
Revised: Additional interpolations may be found in Appedix D.4.

Original: (a) MNIST: 6 → 3 leads to an intermediate(b) MNIST: 8 →
Revised: (a) MNIST: 6 → 3 leads to an intermediate(b) MNIST: 8 →

Original: 6. D OWNSTREAM CLASSIFICATION
Revised: 6. D OWNSTREAM CLASSIFICATION

Original: To demonstrate that the latent representations learned by NECST are useful for downstream classiﬁcation tasks, we extract the binary features and train eight different classiﬁcation algorithms: knearest neighbors (KNN), decision trees (DT), random forests (RF), multilayer perceptron (MLP),
Revised: To demonstrate that the latent representations learned by NECST are useful for downstream classiﬁcation tasks, we extract the binary features and train eight different classiﬁcation algorithms: knearest neighbors (KNN), decision trees (DT), random forests (RF), multilayer perceptron (MLP),

Original: AdaBoost (AdaB), Quadratic Discriminant Analysis (QDA), and support vector machines (SVM).
Revised: AdaBoost (AdaB), Quadratic Discriminant Analysis (QDA), and support vector machines (SVM).

Original: As shown on the leftmost numbers in Table 1, the simple classiﬁers perform reasonably well in the digit classiﬁcation task for MNIST ( (cid:15) = 0 ).
Revised: As shown on the leftmost numbers in Table 1, the simple classiﬁers perform reasonably well in the digit classiﬁcation task for MNIST ( (cid:15) = 0 ).

Original: We observe that with the exception of KNN, all other classiﬁers achieve higher levels of accuracy when using features that were trained with simulated channel noise ( (cid:15)
Revised: We observe that with the exception of KNN, all other classiﬁers achieve higher levels of accuracy when using features that were trained with simulated channel noise ( (cid:15)

Original: = 0 .
Revised: = 0 .

Original: [Table]
Revised: [Table]

Original: To further test the hypothesis that NECST trained with the noisy channel learns more robust latent features, we freeze the pre-trained model and evaluate classiﬁcation accuracy using a ”noisy”
Revised: To further test the hypothesis that NECST trained with the noisy channel learns more robust latent features, we freeze the pre-trained model and evaluate classiﬁcation accuracy using a ”noisy”

Original: MNIST dataset.
Revised: MNIST dataset.

Original: We synthesized noisy MNIST by adding (cid:15) ∼ N (0 , 0 . 5) noise to all pixel values in
Revised: We synthesized noisy MNIST by adding (cid:15) ∼ N (0 , 0 . 5) noise to all pixel values in

Original: MNIST, and ran the same experiment as above.
Revised: MNIST, and ran the same experiment as above.

Original: As shown in the rightmost numbers of Table 1, we again observe that most algorithms show improved performance with added channel noise.
Revised: As shown in the rightmost numbers of Table 1, we again observe that most algorithms show improved performance with added channel noise.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There has been a recent surge of work applying deep learning and generative modeling techniques to lossy image compression, many of which compare favorably against industry standards such as
Revised: There has been a recent surge of work applying deep learning and generative modeling techniques to lossy image compression, many of which compare favorably against industry standards such as

Original: JPEG, JPEG2000, and WebP (Toderici et al.
Revised: JPEG, JPEG2000, and WebP (Toderici et al.

Original: , Ball ´ e et al.
Revised: , Ball ´ e et al.

Original: (2016), Toderici et al.
Revised: (2016), Toderici et al.

Original: Theis et al.
Revised: Theis et al.

Original: (2017) use compressive autoencoders that learn the optimal number of bits to represent images based on their pixel frequencies.
Revised: (2017) use compressive autoencoders that learn the optimal number of bits to represent images based on their pixel frequencies.

Original: Ball ´ e et al.
Revised: Ball ´ e et al.

Original: (2018) use a variational autoencoder (VAE) (Kingma & Welling (2013)
Revised: (2018) use a variational autoencoder (VAE) (Kingma & Welling (2013)

Original: Rezende et al.
Revised: Rezende et al.

Original: with a learnable scale hyperprior to capture the image’s partition structure as side information for more efﬁcient compression.
Revised: with a learnable scale hyperprior to capture the image’s partition structure as side information for more efﬁcient compression.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: (2017) use adversarial training (Goodfellow et al.
Revised: (2017) use adversarial training (Goodfellow et al.

Original: to learn neural codecs for compressing images and videos using DCGAN-style ConvNets (Radford et al.
Revised: to learn neural codecs for compressing images and videos using DCGAN-style ConvNets (Radford et al.

Original: Yet these methods focus on source coding only, and do not consider the setting where the compression must be robust to channel noise.
Revised: Yet these methods focus on source coding only, and do not consider the setting where the compression must be robust to channel noise.

Original: In a similar vein, there has been growing interest on leveraging these deep learning systems to sidestep the use of hand-designed codes.
Revised: In a similar vein, there has been growing interest on leveraging these deep learning systems to sidestep the use of hand-designed codes.

Original: Several lines of work train neural decoders based on known coding schemes, sometimes learning more general decoding algorithms than before (Nachmani et al.
Revised: Several lines of work train neural decoders based on known coding schemes, sometimes learning more general decoding algorithms than before (Nachmani et al.

Original: (2016), Gruber et al.
Revised: (2016), Gruber et al.

Original: (2017), Cammerer et al.
Revised: (2017), Cammerer et al.

Original: (2017), Dorner et al.
Revised: (2017), Dorner et al.

Original: (Kim et al.
Revised: (Kim et al.

Original: (2018a), Kim et al.
Revised: (2018a), Kim et al.

Original: (2018b))
Revised: (2018b))

Original: parameterize sequential codes with recurrent neural network (RNN) architectures that achieve comparable performance to capacity-approaching codes over the additive white noise
Revised: parameterize sequential codes with recurrent neural network (RNN) architectures that achieve comparable performance to capacity-approaching codes over the additive white noise

Original: Gaussian (AWGN) and bursty noise channels.
Revised: Gaussian (AWGN) and bursty noise channels.

Original: However, source coding is out of the scope for these methods that focus on learning good channel codes.
Revised: However, source coding is out of the scope for these methods that focus on learning good channel codes.

Original: The problem of end-to-end transmission of structured data, on the other hand, is less well-studied.
Revised: The problem of end-to-end transmission of structured data, on the other hand, is less well-studied.

Original: Farsad et al.
Revised: Farsad et al.

Original: (2018) use RNNs to communicate text over a discrete, binary erasure channel (BEC) and are able to preserve the words’ semantics.
Revised: (2018) use RNNs to communicate text over a discrete, binary erasure channel (BEC) and are able to preserve the words’ semantics.

Original: The most similar to our work is that of Bourtsoulatze et al.
Revised: The most similar to our work is that of Bourtsoulatze et al.

Original: (2018), who use autoencoders for transmitting images over the AWGN and slow Rayleigh fading channels, which are continuous.
Revised: (2018), who use autoencoders for transmitting images over the AWGN and slow Rayleigh fading channels, which are continuous.

Original: We provide a holistic treatment of various discrete noise models and show howthe NECST can also be used for unsupervised learning.
Revised: We provide a holistic treatment of various discrete noise models and show how NECST can also be used for unsupervised learning.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We described how NECST can be used to learn an efﬁcient joint source-channel coding scheme by simulating a noisy channel during training.
Revised: We described how NECST can be used to learn an efﬁcient joint source-channel coding scheme by simulating a noisy channel during training.

Original: We showed that the model: (1) is competitive against  industry standard compression and error-correcting codes, (2) learns an extremely fast neural decoder through amortized inference, and (3) learns a latent code that is not only robust to corruption, but also useful for general downstream tasks such as classiﬁcation.
Revised: We showed that the model: (1) is competitive against a combination of industry standard compression and error-correcting codes, (2) learns an extremely fast neural decoder through amortized inference, and (3) learns a latent code that is not only robust to corruption, but also useful for general downstream tasks such as classiﬁcation.

Original: One limitation of NECST is that we need to train the model separately for different code-lengths and datasets; in its current form, it can only handle ﬁxed-length codes.
Revised: One limitation of NECST is that we need to train the model separately for different code-lengths and datasets; in its current form, it can only handle ﬁxed-length codes.

Original: Extending the model to streaming or adaptive learning scenarios that allow for learning variable-length codes is an exciting direction for future work.
Revised: Extending the model to streaming or adaptive learning scenarios that allow for learning variable-length codes is an exciting direction for future work.

Original: Another direction would be to analyze the characteristics and properties of the learned latent codes under different discrete channel models, such as erasure channels.
Revised: Another direction would be to analyze the characteristics and properties of the learned latent codes under different discrete channel models, such as erasure channels.

Original: A PPENDIX A P ROOFS OF THEORETICAL RESULTS A. S TATIONARY D ISTRIBUTION OF M ARKOV CHAIN
Revised: A PPENDIX A P ROOFS OF THEORETICAL RESULTS A. S TATIONARY D ISTRIBUTION OF M ARKOV CHAIN

Original: We follow the proof structure in (Grover & Ermon (2018)) with minor adaptations.
Revised: We follow the proof structure in (Grover & Ermon (2018)) with minor adaptations.

Original: First, we note that the Markov chain deﬁned in Eq.
Revised: First, we note that the Markov chain deﬁned in Eq.

Original: 5 is ergodic due to the BSC noise model.
Revised: 5 is ergodic due to the BSC noise model.

Original: This is because the BSC deﬁnes a distribution over all possible (ﬁnite) conﬁgurations of the latent code Y , and
Revised: This is because the BSC deﬁnes a distribution over all possible (ﬁnite) conﬁgurations of the latent code Y , and

Original: the
Revised: the

Original: Gaussian decoder posits a non-negative probability of transitioning to all possible reconstructions
Revised: Gaussian decoder posits a non-negative probability of transitioning to all possible reconstructions

Original: X .
Revised: X .

Original: Thus given any ( X, Y ) and ( X (cid:48) , Y (cid:48) )
Revised: Thus given any ( X, Y ) and ( X (cid:48) , Y (cid:48) )

Original: such that the density p ( X , Y ) > 0 and p ( X (cid:48) , Y (cid:48) )
Revised: such that the density p ( X , Y ) > 0 and p ( X (cid:48) , Y (cid:48) )

Original: > 0 , the probability density of transitioning p ( X (cid:48) , Y (cid:48)
Revised: > 0 , the probability density of transitioning p ( X (cid:48) , Y (cid:48)

Original: |X , Y ) > 0 .
Revised: |X , Y ) > 0 .

Original: Next, we can rewrite the objective in Eq.
Revised: Next, we can rewrite the objective in Eq.

Original: equation 4 as the following:
Revised: equation 4 as the following:

Original: [Equation]
Revised: [Equation]

Original: We note that for any value of φ , the optimal value of θ = θ ∗ eliminates the KL-divergence term, as it is non-negative and minimized only when its argument distributions are identical.
Revised: We note that for any value of φ , the optimal value of θ = θ ∗ eliminates the KL-divergence term, as it is non-negative and minimized only when its argument distributions are identical.

Original: Then, for any φ , we note that the following Gibbs chain converges to p ( x, y ) if the chain is ergodic:
Revised: Then, for any φ , we note that the following Gibbs chain converges to p ( x, y ) if the chain is ergodic:

Original: [Equation]
Revised: [Equation]

Original: Substituting p ( x | y ; θ ∗ ) for p decoder ( x | y ; θ ) ﬁnishes the proof.
Revised: Substituting p ( x | y ; θ ∗ ) for p decoder ( x | y ; θ ) ﬁnishes the proof.

Original: B A DDITIONAL CHANNEL MODELS B. B INARY E RASURE C HANNEL (BEC)
Revised: B A DDITIONAL CHANNEL MODELS B. B INARY E RASURE C HANNEL (BEC)

Original: The BEC erases each bit at position i in the latent code ˆ y with some i.i.d.
Revised: The BEC erases each bit at position i in the latent code ˆ y with some i.i.d.

Original: probability (cid:15) into a new symbol.
Revised: probability (cid:15) into a new symbol.

Original: Thus if the input to the channel is ˆ y | x ∼ Bern ( σ ( f φ ( x ))) , the BEC induces a 3-way
Revised: Thus if the input to the channel is ˆ y | x ∼ Bern ( σ ( f φ ( x ))) , the BEC induces a 3-way

Original: Categorical distribution where the third category refers to the bit-erasure.
Revised: Categorical distribution where the third category refers to the bit-erasure.

Original: We denote the erased bit as ?, which can be modeled as category 2 in the new alphabet of Y = { 0 , 1 , 2 } m .
Revised: We denote the erased bit as ?, which can be modeled as category 2 in the new alphabet of Y = { 0 , 1 , 2 } m .

Original: This yields:
Revised: This yields:

Original: [Equation]
Revised: [Equation]

Original: (cid:15) )
Revised: (cid:15) )

Original: where: p
Revised: where: p

Original: [Equation]
Revised: [Equation]

Original: y =
Revised: y =

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: = 0 , x
Revised: = 0 , x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( f φ ( x )
Revised: ( f φ ( x )

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ( f φ ( x ))
Revised: ( f φ ( x ))

Original: [Equation]
Revised: [Equation]

Original: (cid:15) p ( y =
Revised: (cid:15) p ( y =

Original: [Equation]
Revised: [Equation]

Original: | ˆ y =
Revised: | ˆ y =

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: φ ( x )
Revised: φ ( x )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: φ ( x
Revised: φ ( x

Original: [Equation]
Revised: [Equation]

Original: · (cid:15) p
Revised: · (cid:15) p

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: p ( y = ?
Revised: p ( y = ?

Original: | ˆ y =
Revised: | ˆ y =

Original: [Equation]
Revised: [Equation]

Original: x ) = (cid:15)
Revised: x ) = (cid:15)

Original: B. D ELETION C HANNEL
Revised: B. D ELETION C HANNEL

Original: The deletion channel is similar to the BEC, except that with some independent probability (cid:15) , the bit at position i is deleted.
Revised: The deletion channel is similar to the BEC, except that with some independent probability (cid:15) , the bit at position i is deleted.

Original: Thus our codewords y may not necessarily be of length m as in ˆ y ∈ { 0 , 1 } m , and the space of y may have variable lengths.
Revised: Thus our codewords y may not necessarily be of length m as in ˆ y ∈ { 0 , 1 } m , and the space of y may have variable lengths.

Original: C NECST ARCHITECTURE AND HYPERPARAMETER CONFIGURATIONS C. MNIST
Revised: C NECST ARCHITECTURE AND HYPERPARAMETER CONFIGURATIONS C. MNIST

Original: For MNIST, we used the static binarized version as provided in (Burda et al.
Revised: For MNIST, we used the static binarized version as provided in (Burda et al.

Original: with train/validation/test splits of 50K/10K/10K respectively.
Revised: with train/validation/test splits of 50K/10K/10K respectively.

Original: • encoder:
Revised: • encoder:

Original: MLP with 1 hidden layer (500 hidden units), ReLU activations
Revised: MLP with 1 hidden layer (500 hidden units), ReLU activations

Original: • decoder: 2-layer MLP with 500 hidden units each, ReLU activations.
Revised: • decoder: 2-layer MLP with 500 hidden units each, ReLU activations.

Original: The ﬁnal output layer has a sigmoid activation for learning the parameters of p noisy enc ( y | x ; φ, (cid:15) )
Revised: The ﬁnal output layer has a sigmoid activation for learning the parameters of p noisy enc ( y | x ; φ, (cid:15) )

Original: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.
Revised: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.

Original: • Adam optimizer with lr=0.
Revised: • Adam optimizer with lr=0.

Original: C. O MNIGLOT
Revised: C. O MNIGLOT

Original: We statically binarize the Omniglot dataset by rounding values above 0.5 to 1, and those below to 0.
Revised: We statically binarize the Omniglot dataset by rounding values above 0.5 to 1, and those below to 0.

Original: • encoder:
Revised: • encoder:

Original: MLP with 1 hidden layer (500 hidden units), ReLU activations • decoder: 2-layer MLP with 500 hidden units each, ReLU activations.
Revised: MLP with 1 hidden layer (500 hidden units), ReLU activations • decoder: 2-layer MLP with 500 hidden units each, ReLU activations.

Original: The ﬁnal output layer has a sigmoid activation for learning the parameters of p noisy enc ( y | x ; φ, (cid:15) )
Revised: The ﬁnal output layer has a sigmoid activation for learning the parameters of p noisy enc ( y | x ; φ, (cid:15) )

Original: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.
Revised: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.

Original: • Adam optimizer with lr=0.
Revised: • Adam optimizer with lr=0.

Original: C. R ANDOM BITS
Revised: C. R ANDOM BITS

Original: We randomly generated length-100 bitstrings by drawing from a Bern (0 . 5) distribution for each entry in the bitstring.
Revised: We randomly generated length-100 bitstrings by drawing from a Bern (0 . 5) distribution for each entry in the bitstring.

Original: The train/validation/test splits are: 5K/1K/1K.
Revised: The train/validation/test splits are: 5K/1K/1K.

Original: • encoder:
Revised: • encoder:

Original: MLP with 1 hidden layer (500 hidden units), ReLU activations • decoder: 2-layer MLP with 500 hidden units each, ReLU activations.
Revised: MLP with 1 hidden layer (500 hidden units), ReLU activations • decoder: 2-layer MLP with 500 hidden units each, ReLU activations.

Original: The ﬁnal output layer has a sigmoid activation for learning the parameters of p noisy enc ( y | x ; φ, (cid:15) )
Revised: The ﬁnal output layer has a sigmoid activation for learning the parameters of p noisy enc ( y | x ; φ, (cid:15) )

Original: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.
Revised: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.

Original: • Adam optimizer with lr=0.
Revised: • Adam optimizer with lr=0.

Original: C. SVHN
Revised: C. SVHN

Original: For SVHN, we collapse the ”easier” additional examples with the more difﬁcult training set, and randomly partition 10K of the roughly 600K dataset into a validation set.
Revised: For SVHN, we collapse the ”easier” additional examples with the more difﬁcult training set, and randomly partition 10K of the roughly 600K dataset into a validation set.

Original: • encoder: CNN with 3 convolutional layers + fc layer, ReLU activations • decoder: CNN with 4 deconvolutional layers, ReLU activations.
Revised: • encoder: CNN with 3 convolutional layers + fc layer, ReLU activations • decoder: CNN with 4 deconvolutional layers, ReLU activations.

Original: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.
Revised: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.

Original: • Adam optimizer with lr=0.
Revised: • Adam optimizer with lr=0.

Original: The CNN architecture for the encoder is as follows:
Revised: The CNN architecture for the encoder is as follows:

Original: conv1 =
Revised: conv1 =

Original: n ﬁlters=128, kernel size=2, strides=2, padding=”VALID”
Revised: n ﬁlters=128, kernel size=2, strides=2, padding=”VALID”

Original: conv2 =
Revised: conv2 =

Original: n ﬁlters=256, kernel size=2, strides=2, padding=”VALID” 3.
Revised: n ﬁlters=256, kernel size=2, strides=2, padding=”VALID” 3.

Original: conv3 =
Revised: conv3 =

Original: n ﬁlters=512, kernel size=2, strides=2, padding=”VALID” 4.
Revised: n ﬁlters=512, kernel size=2, strides=2, padding=”VALID” 4.

Original: fc = 4*4*512 → n bits, no activation
Revised: fc = 4*4*512 → n bits, no activation

Original: The decoder architecture follows the reverse, but without a ﬁnal deconvolution layer as: n ﬁlters=3, kernel size=1, strides=1, padding=”VALID”, activation=ReLU.
Revised: The decoder architecture follows the reverse, but without a ﬁnal deconvolution layer as: n ﬁlters=3, kernel size=1, strides=1, padding=”VALID”, activation=ReLU.

Original: C. CELEB A
Revised: C. CELEB A

Original: We use the celebA dataset with standard train/validation/test splits with minor preprocessing.
Revised: We use the celebA dataset with standard train/validation/test splits with minor preprocessing.

Original: First, we align and crop each image to focus on the face, resizing the image to be (64 , 64 , 3) .
Revised: First, we align and crop each image to focus on the face, resizing the image to be (64 , 64 , 3) .

Original: • encoder: CNN with 5 convolutional layers + fc layer, ELU activations • decoder: CNN with 5 deconvolutional layers, ELU activations.
Revised: • encoder: CNN with 5 convolutional layers + fc layer, ELU activations • decoder: CNN with 5 deconvolutional layers, ELU activations.

Original: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.
Revised: • n bits:• n epochs:• batch size:• L2 regularization penalty of encoder weights: 0.

Original: • Adam optimizer with lr=0.
Revised: • Adam optimizer with lr=0.

Original: The CNN architecture for the encoder is as follows:
Revised: The CNN architecture for the encoder is as follows:

Original: conv1 =
Revised: conv1 =

Original: n ﬁlters=32, kernel size=4, strides=2, padding=”SAME” 2.
Revised: n ﬁlters=32, kernel size=4, strides=2, padding=”SAME” 2.

Original: conv2 =
Revised: conv2 =

Original: n ﬁlters=32, kernel size=4, strides=2, padding=”SAME” 3.
Revised: n ﬁlters=32, kernel size=4, strides=2, padding=”SAME” 3.

Original: conv3 = n ﬁlters=64, kernel size=4, strides=2, padding=”SAME” 4. conv4 = n ﬁlters=64, kernel size=4, strides=2, padding=”SAME” 5.
Revised: conv3 = n ﬁlters=64, kernel size=4, strides=2, padding=”SAME” 4. conv4 = n ﬁlters=64, kernel size=4, strides=2, padding=”SAME” 5.

Original: conv5 = n ﬁlters=256, kernel size=4, strides=2, padding=”VALID” 6.
Revised: conv5 = n ﬁlters=256, kernel size=4, strides=2, padding=”VALID” 6.

Original: fc = 256 → n bits, no activation
Revised: fc = 256 → n bits, no activation

Original: The decoder architecture follows the reverse, but without the ﬁnal fully connected layer and the last deconvolutional layer as: n ﬁlters=3, kernel size=4, strides=2, padding=”SAME”, activation=sigmoid.
Revised: The decoder architecture follows the reverse, but without the ﬁnal fully connected layer and the last deconvolutional layer as: n ﬁlters=3, kernel size=4, strides=2, padding=”SAME”, activation=sigmoid.

Original: D A DDITIONAL EXPERIMENTAL DETAILS AND RESULTS D. F IXED D ISTORTION : JPEG-I DEAL C HANNEL C ODE S YSTEM
Revised: D A DDITIONAL EXPERIMENTAL DETAILS AND RESULTS D. F IXED D ISTORTION : JPEG-I DEAL C HANNEL C ODE S YSTEM

Original: For the BSC channel, we can compute the theoretical channel capacity with the formula C = 1 −
Revised: For the BSC channel, we can compute the theoretical channel capacity with the formula C = 1 −

Original: H b ( (cid:15) ) , where (cid:15) denotes the bit-ﬂip probability of the channel and H b denotes the binary entropy function.
Revised: H b ( (cid:15) ) , where (cid:15) denotes the bit-ﬂip probability of the channel and H b denotes the binary entropy function.

Original: Note that the communication rate of C is achievable in the asymptotic scenario of inﬁnitely long messages; in the ﬁnite bit-length regime, particularly in the case of short blocklengths, the highest achievable rate will be much lower.
Revised: Note that the communication rate of C is achievable in the asymptotic scenario of inﬁnitely long messages; in the ﬁnite bit-length regime, particularly in the case of short blocklengths, the highest achievable rate will be much lower.

Original: For each image, we ﬁrst obtain the target distortion d per channel noise level by using a ﬁxed bitlength budget with NECST.
Revised: For each image, we ﬁrst obtain the target distortion d per channel noise level by using a ﬁxed bitlength budget with NECST.

Original: Next we use the JPEG compressor to encode the image at the distortion level d .
Revised: Next we use the JPEG compressor to encode the image at the distortion level d .

Original: The resulting size of the compressed image f ( d ) is used to get an estimate f ( d ) /C
Revised: The resulting size of the compressed image f ( d ) is used to get an estimate f ( d ) /C

Original: for the number of bits used for the image representation in the ideal channel code scenario.
Revised: for the number of bits used for the image representation in the ideal channel code scenario.

Original: While measuring the compressed image size f ( d ) , we ignore the header size of the JPEG image, as the header is similar for images from the same dataset.
Revised: While measuring the compressed image size f ( d ) , we ignore the header size of the JPEG image, as the header is similar for images from the same dataset.

Original: The plots compare f ( d ) /C with m , the ﬁxed bit-length budget for NECST.
Revised: The plots compare f ( d ) /C with m , the ﬁxed bit-length budget for NECST.

Original: [Figure]
Revised: [Figure]

Original: We ﬁrst elaborate on the usage of the LDPC software.
Revised: We ﬁrst elaborate on the usage of the LDPC software.

Original: We use an optimized C implementation of
Revised: We use an optimized C implementation of

Original: LDPC codes to run our decoding time experiments: ( http://radfordneal.github.io/LDPC-codes/ ).
Revised: LDPC codes to run our decoding time experiments: ( http://radfordneal.github.io/LDPC-codes/ ).

Original: The procedure is as follows:
Revised: The procedure is as follows:

Original: • make-pchk to create a parity check matrix for a regular LDPC code with three 1’s per column, eliminating cycles of length 4 (default setting).
Revised: • make-pchk to create a parity check matrix for a regular LDPC code with three 1’s per column, eliminating cycles of length 4 (default setting).

Original: The number of parity check bits is: total number of bits allowed - length of codeword .
Revised: The number of parity check bits is: total number of bits allowed - length of codeword .

Original: • make-gen to create the generator matrix from the parity check matrix.
Revised: • make-gen to create the generator matrix from the parity check matrix.

Original: We use the default dense setting to create a dense representation.
Revised: We use the default dense setting to create a dense representation.

Original: • encode to encode the source bits into the LDPC encoding • transmit to transmit the LDPC code through a bsc channel, with the appropriate level of channel noise speciﬁed (e.g. 0.1) • extract to obtain the actual decoded bits, or decode to directly obtain the bit errors from the source to the decoding.
Revised: • encode to encode the source bits into the LDPC encoding • transmit to transmit the LDPC code through a bsc channel, with the appropriate level of channel noise speciﬁed (e.g. 0.1) • extract to obtain the actual decoded bits, or decode to directly obtain the bit errors from the source to the decoding.

Original: LDPC-based channel codes require larger blocklengths to be effective.
Revised: LDPC-based channel codes require larger blocklengths to be effective.

Original: To perform an end-to-end experiment with the JPEG compression and LDPC channel codes, we form the input by concatenating multiple blocks of images together into a grid-like image.
Revised: To perform an end-to-end experiment with the JPEG compression and LDPC channel codes, we form the input by concatenating multiple blocks of images together into a grid-like image.

Original: In the ﬁrst step, the ﬁxed rate of m is scaled by the total number of images combined, and this quantity is used to estimate the target f ( d ) to which we compress the concatenated images.
Revised: In the ﬁrst step, the ﬁxed rate of m is scaled by the total number of images combined, and this quantity is used to estimate the target f ( d ) to which we compress the concatenated images.

Original: In the second step, the compressed concatenated image is coded together by the LDPC code into a bit-string, so as to correct for any errors due to channel noise.
Revised: In the second step, the compressed concatenated image is coded together by the LDPC code into a bit-string, so as to correct for any errors due to channel noise.

Original: Finally, we decode the corrupted bit-string using the LDPC decoder.
Revised: Finally, we decode the corrupted bit-string using the LDPC decoder.

Original: The plots compare the resulting distortion of the compressed concatenated block of images with the average distortion on compressing the images individually using NECST.
Revised: The plots compare the resulting distortion of the compressed concatenated block of images with the average distortion on compressing the images individually using NECST.

Original: Note that, the experiment gives a slight disadvantage
Revised: Note that, the experiment gives a slight disadvantage

Original: to NECST as it compresses every image individually, while JPEG compresses multiple images together
Revised: to NECST as it compresses every image individually, while JPEG compresses multiple images together

Original: We report the average distortions for sampled images from the test set.
Revised: We report the average distortions for sampled images from the test set.

Original: Unfortunately, we were unable to run the end-to-end for some scenarios and samples due to errors in the decoding (LDPC decoding, invalid JPEGs etc.).
Revised: Unfortunately, we were unable to run the end-to-end for some scenarios and samples due to errors in the decoding (LDPC decoding, invalid JPEGs etc.).

Original: D. VAE-LDPC SYSTEM
Revised: D. VAE-LDPC SYSTEM

Original: For the VAE-LDPC system, we place a uniform prior over all the possible latent codes and compute the KL penalty term between this prior p ( y ) and the random variable q noisy enc ( y | x ; φ, (cid:15) ) .
Revised: For the VAE-LDPC system, we place a uniform prior over all the possible latent codes and compute the KL penalty term between this prior p ( y ) and the random variable q noisy enc ( y | x ; φ, (cid:15) ) .

Original: The learning rate, batch size, and choice of architecture are data-dependent and ﬁxed to match those of
Revised: The learning rate, batch size, and choice of architecture are data-dependent and ﬁxed to match those of

Original: NECST as outlined in Section C.
Revised: NECST as outlined in Section C.

Original: However, we use half the number of bits as allotted for NECST so that during LDPC channel coding, we can double the codeword length in order to match the rate of our model.
Revised: However, we use half the number of bits as allotted for NECST so that during LDPC channel coding, we can double the codeword length in order to match the rate of our model.

Original: D. I NTERPOLATION IN L ATENT S PACE
Revised: D. I NTERPOLATION IN L ATENT S PACE

Original: We show results from latent space interpolation for two additional datasets: SVHN and celebA. We used 500 bits for SVHN and 1000 bits for celebA across channel noise levels of [0.0, 0.1, 0.2, 0.3, 0.4, 0.5].
Revised: We show results from latent space interpolation for two additional datasets: SVHN and celebA. We used 500 bits for SVHN and 1000 bits for celebA across channel noise levels of [0.0, 0.1, 0.2, 0.3, 0.4, 0.5].

Original: M ARKOV CHAIN IMAGE GENERATION
Revised: M ARKOV CHAIN IMAGE GENERATION

Original: We observe that we can generate diverse samples from the data distribution after initializing the chain with both: (1) examples from the test set and (2) random Gaussian noise x 0 ∼ N (0 , 0 . 5) .
Revised: We observe that we can generate diverse samples from the data distribution after initializing the chain with both: (1) examples from the test set and (2) random Gaussian noise x 0 ∼ N (0 , 0 . 5) .

Original: D. D OWNSTREAM CLASSIFICATION
Revised: D. D OWNSTREAM CLASSIFICATION

Original: Following the setup of (Grover & Ermon (2018)), we used standard implementations in sklearn with default parameters for all 8 classiﬁers with the following exceptions:
Revised: Following the setup of (Grover & Ermon (2018)), we used standard implementations in sklearn with default parameters for all 8 classiﬁers with the following exceptions:

Original: KNN: n neighbors= 2.
Revised: KNN: n neighbors= 2.

Original: DF: max depth= 3.
Revised: DF: max depth= 3.

Original: RF: max depth=5, n estimators=10, max features= 4.
Revised: RF: max depth=5, n estimators=10, max features= 4.

Original: MLP: alpha= 5.
Revised: MLP: alpha= 5.

Original: SVC: kernel=linear, C=0.
Revised: SVC: kernel=linear, C=0.

Original: [Figure]
Revised: [Figure]

Original: D. D ECODING TIME
Revised: D. D ECODING TIME

Original: We show the average decoding times for the randomly generated bitstring data, averaged overruns:
Revised: We show the average decoding times for the randomly generated bitstring data, averaged overruns: