D ENOISING D IFFUSION E RROR C ORRECTION C ODES
Anonymous authors
Paper under double-blind review
A BSTRACT
Error correction code (ECC) is an integral part of the physical communication layer, ensuring reliable data transfer over noisy channels.
Recently, neural decoders have demonstrated their advantage over classical decoding techniques.
However, recent state-of-the-art neural decoders suffer from high complexity and lack the important iterative scheme characteristic of many legacy decoders.
In this work, we propose to employ denoising diffusion models for the soft decoding of linear codes at arbitrary block lengths.
Our framework models the forward channel corruption as a series of diffusion steps that can be reversed iteratively.
Three contributions are made: (i) a diffusion process suitable for the decoding setting is introduced, (ii) the neural diffusion decoder is conditioned on the number of parity errors, which indicates the level of corruption at a given step, (iii) a line search procedure based on the code’s syndrome obtains the optimal reverse diffusion step size.
The proposed approach demonstrates the power of diffusion models for ECC and is able to achieve state of the art accuracy, outperforming the other neural decoders by sizable margins, even for a single reverse diffusion step.
Our code is attached as supplementary material.
I NTRODUCTION
Reliable digital communication is of major importance in the modern information age and involves the design of codes that can be robustly decoded despite noisy transmission channels.
The target decoding is defined by the NP-hard maximum likelihood rule, and the efficient decoding of commonly employed families of codes, such as algebraic block codes, remains an open problem.
Recently, powerful learning-based techniques have been introduced.
Model-free decoders (O’Shea & Hoydis, 2017; Gruber et al., 2017;
Kim et al., 2018) employ generic neural networks and may potentially benefit from the application of powerful deep architectures that have emerged in recent years in various fields.
A Transformer-based decoder that is able to incorporate the code into the architecture has been recently proposed by Choukroun & Wolf (2022).
It outperforms existing methods by sizable margins, at a fraction of their time complexity.
The decoder’s objective in this model is to predict the noise corruption, to recover the transmitted codeword (Bennatan et al., 2018).
Deep generative neural networks have shown significant progress over the last years.
Denoising
Diffusion Probabilistic Models (DDPM)
(Ho et al., 2020b) are an emerging class of likelihood-based generative models.
Such methods use diffusion models and denoising score matching to generate new samples, for example, images (Dhariwal & Nichol, 2021) or speech (Chen et al., 2020a).
The
DDPM model learns to perform a reversed diffusion process on a Markov chain of latent variables, and generates samples by gradually removing noise from a given signal.
One major drawback of model-free approaches is the high space/memory requirement and time complexity that hamper its deployment on constrained hardware.
Moreover, the lack of an iterative solution means that both highly and slightly corrupted codewords go through the same computationally demanding neural decoding procedure.
In this work, we consider the error correcting code paradigm via the prism of diffusion processes.
The channel codeword corruption can be viewed as an iterative forward diffusion process to be reversed via an adapted DDPM.
As far as we can ascertain, this is the first adaptation of diffusion models to error correction codes.
Beyond the conceptual novelty, we make three technical contributions: (i) our framework is based on an adapted diffusion process that simulates the coding and transmission processes, (ii) we further condition the denoising model on the number of parity-check errors, as an indicator of the signal’s level of corruption, and (iii) we propose a line-search procedure that minimizes the denoised code syndrome, in order to provide an optimal step size for the reverse diffusion.
Applied to a wide variety of codes, our method outperforms the state-of-the-art learning-based solutions by very large margins, employing extremely shallow architectures.
Furthermore, we show that even a single reverse diffusion step with a controlled step size can outperform concurrent methods.
R ELATED W ORKS
The emergence of deep learning for communication and information theory applications has demonstrated the advantages of neural networks in many tasks, such as channel equalization, modulation, detection, quantization, compression, and decoding (Ibnkahla, 2000).
Model-free decoders employ general neural network architectures (Cammerer et al., 2017; Gruber et al., 2017;
Kim et al., 2018;
Bennatan et al., 2018).
However, the exponential number of possible codewords makes the decoding of large codes unfeasible.
Bennatan et al.
(2018) preprocess the channel output to allow the decoder to remain provably invariant to the transmitted codeword and to eliminate risks of overfitting.
Model-free approaches generally make use of multilayer perceptron networks or recurrent neural networks to simulate the iterative process existing in many legacy decoders (Gruber et al., 2017;
Kim et al., 2018; Bennatan et al., 2018).
However, many architectures have difficulties in learning the code or analyzing the reliability of the output, and require prohibitive parameterization or expensive graph permutation preprocessing (Bennatan et al., 2018).
Recently, Choukroun & Wolf (2022) proposed the Error Correction Code Transformer (ECCT), obtaining SOTA performance.
The model embeds the signal elements into a high-dimensional space where analysis is more efficient, while the information about the code is integrated via a masked self-attention mechanism.
Diffusion Probabilistic Models were first introduced by Sohl-Dickstein et al.
(2015), who presented the idea of using a slow iterative diffusion process to break the structure of a given distribution while learning the reverse neural diffusion process, in order to restore the structure in the data.
Song &
Ermon (2019) proposed a new score-based generative model, building on the work of Hyv ¨ arinen &
Dayan (2005), as a way of modeling a data distribution using its gradients, and then sampling using
Langevin dynamics (Welling & Teh, 2011).
The DDPM method of Ho et al.
(2020b) is a generative model based on the neural diffusion process that applies score matching for image generation.
Song et al.
(2020b) leverage techniques from stochastic differential equations to improve the sample quality obtained by score-based models;
Song et al.
(2020a)
and Nichol & Dhariwal (2021a) propose methods for improving sampling speed;
Nichol & Dhariwal (2021a) and Saharia et al.
(2021) demonstrated promising results on the difficult
ImageNet generation task, using upsampling diffusion models.
Several extensions to other fields, such as audio (Kong et al., 2020; Chen et al., 2020b), have been proposed.
B ACKGROUND
We provide in this section the necessary background on error correction coding and DDPM.
Coding We assume a standard transmission that uses a linear code C .
The code is defined by the binary generator matrix G of size k × n and the binary parity check matrix H of size ( n − k ) × n defined such that GH T = 0 over the order 2 Galois field GF (2) .
The input message m ∈ { 0 , 1 } k is encoded by G to a codeword x ∈ C ⊂ { 0 , 1 } n satisfying
Hx = 0 and transmitted via a Binary-Input Symmetric-Output channel, e.g., an AWGN channel.
Let y denote the channel output represented as y =
x s + ε , where x s denotes the Binary Phase
Shift Keying (BPSK) modulation of x (i.e., over {± 1 } ), and ε is a random noise independent of the transmitted x .
The main goal of the decoder f : R n → R n is to provide a soft approximation ˆ x = f ( y ) of the codeword.
We follow the preprocessing of Bennatan et al.
(2018); Choukroun & Wolf (2022), in order to remain provably invariant to the transmitted codeword and to avoid overfitting.
The preprocessing transforms y to a vector of dimensionality 2 n − k defined as
[Equation]
where, [ · , · ] denotes vector concatenation, | y | denotes the absolute value (magnitude) of y and s ( y ) ∈ { 0 , 1 } n − k denotes the binary code syndrome .
The syndrome is obtained via the GF (2) multiplication of the binary mapping of y with the parity check matrix such that
[Equation]
The induced parameterized decoder ϵ θ : R 2 n − k
→ R n with parameters θ aims to predict the multiplicative noise denoted as ˜ ε and defined such that y = x s · ˜ ε .
The prediction of the multiplicative noise instead of the additive physical one is done in order to remain invariant to the transmitted codeword ( | y | = | x s ˜ ε | = | ˜ ε | ), thereby avoiding the risk of code overfitting, as described by Bennatan et al.
(2018) and the proof of lemma 1 of Richardson & Urbanke (2001).
The final prediction takes the form ˆ x s = sign( y · ϵ θ ( | y | , Hy b ) ).
Denoising Diffusion Probability Model (DDPM)
Ho et al.
(2020a) assume a data distribution x 0 ∼ q ( x ) and a Markovian noising process q that gradually adds noise to the data to produce noisy samples { x i } Ti =1 .
Each step of the corruption process adds Gaussian noise according to some variance schedule given by β t such that
[Equation]
q ( x t | x 0 ) can be expressed as a Gaussian distribution such that, with α t := 1 − β t and ¯ α t :=
{+(cid:81) ts =
+}
α s , we have
[Equation]
The intractable reverse diffusion process q ( x t − 1 | x t ) approaches a diagonal Gaussian distribution as β t −−−→ t →∞ 0 (Sohl-Dickstein et al., 2015) and can be approximated using a neural network p θ ( x t ) in order to predict the Gaussian statistics.
The model is trained by stochastically optimizing the random terms of the variational lower bound of the negative log-likelihood function.
One can find via Bayes’ theorem that the posterior q ( x t − 1 | x t , x 0 ) is also Gaussian, making the objective a sum of tractable KL divergences between Gaussians.
Ho et al.
(2020a) found a more practical objective, defined via the training of a model ϵ DDPMθ ( x t , t ) that predicts the additive noise ε from Eq.
4 as follows
[Equation]
The distribution q ( x T ) is assumed to be a nearly isotropic Gaussian distribution, such that sampling x T is trivial.
Thus, the reverse diffusion process is given by the following iterative process
[Equation]
D ENOISING D IFFUSION E RROR C ORRECTION C ODES
We present the elements of the proposed denoising diffusion for decoding and the proposed architecture, together with its training procedure.
An illustration of the coding setting and the proposed decoding framework are given in Figure 1.
4. D ATA T RANSMISSION AS A F ORWARD D IFFUSION P ROCESS
Given a codeword x 0 sampled from the Code distribution x 0 ∼ q ( x ) , we propose to define the codeword transmission procedure y =
x 0 + σε as a forward diffusion process adding a small amount of Gaussian noise to the sample in t steps with t ∈ (0 , . .
. , T ) , where the step sizes are controlled by a variance schedule { β t } Tt
=0 .
In our setting, we propose the following unscaled forward diffusion
[Equation]
Thus, for a given received word y and a corresponding t , we consider y as a codeword that has been corrupted gradually, such that for ε ∼ N (0 , I )
[Equation]
where ¯ β t = (cid:80) ti =
1 β i and σ defines the level of corruption of the AWGN channel.
Thus, the transmission of data over noisy communication channels can be defined as a modified iterative diffusion process to be reversed for decoding.
4. D ECODING AS A R EVERSE D IFFUSION P ROCESS
Following
Bayes’ theorem, the posterior q ( x t − 1 | x t , x 0 ) is a
Gaussian such that q
[Equation]
x t | x t − 1 , x 0 ) ∼ N ( x t ; ˜ µ t ( x t , x 0 ) , ˜ β t I ) , where, according to Eq. 8, we have
[Equation]
The full derivation is given in the Appendix A. Similarly to (Sohl-Dickstein et al., 2015; Ho et al., 2020b), we wish to approximate the intractable Gaussian reverse diffusion process q ( x t − 1 | x t ) such that
[Equation]
with fixed variance ˜ β t .
Following the simplified objective of Ho et al.
(2020b), one would adapt the negative log-likelihood approximation such that the decoder predicts the additive noise of the adapted diffusion process and
[Equation]
One interesting property of the syndrome-based approach of Bennatan et al.
(2018) is that, similarly to denoising diffusion models, in order to retrieve the original codeword, the decoder’s objective is to predict the channel’s noise.
However, the syndrome-based approach enforces the prediction of the multiplicative noise ˜ ε instead of the additive noise ε , in contrast to classic diffusion models.
We note, however, that the exact value of the multiplicative noise is not important for hard decoding, but only its sign since x s = sign( y ˜ ε ) .
two only signed codewords: ± (1 , 1 , 1) .
The colors are defined by Maximum Likelihood decoding.
Evidently, the denoising diffusion model reverses noisy codes towards the right distribution.
An illustration of the forward process for this code is provided in Appendix B.
Therefore, we propose to learn the hard (i.e., the sign) prediction of the multiplicative noise using the binary cross entropy loss as a surrogate objective, such that
[Equation]
(12) where the target binary multiplicative noise is defined as ˜ ε b = bin (cid:0) x 0 ( x 0 + (cid:112) ¯ β t ε ) (cid:1) , and BCE denotes the binary cross entropy loss.
4. D ENOISING VIA P ARITY C HECK C ONDITIONING
The reverse denoising process of traditional DDPM is conditioned by the time step.
Thus, by sampling Gaussian noise, which is assumed as equivalent to step t = T , one can fully reverse the diffusion by up to T iterations.
In our case, we are not interested in a generative model, but in an exact iterative denoising scheme, where the original signal is only corrupted to a measured extent.
Moreover, a given noisy code conveys information about the level of noise via its syndrome, since s ( y ) =
Hy =
Hx + Hz =
Hz .
Fig. 3 illustrates the impact of noise on the number of parity check errors.
Evidently, one can approximate an injective function between the number of parity check errors and the amount of noise.
Such a function is a direction indication of the proximity of the current iterate to a solution (codeword).
Therefore, we suggest conditioning the diffusion decoder according to the number of parity check errors e t , such that e t :=
e ( x t ) =
(cid:80) n − k i
=
1 s ( x t ) i ∈ { 0 , . . .
n − k } .
The resulting training objective is now given by
[Equation]
Following this logic, the number of required denoising steps T = n − k is set as the maximum number of parity check errors.
Similarly to the classical DDPM training procedure, sampling a time step t ∼ U (0 , . . .
, T ) produces noise, which in turn induces a certain number of parity errors.
The training procedure of our method is given in Alg.
The framework assumes a random ”time” sampling, producing a noise and then a syndrome to be corrected.
Note that, our model-free solution is invariant to the transmitted codeword, and the diffusion decoding can be trained with one single codeword (Alg. 1 line 1).
Since the denoising model predicts the multiplicative noise ˜ ε , at inference time it needs to be transformed into its additive counterpart ε in order to perform the gradient step in the original additive 1: x 0 ∈ C 2: Input: Parity check matrix H , noise schedule β 1 , ..., β T 3: repeat 4: t ∼ U ( [Equation] 1 , ..., T } ) 5:
[Equation] N (0 , I ) 6: [Equation] Take gradient descent step on:
BCE ( [Equation] 8: until converged 1: Input: Parity check matrix H , channel’s output y 2: for n − k iterations do 3: γ = e ( bin ( y )) 4: if γ = 0 then 5: return bin ( y ) 6: ˆ˜ ε =
ε θ
( y, γ ) ; ˆ ε = y − sign (ˆ˜ εy ) 7:
Get λ according to Eq.8: y = y − λ √ ¯ β γ βγ ¯ β γ + β γ ˆ ε 9: return bin ( y ) diffusion process domain.
We obtain the additive noise by subtracting the modulated predicted codeword sign (ˆ x ) from the noisy signal, such that
[Equation]
Therefore, following Eq. 9, at inference time the reverse process is given by
[Equation]
The inference procedure is defined in Alg.2.
If the syndrome is non-zero, we predict the multiplicative noise, extract the corresponding additive noise, and perform the reverse step.
We illustrate in
Fig.2 the reverse diffusion dynamics (gradient field) for a (3 , 1) repetition code, i.e., G = (1 , 1 , 1) .
4. S YNDROME - BASED L INE S EARCH FOR R EVERSE D IFFUSION S TEP S IZE
One major limitation of the generative neural diffusion process is the large number of diffusion steps required - generally a thousand - in order to generate high-quality samples.
Several methods proposed faster sampling procedures in order to accelerate data generation via schedule subsampling or step size correction (Nichol & Dhariwal, 2021b; San-Roman et al., 2021).
In our configuration, one can assess the quality of the denoised signal via the value of its syndrome, i.e., the number of parity check errors, while a zero syndrome means a valid codeword.
Therefore, we propose to find the optimal step size λ by solving the following optimization problem
[Equation]
[Equation] where s ( · ) denotes the syndrome computed over GF (2) as in Eq. 2.
While many line-search (LS) methods exist in numerical optimization (Nocedal & Wright, 2006), since the objective is highly non-differentiable, we suggest adopting a grid search procedure such that the search space becomes restricted to λ ∈
I where I is a predefined discrete segment.
This parallelizable procedure reduces the number of iterations by a sizable factor, as shown in Section 5.
Details regarding the grid-search procedure are discussed in Appendix C.
4. A RCHITECTURE AND T RAINING
The state-of-the-art ECCT architecture of Choukroun & Wolf (2022) is used as ϵ θ .
In this architecture, the capacity of the model is defined according to the chosen embedding dimension d and the number of self-attention layers N .
In order to condition the network by the number of parity errors e t ∈ { 0 , . . . , n − k } , we employ a d dimensional one hot encoding multiplied via Hadamard product with the initial elements’ embedding of the ECCT.
Denoting the ECCT’s embedding of the i element as ϕ i , the new embedding is defined as ˜ ϕ
i = ϕ
i ⊙ ψ ( e t ) , ∀ i , where ψ denotes the n − k one hot embedding.
As a transformation of the syndrome, e t remains also invariant to the codeword.
Additional details on the DDECCT architecture are given in Appendix F.
[Table] The discrete grid search of λ is uniformly sampled over I =
[1 , 20] with 20 samples, in order to find the optimal step size.
A denser or a code adaptive sampling may improve the results, according to a predefined computation-speed trade-off.
We show the distribution of optimal λ in Appendix C.
The Adam optimizer (Kingma & Ba, 2014) is used with 128 samples per mini-batch, forepochs, with 1000 mini-batches per epoch.
The noise scheduling is constant and set to β t = 0 .
01 , ∀ t .
An extended discussion regarding the β scheduler can be found in Appendix G. We initialized the learning rate to 10 − 4 coupled with a cosine decay scheduler down to 5 · 10 − 6 at the end of training.
No warmup (Xiong et al., 2020) was employed.
Training and experiments were performed on a 12GB Titan V GPU.
The total training time ranged from 12 to 24 hours depending on the code length, and no optimization of the self-attention mechanism was employed.
Per epoch, the training time was in the range of 19-40 and 40-102 seconds for the N = 2 , 6 architectures, respectively.
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Figure]
To evaluate our method we train the proposed architecture with three classes of linear block codes: Low-Density Parity Check (LDPC) codes (Gallager, 1962), Polar codes (Arikan, 2008) and
The proposed architecture is defined solely by the number of encoder layers N and the dimension of the embedding d .
We compare our method with the BP algorithm (Pearl, 1988), the recent Autoregressive hyper-network BP of Nachmani & Wolf (2021) (AR BP) and the SOTA ECCT (Choukroun & Wolf, 2022).
Since our decoder is based on the ECCT, the contribution of the diffusion model scheme is pertinent in comparing our results with ECCT since they have similar architectures and capacities.
Our method’s overhead over ECCT is by a factor of the number of diffusion steps L , i.e., a complexity of O ( LN ( d 2 (2 n − k ) + hd )) , where h is the complexity of the self-attention module.
We refer the reader to Choukroun & Wolf (2022) for a detailed complexity analysis of the ECCT.
Details about the computational overhead of the DDECCT are given in Appendix F. Note that LDPC codes are designed specifically for BP-based decoding (Richardson et al., 2001).
The results are reported as bit error rates (BER) for different normalized SNR values ( Eb/N 0 ).
We follow the testing benchmark of (Nachmani & Wolf, 2019; Choukroun & Wolf, 2022).
During testing, our decoder decodes at least 10 5 random codewords, to obtain at least 500 frames with errors at each SNR value.
All baseline results were obtained from the corresponding papers.
The results are reported in Tab. 1, where we present the negative natural logarithm of the BER.
For each code, we present the results of the BP-based competing methods for 5 and 50 iterations (first and second rows), corresponding to a neural network with 10 and 100 layers, respectively.
As in (Choukroun & Wolf, 2022), our framework’s performance with Line Search (LS) as described in
Section 4.4 is evaluated for six different architectures, with N = { 2 , 6 } and d = { 32 , 64 , 128 } , respectively (first to third rows).
A comparison to the SCL Polar decoder of Tal & Vardy (2015) is given in Appendix H. BER plots with respect to the SNR are given in Appendix I.
As can be seen, our approach outperforms the current SOTA results (obtained by ECCT) by extremely large margins on several codes, at a fraction of the capacity.
Especially for shallow models, the difference can be an order of magnitude.
Performance is closer with short high-rate codes, for which ECCT performance is already very high.
We present in Figure 5 the performance of the proposed DDECCT on larger codes.
As can be seen, DDECCT can learn to efficiently decode larger codes and outperforms ECCT.
We present in Table 2 the difference in accuracy ∆ between the line search procedure and the regular reverse diffusion.
We also present convergence statistics (mean and standard deviation of the number of iterations) for the regular reverse diffusion and the line search procedure.
The full table with the statistics for all of the codes is given in Appendix E. Evidently, the line search procedure enables extremely fast convergence, requiring as little as one iteration for high SNR.
Note that we measure the number of iterations required to reach a syndrome of zero failed checks.
We do not apply early stopping to the decoding, which could reduce the average number of iterations even further if the decoder stagnates and does not converge to zero syndrome.
5. N ON -G AUSSIAN C HANNEL
We test our framework on a non-Gaussian Rayleigh fading channel, which is often used for simulating the propagation environment of a signal, e.g., for wireless devices.
In this fading model, the transmission of the codeword x ∈ { 0 , 1 } n is defined as y =
hx s + z , where h is an n -dimensional i.i.d.
Rayleigh-distributed vector with a scale parameter α , and z ∼ N (0 , σ 2 I n ) .
In our simulations, we assume a high scale α = 1 in order to easily compare and reproduce the results, while the level of Gaussian noise and the testing procedure remain the same as described in the paper.
The overall variance of the transmitted codeword y in the Rayleigh channel is roughly twice the AWGN’s on the tested SNR range.
The results are presented in Figure 4.
As can be observed, our method is still able to learn to decode, even under these very noisy fading channels.
5. BER EVOLUTION THROUGH ITERATION / TIME
We illustrate in Figure 6 the denoising process for several codes.
We show how the BER decreases with time for the regular proposed method and the augmented line search procedure.
We can observe the very fast convergence of the line search approach.
We further provide in Appendix D the performance of the proposed framework for one, two and three iteration steps.
We can see that LS enables outperforming the original ECCT, even with one step only.
C ONCLUSIONS
We present a novel denoising diffusion method for the decoding of algebraic block codes.
It is based on an adapted diffusion process that simulates the channel corruption we wish to reverse.
The method makes use of the syndrome as a conditioning signal and employs a line-search procedure to control the step size.
Since it inherits the iterative nature of the underlying process, both training and deployment are extremely efficient.
Even with very low-capacity networks, the proposed approach outperforms existing neural decoders by sizable margins for a broad range of code families
A U NSCALED D IFFUSION D ERIVATION
According to Bayes’ rule
[Equation]
where C ( x t , x 0 ) represents the constant term of the second-order equation.
Following the standard
Gaussian density function
[Equation]
the mean and variance can be parameterized as follows
[Equation]
B F ORWARD D IFFUSION P ROCESS V ISUALIZATION
We provide a visualization of the forward diffusion process as described in Section 4.1, using the three dimensional repetition code as discussed in Section 4.3.
Figure 7 presents random diffusion processes from one of the two valid codewords through time.
As can be seen, there is a migration from the valid codewords (depicted by either a blue or a red cross) to the vicinity of invalid words (black crosses).
C L INE S EARCH H ISTOGRAMS AND C OMPLEXITY C. H ISTOGRAMS
Figure 8 presents the distribution of the optimal step size λ for several codes.
Each code presents a different distribution of the optimal step sizes, as can be seen from the high variance of the x-axis. C.
C OMPLEXITY OVERHEAD OF THE LINE SEARCH PROCEDURE
The computation of the syndrome consists of a series of efficient binary operations (xor) inducing a computational complexity that is proportional to the density of the code. The line search consists of the parallel computation of the syndrome, over the multiple words obtained for different λ values sampled over a predefined grid. Thus, the time complexity of the line-search procedure can be reduced to the very efficient computation of the syndrome which can be assumed as constant.
Without parallelization, the complexity is linear with the grid size, but the overall process remains extremely efficient.
Table 3 presents the performance of the proposed framework for one, two, and three iteration steps with the regular reverse diffusion method and with the proposed line-search approach.
As can be seen, the line-search approach improves over the regular reverse diffusion by orders of magnitude
[Table] [Table]
E N UMBER OF D IFFUSION S TEPS
Table 4 presents the convergence statistics (mean and standard deviation of the number of iterations) for the regular reverse diffusion and the line search procedure.
Evidently, the line-search approach substantially reduces the number of steps, especially for low SNRs where the improvement can reach one order of magnitude.
[Table]
[Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
F D ENOISING D IFFUSION E RROR C ORRECTION C ODE T RANSFORMER F. P ARITY - CHECK C ONDITIONING A RCHITECTURE
The DDECCT architecture is depicted in Figure 9.
In order to condition the network on the number of parity errors e t ∈ { 0 , . . . , n − k } , we employ a d dimensional one hot encoding multiplied via Hadamard product with the initial elements’ embedding of the ECCT.
Denoting the ECCT’s embedding of the i element as ϕ i , the new embedding is defined as ˜ ϕ
i = ϕ
i ⊙ ψ ( e t ) , ∀ i , where ψ denotes the n − k one hot embedding.
F. DDECCT C OMPUTATIONAL C OMPLEXITY
The computational overhead consists of the conditioning, which is a negligible Hadamard product of the initial ECCT embedding with the one-hot encoding of the number of parity-check errors, the parallel line-search procedure (which scales linearly with the density of the code, and most codes are sparse), and the number of reverse diffusion iterations, which is reduced to extremely few iterations
Therefore, the large improvement of DDECCT over ECCT is obtained while employing a modest increase in computational complexity.
Most importantly, the space complexity, determining the capacity of the network, is extremely reduced with the DDECC framework, since even shallow
DDECCT architectures outperform deep ECCT models.
It should also be mentioned that as described in the introduction, the iterative DDECC framework supports a differential treatment of the samples based on their level of corruption, what is not possible with ECCT.
G B ETA SCHEDULING
The choice of the scheduling range, i.e. t ∈ { 0 , 1 , . .
, T } is explained as follows.
In contrast to classical denoising diffusion models used as generative models, the conditioning in our model is performed over the number of parity-check errors in order to obtain an estimate of the proximity to the solution.
Thus, T is now defined as the maximum possible number of parity-check errors, i.e. n − k .
The values of the beta schedule have been set empirically such that the cumulative sum of the diffusion noise roughly corresponds to the average training noise statistics, i.e., (cid:112) ¯ β t
≈ σ such that y = x + σz , z ∼ N (0 , I ) , as described in Eq. 8.
In addition to the empirical support, constant scheduling has been chosen to induce a uniform treatment of the noise space.
Finally, the full method uses the line-search procedure, reducing the need to precisely define or tune the scheduler.
H C OMPARISON WITH S UCCESSIVE C ANCELLATION L IST (SCL) P OLAR D ECODER
Table 5 compares the performance of ECCT and DDECCT to the SOTA SCL Polar decoder Tal & Vardy (2015) for several Polar Codes.
The SCL decoder has a time and space complexity of
O ( LN log N ) and O ( LN ) ,
respectively.
We tested the SCL algorithm for L = { 1 , 4 } and we sampled only 10 5 noisy codewords because of the high complexity and non-parallel application of the SCL algorithm.
The proposed shallow ECCTs are able to compete and even surpass the SCL for some of the codes and SNRs. The corresponding DDECCT outperforms the other methods by very large margins.
[Table]
Increasing the capacity of the network, especially with more layers, is expected to lead to better results as demonstrated for LDPC codes. Similarly, SCL with bigger lists would obtain improved accuracy
[Table]
I R ESULTS OVER MULTIPLE SNR VALUES
Figure 10 depicts BER plots comparing the performance of ECCT and DDECCT for several codes.