C OMMUNICATION A LGORITHMS VIA D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Coding theory is a central discipline underpinning wireline and wireless modems that are the workhorses of the information age.
Progress in coding theory is largely driven by individual human ingenuity with sporadic breakthroughs over the past century.
In this paper we study whether it is possible to automate the discovery of coding and decoding algorithms via deep learning.
We study a family of sequential codes parametrized by recurrent neural network (RNN) architectures.
We show that creatively designed and trained RNN architectures can decode well known sequential codes such as the convolutional and turbo codes with close to optimal performance on the additive white Gaussian noise (AWGN) channel, which itself is achieved by breakthrough algorithms of our times (Viterbi and BCJR decoders, representing dynamic programing and forward-backward algorithms).
We show strong generalizations, i.e., we train at a speciﬁc signal to noise ratio and block length but test at a wide range of these quantities, as well as robustness and adaptivity to deviations from the AWGN setting.
Finally, we use the RNN architectures to design new nonlinear codes that represent a major progress in the long standing open problem of communicating reliably over the AWGN channel with noisy output feedback.
I NTRODUCTION
Reliable digital communication, both wireline (ethernet, cable and DSL modems) and wireless (cellular, satellite, deep space), is a primary workhorse of the modern information age.
A critical aspect of reliable communication involves the design of codes that allow transmissions to be robustly (and computationally efﬁciently) decoded under noisy conditions.
This is the discipline of coding theory; over the past century and especially the past 70 years (since the birth of information theory (Shannon, 1948)) much progress has been made in the design of near optimal codes.
Landmark codes include convolutional codes, turbo codes, low density parity check (LDPC) codes and, recently, polar codes.
The impact on humanity is enormous – every cellular phone designed uses one of these codes, which feature in global cellular standards ranging from the 2nd generation to the 5th generation respectively, and are text book material (Richardson & Urbanke, 2008).
The canonical setting is one of point-to-point reliable communication over the additive white Gaussian noise (AWGN) channel and performance of a code in this setting is its gold standard.
The
AWGN channel ﬁts much of wireline and wireless communications although the front end of the receiver may have to be speciﬁcally designed before being processed by the decoder (example: intersymbol equalization in cable modems, beamforming and sphere decoding in multiple antenna wireless systems); again this is text book material (Tse & Viswanath, 2005).
There are two long term goals in coding theory: ( a ) design of new, computationally efﬁcient, codes that improve the state of the art (probability of correct reception) over the AWGN setting.
Since the current codes already operate close to the information theoretic “Shannon limit”, the emphasis is on robustness and adaptability to deviations from the AWGN settings (a list of channel models motivated by practical settings, (such as urban, pedestrian, vehicular) in the recent 5th generation cellular standard is available in Annex B of 3GPP TS 36.101.)
(b) design of new codes for multi-terminal (i.e., beyond point-to-point) settings – examples include the feedback channel, the relay channel and the interference channel.
Progress over these long term goals has generally been driven by individual human ingenuity and, beﬁttingly, is sporadic.
For instance, the time duration between convolutional codes (2nd generation cellular standards) to polar codes (5th generation cellular standards) is over 4 decades.
Deep learning is fast emerging as capable of learning sophisticated algorithms from observed data (input, action, output) alone and has been remarkably successful in a large variety of human endeavors (ranging from language (Mikolov et al., 2013) to vision (Russakovsky et al., 2015) to playing
Go (Silver et al., 2016)).
Motivated by these successes, we posit that deep learning methods can play a crucial role in solving both the aforementioned goals of coding theory and show that we can make signiﬁcant progress on both these goals in this paper.
While the learning framework is clear and there is virtually unlimited training data available, there are two main challenges: ( a ) The space of codes is very vast and the sizes astronomical; for instance a rate 1/2 code over 100 information bits involves designing 2 100 codewords in a 200 dimensional space.
Computationally efﬁcient encoding and decoding procedures are a must, apart from high reliability over the AWGN channel.
( b ) Generalization is highly desirable across block lengths and data rate that each work very well over a wide range of channel signal to noise ratios (SNR).
In other words, one is looking to design a family of codes (parametrized by data rate and number of information bits) and their performance is evaluated over a range of channel SNRs.
In part due to these challenges, recent deep learning works on coding theory focus on decoding known codes using data-driven neural decoders and have limited themselves to small block lengths (O’Shea & Hoydis, 2017; D¨orner et al., 2017; Gruber et al., 2017).
Even here, generalization is difﬁcult (for instance, even when nearly 90% of the codewords of a rate 1/2 polar code overinformation bits are exposed to the neural decoder (Gruber et al., 2017)).
The main challenge is to restrict oneself to a class of codes that neural networks can naturally encode and decode.
In this paper, we restrict ourselves to a class of sequential encoding and decoding schemes, of which convolutional and turbo codes are part of.
These sequential coding schemes naturally meld with the family of recurrent neural network (RNN) architectures, which have recently seen large success in a wide variety of time-series tasks.
The ancillary advantage of sequential schemes is that arbitrarily long information bits can be encoded and also at a large variety of coding rates.
Working within sequential codes parametrized by RNN architectures, we make the following contributions.
(1) Focusing on convolutional codes we aim to decode them on the AWGN channel using RNN architectures.
Efﬁcient optimal decoding of convolutional codes has represented historically fundamental progress in the broad arena of algorithms; optimal bit error decoding is achieved by the ‘Viterbi decoder’ (Viterbi, 1967) which is simply dynamic programming or Dijkstra’s algorithm on a speciﬁc graph (the ‘trellis’) induced by the convolutional code.
Optimal block error decoding is the BCJR decoder (Bahl et al., 1974) which is part of a family of forward-backward algorithms.
While early work had shown that vanilla-RNNs are capable in principle of emulating both Viterbi and BCJR decoders (Wang & Wicker, 1996; Sazl & Ik, 2007) we show empirically, through a careful construction of RNN architectures and training methodology, that neural network decoding is possi ble at very near optimal performances (both bit error rate (BER) and block error rate (BLER))
The key point is that we
train a RNN decoder at a speciﬁc
SNR
and
over
short information bit lengths
100 bits) and show strong generalization capabilities by testing over a wide range of SNR and block lengths (up to 10,000 bits). The speciﬁc training
SNR is closely related to the Shannon limit of the AWGN channel at the rate of the code and provides strong information theoretic collateral to our empirical results.
(2) Turbo codes are naturally built on top of convolutional codes, both in terms of encoding and decoding.
A natural generalization of our RNN convolutional decoders allow us to decode turbo codes at BER comparable to, and at certain regimes, even better than state of the art turbo decoders on the AWGN channel.
That data driven
, SGD-learnt, RNN architectures can decode comparably is fairly remarkable since turbo codes already operate near the Shannon limit of reliable communication over the AWGN channel.
(3) We show the afore-described neural network decoders for both convolutional and turbo codes are robust to variations to the AWGN channel model.
We consider a problem of contemporary interest: communication over a “bursty” AWGN channel (where a small fraction of noise has much higher variance than usual) which models inter-cell interference in OFDM cellular systems (used in 4G and 5G cellular standards) or co-channel radar interference.
We demonstrate empirically the neural network architectures can adapt to such variations and beat state of the art heuristics comfortably (despite evidence elsewhere that neural network are sensitive to models they are trained on (Szegedy et al., 2013)).
Via an innovative local perturbation analysis (akin to (Ribeiro et al., 2016)), we demonstrate the neural network to have learnt sophisticated preprocessing heuristics in engineering of real world systems (Li et al., 2013).
(4) So far we have restricted ourselves to neurally decoding ﬁxed encoding schemes (i.e., convolutional and turbo codes).
In the ﬁnal section of this paper, we demonstrate new RNN-driven encoders (with matching decoders) that operate signiﬁcantly better than state of the art on the AWGN channel with (noisy) output feedback.
While feedback does not improve the Shannon capacity of the AWGN channel (Shannon, 1956), it is known to provide better reliability at ﬁnite block lengths (Schalkwijk & Kailath, 1966), although very sensitive to even tiny amounts of noise in the output feedback; more generally any linear code incorporating the noisy output feedback cannot achieve a non-zero reliable rate of communication (Kim et al., 2007b) – this is very troubling since all practical codes are linear and linear codes are known to achieve capacity (without feedback) (Elias, 1955).
Our
RNN parameterized encoders are inherently nonlinear and map information bits directly to realvalued transmissions.
Their performance vastly improves the state of the art on the long standing open problem in information theory on communicating over the AWGN channel with noisy output feedback.
RNN DECODERS FOR SEQUENTIAL CODES
Among diverse families of coding scheme available in the literature, sequential coding schemes are particularly attractive.
They ( a ) are used extensively in mobile telephone standards including satellite communications, 3G, 4G, and LTE; ( b ) provably achieve performance close to the information theoretic limit; and ( c ) have a natural recurrent structure that is aligned with an established family of deep models, namely recurrent neural networks.
We consider the basic sequential code known as convolutional codes , and provide a neural decoder that can be trained to achieve the optimal classiﬁcation accuracy.
A standard example of a convolutional code is the rate-1/2 Recursive Systematic Convolutional (RSC) code .
The encoder performs a forward pass on a recurrent network shown in Figure 1 on binary input sequence b =
( b 1 , . . .
, b K ) , which we call message bits , with binary vector states ( s 1 , . . .
, s K ) and binary vector outputs ( c 1 , . .
. , c K ) , which we call transmitted bits or a codeword .
At time k with binary input b k ∈ { 0 , 1 } and the state of a two-dimensional binary vector s k =
( s k 1 , s k 2 ) , the output is a two-dimensional binary vector c k
= ( c k 1 , c k 2 ) =
(2 b k − 1 , 2( b k ⊕ s k 1 ) − 1) ∈ {− 1 , 1 } 2 , where x ⊕ y =
| x − y | .
The state of the next cell is updated as s k +
=
( b k ⊕ s k 1 ⊕
s k 2 , s k 1 ) .
Initial state is assumed to be 0, i.e., s
=
The 2 K output bits are sent over a noisy channel, with the canonical one being the AWGN channel: the received binary vectors y
=
( y 1 , . . .
, y K ) , which are called the received bits , are y ki
= c ki + z ki for all k ∈
[ K ] and i ∈ { 1 , 2 } , where z ki ’s are i.i.d.
Gaussian with zero mean and variance σ 2 .
Decoding a received signal y refers to (attempting to) ﬁnding the maximum a posteriori (MAP) estimate.
Due to the simple recurrent structure, efﬁcient iterative schemes are available for ﬁnding the MAP estimate for convolutional codes (vit; Bahl et al., 1974).
There are two MAP decoders depending on the error criterion in evaluating the performance: bit error rate (BER) or block error rate (BLER).
BLER counts the fraction of blocks that are wrongly decoded (assuming many such length- K blocks have been transmitted), and matching optimal MAP estimator is ˆ b = arg max b Pr( b | y ) .
Using dynamic programming
, one can ﬁnd the optimal MAP estimate in time linear in the block length
K , which is called the Viterbi algorithm .
BER counts the fraction of bits that are wrong, and matching optimal MAP estimator is ˆ b k = arg max b k Pr( b k | y ) , for all k = 1 , · · · , K .
Again using dynamic programming, the optimal estimate can be computed in O ( K ) time, which is called the
BCJR algorithm .
In both cases, the linear time optimal decoder crucially depends on the recurrent structure of the encoder.
This structure can be represented as a hidden Markov chain (HMM), and both decoders are special cases of general efﬁcient methods to solve inference problems on HMM using the principle of dynamic programming (e.g. belief propagation).
These methods efﬁciently compute the exact posterior distributions in two passes through the network: the forward pass and the backward pass.
Our ﬁrst aim is to train a (recurrent) neural network from samples, without explicitly specifying the underlying probabilistic model, and still recover the accuracy of the matching optimal decoders.
At a high level, we want to prove by a constructive example that highly engineered dynamic programming can be matched by a neural network which only has access to the samples.
The challenge lies in ﬁnding the right architecture and showing the right training examples.
Neural decoder for convolutional codes .
We treat the decoding problem as a K -dimensional binary classiﬁcation problem for each of the message bits b k .
The input to the decoder is a length2 K sequence of received bits y ∈ R 2 K each associated with its length- K sequence of “true classes” b ∈ { 0 , 1 } K .
The goal is to train a model to ﬁnd an accurate sequence-to-sequence classiﬁer.
The input y is a noisy version of the class b according to the rate-1/2 RSC code deﬁned in earlier in this section.
We generate N training examples ( y ( i ) , b ( i ) ) for i ∈
[ N ] according to this joint distribution to train our model.
We introduce a novel neural decoder for rate-1/2 RSC codes, we call N-RSC.
It is two layers of bi-direction Gated Recurrent Units (bi-GRU) each followed by batch normalization units, and the output layer is a single fully connected sigmoid unit.
Let W denote all the parameters in the model whose dimensions are shown in Figure 2, and f W ( y ) ∈
[0 , 1] K denote the output sequence.
The k -th output f W ( y ) k estimates the posterior probability Pr( b k = 1 | y ) , and we train the weights W to minimize the L w error with respect to a choice of a loss function (cid:96) ( · , · ) speciﬁed below:
[Equation]
As the encoder is a recurrent network, it is critical that we use recurrent neural networks as a building block.
Among several options of designing RNNs, we make three speciﬁc choices that are crucial in achieving the target accuracy: ( i ) bidirectional GRU as a building block instead of unidirectional
GRU; ( ii ) 2-layer architecture instead of a single layer; and ( iii ) using batch normalization.
As we show in Table 1 in Appendix C, unidirectional GRU fails because the underlying dynamic program requires bi-directional recursion of both forward pass and backward pass through the received sequence.
A single layer bi-GRU fails to give the desired performance, and two layers is sufﬁcient.
We show how the accuracy depends on the number of layer in Table 1 in Appendix C. Batch normalization is also critical in achieving the target accuracy.
Layer
Output dimension
Input (K, 2) bi-GRU (K, 400)
Batch Normalization (K, 400) bi-GRU (K, 400)
Batch Normalization (K, 400)
Dense (sigmoid)
[Equation]
K, 1)
Training.
We propose two novel training techniques that improve accuracy of the trained model signiﬁcantly.
First, we propose a novel loss function guided by the efﬁcient dynamic programming, that signiﬁcantly reduces the number of training example we need to show.
A natural L loss (which gives better accuracy than cross-entropy in our problem) would be (cid:96) ( f W ( y ( i ) )
k , b ( i ) k ) =
( f W ( y ( i ) )
k − b ( i ) k ) 2 .
Recall that the neural network estimates the posterior Pr( b k = 1 | y ( i ) ) , and the true label b ( i ) k is a mere surrogate for the posterior, as typically the posterior distribution is simply not accessible.
However, for decoding RSC codes, there exists efﬁcient dynamic programming that can compute the posterior distribution exactly.
This can signiﬁcantly improve sample complexity of our training, as we are directly providing Pr( b k = 1 | y ( i ) ) as opposed to a sample from this distribution, which is b ( i ) k .
We use a python implementation of BCJR in Taranalli (2015) to compute the posterior distribution exactly, and minimize the loss
[Equation]
Next, we provide a guideline for choosing the training examples that improve the accuracy.
As it is natural to sample the training data and test data from the same distribution, one might use the same noise level for testing and training.
However, this is not reliable as shown in Figure 3.
Channel noise is measured by Signal-to-Noise Ratio (SNR) deﬁned as − 10 log 10 σ 2 where σ 2 is the variance of the Gaussian noise in the channel.
For rate-1/2 RSC code, we propose using training data with noise level SNR train =
min { SNR test , 0 } .
Namely, matched training SNR if test SNR is below one, and otherwise ﬁx training SNR at one independent of the test SNR.
In Appendix E, we give a general formula for general rate- r codes, and provide an information theoretic justiﬁcation and empirical evidences showing that this is near optimal choice of training data.
[Figure]
In examples everywhere
Perhaps surprisingly, use the neural decoder trained on length 100 codes, and apply it directly to codes of length 10 ,and still meet the optimal performance.
Note that we only give 12 , 000 training examples, while the number of unique codewords is 2 10 , 000 .
This shows that the proposed neural decoder ( a ) can generalize to unseen codeword; and ( b ) seamlessly generalizes to signiﬁcantly longer block lengths.
More experimental results including other types of convolutional codes are provided in Appendix A.
Turbo codes are naturally built out of convolutional codes (both encoder and decoder) and represent some of the most successful codes for the AWGN channel (Berrou et al., 1993).
A corresponding stacking of multiple layers of the convolutional neural decoders leads to a natural neural turbo decoder which we show to match (and in some regimes even beat ) the performance of standard state of the art turbo decoders on the AWGN channel; these details are available in Appendix B. Unlike the convolutional codes, the state of the art (message-passing) decoders for turbo codes are not the corresponding MAP decoders, so there is no contradiction in that our neural decoder would beat the message-passing ones.
The training and architectural choices are similar in spirit to those of the convolutional code and are explored in detail in Appendix B.
N ON -G AUSSIAN CHANNELS : R OBUSTNESS AND A DAPTIVITY
In the previous sections, we demonstrated that the neural decoder can perform as well as the turbo decoder.
In practice, there are a wide variety of channel models that are suited for differing applications.
Therefore, we test our neural decoder under some canonical channel models to see how robust and adaptive they are.
Robustness refers to the ability of a decoder trained for a particular channel model to work well on a differing channel model without re-training .
Adaptivity refers to the ability of the learning algorithm to adapt and retrain for differing channel models.
In this section, we demonstrate that the neural turbo decoder is both adaptive and robust by testing on a set of non-Gaussian channel models.
[Figure]
Robustness The robustness test is interesting from two directions, other than obvious practical value.
Firstly, it is known from information theory that Gaussian noise is the worst case noise among all noise distributions with a given variance Cover & Thomas (2012); Lapidoth (1996).
However, practical decoders can have much worse performance on channels different than the ones they are trained for.
Second, this test poses an interesting challenge for neural decoders since deep neural networks are known to misclassify when tested against small adversarial perturbations Szegedy et al.
(2013); Goodfellow et al.
While we are not necessarily interested in adversarial perturbations to the input in this paper, it is important for the learning algorithm to be robust against differing noise distributions.
For the non-Gaussian channel, we choose the t-distribution family parameterized by parameter ν .
We test the performance of both the neural and turbo decoder in this experiment when ν =in Figure 5a and observe that the neural decoder performs signiﬁcantly better than the standard
Turbo decoder (also see Figure 19a in Appendix F).
In order to understand the reason for such a bad performance of the standard Turbo decoder, we plot the average output log-likelihood ratio (LLR) log p ( b k = 1) − log p ( b k = 1) as a function of the bit position in Figure 5b, when the input is all-zero codeword.
The main issue for the standard decoder is that the LLRs are not calculated accurately (see Figure 19b in Appendix F): the LLR is exaggerated in the t-distribution while there is some exaggeration in the neural decoder as well, it is more modest in its prediction leading to more contained error propagation.
[Figure]
Adaptivity : A great advantage of neural channel decoder is that the neural network can learn a decoding algorithm even if the channel does not yield to a clean mathematical analysis.
Consider a scenario where the transmitted signal is added with a Gaussian noise always, however, with a small probability, a further high variance noise is added.
The channel model is mathematically described as follows, with y i describing the received symbol and x i denoting the transmitted symbol at time instant i : y i
=
x i + z
i + w i , z i ∼ N (0 , σ 2 ) , and w i ∼ N (0 , σ 2 b ) with probability ρ and w i =with probability 1 − ρ , i.e., z i denotes the Gaussian noise whereas w i denotes the bursty noise.
This channel model accurately describes how radar signals (which are bursty) can create an interference for LTE in next generation wireless systems.
This model has attracted attention in communications systems community due to its practical relevance Sanders et al.
; Sanders (2014).
Under the aforesaid channel model, it turns out that standard Turbo coding decoder fails very badly SafaviNaeini et al. (2015).
The reason that the Turbo decoder cannot be modiﬁed in a straight-forward way is that the location of the bursty noise is a latent variable that needs to be jointly decoded along with the message bits.
In order to combat this particular noise model, we ﬁne-tune our neural decoder on this noise model, initialized from the AWGN neural decoder, and term it the bursty neural decoder.
There are
We demonstrate the performance of our AWGN neural decoder (trained on Gaussian noise) as well as standard turbo decoder (for Gaussian noise) on this problem, shown in Figure 6 when σ b =
We summarize the results of Figure 6: (1) standard turbo decoder not aware of bursty noise will result in complete failure of decoding.
(2) standard neural decoder still outperforms standard
Turbo Decoder.
(3) Bursty-neural-decoder outperforms Turbo Decoder using both state-of-the-art heuristics at σ 2 b = 3 . 5 and obtains performance approaching that of the better of the two schemes at other variances.
[Figure]
Interpreting the Neural Decoder
We try to interpret the action of the neural decoder trained under bursty noise.
To do so, we look at the following simpliﬁed model, where y i =
x i + z
i + w i where x i , y i , z i are as before, but w i
=
B during the 50 -th symbol in a 100 -length codeword.
We also ﬁx the input codeword to be the all-zero codeword.
We look at the average output LLR as a function of position for the one round of the neural decoder in Figure 7a and one round of BCJR algorithm in Figure 7b (the BER as a function of position is shown in Figure 20 in Appendix F).
A negative
LLR implies correct decoding at this level and a positive LLR implies incorrect decoding.
It is evident that both RNN and BCJR algorithms make errors concentrated around the mid-point of the codeword.
However, what is different between the two ﬁgures is that the scale of likelihoods of the two ﬁgures are quite different: the BCJR has a high sense of (misplaced) conﬁdence, whereas the
RNN is more modest in its assessment of its conﬁdence.
In the later stages of the decoding, the exaggerated sense of conﬁdence of BCJR leads to an error propagation cascade eventually toggling other bits as well.
N EW C ODES : AWGN CHANNEL WITH FEEDBACK
In previous sections, we have focused on decoding, via carefully designed and trained RNN architectures, of existing codes (convolutional and turbo) over the AWGN channel and variants.
One reason is that for the AWGN channel it is known that linear codes are known to be optimal (Elias, 1955; Urbanke & Rimoldi, 1998).
Designing new encoding schemes that outperform the strong state of the art (turbo, LDPC and polar codes) would represent major progress and remains an open problem.
In this section, we consider an important communication scenario:
AWGN channel with output feedback .
The decoder receives y k
=
c k + z k , z k ∼ N (0 , σ 2 ) at time k , which is fed back causally to the encoder; so the encoder has access to ˜ y k
= y k + w k , w k ∼ N (0 , σ 2 F ) at time k
+ 1 , which it could use to design its transmissions c t at times t ≥ k + 1 .
Here σ 2 F is the variance of the noise in the feedback received by the encoder (modeling receiver quantization losses and feedback transmission losses).
Although output feedback (with or without noise) does not improve the Shannon capacity of the
AWGN channel (Shannon, 1956), it is known to improve BER performance at ﬁnite block lengths.
This was sharply brought to focus by the work of Schalkwijk & Kailath (1966) which considered the case of perfect output feedback.
However, the celebrated scheme of Schalkwijk & Kailath (1966) is very sensitive to noise in the feedback (even precision errors when implemented in ﬂoating point arithmetic set the performance back dramatically).
More generally, it is known that any linear scheme that incorporates the noisy output feedback with a ﬁnite rate cannot have BER be vanishingly small, no matter how long the blocklength is (Kim et al., 2007a; 2010).
Practical efforts at incorporating output feedback has been less than successful, at times performing even worse than if the feedback were ignored.
As such, designing codes for the AWGN channel that efﬁciently makes use of the noisy output feedback is a long standing open problem in coding and communication theory; this is the focus of this section.
Architecture and Training .
We focus on the setting of rate 1/3 and information block length offor concreteness.
The key innovation is in the design of a new encoding scheme that matches the beneﬁts of noisy output feedback with the nonlinear representation capabilities of RNN architectures.
The scheme progresses in two stages.
In the ﬁrst stage, the information bits are sent raw (uncoded) over the AWGN channel.
In the second stage, the redundant bits are transmitted via a single-directional LSTM encoder which uses both the information bits and (delayed) output feedback from the earlier raw information bit transmissions.
The sequential nature of the input to the LSTM encoder ensures we respect the causality involved in using the feedback.
The decoder is a 2layered bi-directional GRU.
The architectures of encoder and decoder are shown in Figure 16,17 in Appendix D. Both the encoder and decoder are trained jointly using binary cross-entropy as the loss function over 10 7 examples.
In training, we set the forward SNR equal to be 0 dB and feedback SNR to be the test feedback SNR.
We randomly initialize weights of the encoder and the decoder.
We observed that random initialization of encoder-decoder was better than other initializations motivated by convolutional codes – codes that are designed for channels without feedback.
Results .
The empirical BER ﬁndings are reported in Figure 8 in comparison to several baseline schemes: (a) the performance of a state of the art rate 1/3 turbo code that does not use the feedback information; (b) the performance of (Schalkwijk & Kailath, 1966) implemented using (default)bit ﬂoating point arithmetic in Matlab and with noisy feedback; (c) recent codes for the feedback channel derived from (Chance & Love, 2011).
The main ﬁnding is that our RNN driven encoder and decoder scheme signiﬁcantly outperforms all the baselines.
Signiﬁcantly, we outperform even (Schalkwijk & Kailath, 1966) with noisless output feedback, which has been the gold standard in the literature for BER performance on the AWGN channel with output feedback.
C ONCLUSION
In this paper we have demonstrated that appropriately designed and trained RNN architectures can ‘learn’ the landmark algorithms of Viterbi and BCJR decoding based on the strong generalization capabilities we demonstrate.
This is similar in spirit to recent works on ‘program learning’ in the literature (Reed & De Freitas, 2015; Cai et al., 2017).
In those works, the learning is assisted signiﬁcantly by a low level program trace on an input; here we learn the Viterbi and BCJR algorithms only by end-to-end (input, output) training samples; we conjecture that this could be related to the strong “algebraic” nature of the Viterbi and BCJR algorithms.
We have also demonstrated that appropriate RNN architectures can ﬁnd new encoding schemes, as seen via the progress made on the challenging problem of communicating over the AWGN channel with noisy output feedback.
These encoding and decoding capabilities of the RNN architectures suggest that new codes could be found on the AWGN channel itself and improve the state of the art (constituted by turbo, LDPC and polar codes).
Also interesting is a new look at classical multi-terminal communication problems, including the relay and interference channels.
Both are active areas of present research.
A PPENDIX A N EURAL DECODER FOR OTHER CONVOLUTIONAL CODES
The rate-1/2 RSC code introduced in Section 2 is one example of many convolutional codes.
In this section, we show empirically that neural decoders can be trained to decode other types of convolutional codes as well as MAP decoder.
We consider the following two convolutional codes.
Unlike the rate-1/2 RSC code in Section 2, the convolutional code in Figure 9 ( a ) is not recursive, i.e., state does not have a feedback.
Also, it is non-systematic, i.e., the message bits can not be seen
[Figure]
immediately from the coded bits.
The convolutional code in Figure 9 ( b ) is another type of rate-1/
RSC code with a larger state dimension (dimension 3 instead of 2).
Figure 9 show the architecture of neural network we used for the convolutional codes in Figure 9.
For the code in Figure 9(a), we used the exact same architecture we used for the rate-1/2 RSC code in Section 2.
For the code in Figure 9(b), we used a larger network (LSTM instead of GRU andhidden units instead of 400).
This is due to the increased state dimension in the encoder.
[Figure]
(K, 2) bi
[Figure]
GRU (K, 400)
Batch Normalization (K, 400) bi-GRU (K, 400)
Batch Normalization (K, 400)
Dense (sigmoid) (K, 1) Layer Output
[Figure]
Input (K, 2) bi-
[Figure]
(K, 800)
[Figure]
(K, 800)
[Figure]
(K,
[Figure] [Figure]
(K, 800)
Dense (sigmoid) (K, 1)
Performance.
In Figures 11 , we show the BER and BLER of the trained neural decoder for convolutional code in Figure 9(a) under various SNRs and block lengths.
As we can see from these ﬁgures, neural decoder trained on one SNR (0dB) and short block length (100) can be generalized to decoding as good as MAP decoder under various SNRs and block lengths.
Similarly in Figure 12, we show the BER and BLER performances of trained neural decoder for convolutional code in Figure 9(b), which again shows the generalization capability of the trained neural decoder.
B N EURAL DECODER FOR TURBO CODES
Turbo codes, also called parallel concatenated convolutional codes, are popular in practice as they signiﬁcantly outperform RSC codes.
We provide a neural decoder for turbo codes using multiple layers of neural decoder we introduced for RSC codes.
An example of rate-1/3 turbo code is shown in Figure 13.
Two identical rate-1/2 RSC encoders are used, encoder 1 with original sequence b as input and encoder 2 with a randomly permuted version of b as input.
Interleaver performs the random permutation.
As the ﬁrst output sequence c 1 (1) of encoder 1 is identical to the output sequence c 1 (2) of encoder 2, and hence redundant.
So the sequence c 1 (2) is thrown away, and the rest of the sequences ( c 1 (1) , c 2 (1) , c 2 (2)) are transmitted; hence, rate is 1/3.
The sequences ( c 1 (1) , c 2 (1) , c 2 (2)) are transmitted over AWGN channel, and the noisy received sequences are ( y 1 (1) , y 2 (1) , y 2 (2)) .
Due to the interleaved structure of the encoder, MAP decoding is computationally intractable.
Instead, an iterative decoder known as turbo decoder is used in practice, which uses the RSC MAP decoder (BCJR algorithm) as a building block.
At ﬁrst iteration the standard BCJR estimates the posterior Pr( b k | y 1 (1) , y 2 (1)) with uniform prior on b k for all k ∈
[ K ] .
Next, BCJR estimates Pr( b k | π
( y 1 (1)) , y 2 (2)) with the interleaved sequence π ( y 1 (1)) , but now takes the output of the ﬁrst layer as a prior on b k ’s. This process is repeated, reﬁning the belief on what the codewords b k ’s are, until convergence and an estimation is made in the end for each bit.
Training.
We propose a neural decoder for turbo codes that we call N-Turbo in Figure 13.
Following the deep layered architecture of the turbo decoder, we stack layers of a variation of our N-RSC decoder, which we call N-BCJR.
However, end-to-end training (using examples of the input sequence y
( i ) ’s and the message sequence of b ( i ) ’s) of such a deep layers of recurrent architecture is challenging.
We propose ﬁrst training each layer separately, use these trained models as initializations, and train the deep layered neural decoder of N-Turbo starting from these initialized weights.
We ﬁrst explain our N-BCJR architecture, which is a new type of N-RSC that can take ﬂexible bit-wise prior distribution as input.
Previous N-RSC we proposed is customized for uniform prior distribution.
The architecture is similar to the one for N-RSC.
The main difference is input size ( instead of 2) and the type of RNN (LSTM instead of GRU).
To generate N training examples of { ( noisycodeword, prior ) , posterior } , we generate N/ 12 examples of turbo codes.
Then we ran turbo decoder for 12 component decoding - and collect input output pairs from the 12 intermediate steps of Turbo decoder, implemented in python Taranalli (2015) shown in Figure 14.
We train with codes with blocklength 100 at ﬁxed SNR -1dB.
We use mean squared error in (2) as a cost function.
To generate training examples with non-zero priors, i.e. example of a triplet (prior probabilities { Pr( b k ) }
Kk =1 , a received sequence y , and posterior probabilities of the message bits { Pr( b k | y ) }
Kk =1 ), we use intermediate layers of a turbo decoder.
We run turbo decoder, and in each of the intermediate layers, we take as an example the triplet: the input prior probability, the input sequence, and the output of the BJCR layer.
We ﬁx training SNR to be -1dB.
We stack 6 layers of
BCJR decoder with interleavers in between.
The last layer of our neural decoder is trained slightly differently to output the estimated message bit and not the posterior probability.
Accordingly, we use binary crossentropy loss of as a cost function.
We train each N-BCJR layer with 2,000 examples of length 100 turbo encoder, and in the end-to-end training of N-Turbo, we train with 1,000 examples of length 1,000 turbo encoder.
We train with 10 epochs and ADAM optimizer with learning rate 0.001.
For the end-to-end training, we again use a ﬁxed SNR of noise (-1dB), and test on various
SNRs.
The choice of training SNR is discussed in detail in the Appendix E.
Performance.
As can be seen in Figure 15, the proposed N-Turbo meets the performance of turbo decoder for block length 100, and in some cases, for test SNR = 2 , it achieves a higher accuracy.
[Figure]
Similar to N-RSC, N-Turbo generalizes to unseen codewords, as we only show 3 , 000 examples in total.
It also seamlessly generalizes in the test SNR, as training SNR is ﬁxed at − 1 dB.
C O THER NEURAL NETWORK ARCHITECTURES FOR N-RSC AND N-BCJR
In this section, we show the performances of neural networks of various recurrent network architectures in decoding rate-1/2 RSC code and in learning BCJR algorithm with non-zero priors.
Tableshows the BER of various types of recurrent neural networks trained under the same condition as in N-RSC (120000 example, code length 100).
We can see that BERs of the 1-layered RNN and single-directional RNN are order-wise worse than the one of 2-layered GRU (N-RSC), and two layers is sufﬁcient.
Table 2 shows the performance of neural networks of various recurrent network architectures in BCJR training.
Again, we can see that 2-layers are needed and single directional
RNN does not work as well as bi-directional RNNs.
D A RCHITECTURE OF RATE -1/3 NEURAL ENCODER AND DECODER FOR NOISY FEEDBACK CHANNEL
In this section, we show the architectures for rate-1/3 recurrent neural encoder and decoder for channels with noisy feedback in Figure 16 and 17.
The number of message bits is denoted by K .
The encode maps K message bits to a codeword of length 3( K + 1) .
Note that unlike in RSC or Turbo codes, the codewords are not restricted to be binary.
[Figure]
The encoding process has two phases.
In the ﬁrst phase, the encoder simply transmits the raw message bits padded by 0. That is, the encoder maps b i to c i = 2 b i − 1 for i = 1 , · · · , K + 1 , where b K +1 = 0 , and collects the feedback ˜ y 1 , · · · , ˜ y K ,
˜ y K +1 . In the second phase, the encoder generates a coded sequence of length 2( K +1) through a single directional LSTM.
Each k -th LSTM cell generates two coded bits c k 1 , c k 2 .
The input to the k -th LSTM cell is of size 4: b k , ˜
y k − c k (the noise added to k -th message bit in phase 1) and the most recent two noise feedback from phase 2: ˜
y ( k − 1)1 − c ( k − 1)1 and ˜ y ( k − 1)2 − c ( k − 1)2 which are available fed back by the receiver (Notation: ˜ y ( ij ) denotes the noisy version of c ij ).
This architecture naturally assures the causality of using feedback in the encoder.
For the decoder, we use a 2 layered bidirectional GRU.
Layer
Output dimension Input (K+1, 4) LSTM (K+1, 100) Dense (sigmoid) (K+1, 2)
E G UIDELINES FOR CHOOSING THE TRAINING SNR FOR NEURAL DECODERS
As it is natural to sample the training data and test data from the same distribution, one might use the same noise level for testing and training.
However, this matched SNR is not reliable as shown in Figure 3.
We give an analysis that predicts the appropriate choice of training SNR that might be
[Table] [Table]
different from testing SNR, and justify our choice via comparisons over various pairs of training and testing SNRs.
We conjecture that the optimal training SNR that gives best BER for a target testing SNR depends on the coding rate.
A coding rate is deﬁned as the ratio between the length of the message bit sequence
K and the length of the transmitted codeword sequence c .
The example we use in this paper is a rate r = 1 / 2 code with length of c equal to 2 K .
For a rate r code, we propose using training SNR according to
SNR train =
min { SNR test , 10 log 10 (2 2 r − 1) } , (3) and call the knee of this curve f ( r ) = 10 log 10 (2 2 r − 1) a threshold.
In particular, this gives
SNR train =
min { SNR test , 0 } for rate 1 / 2 codes.
In Figure 18 left, we train our neural decoder for
RSC encoders of varying rates of r ∈ { 1 / 2 , 1 / 3 , 1 / 4 , 1 / 5 , 1 / 6 , 1 / 7 } whose corresponding f ( r ) =
Layer
Output dimension
Input (K+1, 3) bi-GRU (K+1, 400)
Batch Normalization (K+1, 400) bi-GRU (K+1, 400)
Batch Normalization (K+1, 400)
Fully connected (sigmoid) (K+1, 1)
We derive the formula in (3) in two parts.
When the test SNR is below the threshold, then we are targeting for bit error rate (and similarly the block error rate) of around 10 − 1 ∼ 10 − 2 .
This implies that signiﬁcant portion of the testing examples lie near the decision boundary of this problem.
Hence, it makes sense to show matching training examples, as signiﬁcant portion of the training examples will also be at the boundary, which is what we want in order to maximize the use of the samples.
On the other hand, when we are above the threshold, our target bit-error-rate can be signiﬁcantly smaller, say 10 − 6 .
In this case, most of the testing examples are easy , and only a very small proportion of the testing examples lie at the decision boundary.
Hence, if we match training SNR, most of the examples will be wasted.
Hence, we need to show those examples at the decision boundary, and we propose that the training examples from SNR 10 log 10 (2 2 r − 1) should lie near the boundary.
This is a crude estimate, but effective, and can be computed using the capacity achieving random codes for
AWGN channels and the distances between the codes words at capacity.
Capacity is a fundamental limit on what rate can be used at a given test SNR to achieve small error.
In other words, for a given test SNR over AWGN channel, Gaussian capacity gives how closely we can pack the codewords (the classes in our classiﬁcation problem) so that they are as densely packed as possible.
This gives us a sense of how decision boundaries (as measured by the test SNR) depend on the rate.
It is given by the Gaussian capacity rate = 1 / 2 log(1 + SNR ) .
Translating this into our setting, we set the desired threshold that we seek.
F S UPPLEMENTARY FIGURES
[Figure] SECTION
[Figure]