Paper: A Foundation Model for Error Correction Codes (ICLR-2024)

Reviewer comment (Theory):
Interpretability: FECCT should have been learned some interesting algorithm, that can be interpreted as an advanced version of BP. We do see some part of interpretations in the appendix, but not solid enough to get insight on what FECCT's algorithm means.

Author response:
## Interpretability
We provide in the revision’s Appendix G the depiction of typical learned self-attention matrices.  As can be observed, initially the attention is mostly focused on the syndrome values in order to perform the decoding, while in the final layer, the focus is also transferred to the information bits.

Reviewer comment (Evaluation):
Complexity of network: attention-based neural network are very complex. Deploying the FEECT to any real world production requires some hard work on complexity reduction. In its current form, I am not seeing FECCT can be deployed to modem in short time. Note that channel coding are heavily used in all modern wireless communication systems, which requires minimal latency, high throughput, and low cost.

Author response:
## Integration
While being out of the scope of the paper, the deployment of the FECCT is indeed a challenge, requiring the application of many deep-learning acceleration techniques such as quantization (binarization) and sparsification (mainly of the self-attention module) as well as its FPGA/ASIC integration.

Reviewer comment (Novelty):
Although the design of foundational decoders is a very interesting intellectual exercise, the real-world impact of it is close to none if not zero. The reason is that error correcting codes are designed and deployed once and their training is not a big deal even if someone takes deep neural network decoders as opposed to classical BP methods. More importantly, capacity-achieving codes exist for long codes and their BP decoders are close to ML performance, i.e., optimal decoders, so there is no gain of these deep neural decoders in long codes. The focus should be short codes, for which we do not have good BP decoders. However, for short codes, we really do not need foundational decoders as one can design and easily train specialized neural decoders for the short codes that will very likely beat the performance of generalist decoders for all lengths. It is clear to believe that a generalist decoder will not be able to outperform a specialized deep neural decoder for short lengths, unless the authors can show their generalist decoder can beat the performance of state-of-the-art short length (less than 200 bits) code decoders, specialized for that specific code length.

Author response:
>“The reason is that error correcting codes are designed and deployed once and their training is not a big deal even if someone takes deep neural network decoders as opposed to classical BP methods”

While we agree with the reviewer that training is not a major limitation compared to deployment, we respectfully disagree with the reviewer regarding the integration/deployment side. Current error-correction systems (e.g., base stations) can/need to decode codes of different lengths and/or different families. For example, LDPC codes of different lengths are decoded on HW by omitting some of the integrated processing engines of the BP implementation for shorter codewords.

Moreover, deploying one single neural decoder for several types of codes and/or lengths is fundamentally different from integrating several neural decoders, especially in terms of die space and storage/memory (one model’s weights vs many models’ weights).

We remind here that existing neural decoders are designed and trained on one single code to be integrated into dedicated HW.

This point is emphasized in the revision.

>“However, for short codes, we really do not need foundational decoders as one can design and easily train specialized neural decoders for the short codes that will very likely beat the performance of generalist decoders for all lengths.”

Following our previous answer, the deployment of a specialized neural decoder has major integration drawbacks.

 Most importantly, the main goal of our work was to show our method, while being code/length-invariant (referred to as “generalist decoder” by the reviewer), can match and even outperform the “specialized” state-of-the-art neural decoder.

In addition to the points above that address the direct utility, foundation models support downstream tasks. Specifically, the ability to move quickly between codes enables the sampling and evaluation of random codes. Moreover, since our method defines a universal differentiable neural decoder, it opens the door to the optimization and definition of new families of codes.

These points are emphasized in the revision.

Reviewer comment (Evaluation):
What is the performance reported in the tables? I see that 3 different values of Eb/N0 are used as the channel input signal (bit) power to noise ratio, but what are the reported numbers as performance? I would like to see how these reported numbers translate to the error rate performance, as it is the only thing that matters in communication. It does not look like the authors picked a particular error rate and reported the corresponding Required Eb/N0 to achieve such an error rate, because in that case the lower number would be associated with the better scheme, not the higher (as stated in the paper).

Author response:
>“What is the performance in the tables?What about reported numbers as performance. What are they? I like to see how these reported numbers translate to the error rate performance”

As described in Section 5, third paragraph, the reported numbers are the “negative natural logarithm bit error rates (BER) for three different normalized SNR values (Eb/N0)”, meaning we provide $-\ln(BER)$ (such that higher is better) for three Eb/N0 values. This dense presentation for multiple codes of the BER on the SNR of most interest has been borrowed from many of the previous neural decoding works.

Reviewer comment (Experiments):
The authors need to compare their scheme with Choukroun & Wolf (2022b), which is shown to be superior to ECCT.

Author response:
>“it would be helpful to compare your proposed work with that of Choukroun & Wolf (2022b) which extends and enhances ECCT via Denoising diffusion in error correction codes and have far superior performance than ECCT.”

We wish to remind the main difference between ECCT and DDECC, as described in Section 6 paragraph 2. While ECCT focuses on the design of a new neural *decoding architecture*, DDECC is a generic iterative *decoding algorithm* applied to a given backbone architecture. As described in [2] Section 4.5, the ECCT has been chosen since it is the state-of-the-art decoding architecture.

Our model focuses on the first family of work (i.e., decoding architecture) that can, same as ECCT, be coupled with DDECC and makes it code and length invariant too. While these results would be interesting to see, they require many more experiments and much heavier resources.

This point is emphasized in the revision.

Reviewer comment (Experiments):
Please compare the proposed generalist decoder at short lengths (less than 200 bits) with that of specialized decoders at those lengths, because as pointed out, this is where these foundational decoders would show value if any.

Author response:
> “please compare the proposed generalist decoder at short lengths (less than 200 bits) with that of specialized decoders at those lengths. Because as I pointed out this is where these foundational decoders would show value if any.”

While our work focuses on the development of neural decoders, the previous work cited by the reviewer (e.g., [1],[2])  already provided a comprehensive comparison with specialized decoders. In our work, we compare with it. 

The revised text now points to these results.

Reviewer comment (Experiments):
It would be helpful to compare the proposed work with that of Choukroun & Wolf (2022b), which extends and enhances ECCT via Denoising Diffusion in error correction codes and has far superior performance than ECCT.

Author response:
>“it would be helpful to compare your proposed work with that of Choukroun & Wolf (2022b) which extends and enhances ECCT via Denoising diffusion in error correction codes and have far superior performance than ECCT.”

We wish to remind the main difference between ECCT and DDECC, as described in Section 6 paragraph 2. While ECCT focuses on the design of a new neural *decoding architecture*, DDECC is a generic iterative *decoding algorithm* applied to a given backbone architecture. As described in [2] Section 4.5, the ECCT has been chosen since it is the state-of-the-art decoding architecture.

Our model focuses on the first family of work (i.e., decoding architecture) that can, same as ECCT, be coupled with DDECC and makes it code and length invariant too. While these results would be interesting to see, they require many more experiments and much heavier resources.

This point is emphasized in the revision.

Reviewer comment (Writing):
Conclusion should be rewritten based on the results presented mentioning the future scope.

Author response:
We thank the reviewer for the very supportive review.

The conclusion is rewritten and includes the future scopes of research as follows.

*“Next steps should focus on the deployment of the method on existing error correction embedded systems via the utilization of Transformer acceleration methods along with extensive tuning and training of the codes and channels of interest. Additionally, the definition of a universal differentiable neural decoder may open the door to the optimization of codes and even to the learning of new families of codes.”*

Reviewer comment (Writing):
Conclusion should be rewritten based on the results presented mentioning the future scope.

Author response:
We thank the reviewer for the very supportive review.

The conclusion is rewritten and includes the future scopes of research as follows.

*“Next steps should focus on the deployment of the method on existing error correction embedded systems via the utilization of Transformer acceleration methods along with extensive tuning and training of the codes and channels of interest. Additionally, the definition of a universal differentiable neural decoder may open the door to the optimization of codes and even to the learning of new families of codes.”*