Paper: Understanding Addition in Transformers (ICLR-2024)

Reviewer comment (Theory):
In the integer addition task, digit 0 should be treated as a special case, since intuitively, when humans perform integer calculations, the more zeros there are, the easier the calculation becomes. In other words, the digit 0 requires special attention. So it might be interesting to include Make Sum 0 and Use Sum 0 in the mathematical framework.

Author response:
__Weakness: In the integer addition task, digit 0 should be treated as a special case, since intuitively, when humans perform integer calculations, the more zeros there are, the easier the calculation becomes. In other words, the digit 0 requires special attention. So it might be interesting to include Make Sum 0 and Use Sum 0 in the mathematical framework.__

The Make Sum 0 ideas was indeed considered by the authors. Make Sum 0 is a useful efficiency gain for humans doing a specific sub-class of addition questions. Our intuition is that the model is calculating bigrams of tokens without any real understanding of numbers - it treats the digits as string tokens "3" + "4" = "7". The model can efficiently map all 100 pairings without needing to or benefiting from treating "0" as a special case. Our intuition is that if the model treated "0" as a special case, this would increase the algorithm complexity without increasing its accuracy and so the model would not benefit from adding this rule.

Reviewer comment (Evaluation):
What's the performance of the trained model on the test data?

Author response:
__Q1. What's the performance of the trained model on the test data?__

The overall response and the new Appendix 17 provide this detail.

Reviewer comment (Experiments):
The adaptability of the methodology in this paper is limited, as it only applies to a one-layer Transformer model. Perhaps further analysis on two-layers or even more complex models would be beneficial. Moreover, the study solely focuses on integer addition, making it challenging to extend to other operations like subtraction or multiplication.

Author response:
__Weakness: The adaptability of the methodology in this paper is limited, as it only applies to a one-layer Transformer model. Perhaps further analysis on two-layers or even more complex models would be beneficial. Moreover, the study solely focuses on integer addition, making it challenging to extend to other operations like subtraction or multiplication.__

Feedback includes that analysis should be extended to two-layer or more complex models. We agree with this direction but see this work as outside the scope of this paper. When this paper was written we did not understand the more-complex 2-layer addition algorithm well. (Our analysis of this model is now well advanced. The 2-layer model utilizes the 1-layer algorithm but also includes an additional algorithm covering the “cascading US9” high-loss edge case. This will be the subject of a future paper.). We see this paper is a solid first step on a long journey explaining ever more complex algorithms (e.g. multiplication) partially by re-using well-understood foundational components (e.g. addition).

The concern is raised that the paper’s methodology is not reusable and the applicability of the paper is limited. We respectfully disagree. Table 5 in the new Appendix 17 “The loss function and loss measures” shows one instance of this approach, showing the output from an automated test framework. 

The concern is raised that the methodology only applies to one-layer transformer model. While, it is very likely that 1-layer addition, 2-layer addition, subtraction, multiplication implement substantially different algorithms, this paper’s methodology provides a way to gain many insights into any algorithm, before the algorithm’s implementation detail is understood. While these insights do not explain the algorithm’s implementation directly, from experience they prove to be very useful in constraining the feasible algorithm space, leading to further researcher insights.

Reviewer comment (Writing):
The writing of this paper is not comprehensive. For instance, the descriptions for Figure 4 and 8 are difficult to comprehend.

Author response:
__Weakness: The writing of this paper is not comprehensive. For instance, the descriptions for Figure 4 and 8 are difficult to comprehend.__

The concern was raised that the paper was not comprehensive. In this paper we:
- Provide an alternative mathematical framework for addition suited to neural networks
- Show not only that the model can do addition and is low loss, but explains the edge-case causing the loss in terms of the mathematical framework
- Propose an explanation for the model’s implementation of the addition algorithm in terms of the mathematical framework
- Validate the proposed algorithm with experimental results. Details of this validation are in the new “Loss function and loss measures” appendix.

A concern was raised that Figure 4 is difficult to understand. The addition algorithm that the model learnt is more complex than the traditional human process for addition. Understanding the model’s approach requires us to ignore our existing process, disassemble the addition process into its foundational components and then reassemble an alternative process. Figure 4, supported in the paper, documents this alternative approach. It is non-trivial to understand but led to insights key to understanding the model’s algorithm. We believe Figure 4 is useful and instructive.

A concern was raised that Figure 8 is difficult to understand. The addition algorithm that the model learnt is complex - more complex than the traditional human approach to addition. Some complexity in description is unavoidable. Figure 1 introduces the algorithm concepts and shows a simplified representation of the algorithm. Figure 8 fleshes out the algorithm’s detail. The two figures work together, supported in the paper, to build out the concepts. We believe these figures are very good representations of the algorithm, and that further simplification in the diagram would require omitting pertinent information. To aid in comprehension, a new Appendix 16 “Model Algorithm as Pseudocode” reproduces the algorithm in Figure 8 as pseudocode.

Reviewer comment (Experiments):
The experiments conducted are not exhaustive. In the "Prediction Analysis" section, the authors failed to provide a specific metric and results compared to baselines.

Author response:
__Weakness: The experiments conducted are not exhaustive. In the "Prediction Analysis" section, the authors failed to provide a specific metric and results compared to baselines__

The overall rebuttal and the new Appendix 17 “Loss function and loss measures” provide this detail.

Reviewer comment (Theory):
The analysis framework and analysis lacks mathematical rigidity.

Author response:
__Weakness: The analysis framework and analysis lacks mathematical rigidity.__

A concern was raised that the conclusion is not well established based on rigorous mathematical framework. Please refer to the overall rebuttal and the new Appendix 17 that provides detail on the experimental results that underpin the paper’s claims.

Reviewer comment (Theory):
The conclusion is not well established based on rigorous mathematical framework.

Author response:
__Weakness: The conclusion is not well established based on rigorous mathematical framework__

A concern was raised that the analysis framework and analysis lacks mathematical rigidity. Please refer to the overall rebuttal and the new Appendix 17.

Reviewer comment (Experiments):
The paper does not fully utilize/choose the most relevant aspect of transformers for addition tasks.

Author response:
__Weakness: The paper does not fully utilize/choose the most relevant aspect of transformers for addition tasks.__

A concern is raised that the paper does not fully utilize/choose the most relevant aspect of transformers for addition tasks. We’re not clear on what this refers to. Assuming this refers to using a 1-layer rather than a 2-layer model, when this paper was written we did not understand the more-complex 2-layer addition algorithm well. Our analysis of the 2-layer addition model is now well advanced, and leverages some of the methodologies and findings of this paper. We see this paper is a solid first step on a long journey explaining ever more complex algorithms and transformers.

see below Response to reviewer wEos's review - Questions [2/2] for responses to questions.

Reviewer comment (Writing):
Page 3, section 3, paragraph 4, "techniques in works like symbolically … ", a grammar error?

Author response:
__Q1, Q2, Q3, Q4. Various small issues__

Thank you for pointing out these issues. They have been resolved in the paper.

Reviewer comment (Writing):
Page 3, section 3, paragraph 7, "Surveys like  overview techniques…", missing citations after "Surveys like"?

Author response:
__Q1, Q2, Q3, Q4. Various small issues__

Thank you for pointing out these issues. They have been resolved in the paper.

Reviewer comment (Theory):
Page 4, section 4, paragraph 4, "Transformer models always process text from left to right…", this is not true. It is just an artifact of GPT-style attention masking. For example, we can do config the attention mask to enable full order attention over the two addends and generate the outputs in all kinds of order, e.g. from the tens digit to higher value digits, from the middle digit to two ends, and so on. We can also do non-autogression generation, e.g. incremental masking output generation.

Author response:
__Q6. Page 4, section 4, paragraph 4, "Transformer models always process text from left to right…"__

A concern was raised about the statement “Transformer models always process text from left to right”. This has been fixed in the paper to read: This autoregressive transformer model processes text from left to right.

Reviewer comment (Presentation):
Page 4, figure 3 caption "..After the question is fully revealed (at layer 11)..", by "layer 11" do you mean the 11th row? To avoid ambiguity, it is better to number the attention matrix and refer to them the row or column number across the paper. Also what are the sub-figures of 0.0, 0.1, 0.2? Different heads? What are the labels?

Author response:
__Q7. Page 4, figure 3 caption "..After the question is fully revealed (at layer 11)..", by "layer 11" do you mean the 11th row?__

A concern was raised about the inconsistent use of the term “layer” in explanations. We have fixed the paper to consistently use the terms: “layer” only for number of layers in the Transformer and “row” for each (vertical) step in the attention pattern.

The meaning of the labels in the attention pattern figure was queried. The Figure image has been improved in the paper to clarify the meaning of the image labels as Layer and Head.

Reviewer comment (Theory):
Page 4-5, section 5, please clarify whether the  "mathematical framework" is  for characterizing (grouping) addition data instances-digits only? Or is there a link to the loss? If so, please formulate the framework and what kind of mathematical hypotheses this framework can verify formally in mathematical terms? Also please detail the loss on each digits formally. Also please detail the statistics of your training and valid datasets in terms of your classification of digits in your framework.

Author response:
__Q8. Page 4-5, section 5, please clarify whether the "mathematical framework" is for characterizing (grouping) addition data instances-digits only? Or is there a link to the loss?__

The concern about the independence (or otherwise) of the mathematical framework and the loss calculations is covered by the “Weaknesses” section above and the new Appendix 17. The “Mathematical Framework” section now concludes with: *We use this mathematical framework solely for analysis to gain insights. The model training and all loss calculations are completely independent of this mathematical framework.*

Reviewer comment (Reproducibility):
Page 4-6, please detail how the loss is being average. Are they per digits or per digit average?

Author response:
__Q9. Page 4-6, please detail how the loss is being average. Are they per digits or per digit average?__

There is a loss function used with per-digit graphs, and a separate "mean" loss function used with all-digit graphs. Both loss functions are defined in the overall feedback and in the new Appendix 17.

Reviewer comment (Writing):
Page 6, please introduce or define or describe phase 1, 2, 3 formally?

Author response:
__Q10. Page 6, please introduce or define or describe phase 1, 2, 3 formally?__

The suggestion is made to introduce or define the phases 1, 2 & 3 formally. We are hesitant to make strong claims about these phases at this time. Experimentally, and as per the literature, they seem to correspond to “memorization”, “algorithm refinement” and “clean-up” phases. The “Training Analysis” figure description now includes: The 3 phases seem to correspond to “memorization”, “algorithm discovery” and “clean-up”.

Reviewer comment (Theory):
Page 7, section 7, "During model prediction we overrode … the model memory (residual stream)…", please detail the approach formally? Are your conclusions/assertions based on checking the attention scores? Please discuss explicitly with formal treatment. Otherwise, the plain English language analysis in Section 7 is difficult to follow and justify. Also formally define "independent of every other digit", "most impact on loss" and define it based on measure statistics during model inference time.

Author response:
__Q11.Page 7, section 7, "During model prediction we overrode … the model memory (residual stream)…", please detail the approach formally__

A request for a formal treatment of the conclusion/assertions including how the model prediction was overridden was requested is covered in Appendix 17.

A request was made for a formal definition of the independent calculation of each digit. The new Appendix 16 “Model Algorithm as Pseudocode” shows how the algorithm calculates each digit independently.