From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper introduces a new toy task, called LEGO, to investigate how transformer networks perform iterative reasoning. Using this task, the authors then perform an empirical analysis of different architecture variants of transformers, with and without pre-training, and show that different roles are played by different attention heads.

There is a wide range of scores for this paper, and I thus had a thorough discussion with the reviewers. On the positive side, the reviewers enjoyed the direction of this paper, and the analysis provided on the transformer models regarding their abilities to perform reasoning, to generalise to longer sequence than seen during training, and the role of pre-training. On the other hand, some reviewers raised concerns regarding the discussion of previous works that also study how transformers works. More precisely, one of the main concerns around what one could learn by using this new toy task, compared to existing work. I tend to agree with this concern: while the analysis of this paper is well made, I also feel that I did not learn anything new by reading the paper, since most of the observations were already made in other papers. I also believe that the proposed task is a bit naive (eg compared to previous considered ones), and I am not sure what it can add to the field. Moreover, the extensive literature on the subject is not well discussed. For this reason, I recommend to reject this paper.
justification_for_why_not_higher_score: I agree with the concerns raised by reviewers, about the limited impact of the proposed toy task. I think that the paper does not do a good job at situating its contributions compared to existing work, and in particular, what was, or could be, learned by using the proposed LEGO task about reasoning abilities of transformer networks. Moreover, I think that this toy task is not well motivated compared to other toy tasks that were previously considered in the field of ML and NLP.
justification_for_why_not_lower_score: n/a

From: ICLR.cc/2023/Conference/Paper5434/Authors
title: Response
comment: We thank the reviewer for the detailed comments and the encouraging evaluations. We have added a discussion of reference [1] in the revision.

To answer the question over "shortcut":

In fact, we have a detailed discussion about it in Appendix A for why it is an undesirable phenomenon. In particular, we have controlled experiments regarding the harmfulness of shortcuts (Figure 10). The results suggest that models that have learned shortcuts perform significantly worse than the models that do not learn shortcuts, in terms of length generalization.

From: ICLR.cc/2023/Conference/Paper5434/Authors
title: Response
comment: We thank the reviewer for the comments.

To answer the question 'how well would RNNs work on LEGO':

We believe LEGO will be much more challenging for RNNs than for Transformers (to an extent that it almost becomes an unfair comparison).

Solving our LEGO task requires the model to implement a certain long-range operation (i.e. association) since the clauses are shuffled in every LEGO sequence. This operation is perceivably challenging for RNNs due to their well-known limitations of short context length [1]. On the other hand, long-range dependency is indeed the strength of Transformer models thanks to the attention modules.



[1] Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context. Khandelwal, et al, 2018

From: ICLR.cc/2023/Conference/Paper5434/Authors
title: Response
comment:  

We thank the reviewer for the detailed comments. We have added a discussion of the ‘universal transformer’ in the revision. 

 

We completely agree with the reviewer that our proposed task LEGO does not capture certain aspects of general NLP, and here are certainly more aspects of Transformers’ power beyond logical reasoning. 

In this work, we have mostly focused on analyzing Transformers’ capabilities to perform logical reasoning,  

However, we believe our methodology of gathering insights by constructing controllable synthetic benchmarks to reflect certain important aspects of complex real-world tasks is generalizable to many other areas too. 

 
Surprisingly, the seemingly simplistic LEGO task has led to various insights that are transferable to more general NLP tasks. Among them, one significant discovery is the association head from pretrained models. To the best of our knowledge, it is the first global, sparse, input-dependent attention pattern that the community has ever identified, and we demonstrate its importance in pretraining and SQuAD. We agree with the reviewer that it is an exciting future direction to explore the limit of hardcoding this head and also identify other structured attention patterns that captures more aspects of general NLP. We view our current work as a solid step in this direction.

From: ICLR.cc/2023/Conference/Paper5434/Authors
title: Our global association pattern is NOVEL
comment: We thank the reviewer for the comments and for expertly pointing us to useful related works. 

We’d like to kindly point out hat the reviewer might have inadvertently overlooked our finding of the **association pattern**. 

### 1) Our finding of association head is significantly novel:  

To our best knowledge, we are the first to discover this pattern’s existence in pretrained models and to demonstrate its benefits. Specifically, such a pattern is NOT mentioned in any of the papers the reviewer listed above. 

**Our association pattern is sparse and global as it attends to different occurrences of identical tokens**. Moreover, its pattern is NOT fixed and indeed depends on the input. Thus, it fundamentally differs from any fixed or local patterns from previous work.  

As we show in the paper, the association head is vital for both LEGO and real NLP tasks such as pretraining and SQuAD. 


### 2) LEGO led us to the finding of such a global association pattern.   

In Section 4.2.1, we argued that association operators would be useful to solve LEGO, and this led us to examine pretrained models and discovered the existence of association heads. A priori, it might not be as easy to observe such a pattern on natural language input where repeated keywords may be rare. 

### 3) Fixed Local attention patterns alone DO NOT solve LEGO.

To further stress our association pattern's importance, during the rebuttal, we followed the reviewers’ pointers and tried out on LEGO the Hardcoded Gaussian Attention [5] as well as other fixed local attention patterns.

In these experiments, we instantiate the latter by replacing BERT’s attention layers with only a linear value map and 1D temporal convolutions with various convolutional kernel sizes (e.g., 5, 11, 21, 43 where input length is 62.) and a single filter is shared across all channels (to simulate attention patterns).  

On the same LEGO setting (12 variables, train on the first 6), none of the Gaussian Attention models can fit the second variable along the chain (its training accuracy remains at 50%). While the other local attention models fit all 6 variables in training, their test accuracies on the third variable and beyond stay at 50%, indicating overfitting. None of these models exhibits any length extrapolation.   

On the contrary, our model with both local attention and global association generalizes well and performs non-trivial length extrapolation. 


## To sum up 

Our finding of association pattern is significantly novel (non-existent in previous literature).

We acknowledge the reviewer’s expertise in analyzing attention heads. We believe the reviewer sees the importance of input-dependent, global, sparse attention patterns (beyond the local ones) for general NLP. We view our finding as an important step towards understanding them.

We hope the above helps address the concern regarding the novelty, and we hope it may potentially update your overall evaluation of our work. We are looking forward to further responses/exchanges.

From: ICLR.cc/2023/Conference/Paper5434/Reviewer_drii
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a synthetic task that aims to capture aspects of multi-hop reasoning, and uses it to characterize the behavior of Transformer models. It finds that pre-trained transformers learn the task more rapidly, and attributes this to specific patterns of attention in the network. Introducing mechanisms that emulate these specific attention patterns helps close this gap in performance without the need for expensive pre-training. Models struggle with extrapolating to longer chains of reasoning than required during training. This deterioration can be mitigated by repeating layers with the same parameters as well as training with stochastic depth. 
strength_and_weaknesses: Strengths:
- The proposed task is useful as a tool for understanding some ways in which a Transformer can perform multi-hop reasoning
- The proposed operationalization of requiring length extrapolation is effective at disentangling issues with positional encoding from the question of whether the model can extrapolate to longer chains of reasoning 
- Experiments help provide an understanding for how pre-training, as well as stochastic depth, help improve certain aspects of the models' capabilities
- Hardcoded attention patterns are shown to be just as effective for solving this specific task
- The writing of the paper is clear and well-structured

Weaknesses:
- One of the strengths of the attention mechanism is that it can learn any attention pattern a task may require, and the multi-head version can learn when (at what layer) and how much to use each pattern. The hardcoded attention patterns seem sufficient for the proposed synthetic task -- almost by construction of the LEGO task. There are some preliminary findings that they may also be effective for masked language modeling and at least SQuAD, but this direction is underexplored and perhaps not all of the necessary components for real NLP tasks are captured.
- The proposed task is perhaps not the best test-bed of *pre-training* specifically, because the approach used is to pre-train on natural language but then test on synthetic expressions. It would not be surprising for language-specific reasoning to not undergo transfer in this case, potentially leaving only the most general of structural patterns, like the attention patterns revealed in this work.
- Overall, it's not clear how much of the Transformers' capabilities are stressed by the proposed synthetic task. It could be that understanding multi-hop computation in the style of the LEGO task is the key for explaining current behaviors and proposing improvements. Alternatively, multi-hop reasoning may just not be a crucial component of many NLP tasks as currently formulated, or LEGO might not be capturing the actual character of chain-based reasoning as required for real-world task.
clarity,_quality,_novelty_and_reproducibility: See above

Minor:
- Regarding Transformers that repeat layers: in addition to ALBERT, it would be helpful to cite other instances such as Universal Transformers (Dehghani et al)
summary_of_the_review: My overall recommendation is based on weighing the thorough and well-written exposition/investigation of the LEGO task with the potential downsides of any synthetic benchmark, namely that it may or may not capture the key elements of actual real-world tasks, and can therefore be limited in its impact.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5434/Reviewer_s6py
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: LEGO is a synthetic reasoning task that targets chain reasoning. Think Markov chain, where the current state only depends on the previous state. They specifically evaluate BERT and ALBERT. ALBERT is particularly interesting since weights are shared across all layers in ALBERT and is hypothesized that it may be particularly suited for this type of reasoning. Indeed, it outperforms BERT. Various ablations are carried out to understand why.
strength_and_weaknesses: The task and setup are framed well. Ablations such as stochastic depth length extrapolation, pretraining, and probing classifier are creative ways at understanding the inductive biases of the networks.

The task seems not that specific to transformers. I do wonder how other RNNs would fare.
clarity,_quality,_novelty_and_reproducibility: I found the paper clear. I appreciated the language of abstract algebra, but I am not sure if all readers will. I thought the figures were good. The experiments are well done and should be simple to reproduce. There is not much originality in the modeling, but the task and analysis are novel
summary_of_the_review: I think the paper adds to the understanding of the various inductive biases that different transformer architectures have. There is much interest int the mathematical ability of transformers. I would like to see more models benchmarked on this task.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5434/Reviewer_3VMG
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a synthetic task, LEGO, and uses it to understand how Transformers operate. The task is in following a chain of simple reasoning, in particular for simple mathematical operations with variables valued either +1 or -1 and expressions of the kind a=+b or a=-b. With this task, the authors consider BERT’s and ALBERT’s abilities to generalize to longer inputs. They find specific attention heads (e.g., local attention patterns) and propose modification with some of the heads fixed to these patterns.
strength_and_weaknesses: While the paper introduces the new task to help understanding Transformers, the task does not lead to new observations not widely known before. Specifically, it’s well-known that the most important Transformer attention heads play interpretable roles, including positional ([1], [2], [3], etc). This was used extensively to improve/simplify models: [4] fixes attention window size for some of the heads and gets improvement for low-resource settings (similar to this work), [5] develops hard-coded attention without learned parameters, to name a few. From this perspective, (1) neither finding these attention patterns nor proposing fixed attention is novel; (2) doing it on a synthetic task when this has already been done before for different real-world tasks does not make much sense.


[1] ACL 2019: Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.

[2] EMNLP 2019: ADaptively Sparse Transformers.

[3] BlackBoxNLP 2019: What does BERT Look at? An Analysis of BERT’s Attention

[4] EMNLP findings 2020: Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

[5] ACL 2020: Hard-Coded Gaussian Attention for Neural Machine Translation

clarity,_quality,_novelty_and_reproducibility: See comments above.
summary_of_the_review: The paper introduces a synthetic task to analyse Transformers. In the analysis, finds specific attention patterns and proposes model modification with some of the heads being fixed. Most of the observations done in the paper and model modification have already been done before without resorting to synthetic tasks.
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5434/Reviewer_7SQ8
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper provides empirical investigation of transformer by constructing a synthetic reasoning task -- LEGO (Learning Equality and Group Operations). The LEGO tasks consist of group operations and a chain of reasoning. The major purpose of the synthetic experiment is to provide a controlled setting and unveil what transformer learns. To do so, the paper conduct a series of experiments on the both classic setting (training and testing distribution are the same) and the OOD setting (out of distribution) and they found

(1) Classical generalization happens all the architecture.

(2) OOD generalization depends on the architecture and data preparation.

(3) The non-extrapolating models (architecture fails for OOD) learns short-cut solution.

The author also propose hardcode attention layer that incorporates knowledge over the dataset that seems to work well on LEGO. 
strength_and_weaknesses: Strength. The paper provides solid contribution towards understanding the Transformer architecture. By constructing a synthetic benchmark, it provides a series experiment that provide new insight and confirm existing observations. 

Weakness. There is no major weakness.

Minor comment. The related work section can be improved, in particular, there is a previous line of work "unveils" transformer over regular language ( in particular dyck language) that should be cited and discussed. Examples include [1] (and see the reference therein), which studies the dyck language and provide some theory and empirical investigation on what transformer learns.

[1] Self-Attention Networks Can Process Bounded Hierarchical Languages. Shunyu Yao, Binghui Peng, Christos Papadimitriou, Karthik Narasimhan. ACL'2021






clarity,_quality,_novelty_and_reproducibility: Yes
summary_of_the_review: The paper provides solid contribution to the literature and I vote for acceptance.


Some minor questions over "shortcut":
The paper finds transformer sometime learns short-cut solution to the reasoning tasks. However, it is unclear to me why this is a bad thing. I feel it is not equivalent to spurious feature because some reasoning tasks involves short-cut solution or parallelizable solution, e.g., it is possible that a linear size circuit only has depth $O(\log n)$ (i.e., do not follow the chain of reasoning). Some further discussions should be a good addition on the finding!



------------

Post rebuttal

I have read the rebuttal and my evaluation remains.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper