From: ICLR.cc/2023/Conference/Paper1834/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: Explanation of Section4.7
comment: **Explanation:** 

The standard Transformer has a higher token correlation even in the first layer, which harms the learning of later layers. In contrast, our \textsc{Eit} in the shallow layers can maintain a low-level token correlation. Thus, they can collaborate well even with only one EIT encoder layer.

**Experiment Setup:**

We also train a 6-layer big model with 2 EIT encoder layers. We set the batch size to 4096*2. We train it with 56979 updates (only 24 epochs, costing only 7.12h on 8 3090 cards). For more details about the training details, please see Appendix A.

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: Response to Reviewer vEt7 
comment: Thanks for your comments. Following your suggestions, we are rewriting this paper to make it more clear. We will upload it on time.

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: Response to Reviewer Bhuf 
comment: Thanks for your constructive review. I am glad to answer your questions. By the way, we add some new experiments and analysis to further validate the effectiveness of our EIT and verify the improvement is from the capturing of interaction among multi-heads. You can find them in the general response.


**Response to Question1:** 

Now, I will give more clear explanations of this.

There is no contradiction between Section4.3 and Figure10. Section 4.3 describes the attention maps after RFE while Figure 10 visualizes the attention maps after CSI (final attention maps). 

When one wants to benefit from the enlarged subspaces, two steps are needed: 1) generating a large number of attention maps. 2) modeling the well-defined interactions among them to make full use of them. 

In the first step, these large number of attention maps should have a low similarity. And in the second step, the final attention maps after interaction should have higher similarity, and at this point, high similarity is good.

To validate this, we conduct two experiments on the En-De task: 1) replacing the RFE module with attention expansion method. 2) using regularization method to explicitly enlarge the head distances among the final attention maps and setting the regularization ratio to 1.
The results of the first experiment are listed in the first table. As for the second experiment, we exhibit the performance of these methods as well as their corresponding head similarity in the second table and the third table, respectively.  The results verify our statements.


|Model|θ (M)|BLEU|sBLEU|
|---|---|---|---|
|EIT|62M|28.00| 26.9|
|EIT replace with attention expansion |62M|27.41| 26.3|

|Model|θ (M)|BLEU|sBLEU|
|---|---|---|---|
|EIT|62M|28.00| 26.9|
|EIT-reg |62M|27.65| 26.6|

|Model|1|2|3|4|5|6|
|---|---|---|---|---|---|---|
|EIT|0.53|0.71| 0.67| 0.70| 0.81 | 0.79|
|EIT-reg |0.47|0.47| 0.40|0.45|0.37|0.47 |


**Response to Question2:** 

There are two main differences between our high similarity attention maps after CSI and the common redundant attention maps.
1. The high similarity is a result of modeling the interactions of a large number of attention maps generated by RFE. 
2. Compared with the simple redundant attention maps, we can get a confident decision rather than pay attention to numerous positions.

To further eliminate your concerns, we adopt the regularization technology (li-2018) to encourage the distance among attention maps after the CSI. The regularization ratio is set to 1. We conduct experiments on the WMT En-De task and setups are the same as that in our EIT. The results as well as their head similarity are listed in response to Q1. The experimental results also verify our assumption.


**Response to Question3:**

In fact, we have visualized the attention maps of different tasks such as the En-De translation task (Figure 10)and abstractive summarization task (Figure 18). To eliminate your concerns, I also plot the token correlation metric on abstractive summarization task. The results are listed as follows:

|Model|1|2|3|4|5|6|
|---|---|---|---|---|---|---|
|Transformer|0.07|0.12| 0.13| 0.15| 0.15| 0.15|
|EIT |0.04|0.08| 0.12|0.14|0.14|0.14 |


We can conclude that on both tasks, EIT can generate attention maps with more confident decisions and learn features with lower token correlation. As for the local information or global information, we could not find significant enhancement of them on both tasks (see Point 2 in general response). However, we can find by modeling the interaction among multi-head, some layers in Transformer with lower localness turn to significantly higher localness and some layers in Transformer with higher localness turn to significantly lower localness on the abstractive summarization task.  For example, localness value (w=0.1T+1) of the first layer in EIT is 0.90 while the value of the first layer in Transformer is 0.35.



**Response to Question4:**

There are no errors in that table. As you can see, we have added a lot of analysis to reveal why EIT works. Due to the page limitation, I have to leave something in our main paper. However, I will add it to the appendix. The new version of our paper will be uploaded soon.

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: Response to Reviewer zUXt 
comment: 
Thanks for your constructive feedback, we respond to the comments as follows. By the way, we add some new experiments and analysis to further validate the effectiveness of our EIT and verify the improvement is from the capturing of interaction among multi-heads. You can find them in the general response.


**Response to Weakness 1:**

In all three tasks, our EIT can improve the performance, especially for the machine translation task and grammar error task. Notice that our model is just a head modification method without restricting the relationship among the tokens which can be served as a general method (See Point 1 in general response). To further validate, we also evaluate our EIT on the other two kinds of tasks (See Point 2 in the general response).


**Response to Weakness 2:**
In fact, our direct related research field is head modification and we have given a broad comparison (selecting all representative methods) in Table 4. To give a more broad comparison, we add several latest comparisons such as FishFormer (Neurips2022), MoA (EMNLP 2022) and UMST (ICML 2022). The results are listed in the following tables. We can see that our EIT can also outperform them.

|Type|Model|θ (M)|BLEU|sBLEU|
|---|---|---|---|---|
|Head Modification|Refiner (2021b)|-|27.62|26.5|
||Talking-Head (2020)|-|27.51|26.4|
||Collaboration (2020)|-|27.55|26.4|
||DYROUTING (2019)|297M|28.96|-|
||DISAGREE (2018)|-|29.28|-|
||MoA (2022)|200M|29.40|-|
||FISHformer (2022)|-|29.57|-|
|Localness|DMAN (2021)|211M|28.97|27.8|
||CSAN (2019)|-|28.74|-|
||UMST (2022)|242M|29.75|-|
|Deep Transformer|DLCL (2019)|137M|29.30|-|
||ADMIN (2020b)|262M|30.10|-|
||MSC (2020)|272M|30.19|-|
|Our System (Pre-Norm)|Transformer base|62M|27.13|26.0|
||EIT base|62M|**28.00**|26.9|
||E-EIT base|62M|27.72|26.7|
||Transformer deep-48|194M|29.60|28.5|
||EIT deep-48|194M|**30.25**|29.2|
||E-EIT deep-48|194M|30.16|29.1|
||Transformer big|211M|28.80|27.7|
||EIT big|212M|**29.79**|28.7|
||E-EIT big|211M|29.61|28.5|


|Type|Model|θ (M)|BLEU|
|---|---|---|---|
|Basic Baseline|ConvS2S (2017)|-|29.90|
||Transformer (2020c)|-|34.30|
||Transformer (2020b)|-|34.16|
||FlowSeq(2019)|-|31.97|
||Int-TF (2021)|-|32.60|
||DELIGHT (2021)|53M|34.70|
|Head modification|Refiner (2021b)|54M|34.25|
||Talking-Head (2020)|54M|34.35|
||Collaboration (2020)|54M|34.64|
||FISHformer (2022)|49M|34.42|
||MoA (Zhang et al., 2022)|56M|34.39|
|Localness|DMAN (Fan et al., 2021)|-|34.49|
||UMST(Li et al., 2022)|60M|34.81|
|Our System (Pre-Norm)|Transformer base|54M|34.23|
||EIT base|54M|**35.10**|
||E-EIT base|54M|35.01|
||Transformer deep-24|111M|35.00|
||EIT deep-24|111M|**35.40**|
||E-EIT deep-24|111M|35.35|
||Transformer big|196M|34.44|
||EIT big|196M|**34.91**|
||E-EIT big|196M|34.67|

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: Response to Reviewer wR9p (2/2) 
comment: **Response to Reject Reason4:** 

Through Table 6, we can find that the key point of inefficiency lies in the unfriendly support of group convolution in PyTorch. So we replace the mlp-like convolution (in CSI) and mlp-like group convolution (in ISI) with a single convolution and single group convolution, respectively. In other words, a two-layer (group) convolution is replaced by a one-layer (group) convolution, and the setup of them are listed in Appendix.  Thus, through the simple modification, it is possible to greatly increase the speed. 


**Response to Reject Reason5:** 

There is no contradiction between Section4.4 and Figure 5. In fact, we have given a detailed investigation of the head distance among attention maps at each stage of the whole process, e.g., after RFE, after ISI and after CSI. To be more specific, section 4.4 describes the attention maps after CSI (final attention maps) while Figure 5 describes attention maps after RFE. 

Due to the page limit, we remove some figures, which makes you confused about some of our conclusions. In fact, the final attention maps of EIT indeed own higher similarity than other models. To further validate this, we list the head similarity of final attention maps of other models such as Refinere (Zhou et al., 2021b), Talking-heads (Shazeer et al., 2020). and Transformer (Vaswani et al., 2017), under the configuration of 6 in the following table. And again, I want to stress the value list in this table describes the final attention maps rather than the attention maps after RFE.

|Model|1|2|3|4|5|6|
|---|---|---|---|---|---|---|
|Transformer|0.53|0.47|0.41|0.42|0.38|0.36|
|Talking-head| 0.42 | 0.41 | 0.47| 0.37| 0.40| 0.45|
|Refiner|0.49|0.30|0.41|0.36|0.38|0.36|
|EIT|0.53|0.71| 0.67| 0.70| 0.81 | 0.79|

Through the results in this table, we conclude that EIT owns the highest average similarity among attention maps from different heads, compared to other selected models.

Besides, reviewer Bhuf also asks similar questions and we have conducted some new experiments on this part. You can find them in response to Q1 of reviewer Bhuf, which could be helpful for you to understand this part further.

**Response to Reject Reason6:** 

Here I will give more explanations about these two experiments.

The EMHA in both EIT and EIT (selected attn) follows the same calculation to obtain the final attention maps: $[A^1,...,A^M]=Softmax(DEI(RFE([Q^1,...,Q^M],[K^1,...,K^M])))$. The main difference is the aggregation process. The EMHA in EIT obtains the final output as follows: $ O = [A^1V^1, ..., A^MV^M]W^o$, while the EMHA in EIT (selected attn) calculates the final output as follows: $O = [A^1V^1, ..., A^1V^M]W^o$. Similar processes are conducted in E-EIT (selected attn). All the training settings are kept constant, and you can find them in our Appendix. 

As for the experiment in Section 4.7, instead of adopting the EIT encoder layer in all encoder layers, we only adopt it in some layers. For example, if the number of  EIT layers is set to 3, the first three encoder layers will adopt the EIT encoder layer. In this experiment, we still keep the training hyper-parameters constant and you can find them in the Appendix. 

In fact, 5-layer EIT indeed achieves such a result. 

**Response to Reject Reason7:** 

Other errors have been corrected in the new version of our paper.

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: Response to Reviewer wR9p (1/2)
comment: 
Thanks for your constructive comments, below we provide a point by point reply. By the way, we add some new experiments and analysis to further validate the effectiveness of our EIT and verify the improvement is from the capturing of interaction among multi-heads. You can find them in the general response.

**Response to Reject Reason1:** 

Thanks for pointing out this kind of redundancy. A new summarization of our contractions is as follows:

1.  We propose a many-to-many mapping scheme that can generate a large number of attention maps more efficiently.  For example, the attention maps generated by many-to-many mapping are very diverse.
2.  We propose an EIT architecture that can outperform Transformer on three kinds of sequence generation tasks including machine translation, abstractive summarization and grammar error correction, one kind of language modelling task and one kind of brain disease automatic diagnosis task, demonstrating the superiority of EIT.
3.   EIT can attain persistent gains from enlarged heads as well as keep high expressiveness. It can also simulate some property of attention with numerous heads, e.g., anti-oversmoothing. For example, features learned by EIT have a lower token correlation. 
4.  The final attention maps generated by EIT are more confident in their decision, e.g. each token pays more attention to a few positions rather than learning a smoothing view over all positions, though these attention maps own high similarity among each other.

**Response to Reject Reason2:**

In this paper, we default r = M in our EIT and E-EIT settings. However, the choice of r can be served as an efficiency-performance trade-off strategy, so we give the general definition of receptive field with any r in Section 2.2.1 and discuss them in Section 4.6.


**Response to Reject Reason3:**

In fact, we connect the number of attention maps with the size of interaction space. As defined in section 2.2.1, the interaction space is the set that consists of several query-key mappings. For example, an 8-head standard multi-head attention has an interaction space size 8. In attention, the attention maps are usually computed by applying similarity functions to query-key mappings, e.g., cos() or dot-multiplication. So, we say the larger the interaction space is, the more attention maps we can obtain.

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: General Response (2/2)
comment: **Point3: Performance of EIT on other two tasks (encoder only and decoder only).**

To further validate the effectiveness of our EIT, we can also evaluate our EIT on a real-world disease automatic diagnosis task: ABIDE and a language modeling task: wikitext-103. 

For the ABIDE, we follow the setting of BrainNetTF (Neurips2022) and build our BrainNetEITF. Through the results, our method can outperform them with 2 points in terms of accuracy. Note that the experiments are conducted five times and We list the mean and standard deviation. The performance improvements of our BrainNetEITF over the BrainNetTF go through the t-test. We will upload the training logs in the supplementary material soon.  More details will be included in our paper.


|Model|Accuracy|AUC|Sensitivity|Specificity|
|---|---|---|---|---|
|Mvs-GCN(CIBM 2022)|69.4|69.0|69.31|64.45|
|BrainNetTF(Neurips 2022)|71.8±3.0|80.9±2.6|71.1±4.1|72.5±1.9|
|BrainNetEITF(Ours)|73.8±3.2|81.3±2.7|72.0±4.9|76.0±5.2|

As for the wikitext-103 task, we build our EIT on Adaptive Input Transformer (Baevski et al., 2019). However, due to the limited time for rebuttal, we only train an 8-layer model instead of a 16-layer one. Besides, we only train both Transformer and EIT with 169692 updates instead of normal 286000 updates. The results are as follows:

the results are as follows:

|Model|Test PPL|
|---|---|
|Transformer(Adaptive Input)|22.50|
|EIT|21.34|
It can be observed that our EIT can also outperform Transformer with 1.16 PPL.

From: ICLR.cc/2023/Conference/Paper1834/Authors
title: General Response (1/2)
comment: 
Thanks for spending your time reviewing this work. There are some points I want to claim here to help you understand our work. Since we have added many experiments, we submit the response now.

**Point1: Contribution of this work**

Multi-head self-attention (MHSA) is a very common strategy. Previous work thinks that there is no need to own too many heads in MHSA, due to the high redundancy among attention maps. Thus, tremendous pruning heads methods (Michel et al., 2019; Behnke & Heafield, 2020). have been proposed. However, MHSA with an extremely large number of heads theoretically has several good properties such as anti-oversmoothing (Wang et al., 2022), a property highly related to the expressiveness of model, and high efficiency for efficient attention (Bolya et al., 2022), e.g., O($Td^2/M$). These properties are all very vital for Transformer. Unfortunately, simply enlarging the number of heads in MHSA suffers performance degradation. So it is very crucial to dig out how to enable MHSA to efficiently benefit from a large number of heads. This paper aims to answer these questions under the situation of $O(T^2)$ attention. 

In this work, we propose a new many-to-many mapping scheme to generate a large number of diverse attention maps, which is the first step to benefit from enlarged subspaces.  Besides, we establish two relationships among those materials. The inner-subspace interaction (ISI) describes the connection among the attention maps that are generated by the same query. These attention maps have a closer relationship with each other, which provides a suitable solution to distinguish the useful information and abandon the useless parts, since from the perspective of clustering, similar information can be grouped into one. Cross-subspace interaction (CSI) relationship describes the collaboration among different subspaces. It exists in the attention maps generated by different queries. By modeling these relationships, we make full use of these materials. In fact, our EIT indeed has the property of anti-oversmoothing (see Section 4.4, lower token correlation).

Moreover, this work does not contradict those pruning heads methods. One can also apply those methods to our EIT but this is not the focus of our work.

**Point2: Whether the improvements come from the head modification.**

In this work, we adopt group convolution and convolution to model two kinds of interactions though they are only applied to the attention maps. In this process, there is a risk of explicitly introducing local biases into our model. To eliminate this concern, we define a metric to measure the localness. Inspired by DMAN (Fan et al., 2021), given an attention distribution $A$ and a local region $w$, we only keep the attention scores of each token within the local region and ignore the rest. Then we average all the tokens, all the heads and all the samples of test set. The results of the abstractive summarization task (first three tables) and WMT En-De task (last three tables) are as follows:


|Model (w=0.1T+1)|1|2|3|4|5|6|Avg | 
|---|---|---|---|---|---|---|---|
|Transformer|0.35|0.65|0.46|0.44|0.39|0.24| 0.422|
|EIT|0.90|0.16|0.29|0.59|0.44|0.17| 0.425 |


|Model (w=0.2T+1)|1|2|3|4|5|6|Avg | 
|---|---|---|---|---|---|---|---|
|Transformer|0.43|0.71|0.53|0.52|0.47|0.33| 0.498|
|EIT|0.91|0.26|0.37|0.67|0.51|0.26|0.497|

|Model (w=0.3T+1)|1|2|3|4|5|6| Avg|
|---|---|---|---|---|---|---| ---|
|Transformer|0.49|0.76|0.58|0.58|0.53|0.42| 0.560 |
|EIT|0.93|0.36|0.44|0.70|0.57|0.34|0.557 |

|Model (w=0.1T+1)|1|2|3|4|5|6|…|43|44|45|46|47|48|Avg|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Transformer|0.14|0.27|0.25|0.23|0.25|0.34|…|0.18|0.22|0.22|0.18|0.20|0.18| 0.232|
|EIT|0.12|0.13|0.35|0.27|0.24|0.36|…|0.15|0.12|0.10|0.14|0.20|0.15| 0.185|


|Model (w=0.2T+1)|1|2|3|4|5|6|…|43|44|45|46|47|48|Avg|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Transformer|0.24|0.43|0.38|0.36|0.39|0.55|…|0.33|0.37|0.39|0.32|0.35|0.32| 0.385|
|EIT|0.24|0.23|0.52|0.42|0.39|0.50|…|0.27|0.26|0.20|0.27|0.36|0.26| 0.323|


|Model (w=0.3T+1)|1|2|3|4|5|6|…|43|44|45|46|47|48|Avg|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Transformer|0.32|0.51|0.45|0.45|0.47|0.63|…|0.43|0.46|0.48|0.42|0.45|0.42| 0.475|
|EIT|0.32|0.32|0.59|0.50|0.47|0.56|…|0.36|0.36|0.27|0.37|0.46|0.35| 0.409|


Through the results (Column Avg), we can see that there is no local phenomenon on both tasks. Note that the attention maps in the first layer of EIT on the abstractive summarization have a strong local pattern, but the kernel sizes are all set to 1 on this task. So we can conclude that the improvements do not come from introducing local biases. Consequently, we can see that EIT is a general technique without modifying the relationship among tokens and can be applied to different applications not only the sequence generation. To further validate this, we also evaluate our EIT on the other two kinds of tasks (See Point 3).

From: ICLR.cc/2023/Conference/Paper1834/Reviewer_wR9p
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper improves interactive MT via alleviating head degradation in Transformer attention layers. The enhanced interactive transformer (EIT) replaces multi-head self-attention with enhanced multi-head attention (EMHA) for many-to-many mapping strategy. Experiments show that EIT outperforms the Transformer on various configurations on three major seq-seq tasks.
strength_and_weaknesses: Reasons to accept:
- The extensive experiments verify the proposed model properties. 
- The efficiency/performance trade-offs are considered. The proposed efficient version of EIT achieves similar performance compared with standard EIT using fewer costs.

Reasons to reject
Many parts of the paper are not clear enough, bringing difficulties for readers to understand.
  1. The generalization of contributions in Section1 is confusing. The first and third points both refer to the performance on machine translation task, which is repetitive redundancy.
  2. The definition in 2.2.1 is problematic. It seems EMHA uses a receptive field (r) of 8, but it is not clearly stated.
  3. For the first sentence in section 2.2.2, the statement is weird because the number of attention maps is the definition of interaction space.
  4. As an important variants, the description in EFFICIENT VERSION OF EIT is rather brief. It's confusing how to design the model to achieve the efficiency.
  5. "As plotted in Figure 6(left), EIT owns the highest average similarity among attention maps from different heads, compared to all other models." Section4.4 contradicts to Figure5 and can not be inferred from Figure6.
  6. Explanation in Section4.2 and Section4.7 is confusing. The setting of the experiment were poorly explained. Furthermore, the results of the 5-layer EIT in Section4.7 looks strange in Figure9.
  7.  There's a large number of symbols in the paper are not fully explained, which brings reading difficulties, e.g., formula in 2.2.1, 2.2.3.

There are also many errors in the paper:
  1. There is a reference to Figure3.b in Section 2.2.3, but there is no picture numbered b.
  2. Speed in Table6 should be replaced with "Computing time". REF,ISI,CSI will increase the computation, resulting in slower speed.
  3. The right part of Figure6 has two "Baseline f =FFN" which should be a mistake.

clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is a major problem. The motivation is clear but how the interaction between different queries and keys are not well described. 
summary_of_the_review: I think the current version needs to be improved in both description of technical details and the experiments and analysis.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1834/Reviewer_Bhuf
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This article proposed an enhanced interactive transformer (EIT) architecture for sequence generation to simulate the interaction among multi-head subspaces. The proposed enhanced multi-head attention (EHMA) is expected to simulate the many-to-many mapping instead of the existing one-to-one mapping among queries and keys in multiple subspaces and is applied to the self-attention and crossing-attention modules in the standard Transformer architecture. Authors reported the improvements on several sequence generation tasks in terms of BLEU, Precision, Recall, and F value.


strength_and_weaknesses: Strength:
Authors reported the improvements on several sequence generation tasks in terms of BLEU, Precision, Recall, and F value. 


Questions:

1. the analysis in section 4.3 seems to conflict with the result in figure 10. I am confused that whether a high similarity between heads is good or bad, because sometimes you say it is not good, and sometimes you say under certain conditions it does not matter. more clear explanations are needed.

2. the attention maps after the CSI module are nearly the same, from my perspective, the redundant attention is useless, which means the method could be further developed. 

3. I'd like to know if this interaction among multi-heads carries the same information in different tasks, but I do not find it.

4. there is some content removed in this version of the paper, for example, a table that analysis the effect on the placement of DEI on two tasks is removed, is there something wrong with that table?
clarity,_quality,_novelty_and_reproducibility: The contribution of this study was limited, and the experiment did not focus on what kind of useful information was captured by the proposed approach.
summary_of_the_review: Please see the review in the previous sections.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1834/Reviewer_zUXt
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper tackles enhancing the standard Transformers by modifying the multi-head attention mechanism, which this paper call enhanced multi-head attention.
The basic idea of the method is to consider the interaction among the separated heads instead of individual computation of query-key mapping for each individual head.
In other words, the proposed method considers the many-to-many (or all-to-all) interaction among heads, while the baseline method considers only one-to-one interaction.
The experimental results show that the proposed method improved the performance on several text generation tasks, such as machine translation, text summarization, and grammatical error correction. 

 
strength_and_weaknesses: Strength:
* This paper proposes a simple extension of the multi-head attention module that can be used as a default setting of the existing Transformer models.
* The proposed method seems to lead to no harmful effects.


Weaknesses:
* The improvements demonstrated in this paper are limited (not significantly improving the performance for all the tasks).
* I think many other attention modification methods have been proposed in the literature so far. I am not very confident about the experiments since the methods directly compared with the proposed methods are limited and not the latest ones.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The reader can follow the paper, but difficult to understand some of the equations and notations.
Figure 3 can help understanding the proposed method.

Novelty:
To the best of my knowledge, I have not heard a similar approach of modifying the multi-head attention mechanisms like the proposed method. However, I am not following the line of studies for modifying the attention. Therefore, my justification may not extremely accurate.


Reproducibility:
If the authors release their code to the public, it is easy to check the reproducibility (and repeatability) of the proposed method. Currently, we cannot judge this point.
summary_of_the_review: There are no significant flaws in this paper. 
However, I found no apparent large positive novelty of findings that I should strongly recommend this paper to be accepted to the conference.
The performance gains for all the experiments shown in this paper are marginal and not strongly convincing of the effectiveness of the proposed method.


correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: I found no ethical concerns.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1834/Reviewer_vEt7
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a new architecture in which the standard multihead self-attention is replaced with the enhanced multi-head attention (EMHA). By removing the one-to-one mapping constraint between queries and keys in multiple subspaces, EMHA allows queries to attend to multiple keys. 
strength_and_weaknesses: 
Strength: 
There is a significant improvement over the baseline and previous work on a variety of tasks in this paper.

Weaknesses: 
The paper is difficult to read and understand in its current form. Even after reading the paper three times, I am still not sure I understand it completely. 
clarity,_quality,_novelty_and_reproducibility: The idea seems novel. However, as I mentioned the idea is not clear and not reproducible. 
summary_of_the_review: At this point, I would vote to reject the paper as it is. It needs to be rewritten. However, the idea of the paper is very interesting, and I think the results are strong, so after rewriting, I encourage the authors to resubmit it. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough