From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper got three 5s (marginally below threshold) and one 3 (reject). The major challenges include insufficient experimental valdations, misleading contributions, flaws in assumptions and the proofs of theorems, lack of varification on the effectiveness of relaxation, etc. The author rebuttal only addressed the reviewers' concerns partially and all the reviewers maintained their scores. In particular, the paper only address the modified model Eqn. (4), not the original one Eqn. (3), making the analysis less attractive. By the overall scores, the AC recommended rejection.

justification_for_why_not_higher_score: This paper should be rejected for sure.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A

From: ICLR.cc/2023/Conference/Paper4671/Area_Chair_xJtC
title: Please respond to author rebuttals
comment: Dear Reviewers,

The authors have submitted their rebuttals. Please have a look and respond to their efforts. This will be a respect to their hard work. Many thanks!

Area Chair

From: ICLR.cc/2023/Conference/Paper4671/Reviewer_MHEr
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a convex alternative to the attention operation in standard Transformers with the benefits of better optimization and interpretability. The analysis is insightful and deep, and can certainly enlighten further research in the community. 
strength_and_weaknesses: [ Strength ]
+ The motivation is clear and important. The interpretability of Transformers remains an open question in existing research, and it is important to the scientific understanding and design of each module. The paper follows the exploration line and steps into the problem from a new perspective, i.e., making the attention operation convex and easier to be optimized.
+ The presentation is logical. It is easy to know the logic of the paper as it is very clear. Also, concrete proofs are provided, making the proposed method mathematically solid.
+ Experimental results can well support the effectiveness of the proposed method.

[ For further improvements ]
- There is lack of complexity or efficiency analysis. 
- It would be more convincing if the authors can validate the proposed method on more downstream tasks. 
clarity,_quality,_novelty_and_reproducibility: The paper is well presented.
summary_of_the_review: I think the experimental analysis is not sufficient. Although the theoretical analysis is nice, I still don't have enough confidence with the effectiveness of the method.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4671/Reviewer_5uWH
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors study a relaxation of the attention mechanism. For this relaxation (which is still a non-convex problem) they propose a convexification. For single output models, they show how to solve the original problem using the convex equivalent problem. 
strength_and_weaknesses: Strength:
- the paper is in general easy to follow

Weaknesses: 
- The paper's contributions (as listed on the title, abstract, intro) are misleading. The papers does *not* analyze the self-attention mechanism (eqn (2)) but something different (eqn (5)). (W_1 is independent of X_i which introduces something different from self-attention. It means that the authors are trying to convexify another problem, not self-attention layer.) 
I understand that (2) is challenging. I can also understand that potentially (5) is already challenging and of interest, but then (a) the wording and the claims have to be adjusted and (b) it would be needed to show that (5) is (practically) interesting/relevant (Again, in experiments, they compare their results with equation (8) which is something different from self-attention layer.)

- In Prop. 1 it is assumed that there are h non-zero rows on Z. There is no guarantee for that and is unclear how the prop changes otherwise. No analogue of Prop. 1 is given for vector case.
- In the proofs of the main theorems, I have a question about the use of Sion's min-max theorem. Take for example Thm1 where this step is explicitly stated in the proof. As far as I know, Sion's minimax theorem is valid provided at least one set is compact, but 3 equations below eqn (20), but sets appear unbounded. 
- Also in proof of Thm 1, some more details should be given  the last part of
proof when they use KKT conditions.
- Some typos in the paper - e.g., equation (2). 
- Question: In the experiments, how do you measure test/train error of the convexified problem? It suffices to know Z and you don't need to solve for w's (with an alike of Prop 1)?
clarity,_quality,_novelty_and_reproducibility: Please see detailed comments above
summary_of_the_review: I have concerns about the presentation of the results (see first bullet in Weaknesses) as well as the justification of some of the theory claims (see second and third bullets). In view of these, I am afraid I have to recommend rejection.
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4671/Reviewer_qfuJ
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper derives a convex formulation for self-attention in transformers. The authors extend the formulation to various transformers architectures and then apply this convex formulation with an additional regularization for training transformers to find a globally optimal solution. The proposed formulation also suggests an implicit regularization mechanism induced by our attention mechanism. The authors verify the advantage of their convex formulation via fine-tuning experiments with BERT and algorithmic benchmarks.
strength_and_weaknesses: **Strong points:**

1. The paper proposes an interesting convex formulation for self-attention. This convex formulation open up a new approach to studying attention mechanism and transformer. 

2. The paper is well-motivated and discusses relevant related works.

3. The paper is well-written, and I really enjoy reading the paper.

**Weak points:**

1. My main concern is the proposed convex formulation is very similar to the convex formulation proposed in [1]. Even though [1] only discusses linear and ReLU activation multi-head self-attention, the techniques in both papers are quite the same. [1] also extends their results to other mixing mechanisms such as the MLP mixer and Fourier Neural Operator. 

2. My second concern is that the paper needs more experimental results on different benchmarks and applications to justify the advantage of the proposed convex formulation.

**References:**

[1] Sahiner, Arda, Tolga Ergen, Batu Ozturkler, John Pauly, Morteza Mardani, and Mert Pilanci. "Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers." ICML (2022).


clarity,_quality,_novelty_and_reproducibility: The paper is well-written. The quality is high, but the novelty of the paper is questionable. The details on the experiment settings are useful, and the submitted code helps reproduce the results reported in the paper.

summary_of_the_review: Overall, this paper could be an interesting algorithmic contribution. However, my main concern is the novelty of the paper. Also, more experimental results are needed to justify the advantage of the proposed convex formulation.

Currently, I am leaning toward rejecting the paper. However, given additional clarifications on the two main concerns above in an author response, I would be willing to increase the score.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: I have no ethics concerns for this paper.

recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4671/Reviewer_Dm3W
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper relaxes the softmax function to a linear operation with unit simplex constraints. Then for the self-attention-only network with a scalar or vector output, the paper shows that the nonconvex optimization problem could be cast as a convex one. It also demonstrates that the convexified problem has an implicit regularization mechanism, which promotes sparsity across tokens. For the self-attention network with an FCN layer, the paper shows that the nonconvex optimization problem with the gated ReLU activation is equivalent to a convex optimization problem. Experiments with the nonconvex and convex approaches are presented. Interestingly, the paper shows that the convex approach could mitigate the impact of the grokking phenomenon. 
strength_and_weaknesses: Strength:
By the observation that softmax function output a probability distribution, the paper presents an convex alternative to the self-attention mechanism and reformulated the training problem as convex optimization problem. The paper reveals that the convex optimization problem has the sparsity-inducing regularization. It also shows that the convex problem could mitigate the impact of the grokking phenomenon.

Weaknesses:
There are three main comments below, which need to consider carefully.
First, the relaxation of the softmax function is the fundamental assumption, i.e., the reformulation in Eq.(3). The relaxed linear operation does not depend on X_i, which may be not an appropriate approximation. Also, the interpretability of the approach does not explain clearly. Moreover, the paper does not verify the effectiveness of the relaxation. For example, it is better to compare with the regularized training problem in (1) using SGD.
Second, the shapes of the input and output of a Transformer block are usually the same. However, the paper only demonstrates the self-attention-only networks with the scalar and vector output.  What about the matrix output with the same shape of X_i?
Third, the paper tests with deep settings by stack the convex transformer layers in the experiments. It is better to give more detailed explanations on this issue. Is there any difficulty to extend to deep Transformers?
Other minor comments:
1.	There are two “of “ letters at the second line of the first paragraph in page 2.
2.	In (2), “X” should be “X_i”. 
3.	At the first line of the second paragraph in page 4, “varios”should be “various”.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and it is easy to understand the idea. However, the basic assumption of the paper is not so theoretically sound. It is better to investigate the effectiveness of the assumption thoroughly. However, the paper does not consider this as a crucial problem. 
summary_of_the_review: The paper proposes the convex optimization for self-attention-only network with and without FCN layers. The paper also reveals the sparsity-inducing regularization and advantage in mitigating the impact of the grokking phenomenon. However, it does not compare with the regularized training problems in (2) with the variants of SGD. Also, the interpretability of the learned W_1 does not present. So the effectiveness of the proposed approach is not clear. This is the main weakness of the paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: The paper does not have ethics concerns.
recommendation: 5: marginally below the acceptance threshold