Paper: LOLAMEME: LOGIC, LANGUAGE, MEMORY, MECHANISTIC FRAMEWORK (ICLR-2024)

Reviewer comment (Presentation):
I would suggest that an illustration figure be provided to clearly show the main idea of the LOLAMEME framework, which will make this work easier to understand.

Author response:
3. Why not provide an illustration figure to show the model framework?
    1. Thanks for the feedback, we will definitely add model architecture as well as LOLAMEME framework illustration in the next revision. This is the anonymous link to a draft of LOLAMEME framework illustration:  https://ibb.co/qWCPRnW . We will add a revised version to the paper.

Reviewer comment (Novelty):
What are the differences among T HEX, GPT-2, and Hyena?

Author response:
1. What are the differences among T HEX, GPT-2, and Hyena?
    1. In the paper, we use GPT-2 model which uses attention operator, Hyena model which uses hyena operator and T-Hex which is similar to hyena model in which 1 layer of hyena operator is replaced with attention operator.

Reviewer comment (Novelty):
What are the advantages of T HEX?

Author response:
2. What are the advantages of T HEX?
    1. Using LOLAMEME framework, we tested the GPT-2 and hyena model architectures. We identified various areas where GPT-2 using attention operator works better and Hyena using hyena operator works better. Guided by the experiment results using LOLAMEME framework, we show that attention and hyena have complementary strengths using T-HEX. Using attention operator together with hyena operator, boosts the model architecture’s performance on various aspects such as memorization, in-context learning, language mixing, longer input length and more listed in the paper. 
    2. We show T-Hex performs better than Hyena and GPT-2 on various aspects of the language and we also show results on existing dataset “Listops”. Using LOLAMEME framework, better architectural design changes can be developed and tested on various aspects of the language.

Reviewer comment (Novelty):
The motivation and problem formulation of this work is unclear. The novelty and contribution of this paper are somewhat limited. The proposed new architectures are simply constructed by replacing certain layers of the Hyena model with the GPT-2 layer. Although some experiments demonstrate better performance on the proposed two test datasets, there may be a lack of validation experiments on other existing datasets. Additionally, providing some interesting findings or interpretations about the experiments through a deeper analysis of the proposed architectures should be better.

Author response:
1. The motivation and problem formulation of this work is unclear. Natural languages has several aspects to it and it has become extremely difficult to test a model architecture on one or a set of aspects of the natural language. LOLAMEME framework is built to evaluate model architectures on various aspects of language such as grammer, vocabulary, logic/reasoning, memorization, multiple languages and more listed in the paper.  
2. The proposed new architectures are simply constructed by replacing certain layers of the Hyena model with the GPT-2 layer. Although some experiments demonstrate better performance on the proposed two test datasets, there may be a lack of validation experiments on other existing datasets. Additionally, providing some interesting findings or interpretations about the experiments through a deeper analysis of the proposed architectures should be better.  
    1. Using LOLAMEME framework, we tested the GPT-2 and hyena model architectures. We identified various areas where GPT-2 using attention operator works better and Hyena using hyena operator works better. Guided by the experiment results using LOLAMEME framework, we show that attention and hyena have complementary strengths using T-HEX. Using attention operator together with hyena operator, boosts the model architecture’s performance on various aspects such as memorization, in-context learning, language mixing, longer input length and more listed in the paper. 
    2. We show T-Hex performs better than Hyena and GPT-2 on various aspects of the language and we also show results on existing dataset “Listops”. Using LOLAMEME framework, better architectural design changes can be developed and tested on various aspects of the language.

Reviewer comment (Writing):
There are quite a few typo errors, including grammar and table issues, in this paper. For example, in the abstract, it states "We propose the hybrid architecture T HEX and use LOLAMEME framework is used to compare three architectures." There are also typo errors in tables 3, 4, and 5. The grammar, figures, and tables in this paper may require some polishing.

Author response:
4.  There are quite a few typo errors, including grammar and table issues, in this paper. For example, in the abstract, it states "We propose the hybrid architecture T HEX and use LOLAMEME framework is used to compare three architectures." There are also typo errors in tables 3, 4, and 5. The grammar, figures, and tables in this paper may require some polishing. Thank you for your feedback, we will address this in the next revision.

Reviewer comment (Novelty):
The related work on mechanistic interpretability is not comprehensive. In fact, there is a considerable amount of work such as [1], [2], [3] attempting to interpret and understand the mechanisms of LLMs.

Author response:
5. The related work on mechanistic interpretability is not comprehensive. In fact, there is a considerable amount of work such as [1], [2], [3] attempting to interpret and understand the mechanisms of LLMs.  We realize the importance of a comprehensive review of related work. Our paper will be updated to include a more exhaustive list of research in the domain of LLMs interpretability, referencing works such as [1], [2], [3] cited by the reviewer