From: ICLR.cc/2023/Conference/Paper3428/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper3428/Reviewer_aDFb
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The author(s) used four binary and fine-grained public sentiment analysis datasets to experiment their proposed novel contrastive logical knowledge learning (CLK) framework. It was evaluated with baseline model and interesting comparative results were recorded. 
strength_and_weaknesses: The paper is well written, detailed and theory based. The methods applied are novel and the flow is good, showing good promises of reproducibility. 

Weaknesses: 
1. The actual number of the baseline model in Section 4 is unclear (4 or 3), and why these?
2. The demonstration of how the proposed method can combine with variants of the base models to form interpretable ones was not shown
3. The interpretable parts are unclear 
clarity,_quality,_novelty_and_reproducibility: The quality of the paper is Ok. The novelty it demonstrates is good. 
summary_of_the_review: The ideas in the paper is empirically demonstrated and the theories were presented with clarity. It would be improved if the weaknesses identified are addressed. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3428/Reviewer_cdQ3
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper designs a new training methodology for sentiment classification. The goal of this paper is to make the model's decision making interpretable, using Talmudic public announcement logic (TPK). TPK seems to be some kind of a semantic graph representation between tokens of a sentence. The paper adopts contrastive learning to learn word representations which can be mapped onto nodes of the TPK tree/graph for reasoning / inference. The paper tests their approach on SST2 and IMDb.
strength_and_weaknesses: 1. The paper is poorly written and hard to understand. Section 2 makes little sense, what is modal logic and public annoucement logic? I didn't understand the need for LDA in Section 3.1 (why not take word vectors directly, or BERT CLS vectors for label representations?) Several variables of the loss function equation were not defined, which are critical to understand the approach (for instance, in Section 3.3 what exactly is bold faced p? What does ground truth of X_i mean?) The paper also assumes readers are familiar with Chen et al. 2022, which the work is heavily derived from.

2. It's unclear to me how this method compares to other interpretability methods in NLP, such as token salience maps (see https://arxiv.org/abs/1909.09251 for an overview of some methods), or generating explanations by large LMs (joke explanation generation in https://arxiv.org/abs/2204.02311). The proposed method seems unnecessarily complex for interpreting neural networks, and it's not compared against any prior work on interpretable NLP.

3. The baselines in the paper look very weak. 73.5% on SST2 with BERT is quite strange, the BERT paper reports 93-94% accuracy. Even 1 layer CNNs from 2014 (https://arxiv.org/pdf/1408.5882.pdf) get around 88%. Moreover, the field rarely evaluates on SST given the performance of current models is much better than humans on SST (https://gluebenchmark.com/leaderboard).

4. The metrics in the paper don't make much sense to me. What is the macro f1 between? What is "model information" in Section 4.2? Is the F1 score for the binary classification performance itself (seems so, given the similarity between f1/accuracy scores)? If yes, exactly is this an interpretability metric? No other metrics have been reported for interpretability, so the evaluation seems to be weak.
clarity,_quality,_novelty_and_reproducibility: *Clarity* - I didn't understand the details of the methodology (despite reading Section 2/3 three times), my summary above is at best an educated guess.

*Quality* - The paper is not very relevant to modern NLP (see weakness #2, #3), and metrics seem to be weak (weakness #4)

*Novelty* - Not sure, but I've never seen TPK being used in neural networks before this

*Reproducibility* - Probably hard to reproduce unless code is released
summary_of_the_review: I had a hard time understanding the details of the methodology in this paper (weakness #1). However parts of the experimental setup did make sense, and I think it's not very relevant to modern NLP (weakness #2,3,4). Based on this, I'm voting rejection.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3428/Reviewer_dFzD
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work aims to investigate the model interpretability in the sentiment analysis task. Specifically, they introduce a contrastive logical knowledge learning method which consists of a knowledge acquisition module, a knowledge representation module, a knowledge learning module, and a knowledge reasoning module. The knowledge acquisition module is able to extract logical knowledge from the existing datasets. The knowledge representation module employs TPK to represent the logic relationship involved in the deep learning model. The knowledge learning module is enhanced via contrastive learning mechanism to address the anisotropy and uncertainty of logical knowledge. Finally, the knowledge reasoning module involves a structural causal model (SCM) which can generate human-readable explanations.




**Strengths**

1. the paper addresses an important problem. 
2. The method is intuitive, and the TPK is novel. 
3. The analyses presented in the paper do lead to several findings about the model interpretability, thus they back the claims risen in the paper.






**Weaknesses**

1. The motivation is ill-defined. The authors did not explain clearly what logical knowledge is in the sentiment analysis task and why logical knowledge is helpful to improve the interpretability of the sentiment analysis models. Specifically, in the Introduction section: 
	- "For instance, when a physician utilizes a personalized medical recommendation ... whether the system is reliable." I don't think the example is able to well show the criticality of model interpretability.
	- "According to (Chen et al. 2022), there s a universal contextual semantic logical relationship in bidirectional deep neural networks, ... model context dependency." What do the authors mean by "universal contextual semantic logical relationship"? Why does the logical relationship only exist in bidirectional deep neural networks (BDNN)? If the logical relationship has existed in BDNN, why TPK is needed for representation？ Are there any other methods？ 
	- "Considering the sequence ... associated with words "terrible" and "bad"." The correlation between words and labels is the contextual semantic logical relationship? The authors claim that TPK is applied to represent the logical relationship due to TPK's temporal characteristics and capability of modeling context dependency. However, I argue whether extracting the correlation between words and labels needs temporal features. Besides, it needs to focus on the dependency between words and labels rather than context.
	- "(1) how to acquire and represent the logical knowledge ... ?" According to the aforementioned claim, TPK can represent logical knowledge, why do the authors raise this question?.
	- The two research questions the authors drew commonly exist in all kinds of NLP tasks, not only in sentiment analysis. However, the title shows that they only pay attention to sentiment analysis.  
2. There are many errors and typos in the Method section, resulting in difficulty in analyzing the reasonableness of the proposed methods. For example, 
	- \pi can relect the information distribution of each word (or sequence), and Distribution(\matchcal{Y}) dnotes the information distribution of \matchcal{Y}, \matchcal{Y} denotes the information of each state. So, when \pi= Distribution(\matchcal{Y}), does it mean each state consists of word or sequence, i.e., s_i = \mathcal{Y}?  
	- "the same semantic knowledge should be as close as possible..." what is semantic knowledge?
	- how to obtain \bm{p}_i?
	- what does M, M^{'}, \mathbb{M}, \mathbb{M}_{i, j} mean, respectively?
	- In Eq. 6, How to compute in "\mathbb{M} \cup \bm{h}" ? 
	- In section 3.1, there are two symbols that represent the document of i-th label category, \mathbb{D}_i, and \mathcal{D}_i.
	- "Let \matchcal{Y} denote..." should be "Let \matchcal{Y} denotes..." 
	- repeat sentence. For example, the sentence, "e_{D_i} denotes the embedding vector of i-th label", appears twice.
	- h_{l} = [h_{e^{D_1}}, ..,h_{e^{D_2}}] should be h_{l} = [h_{e^{D_1}}, ..,h_{e^{D_{|N^{l}|}}}].
	- z_3 and z_4 were not used. Besides, [z_1, ..., z_k] is a topic vector, resulting in conflicts.
3. Many paradoxical claims. For example, the authors show that explanations of the model can be easily provided with an interpretable structure of TPK. But they paradoxically claim that it is difficult to measure the interpretability of the model. Besides, they propose a structural causal model (SCM) to characterize the logic behind sentiment analysis tasks. However, they still visualize the TPK model in Case Study. I did not see the necessity and effectiveness of SCM.
4. Evaluation metrics are ambiguous. The authors did not clearly demonstrate the how F1-score is able to evaluate the model interpretability. To my knowledge, the vanilla F1-score is only used to evaluate the model performance. If there is another calculation of F1-score, the authors should provide a detailed calculation process.
5. The analysis of experimental results is not deep enough. For example, the authors just show that the higher Acc results demonstrate the stability of CLK, but they fail to explain the reason in detail.
The lower P, R, and F1 denote the low consistency between the model interpretability and the model information. However, CLK is responsible for the knowledge reasoning that provides the model interpretability. Therefore, I argue whether higher Acc results can demonstrate the stability of CLK.
strength_and_weaknesses: See above.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: This paper introduces the task of interpretable sentiment analysis, and presents models, metrics and empirical results on it. 
While I appreciate the pivotal problem, I am concerned about the ill-defined motivation, paradoxical claims, bad presentation, and superficial experimental analysis.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough