A N INTERPRETABLE CONTRASTIVE LOGICAL KNOWLEDGE LEARNING METHOD FOR SENTIMENT ANALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Current interpretable sentiment analysis (ISA) methods frequently underperform state-of-the-art models, and few of them cast light on the inner working of pretrained models.
In this work, we fill the gap by addressing four key research challenges in ISA—knowledge acquisition, knowledge representation, knowledge learning and knowledge reasoning—in one unified framework.
Theoretically, we propose a novel contrasitive logical knowledge learning (CLK) framework that can visualize the decisions made through deterministic Talmudic public announcement logic semantics.
We apply CLK to current popular sentiment analysis models to obtain CLK based interpretable ones.
Empirically, experimental results of both binary sentiment analysis tasks and fine-grained sentiment analysis tasks indicate that CLK can achieve an effective trade-off between accuracy and interpretability.
Furthermore, we find that CLK can reduce the uncertainty of logical knowledge for discriminative labels in unbalanced fine-grained sentiment analysis tasks.
Besides, we carry out a case study to investigate the fidelity of model interpretability through knowledge reasoning, which demonstrates that the explanations provided by our method are causally effective for sentiment analysis tasks.
I NTRODUCTION
Sentiment analysis (also known as opinion mining) is a primary task in natural language processing that identifies the emotional inclination of humans toward particular topics (Zitnik et al., 2022), it is commonly used in marketing (Valle-Cruz et al., 2022), customer service (Li et al., 2019), medical healthcare (Sanglerdsinlapachai et al., 2021), and social media (Arias et al., 2022).
Investigating model interpretability is critical to obtain user trust in high-stake domains (Ito et al., 2020; Zhang et al., 2013).
Several approaches have been proposed for interpretability in sentiment analysis, such as Tsetlin Machine (Yadav et al., 2021), Hidden Markov Model based methods (Perikos et al., 2021), and fuzzy rule based methods (Liu & Cocea, 2017).
However, while the existing approaches can provide explanations, the fidelity and rationality of model interpretability are unclear for sentiment analysis tasks, which is insufficient for user trust.
In addition, despite that deep learning models have shown promising performance (Alaparthi & Mishra, 2020), few interpretable methods have involved them in sentiment analysis tasks.
According to (Chen et al., 2022), the logical knowledge implicitly in bidirectional recurrent neural networks (BRNNs) can be represented by Talmudic public announcement logic (TPK) (Abraham et al., 2013), since TPK has natural temporal characteristics and can model context dependency.
In light of this, it’s possible to learn the interpretable TPK from bidirectional deep neural networks (BDNNs) by modeling word-label dependency, since the word-label dependency in sequential data can also be represented by temporal logic.
Considering the sequence “I like this novel very much, but the film is terrible, so I will give a bad review”, label semantics are associated with correlated words in the sequence.
Specifically, the label “positive” is associated with the words “like”, and the label “negative” is associated with the words “terrible” and “bad”, the predicted label is determined by the temporal characteristics of these correlated words.
A few studies have looked into the correlation between words and labels in sentiment analysis (Ito et al., 2020), the logical knowledge implicitly in the trained deep learning models is still unspecified although it can help us to reveal how a BDNN operates.
It is worth noting that the knowledge representation of deep learning models is challenging due to the anisotropy and uncertainty of logical knowledge, where anisotropy is produced by the vector representation derived from transformer when using pre-trained models, meaning that the vectors will be unevenly spaced and filled in a small conical area (Cai et al., 2020; Ethayarajh, 2019).
The uncertainty stems from two avenues.
On the one hand, the knowledge of labels with similar meanings (like ”slightly positive” and ”positive”) has similar logic knowledge, which is easily confused; on the other hand, logical knowledge trained with model during the training stage leads to randomness.
As a result, the following research questions emerge naturally to fill the research gap: (1) how to acquire effective logical knowledge and integrate it into model training in sentiment analysis tasks?
(2) how to generate human-understandable explanations for sentiment analysis tasks and evaluate the fidelity of model interpretability?
To answer the first research question, we propose a contrastive logical knowledge learning (CLK) method based on the Talmudic public announcement logic (TPK) and contrastive learning.
CLK can be regarded as a component that can be added to any deep learning models, and the challenge is that models constructed in such a way may suffer from lower performance when the transparency is brought in, thus the logical knowledge need to be accurate and effective.
To address this challenge, we design a knowledge acquisition module (Figure 1 (a)), a knowledge representation module (Figure 1 (b)) and a knowledge learning module (Figure 1 (c)) in CLK.
The logical knowledge is obtained from two avenues: firstly, we employ the latent Dirichlet allocation to extract the semantic knowledge from existing datasets, and further utilize this logical knowledge to form a good label representation (Section 3.1).
Secondly, we learn the potential public announcements of TPK by calculating the similarity of label representation and word representation (Section 3.3).
Based on the learned public announcements, the logical knowledge can be represented in the form of TPK (Section 3.2).
To address the anisotropy and uncertainty of logical knowledge, we aggregate it with contrastive learning to enhance the uniformity and reduce uncertainty, thus encouraging the model to learn more discriminative representations for similar labels (Section 3.3).
To answer the second question, we design a knowledge reasoning module in CLK (Section 3.4), which can generate human-readable explanations based on TPK semantics and validate the fidelity of the model interpretability by causal reasoning.
By displaying how sentiment changes from the beginning to the end of sequences, CLK can visualize decisions made in the form of TPK models.
Through reasoning on a structural causal model for sentiment analysis tasks, the effectiveness and rationality of the model interpretability are investigated.
In addition, we conduct extensive experiments on both binary sentiment analysis datasets and fine-grained sentiment analysis datasets, and the results demonstrate that CLK can achieve an effective trade-off between accuracy and interpretability by leveraging the benefits of both logical knowledge and contrastive learning, especially compared with pre-trained models on Weibo-8 and Yelp-2 dataset (Section 5).
Besides, we conduct a ablation study to show how knowledge acquisition module and knowledge learning module influence model output (Section 5.3).
D ETERMINISTIC T ALMUDIC P UBLIC A NNOUNCEMENT LOGIC
In this section, we briefly introduce the syntax and semantics of of deterministic Talmudic Public Announcement logic (TPK), and the detailed background of TPK can be found in Appendix B.
Syntax The syntax of TPK is inspired by modal logic and public announcement logic.
Fix the non-empty sets of propositions and modal operators □ , ⊟ , Y and Y , the minimal syntax of TPK can be specified in Backus–Naur form as follows:
[Equation]
where p i is a propositional letter for i ∈ N , ϕ and ψ are well-formed TPK formulas.
Notice that □ is the standard necessity operator, Y is a yesterday operator for □ , ⊟ is a modal operator corresponding to function ρ , and Y is a yesterday operator for ⊟ .
ρ is a functional relation that denotes the public announcement in TPK, which is specified as below.
Definition Based on the above language of TPK, a deterministic TPK (Taimudic public announcement logic) model
(Abraham et al., 2013) is defined as a 6-tuple ⟨ S, R, R ∗ , ρ, s 0 , π ⟩ where ( S, R, s 0 ) is a directed tree with root s 0 and tree successor relation R , R ∗ is the transitive closure of R , ρ :
S → S is a functional relation, π is the valuation function.
For ∀ x, y, z, v ∈ S , we say ρ is a public announcement of TPK
iff ρ satisfies the following properties:(a)
xρy
→ y ̸ = s 0 .
(b) Let zRx , the successors of z are publicly clarified to be v , where zRv holds, if we have xρy , with vR ∗ y .
(c) For deterministic actions,
xρy ∧ xρz → y = z .
(d) Since every z has a unique successor which is publicly clarified at some point, this means that there exists a path the model deterministic.
(e) Let D be the distance from the root and u, v ∈ S , whenever uRv then D ( v ) =
D ( u ) + 1 .
xρy ∧ ¬ ( yRx ) ∧
y ̸
= x → D ( y ) =
D ( x ) 1 .
Semantics Formulas of TPK are interpreted over state transition systems whose transitions are determined by the non-deterministic (deterministic) actions of agents.
In TPK, the modal operators can describe the property of states
i.e., the path formula s ⊨
□ ϕ is read “ ϕ holds in each state that is accessible to s ”.
For ∀ t, s ∈ S , the semantics of TPK are as follows: as for the relation R : t ⊨ □ A iff for all s such that tRs we have s ⊨ A ; t ⊨ Y
A iff for all s such that sRt we have s ⊨ A ; If t = sthen Y A = ⊥ .
As for the functional relation ρ : t ⊨ ⊟
A iff for all s such that tρs we have s ⊨ A ; t ⊨ Y
A iff for all s such that sρt we have s | =
A and if no such s exists then ⊥ .
Now we can write axioms and attempt to prove completeness based on above operators, but it is not our purpose in this paper, the interested reader can find more details in (Abraham et al., 2013).
M ETHOD
3. K NOWLEDGE ACQUISITION
Latent Dirichlet Allocation (LDA) is a topic model which can give the topic of each document in the document set in the form of a probability distribution (Blei et al., 2003).
There is only one corresponding label for each input sequence in our sentiment analysis datasets.
Firstly, we classify all sequences into documents based on their labels and obtain corresponding documents.
Let D i represent the sequence-level document of i -th label category, we have D i =
{ X 1 , X 2 , ...
X V } , where X v denotes the v -th sequence in the document, V denotes the number of sequences labeled as D i in the datasets, and N v is the number of words in v -th sequence.
By dividing each sequence into a series of tokens (words) and utilizing these tokens to represent the document, we obtain D i
= { x 1 , x 2 , ...
x M } , where M is the number of tokens and x m indicates the m -th token, D i denotes the token -level document of i -th label category.
Suppose the document D
i has K ( K > 0) topics, the topic probability distribution of X v is denoted as θ v = P
( z | X v ) , where z
=
[ z 1 , z 2 , . .
. , z k ] is a topic vector.
Thus the topic probability distribution of all sequences in D i can be represented as Θ = { θ v } Vv =1 ∈ R V × K , where the prior distribution of θ v is a Dirichlet distribution with prior parameter α such that P
θ v | α ) ∼ Dirichlet ( θ v | α ) .
Assume that there are N v words in sequence X v , and let vector n v = [ n 1 v , n 2 v , . . . , n kv ] denote the number of occurrence of each topic in X v , where n kv denotes the number of occurrence of topic z k .
Then n v ∼ multi ( n v | θ v , N v ) , where multi ( · ) represents the multinomial distribution function.
Since a Dirichlet distribution is the prior distribution of a multinomial distribution (see Appendix E.1), the posterior distribution of θ v can be computed by P ( θ v | n v , α ) ∼ Dirichlet ( θ v | ( α + n v ) ) .
Since all sequences in document D i are independent of each other, the probability of the topic in the corpus can be obtained by (see Appendix E.2 for more details):
[Equation]
Let ϕ =
P ( w | z k ) represent the word probability distribution of z k , then the word probability distribution of all topics is given by: Φ = { ϕ k } Kk =1 ∈ R K × M .
Similarly, the prior distribution of ϕ k is given by a Dirichlet distribution with prior parameter β such that P ( ϕ k | β ) ∼ Dirichlet ( ϕ k | β ) .
Assume that there are N k words in topic z k , and let vector n k = [ n 1 k , n 2 k , . . . , n mk ] denote the number of occurrence of each words in topic z k , then n k ∼ multi ( n k | ϕ k , N k ) .
Similarly, the posterior distribution of ϕ k is given by P ( ϕ k | n k , β ) ∼ Dirichlet ( ϕ k | ( β + n k ) ) .
Since these topics in the document are independent of each other, the word probability distribution in D i can be obtained as follows (Blei et al., 2003)
(see Appendix E.3 for more details):
[Equation]
Above information allows us to derive the following formula for joint probability distribution of word-topic:
[Equation]
where P ( x, z | α, β ) represents the topic-word probability.
This paper sets α = 0 .
01 , β = 0 .and
utilizes variational inference-based EM algorithm (Wang et al., 2018) to produce the probability distributions, then the probability P ( x, z | α, β ) is solved iteratively.
Based on the solved
P ( x, z | α, β ) , we can get the words with high probability under a certain topic by sorting the probability values.
Since we have divided the training data into several documents according to their labels, we can directly extract T words with the highest probability from these documents based on
P ( x, z | α, β ) , then utilize them to represent the i -th label such that D
′ i
= { w 1 , w 2 , . .
. , w T } .
Glove vectors is a word representation tool that considers the statistics of words ( by co-occurrence matrix) in the corpus to learn word representation (Pennington et al., 2014).
To obtain a better word representation for above T words, we introduce a pre-trained glove model provided by (Pennington et al., 2014), and get a set of word vectors h D ′
i
= { h w 1 , h w 2 , ...
h w T } , where h w i is the word vector of w i based on the pre-trained Glove model.
Remark that the total number of these vectors are T since we have extracted top T ( T ≥ 0 ) words based on P ( x, z | α, β ) .
Then, the average of vectors of all w i can represent D i as h D i = 1 T (cid:80)
Tj =1 h w j , where h D i denotes the average vector of D i .
This paper set T = 25 and use h D i it to initial label embeddings.
3. K NOWLEDGE REPRESENTATION
For each input sequence X =
( x 1 , x 2 , ..., x n ) , we get the corresponding feature vectors (hidden states)
h e i of each word by a base model (or baseline), denoted as h =
[ h e 1 , h e 2 , ...
h e n ] , where e i denotes the text embedding vector of x i by the Embedding layer and h e i denotes the hidden state of word x i through a deep learning network.
Then we employ two dense layers with different activation functions to get p = softmax ( relu ( h )) , where function relu ( x ) = max (0 , x ) and function softmax ( z i ) =
e z i (cid:80) ki =1 e zi .
The base model just computes the loss by p and true labels.
Despite knowing this training process, we still know nothing about the logic relationship of the hidden states in the base model.
Inspired by the work of (Chen et al., 2022), we employ the deterministic Talmudic public announcement logic (TPK) to represent this kind of logical knowledge.
As stated in Section 2, TPK is composed of a 6-tuple ⟨ S, R, R ∗ , ρ, s 0 , π ⟩ .
Thus, for the state set S of a TPK model, we construct them by the obtained sequence of DNNs
such that s i
= { h e 1 , ..., h e i , r } ( i > 0) , where s i ∈ S denotes the i -th state; r denotes a propositional letter generated by the learned TPK; the root s 0 is an empty state and denotes the start of the model.
Similarly to (Chen et al., 2022), the connectivity of these states is represented by the successor relation R and transitive closure R ∗ in TPK.
Public announcement function ρ in TPK can represent the logic relationship of hidden states by showing how current states are determined by future information.
Notice that, this kind of logical relationship always changes during training, necessitating the training of the model with public announcements.
In next subsection, we will discuss how to learn the public announcements of TPK and how to employ the logical knowledge to guide model training.
The assignment function π can reflect the information distribution of each word (or sequence) since it stores the true value of each proposition variable.
Let Y denote the information of each state, then π = Distribution {Y} , where Distribution {Y} denotes the information distribution of Y , which is established by M specified in next subsection.
3. C ONTRASTIVE LOGICAL KNOWLEDGE LEARNING
The uncertainty of logical knowledge is the main obstacle in learning effective representations.
Different from the work in (Chen et al., 2022), this paper learns potential public announcements through similarity computing and contrastive learning.
Since the label vector h D i contains the logical knowledge from datasets, we use these h D i to initialize label embeddings and then get e l =
[ e D 1 , e D 2 , ...
e D n ](1 ≤ n ≤ N l ) , where e D i denotes the embedding vector of i -th label.
Notably, for a non-pretrained based model, we use a DNN (BLTM or BGRU) to encode each label vector after label embedding, and obtain the label vector h l =
[ h e D 1 , h e D 2 , ... h e Dn ](1 ≤ n ≤ N l ) .
Then we calculate the similarity matrix M ∈ R n ×| N l | of each word vector h e j ∈ h and label vector h e Di ∈ h l in CLK as below:
[Equation]
where f 1 and f 2 are trained parameters.
Remark that, for pre-trained models, we directly use e l and h to calculate the similarity of word vectors and label vectors since the model has been trained.
The public announcements information of a sequence
X i is stored in M i,j .
Since the public announcements change the decision of a TPK model, we can obtain them by comparing and analyzing the maximum values of elements in each row of M i,j .
And a public announcement can be a word or a phrase (comprise of several words) that changes model’s choices into a another direction based on M .
Theoretically, the logical knowledge inferred from datasets should match what is actually known.
However, there are some distinctions between logical knowledge and real knowledge in embedding space because of uncertainty and anisotropy.
To address this issue, CLK aggregates the logical knowledge with a contrastive term from the standpoint of distance, thus confining it to a space congruent with the real knowledge.
Given that CLK may shorten the gap between pieces of similar knowledge in the embedding space through contrastive learning, a contrastive objective L c is introduced to guide model training (Hadsell et al., 2006).
To make use of the correlation information of the word-label pairs and local information from base model, the contrastive logical knowledge
˜
q
i is computed by:
[Equation]
where w T and b are two trained parameters.
Let q i be the ground truth of X i , then the contrastive objective L c is given by:
[Equation]
margin ∈ [0 , 1] is a given parameter to control the strength of contrastive learning.
Through the objective function L c , the semantic knowledge of similar labels in embedding space should be as close as possible, while the different ones should be as far as possible.
As stated in (Wang & Isola, 2020), a good contrastive learning system possess both alignment and uniformity, while it may collapse when the information is extremely uneven.
To avoid this issue, we also introduce a cross entropy function L sim ( p i , ˜ q i ) =
( − (cid:80) ni
=1 ˜ q
i log p i ) ( p i ∈ p ) into the objective function.
Integrating L sim and L c into the loss computation, we jointly train the classifier as well as the contrastive logical knowledge, and the ultimate loss function is given by:
[Equation]
where e is a given parameter that control the degree of contrastive loss, larger e indicates that the model relies on contrastive learning more.
3. K NOWLEDGE REASONING
Based on the knowledge representation and contrastive logical knowledge learning, an interpretable structure M = ⟨ S, R, R ∗ , ρ, s 0 , π ⟩ is learned from the CLK-based deep learning models.
Specifically, for an arbitrary input sequence X v , a classical TPK model has a form of tree where each word denotes the time step.
With the logical knowledge embedd into models, explanations for sentiment analysis can be obtained based on the semantics of TPK.
For example, as shown in Figure 1 (c) , the world under state s 3 is non-deterministic since there are different states on the same time step, and there is a public announcement at sate s ′ 3 such that s ′ 3 ρs 3 , so the model will change it’s decision and go to another one.
Based on such interpretable structures of TPK, we can provide explanations for users.
Human-level knowledge dictates that explanations be logical, effective, and trustworthy; causal reasoning can probe these qualities in depth.
As a solution, we map a TPK tree to a structural causal model (SCM) (Shanmugam, 2001) to characterize the logic behind sentiment analysis tasks.
Based on the learned TPK model, we decompose each sequence into the two variables: simple word I and public announcement T .
Then we assume the following SCM:
[Equation]
where C is the confounder that influences I and T ; X is the input sequence influenced by I and
T , Y is the predicted label of X and U ∗ represents the unmeasured variable.
As shown in Figure 1 (d), SCM is represented intuitively by using a directed acyclic graph, where vertices are random variables, and directed edges represent the direct causality between these variables.
Following the Lewis’s notion of causality (Lewis, 1973; 1977; 1981), we verify the rationality and fidelity of explanations by generating counterfactual public announcements through a mathematical operator do ( t 0 ) interventions on learned public announcements.
For instance, let ρ be a public announcement of s 1 at state s n such that s n ρs ′ n and s 1 R ∗ s n and s 1 R ∗ s ′ n ; we can apply an intervention on variable T by fixing the value of T as t 0 , denoted as t := t 0 .
Thus the intervention t := t 0 blocks the influence of the original T on X . The explanation is considered to be effective and trustworthy iff ¬ T → ¬ X , since ¬ T → ¬ X means the SCM corresponding to the explanations is causally effective (Lewis, 1973; 1981).
E XPERIMENTS 4. B ASELINES
We adopt both glass-box models and black-box models as our baselines, where the glass models includ Linear Regression( LR ) (Molnar, 2022), K Nearest Neighbors ( KNN ) (Koech & Dombeu, 2021), Decision Tree ( DT ) (Zhu & Yang, 2016) and Explainable Boosting Machine ( EBM )
(Ntekouli et al., 2022).
Black-box baselines include BLSTM , BGRU and Bert , where BLSTM (Thireou & Reczko, 2007) and BGRU (Kim & Lee, 2017) are two widely used non-pretrained models for sentiment analysis; Bert is a pre-trained state-of-art model proposed in (Devlin et al., 2019), which has achieved high accuracy in many NLP tasks. For comparison, we also compare our method with other black-box explainability techniques such as Partial Dependence (Moosbauer et al., 2021),
LIME (Ribeiro et al., 2016) and SHAP (Kokalj et al., 2021).
Remark that, we understand that there are already many models achieving comparable performance combined with BLSTM (or BGRU), such as BLSTM-CNN (Shen et al., 2017) and BGRU-CNN (Wang et al., 2021).
And there are also other variants of pre-trained models, such as Roberta (Tan et al., 2022).
However, this paper aims to evaluate how interpretable methods influence base models (baselines), i.e., whether model performance is sacrificed when transparency is brought in.
Therefore, we only employ the original deep learning models as our base models, even though our proposed method can combine with those variants to form interpretable ones by replacing the DNN model (in Figure 1 (a)) with them.
4. D ATASETS AND E VALUATIONS
For model training and evaluation, we compiled four public sentiment analysis datasets including binary datasets and fine-grained datasets.
Specifically, IMDB 2 and Yelp-2 3 are binary datasets with 2 label categories.
Weibo-8 and Yelp-4 are fine-grained datasets with more than 3 label categories.
Note that Yelp-2 , Yelp-4 and Weibo-8 are also unbalanced datasets and the datasets description can be found in Appendix C.1.
As we mentioned in knowledge reasoning, it is important to measure not only how well a model performs but also how well it can be understood.
For model performance, we employ the accuracy as evaluation metrics.
For model interpretability, we employ evaluation metrics and causal reasoning to investigate it, where evaluation metrics are used to evaluate the degree of fit between the predicted labels by the model and the true labels in the datasets.
Following (Dzikovska et al., 2012; 2013), we use macro average F 1 -score as the primary evaluation metric since it is suitable for evaluating unbalanced class distributions (see Appendix C.1).
4. D ETAILS OF I MPLEMENTATION
In our experiments, we set batch size as 128.
For all input sequences, we set the maximum sequence lengths as 100 tokens.
We set embedding size as 100 for non-pretrained models. For pre-trained models, we set the embedding size as 100 for English datasets and 300 for Chinese datasets. We used BERT-tiny Turc et al. for English datasets and ALBERT Lan et al.
for Chinese datasets, which are both denoted as “Bert” in our experimental results.
All models are trained with Adam Optimizer (Kingma & Ba, 2015) on GTX 2080 Ti and RTX 3090.
We train our model’s parameters with the default learning rate of adam in Keras.
In our main experiments, we set the maximum epochs as 200, and we employ earlystopping to control model training with min delta = 1 e − 5 and patience = 6 (Graziotin & Abrahamsson, 2013).
In our main experiments, parameter for contrastive logical knowledge learning are set as e = 0 .
9 and margin = 0 . 1 or margin = 0 .
We run 5 times and use the averaged performance with variance given in Appendix C.1.
R ESULTS AND DISCUSSION
This part demonstrates how CLK performs on both binary sentiment analysis and fine-grained sentiment analysis, together with discusses the sensitivity of parameters and model interpretability.
5. A NALYSIS ON CLASSIFICATION RESULTS
[Table]
As shown in Table 1, our proposed CLK obtains quite similar or higher accuracy compared to the black-box baselines (BLSTM, BGRU and BERT), while offering a transparent view of the learning mechanism.
When compared to glass-box models, CLK outperforms baselines in binary sentiment analysis tasks (IMDB and Yelp-2), while demonstrates a lower accuracy in fine-grained analysis tasks (Weibo-8 and Yelp-4).
Due to the fact that Weibo-8 and Yelp-4 are fine-grained and unbalanced datasets, it is unreasonable to evaluate model performance solely based on accuracy. If readers pay close attention to the macro F 1 score, they may notice that CLK provides a significantly higher macro F 1 score than all glass-box models.
Given that all black-box baselines achieve lower accuracy and a significantly higher macro F 1 score in fine-grained analysis tasks than glass-box baselines, it is possible to conclude that glass-box models are more likely to classify samples as the class with high proportion in unbalanced datasets.
Comparing with the explainability techniques like LIME, such as SHAP and Partial Dependence (denoted as LIME ⋆ in Table 1), we find that such methods are unable to enhance model performance even though they bring transparency to black-box models.
In contrast, our method can boost model performance while improving model interpretability.
It is also worth noting that although the precision (P) of CLK is lower than the some black-box baselines on Yelp-2 and Weibo-8, the recall (R), F 1 -score and Acc results are always higher than baselines, which also demonstrates the advantages of CLK.
5. S ENSITIVITY ANALYSIS OF margin
To study the sensitivity of margin in L c (as stated in Equation 7), we apply our CLK in three deep learning models including BLSTM, BGRU and Bert.
Figure 2 (a)-(f) shows the sensitivity of margin in binary sentiment analysis datasets.
When margin > 0 .
5 , we observed that the performance of most CLK-based models rapidly declined, with the F 1 score dropping more quickly than the Acc.
Figure 2 (i)-(n) illustrates the sensitivity of margin in fine-grained sentiment analysis datasets.
The performance of CLK based models slightly fluctuate on Weibo-8 with F 1 score much lower that accuracy. As for Yelp-4 dataset, the performance of CLK based BLSTM (or BGRU) rapidly degrades when margin > 0 . 7 , while CLK-based Bert varies with the value of margin .
Note that macro F 1 -scores of all CLK based models are lower than Acc except for CLK-Based
BLSTM on Yelp-4.
Besides, despite the significance of contrastive learning, it should be weighted with a proper value for good performance.
5. A BLATION TEST
[Table]
where “+KA” indicates baselines equipped with knowledge acquisition, “+KL” indicates baselines equipped with knowledge learning, “+IE” indicates baselines equipped with random initial label embedding and “+CLK” means “+KA+KL”.
We conduct an ablation test on Weibo-8 dataset since Weibo8 is a fine-grained unbalanced sentiment analysis datatset, we mainly focus on the macro F 1 -score. During training stage, our method (CLK) mainly includes knowledge acquisition (denoted as KA) and knowledge learning (KL). We demonstrate the effect of each component. As shown in Table 2, BLSTM benefits a lot from adding knowledge extraction (KA), whereas BGRU and
BERT suffer when equipped with KA but not knowledge learning (KL). And the performance of all baselines drops when only equipped with the initial label embedding (IE) method. This phenomenon is triggered by the properties of the data; Weibo8 is a fine-grained dataset with some indistinguishable labels, such as happiness and like, disgust and sadness. In addition,
Weibo-8 is also an unbalanced dataset, making it more challenging to classify sequences into indistinguishable labels.
Readers paying close attention to Table 2 may notice that IE and KA reduce the performances of BERT to the same level, this is mainly due to the anisotropy and uncertainty of logical knowledge as we mentioned in Section 1, where the anisotropy is mainly produced when using pre-trained models (Cai et al., 2020; Ethayarajh, 2019).
What’s more, when equipped with knowledge learning, BGRU and BERT can be improved by a large margin, which indicates that ineffective knowledge not only does not improve the model, but also rather hinders its performance by clouding its decision-making.
Hence, contrastive knowledge learning is essential and effective when learning high-quality representations for indistinguishable labels with unbalanced distribution.
5. A CASE STUDY FOR INTERPRETABILITY THROUGH KNOWLEDGE REASONING
Interpretability analysis makes it possible for us to quickly and simply understand how the model arrives at its predictions.
In this case study, we demonstrate the interpretable result from a trained model (Bert+CLK) with a real positive example in Yelp-2 test data.
Example 1 Consider the sequence task 1001 =
“We went for lunch. The waitress took the bread basket from another table with dirty dishes on it and put it on our table. Open Grated cheese dish goes around from table to table . This is not sanitary.”
Question: Why the corresponding prediction is “negative” rather than “positive”?
Explanation: Because the “not sanitary” (public announcement) appears in the following words, so the model classifies this sequence into “negative”.
As shown in Figure 3, we can visualize the explanation of this example in the form of TPK based on the knowledge representation in Section 3.
In the diagram of TPK models, the arrow → represents successor relation R , double arrow ↠ represents the relation ρ , and s 0 is the root of the tree structure.
Let the proposition letter p denote that the sequence is recognized as “positive”, proposition letter q denote that the sequence is recognized as “negative”.
Then task 1003 is correctly classified iff ( s ′ 43 ⊨ q ) , where s i and s ′
i denote the states of i
th word in the sequence.
The interpretable results are shown in Figure 3, where s 40 ⊨ Y p .
At state s 42 , we recognise that the reason path with { s 0 , s 1 , s 3 , . . .
, s 42 } contains an incompatibility because it uses the choice of “positive” and this is incompatible with “not sanitary”.
Thus, there is a public announcement that s [-
 ρ 1 s-]42 ρs ′ 42 , so we have s 42 ⊨ ⊟ q and s ′
⊨ Y p .
Then, the reason path becomes { s 0 , s 1 , . . .
, s 42 , s ′ 42 , . . .
, s ′ 43 }[-. Similarly, there is another public announcement that s ′ 
 ρ 2 s ′′ 28 at state s ′ 28 with s ′ 28 ⊨ ⊟ p-] and ultimately the task 1003 is classified as “negative” with s ′ 43 ⊨ q .
From the perspective of human knowledge, it is clear that task 1003 is correctly classified, but it’s still unknown that whether the explanation based on the TPK model in Figure 3 is trustworthy. As mentioned in subsection 3.4, we investigate the fidelity of the explanation by counterfactual reasoning.
Here we generates a counterfactual public announcement for ρ through a mathematical operator do ( t 0 ) interventions on task 1003 .
Then we can obtain a counterfactual TPK model as shown in Figure 4, where s 43 ⊨ Y p and s [-′ 
-]1 ⊨ □ [-q . Moreover, there is only one public announcement ρ 1 such that s 
 ρ 1 s ′ 16-]p , and the corresponding reason path is { s 0 , s 1 , . . .
, s 43 } .
In this counterfactual case, task 1003 is classified as “negative”, which indicates that the public announcement ρ is causally effective since ¬ T → ¬ C .
Therefore, we can conclude that the model derives the prediction “negative” because [-there is a word “knock-offs” in task 1001 , which results in s ′ 
 ρ 2 s ′′ 28 ;-]the “not sanitary” (public announcement) appears; and the model would classify task 1003 as “positive” without the public announcement.
Based on above analysis, it’s possible to derive some intuitions from the point of model interpretability. In contrast to existing interpretable methods like LIME, SHAP, and PD, our method generates explanations by learning interpretable structures (TPK) from deep learning models. Besides, our method is proven to be causally successful by counterfactual reasoning, which is a key distinction between CLK and other interpretable methods like LIME.
C ONCLUSION
In this paper, we present a novel contrastive logical knowledge learning (CLK) method to learn interpretable TPK models and generate explanations for sentiment analysis tasks.
To the best of our knowledge, this is the first work that uses contrastive learning to optimize the learning of logical knowledge from datasets and trained models.
Our method achieves comparable performance while adding transparency and interpretability to deep learning models by leveraging the benefits of both logical knowledge and contrastive learning.
Empirically, we conducted extensive experiments to show the effectiveness of CLK, and our results are competitive both in binary sentiment analysis tasks and fine-grained sentiment analysis tasks.
The case study for model interpretability shows that
CLK can generate human-understandable explanations, and the knowledge reasoning of explanations demonstrates the rationality and fidelity of our method.
Our research aims to raise awareness of the potential of modal logic and contrastive learning in delivering interpretable sentiment analysis.
A R ELATED WORK
Our work adds transparency to deep learning models by learning interpretable TPK models and generating explanations for sentiment analysis tasks.
Additionally, our work extends the logic-based interpretable method and provides insight into the inner working of deep learning models beyond visualisation and is adaptable to a variety of explanatory settings.
In this section, we briefly review interpretable approaches for sentiment analysis tasks, contrastive learning and logic-based methods for providing explanations.
Interpretable sentiment analysis There have been numerous studies on sentiment analysis.
Early sentiment analysis mainly uses lexicon-based methods for text classification, which mainly depend on dictionaries and corpora (Ravi & Ravi, 2015).
Because of the language’s adaptability and lack of formality, it is difficult to construct a general and efficient rule applicable to all contexts.
Many works utilize machine learning and deep learning models to accomplish sentiment analysis, for example, Basiri et al.
(2021) uses a bidirectional neural network architecture for sentiment analysis,
Wei et al.
(2020) uses a BLSTM with multi-polarity orthogonal attention, and Yang et al.
(2016) uses a hierarchical attention network (HAN).
In recent years, with the development of pre-trained language model, Reimers & Gurevych (2019) utilizes the ELMo embeddings and Munikar et al.
(2019) utilizes BERT deep learning model to improve the performance of sentiment analysis tasks.
Although these models can effectively capture the semantic relationships between words in context, they lack interpretability for sentiment analysis.
Consequently, several studies have been conducted to investigate the interpretability of models for sentiment analysis.
Liu & Cocea (2017) proposes the fuzzy rule based systems, which can reduce the computational complexity and increase the interpretability.
Luo et al.
(2018) proposes a query-driven attention mechanism to discover different spotlights for queries from different aspects and provide explainable results.
Perikos et al.
(2021) introduces an efficient method based HMM for sentiment analysis, and Yadav et al.
(2021) provides an interpretable learning approach for the aspect-based sentiment analysis tasks.
Despite their successes, none of the existing work attempted to leverage deep learning models (particularly pretrained language models) with human-readable explanations for sentiment analysis tasks, achieving the state-of-the-art performance.
Contrastive learning.
Contrastive learning is a learning paradigm which aims to teach models to distinguish the targed sample from its corresponing negative samples.
This method has been extensively used in numerous research fields.
In the field of natural language process, numerous contrastive approaches have been utilized on sentence-level.
For example, Gao et al.
(2021) presents
SimCSE, a simple contrastive learning framework that greatly enhances state-of-the-art sentence embeddings on semantic textual similarity tasks.
Giorgi et al.
(2021) proposes a simple framework,
DeCLUTR, to shorten the distance between sentence embeddings in the same text and to widen the distance between sentence embeddings in different text.
Inspired by equivariant contrastive learning (Dangovski et al., 2021), the generalization of contrastive learning in computer vision, Chuang et al.
(2022) propoeses DiffCSE to learn the sentence embeddings between the original sentence and the edited sentence.
Unlike these previous works, this paper utilizes contrastive learning for the first time (to the best of our knowledge) to enhance the uniformity and reduce the uncertainty of the logical knowledge so as to make logical explanation consistent with real knowledge.
Logic-based interpretable methods.
There have been plenty of studies employing logic-based methods to provide explanations in explainable artificial intelligence domains.
For example, Ferreira et al.
(2022) develops a procedure to induce logic-based explanations for a given neural network model.
Liu & Lorini (2021) presents a modal language of a ceteris paribus nature which can provide explanations for binary input classifiers.
Barbiero et al.
(2022) proposes an end-to-end differentiable approach which can obtain logic explanations from neural networks using the formalism of First-Order Logic.
Although these work, the mechanism of deep learning models is still unknown.
To address this issue, (Chen et al., 2022) first investigates the working mechanism of
Gate BRNNs through Talmudic Public Announcement logic.
Enlightened by it, this paper employs
Talmudic Public Announcement logic to generate explanations for sentiment analysis, and further applies proposed method to pre-trained models (Bert) to get interpretable ones, which is missing from existing works.
B T HE DETAILED BACKGROUND OF T ALMUDIC PUBLIC ANNOUNCEMENT LOGIC
This part provides a general introduction of the kripke model in Modal logic as a preliminary knowledge, and then introduce the basic knowledge of TPK.
B. KRIPKE MODEL
Modal logic (Bull, 1979) is typically used to describe the systems which have the expression related to “necessarily” or “possibly”.
And it may be broadly used in belief, deontic, temporal and other systems.
Modal logic K is formulated in the language of classical logic with the added unary operator □ .
There are a range of different types of modal logics, all governed by a similar set of logical axioms and rules.
Kripke (1963) (Kaplan, 1966) defined semantics which can be applied over all the modal logics.
Here, we introduce the Kripke model for time action logic.
A Kripke model (or Kripke structure)
M for time action logic is a tuple ( S, R, Ω , h ) , where ( S, R, Ω) is a tree with root Ω ,
• S :
A set of possible worlds.
• R : The accessibility binary relation among the worlds, i.e. R ⊆ S × S .
We write s 1 Rs 2 to indicate that ( s 1 , s 2 ) ∈ R ; • h : An assignment giving for each atom q of the language a subset h ( q ) ∈ S .
Satisfaction is of the form t ⊨ A , where t ∈ S and A a wff and is defined recursively as follows:
t ⊨ q iff t ∈ h ( q ) for q atomic.
t ⊨ A ∧ B iff t ⊨ A and t ⊨ B . 3.
t ⊨ ¬
A iff t ̸| =
A . 4.
t ⊨
□ A iff for all s such that s is an immediate successor of t in the tree we have s |
=
A . 5.
A holds in the model ( S, R, Ω , h )
iff Ω | =
A .
So the Kripke model of time action logic could always be depicted as a tree structure using successor functions, which is called a time action model m =
( R , R, Ω) ,
• R = r 1 , r 2 , . . .
is a set of unary successor functions capable of opereting on Ω .
Thus t = Ω a 1 a 2 a 3 . . .
a n is the form of element sequence.
The elements of R could be regarded as actions a ∈ R moving the agent from any state t to a new state t a .
• R is the accessibility binary relation among the states.
tRs is hold iff for some a ∈ R we have s = t a • Ω is the initial state.
Time comes into the model if we take the view that time moves one unit when the agent perform any action.
So the states can also represent moments, i.e. Time 0, Time 1 ...and the paths represent how time moves.
There is always one root in the graph, which means the initial state, time 0.
The ordinary dynamic logics deal with actions upon states, whose possible results are clear cut.
However, in some certain cases, actions may depend on the future and therefore may be not clear cut at the present and need future clarifications.
Thus, a public announcement mechanism has been put forward in (Abraham et al., 2013).
B.1. PUBLIC ANNOUNCEMENT
To deal with the future clarifications, a set of public announcements P is defined in a Talmudic K frame(Abraham et al., 2013).
A Talmudic K frame has the form ( S, R , P ) where
• S is a non-empty set of possible worlds.
• R is a multi-valued accessibility relation of the form xRY x , where Y x = x 1 , . . .
, x n ( x ) , which means: one of x i in Y x is accessible to x
but it is not clear which one and need to await a public announcement clarification.
The set Y x = x 1 , . . . , x n ( x ) is unique in its R relation to x .
( S, R ) is a tree if for any y in S there is at most one x in S such that y is in Y x and x R Y x holds.
And define n ( x ) , for x in S the accessibility branching parameter of x . • P is a set of public announcements with the deterministic form: α = ( x, x 1 , ..., x n ( x ) , y ) , y ∈ x 1 , ..., x n ( x ) where ( x, x 1 , ..., x n ( x )
, y ) ∈ R means: it is hereby announced that y is the element accessible to x. While the public announcement can be non-deterministic if it chooses a subset Y of x 1 , ..., x n ( x ) .
It therefore has the form
α = ( x, x 1 , ..., x n ( x ) , Y ) , Y is a subset of x 1 , ..., x n ( x ) .
If Y is allowed to be empty, this means that it is announced that x has no accessible points.
The public announcement in Talmudic frame is a revision on the relation R , say, we have ( S, R , t ) and announce α , then we move to ( S, R α , t ) .
A certain α 1 not only expresses the public announce, but also tells where the current state is supposed to go to, and also give the information about what time and place the public announcement is made.
B. D EFINITION OF A TPK MODEL
Based on above knowledge, a Talmudic public announcement logic model is defined as shown in
Section 2.
Remark that, a deterministic TPK model is assumed to be deterministic because the public announcement ρ leads to at most one successor state when there is a public announcement, i.e., one state should be the unique clarified successor of the state where public announcement occurs.
To make it clear, we now give an example of deterministic TPK and non-deterministic TPK, respectively.
An example of non-deterministic TPK model is shown in shown in Figure 5.
Here the arrow → represents successor relation R , double arrow ↠ represents the relation ρ , and s 0 is the root of the tree structure.
• It is not a deterministic model, because ( s 1 ρy ′ ∧ s 1 ρx ∧ y ′ ̸
= x ) holds.
(or the same for s 2 ) • Some equivalence relation of distance could be raised, like D ( y )
=
D ( x ) , since we have xρy ∧ ¬ ( yRx )
∧ y ̸
= x . • t is publicly clarified at z 2 to z 1 , expressed by s 1 ↠ y ′ or s 2 ↠ y ′ or x ↠ y , since we have s 1 ρy ′ , s 2 ρy ′ , xρy , with z 1 Ry ′ , z
1 Ry .
• The double arrow ↠ also shows that where the announcement points are supposed to go to, i.e. s 2 or s 2 are supposed to go to y ′ when the public announcement at z 2 is made.
An example of deterministic TPK model is shown in shown in Figure 6.
Similarly, the arrow → represents successor relation R , double arrow ↠ represents the relation ρ , and s 0 is the root of the tree structure.
• It is a deterministic TPK model, because the public announcement ρ is deterministic and leads to at most one successor state, i.e. s ′ 3 ρs 3 .
• Some equivalence relation of distance could be raised, like D ( s 3 ) =
D ( s ′ 3 ) , since we have s ′ 3 ρs 3 ∧ ¬ ( s ′ 3 Rs 3 )
∧ s 3 ̸ = s ′ 3 .
• The double arrow ↠ also shows that where the announcement points are supposed to go to, i.e. s 1 is supposed to go to s 2 when the public announcement at s ′ 3 is made.
C T HE SUPPLEMENT OF EXPERIMENTS C. DATASET DESCRIBTION AND EVALUATION
For model training and evaluation, we compiled four public sentiment analysis datasets including binary datasets and fine-grained datasets. • IMDB 4 is a common used dataset for binary sentiment analysis containing 5,0000 available sequences, where “positive:negative=1:1”. • Yelp-2 5 is a review corpus obtained from Yelp website.
We process the raw data and extract 10937 available sequences with 2 label categories, where “positive:negative=9573:1364”.
• Weibo-8 is a fine-grained dataset provided by CCF for fine-grained sentiment analysis.
We extract available sequences with label categories, where “none:like:disgust:sadness:happiness:anger:surprise:fear= 8313:1224:1004:838:728:718:310:115”.
• Yelp-4 6 is a review corpus obtained from Yelp website.
We process the raw data and extract 10937 available sequences with 4 label categories, where “positive:slightly positive:slightly negative:negative= 2391:4132:3050:1364”. The division of training samples, validation samples, and test samples is shown in Table 3.
Macro average F 1 -score As shown in Table 3, Yelp-2, Yelp-4, and Weibo-8 are imbalanced datasets, especially Weibo-8.
[-- s 0 is the - a 1 = { getting a new token “day”, and recognizing the obtained sequence as “negative” } - s 1 = { “day”, p } - a 2 = { getting a new token “is”, and recognizing the obtained sequence as “negative” } - s 
 = { “day is”, p } - ... - a 15 = { getting a new token “movie”, and recognizing-]Therefore, appropriate metrics should be used to evaluate the [-obtained sequence as “negative” } - s 
-]performance of interpretable models.
Macro average F 1 -score is usually used to evaluate the model performance of unbalanced datasets, which is also widely used to evaluate the [-obtained sequence as “positive” } - s 
-]model performance
[Table]
[-{ “day is not a great bond movie , but it is a good bond movie,”, p } - a ′ 
-]in interpretable learning tasks (Fan et al., 2022; Dzikovska et al., 2012; 2013).
For a set of classes D ,each represented with N d instances in the [-obtained sequence as “positive” } - s ′ 
 = { “day is not a great bond movie , but it is a good bond movie,”, q } - s 16 ρ-]test samples, the Macro average F 1 -score is given by:
[Equation]
[-{ getting a new token “bond”, and recognizing the obtained sequence as “positive” } - s ′ 
 = { “day is not a great bond movie-]As mentions in (Grandini et al., 2020) , [-but it is a good bond movie, which still makes it much better than your typical bond”, q } - a ′ 
 = { getting a new token “knock-offs”, and recognizing-]since the [-obtained sequence as “positive” } - s ′ 
 = { “day is not a great bond movie , but it is a good bond movie, which still makes it much better than your typical bond knock-offs”, q } - a ′′ 
-]numerators of Macro Average Precision and Macro
Average Recall are formed of values in the range [0, 1], Macro-Average methods often compute an overall mean of different measures. There is no correlation between class size and the numerator weighting, which makes [-it much better than your typical bond knock-offs”, p } - s ′ 
-]classes of varying sizes equivalent.
This means that large classes are just as consequential as small ones.
The derived measure provides a class-based evaluation of the algorithm; a high Macro- F 1 value indicates that the algorithm performs well across all classes, while a low value indicates that some classes were poorly predicted.
Assume that all classes is equally important for users, it is possible to derive some intuitions from the equation. The macro average F 1 -score is harmonistic and can reflect the degree of fit between the predicted labels by the model and the true labels in the datasets, especially for imbalanced datasets. Thus, macro average F 1 -score is used to evaluate the rationality of interpretable models while the precision and recall are also reported in all experiments.
C. B INARY SENTIMENT ANALYSIS
Table 4 demonstrates the experimental results of non-pretrained models and Bert for binary sentiment analysis with variance given in the subscript.
[-Therefore, we can derive that s 
-]C. F INE - GRAINED SENTIMENT ANALYSIS
Table 5 demonstrates experimental results of non-pretrained models and Bert for fine-grained sentiment analysis with variance given in the subscripts.
D D ISCUSSION AND COMPARISON WITH OTHER LOGIC D. C OMPARING WITH BRANCHING TIME TEMPORAL LOGIC
TPK is different from the existing branching time temporal logic such as Computation Tree Logic (CTL) in essence (Heljanko et al., 1997).
On the one hand, temporal operators in CTL are restricted to the possible future paths from a given state, and some temporal operators in CTL, such as next, globally, and until, are not available in TPK.
It is worth noting that the true value of the formula in the current state in CTL will not be affected by future states, i.e., time cannot be reversed, nor can historical results be modified by returning from the future.
On the other hand, public announcement is the key of TPK, i.e., a current state depends on the future and therefore may be not clear cut at the present and need future clarifications.
So TPK and CTL aren’t directly comparable, but there are some common ideas between these two in terms of the representation of the past and future of a state. [Table] [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] D. C OMPARING WITH EPISTEMIC LOGIC
Talmudic logic treats states much like the quantum superposition of states and when there is a public announcement we get a collapse onto a pure state.
This is similar to how the epistemic model of the agent changes after a public announcement occurs in epistemic logic (EL) and dynamic epistemic logic (DEL) ( ˚Agotnes & Alechina, 2006).
One of the main purposes of the public announcement both in TPK and epistemic logic is to update the existing model and increase its certainty.
But this idea is implemented by different ways in two types of logical systems.
Besides, a public announcement in EL or DEL deletes some possible worlds after an announcement, while a public announcement in TPK deletes some accessibility links after an announcement.
Technically, these two approaches are similar but not equivalent.
E PROOF OF SOME EQUATIONS IN K NOWLEDGE ACQUISITION E. C ONJUGATION OF D IRICHLET DISTRIBUTION AND M ULTINOMIAL DISTRIBUTION
[Equation]
Likelihood function of multinomial distribution:
[Equation]
Thus:
[Equation]
We can write it as below:
[Equation]
E. PROOF OF E QUATION
The prior distribution of θ v is Dirichlet ( θ v | α ) as below:
[Equation]
Thus
[Equation]
the posterior distribution of θ v is given by:
[Equation]
Then the topic probability distribution of X v is given by:
[Equation]
Thus
[Equation]
the topic probability distribution is as below :
[Equation]
E. PROOF OF E QUATION
Similarly, the prior distribution of ϕ k is a Dirichlet distribution with prior parameter β
such that:
[Equation]
And, the posterior distribution of ϕ k is given by:
[Equation]
So we can obtain the word probability distribution of the k -th topic by:
[Equation]
Thus, when given the topic the word distribution is calculated by:
[Equation]
E. PROOF OF E QUATION
Finally, the joint probability distribution for topic-word is as follows:
[Equation]