DARE: Disentanglement-Augmented Rationale Extraction
Abstract
Rationale extraction can be considered as a straightforward method of improving the model explainability, where rationales are a subsequence of the original inputs, and can be extracted to support the prediction results.
Existing methods are mainly cascaded with the selector which extracts the rationale tokens, and the predictor which makes the prediction based on selected tokens.
Since previous works fail to fully exploit the original input, where the information of non-selected tokens is ignored, in this paper, we propose a Disentanglement-Augmented Rationale
Extraction (DARE) method, which encapsulates more information from the input to extract rationales.
Speciﬁcally, it ﬁrst disentangles the input into the rationale representations and the non-rationale ones, and then learns more comprehensive rationale representations for extracting by minimizing the mutual information (MI) between the two disentangled representations.
Besides, to improve the performance of MI minimization, we develop a new MI estimator by exploring existing MI estimation methods.
Extensive experimental results on three real-world datasets and simulation studies clearly validate the effectiveness of our proposed method.
Code is released at https://github.com/yuelinan/DARE .
Introduction
Although the performance of deep neural networks (DNNs) has signiﬁcantly improved across a range of natural language understanding tasks [43, 49, 50], the inability of DNNs to provide the explainability for their predictions still remains a serious risk.
To this end, several researchers [35, 26, 28] have focused on improving the explainability of DNNs.
Among them, the rationale extraction method [24, 4] can provide an intuitive explanation by identifying the important features of inputs.
Speciﬁcally, it extracts a short and coherent part of original inputs (i.e., the rationale ) as an explanation to support the prediction results when yielding them.
Traditional rationale extraction approaches [24, 4, 44, 33] cascade the selector and the predictor .
As shown in Figure 1(a), we illustrate this type of approaches with an example of the charge prediction, where charges are automatically predicted based on the case fact.
Speciﬁcally, the selector ﬁrst selects a subsequence of the fact description (i.e., the bolded tokens in Figure 1(a).
Then, the predictor yields the charge result based on the selected tokens.
And the extracted subsequence is deﬁned as the rationale.
However, since the extractive rationales only depend on the comparison between the prediction results and labels, this kind of approach may fail to fully exploit the information of the original input.
To solve that, Sha et al.
[37] add an external guider (i.e., Figure 1(b)) which takes the whole text as the input and generate the accurate but uninterpretable representations as most DNNs do to predict the result.
Then, the guider utilizes the above representations to guide the predictor to yield more comprehensive task-related representations with an adversarial-based method.
Since this method fails to utilize the information of the original text, where the non-rationale tokens are ignored, we argue that this “ guidance pattern ” can be further explored to improve the rationale extraction.
[Figure]
Along this research line, in this paper, we propose a “ selfguided ” method, D isentanglement- A ugmented R ationale
E xtraction (DARE), which augments traditional rationale extraction approaches with the disentangled representations learning.
Different from the previous model that requires external guidance, DARE aims to guide itself to extract more comprehensive rationales by squeezing more information from the input.
Speciﬁcally, we ﬁrst disentangle the original inputs into two parts: the rationale tokens and the nonrationale ones by the selector , where the two type tokens should be independent ideally.
Then, in the predictor , consisting of an encoder and a classiﬁer, these two type tokens are passed into a shared encoder to obtain the corresponding representations.
Finally, DARE utilizes rationale representations to yield results.
Meanwhile, the guider adopts non-rationale representations to guide the rationale ones to be more comprehensive by reducing the dependency between the two representations (i.e., minimizing their Mutual Information (MI)).
Besides, to improve the performance of minimizing MI, based on the state-of-the-art MI minimization method, Contrastive
Log-ratio Upper Bound (CLUB)
[11, 12], we propose a new implementation of CLUB (i.e., CLUB_NCE) by exploring the relationship between CLUB and InfoNCE (a classical maximizing MI method [32]).
Following the experimental setup of [34, 11], we evaluate the effectiveness of CLUB_NCE on two simulation studies by comparing with other MI minimization methods.
To validate the effectiveness of our proposed DARE, we perform extensive experiments on the
BeerAdvocate dataset [29] which is utilized to the multi-aspect sentiment analysis, a movie review dataset [48], and a legal judgment prediction dataset (CAIL2018)
[41] which is adopted to predict judgment results including charges, law articles and terms of penalty.
The experimental results empirically show that DARE has achieved better performance on both extracting rationales and predicting task results compared to other state-of-the-art rationale extraction methods.
Disentanglement-Augmented Rationale Extraction 2.1 Problem Deﬁnition
[Figure]
Given a text input x = { x 1 , x 2 , . . .
, x n } consisting of n tokens and the ground truth y , our goal is ﬁrst to learn a mask variable m
= { m 1 , m 2 , . .
, m n } , and then learn a model which adopts the rationale z
= m  x = { m 1 · x 1 , m 2 · x 2 , . . .
, m n · x n } to yield the prediction results.
Take the case in Figure 1 for example, based on the fact description x , DARE aims to yield the charge result y while extracting the rationale z to support this result.
Next, we will present the details of our proposed “ self-guided ” method, DARE, which consists of the selector , predictor and guider .
2.2 Architecture of DARE
To utilize the non-rationale representations to enhance the rationale ones, and further extract more accurate rationales, we propose the DARE method which is shown in Figure 2.
In DARE, we ﬁrst utilize the selector to select the rationale tokens and the non-rationale ones, and adopt the shared encoder to generate the corresponding representations.
Then, the classiﬁer in the predictor can yield the prediction results based on the rationale representations.
Meanwhile, the guider considered as a regularizer minimizes the MI between the above two types of representations to ensure they are explicitly disentangled, which can make the rationale representations more comprehensive.
2.2.1 Selector
To select the rationale tokens, the selector takes x = { x 1 , x 2 , . . .
, x n } as the input to generate a binary variable m
= { m 1 , m 2 , . .
, m n } , and adopts m to sample the rationale and non-rationale tokens.
Speciﬁcally, we ﬁrst utilize the selector to generate a probability distribution p ( m | x ) which represents the probability of sampling each x i as the part of the rationale.
To ensure this operation of sampling is differentiable, we adopt HardKuma reparameterization trick [4] to generate the p ( m | x ) : [Equation]
[Equation]
where u j is random sampled from the uniform distribution U (0 , 1) , F  1 Kuma ( u ; a, b ) =
1  (1  u ) 1 /b  1 /a is the inverse c.d.f.
of the Kumaraswamy distribution [23], a and b are parameterized by DNNs which takes x as the input, and r and l are ﬁxed hyperparameters.
Finally, we can derive the rationale tokens as z 1 =
m  x and the non-rationale ones as z 2 = (1  m )  x .
Predictor
The predictor q ( y | z 1 ) which consists of an encoder and a classiﬁer outputs the prediction results based on the rationale tokens z 1 .
Speciﬁcally, we ﬁrst re-embed the z 1 into continuous hidden states by the encoder to obtain the rationale representations v z 1 2 R d , where d is the hidden size.
Then, we pass the v z 1 into the classiﬁer to yield the results y which is predicted based exclusively on the rationale tokens extracted by selector , and the task loss function can be formulated as :
[Equation]
where p  is the neural network of selector .
2.2.3 Guider
While predicting results, to guide the model itself to extract more comprehensive rationales, we augment the disentangled representations learning (DRL) into the guider .
Speciﬁcally, we pass non-rationale tokens z 2 to the shared encoder described in section 2.2.2 to obtain the non-rationale representations v z 2 .
Then, we hope v z 1 and v z 2 should be independent ideally so that the v zcan contain rich rationale information.
To achieve this goal, we propose the mutual information minimization method which reduces the dependency between v z 1 and v z 2 to disentangle them (i.e., min I ( v z 1 ; v z 2 ) ).
Besides, following the application of MI in DRL [12, 36, 20], we should maximize
I ( v x ; v z 1 , v z 2 ) (a reconstruction loss) to ensure v z 1 and v z 2 encapsulate information from the input x sufﬁciently, where v x is the original input representations.
However, since v z 1 and v z 2 are obtained from the separate input, we argue that v z 1 and v z 2 have preserved the original input itself, and can remove this regularization.
In section 2.3, we will describe the MI minimization method in detail.
2.3 Mutual Information Minimization
In this section, we ﬁrst review some concepts of MI, and then present a new MI minimization method (i.e., CLUB_NCE) by discussing the relationships between InfoNCE and CLUB.
2.3.1 Mutual Information
In probability theory and information theory, the mutual information (MI) of two random variables (e.g., X and Y ) is a measure of the mutual dependence between the two variables
I ( X ; Y ) =
E p ( x,y ) h log p ( x,y ) p ( x ) p ( y ) i .
However, directly computing MI values is difﬁcult [5], especially when X and Y are continuous and high-dimensional.
To approximate MI, for MI maximization tasks, Oord et al.
[32] derive a lower-bound MI estimation (InfoNCE):
[Equation]
where { ( x i , y i ) } Ni =1 is a batch of sample pairs and f ( x, y ) is a learnable score function.
For MI minimization tasks, Cheng et al.
[11, 12] propose a Contrastive Log-ratio Upper Bound (CLUB) method:
[Equation]
where p ( y | x ) is a conditional distribution.
2.3.2 CLUB_NCE
Although p ( y | x ) in CLUB can be any neural network, intuitively, an easy way to parameterize p ( y | x ) is with an MLP.
In practical, p ( y | x ) is always parameterized by a Gaussian family 2 , where one potential reason for not employing MLP is that it is difﬁcult to converge in CLUB.
Therefore, it is signiﬁcantly to design a function that can parameterize p ( y | x ) with any network and be easy to converge.
To this end, we ﬁrst discuss the relationship between InfoNCE and CLUB.
Speciﬁcally, by applying the Jensen’s inequality in the Eq (3), we can get: I [Equation]
[Equation]
where we note this inequality is similar to Eq (4) and the main difference is the score function.
Interestingly, we ﬁnd for any critic f
( x, y ) in InfoNCE, the optimal critic is f ⇤ ( x, y )
=
log( p ( y | x )) [34] which is equivalent to the critic function in Eq (4).
for each training iteration do
Sample { ( x i , y i ) } Ni =1 from g  ( x, y ) .
Compute the InfoNCE loss as:
I nce = 1 N P
Ni =1 log exp( f ✓ ( x i ,y i )) N
P Nj =1 exp( f ✓ ( x i ,y j )) .
Update the f ✓ ( x, y ) by maximizing I nce .
for i = 1 ; i < =
N ; i + + do
U i =
f ✓ ( x i , y i )  1 N
P
Nj =1 f ✓ ( x i , x j ) .
end for
Update the g  ( x, y ) by minimizing :
I club = 1 N
P
{+Ni =
+} U i .
end for
Inspired by this observation, we combine CLUB with InfoNCE empirically and propose a new implementation of CLUB (i.e., CLUB_NCE) to minimize MI, which ﬁrst adopts the trained f ( x, y ) by InfoNCE to replace the log( p ( y | x )) in CLUB to calculate the value of I club , and then minimizes it to minimize MI.
We illustrate the process of MI Minimization with CLUB_NCE in Algorithm 1.
At each training iteration, we ﬁrst sample { ( x i , y i ) } Ni =1 from the generator g  ( x, y ) .
Then, we update the f ✓ ( x, y ) by maximizing I nce .
Next, the value of I club can be calculated as described in Eq (5) and g  ( x, y ) can be updated.
The neural network f ✓ and g  are trained alternately.
Speciﬁcally, in DARE, we set the shared encoder as the g  , and the guider which is parameterized with a full connection layer as the f ✓ .
2.4 Training
During training, following the setup of [4], we ﬁrst penalize L sh = 1 n
P nj =1 [1  p ( m j = 0 | x )] to ensure the model extracts short rationales, where p ( m j = 0 | x ) is the probability of the token not being selected.
Then, we add a regularizer to encourage coherence of selected tokens 3 :
[Equation]
By combining Eq (2), L sh , L re and the I club calculated by CLUB_NCE between v z 1 and v z 2 , our ﬁnal objective of DARE is deﬁned as [-L = λ 1 L task + λ 2 |L sh − l r | + λ 3 L re + λ 4 I club , where λand λ 4 are ﬁxed hyperparameters, λ 
-][Equation]
, where  1 and  4 are ﬁxed hyperparameters,  2 and  3 are Lagrangian multipliers which will be updated while training, and l r represents that how many tokens DARE will select.
[Table]
Experiments
In this section
[Table]
we ﬁrst compare DARE with some baselines on a beer reviews dataset, a movie reviews dataset, and a legal judgment prediction dataset.
Then, to demonstrate the effectiveness of CLUB_NCE
[Table]
we compare it with some classical MI minimization methods on both simulation studies and real-world datasets.
3.1 Beer Reviews Prediction
Beer Reviews Prediction is formulated as a multi-aspect sentiment analysis task, which predicts the ratings (on a scale of 0 to 5 stars) for multiple aspects (e.g., smell, palate) based on the beer reviews.
We use the BeerAdvocate [29] dataset containing more than 220,000 beer reviews as our dataset.
For the fair comparison, we replicate the pre-process of [24] and [37] where the dataset has been divided into three aspects including appearance, smell, and palate.
We normalize the ratings to [0 , 1] and adopt them as the ground truth for the regression.
For testing, we take 994 reviews for three aspects as our test set, where the aspect-related rationales are annotated by human.
3.1.1 Comparison Methods and Experimental Setup
• Bernoulli [24] generates rationales by yielding the Bernoulli distribution of each token.
• HardKuma [4] proposes a HardKuma distribution for reparameterized gradient estimates while selecting rationale tokens.
• InfoCal [37] introduces an adversarial-based method to guide the selector-predictor model to extract rationales.
Since InfoCal proposed a new LM regularizer to encouraging the consecutiveness of rationales, for a fair comparison, we adopt one of the versions of InfoCal (i
e., InfoCal (HK 4 ) in [37]), which applies the same regularizers L sh and L re to ensure the rationales are short and coherence.
• InfoCal_IB [37] adopts an Information Bottleneck regularizer to manage the trade-off between yielding short rationales and accurate results by removing the adversarial module in InfoCal.
We replicate the setup of [37] where the architecture of selector and encoder are RCNN and the loss function is the mean-squared error (MSE) loss.
Besides, we adopt the precision, recall, and F1-score to evaluate the performance of the selected rationales.
Among them, the precision is deﬁned as how many the selected tokens are in annotated rationales, and the recall represents how many annotated rationale tokens are selected.
For a fair comparison, we set the l r as {0.13, 0.07, 0.07} in three aspect datasets, respectively, and initialize the epoch as 50, which are both consistent with the previous methods [4, 37].
Several results of baselines in Table 1 are directly taken from [37].
3.1.2 Experimental Results
To demonstrate the effectiveness of DARE, we compare it with baselines on the beer review prediction task, and experimental results are shown in Table 1.
From the results, we can ﬁnd DARE achieves better performance on the three aspects of BeerAdvocate datasets, especially on recall and F1 scores, which demonstrates that DARE can select more comprehensive rationales.
Speciﬁcally, compared with the traditional selector - predictor models (i.e., Bernoulli, HardKuma and InfoCal_IB), DARE signiﬁcantly outperforms them, indicating the effectiveness of our method on utilizing the information of non-rationale tokens, where this type tokens are ignored in previous models.
As the selector and predictor in DARE are based on HardKuma, we can consider HardKuma is a variant of DARE, which removes the disentanglement operation.
Our model performs better than it, we can conclude that our disentanglement augmented method with MI minimization is effective.
Besides, we analyze the experimental results between DARE and InfoCal(HK) which is also a “guided” model but with an external guider (i.e., Figure 1(b)).
DARE does not outperform the
InfoCal(HK) on all metrics and some results are close (e.g., F1 in Palate prediction).
This is probably because InfoCal(HK) exploits the strength of the external guider , a black-box neural network that can yield more accurate but uninterpretable representations.
However, as a whole, InfoCal(HK) does not perform as well as our model, which demonstrates the effectiveness of squeezing the non-rationale information from input.
We also analyze the effectiveness of CLUB_NCE by comparing other MI minimization methods (e.g., CLUB [11] and L1Out [34]), and the results are shown in section 3.4.4.
In the section 2.2.3, we argue that the separable tokens have preserved the information of the original input and DARE does not need the reconstruction loss.
To validate this argument, we implement
DARE with the reconstruction loss by maximizing I ( v x ; v z 1 , v z 2 ) .
Among them, the token F1 scores are 82.6 , 61.4 and 69.8 for appearance, smell and palate, respectively, and so far the results are similar to DARE.
Although DARE with reconstruction loss improves on some metrics over DARE, the improvement is very limited, moreover, DARE with reconstruction loss increases the parameters of the model.
Therefore, we consider that the reconstruction loss is not necessary.
Since RCNN is used as the selector - predictor architecture, to illustrate DARE is agnostic of the model structure, we implement HardKuma and DARE with pretrained models (i.e., adopting BigBird [47] to replace RCNN) on the appearance aspect.
The token F1 of HardKuma is 84.4 and DARE is 87.5 , where DARE still outperforms HardKuma, demonstrating the effectiveness of the DARE structure.
3.2 Movie Reviews Prediction
[Table]
Besides the BeerAdvocate dataset, we also make experiments on a movie review dataset [48] of the ERASER benchmark [15], which also contains token-level human rationale annotations.
The movie reviews prediction task is formulated as a binary classiﬁcation, yielding the sentiment of movie reviews (i.e., positive or negative).
Table 2 gives results on this task.
From the observation, we ﬁnd DARE achieves the best performance on the three metrics, which further validates the effectiveness of DARE.
3.3 Legal Judgment Prediction
Legal Judgment Prediction (LJP) can be formulated as a text classiﬁcation problem, which adopts the case fact to yield the judgment results (i.e., charge, law articles, and terms of penalty).
We conduct our experiments on publicly available datasets of the C hinese AI and L aw challenge CAIL2018 [41].
CAIL2018 contains criminal cases which consists of fact description and corresponding charges, law articles, and terms of penalty results.
Following the data process of [46], we divide the terms into nonoverlapping intervals.
Then, the LJP task can be formulated as a multi-class classiﬁcation problem.
In addition to comparing the methods in the rationale extraction, we also compare with some classical baselines in LJP task including FLA [27], TopJudge [51], LADAN [42] and NeurJudge [46].
The above baselines are trained with a multi-task framework which exploits the dependence among the sub-tasks in LJP.
Besides, we conduct experiments on one of versions of CAIL2018 (denoted by
CAIL) which contains 134,739 cases [46].
We set the l r as {0.14, 0.14, 0.14} for training.
For evaluating, as the CAIL dataset does not contain annotations of rationales, we ﬁrst adopt the accuracy (Acc), macro-precision (MP), macro-recall (MR), and macro-F1 (F1) to evaluate the performance of yield judgment results.
Then, we provide a qualitative analysis on the rationale extraction.
Detailed description of comparison methods and experimental setups can be found in Appendix A.
3.3.1 Experimental Results
To evaluate the performance of our model on LJP, we show the experimental results from three aspects.
First, comparing with the traditional rationale extraction methods, we can ﬁnd DARE performs well [Table] [Table] [Table]
[Table]
DARE
[Table]
[Table]
[Table]
DARE
[Table]
[Figure]
[Table]
is the predicted rationales.
More examples can be found in Appendix B.1.
on LJP tasks, which further validates the effectiveness of our proposed model based on the previous beer reviews prediction experimental results.
Meanwhile, the analysis of the comparison between
CLUB_NCE and other methods of minimizing MI are shown in section 3.4.4.
Then, by comparing with methods in LJP (e.g., NeurJudge), we observe DARE still achieves good overall performance, second only to NeurJudge (the state-of-the-art model in LJP) in most metrics.
It is worth noting that these LJP methods are modeled based on the multi-task learning, where the charge, article and term of penalty prediction tasks are related and can enhance each other while
DARE is a single-task model (i.e., we need to train a separate model for each task).
It demonstrates that DARE can extract effective rationale tokens for prediction.
Besides, although these LJP methods achieve promising performance, the judgment results still remain unreliable and difﬁcult to explain.
In contrast, DARE can achieve a trade-off between the accuracy and explainability.
[Figure]
As there exist no annotations of rationales in
CAIL dataset, we provide a qualitative analysis on rationales extracted by DARE.
First, Figure 3 shows an example of the extracted rationale for the charge prediction with different methods, where the underlined tokens represent the practical rationales annotated by human judge.
From the observation, DARE can select key rationales which could be adopted to support the prediction results.
Besides, comparing with other extracted performance of baselines, DARE can select more rationale tokens, which indicates the strength of utilizing the non-rationale representations.
Then, to verify that DARE has effectively disentangled the textual input into the rationale representations and the non-rationale ones, we visualize these representations spaces using t-SNE [40] in
Figure 4.
Speciﬁcally, we visualize the same case (in Figure 3) predicted with HardKuma and DARE.
Figure 4(a) shows the visualization of HardKuma.
Among them, the blue circle selected represents the tokens that are predicted correctly by HardKuma; and the red circle selected denotes the tokens that are predicted incorrectly, and the position of the red circle is very close to the blue circle.
For
Figure 4(b), the blue circle selected represents the tokens that are predicted correctly by DARE; and the red circle selected denotes the tokens that are predicted incorrectly by HardKuma, where our
DARE identify that these tokens are non-rationale tokens.
From Figure 4(b), we can ﬁnd the tokens predicted incorrectly by HardKuma are accurately predicted to be non-rationale tokens.
It can be seen that after the disentanglement operation, these tokens can be obviously separated and be farther away from the tokens circled in blue.
Besides, we calculate the Euclidean(EU) distance between the blue circle and the red circle in Figure 4(a) and 4(b) for measuring their distance.
Among them, the
EU distance in Figure 4(a) is 7.62 , and in Figure 4(b) is 12.18 which is higher than it in Figure 4(a).
The above visualizations demonstrate that DARE can well disentangle rationale representations.
[Table]
Furthermore, we add a human evaluation to further evaluate extracted rationales in LJP.
We sampled 100 examples for the charge prediction task by comparing DARE with HardKuma and InfoCal(HK).
Speciﬁcally, following [37], we evaluate the rationales with three metrics: usefulness (U), completeness (C), and ﬂuency (F).
Among them, each scored from 1 (lowest) to(e.g.
2.0 and 4.9).
The scoring standard for human annotators can be found in Appendix A.3.
Besides, we ask people to evaluate the extracted rationales and the results are report in the table.
From the observation on Table 4, we can ﬁnd DARE outperforms HardKuma and
InfoCal(HK) in all metrics, illustrating the effectiveness of DARE.
3.4 MI Estimation Quality
In this section, we ﬁrst evaluate CLUB_NCE on simulation studies with two toy tasks.
Then, we show the performance of CLUB_NCE and other MI minimization methods on two real-world applications which described previously.
3.4.1 Simulated Studies
Here, we introduce the experimental simulated datasets and comparison methods.
First, following the setup from [34, 11], we experiment a dataset where x and y are sampled from a correlated
Gaussian distribution: x ⇠ N (0 , I d ) , y ⇠ N  ⇢ x,  1  ⇢ I d  , where ⇢ is the correlation and d is the dimensionality.
In the ﬁrst toy task (denoted by Gaussian ), we sample each dimension of ( x ; y ) from this simulated dataset, and the second task ( Cubic ) is the same as the ﬁrst while applying the transformation y !
y 3 (i.e., sample ( x ; y 3 ) from the same Gaussian distribution).
Meanwhile, the value of this mutual information is invariant (i.e., I ( x, y )
=
I ( x, y 3 ))
Under Gaussian distributions, the ground truth of MI is tractable and can be calculated as I ( x, y ) =
d 2 log  1  ⇢ [34].
For both of tasks, we set the dimension to d = 20 , hidden size to 128 and batch size to 64.
Besides, we set the initial MI value as 2 and increase it by 2 every 4k iterations, for a total of 20k iterations.
Then, we present the comparison methods including InfoNCE [32], CLUB [11], VarUB [1], and
L1Out [34].
Among them, CLUB and InfoNCE have been described in section 2.3.1, and VarUB ( I varub =
E p ( x,y ) h log p ( y | x ) r ( y ) i ) and L1Out ( I l 1 out = {+
+} N
P Ni = h log p ( y i | x i ) N  1 P j 6 = i p
( y i | x j ) i ) are both MI minimization methods which are adopted to estimate the upper bound of MI, where p ( y | x ) is commonly parameterized by a Gaussian family [11] and r ( y ) is ﬁxed as a standard normal distribution.
3.4.2 Simulated Studies
Here, we introduce the experimental simulated datasets and comparison methods.
First, following the setup from Poole et al.
[34] and Cheng et al.
[11], we experiment a dataset where x and y are sampled from a correlated Gaussian distribution: x ⇠ N (0 , I d ) , y ⇠ N  ⇢ x,  1  ⇢
I d  , where ⇢ is the correlation and d is the dimensionality.
In the ﬁrst toy task (denoted by Gaussian ), we sample each dimension of ( x ; y ) from this simulated dataset, and the second task ( Cubic ) is the same as the ﬁrst while applying the transformation y !
y 3 (i.e., sample ( x ; y 3 ) from the same Gaussian distribution).
Meanwhile, the value of this mutual information is invariant (i.e.,
I ( x, y )
=
I ( x, y 3 ))
Under Gaussian distributions, the ground truth of MI is tractable and can be calculated as I ( x, y ) =
d 2 log [Equation]  [34].
For both of the tasks, we set the dimension to d = 20 ,
[Figure]
hidden size to 128 and batch size to 64.
Besides, we set the initial MI value as 2 and increase it byevery 4k iterations, for a total of 20k iterations.
Then, we present the comparison methods including InfoNCE [32], CLUB [11], VarUB [1], and
L1Out [34].
Among them, CLUB and InfoNCE have been described in section 2.3.1, and VarUB ( I varub =
E p ( x,y ) h log p ( y | x ) r ( y ) i ) and L1Out ( I l 1 out = {+
+}
N P [Equation]
h log [Equation] [Figure] [Equation]
y [Equation] x j ) i ) are both MI minimization methods which are adopted to estimate the upper bound of MI, where p ( y | x ) is commonly parameterized by a Gaussian family [11] and r ( y ) is ﬁxed as a standard normal distribution.
3.4.3 Experimental Results on Simulated Studies
[Figure]
In this section, we mainly report the experimental analysis on the Gaussian task, the analysis on Cubic is similar and can be found in Appendix B.2.
Figure 5 shows the performance of our CLUB_NCE and other MI estimation approaches.
Among them, the joint critic is a single Multilayer Perceptron (MLP) and takes [ x ; y ] as the input, where “ ; ” represents the concatenate operation, and the separable critic is f ( x, y ) =
g 1 ( x ) T g 2 ( y ) , where g 1 and g 1 are two different MLPs.
The gaussian critic is parameterized by a Gaussian family (i.e.,
N  y | µ ( x ) ,  2 ( x ) · I  ), where the mean and variance are calculated by neural networks.
We can observe that the InfoNCE values are under the true MI while CLUB_NCE are above it, which illustrates CLUB_NCE is an upper bound of MI.
The estimation of VarUB has low value, indicating this method could not remain an upper bound in some cases.
The CLUB and L1Out has high bias and variance when the ground truth MI is high, whereas CLUB_NCE has the lower ones, especially the variance.
It might have beneﬁted from the strength of InfoNCE which has a low variance.
To further validate the inﬂuence of InfoNCE on CLUB_NCE, we ﬁrst analyze the relationship between them theoretically, and then conduct a comparison among the CLUB_NCE with different batch sizes.
Speciﬁcally, we review the InfoNCE shown in Eq (3):
[Equation]
where the ﬁrst term is less than
and I nce is upper bounded by log N [32].
Therefore, the performance of InfoNCE is inﬂuenced by batch sizes, where the larger the batch size N , the lower variance between the estimated MI and the true one [34].
In other words, since CLUB_NCE adopts the trained critic by
InfoNCE (Algorithm 1), the value of I club should have lower variance as the batch size increases.
Therefore, we conduct an experiment to estimate CLUB_NCE by adopting various batch sizes.
As shown in Figure 6, we calculate the variance between the estimated MI values I club and the theoretical
MI ones on the Gaussian task with different batch sizes.
From the observation, we can conclude that CLUB_NCE with larger batch size achieves the lower variance, which supports the previous theoretical analysis.
3.4.4 Real-world Applications
We compare CLUB_NCE with CLUB, L1Out and VarUB on beer reviews prediction and legal judgment prediction.
Speciﬁcally, we replace CLUB_NCE in DARE with other MI minimization methods, and denote them as DARE (CLUB), DARE (L1Out) and DARE (VarUB).
The experimental results are reported in Table 1 and Table 3.
Among them, as DARE (VarUB) fails to converge within epochs in the experimental setup, we do not present its result in the table.
From the observation,
DARE performs the best which indicates our CLUB_NCE is more effective than others methods at minimizing MI on real-world applications.
Besides, these variants of DARE fail to improve on some tasks (e.g., DARE (L1Out) in Palate prediction and Terms of Penalty prediction), and are even less effective than other selector - predictor models, illustrating it is crucial for DARE to choose a superior
MI estimator to effectively minimize the MI.
Related Work
Rationale Extraction.
Rationale extraction methods can provide the model explainability by extracting important features of inputs and many researchers [24, 4, 44, 8, 9, 39, 45] have been attracted.
Among them, Lei et al.
[24] proposed a classical framework for rationales extraction with a selector and predictor.
Following this framework, Bastings et al.
[4] introduced a HardKuma distribution for reparameterized gradient estimates and facilitated end-to-end differentiability of this framework.
Paranjape et al.
[33] and Chen et al.
[10] managed trade-off between extracting sparse rationales and generating accurate results by the information bottleneck theory.
Since this type of framework failed to exploit the information of the original text, several guider methods [37, 7, 19] have been proposed, which adopted the whole text to guide the rationale extraction.
Therein, Sha et al.
[37] introduced an adversarial-based technique to make the selector-predictor model learn from the guider model.
Cao et al.
[7] minimized the KL divergence between the two types models to reduce their difference.
Although extensive research has been carried out on the rationale extraction, few considered the information of non-rationale tokens containing rich information for extracting rationales.
Disentangled Representation Learning.
Disentangled representation learning (DRL) [6, 18, 16, 3] maps different aspects of data into independent latent vectors by adding constraints on the embedding space and has attracted much attention.
Recently, information theory has been widely adopted in the applications of DRL.
Among them, Sanchez et al.
[36], Cheng et al.
[12] and Huang et al.
[20] maximized MI between the original input and its different aspects to retain as much as possible information of the input.
And Cheng et al.[11] and Colombo et al.
[14] reduced the dependency among the different aspects of the input to make them independent by minimizing the MI.
Mutual Information.
Mutual information (MI) is adopted to measure the dependency between random variables.
Since computing the values of MI is challenging, and many researches utilized the
MI as a regularizer in loss functions, and not concerned with its precise value, recent works focused on MI maximization and minimization.
Nguyen et al.
[31] studied a lower bound of MI based on the MI f-divergence representation for maximizing MI.
Belghazi et al.
[5] proposed a mutual information estimator (MINE) which estimated the lower bound of MI.
Similarly, InfoNCE was introduced by Oord et al.
[32], which adopted the negative sampling estimation to maximize MI.
For
MI minimization, Poole et al.
[34] studied an upper bound of MI estimator L1Out based on reviewing the existing variational bounds on MI.
And a contrastive log-ratio upper bound (CLUB) was proposed by [12, 11] to minimize MI.
Besides, based on the KL divergence and Renyi divergence, Colombo et al.
[14] derived a MI estimator on the basis of a variational upper bound.
Conclusion
In this paper, we proposed a disentanglement-augmented rationale extraction method (DARE) which squeezed more information from the original input by exploiting the non-rationale tokens.
To be speciﬁc, DARE disentangled the textual input into two parts (i.e., the rationale representations and the non-rationale ones), and minimized the mutual information (MI) between the two disentanglements for extracting more comprehensive rationales.
Besides, to improve the performance of minimizing
MI, we introduced a new MI minimization method CLUB_NCE based on exploring the relationship between InfoNCE and CLUB.
Experimental results on three real-world datasets and a simulated dataset demonstrated the effectiveness of our proposed method
Acknowledgements.
This research was partially supported by grants from the National Key Research and Development Program of China (No.2021YFF0901003), the National Natural Science Foundation of China (Grants No.61922073 and U20A20229).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We discuss the limitations in Appendix C. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss the broader impacts in Appendix C. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please see Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] Please see Section 2. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Please see Table 1, Table 2 and Table 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We run all experiments on a single V100 GPU. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] Although the original BeerAdvocate has been removed by the authors of this dataset, we can obtain BeerAdvocate from http://people.csail.mit.edu/taolei/beer/ , which is published by [24].
CAIL2018 has been published in https://github.com/ china-ai-law-challenge/CAIL2018 .
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] To our best knowledge, their exists no personally identiﬁable information in BeerAdvocate, MovieReview and CAIL.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]