I NTERPRETABLE S INGLE /M ULTI - LABEL T EXT C LASSIFICATION WITH U NSUPERVISED C ONSTITUENT - LABEL ALIGNMENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks based on layer-stacking architectures have historically suffered from poor inherent interpretability.
Meanwhile, symbolic probabilistic models function with clear interpretability, but how to combine them with neural networks to enhance their performance remains to be explored.
In this paper, we try to marry these two systems for text classification via a structured language model.
We propose a Symbolic-Neural model that can learn to explicitly predict class labels of text spans from a constituency tree without requiring any access to spanlevel gold labels. As the structured language model learns to predict constituency trees in a self-supervised manner, only raw texts and sentence-level labels are required as training data, which makes it essentially a general constituent-level self-interpretable classification model.
Our experiments demonstrate that our approach could achieve good prediction accuracy in downstream tasks. Meanwhile, the predicted span labels are consistent with human rationales to a certain degree.
I NTRODUCTION
Lack of interpretability is an intrinsic problem in deep neural networks based on layer-stacking for text classification.
Many methods have been proposed to provide posthoc explanations for neural networks (Lipton, 2018; Lundberg & Lee, 2017; Sundararajan et al., 2017).
However, these methods have multiple drawbacks.
First, there is only word-level attribution but no high-level attribution such as those over phrases and clauses.
Take sentiment analysis as an example, in addition to the ability to recognize the sentiment of sentences, an ideal interpretable model should be able to identify the sentiment and polarity reversal at the levels of words, phrases, and clauses.
Secondly, as argued by
Rudin (2019), models should be inherently interpretable rather than explained by a posthoc model.
A widely accepted property of natural languages is that ”the meaning of a whole is a function of the meanings of the parts and of the way they are syntactically combined” (Partee, 1995).
Compared with the sequential outputs of layer-stacked model architectures, syntactic tree structures naturally capture features of various levels because each node in a tree represents a constituent span.
Such a characteristic motivates us to think about whether the representations of these internal nodes could be leveraged to design an inherently constituent-level interpretable model.
One challenge faced by this idea is that traditional syntactic parsers require supervised training and have degraded performance on out-of-domain data.
Fortunately, with the development of structured language models (Tu et al., 2013; Maillard et al., 2017; Choi et al., 2018;
Kim et al., 2019), we are now able to learn hierarchical syntactic structures in an unsupervised manner from any raw text.
In this paper, we propose a general selfinterpretable text classification model that can learn to predict span-level labels unsupervisedly as shown in Figure 1.
Specifically, we propose a novel label extraction framework based on a simple inductive bias for inference.
During training, we maximize the probability summation of all potential trees whose extracted labels are consistent with a gold label set via dynamic programming with linear complexity.
By using a structured language model as the backbone, we are able to leverage the internal representations of constituent spans as symbolic interfaces, based on which we build transition functions for the dynamic programming algorithm.
The main contribution of this work is that we propose a Symbolic-Neural model, a simple but general model architecture for text classification, which has three advantages:
Our model has both competitive prediction accuracy and self-interpretability, whose rationales are explicitly reflected on the label probabilities of each constituent.
Our model can learn to predict span-level labels without requiring any access to span-level gold labels.
It handles both single-label and multi-label text classification tasks in a unified way instead of transferring the latter ones into binary classification problems (Read et al., 2011) in conventional methods.
To the best of our knowledge, we are the first to propose a general constituent-level self-interpretable classification model with good performance on downstream task performance. Our experiment shows that the span-level attribution is consistent with human rationales to a certain extent.
We argue such characteristics of our model could be valuable in various application scenarios like data mining, NLU systems, prediction explanation, etc, and we discuss some of them in our experiments.
P RELIMINARY 2. E SSENTIAL PROPERTIES OF STRUCTURED L ANGUAGE MODELS
Structured language models feature combining the powerful representation of neural networks with syntax structures.
Though many attempts have been made about structured language models (Kim et al., 2019; Drozdov et al., 2019; Shen et al., 2021), three prerequisites need to be met before a model is selected as the backbone of our method.
Firstly, it should have the ability to learn reasonable syntax structure in an unsupervised manner.
Secondly, it computes an intermediate representation for each constituency node.
Thirdly, it has a pretraining mechanism to improve representation performance.
Since Fast-R2D2 (Hu et al., 2022; 2021) satisfies all the above conditions and also has good inference speed, we choose Fast-R2D2 as our backbone
. 2. F AST -R2D
Overall, Fast-R2D2 is a type of structured language model that takes raw texts as input and outputs corresponding binary parsing trees along with node representations.
The representation e i,j representing a text span from the i th to the j th word is computed recursively from its child node representations via a shared composition function, i.e., e i,j = f ( e i,k , e k
+1 ,j ) , where k is the split point given by the parser and f ( · ) is an n-layered Transformer encoder.
When i =
j , e i,j is initialized as the embedding of the corresponding input token.
Please note the parser is trained in a self-supervised manner, so no human-annotated parsing trees are required.
S YMBOLIC -N EURAL M ODEL 3. L ABEL EXTRACTION FRAMEWORK FOR I NFERENCE
During inference, we first use Fast-R2D2 to produce a parsing tree, then predict the label of each node in the parse tree and output a final label set by the yield function introduced below.
Inductive bias.
Through observing cases in single/multi-label classification tasks, we propose an inductive bias that a constituent in a text corresponds to at most one label.
As constituents could be seen as nodes in a binary parsing tree, we can associate the nodes with labels.
Nodes with multiple labels could be achieved by assigning labels to non-overlapping child nodes.
Please note such an inductive bias is not applicable for special cases in which a minimal semantic constituent of a text is associated with multiple labels, e.g., the movie “Titanic” could be labeled with both ‘disaster’ and ‘love’.
However, we argue that such cases are rare because our inductive bias works well on most single/multi-label tasks as demonstrated in our experiments.
Label Tree.
A label tree is transferred from a parsing tree by associating each node with a label. A label tree example is illustrated in Figure 2.
During inference, we predict a probability distribution of labels for each node(to be formulated in Section 3.2) and pick the label with the highest probability.
Yield function.
We design a yield function that traverses a label tree in a top-down manner and gathers task-defined labels of nodes.
We divide the labels into two categories: terminal labels and non-terminal labels, which indicate whether the yield function should stop or continue respectively when it traverses to a node.
Considering some nodes may not be associated with any task-defined labels, we introduce empty labels denoted as ϕ T and ϕ NT for terminal and non-terminal ones respectively.
For simplicity, we don’t discuss nesting cases 1 in this paper, so there is only one unique nonterminal label which is ϕ NT and all task-defined labels are terminal labels.
However, our method can be naturally extended to handle nesting cases by allowing non-terminal labels to be associated with task labels.
The yield function is defined by the pseudo-code in Algorithm 1.
Figure 2 illustrates how the yield function traverses the label tree and gathers task-defined labels.
3. T RAINING OBJECTIVE .
During the training stage, though tree structures can be predicted by Fast-R2D2, the difficulty here is how to associate each node with a single label without span-level gold labels.
To constrain our model to predict label trees whose extracted labels are consistent with sentence-level gold labels, we define our training objective as follows:
Training objective Given a sentence S whose length is | S | and its gold label set T = { l 1 , ..., l m } , t is its best parsing tree given by the unsupervised parser of Fast-R2D2 and ˆ t is a label tree transferred from t . ˆ t [ C ] denotes ˆ t satisfying condition C .
For brevity, we use Y short for the yield function.
The training objective is to minimize − log P (ˆ t [ Y (ˆ t )
= T ] | t ) .
Before we get into the specifics, several key aspects are defined as follows:
(1) Denotations : t i,j denotes the subtree spanning from i to j (both indices are inclusive), whose root, left and right subtree are n i,j , t i,k and t k +1 ,j respectively in which k is the split point.
(2) Symbolic Interface : P ( l | n i,j ) is the probability of a single node n i,j being associated with the specified label l where P ( ·| n i,j ) = softmax(MLP( e i,j )) .
Thus, the probability of t transferring to a specific label tree ˆ t is the product of all the probabilities of nodes being associated with the corresponding labels in ˆ t . Obviously, it is intractable to exhaust all potential ˆ t to estimate P (ˆ t
[ Y (ˆ t )= T ] | t ) .
Our core idea is to leverage symbolic interfaces to estimate P (ˆ t [ C ] | t ) via dynamic programming.
We start with an elementary case: for a given label l , estimate P (ˆ t [ l ∈Y (ˆ t i,j )]
i,j | t i,j ) abbreviated as Y li,j . As the recursive formulation illustrated in Figure 3, we have:
[Equation]
However, for a given label set M , if we try to estimate P (ˆ t
[ Y (ˆ t i,j )=
M ] i,j | t i,j ) in the same way, we will inevitably exhaust all potential combinations as illustrated in Figure 4(a) which will lead to exponential complexity.
To tackle the problem of exponential complexity, we try to divide the problem of estimating P (ˆ t
[ Y (ˆ t )=
M ] i,j | t i,j ) to estimating Y li,j for each label l in M . Let F denote the set union of all the task labels and { ϕ T , ϕ NT } , O denote F \T .
By making conditional independence assumptions, we have:
[Equation]
We do not approximate P (ˆ t [ O∩Y (ˆ t )
̸
= ϕ ] | t ) as it could be computed directly.
The above function premises that multiple non-overlapping spans could associate with the same label.
In some cases, if there is a mutual-exclusiveness constraint that two non-overlapping spans are not allowed to associate with the same task label as shown in Figure 4(b), the function becomes:
[Equation]
P ( l | n [Equation] j ) if i = j (3) Regarding P (ˆ t [ O∩Y (ˆ t ) ̸
= ϕ ] | t ) , Y (ˆ t ) containing any label l ∈ O would satisfy the condition. We denote it as Y O i,j in short.
Similar to Equation 1 we have:
[Equation]
Thus P (ˆ t [ Y (ˆ t )= T ] | t ) = (cid:81) l ∈T Y l 1 , | S | · (1 − Y O 1 , | S | ) and the objective function given a parsing tree is:
[Equation]
Because it has been verified in the prior work (Hu et al., 2022) that models could achieve better downstream performance and domain-adaptivity by training along with the self-supervised objective of Fast-R2D2 denoted as L self , we design the final loss as follows:
[Equation]
where q ϕ ( z | S ) is the probability of a tree estimated by the parser in Fast-R2D2.
3. T OP - DOWN E NCODER
As recursive language models follow a bottom-up hierarchical encoding process, context outside a span is invisible to the span, which may make low-level short spans unable to predict correct labels because of a lack of information.
Here we draw on the ideas proposed by Le & Zuidema (2014);
Drozdov et al.
(2020) to allow information to flow in parse trees from top to down.
The overall idea is to construct a top-down process to fuse information from both inside and outside of spans.
For a given span ( i, j ) , we denote the top-down representation as e ′
i,j .
We use the Transformer as the top-down encoder function.
The top-down encoding process starts from the root and functions recursively on the child nodes.
For the root node, we have [ · , e ′ 1 , | S | ] = f
([ e root , e 1 , | S | ]) where e root is embedding of the special token [ROOT].
Once the top-down representation e ′
i,j is ready, we compute its child representations recursively via [ · , e ′ i,k , e ′ k
+1 ,j ] = f
([ e ′ i,j , e i,k , e k
+1 ,j ]) as illustrated in
Figure 5.
To enable the top-down encoder in the Symbolic-Neural model, we can simply replace e i,j with e
′ i,j in Sec 3.2.
E XPERIMENTS 4. D OWNSTREAM TASKS
In this section, we compare our interpretable symbolicNeural model with models based on uninterpretable dense sentence representation. All systems are trained on raw texts and sentence-level labels only.
Data set.
We report the results on the development set of the following datasets: SST-2, CoLA (Wang et al., 2019), ATIS (Hakkani-Tur et al., 2016), SNIPS (Coucke et al., 2018), StanfordLU
(Eric et al., 2017).
Please note that SST-2, CoLA, and SNIPS are single-label tasks and ATIS, StanfordLU are multi-label tasks.
There are three sub-fields in StanfordLU including navigator, scheduler, and weather.
Baselines.
To fairly compare our method with other systems, all backbones such as Fast-R2D and BERT (Devlin et al., 2019) are pretrained on the same corpus with the same vocabulary and epochs.
We record the best results of running with 4 different random seeds and report the mean of them.
Because of GPU resource limit and energy saving, we pretrain all models on Wiki-103 (Merity et al., 2017), which contains 110 million tokens 3 .
To compare our model with systems only using whole sentence representations, we include BERT and Fast-R2D2 using root representation in our baselines.
To study the reliability of the unsupervised parser, we include systems with a supervised parser Zhang et al.
(2020) that uses BERT or a tree encoder as the backbone.
For the former, we take the average pooling on representations of words in span (i,j) as the representation of the span.
For the latter, we use the pretrained R2D2 tree encoder as the backbone.
To compare with methods dealing with multi-instance learning (MIL) but without structure constraints, we extend the multi-instance learning framework proposed by Angelidis & Lapata (2018) to the multi-instance multi-label learning (MIMLL) scenario.
Please find the details about the MIL and MIMLL in Appendix A.6.
We also conduct ablation studies on systems with or without the top-down encoder and the mutual-exclusiveness constraint.
For the systems using root or [CLS] representations on multilabel tasks, outputs are followed by a sigmoid layer and filtered by a threshold that is tuned on the training set.
Hyperparameters.
Our BERT follows the setting in Devlin et al.
(2019), using 12-layer Transformers with 768-dimensional embeddings, 3,072-dimensional hidden layer representations, andattention heads.
The setting of Fast-R2D2 follows Hu et al.
Specifically, the tree encoder uses 4-layer Transformers with other hyper-parameters same as BERT and the top-down encoder uses 2-layer ones.
The top-down parser uses a 4-layer bidirectional LSTM with 128-dimensional embeddings and 256-dimensional hidden layers.
We train all the systems across the seven datasets for 20 epochs with a learning rate of 5 × 10 − 5 for the encoder, 1 × 10 − 2 for the unsupervised parser, and batch size 64 on 8 A100 GPUs.
[Table]
in Appendix A.1.
Results and discussion.
We make several observations from Table 1.
Firstly, We find that our models overall achieve competitive prediction accuracy compared with strong baselines including
BERT, especially on multi-label tasks.
The result validates the rationality of our label-constituent association inductive bias.
The significant gap compared to MIMLL fully demonstrates the superiority of building hierarchical relationships between spans in the model.
Secondly, when using sentence representation, the models with the unsupervised parser achieve similar results to those with the supervised parser on most tasks but significantly outperform the latter on CoLA.
A possible reason for the poor performance of the latter systems on CoLA is that there are many sentences with grammar errors in the dataset which are not covered by the training set of the supervised parser.
While the unsupervised parser can adapt to those sentences as L bilm and L KL are included in the final loss.
The result reflects the flexibility and adaptability of using unsupervised parsers.
Thirdly, ‘parser+TreeEnc.’
in Symbolic-Neural architectures does not perform as well as ‘parser+TreeEnc.’
using sentence representation, while the systems using the unsupervised parser show opposite results.
Considering that the Symbolic-Neural model relies heavily on the representation of inner constituents, we suppose such results ascribe to the tree encoder having adapted to the trees given by the unsupervised parser during the pretraining stage of Fast-R2D2, which leads to the self-consistent intermediate representations.
This result also verifies the structured language model that learns latent tree structures unsupervisedly is mature enough to be the backbone of our method.
4. I NTERPRETABILITY & C ONSISTENCY WITH HUMAN RATIONALES
To evaluate the consistency of the span labels learned by our model with human rationales, we design a constituent-level attribution task.
Specifically, we hide the gold span positions in NER and slot-filling datasets and gather span labels as sentence-level labels set. So only raw text and sentence-level gold labels are visible to models.
We then train models as multi-label classification tasks and evaluate span positions learned unsupervisedly by models. Though we argue that our model is self-interpretable, we still conduct an analysis on interpretability in Appendix A.4.
Data set.
We report F1 scores on the following data sets: ATIS (Hakkani-Tur et al., 2016),
MITRestaurant (Liu et al., 2013a) and MITMovie (Liu et al., 2013b).
ATIS is a slot-filling task and the others are NER tasks.
Baselines.
We include two baselines with attribution ability on multi-label tasks: integratedgradient(IG) (Sundararajan et al., 2017) and multi-instance learning (Angelidis & Lapata, 2018).
We follow the setup in Sec 4.1 and report the results of the last epoch.
For IG, we set the interpolation steps as 200 and use the same BERT in the last section as the encoder, filter the attribution of each token by a threshold and select filtered positions as outputs. We use zero vectors and [MASK] embeddings as the baselines for IG as Bastings et al.
(2021) find the latter one could significantly improve its performance.
Considering IG scores not having explicit meaning, we allow IG to adjust thresholds according to the test datasets.
We report the best results of both baselines and corresponding thresholds.
Please find the full version of the table in Appendix A.3.
For MIMLL, we select the span with the max attention score for a specified label.
Please find details in Appendix A.6.
Metrics.
We denote the predicted span set as P and gold span set as G and the overlap of P and G with the same labels as O .
Then we have:
[Equation]
[-Slot-filling sls-movie-eng length all 1 −2 −3 −all 1 −
-]
[Table]
∗
[Table]
[Table]
[Table]
[Table]
Results and discussion.
From Table 3, one observation is that models with the mutualexclusiveness constraint achieve better F1 scores.
Such results illustrate that a stronger inductive bias is more helpful for models to learn constituent-label alignments.
Besides, we find the NeuralSymbolic models significantly outperform the MIMLL and IG baselines on the NER datasets but trail the IG on the slot-filling task.
Through studying the outputs of our method, with a sample shown in Figure [Table] we find that our model tends to recall long spans while the ground truths in ATIS tend to be short spans.
We also find that on sls-movie-trivial, MIMLL significantly outperforms IG.
So we hypothesize that the distribution of golden span lengths may affect results.
We divide sentences into buckets according to the average golden span length and compute F1 scores for each bucket, as shown in Table 3.
Interestingly, we find that the scores of IG decline significantly with increasing span lengths, while our method performs well on all the buckets.
In addition, we argue that the F scores on the NER datasets can reflect interpretability more objectively, because the boundaries of proper nouns are clear and objective, while the choice of slots is relatively ambiguous about whether to include prepositions, modal verbs, etc.
4. C ASE S TUDY & P OTENTIAL A PPLICATIONS
We output the label trees generated by our model trained on the Navigator, SST-2, and CoLA to observe whether the model has sufficient interpretability.
From Figure 7 we can find our method is able to learn potential alignments of intents and texts and show them explicitly.
This can be used in multi-intent NLU systems to help determine the attribution of slots to corresponding intents.
We also study the difference between generated label trees of the vanilla Symbolic-Neural model and the Symbolic-Neural topdown .
The cases could be found in Appendix A.9.
We find the vanilla Symbolic-Neural model fails to deal with multi-intent cases.
Such an observation verifies the necessity of introducing the top-down encoder.
For SST-2, as there are no neutral samples, we randomly sampled sentences from Wiki-103 as neutral texts and force all nodes to be ϕ NT by the mean squared error loss.
Figure 8(a) shows the sentiment polarity of each constituent and the polarity reversal of “never”.
Such a characteristic could be used for text mining by gathering the minimal spans of a specified label.
We also study the generated label trees on CoLA, a linguistic acceptance data set.
We transfer the task to a grammar error detection problem by converting the label “1” to ϕ as “1” means no error is found in a sentence.
Figure 8(b) shows it’s able to detect incomplete constituents and may help in applications like grammar error location.
More cases could be found in the Appendix.
R ELATED WORKS
Structured language models.
Many attempts have been made to develop structured language models.
Pollack (1990) proposed to use RvNN as a recursive architecture to encode text hierarchically, and Socher et al.
(2013) showed the effectiveness of RvNNs with gold trees for sentiment analysis.
However, both approaches require annotated trees.
Gumbel-Tree-LSTMs (Choi et al., 2018) construct trees by recursively selecting two terminal nodes to merge and learning composition probabilities via downstream tasks.
CRvNN (Chowdhury & Caragea, 2021) makes the entire process end-to-end differentiable and parallel by introducing a continuous relaxation.
However, neither Gumbel-Tree-LSTMs nor CRvNN mentions the pretraining mechanism in their work.
URNNG (Kim et al., 2019) proposed the first architecture to jointly pretrain a parser and an encoder based on RNNG (Dyer et al., 2016).
However, its O ( n 3 ) time and space complexity makes it hard to pretrain on large-scale corpora.
ON-LSTM and StructFormer (Shen et al., 2019; 2021) propose a series of methods to integrate structures into LSTM or Transformer by masking information in differentiable ways.
As the encoding process is still performed in layer-stacking models, there are no intermediate representations for tree nodes.
Maillard et al.
(2017) propose an alternative approach, based on a differentiable CKY encoding.
The algorithm is differentiable by using a softgating approach, which approximates discrete candidate selection by a probabilistic mixture of the constituents available in a given cell of the chart.
While their work relies on annotated downstream tasks to learn structures,Drozdov et al.
(2019) propose a novel auto-encoder-like pretraining objective based on the inside-outside algorithm Baker (1979); Casacuberta (1994) but is still of cubic complexity.
To tackle the O ( n 3 ) limitation of CKY encoding, Hu et al.
(2021) propose an MLMlike pretraining objective and a pruning strategy, which reduces the complexity of encoding to linear and makes the model possible to pretrain on large-scale corpora.
Multi-Instance Learning.
Multi-Instance learning (MIL) deals with problems where labels are associated with groups of instances or bags (spans in our case), while instance labels are unobserved.
The goal is either to label bags Keeler et al.
; Dietterich et al.
; Maron & Ratan (1998) or to simultaneously infer bag and instance labels Zhou et al.
; Kotzias et al.
Angelidis & Lapata (2018) apply MIL to segment-level sentiment analysis based on an attention-based scoring method.
In our work, we refine instances to different semantic granularities and consider hierarchical relationships between instances.
Model Interpretability.
In the line of work on model interpretability, many approaches have been proposed.
Ribeiro et al.
; Lundberg & Lee (2017) try to generate explanation for prediction.
Baehrens et al.
; Simonyan et al.
; Sundararajan et al.
(2017) analyze attribution by gradients.
The above-mentioned methods are all posthoc.
Kim et al.
(2020); Cao et al.
Chen & Ji (2020) apply masks on the model input in text classification to obtain token weights, but single-dimensional weights are not enough to reflect multi-label interpretation.
Alvarez-Melis & Jaakkola (2018); Rudin (2019) argue interpretability should be an inherent property of a deep neural network and propose corresponding model architectures.
However, all the above-mentioned methods are not able to generate constituent-level interpretability.
C ONCLUSION , L IMITATION AND F UTURE WORK
In this paper, we propose a novel label extraction framework based on a simple inductive bias and model single/multi-label text classification in a unified way.
We discuss how to build a probabilistic model to maximize the valid potential label trees by leveraging the internal representations of a structured language model as symbolic interfaces.
Our experiment results show our method achieves inherent interpretability on various granularities.
The generated label trees could have potential values in various unsupervised tasks requiring constituent-level outputs.
Regarding to the limitation of our work, we require that the labels corresponding to the texts in the dataset have a certain degree of diversity, thus forcing the model to learn self-consistent constituentlabel alignments.
For example, in ATIS, almost all training samples have the same labels like “fromloc.city name” and “toloc.city name”.
That’s why our model fails to accurately associate these two labels with correct spans in Figure 6.
In the future, we will explore more unsupervised application scenarios, such as text-mining combined with VAE (Kingma & Welling, 2014) and image-text alignment tasks, to make the architecture more versatile.
R EPRODUCIBILITY S TATEMENT
In the supplemental, we include a zip file containing our code and datasets downloading linkage.
We’ve also included in the supplemental the scripts we run all baselines and the Symbolic-Neural models.
A A PPENDIX A. D YNAMIC P ROGRAMMING BASED ON FULL PERMUTATION
A naive way to estimate P (ˆ t [ Y ( M )] i,j | t i,j ) is to enumerate all possible state spaces and sum them up via dynamic programming.
We use X M i,j short for P (ˆ t [ Y ( M )]
i,j | t i,j ) .
Let M l and M r denote a pair of sets subject to M l ∪ M r = M .
Let C ( M ) denote the set containing all valid M l and M r pairs.
Figure 4(a) discusses all potential combinations of M l and M r when |M| > 1 .
If |M| > 1 , let C ( M ) be the set of all potential pairs
where Y (ˆ t
i,k ) ∪ Y (ˆ t k +1 ,j ) =
M .
If |M| = 1 , it’s similar to the case described in Figure 3.
If M = ϕ , n i,j could only be associated with ϕ T or ϕ NT with M l = ϕ and M r = ϕ .
Finally the transition function for t i,j where i < j is:
[Equation]
When i =
j , we have:
[Equation]
The transition function works in a bottom-up manner and iterates all possible M ⊆ T .
X T 1 , | S | is the final probability.
Even though, iterating C ( M ) and all M ⊆ T is of exponential complexity, so it only works when |T | is small.
A. P RETRAIN BERT AND F AST -R2D2 FROM SCRATCH
The dataset WikiBooks originally used to train BERT (Devlin et al., 2019)
is a combination of English Wikipedia and BooksCorpus (Zhu et al., 2015).
However, BooksCorpus is no longer publicly available.
So it’s hard to pretrain Fast-R2D2 on the same corpus, making it impossible to compare fairly with the publicly available BERT model.
Considering the limited GPU resources, we pretrain both BERT and Fast-R2D2 from scratch on Wiki-103.
We train BERT from scratch following the tutorial by Huggingface 4 with the masked rate set to 15%.
The vocabulary of BERT and Fast-R2D is kept the same as the original BERT.
As demonstrated in RoBERTa (Liu et al., 2019) that the NSP task is harmful and longer sentence is helpful to improve performance in downstream tasks, we remove the NSP task and use the original corpus that is not split into sentences as inputs.
For FastR2D2, WikiText103 is split at the sentence level, and sentences longer than 200 after tokenization are discarded (about 0.04‰ of the original data).
BERT is pretrained for 60 epochs with a learning rate of 5 × 10 − 5 and batch size 50 per GPU on 8 A100 GPUs.
Fast-R2D2 is pretrained with learning rate of 5 × 10 − 5 for the transformer encoder and 1 × 10 − 3 for the parser.
Please note that the batch size of Fast-R2D2 is dynamically adjusted to ensure the total length of sentences in a batch won’t exceed a certain maximum threshold, to make the batch size similar to that of BERT, the maximum threshold is set to 1536.
Because the average sentence length is around 30 for Wiki103, the average batch size of Fast-R2D2 is around 50 which is similar to that of BERT.
[Table]
[Table]
[Table]
A NALYSIS OF I NTERPRETABILITY .
Bastings et al.
(2021) propose a method that “poisons” a classification dataset with synthetic shortcuts, trains classifiers on the poisoned data, and then tests if a given interpretability method can pick up on the shortcut.
Setup.
Following the work, we define two shortcuts with four continuous tokens to access the faithfulness of predicted span labels: #0#1#2#3 and #4#5#6#7 indicate label 1 and 0 respectively.
We select SST2 and CoLA as the training sets, with additional 20% synthetic data.
We create a synthetic example by (1) randomly sampling an instance from the source data, (2) inserting the continuous tokens at random positions, and (3) setting the label as the shortcut prescribes.
Verification steps.
The model trained on the synthesis data could achieve 100% accuracy on the synthetic test data and the model trained on the original dataset achieves around 50% on the synthetic test set.
Sorting tokens.
Since our model does not produce a heatmap for input tokens, it lacks an intuitive way to get top K tokens as required in the shortcut method.
So we propose a simple heuristic tree-based ranking algorithm.
Specifically, for a given label, we start from the root denoted as n and compare P ( l | n left ) and P ( l | n right ) where n left and n right are its left and right children.
If
P ( l | n left ) > P ( l | n right ) , all descendants of the left node are ordered before the descendants of the right child, and vice versa.
By recursively ranking according to the above rule, we could have all tokens ranked.
We additionally report the precision of shortcut span labels in the predicted label trees.
A shortcut span label is correct only if the continuous shortcut tokens are covered by the same span and the predicted label is consistent with the shortcut label.
Results.
From the table 3, we have an interesting finding that the precision declines with the increase of training epochs.
We think the reason for this phenomenon is that shortcut spans are the easiest to learn at early epochs, so almost all top tokens are short-cut tokens.
With continuous training, the model gradually learns the semantics of texts from the original data.
Although the
[Table]
label for a sentence in the synthetic data is random, there is still around 50% probability that it is semantically consistent with the text and hence the label probability of a certain span may exceed the probability of the shortcut span.
Please note the precision of shortcut span labels predicted by our model is 100%.
Such results demonstrate again that our model is self-interpretable and could reflect the model’s rationales by span labels.
Samples of label trees with shortcut tokens are shown as follows:         .
title  better far a  been   ve ’ would  turkey
#   cold the journey   #  #   #  #     affecting often  and charming  a   s ’ it
A
M ULTI -L ABEL L EARNING BASED ON F AST -R2D
We adopt a canonical multi-instance learning framework used in text classification proposed by
Angelidis & Lapata (2018), in which each instance has a representation and all instances are fused by attention.
The original work produces hidden vectors
h i for each segment by GRU modules and computes attention weights a i as the normalized similarity of each h i with h a . [Equation]
[Equation]
where C is the total class number, p i is the individual segment label prediction, p d is document level predictions.
They use the negative log-likelihood of the prediction as an objective function:
L cls = − (cid:80) d log p ( y d ) d .
We simply replace segment representations with span representations in our work as the experiment baseline.
Specifically, we use the top-down representation e
′ i,j as the tensor to be attended to and predict the label by e i,j :
[Equation]
(11) where D is the span set for a parsing tree.
Please note the MIL model in our baselines is trained together with L bilm and L KL , whose final loss is L cls + L self .
A. M ULTI -L ABEL M ULTI -I NSTANCE L EARNING BASED ON F AST -R2D
To support multi-label multi-instance learning, we refactor the above equations to enable them to support attention on different labels.
For each label there is vector h ( c ) a , based on which attention weights a ( c ) i is computed as following:
[Equation]
The final objective function is L = − (cid:80) c
∈T log p ( c ) − (cid:80) c
∈F\T log(1 − p ( c ) ) .
In the semi-supervised slot-filling and NER tasks, we let the model predicts labels first and then pick the span with the max attention weight for each label.
A. R EAL L ABEL T REES SAMPLED FROM S YMBOLIC -N EURAL − t/ − e ON SST-
A.
R EAL L ABEL T REES SAMPLED FROM S YMBOLIC -N EURAL − t/ − e ON C O LA
T T T
T T T . T ##por
T stu
T a into  us talked
T
T professor
T the
T .
us talked
T
T professor
T the
T
T
T
T
T .
T
T monster
T a into
T him made
T they
T
T
T .
T exhaustion
T to
T him made
T they
[Figure]
[Figure]
S AMPLED L ABEL T REES IN ATIS
We sample label trees from Neural-Symbolic − t/ − e and Neural-Symbolic + t/ − e respectively for ob- servation.
Ground truths are annotated in brackets.
A.
S AMPLED L ABEL T REES IN N AVIGATOR 2:request route, 3:appreciate, 4:request address, 6:navigate