Explaining Interactions Between Text Spans (2023)

Abstract: Reasoning over spans of tokens from different parts of the input is essential for natural language understanding (NLU) tasks such as fact-checking (FC), machine reading comprehension (MRC) or natural language inference (NLI). However, existing highlight-based explanations primarily focus on identifying individual important features or interactions only between adjacent tokens or tuples of tokens. Most notably, there is a lack of annotations capturing the human decision-making process with respect to the necessary interactions for informed decision-making in such tasks. To bridge this gap, we introduce SpanEx, a multi-annotator dataset of human span interaction explanations for two NLU tasks: NLI and FC. We then investigate the decision-making processes of multiple fine-tuned large language models in terms of the employed connections between spans in separate parts of the input and compare them to the human reasoning processes. Finally, we present a novel community detection based unsupervised method to extract such interaction explanations. We make the code and the dataset available on [Github](https://github.com/copenlu/spanex). The dataset is also available on [Huggingface datasets](https://huggingface.co/datasets/copenlu/spanex).

## introduction large language models (llms) employed for natural language understanding (nlu) tasks are inherently opaque. this has necessitated the development of explanation methods to unveil their decision-making processes. highlight-based explanations @xcite are common: they produce importance scores for each token of the input, indicating its contribution to the model's prediction. in many nlu problems, however, the correct label depends on interactions between tokens from separate parts of the input. for instance, in fact checking (fc), we verify 54 novels novels, king stephen king 54 video games. ## model and human explanations agreement in §2, we discussed how the annotators modeled the spans and their interactions that led to the classification decision. we next investigate if fine-tuned llms use the same decision-making process by comparing the human annotations with two baselines. the random phrase baseline randomly samples both spans of the interaction from each of the two parts of the input. the part phrase baseline selects one span from the human annotations, and samples the other one at random from the remaining part. if the models follow the same reasoning as the annotators, the human explanations will have a significantly higher score than the baselines. however, if the annotated interactions are not important for the model, or only one part of the input is sufficient, we will see no such difference. ## extracting interactive explanations an explanation method should output interaction pairs of sets of tokens from the parts of the input p @xmath0, {x part2 j }, v⟩|@xmath1, |part1|], @xmath2, |part2|]}, where each pair is further assigned a significance value v depending on its influence on the prediction of the model. interactions between features, e.g., tokens, in ml models are most commonly learned with an attention mechanism @xcite . hence, we generate a directed bipartite interaction graph g @xmath3, e) with tokens from two parts of the input. the weights for the edges come from the attention matrix. we keep only the attention weights between tokens that belong to different parts of the input, thus, creating two vertex partitions. while we create the interaction graph using attention weights, it can be built using other explanation techniques producing token interaction scores. we use the top layer attention scores as they dictate how the final representation before the classnli-aopc-comp fever-aopc-comp 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 mean fever-refutes, snli-contradiction, antonym, low snli-aopc-comp fever-aopc-comp mean fever-supports, snli-entailment, synonym, low snli-aopc-comp fever-aopc-comp mean fever-nei, snli-neutral, hypernym-p1-p2, low disagreement 3-agreement 2-agreement figure 4: aopc-comp scores for the most relevant interactions split by inter-annotator agreement. in another approach, scalar mix, we freeze the parameters of the encoders and train new models with a set of parameters [λ 1 ..λ a ] on top of the frozen cls representations. the resulting linear classifier (w ′ , (l × m)) in this model uses the scalar mixed @xcite cls vector a @xmath4. argmax i (λ i ) determines the most important attention head. we use community structure detection algorithms @xcite on g i to find groups of nodes (tokens) with dense inter-group and sparse intra-group connections. these algorithms are computationally optimized for large social @xcite and biological networks @xcite and hence overcome the limitation of existing perturbation and simplification-based explanations that rely on the occlusion of groups of input tokens, which leads to a combinatorial explosion when considering span interactions. we use the louvain algorithm @xcite , see app. §d) which has been used in directed graphs such as ours. the bipartite nature of g i ensures that the explanation tokens are from two parts of the input, which are then combined to produce spans based on their positions. finally, a list of span pairs is generated by a cartesian product of the generated spans. the score for each span pair is the sum of the edge weights between the nodes in them. the ranked list of span pairs constitutes the explanation. ## related work existing work mainly explores interactions between tuples of tokens. @xcite propose a method to detect grouped pairwise token interactions, where the only interactions occur between tokens in the same group. @xcite creates attention attribution scores using integrated gradient (ig) over attention matrices and then constructs self-attention attribution trees. the latter is extended by @xcite to multiple layers. several approaches also extend highlight-based explanations to detect interactions between tuples of tokens. one line of work @xcite @xcite , introduces axioms and methods to obtain interaction shapley scores. janizek et al. (2021) extend ig by assuming that the ig value for a differentiable model is itself a differentiable function, thus, can be applied to itself. @xcite extend univariate explanations to produce bivariate shapley value explanations. additionally, @xcite find groups of correlated tokens from different input parts, but the tokens are found at arbitrary positions and the produced explanations are not necessarily semantically coherent. in contrast, we investigate interactions between token spans that bear sufficient meaning and are semantically coherent and thus plausible to end users. finally, there is a stream of work on explainability methods for constructing hierarchical interaction explanations. @xcite compute importance scores in a bottom-up manner starting from the individual embedding dimensions, working its way up to tokens, words, phrases, and finally the sentence. @xcite build interpretable interaction trees, where the interaction is again defined based on shapley values. while these methods produce spans of tokens that are part of an interaction, the hierarchical nature of the explanation limits the interactions only to neighboring spans. in contrast, we are interested in spans that can appear in the different parts of the input for nlu tasks and are not necessarily neighboring. ## conclusion we introduce spanex, a multi-annotator explanation dataset that captures the interactions between semantically coherent spans from different inputs in pairwise nlu tasks, here, nli and fc. spanex maps the implicit human decision-making process for these tasks to explicit lexical units and their interactions, opening up new research directions in explainability. using this dataset, we show that fine-tuned llms share the human inductive bias as evidenced by their relatively higher scores on established explainability metrics compared to random baselines. we also propose novel community detection-based methods to extract such explanations with modest success. we hope this work will pave the way for further research in the nascent area of interaction explanations.