Feature Interactions Reveal Linguistic Structure in Language Models (2023)

Abstract: We study feature interactions in the context of feature attribution methods for post-hoc interpretability. In interpretability research, getting to grips with feature interactions is increasingly recognised as an important challenge, because interacting features are key to the success of neural networks. Feature interactions allow a model to build up hierarchical representations for its input, and might provide an ideal starting point for the investigation into linguistic structure in language models. However, uncovering the exact role that these interactions play is also difficult, and a diverse range of interaction attribution methods has been proposed. In this paper, we focus on the question which of these methods most faithfully reflects the inner workings of the target models. We work out a grey box methodology, in which we train models to perfection on a formal language classification task, using PCFGs. We show that under specific configurations, some methods are indeed able to uncover the grammatical rules acquired by a model. Based on these findings we extend our evaluation to a case study on language models, providing novel insights into the linguistic structure that these models have acquired.

## introduction feature attribution methods (fams) are a popular family of tools for explaining the behaviour of deep learning models, by explaining a prediction in terms of contributions of individual features @xcite . there are many such methods proposed, and mathematical results (such as axiomatic approaches based on game theory) and theoretical frameworks (such as @xcite 's 'explaining by removing') are starting to offer a good understanding of how different methods relate to one another. however, there are also some important shortcomings. perhaps most importantly, popular fams mostly ignore the existence of interactions between the effects of features on the prediction. this is problematic, because feature interactions are widely seen as a major factor in the success of neural networks @xcite . this is all the more important in domains such as language and music processing, because feature interactions allow neural networks to model hierarchical representations of their input, which is considered a key design feature of language and music. to address these shortcomings, there is now an emerging literature on feature interaction detection and attribution methods (fidams) that explain model predictions in terms of interacting features @xcite . however, assessing the faithfulness of fidams is even more challenging than assessing the faithfulness of feature attribution methods more generally @xcite . in this paper, we present a systematic framework to characterise fi-dams, and derive several new fidams based on that framework. we then proceed with creating an evaluation pipeline that measures a fidam's ability to recover the structural rules for which we have good evidence that they play an important role in the target model's performance (figure 1 ). we first test this on a set of small-scale formal language tasks, that provide stronger faithfulness guarantees. finally, we present a case study of a large language model on the cola task for linguistic acceptability. we find that the performance of fidams is very variable, and that the performance on the smallscale formal language tasks may not be predictive of the performance of methods on the large-scale natural language task. this is an illustration of what we call the attribution generalisation problem. we argue that this problem remains a key open problem in the study of explanation methods in general. pcfg @xmath0... [[[]]( * ][]] []( * [(()))) [()()[] * ) [ * [(()())]] ... 4. evaluate 1. generate language ## a framework for characterising fidams feature attribution methods typically decompose a model prediction into a sum of feature contributions @xcite . a large contribution then indicates that this feature played an important role in a model's prediction. although feature attributions can provide meaningful insights into the inner model dynamics, they paint a fairly limited picture of the model be-haviour. most importantly, interactions between features are lumped together, making it impossible to discern whether a large contribution of a feature stemmed from that feature alone, or from its interaction with neighbouring features. to address this, multiple methods have been proposed that decompose a model prediction into a sum of feature interactions, based on similar mathematical formalism as those of feature attributions. notation a neural network is represented as a single function f . the input to f is denoted as x, which consists of n input features. a partial input x s only consists of input features s ⊆ n . a value function v(x s ) quantifies the model output on the partial input x s . padding the missing features in x s with replacement features x ′ \s is denoted as x s ∪ x ′ \s . the attribution value of feature i is denoted as ϕ i , and the interaction effect of a set of features i is denoted as γ i . attribution dimensions attribution methods can generally be characterised along two dimensions @xcite : 1) how the method deals with feature removal, and 2) how the impact of removing a feature is quantified. fidams are built on the same principles as fams, and can be categorised along the same two dimension. by discerning these two dimensions we can separately evaluate their impact on the faithfulness of the attribution method. furthermore, we can combine feature removal procedures with influence quantification methods in order to obtain novel attribution methods, an observation that has also been made in the context of fidams by @xcite , who, concurrent to our work, provide a general framework for characterising fi-dams. ## fidams we now address a series of interaction methods that we use in our own experiments. ## evaluating fidams the final component of our framework is a methodology for evaluating the faithfulness of fidams. to lay a robust foundation for such work, we propose to evaluate a range of interaction methods and baselines on smaller deep learning models (using lstm and transformer architectures) that have been trained to recognise formal languages, based on a probabilistic context-free grammar (pcfg). our models are trained on a binary language classification task, in which a model needs to learn to discern between well-formed strings and minimally corrupted counterparts. models are trained to perfection (100% accuracy) on both train and test set. to obtain perfect performance, a model must rely solely on the grammatical rules that underlie the language, without resorting to spurious heuristics, because only these results allow completely solving the task. this way, due to the controlled nature of the task, we obtain a high degree of confidence about the model's behaviour. the goal of our experimental approach is to recover the structure of the language based on the trained model itself. this is achieved by the fi-dams outlined in §4. we aim to uncover whether a structural dependency between two features results in a high interaction effect. since our models have been trained to perfection, this allows us to employ our setup as a way of measuring the faithfulness of a fidam. a method that assigns a high interaction effect to features that contain a dependency in the original grammar is able to provide a faithful reflection of a model's understanding of the task. by testing a wide range of fidams and baselines we can uncover which configuration yields the most faithful explanations. a graphical overview of our approach is depicted in figure 1 . task the binary language classification task is set up by generating positive examples d + , based on some pcfg, and negative examples d -, derived from minimally corrupting the positive examples. we split the union of these two sets into a random train/test split of 80/20%. we train our models with a default cross-entropy loss, using the adamw optimiser @xcite , a learning rate of 0.01, and a batch size of 48. models our pipeline permits the use of any kind of neural model architecture, in our experiments we considered both lstms @xcite and transformers @xcite . in our experiments we report the results of the lstm model, but we observed similar results for transformers: due to the black-box approach of our explanation procedure the architecture itself is not of great importance. the models are deliberately small: we use an embedding size that is equal to the number of symbols in the language it is trained on, a hidden state size of 20, and a single layer. this results in models that provide a compute-friendly test bed for evaluating the fidams. ## experiments on formal languages we apply the evaluation procedure of §5 to two formal languages: the identity rule language and the dyck-2 language. in the appendix ( §a) we also present results on a palindrome language. ## a natural language case study: cola as a case study on a larger scale natural language task, we apply our methodology to language models fine-tuned on the cola task @xcite . cola is part of the glue benchmark @xcite , and is defined as a binary classification task of determining the linguistic acceptability of a single input sentence. the task consists of linguistically valid sentences, and sentences that contain either a syntactic, semantic, or morphological violation. a model that performs well on this task must have a thorough grasp of grammatical structure, and as such it provides a useful test bed for our fidam evaluation procedure. in the previous experiments there was a degree of certainty about the structure that must be encoded by the model. in the natural language domain, however, we do not have such certainty, and should therefore be careful of making strong claims about faithfulness. furthermore, natural language is highly multi-faceted and can not be captured by a single hierarchical structure that covers all these facets. nonetheless, we consider it valuable to test our setup on a natural domain in order to see if interesting differences between fidams arise, and whether particular facets of language such as syntactic dependency structure can be extracted. ## discussion & conclusions in this paper, we have presented a framework for characterising fidams and evaluating their faithfulness. for the characterisation we set out two dimensions, feature removal and feature influence, along which existing fidams can be characterised, by extending the 'explaining by removing' framework of @xcite to also apply to fidams. this allows us to place each of the known fidams in a two-dimensional grid, and to define novel variants of these models. as such, many of the methods that we incorporated in our experiments are novel fidams, such as combining archipelago with expected explanations and stii with an observational baseline. to assess the faithfulness of fidams, we made use of formal language theory and 'grey box mod-els'. we use formal grammars to generate multiple datasets, each with known feature interactions, and train deep learning models to perfection on those datasets. using fidams, we can then extract the learned feature interactions based on the model itself, and compare these interactions to the dependencies in the original grammar. we demonstrate that only specific combinations of fidams and baselines are able to retrieve the correct interactions, while methods such as archipelago and integrated hessians consistently fail to do so. finally, we tested our methodology on a natural language case study using a model fine-tuned on the cola task for linguistic acceptability. our results on the formal language tasks either did not turn out to be predictive of this experiment or, alternatively, the results were predictive but the lms made less use of dependency graph information than we might have expected. this illustrates the challenge of the attribution generalisation problem, and the open question remains how we can transfer faithfulness guarantees from a synthetic, controlled context to the domain of natural language and llms. we do show, however, that under certain configurations feature interactions align to some degree with the (syntactic) dependency structure of a sentence. this paves the way for revealing linguistic structure in a more direct way than, for instance, can be achieved with structural probes @xcite . investigating whether different methods and baseline configurations are able to retrieve different aspects of structure is an exciting next step that we look forward to exploring in more detail. this could be examined, for instance, through the lens of contrastive explanations @xcite , a procedure that demonstrates that different baselines can reveal different aspects of linguistic structure. furthermore, investigating the role that attention plays in modelling interactions could be a fruitful line of work, for instance by incorporating context mixing methods to our pipeline, such as value zeroing @xcite and alti @xcite .