Analyzing the Inner Workings of Transformers in Compositional Generalization (2025)

Abstract: The compositional generalization abilities of neural models have been sought after for human-like linguistic competence.The popular method to evaluate such abilities is to assess the models’ input-output behavior.However, that does not reveal the internal mechanisms, and the underlying competence of such models in compositional generalization remains unclear.To address this problem, we explore the inner workings of a Transformer model byfinding an existing subnetwork that contributes to the generalization performance and by performing causal analyses on how the model utilizes syntactic features.We find that the model depends on syntactic features to output the correct answer, but that the subnetwork with much better generalization performance than the whole model relies on a non-compositional algorithm in addition to the syntactic features.We also show that the subnetwork improves its generalization performance relatively slowly during the training compared to the in-distribution one, and the non-compositional solution is acquired in the early stages of the training.

## introduction compositional generalization, the ability to understand the meaning of a novel language expression based on the composition of known words and syntactic structures @xcite , is a crucial aspect for robustness against unseen language data. to assess the compositional generalization abilities of neural models, most existing studies have primarily focused on evaluating model outputs in compositional generalization benchmarks @xcite @xcite . however, the model outputs do not necessarily reflect the underlying competence because good performance in the benchmarks does not guarantee that the model implements a solution that generalizes based on compositional rules (i.e., compositional syntax) and vice versa. in addition, while a growing body of work on model interpretability has investigated the inner workings of transformer @xcite -based models @xcite , compositional generalization has rarely been the focus of such studies. @xcite and @xcite analyzed the internal mechanisms in compositional generalization but did not focus on the usage of syntactic features, which is the central part of compositional generalization. thus, the internal mechanisms of the model in compositional generalization are still unclear, and unveiling them would enhance the understanding of the model's competence. in this work, we analyze the inner workings of neural models to understand what type of syntactic features the models depend on in tasks requiring compositional generalization. our analysis method 1 consists of (i) identifying subnetworks within the model that perform well in the generalization and (ii) investigating how syntactic features causally affect the original model and its subnetwork, as shown in figure 1 . the causal analysis involves removing the linguistic concept of interest from the model and comparing the generalization performance before and after the removal. to rigorously evaluate and analyze the compositional generalization abilities, we focus not on pretrained models but on a transformer model trained from scratch; this is because pretraining data contain syntactic structures that should be unseen in this experiment, and pretrained models may not need to generalize compositionally @xcite . we also aim to conduct a detailed analysis by experimenting with various settings. this study employs two commonly used tasks for evaluating compositional generalization: machine translation and semantic parsing. we test the models with two patterns of compositional generalization, pp-iobj and pp-subj, which are similar yet different (see section 4.1 for details). the findings from our experiments suggest that the model and its subnetwork that contributes to the generalization performance indeed leverage syntactic structures. however, intriguingly, we also discover that the subnetwork implements other solutions that do not depend on syntactic structures. from these results, we argue that the solutions that the models employ are partly non-compositional. moreover, analysis of the model at different epochs during the training revealed that the model gradually develops a subnetwork with better generalization performance. the causal analysis of the subnetwork showed that the non-compositional solution was learned during the early phase of the training. we argue that transformer models need a better inductive bias to generalize compositionally utilizing syntactic features. ## analysis method figure 2 presents our method of analyzing the inner workings of a transformer model in compositional generalization. it consists of the following three phases: training a base model, subnetwork probing, and causal analysis. ## conclusion in this work, we investigated the inner mechanisms of a transformer model in compositional generalization tasks. the experimental results showed that the model utilizes syntactic features to some extent in the generalization but that its subnetwork with better generalization accuracy depends on non-syntactic features as well. this indicates that the model develops a non-compositional solution internally and fails to generalize compositionally even when the generalization performance is decent. this paper serves as a foundation for analyzing the underlying mechanisms in compositional generalization from the linguistic perspective. fu-ture work might consider other generalization patterns or other linguistic features to obtain a more profound insight into the linguistic competence of neural models.