Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT (2023)

Abstract: The goal of this paper is to explore how Transformer language models process semantic knowledge, especially regarding the plausibility of noun-verb relations. First, I demonstrate GPT2 exhibits a higher degree of similarity with humans in plausibility processing compared to other Transformer language models. Next, I delve into how knowledge of plausibility is contained within attention heads of GPT2 and how these heads causally contribute to GPT2’s plausibility processing ability. Through several experiments, it was found that: i) GPT2 has a number of attention heads that detect plausible noun-verb relationships; ii) these heads collectively contribute to the Transformer’s ability to process plausibility, albeit to varying degrees; and iii) attention heads’ individual performance in detecting plausibility does not necessarily correlate with how much they contribute to GPT2’s plausibility processing ability.

## introduction transformers are attention-based neural network models @xcite , which have brought breakthroughs in the field of natural language processing achieving state-of-the-art performance in diverse downstream tasks. such great performance is thought to be attributed to transformers' ability to build dependencies even between longdistant words which attention heads are developed for @xcite . to be specific, unlike previous neural network language models (e.g., simple neural networks or recurrent neural networks) that have issues retaining linguistic information coming from distant tokens, attention heads in transformers enable to represent the meaning of tokens by integrating their contextual information without losing information from distant tokens @xcite . provided that transformer language models consist of multiple attention heads that serve different roles, previous studies examined functions that individual attention heads serve and how language processing work is divided inside transformers @xcite @xcite . however, previous studies mostly focused on finding attention heads that process linguistic knowledge intrinsic to language systems such as morphosyntactic rules, and little attention has been paid to semantic knowledge, which requires much of world knowledge going beyond rules in language systems. consequently, we only have limited knowledge of how attention heads contribute to transformers' general ability to process semantic knowledge. a number of studies @xcite @xcite @xcite @xcite @xcite examined how transformers process semantic knowledge in comparison with humans, but their focus was mostly on the models' performance from the final hidden state without answering where the specific type of knowledge is preserved or processed in transformer models. a few studies started investigating how world knowledge is stored in transformers (e.g., @xcite examined how gpt stores factual associations). however, the previous findings are yet generalizable to a broad range of semantic knowledge, and thus more studies are needed to understand how transformers process other types of semantic knowledge. in this regard, the present study aims to advance our knowledge of semantic knowledge processing in transformer language models by closely investigating individual attention heads' ability in processing semantic plausibility and their causal contribution to transformer's performance in plausibility processing. among various types of plausibility, the especial focus of this paper is on the plausible relation between nouns and verbs. while recognizing the importance of considering a broader array of semantic knowledge in future studies, i made this specific choice because the objectives of the present paper are to demonstrate a set of attention heads can be specialized for specific type of semantic knowledge and to introduce a set of analyses that can be used to probe attention heads' role in processing semantic knowledge. the semantic plausibility of the relationship between nouns and verbs can be determined by the degree to which semantic features of nouns and verbs match, as shown in sentences in (1) from @xcite . for instance, in (1a), the syntactic dependent (plate) of the verb (shattered) has a feature [+shatterable], which builds a plausible relation with the verb (shattered). in (1b), however, the syntactic dependent letter does not have a feature @xcite , and thus it is semantically implausible dependent of the verb (shattered). in order to examine how such knowledge is preserved and processed inside transformer-based language models, this paper answers the following questions: (i) how similar are transformer's plausibility processing patterns to humans'?; (ii) how sensitive is each of the attention heads in transformers to plausibility relation?; and (iii) how do these heads make causal effects on transformers' ability to process semantic plausibility? after comparing patterns in plausibility processing between a group of transformer-based language models and humans, it was found that gpt2 tends to process the plausibility between nouns and verbs in a way that is more similar to humans than other language model types. several followup experiments that especially focus on gpt2 answered the last two questions. specifically, it was uncovered that gpt2 has a set of attention heads that detect semantic plausibility, which are relatively diffusely distributed from the bottom layers to the top layers and that they exert causal effects on transformers' semantic plausibility processing ability. gpt2's plausibility processing ability almost disappeared when the plausibility-processing attention heads are pruned, but the effects of removing a plausibility-processing attention head was not balanced nor proportional to the attention heads' performance in detecting plausible nouns. rather, it was found that a single attention head accounts for most of plausibility processing ability of gpt2. in what follows, i will provide a background that relates to the questions i address in this paper. in section 3, i will compare transformer-based language models' and humans' sensitivity to the plausibility of the relation between nouns and verbs. in section 4, i will conduct an experiment to find attention heads that can detect semantic plausibility knowledge and examine how they are distributed inside the model. in section 5, it will be examined how individual attention heads collectively make causal effects of on transformers' sensitivity to plausibility. in section 6, i will summarize the results and discuss the limitations of the study. ## background what roles do attention heads serve? there have been a lot of studies that attempted to explain the language processing mechanism in transformers with analyzing functions that distinct attention heads serve @xcite @xcite . specifically, @xcite found attention heads specialized for a position, syntactic relation, rare words detection; @xcite found attention heads specialized in part-of-speech and syntactic dependency; @xcite found attention heads specialized in coreference resolution; and jo and myaeng (2020) examined how linguistic properties at the sentence level (e.g., length of sentence, depth of syntactic trees and etc.) are processed in attention heads. despite numerous attempts in examining the roles of attention heads, the focus has been mostly on linguistic knowledge intrinsic to language systems which does not require much world knowledge that is indispensable for semantic knowledge processing. thus, it needs to be closely examined how transformers preserve and process such knowledge that facilitates sentence processing. how do we learn attention heads are specialized for certain linguistic knowledge? in previous studies, attention heads are considered to be able to process a certain type of linguistic knowledge if attention distribution patterns in the attention heads are consistent with the linguistic knowledge @xcite @xcite . however, such regional analysis does not explain how much contribution attention heads make to transformers' ability to process linguistic knowledge because such information from the attention heads may fade away or be lumped along with the information flows -from bottom layers to top layers -eventually making little contribution to transformers' ability to process the linguistic knowledge. thus, to rigorously confirm the role of attention heads in processing a certain type of knowledge, it is crucial to analyze the causal effects that they make on transformer's ability to process linguistic information @xcite @xcite . in this sense, this paper will not only examine which attention heads can form attention distributions that are consistent with semantic plausibility knowledge, but also examine how much influence the attention heads can exert on transformers' general ability to process plausibility. ## comparison between humans and transformer language models in plausibility processing patterns this section examines how a set of transformer language models process plausibility of noun-verb relations in comparison with human data. ## plausibility processing attention heads in gpt2 this section will examine whether gpt2 has a specific set of attention heads that can sensitively detect plausiblity of noun-verb relations, irrespective of syntactic dependency relation. experimental stimuli were the same as previous experiment. ## causal effects of plausibility-processing attention heads on gpt2's plausibility sensitivity in the previous experiment, attention heads capable of detecting plausible relations between nouns and verbs were found. the present section examines how such attention heads make causal influence on gpt2's sensitivity to plausibility between nouns and verbs. in particular, i attempt to answer two questions: (i) how gpt2's responses to plausible/implausible verb-noun pairs change when plausibility-processing attention heads are removed? and (ii) how does gpt2's plausibilitysensitivity change as attention heads are gradually pruned? ## conclusion & limitations the present study has shown how semantic plausibility is processed in transformer language models, especially focusing on the role of attention heads. first, i demonstrated that gpt2, whose decoderonly architecture is more aligned with the way humans process sentences, shows greater similarity to humans in plausibility processing compared to other transformer-based models such as bert, roberta and albert. then, a series of experiments showed a set of attention heads are found to process plausibility, and those heads are diffusely distributed across 12 layers in gpt2. moreover, it was observed that they make imbalanced but collective causal contributions to gpt2' plausibilityprocessing ability, which establishes the importance of causal effect analysis in attention-headprobing studies. although the results provide a window into how transformers process semantic knowledge of plausibility, this study has a few limitations to be addressed in future studies. first, the scope of the study is restricted to the plausibility of noun-verb relations although there exist many different types of semantic knowledge. this limitation stems from the present paper's intention to 'initiate' an exploration of transformers' attention heads in handling of semantic knowledge and to exploit diverse and robust techniques for the exploration, rather than serving as a definitive endpoint that accounts for an exhaustive set of semantic knowledge. however, future investigations should expand the current study's scope for better generalizability. moreover, the study does not detail how attention heads interact with other components like hidden states across layers or feed-forward perceptrons. such details would be essential in enhancing our understanding of the attention head roles in plausibility processing by elucidating how these heads impact transformer models' plausibility processing ability. as such, subsequent studies should delve deeper into these interactions for a more accurate understanding of their role in semantic knowledge processing. as these limitations are addressed, i anticipate further advancements in explaining transformer models' capacity for semantic knowledge processing, founded on the novel findings and methods introduced in this study.