Exploring Semantics in Pretrained Language Model Attention (2024)

Abstract: Abstract Meaning Representations (AMRs) encode the semantics of sentences in the form of graphs. Vertices represent instances of concepts, and labeled edges represent semantic relations between those instances. Language models (LMs) operate by computing weights of edges of per layer complete graphs whose vertices are words in a sentence or a whole paragraph. In this work, we investigate the ability of the attention heads of two LMs, RoBERTa and GPT2, to detect the semantic relations encoded in an AMR. This is an attempt to show semantic capabilities of those models without finetuning. To do so, we apply both unsupervised and supervised learning techniques.

## introduction an amr graph, as specified by @xcite , is a representation of the meaning of a sentence in the form of a directed acyclic graph, involving concepts from neo-davidsonian semantics @xcite . a number of datasets of sentences and their corresponding hand-crafted amrs have been published, and various techniques have been developed to automatically build amr graphs from sentences in natural language. these include graph based approaches, which directly predict nodes and edges from the sentences, @xcite , and algorithms based on transition systems @xcite , inspired by dependency parsing algorithms (camr: @xcite . the most recent solutions combine an encoderdecoder pair of a transformer network @xcite to adapt it to the task of transitionbased amr parsing, as structbart does @xcite . amr graphs abstract away meaning from syntactic representations. this is evidenced by the fact that one amr graph can encode several dif-ferently worded sentences, even with different syntaxes. @xcite transformer-based language models, introduced by @xcite , have shown remarkable performance in solving many problems related to automatic natural language processing, but the interpretability of their computations is still subject to active research: @xcite studied the ability of certain attention heads in the bert network @xcite to classify several syntactic relations between words and to resolve coreference, without finetuning bert for any specific task. @xcite studied how constituency grammar is captured by different attention heads in bert. we complement their work and explore the ability of attention heads to classify semantic relations between two words as described by the edge type between two vertices of an amr. we study a representative bidirectional pretrained language model, without finetuning: roberta @xcite , and compare it to gpt-2 @xcite , a pretrained conditional model, using both unsupervised and supervised techniques. our study reveals a striking correlation of these networks' attention heads with semantics. we observed that roberta showed conspicuously better results than gpt-2, probably because of the bidirectional nature of the former. to reproduce our experiments, we made our code publicly available. foot_0 ## dataset design and experimental setup in a nutshell, an amr encodes in a rooted directed acyclic graph who is doing what to whom, where, when and how, in a manner that abstracts away semantics from syntax. in particular, a single amr graph can encode several syntactically different sentences, like "the bears invaded sicily" (a whole clause), "the bears' invasion of sicily" (a noun invade-01 bear arg0 place arg1 name name "sicily" op1 figure 1 : amr for several wordings: "the bears invaded sicily", "the bears' invasion of sicily", "the invasion of sicily by the bears" and "the invasion of the bears in sicily". phrase), "the invasion of sicily by the bears", or "the invasion of the bears in sicily". see figure 1 . in so doing, an amr encodes instances of concepts as vertices in the graph, using propbank framesets @xcite wherever possible foot_1 . relations between instances of concepts are encoded as directed labeled edges in the graph. those relations can be the frame arguments, following propbank conventions (arg0, arg1,...), or other general semantic relations (time, cause, location, etc.). blodgett and schneider (2021) published a dataset of automatic alignments between amrs and the corresponding english sentences in the ldc2020t02 dataset @xcite , which comprises 59,255 sentences. we took advantage of those alignments and built a dataset of edges to test the capability of an lm's attention mechanism to retrieve the semantic relation encoded in a edge from the two connected vertices. in their work, blodgett and schneider labeled their alignments across several categories: • subgraph alignments, a mutually exclusive alignment between consecutive spans of words and subgraphs of the amr, • duplicate subgraph alignments, to account for elliptical construction, where several identical subgraphs in the amr are mapped to the same word span, • relation alignments, providing alignments between a span and a single relation, (an arc in the amr), such as "when" → :time, and • reentrancy alignments, accounting for reentrancy, (the fact that an amr node may have multiple incoming edge). reentrancy alignments provide alignments between reentrant arcs and a word span that triggers the reentrancy. (pronouns, control verb, etc.) we selected the "subgraph alignments", sorting them to keep only those alignments involving a single word in the sentence and a single-vertex subgraph. next, we had the sentences processed by the tokenizers of two pretrained language models with 12 layers and 12 heads per layer : • roberta, a bidirectional encoder @xcite , and • gpt-2, a conditional decoder @xcite . to deal with the case of words split across several tokens, we aligned the sequences of words with the sequences of tokens, keeping only alignments involving a single word aligned with a single token. we thus created a dataset of pairs of tokens aligned with vertices of amr graphs, linked by a semantic relation. to assess the ability to detect the absence of a semantic relation, we included random pairs of tokens corresponding to vertices in the amrs with no edge between them, to create a category "<@xmath0. we then ran the transformers with all sentences as input and computed their representations. for each pair of words, there are possibly two attention directions to be computed: attention from one word to the other, or conversely. we call them st and t s, as they represent attention from the source to the target or from the target to the source, where "target" and "source" denote the direction of the edge in the amr graph. each of those two attentions is represented by 144 scalars, as there are 12 layers, and 12 attention heads per layer. in a transformer, the attention weight from a source token q to a target token k is obtained by taking two affine transformations of the embeddings of q and k, computing the inner product of those, and taking the softmax of that product with respect to all other target tokens. the features we use throughout this study are actually those inner products, before application of the softmax. ## unsupervised analysis: pca the first step of our study was to apply a simple dimension reduction technique to the dataset. we chose to compute a principal component analysis on the inner product dataset. for the dataset computed with roberta, we found that keeping 4 principal components enabled us to explain about 52% of the total variance. we filtered the dataset by relation, and computed kernel density estimations of the distribution of different relations, and looked for dissimilarities between those. to do so, we selected a few relations to be plotted overlaid : (<no_rel>, arg0, arg1, condition, location, time, arg2, quant, polarity, mod, and poss). we found that the first six relations could easily be distinguished by examining only the projection on the second principal component, as their distributions seem very different, although somewhat overlapped (see figure 2 ). the most conspicuous separation is between arg0 and arg1. however, no pair of relations presented completely distinct distributions. the separation of the relations quant, mod and poss is less obvious, and can best be seen on the projection on the fourth principal component. as for the pair of relations (arg1, arg2), they can hardly be distinguished. (plots can be seen on appendix a.1) for the dataset computed with the decoder gpt-2, we found that keeping 4 principal components enabled us to explain 70% of the total variance. (18% more than with roberta). in spite of this difference, we found that gpt-2 was less effective than roberta in distinguishing relations. as an illustration, in figure 3 , we plotted the distributions of the three easiest to distinguish relations for roberta and gpt-2 on the most distinctive axis pc2. (see appendix a.2 for more gpt-2 plots.) this first step tends to show that it is possible to use the attention heads of a transformer network to observe different distributions for pairs of different semantic relations. this behavior of a transformer is more prevalent for a bidirectional encoder (like roberta) than for a conditional decoder (like gpt-2). ## supervised analysis: logistic classifier on the strength of these results, we trained a logistic model to classify the semantic relations of our datasets. to do so, we modified the datasets in the following way : ## conclusion pre-trained lms can, to some extent, code semantic relations in their attention mechanism without need of specialization. bidirectional networks, as roberta, show better ability to distinguish between different semantic roles than conditional networks, as gpt-2. linear methods used in this work unveils an important fact. pre-trained lms encode not only the syntactic structure, but also the semantic structure of the text so that it can be exploited in a linear fashion.