The argument–adjunct distinction in BERT : A F rame N et-based investigation (2023)

Abstract: The distinction between arguments and adjuncts is a fundamental assumption of several linguistic theories. In this study, we investigate to what extent this distinction is picked up by a Transformer-based language model. We use BERT as a case study, operationalizing arguments and adjuncts as core and non-core FrameNet frame elements, respectively, and tying them to activations of particular BERT neurons. We present evidence, from English and Korean, that BERT learns more dedicated representations for arguments than for adjuncts when fine-tuned on the FrameNet frame-identification task. We also show that this distinction is already present in a weaker form in the vanilla pre-trained model.

## introduction the widely used transformer-based contextualized language model bert @xcite has been extensively studied regarding its capability to uncover linguistic patterns from raw text, with analyses focused mostly on syntax. both constituency and dependency trees were either found encoded inside the model or were used to probe for syntactic rules such as agreement @xcite . in this paper, we shift the focus of bert analysis to the syntax-semantics interface, considering the foundational distinction between arguments and adjuncts. according to @xcite , arguments and adjuncts differ in two crucial ways: arguments describe necessary participants in the event described by the verb and are therefore both obligatory, i.e. they have to be realized by default, and specific, i.e. they express idiosyncratic properties of the event or the event class. in contrast, neither is necessarily true for adjuncts. for example, in the sentence peter praised his colleague repeatedly, the praising event is accompanied by two necessary, specific participants, namely a communicator, peter, and an evaluee, the colleague; in contrast, the adverb repeatedly, which specifies the frequency, could be left out and applies to a very broad range of events. the argument-adjunct distinction has played a major role in linguistic theory @xcite @xcite and has implications for human language processing @xcite and semantic nlp @xcite . we empirically assess the status of the argumentadjunct disinction in bert by making use of framenet @xcite ) -an implementation of frame semantics @xcite , a theory of predicate-argument structure, which describes predicate meaning in terms of frames (prototypical situations) and frame elements (the situations' participants). framenet maintains a distinction between core elements and non-core elements, which maps onto the argument-adjunct distinction (see section 2 for details). we use a modification of the method of model analysis proposed by @xcite for associating neurons inside neural-network models with features they are particularly attuned to. in our main analysis, we use framenet annotations to fine-tune bert for a task -frame identification, -for which frame elements are informative, without exposing the frame-element labels to the model, and then correlate the learned model representations with the presence of these labels. we also repeat the correlational analysis on the vanilla (pre-trained) bert model. our contribution is twofold: (a) we extend rethmeier et al.'s methodology, which targeted lstms, to bert and, instead of constructing a probability distribution of features a given neuron is attuned with, we extract tight neuron-feature combinations using correlation analysis, reminiscent of the larger neuroscience literature on input-specific neural activations @xcite ; (b) we use this method for an analysis of the representation of arguments vs. adjuncts in multilingual bert (mbert) based on english and korean data. we find that even though bert representations are dominated by frequency effects, with common input patterns more robustly tracked by individual neurons, arguments and adjuncts differ in their activation patterns (arguments produce relatively more robust activations while adjuncts generally lack highly specialized neurons that track them) and that this distinction is already present, to a lesser extent, in a vanilla pre-trained model. foot_0 ## frame semantics and framenet frame semantics @xcite posits that a key element of the understanding of an utterance is knowledge about the situations that the predicates in it evoke. this knowledge is captured through frames, schemas that associate predicates (frameevoking elements / fees) with situations, their inferences, and their relevant participants, which are realized in language as so-called frame elements. frame-semantic resources were first developed for english (framenet; @xcite but have been extended to other languages @xcite . the example given in the introduction, peter praised his colleague repeatedly, evokes the judgment_communication frame where a communicator expresses an evaluation of an evaluee. these are two of the core elements (ces) of this frame, which generally meet both of koenig et al.'s criteria for argumenthood: they are obligatory (unless they are null-instantiated, cf. fillmore 1986) and they are specific to frames (or groups of closely related frames, cf. @xcite . in contrast, the judgment_communication frame contains a number of non-core elements (nces), which do not meet at least one of the two criteria and thus show adjunct behavior: they are either not specific (manner, frequency) or not obligatory (grounds: the basis for the judgment; role: the capacity of the evaluee). a similar situation obtains with many other frequently found frames, and we assume that the core vs. non-core distinction largely mirrors the argument/adjunct dichotomy. ## experimental setup fine-tuning bert we start from a pre-trained bert model and fine-tune it to assign a single frame to each sentence @xcite in line with the framenet annotation (cf. section 2). we experiment with two variants of the task. in the fee present setting, the model is shown complete sentences, including the fees, but no frameelement annotation. this task aims at encouraging the model to connect fees with arguments, which are known to be relevant for frame identification @xcite . adjuncts are expected to be less relevant (as they are unspecific) or less reliable (as they are optional). to select the frame, we feed the first subword of the first fee token to a fully-connected 50-neuron layer (corresponding to the 50 frames) and obtain a prediction by applying the usual softmax. 2 in the fee masked setting, all fee tokens are replaced with the [mask] token, so that the model has to rely on the sentential context to identify the frame. our hypothesis is that this version of the task incentivizes the model to more actively focus on extracting arguments. in this case, we feed the embedding of the first masked token into the frame classification head as above. in both variants, the model is trained end-to-end using cross-entropy loss for twenty epochs with early stopping when the performance on the test set decreases. we use the pre-trained mbert model provided by huggingface @xcite . for english, we report results for the test set. for korean, we adopt a zero-shot setting and, after checking that mbert fine-tuned on english has some success in identifying korean frames, analyze the activations that korean sentences produce in it. probing analysis once the bert model has been fine-tuned, we can analyze the activation patterns of different layers of the model @xcite . on the data side, we cast input sentences as a binary matrix whose columns correspond to the presence or absence of each ce or nce in these sentences, i.e. to their indicator functions. on the model side, we associate each input sentence with a @xmath0, depending on the setting, for a selected subset of bert layers. we then carry out correlational analysis to identify, for each ce or nce indicator function and for each layer of interest, the neuron whose activations are most strongly correlated with these functions. to choose layers for the analysis, we evaluated english model predictions based on the representations in each layer. the results are shown in figure 1 . for both variants of the task, we find similar results: the outputs of the 11th layer are close to the final layer, and there is a swift increase in prediction accuracy from the 7-8th layer onward. on this basis, we probe the activation patterns of layer 11 (near-convergence) and layer 9 (start of competitive performance). analysis of neural activity was performed in a similar fashion by @xcite . they, however, extract activations in the context of specific tasks, such as pos tagging and syntactic chunking, instead of feeding sentences to a headless embedding model in an unsupervised setting. ## results and discussion english the results of the correlation analysis are presented in the scatterplots in figure 2 . individual points show, for a frame element with a given frequency, how large the correlation with the most attuned neuron activation vector in the respective model is. the left plot shows core elements, the right plot non-core elements. the plots show that frequency is the dominating factor: high-frequency frame elements tend to have (more or less) dedicated neurons tracking them, with correlations of 0.4 and above, while this is not true for low-frequency frame elements. this is to be expected given the maximum-likelihood training objective. however, there still is a clear difference between ces and nces: even the most frequent nces do not attain correlations above 0.3, and only a handful show correlations above 0.2, in both the standard and masked settings. in contrast, the correlations for ces with frequencies above 100 are all higher than 0.2. this shows the model's low reliance on nces for frame identification. comparing the behaviors at layers 9 (red) and 11 (turquoise), we do not see major differences: in particular for nces, the plots are extremely similar. comparing the two variants of the task (solid vs. dashed), we see that the masked-task model learns less dedicated representations for the ces but spends some more effort on representing highfrequency nces -contrary to the expectation we formulated in section 3. the global advantage of ces over nces in all settings leads us to believe that the model simply relies on arguments in either case, and that in the masked setting the model just struggles more to identify where they are. -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 figure 3: correlations between frames (rows) and their core elements (columns). this interpretation is corroborated by an analysis of ce information content. figure 3 shows a matrix of correlations between frames with nonmasked fees (rows) and their ces (columns). some frames are in a nearly one-to-one correspondence with their ces, but other ces can be found with several frames. arguably, when fees are present, they form a strong signal together with the ces pointing towards particular frames. when fees are masked, however, frequent ces -precisely those that are found with many different frames -become less informative, and the model shifts some of the weight towards nces. ## conclusion our study asked whether bert can distinguish between arguments and adjuncts and operationalized these concepts via framenet's core vs. non-core frame-element distinction. for both english and korean, our analysis of the presence of dedicated 4 two most-frequent frames, agent and theme, are very general and unsurprisingly display weaker correlations. by comparison, the next three most-frequent frames, speaker, goal, and time, are much richer semantically and have more dedicated representations. on the neural-language-model side, our study confirms the ability of such models to recover 'deep' linguistic categories in an unsupervised manner. on the framenet side, our results have bearing on the status of borderline-core frame elements @xcite , for which the behaviour of the model may serve as a heuristic. a promising avenue for future work would be to turn around our setup and to explore bert representations in order to identify a set of properties that differentiate arguments and adjuncts from the model's point of view, à la @xcite . this work has focused on framenet. other frameworks giving access to semantic-role information, such as the propbank annotation scheme @xcite , amr @xcite , and ucca @xcite , also may be fruitful for this type of analysis.