BERT ering RAMS : What and How Much does BERT Already Know About Event Arguments? - A Study on the RAMS Dataset (2020)

Abstract: Using the attention map based probing framework from (Clark et al., 2019), we observe that, on the RAMS dataset (Ebner et al., 2020), BERT’s attention heads have modest but well above-chance ability to spot event arguments sans any training or domain finetuning, varying from a low of 17.77% for Place to a high of 51.61% for Artifact. Next, we find that linear combinations of these heads, estimated with approx. 11% of available total event argument detection supervision, can push performance well higher for some roles — highest two being Victim (68.29% Accuracy) and Artifact (58.82% Accuracy). Furthermore, we investigate how well our methods do for cross-sentence event arguments. We propose a procedure to isolate “best heads” for cross-sentence argument detection separately of those for intra-sentence arguments. The heads thus estimated have superior cross-sentence performance compared to their jointly estimated equivalents, albeit only under the unrealistic assumption that we already know the argument is present in another sentence. Lastly, we seek to isolate to what extent our numbers stem from lexical frequency based associations between gold arguments and roles. We propose NONCE, a scheme to create adversarial test examples by replacing gold arguments with randomly generated “nonce” words. We find that learnt linear combinations are robust to NONCE, though individual best heads can be more sensitive.

## introduction the nlp representation paradigm has undergone a drastic change in this decade -moving from lin-guistic/task motivated 0-1 feature families to perword-type pretrained vectors @xcite to contextual embeddings @xcite . contextual embeddings (ces) produce incontext representations for each token -the representation framework being a large, pretrained encoder with per-token outputs. the typical procedure to use ces for a downstream task is to add one or more task layers atop each token, or for a designated token per-sentence, depending on the nature of the task. the task layers (and optionally, the representation) are then "finetuned" using a task specific loss, albeit with a slower training rate than would be used for from-scratch training. elmo @xcite was an early ce. the three-fold recipe of a transformer based architecture, masked language modelling objective and large pre-training corpora, starting with bert @xcite led to ces which were vastly effective for most tasks. the strong performance of contextual representations with just shallow task layers and minimal finetuning drove the urge to understand what and how much these models already knew about aspects of syntax and semantics. the study of methods and analysis to do this has come to be called probing. besides "explaining" ce featurization, probing can aid in finding lacunae to be addressed by future representations. @xcite , one of the early works on probing, evaluated whether language models could predict the correct verb form agreeing with the noun. @xcite generalized this approach beyond single-word gaps with a larger suite of "minimal pairs". they also control for lexical confounding and expand the probing to new aspects such as reflexive anaphora and npis. @xcite evaluate subject-verb agreement but only through "nonce" sentences to con-trol for both lexical confounding and memorization 3 . @xcite isolate units of lstm language models whose activations closely track verb-noun number agreement, particularly for hard, long-distance cases. @xcite , whose probing methods we adopt, examine if bert attention heads capture dependency structure. in this work, we probe what and how much a pretrained bert representation already knows about event roles and their arguments. understanding how well event arguments are represented can be a first foray into understanding other aspects about events. extraction of event arguments is often a prerequisite for more complex event tasks. some examples are event coreference @xcite , detecting event-event temporal @xcite and causal relations @xcite , sub-event structure @xcite and generating approximate causal paths @xcite . tuples of event-type and arguments are one way of inducing script like-structures @xcite . in summary, our work makes the following contributions: to do this, we propose nonce, a method to perturb test examples to dampen such associations ( §2.5.3). we find that the linear approach is robust to nonce perturbation, while besthead is more sensitive. ## related work a complete description of the large body of work on probing is beyond the scope of this paper. besides those discussed earlier, other aspects studied include filler-gap dependencies @xcite , function word comprehension @xcite , sentence-level properties @xcite and negative polarity items @xcite . probing is not limited to examining pairwise word relations or sentence properties. @xcite find that bert token representations are linearly projectable into a space where they embed constituency structure. recently, @xcite probed transformer based language models for coreference. however, they restrict themselves to entity coreference. further- ## conclusion we showed how bert's attention heads have modest but well above chance ability to detect arguments for event roles. this ability is achievable either with only i) 2 parameters per role (besthead) ii) 289 parameters per role (linear). furthermore, the supervision required to reach this is just ≈ 11% of full training set size. secondly, we propose a method to learning separate heads (combinations) for cross-sentence argument detection. our experiments show that the heads so learnt have higher cross-sentence accuracy. thirdly, we show that linear performance is robust to a perturbed nonce test setting with weakened lexical associations. in future, we plan to extend our probing to other event aspects like coreference and subevents.