Verb Knowledge Injection for Multilingual Event Processing (2021)

Abstract: Linguistic probing of pretrained Transformer-based language models (LMs) revealed that they encode a range of syntactic and semantic properties of a language. However, they are still prone to fall back on superficial cues and simple heuristics to solve downstream tasks, rather than leverage deeper linguistic information. In this paper, we target a specific facet of linguistic knowledge, the interplay between verb meaning and argument structure. We investigate whether injecting explicit information on verbs’ semantic-syntactic behaviour improves the performance of pretrained LMs in event extraction tasks, where accurate verb processing is paramount. Concretely, we impart the verb knowledge from curated lexical resources into dedicated adapter modules (verb adapters), allowing it to complement, in downstream tasks, the language knowledge obtained during LM-pretraining. We first demonstrate that injecting verb knowledge leads to performance gains in English event extraction. We then explore the utility of verb adapters for event extraction in other languages: we investigate 1) zero-shot language transfer with multilingual Transformers and 2) transfer via (noisy automatic) translation of English verb-based lexical knowledge. Our results show that the benefits of verb knowledge injection indeed extend to other languages, even when relying on noisily translated lexical knowledge.

## introduction large transformer-based encoders, pretrained with self-supervised language modeling (lm) objectives, form the backbone of state-of-the-art models for most nlp tasks @xcite @xcite . recent probes showed that they implicitly extract a non-negligible amount of linguistic knowledge from text corpora in an unsupervised fashion @xcite @xcite . in downstream tasks, however, they often rely on spurious correlations and superficial cues @xcite rather than a deep understanding of language meaning @xcite , which is detrimental to both generalisation and interpretability @xcite . in this work, we focus on a specific facet of linguistic knowledge: reasoning about events. 1 identifying tokens in the text that mention events and classifying the temporal and causal relations among them is crucial to understand the structure of a story or dialogue @xcite and to ground a text in real-world facts. verbs (with their arguments) are prominently used for expressing events (with their participants). thus, fine-grained knowledge about verbs, e.g., the syntactic patterns in which they partake and the semantic frames, may help pretrained encoders to achieve a deeper understanding of text and improve their performance in event-oriented downstream tasks. there already exist some expert-curated computational resources that organise verbs into classes based on their syntactic-semantic properties @xcite . in particular, here we consider english verbnet and framenet as rich sources of verb knowledge. expanding a line of research on injecting external linguistic knowledge into pretrained lms @xcite @xcite , we integrate verb knowledge into the lms for the first time. we devise a new method to distil verb knowledge into dedicated adapter modules @xcite , which reduce the risk of (catastrophic) forgetting of and allow seamless modular integration with distributional knowledge. we hypothesise that complementing pretrained lms with verb knowledge should benefit model performance in downstream tasks that involve event extraction and processing. we first put this hypothesis to the test in english monolingual event identification and classification tasks from the tempeval @xcite and ace @xcite datasets. we report modest but consistent improvements in the former, and significant performance boosts in the latter, thus verifying that verb knowledge is indeed paramount for a deeper understanding of events and their structure. moreover, expert-curated resources are not available for most of the languages spoken worldwide. therefore, we also investigate the effectiveness of transferring verb knowledge across languages; in particular, from english to spanish, arabic and chinese. the results demonstrate the success of the transfer techniques, and also shed some light on an important linguistic question: to what extent can verb classes (and predicate-argument structures) be considered cross-lingually universal, rather than varying across languages @xcite ? overall, our main contributions consist in 1) mitigating the limitations of pretrained encoders regarding event understanding by supplying external verb knowledge; 2) proposing a new method to do so in a modular way through verb adapters; 3) exploring techniques to transfer verb knowledge to resource-poor languages. the performance gains across four diverse languages and several event processing tasks and datasets validate that complementing distributional knowledge with curated verb knowledge is both beneficial and cost-effective. ## verb knowledge for event processing figure 1 illustrates our framework for injecting verb knowledge from verbnet or framenet and leveraging it in downstream event processing tasks. first, we inject the external verb knowledge, formulated as the so-called lexical constraints @xcite (in our case -verb pairs, see §2.1), into a (small) additional set of adapter parameters ( §2.2) @xcite . second ( §2.3), we combine the language knowledge encoded in the original lm parameters and the verb knowledge from verb adapters for event processing tasks. to this end, we either a) fine-tune both sets of parameters (1. pretrained lm; 2. verb adapters) or b) freeze both sets of parameters and insert an additional set of task-specific adapter pa- multi-head attention add & normalize feed-forward add & normalize add & normalize verb (vn/fn) adapter verb-pair classifier l verb or [convert, transform, true] [separate, split, false] l task multi-head attention add & normalize feed-forward add & normalize add & normalize verb (vn/fn) adapter ...it also [affects] state small businesses, which [pay] occurrence premiums... event task classifier multi-head attention add & normalize feed-forward add & normalize add & normalize verb (vn/fn) adapter task adapter 1) 2a) 2b) event task classifier l task ...it also [affects] state small businesses, which [pay] occurrence premiums... rameters. in both cases, the task-specific training is informed both by the general language knowledge captured in the pretrained lm, and the specialised verb knowledge, captured in the verb adapters. ## experimental setup event processing tasks and data. in event processing tasks, systems are tasked with detecting that something happened, identifying what type of occurrence took place, as well as what entities were involved. verbs typically act as the organisational core of each such event schema, carrying a lot of semantic and structural weight. therefore, a model's grasp of verbs' properties should have a bearing on final task performance. based on this assumption, we select event extraction and classification as suitable tasks to profile the methods from §2. these tasks and the corresponding data are based on the two prominent frameworks for annotating event expressions: timeml @xcite and the automatic content extraction (ace) @xcite . first, we rely on the timeml-annotated corpus from tempeval tasks @xcite , which targets automatic identification of temporal expressions and relations, and events. second, we use the ace dataset: it provides annotations for entities, the relations between them, and for events in which they participate in newswire text. 5 task 1: trigger identification and classification (tempeval). we frame the first event processing task as a token-level classification problem, predicting whether a token triggers an event and assigning it to one of the following event types: oc-currence (e.g., died, attacks), state (e.g., share, assigned), reporting (e.g., announced, said), i-action (e.g., agreed, trying), i-state (e.g., understands, wants, consider), aspectual (e.g., ending, began), and perception (e.g., watched, spotted). 6 we use the tempeval-3 data for english and spanish @xcite , and the tempeval-2 data for chinese @xcite (see ## results and discussion english event processing. ## related work event extraction. the cost and complexity of event annotation requires robust transfer solutions capable of making fine-grained predictions in the face of data scarcity. traditional event extraction methods relied on hand-crafted, language-specific features @xcite @xcite @xcite (e.g., pos tags, entity knowledge), which limited their generalisation ability and effectively prevented language transfer. more recent approaches commonly resorted to word embedding input and neural text encoders such as recurrent nets @xcite @xcite and convolutional nets @xcite , as well as graph neural networks @xcite and adversarial networks @xcite . most recent empirical advancements in event trigger and argument extraction tasks stem from fine-tuning of lm-pretrained transformer networks @xcite @xcite @xcite . limited training data nonetheless remains an obstacle, especially when facing previously unseen event types. the alleviation of such data scarcity issues was attempted through data augmentation -automatic data annotation @xcite @xcite and bootstrapping for training data generation @xcite . the recent release of the large english event detection dataset maven @xcite , with annotations of event triggers only, partially remedies for english data scarcity. maven also demonstrates that even the state-of-the-art transformer models fail to yield satisfying event detection performance in the general domain. the fact that it is unlikely to expect datasets of similar size for other event extraction tasks and especially for other languages only emphasises the need for external event-related knowledge and transfer learning approaches, such as the ones introduced in this work. ## conclusion we investigated the potential of leveraging knowledge about semantic-syntactic behaviour of verbs to improve the capacity of large pretrained models to reason about events in diverse languages. we proposed an auxiliary pretraining task to inject verbnet-and framenet-based lexical verb knowledge into dedicated verb adapter modules. we demonstrated that state-of-the-art pretrained models still benefit from the gold standard linguistic knowledge stored in lexical resources, even those with limited coverage. crucially, we showed that the benefits of the knowledge from resourcerich languages can be extended to other, resourceleaner languages through translation-based transfer of verb class/frame membership information.