E vent BERT : Incorporating Event-based Semantics for Natural Language Understanding (2022)

Abstract: “Natural language understanding tasks require a comprehensive understanding of natural language and further reasoning about it, on the basis of holistic information at different levels to gain comprehensive knowledge. In recent years, pre-trained language models (PrLMs) have shown impressive performance in natural language understanding. However, they rely mainly on extracting context-sensitive statistical patterns without explicitly modeling linguistic information, such as semantic relationships entailed in natural language. In this work, we propose EventBERT, an event-based semantic representation model that takes BERT as the backbone and refines with event-based structural semantics in terms of graph convolution networks. EventBERT benefits simultaneously from rich event-based structures embodied in the graph and contextual semantics learned in pre-trained model BERT. Experimental results on the GLUE benchmark show that the proposed model consistently outperforms the baseline model.”

## introduction recent years have witnessed deep pre-trained language models (prlm) such as elmo @xcite , bert @xcite , xlnet @xcite and ernie @xcite significantly prospering the performance of a wide range of natural language understanding (nlu) tasks. the remarkable advancements brought by prlm have shown the effectiveness of leveraging contextualized representation. however, they mainly rest on extracting context-sensitive statistical patterns without explicitly modeling linguistic information such as semantic relationships in natural language. it is clear that natural language itself abounds with ample, multi-level linguistic information. although prlms like bert implicitly represent linguistic knowledge more or less @xcite , studies disclose that linguistic knowledge is far from fully absorbed @xcite . therefore, there emerges a series of derivatives of prlm intending to fuse explicit linguistic knowledge so as to acquire better language representation, including syntactic @xcite @xcite and semantic information @xcite @xcite . in cognition practice, human needs to distill semantics of different levels to gain a comprehensive understanding, whereas neural language models learn semantic representation to deal with downstream tasks @xcite . thus, effective learning of semantic knowledge plays a crucial role in nlu tasks and has gained growing attention recently. for instance, @xcite proposed sembert, which directly connects multiple predicate-argument structures acquired by semantic role labeler (srl) to get the joint representation. the essence of srl @xcite lies in that every sentence possesses multiple predicatespecific structures which can represent different frames of events, while semantic roles express the abstract role that arguments of a predicate can take in the event. besides, the events inside a sentence have interactions with each other that serve together to present the overall semantic knowledge. as shown in figure 1 , srl parses every sentence with multiple predicate-specific structures which can serve as events inferring who did what to whom, when and why. each event has an inner structure centered on the predicate to which several arguments are associated such as hoy[arg0], the woman's age[arg1] and tuesday[argm-tmp] connected to confirmed @xcite . meanwhile, the multiple events work together to give a comprehensive meaning of a sentence, like the events centered on said, confirmed and left. with regard to delving into the inner interactions between the events and effectively capturing multiple objects, we are motivated to build a graph to reveal the intrinsic structures between and inside the events. our proposed model works in three steps: it first applies an off-the-shelf srl toolkit to parse every sentence with semantic role labels; then it constructs event-based graphs and employs graph convolutional networks (gcns) @xcite to propagate and aggregate information from neighboring nodes on the graph; at last, it combines the contextualized representation acquired by bert encoder together with the graph-level representation to obtain an event-based contextualized representation. the key contributions of our work are summarized as follows: ## conclusion in this work, we propose eventbert, an event-based semantic representation model that builds on bert architecture and incorporates event-based structural semantics in terms of graph network modeling for fine-grained language representation. experiments on a wide range of nlu tasks show the effectiveness of our model by consistently surpassing the baseline. while most existing works focus on fusing accurate semantic signals to enhance semantic information, we open up a novel perspective to model intrinsic structural semantics for deeper comprehension and inference in an intuitive and explicit way.