TAGPRIME : A Unified Framework for Relational Structure Extraction (2023)

Abstract: Many tasks in natural language processing require the extraction of relationship information for a given condition, such as event argument extraction, relation extraction, and task-oriented semantic parsing. Recent works usually propose sophisticated models for each task independently and pay less attention to the commonality of these tasks and to have a unified framework for all the tasks. In this work, we propose to take a unified view of all these tasks and introduce TAGPRIME to address relational structure extraction problems. TAGPRIME is a sequence tagging model that appends priming words about the information of the given condition (such as an event trigger) to the input text. With the self-attention mechanism in pre-trained language models, the priming words make the output contextualized representations contain more information about the given condition, and hence become more suitable for extracting specific relationships for the condition. Extensive experiments and analyses on three different tasks that cover ten datasets across five different languages demonstrate the generality and effectiveness of TAGPRIME.

## introduction there are many tasks in natural language processing (nlp) that require extracting relational structures from texts. for example, the event argument extraction task aims to identify event arguments and their corresponding roles for a given event trigger @xcite . in entity relation extraction, the model identifies the tail-entities and head-entities that forms specific relations @xcite . in taskoriented semantic parsing, the model predicts the slots and their semantic roles for a given intent in an * the authors contribute equally. in this work, we take a unified view of these nlp tasks. we call them relational structure extraction (rse) tasks and formulate them as a unified task that identifies arguments to a given condition and classifies their relationships. the condition could be a textual span, such as an event trigger for event argument extraction, or a concept, such as an intent for task-oriented semantic parsing. we present tagprime, a simple, unified, and strong model, which follows a sequence tagging paradigm with a priming technique, which is proposed by @xcite . tagprime inherits the strength of sequence tagging models to unifiedly address rse by converting the relational structure into a sequence of predictions by sequentially labeling tokens in the input passage. tagprime further improves this framework's performance by better incorporating information about the given condition via priming. traditional sequence tagging models usually leverage learnable feature embeddings to incorporate information about the given condition before the tags are assigned, as illustrated in figure 1(a) . with the priming mechanism, tagprime augments the input text with condition-specific contexts, as illustrated in figure 1 a conventional sequence tagging model that embeds conditional information by adding learnable features to the output representation from a pre-trained language model. in the shown example, the conditional features contain two parts: one is the token embedding representing the conditional word "military", and the other is an entity type embedding. (b) tagprime with condition priming: the conditional information is further applied to the input sequence to induce the output representation from the pre-trained language model to become condition-aware. (c) tagprime with condition & relationship priming: our approach that further append the verbalized relationship to tagprime with condition priming model. for this case, the goal of the tagging model is to make predictions specific to the relationship type in the input. we omit crf layers after mlp layers in this figure for better readability. our contributions can be summarized as follows. (1) we take a unified view of nlp tasks that requires extracting relational structures, including end-to-end event extraction, end-to-end relation extraction, and task-oriented semantic parsing. then, we present tagprime, a unified sequence tagging model with priming that can serve as a strong baseline to various relational structure extraction problems. (2) thorough experiments on three different tasks show that tagprime achieves competitive performance than the current state-of-the-art on ten datasets in five different languages. (3) we propose a novel efficient approximation to speed up tag-prime during inference time without sacrificing too much performance. our code will be publicly accessible at https: //github.com/pluslabnlp/tagprime . ## related work many natural language processing applications require extracting relational structures, including event extraction, relation extraction, coreference resolution, etc. the prevalence of these applications makes us hard to exhaustively list them in this short summary, hence, we mainly focus on related works for the applications we experiment on. ## method we first introduce our view to unify rse problems and then discuss how tagprime approaches this problem under a unified framework of sequence tagging model with priming. ## experiments to study the effectiveness of tagprime, we consider three nlp tasks: (1) end-to-end event extraction, (2) end-to-end relation extraction, and (3) taskoriented semantic parsing. all the results are the average of five runs with different random seeds. ## analysis in this section, we study two questions: (1) what is the effectiveness of priming techniques compared to learnable features? (2) relationship priming boosts the performance of tagprime, but the task decomposition could slightly slow down the inference speed. can we mitigate this issue? to answer the first question, we conduct ablation experiments on sequence tagging models using different combinations of learnable features or/and adding information through the priming technique (section 5.1). for the second question, we propose a simple modification to tagprime so that we can flexibly control the number of layers to fuse priming information to contextualized representations. the modified tagprime can serve as an efficient approximation of tagprime (section 5.2). ## conclusion in this work, we take a unified view of tasks requiring extracting relational structures and present tag-prime, a simple, unified, effective, and general sequence tagging model. the key idea is applying priming, a small trick to make the representations task-specific by appending condition-related and relationship-related strings to the input text. our experimental results demonstrate that tagprime is general to different tasks in various languages and can serve as a strong baseline for future research on relational structure extraction.