TERL : Transformer Enhanced Reinforcement Learning for Relation Extraction (2023)

Abstract: “Relation Extraction (RE) task aims to discover the semantic relation that holds between two entitiesand contributes to many applications such as knowledge graph construction and completion. Reinforcement Learning (RL) has been widely used for RE task and achieved SOTA results, whichare mainly designed with rewards to choose the optimal actions during the training procedure,to improve RE’s performance, especially for low-resource conditions. Recent work has shownthat offline or online RL can be flexibly formulated as a sequence understanding problem andsolved via approaches similar to large-scale pre-training language modeling. To strengthen theability for understanding the semantic signals interactions among the given text sequence, thispaper leverages Transformer architecture for RL-based RE methods, and proposes a genericframework called Transformer Enhanced RL (TERL) towards RE task. Unlike prior RL-basedRE approaches that usually fit value functions or compute policy gradients, TERL only outputsthe best actions by utilizing a masked Transformer. Experimental results show that the proposedTERL framework can improve many state-of-the-art RL-based RE methods.”

## introduction relation extraction (re) aims to discover the binary semantic relation between two entities in a sequence of words. e.g., given a sentence "• • • carey will succeed cathleen p. black, who held the position for 15 years and will take on a new role as chairwoman of hearst magazines, the company said• • • " @xcite , and we aim to predict the relation type between two entities "cathleen p. black" and "chairwoman" and the result is "per:title". deep neural network (dnn) driven methods have gained decent performance when labeled data is available @xcite . while reinforcement learning (rl) based re methods gain a lot of attention recently and show encouraging effects @xcite @xcite , especially in low-resource and few-shot conditions. since this kinds of work requires fewer labeled data or could expand limited labeled data by exploiting information on unlabeled data to iteratively improve the performance @xcite . recent works have shown transformers @xcite can model high-dimensional distributions of semantic concepts at scale, and several attempts have demonstrated the combination between transformers and rl architecture @xcite @xcite . these works have shown that the transformer's efficiency for modeling beneficial semantic interactions in the given sequence @xcite , which is very enlightening for re task. given the diversity of successful applications of such models @xcite , this paper seeks to investigate their application to sequential re problems formalized as rl, because of the three main advantages of transformers: (i) its ability to model long sequences has been demonstrated in many tasks; (ii) it could perform long-term credit assignment via self-attention strategy, contrary to bellman backups @xcite which slowly propagate rewards and are prone to distractor signals @xcite in q-learning, which could enable transformer-based architecture to still work effectively in the presence of distracting rewards @xcite ; and (iii) it can model a wide distribution of behaviors, enabling better generalization @xcite . hence, inspired by @xcite c c l 2 0 2 3 @xcite , we try to view the rl-based re as a conditional sequence understanding problem. especially, we model the joint distribution of the sequence of states, actions and rewards, and discuss whether generative sequence understanding could can serve as a substitute for traditional rl algorithms in re task. overall, we propose transformer enhanced reinforcement learning (terl), which abstracts rl paradigm as autoregressively sequence understanding and utilize transformer architecture in bert 1 to model text sequences with minimal modification to native transformer's architecture, and we investigate whether the sequence understanding paradigm can perform policy optimization by evaluating terl on rl benchmarks in re task. this enables us to leverage the scalability of the transformer's architecture, as well as the related advancements in pre-training language modeling (such as the bert's series). especially, following the backbone proposed in @xcite , we train transformer architecture on collected experience with a sequence understanding objective for re task, instead of training a policy through conventional rl algorithms @xcite . this transformer is trained to predict next token in a sequence of rewards (forward-cumulative-rewards emphasized here), states, and actions. this paper shows that leveraging transformers can open up another paradigm to solve rl-based re problem. the main differences between this work and previous rl-based re methods, can be concluded as follows: (i) rl is transformed into sequence understanding; (ii) we learn the natural projection from reward and state to action, instead of maximizing cumulative discount rewards or only modeling state and action in conventional behavior cloning paradigm @xcite ; (iii) q/vfunctions are no need to be learned, while we directly model it as a sequence problem, wherein as long as given the expected return, we can get the corresponding action; and (iv) bellman backups or other temporal difference frameworks is no need; in re tasks (even relation and entity joint extraction tasks) with our work, the expected target return is highly correlated with the actual observed return. under certain conditions, the proposed terl could successfully generate sequences that almost completely match the required returns. in addition, we can prompt terl with a higher return than the maximum event available in the dataset, indicating that our terl can sometimes be extrapolated. moreover, the proposed framework can also be used as a plug-in unit for any rl-based re architecture, and be extended to relation and entity joint extraction task @xcite . experimental results show that the proposed terl framework can improve many state-of-the-art rl-based re methods. ## related work relation extraction (re) aims to predict the binary relation between two entities in a sequence of words. recent work leverages deep neural network (dnn) for learning the features among two entities from sentences, and then classify these features into pre-defined relation types @xcite . these methods have achieved satisfactory performance when labeled data is sufficient @xcite , however, it's labor-intensive to obtain large amounts of manual annotations on corpus. hence, few-shot (even zero-shot) re methods gained a lot of attention recently, since these methods require fewer labeled data and could expand limited labeled information by exploiting information on unlabeled data to iteratively improve the performance. wherein, reinforcement learning (rl) based methods have grown rapidly @xcite , which has been widely used in nature language processing (nlp) @xcite @xcite . these methods are all designed with rewards to force the correct actions to be chosen during the model's training procedure. for re task, @xcite proposes a rl strategy to generate the false-positive indicator, where it automatically recognizes false positives for each relation type without any supervised information. @xcite addresses the re task by capturing rich contextual dependencies based on the attention mechanism, and using distributional rl to generate optimal relation information representation. @xcite proposes gradient imitation rl method to encourage pseudo label data to imitate the gradient descent direction on labeled data.for relation and entity joint extraction task, @xcite proposes a hierarchical rl framework which decomposes the whole extraction process into a hierarchy of two-level rl policies for relation extraction and entity extraction, respectively. @xcite applies policy gradient method to model future reward in a joint entity and relation extraction task. (wang 1 other transformer architecture is also applicable. ## methodology this section presents the proposed terl for re task, as summarized in fig. 1 . timestep) embedding token-type embedding token-self embedding + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 𝐨 [sep] 𝐨 [sep] output embedding [sep] 𝐩 [sep] the architecture of terl for re task. ## conclusion in this work, we try to combine transformers and reinforcement learning (rl) based sequence relation extraction (re), and extend transformer paradigm to rl. we design a novel framework (terl) that abstracts rl-based re as a sequence understanding task, which could leverage the simplicity and scalability of the transformer-based architecture for understanding textual sequence, as well as the advancements released by pre-training language modeling (such as the bert/gpt series). moreover, the proposed framework can also be used as a plug-in unit for any rl-based re architecture, and be extended to relation and entity joint extraction task. experimental results show that the proposed terl framework can improve many state-of-the-art rl-based re methods. 0 5 1 0 1 5 2 0 2 5 3 0 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 f 1 k ( k ) h r l + t e r l j r l + t e r l s e q 2 s e q r l + t e r l b i r l + t e r l h r l j r l s e q 2 s e q r l b i r l figure 3: effect of context length k on joint extraction task.