ITER : Iterative Transformer-based Entity Recognition and Relation Extraction (2024)

Abstract: When extracting structured information from text, recognizing entities and extracting relationships are essential. Recent advances in both tasks generate a structured representation of the information in an autoregressive manner, a time-consuming and computationally expensive approach. This naturally raises the question of whether autoregressive methods are necessary in order to achieve comparable results. In this work, we propose ITER, an efficient encoder-based relation extraction model, that performs the task in three parallelizable steps, greatly accelerating a recent language modeling approach: ITER achieves an inference throughput of over 600 samples per second for a large model on a single consumer-grade GPU. Furthermore, we achieve state-of-the-art results on the relation extraction datasets ADE and ACE05, and demonstrate competitive performance for both named entity recognition with GENIA and CoNLL03, and for relation extraction with SciERC and CoNLL04.

## introduction in recent years, there has been a shift towards the use of autoregressive methods in many common natural language processing (nlp) tasks. in parallel, there has been an increasing focus on approaching nlp tasks such as relation extraction or (nested) named entity recognition as structured prediction problems. given a sequence of textual input, a given model autoregressively generates outputs that encode the structure contained in the input, providing flexibility since the source and target vocabularies do not need to have any commonalities. flattening the output structure into a single string, preserving the structure information in the input, and using an autoregressive model to learn to generate this adapted target language @xcite , is an implicit approach known to work well across task boundaries @xcite . however, representing the structured output as a string introduces additional complexity when modeling intra-structure dependencies @xcite . more recently, @xcite proposed restricting the autoregressive model to explicit generation of the output structure. however, since inference cannot be parallelized across the sequence dimension, language modeling approaches are prone to low throughput, especially as model size increases @xcite . to counteract this effect, a smaller output sequence length is of critical importance. for asp @xcite , the output is always at least as long as the input, leading to poor real-world performance (see eq. 1 in appendix c). while scaling the model size from hundreds of millions to billions of parameters provides performance gains for @xcite , this scaling may become infeasible in terms of both computational requirements and throughput when using these large models in production. this raises the natural question of whether a non-autoregressive process capable of generating such an output structure can achieve similar performance while addressing the aforementioned limitations of language modeling approaches. this paper presents iter, an encoder-only transformer-based relation extraction model that addresses the limitations of state-of-the-art architectures and shows that the structured prediction problem can be approached without language modeling goals. in summary, the main contributions we have made are as follows: ## related work the goal of relation extraction @xcite , sometimes also referred to as end-to-end relation extraction or joint entity and relation extraction, is to identify the names and types of named entities, within a given text, as well as to classify the relationships between these entities @xcite . initial approaches to relation extraction have been to split the task into named entity recognition (ner) and relation classification, where the named entities are identified first, while the relationships between the found named entities are then classified in a second, separate stage that is being learned independently. this pipeline-based approach is known to be prone to error propagation @xcite . because of this known limitation, joint approaches modeling both tasks simultaneously have been introduced and have shown promising results @xcite @xcite . ## approach we base iter on the work of @xcite to replace the autoregressive component of their approach with our inference process, several modifications to the structured prediction are necessary. we will divide the process into three basic steps: to allow an efficient implementation of our model, each step can be individually parallelized across the sequence dimension. first, we define the set of structure-building actions a: @xmath0, ] } our model must be allowed to perform both [ (i.e., marking the possible beginning of a span) and ] (i.e., ending a span) actions at the same time, in order not to lose model expressiveness. otherwise, it will not be able to correctly classify single-token spans foot_1 . therefore, the structure-building actions a n ⊆ a performed at position n must now be a subset of a, to allow for this behavior. this change is reflected in the definition of the optimal structured output y * that our model will learn to generate: y * ∈ × n @xmath1. the possible actions y n to be performed at step n are defined as follows: to properly handle two or more entities ending at the same position, the bracket-pairing actions are also present in y n as a subset of all possible b n actions. this change comes in combination with two adjustments to the definition of b n itself: at position m, iter is allowed to pair [ actions at positions @xmath2, circumventing single-token named entity issues (1, eq. 2), and each such individual pairing is allowed to have its own named entity type @xmath3, eq. 2). ## experimental results in this section, we give an overview of the datasets used (section 4.1) followed by a discussion of the results from our experiments (section 4.2). details of the hyperparameter search we performed can be found in appendix d. ## conclusion and future work in this work, we propose iter, an efficient, wellperforming relation extraction model. we translate the autoregressive process of liu et al. into a constant, easily parallelizable three-step process, while maintaining the same level of performance and increasing model throughput, especially for longer sequence lengths. our model allows us to perform any kind of structured prediction task with maximum throughput (figure 3 ) and state-of-the-art performance. on ace05, we set a new state-ofthe-art of 71.9 f1 and 91.9 for re and ner. for the biomedical dataset ade, we set a new stateof-the-art for ner (92.2) and re+ (85.6). in our experiments, we find that using the encoders of generative t5 models can yield model performance advantages over using discriminative models such as bert and albert, while also outperforming these models in terms of throughput. the only exception to this finding is deberta, where f1 performance is competitive (re+) or even better (ner). however, in terms of model throughput, the t5 family models still outperform deberta. we also highlight the advantages of encoder-based models over generative approaches in terms of model throughput, with iter being up to 42 times faster than the autoregressive state-of-the-art model asp. to the best of our knowledge iter is the first model to successfully use only the encoder of an autoregressively trained plm in this domain, inspiring further research in this direction. one area of future work may be to develop a large (synthetic) dataset that allows evaluation of the expressiveness of re models with respect to nested entities, since nested entities are a real-world problem, but existing datasets contain only a small fraction of such examples (see