GAD e P o: Graph-Assisted Declarative Pooling Transformers for Document-Level Relation Extraction (2024)

Abstract: Document-level relation extraction typically relies on text-based encoders and hand-coded pooling heuristics to aggregate information learned by the encoder. In this paper, we leverage the intrinsic graph processing capabilities of the Transformer model and propose replacing hand-coded pooling methods with new tokens in the input, which are designed to aggregate information via explicit graph relations in the computation of attention weights. We introduce a joint text-graph Transformer model and a graph-assisted declarative pooling (GADePo) specification of the input, which provides explicit and high-level instructions for information aggregation. GADePo allows the pooling process to be guided by domain-specific knowledge or desired outcomes but still learned by the Transformer, leading to more flexible and customisable pooling strategies. We evaluate our method across diverse datasets and models and show that our approach yields promising results that are consistently better than those achieved by the hand-coded pooling functions.

## introduction document-level relation extraction is an important task in natural language processing, which involves identifying and categorising meaningful relationships between entities within a document, as exemplified in figure 1 . this task is foundational to many applications, including knowledge base population and completion @xcite , information retrieval and extraction @xcite , question answering @xcite and sentiment analysis @xcite , to name a few. standard methods that approach this challenge generally employ pretrained text-based encoders @xcite @xcite , which are responsible for capturing the nuances of information contained in the entity mentions and their contextual surroundings. previous successful methods often then use hand-coded pooling heuristics to aggregate the information learned by the encoder, with some aimed at creating entity representations, while others directly exploiting the pattern of attention weights to capture context aware relations between entity mentions @xcite @xcite . these pooling heuristics can be very effective at leveraging the information in a pretrained encoder. however, as shown in @xcite ; @xcite ; @xcite ; @xcite , the selection of an appropriate pooling function can be model-dependent, task-specific, resource-intensive and time-consuming to determine, thereby limiting flexibility. in this paper, we address these issues with a new approach where we leverage the intrinsic graph processing capabilities of the transformer model @xcite , leveraging insights from the work of @xcite ; @xcite ; @xcite ; @xcite . they argue that attention weights and graph relations are functionally equivalent and show how to incorporate structural dependencies between input elements by simply adding relation features to the attention functions. transformers easily learn to integrate these relation features into their pretrained attention functions, resulting in very successful graphconditioned models @xcite @xcite . given this effective method for integrating explicit graphs with pretrained attention functions, we propose to use the attention function itself for aggregation. we replace the rigid pooling methods with new tokens which act as aggregation nodes, plus explicit graph relations which steer the aggregation. we introduce a joint text-graph transformer model and a graph-assisted declarative pooling (gadepo) method foot_0 that leverages these special tokens and graph relations, to provide an explicit high-level declarative specification for the information aggregation process. by integrating these graphs in the attention functions of a pretrained model, gadepo exploits the pretrained embeddings and attention patterns but still has the flexibility of being trained on data. this enables the pooling to be guided by domain-specific knowledge or desired outcomes but still learned by the transformer, opening up a more customisable but still data-driven relation extraction process. we evaluate our method across diverse datasets and models commonly employed in documentlevel relation extraction tasks, and show that our approach yields promising results that are consistently better than those achieved by the hand-coded pooling functions. ## related work in recent studies, the scope of relation extraction has been expanded to include not only individual sentences but entire documents. this extension, known as document-level relation extraction, presents a more realistic and challenging scenario as it seeks to extract relations both within sentences and across multiple sentences @xcite . transformer-based @xcite models have shown great potential in addressing this task. @xcite and @xcite show that the bilstm-based @xcite baselines lack the capacity to model complex interactions between multiple entities. they propose a more robust approach, which consists of using the pretrained bert @xcite model and a two-step prediction process, i.e., first identifying if a link between two entities exists, followed by predicting the specific relation type. gain @xcite leverages bert as a text encoder and gcns @xcite to process two types of graphs, one at mention level and another at entity level, showing notable performance in inter-sentence and inferential scenarios. mohammadshahi and henderson (2020, 2021 ) propose the g2gt model and show how to leverage the intrinsic graph processing capabilities of the transformer model by incorporating structural dependencies between input elements as features input to the self-attention weight computations. ssan @xcite leverages this idea and considers the structure of entities. it employs a transformation module that creates attentive biases from this structure to regulate the attention flow during the encoding phase. docunet @xcite reformulates the task as a semantic segmentation problem. it employs a u-shaped segmentation module and an encoder module to capture global interdependencies and contextual information of entities, respectively. pl-marker @xcite introduces a method that takes into account the interplay between spans via a neighbourhood-oriented and subject-oriented packing approach, highlighting the importance of capturing the interrelation among span pairs in relation extraction tasks. sais @xcite explicitly models key information sources such as relevant contexts and entity types. it improves extraction quality and interpretability, while also boosting performance through evidence-based data augmentation and ensemble inference. kd-docre @xcite proposes a semisupervised framework with three key components. firstly, an axial attention module enhances performance in handling two-hop relations by capturing the interdependence of entity pairs. secondly, transformer encoder (atlop) text graph joint text-graph transformer encoder (gadepo) classifier classifier previous method proposed method figure 2 : comparison between the previous method atlop (left) and the proposed method gadepo (right), illustrating the document in figure 1 containing two entities (red and blue), each with two mentions. in atlop, the mentions' encoder outputs are aggregated into entity representations h e , and the encoder's attention weights are used to identify which outputs to aggregate for entity-pair representations c (s,o) . in gadepo, the textual input is extended to include the graph special @xmath0, and explicit directional graph relations specify their associated mentions. a joint text-graph transformer model is then used to encode this declarative pooling specification graph and compute the relevant aggregations. an adaptive focal loss solution addresses the class imbalance issue. lastly, the framework employs knowledge distillation to improve robustness and overall effectiveness by bridging the gap between human-annotated and distantly supervised data. dreeam @xcite is a method designed to enhance document-level relation extraction by addressing memory efficiency and annotation limitations in evidence retrieval. it employs evidence as a supervisory signal to guide attention and introduces a self-training strategy to learn evidence retrieval without requiring evidence annotations. sais @xcite , kd-docre @xcite , and dreeam @xcite have been built upon the foundations of atlop @xcite . atlop introduces two innovative techniques, adaptive thresholding, and localised context pooling, to address challenges in multi-label and multi-entity problems. adaptive thresholding employs a learnable entities-dependent threshold, replacing the global threshold used in previous approaches for multi-label classification @xcite @xcite . localised context pooling leverages the attention patterns of a pretrained language model to identify and extract relevant context crucial for determining the relation between entities, using specific hand-coded pooling functions. ## background the foundational work of atlop @xcite has been the basis of many state-of-the-art (sota) models @xcite @xcite . given the problems with handcoded pooling functions, discussed in section 1, we aim to provide a new baseline that can serve as the foundation for future sota models. for this reason, we evaluate our proposed models by comparing them to this established baseline. our goal is to demonstrate that our method not only achieves results comparable to or better than atlop, but also offers a novel approach which addresses its limitations. to provide a better understanding of atlop and its components, we present a detailed breakdown in the left portion of figure 2 , which we elaborate on in this section. ## conclusion in this paper we proposed a novel approach to document-level relation extraction, challenging the conventional reliance on hand-coded pooling functions for information aggregation. our method leverages the power of transformer models by incorporating explicit graph relations as instructions for information aggregation. by combining graph processing with text-based encoding, we introduced the graph-assisted declarative pooling (gadepo) specification, which allows for more flexible and customisable specification of pooling strategies which are still learned from data. we conducted evaluations using diverse datasets and models commonly employed in documentlevel relation extraction tasks. the results of our experiments demonstrated that our approach achieves promising performance that is comparable to or better than that of hand-coded pooling functions. this suggests that our method can serve as a viable basis for other relation extraction methods, providing a more adaptable and tailored approach. in particular, recent methods have improved performance by exploiting information about evidence, which can naturally be incorporated in our graph-based approach.