AIFB - W eb S cience at S em E val-2022 Task 12: Relation Extraction First - Using Relation Extraction to Identify Entities (2022)

Abstract: In this paper, we present an end-to-end joint entity and relation extraction approach based on transformer-based language models. We apply the model to the task of linking mathematical symbols to their descriptions in LaTeX documents. In contrast to existing approaches, which perform entity and relation extraction in sequence, our system incorporates information from relation extraction into entity extraction. This means that the system can be trained even on data sets where only a subset of all valid entity spans is annotated. We provide an extensive evaluation of the proposed system and its strengths and weaknesses. Our approach, which can be scaled dynamically in computational complexity at inference time, produces predictions with high precision and reaches 3rd place in the leaderboard of SemEval-2022 Task 12. For inputs in the domain of physics and math, it achieves high relation extraction macro F1 scores of 95.43% and 79.17%, respectively. The code used for training and evaluating our models is available at: https://github.com/nicpopovic/RE1st

## introduction information extraction systems are a key component in making scientific literature more consumable. with the large amount of scientific works which are constantly being published (e.g., more than 60,000 machine learning papers per year (färber, 2019)), indexing techniques that go beyond keyword searches are becoming more important. while many efforts have focused on the processing of abstracts as a way of building representations of publications @xcite , methods processing full text documents will be needed to accurately capture their contents for use cases such as academic search and recommender systems and scientific impact quantification. the task tackled in this paper @xcite , consisting of linking mathematical symbols to their descriptions in latex documents, is a joint entity and relation extraction task. while earlier work tackled both subtasks sequentially via separate models, more recent approaches tend to use a single joint model @xcite @xcite . in contrast to early approaches, which are based on bi-lstms @xcite @xcite , more recent approaches @xcite ) make use of transformer-based language models, such as bert @xcite . a key challenge in joint models is the computational complexity stemming from pairwise comparisons between entity spans required for relation extraction. previous works tackle this using a span scoring mechanism based on a feed forward neural network, which produces a score indicating the likelihood that a span is in a relation @xcite . relation extraction is then performed on only those spans with the highest scores. for data sets which include span annotations even for entities which are not in any relation, such as docred @xcite , as examined by @xcite , such a scoring mechanism is not necessary, because the entity extraction component of the model can be trained on these annotations. for the task tackled in this paper, complete annotations for entity spans are not provided, making the use of a span scoring mechanism necessary. in this paper, we propose an end-to-end approach for joint entity and relation extraction. the approach is based on a transformer-based language model, following previous work @xcite , but is peculiar in the sense that it incorporates a span scoring mechanism based on dot product similarity which is learned via triplet loss rather than cross entropy loss, making it applicable to datasets which contain annotations only for a subset of all valid entity mention spans. the task tackled in this paper is one of joint entity and relation extraction. this means, given an unannotated text as input, a system needs to (1) return annotations of relevant entity mention spans, (2) perform coreference resolution, (3) entity type classification, and finally (4) relation extraction on the identified spans. the specific task at hand has a number of key features that separate it from similar settings. first, regarding entity extraction, the annotations and, thus, the final scoring are restricted to those entities which participate in relations. this means that a system which correctly identifies all symbols and descriptions in the input will score poorly even on the entity extraction portion of the final benchmark if the relation extraction is incorrect. more importantly from an engineering perspective, the resulting span annotations are incomplete in that they only include a partial set of valid spans for each document. in the entity extraction step we can, therefore, only reliably identify true positives and false negatives, not, however, false positives and true negatives. second, while coreference resolution (i.e., the linking of multiple mentions to a single entity) is part of the task, relation extraction is to be performed on a mention-level rather than the entitylevel. this means that although a system may correctly identify a text span as being the description of a certain symbol, this classification will only be deemed correct in the evaluation if linked to the correct mention of said symbol. as a result, coreference links are interpreted as relations between mentions and thereby as part of the relation extraction subtask, rather than as part of the entity extraction subtask. third, entity types can be reliably inferred from the relations between them, meaning that instances of relations are only found between certain entity types. this feature can be used to inform the design of a system in two ways: either, the task of relation extraction can be simplified by reducing the choices given to a classifier based on the entity types of two spans (i.e., a symbol cannot be the description to another symbol, therefore any such prediction can be disregarded), or the entity type classification can be informed by the relation extraction (i.e., if we identify a span a as the description of another span b, span a must be a description, while span b must be a symbol). ## approach we propose an end-to-end entity and relation extraction system using a transformer-based language model, as illustrated in figure 1. the system consists of 4 modules: (1) the input encoding module tokenizes the input text and produces contextualized embeddings for each token, (2) the soft mention detection module ranks possible token spans by the likelihood with which they contain an entity mention, (3) the relation extraction module extracts relations on a subset of the highest ranked spans from the previous step, and finally (4) the entity type classification module assigns entity types to spans based on the relations detected between them. ## experimental setup for our language model we use scibert @xcite , which is trained on scientific text, via huggingface's transformers library @xcite . for latex preprocessing (see section 3.1) we use pylatexenc foot_2 . as our optimizer, we use adamw @xcite with learning @xmath0, 5e-5, 7e-5], a linear warmup of 1 epoch followed by a linear decay to zero, for a total of 60 epochs foot_3 , a batch size of 4, and apply gradient clipping with a max norm of 1. during training, we randomly downsample the amount of candidate spans for soft mention detection to 1000, while ensuring that all labeled spans are included. during training and development set evaluation, we set k, the number of spans to perform relation classification on, to 50, as preliminary experiments showed this value to yield a good compromise between model performance and training time. for test set evaluation we increase k to 400. training takes approximately 10 hours on a single nvidia v100 gpu using mixed precision. we perform early stopping based on the micro f 1 score for relation extraction on the development set. we train each hyperparameter configuration 3 times using different random seeds and report the median and standard deviation for each metric. as a result of the different combinations of preprocessing and mean-/max-pooling, we examine the performance of 4 configurations on the test set. for our evaluation, we report the micro f 1 scores for ner metrics as used in semeval-2013 task 9.1 (segura-bedmar et al., 2013) 5 . for relation extraction we report micro precision, recall and f 1 scores, unless otherwise indicated. ## conclusion in this paper, we present an end-to-end joint entity and relation extraction approach for linking mathematical symbols to their descriptions in latex documents. our model appears to be sensitive to the domain of the input documents, achieving high macro f 1 scores of 95.43% and 79.17% for physics and math content, respectively, while achieving macro f 1 scores of only 19.23% and 13.84% for computer science and economics related content. we find that the model's predictions are higher in precision than in recall. we perform a detailed error analysis and identify cross-domain generalization as the most critical problem to tackle in future work.