S 2ab EL : A Dataset for Entity Linking from Scientific Tables (2023)

Abstract: Entity linking (EL) is the task of linking a textual mention to its corresponding entry in a knowledge base, and is critical for many knowledge-intensive NLP applications. When applied to tables in scientific papers, EL is a step toward large-scale scientific knowledge bases that could enable advanced scientific question answering and analytics. We present the first dataset for EL in scientific tables. EL for scientific tables is especially challenging because scientific knowledge bases can be very incomplete, and disambiguating table mentions typically requires understanding the paper’s text in addition to the table. Our dataset, Scientific Table Entity Linking (S2abEL), focuses on EL in machine learning results tables and includes hand-labeled cell types, attributed sources, and entity links from the PaperswithCode taxonomy for 8,429 cells from 732 tables. We introduce a neural baseline method designed for EL on scientific tables containing many out-of-knowledge-base mentions, and show that it significantly outperforms a state-of-the-art generic table EL method. The best baselines fall below human performance, and our analysis highlights avenues for improvement.

## introduction entity linking (el) is a longstanding problem in natural language processing and information extraction. the goal of the task is to link textual mentions to their corresponding entities in a knowledge base (kb) @xcite , and it serves as a building block for various knowledge-intensive applications, including search engines @xcite , question-answering systems @xcite , and more. however, existing el methods and datasets primarily focus on linking mentions * work done during an internship at ai2. from free-form natural language @xcite @xcite . some consider tabular data, but focus on tables from the general domain @xcite @xcite . despite significant research in el, there is a lack of datasets and methods for el in scientific tables. linking entities in scientific tables holds promise for accelerating science in multiple ways: from augmented reading applications that help users understand the meaning of table cells without diving into the document @xcite to automated knowledge base construction that unifies disparate tables, enabling complex question answering or hypothesis generation @xcite . el in science is challenging because the set of scientific entities is vast and always growing, and existing knowledge bases are highly incomplete. a traditional "closed world" assumption often made in el systems, whereby all mentions have corresponding entities in the target kb, is not realistic in scientific domains. it is important to detect which mentions are entities not yet in the reference kb, referred to as outkb mentions. even for human annotators, accurately identifying whether a rarely-seen surface form actually refers to a rarely-mentioned long-tail inkb entity or an outkb entity requires domain expertise and a significant effort to investigate the document and the target kb. a further challenge is that entity mentions in scientific tables are often abbreviated and opaque, and require examining other context in the caption and paper text for disambiguation. an example is shown in figure 1 . in this paper, we make three main contributions. first, we introduce s2abel, a high-quality humanannotated dataset for el in machine learning results tables. the dataset is sufficiently large for training and evaluating models on table el and relevant sub-tasks, including 52,257 annotations of appropriate types for table cells (e.g. method, dataset), figure 1 : part of a table in opt: open pre-trained transformer language models @xcite showing relevant context that must be found for entity mentions in the table, and a portion of el results mapping table mentions to the paperswithcode kb. 9,565 annotations of attributed source papers and candidate entities for mentions, and 8,429 annotations for entity disambiguation including outkb mentions. to the best of our knowledge, this is the first dataset for table el in the scientific domain. second, we propose a model that serves as a strong baseline for each of the sub-tasks, as well as end-to-end table el. we conduct a comprehensive comparison between our approach and existing approaches, where applicable, for each sub-task. our method significantly outperforms turl @xcite , a state-of-the-art method closest to the table el task, but only designed for generaldomain tables. we also provide a detailed error analysis that emphasizes the need for improved methods to address the unique challenges of el from scientific tables with outkb mentions. ## entity linking in scientific tables our entity linking task takes as input a reference kb (the papers with code 1 taxonomy in our experiments), a table in a scientific paper, and the table's surrounding context. the goal is to output an entity from the kb for each table cell (or "outkb" if none). we decompose the task into several subtasks, discussed below. we then present s2abel, the dataset we construct for scientific table el. ## method in this section, we describe our approach for representing table cells, papers, and kb entities, as well as our model design for performing each of the sub-tasks defined in section 3.1. cell representation -for each table cell in a document, we collect information from both document text and the surrounding table. top-ranked sentences were retrieved using bm25 @xcite as context sentences, which often include explanations and descriptions of the table cell. the surrounding table captures the row and column context of the cell, which can offer valuable hints, such as the fact that mentions in the same row and column usually refer to the same type of entities. more details about cell representation features are in paper representation -for each referenced paper, we extract its index in the reference section, the last name of the first author, year, title, and abstract. index, author name, and year are helpful for identifying inline citations (which frequently take the form of the index in brackets or the author and year in parens). additionally, the title and abstract provide a summary of a paper which may contain information on new concepts it proposes. kb entity representation -to represent each entity in the target kb, we use its abbreviation, full name, and description from the kb, if available. the abbreviation and full name of an entity are crucial for capturing exact mentions in the text, while the description provides additional context for the entity @xcite . cell type classification -we concatenate features of cell representation (separated by special tokens) and input the resulting sequence to the pretrained language model scibert @xcite . for each token in the input sequence, we add its word embedding vector with an additional trainable embedding vector from a separate embedding layer to differentiate whether a token is in the cell, from context sentences, etc. (subsequent mentions of scibert in this paper refer to this modified version). we pass the average of the output token embeddings at the last layer to a linear output layer and optimize for cross entropy loss. however, because the majority of cells in scientific tables pertain to experimental statistics, the distribution of cell types is highly imbalanced (as shown in appendix a). to address this issue, we oversample the minority class data by randomly shuffling the context text extracted from the paper for those cells at a sentence level. attributed source matching -to enable contextualization between cell context and a potential source, we combine the representations of each table cell and potential attributed source in the document as the input to a scibert followed by a linear output layer. we optimize for the binary cross entropy loss, where all non-attributed sources in the document are used as negative examples for a cell. the model output measures the likelihood that a source should be attributed to given a table cell. candidate entity retrieval -we design a method that combines candidates retrieved by two strategies: (i) dense retrieval (dr) @xcite ) that leverages embeddings to represent latent semantics of table cells and entities, and (ii) attributed source retrieval (asr) which uses the attributed source information to retrieve candidate entities. for dr, we fine-tune a bi-encoder architecture @xcite with two separate sciberts to optimize a triplet objective function. the model is only trained on cells whose gold referent entity exists in the kb. top-ranked most similar entities based on the bm25f algorithm @xcite foot_5 in elasticsearch are used as negative examples. for each table cell t i , the top-k nearest entities o i dr in the embedding space with ranks are returned as candidates. for asr, we use the trained asm model to obtain a list of papers ranked by their probabilities of being the attributed source estimated by the model. the candidate entity sequence o i asr is constructed by fetching entities associated with each potentially attributed paper in ranked order using the paper-relatesto-entity relations in pwc. only entities of the same cell type as identified in ctc are retained. note that including entities associated with lowerranked papers mitigates the errors propagated from the asm model and the problem of imperfect entity and relation coverage that is common in real-world kbs. we finally interleave o i dr and o i asr until we reach a pre-defined entity set size k. entity disambiguation with outkb identification -given a table cell and its entity candidates, we fine-tune a cross-encoder architecture @xcite with a scibert that takes as input the fused cell representation and entity representation, followed by a linear output layer. we optimize for bce loss using the same negative examples used in cer training. the trained model is used to estimate the probability that a table cell matches an entity. if the top-ranked entity for a cell has a matching likelihood lower than 0.5, then the cell is considered to be outkb. work by evaluating their performance on sub-tasks of our dataset (section 5.1). additionally, we report the performance of the end-to-end system to provide baseline results for future work (section 5.2). finally, to understand the connection and impact of each sub-task on the final el performance, we conducted a component-wise ablation study (section 5.3). this study provides valuable insights into the difficulties and bottlenecks in model performance. the experiments are designed to evaluate the performance of methods in a cross-domain setting (following the setup in @xcite , where training, validation, and test data come from different disjoint topics. this ensures that the methods are not overfitting to the particular characteristics of a topic and can generalize well to unseen data from different topics. ## conclusion in this paper, we present s2abel, a high-quality human-annotated dataset for entity linking in machine learning results tables, which is, to the best of our knowledge, the first dataset for table el in the scientific domain. we propose a model that serves as a strong baseline for the end-to-end table el task with identification of outkb mentions, as well as for each of the sub-tasks in the dataset. we show that extracting context from paper text gives a significant improvement compared to methods that use only tables. identifying attributed source papers for a concept achieves higher recall@k compared with dense retrieval for candidate entity generation. however, the best baselines still fall far below human performance, showing potential for future improvement.