Synthetic Context with LLM for Entity Linking from Scientific Tables (2024)

Abstract: Tables in scientific papers contain crucial information, such as experimental results.Entity Linking (EL) is a promising technology that analyses tables and associates them with a knowledge base.EL for table cells requires identifying the referent concept of each cell while understanding the context relevant to each cell in the paper. However, extracting the relevant context from the paper is challenging because the relevant parts are scattered in the main text and captions.This study defines a rule-based method for extracting broad context from the main text, including table captions and sentences that mention the table.Furthermore, we propose synthetic context as a more refined context generated by large language models (LLMs).In a synthetic context, contexts from the entire paper are refined by summarizing, injecting supplemental knowledge, and clarifying the referent concept.We observe this approach improves accuracy for EL by more than 10 points on the S2abEL dataset, and our qualitative analysis suggests potential future works.

## introduction information analysis of scientific papers has numerous applications in accelerating science, such as paper retrieval, reading assistance, and automatic knowledge base construction. in particular in information science, crucial information, such as experimental results, evaluation datasets, tasks, and evaluation metrics, is often recorded in tables within the papers. thus, the analysis of table information is an important research field. for table analysis, entity linking (el) that associates table cells in scientific papers with a knowledge base (kb) is an important task, and various methods and datasets have been proposed for this purpose @xcite @xcite . s2abel @xcite is a large-scale evaluation dataset for el targeting tables in papers for the machine learning field. in table cell cell context from the paper cola the corpus of linguistic acceptability is a binary [...] 2019 ). given a target table cell (e.g., "cola"), a model seeks to link it to the corresponding entity (e.g., "/dataset/cola") in papers with code by considering the contexts in the paper related to the cell. the dataset, each table cell is linked to an entity defined in papers with code (pwc) foot_0 , a free and open kb in the scientific domain, as illustrated in figure 1 . to correctly link the target table cell "cola" to the corresponding entity "/dataset/cola" in pwc, a model needs to understand the concept of cola from the contexts scattered in the main text, captions, and references. however, extracting such contexts relevant to each cell from a paper has three technical challenges. (i) relevant contexts for a cell text are scattered in an entire paper, and mentions are often abbreviated or paraphrased, (ii) the context or explanation for a referent concept of a cell can be insufficient, and (iii) general words such as "ours," "baseline," and "all" are often used in cell texts, and the referent is ambiguous. an example for the first, in the paper of the evolved transformer @xcite , a cell text "et perp" is interpreted as "the perplexity achieved by the evolved transformer", although the term "et perp" does not appear in the main text of the paper. second, explanations for well-known methods such as lstm are often omitted, and thus, sufficient contexts are unavailable in the paper itself. third, the cell text "all" stands for the entire dataset. identifying the dataset requires understanding the context of the main text. however, the word "all" is general and frequently used in irrelevant contexts in the paper. to address these challenges, we propose a data synthesis method for providing an el model with supplemental contexts for table cells by using large language models (llms) such as chat-gpt (openai, 2023) and llama2 @xcite . llms, acquiring specialized knowledge through pre-training, can be utilized as knowledge bases @xcite . they also demonstrate high performance in a zero-shot setting for abbreviation expansion @xcite and coreference resolution @xcite . therefore, we expect that they can generate additional information for table cells, which was not mined by previous methods. in experiments, we use context data for el obtained by our refined rule-based method and llmbased method and confirm consistent improvements by both methods over the baseline proposed in s2abel, including a 10-point improvement in accuracy for el. furthermore, by utilizing synthetic context, we improved the accuracy of a subtask that extracts relevant papers for a given table cell by more than 12 points in terms of top-5 accuracy. this result demonstrates the effectiveness of our approach in linking table cells to entities without relying on existing knowledge bases. our qualitative analysis also reveals that synthetic context data captures better supplemental information through context completion and knowledge completion by llms foot_1 . ## method in this study, we aim to improve the overall accuracy of el by enhancing the context of cells, which is the most important feature for both asm and ed tasks in the baseline method. to this end, we first improve the rule-based context extraction method and then generate synthetic data to refine the context. ## experiments the experiments follow the setup of s2abel, where training, validation, and test data are created from different topics. the results below represent the cross-validation average on the topics in s2abel. this allows us to compare the generalization performance of the models without overfitting to specific topics. ## conclusion in this study, we proposed new context extraction methods from the main text for entity linking of table cells of scientific papers. first, we propose a rule-based context extraction method (raw context) to collect broad context from a paper. then, we introduce the synthesized data using an llm to refine the raw context (synthetic context). by employing raw context and synthetic context, we improved the accuracy of entity linking by more than 10 points. in the qualitative analysis, we observe the llm refines raw context by supplementing context and completing information. practicality. our proposed method outperforms the existing method and achieves 53.1% @top1 accuracy in el, 55.6% for @top1, and 75.7% for @top5 in asm. these results suggest that fully automating the linking of table cells in papers is still challenging; however, it could potentially be used to assist manual annotation for table cells. model bias. synthetic context depends on llms' generative capabilities and knowledge, making it susceptible to the model's bias. this study targets only english-language papers in the machine learning domain, which may limit generalization to other languages and fields.