TELIN : Table Entity LIN ker for Extracting Leaderboards from Machine Learning Publications (2022)

Abstract: Tracking state-of-the-art (SOTA) results in machine learning studies is challenging due to high publication volume. Existing methods for creating leaderboards in scientific documents require significant human supervision or rely on scarcely available LaTeX source files. We propose Table Entity LINker (TELIN), a framework which extracts (task, model, dataset, metric) quadruples from collections of scientific publications in PDF format. TELIN identifies scientific named entities, constructs a knowledge base, and leverages human feedback to iteratively refine automatic extractions. TELIN identifies and prioritizes uncertain and impactful entities for human review to create a cascade effect for leaderboard completion. We show that TELIN is competitive with the SOTA but requires much less human annotation.

## introduction advances in the field of machine learning (ml) are typically evidenced by producing better empirical results on benchmark datasets. with over 334k ai papers published in 2021 @xcite , automated approaches to extract and categorize empirical results would help practitioners track progress in the field. leaderboard extraction is challenging because there is no universal lexicon, taxonomy, or structure for reporting empirical results in ml publications. new benchmark datasets and tasks are frequently introduced, and established datasets are updated or repurposed for new tasks or metrics. for example, a publication with a table containing numerical results on "imagenet" could refer to any particular lsvrc challenge year @xcite @xcite @xcite @xcite , task (e.g., classification, object detection, localization), number of classes, dataset version, evaluation metric, etc. these necessary details could be specified in table header cells, table captions, paragraphs referencing the table, or elsewhere in the paper. additionally, ml publications are often only available in pdf format which infrequently explicitly encodes the underlying document paragraph and table structures. prior work on scientific leaderboard construction suffer from the following weaknesses: our evaluation on the pwcleaderboards dataset @xcite shows that telin uses significantly less human supervision on pdf inputs to achieve comparable accuracy with the state-of-the-art leaderboard extraction system, axcell @xcite , which requires l a t e x source file inputs. while their accuracy is similar, we conclude that telin is likely a more practical tool for leaderboard extraction since it requires less human annotation and can be applied to any publication avaiable in pdf. since spert is trained on full sentences, it performs poorly on short non-sentence text such as table header cell text. therefore, telin takes the nes from the main text and compares them with table cell text and propagates ne labels for closely matching text. the implementation of computing character level similarity ratio is able to draw comparison between acronyms. the cosine similarity between the sentence representations indicates how close the strings are semantically. note that common entities, such as accuracy (as metric) and coco (as dataset), do not automatically belong to this category. the proposed design of this task is inspired by identifying influential nodes in a network @xcite @xcite . first, we compute the uncertainty of a cell by calculate the entropy of the predicted entity type distribution: next, we construct a heterogeneous network for the purpose of computing the potential of a cell to cause a cascade. each confirmed entity is a node and edges are formed when two entities appear in the same table header row/col. the "spreading ability" @xcite of a cell is computed as: finally, the influence score of an entity e v can be acquired by: telin selects tables including the entities with the highest influence scores for human review. the users are able to confirm or correct the types of the entities on a row/column basis. the user can also label any useful entities in the caption of the table. ## experiments and results we evaluate telin's end-to-end performance on task, dataset, metric, score (tdms) quadruple extraction on the pwcleaderboards @xcite task and compare it to the state-of-the-art ax-cell model @xcite . we select axcell as our main competitor due to its superiority against other existing work @xcite . pwcleaderboards include 731 papers and 3,445 leaderboards, which include the unique tdms quadruples in every paper. we follow kardas et al. for evaluation metrics. we also investigate the performance improvement from the human feedback phase. ## discussion while telin presents promising performance, it still does not exceed the state-of-the-art accuracy in extracting leaderboards from machine learning research papers. our method relies on the propagation of discoveries from one paper to another. the relatively small data size (731 papers) of pwcleaderboard dataset limits the capability of telin. we will investigate whether introducing more data helps the performance of telin. moreover, unlike existing studies relying on taxonomy of leaderboards known in advance, telin operates without any assumptions of taxonomy. we are interested in analyzing the capacity of telin for novel taxonomy discovery. extracting leaderboards from the scientific papers on the web is an example of integrating artificial intelligence in conceptual modeling @xcite . conceptual modeling is a vessel for humans to transform the noise in the nature to structured or semi-structured presentations. while automatic machine extraction has been utilized to collect and organize data from a wide variety of sources in conceptual modeling @xcite @xcite , the role of deep learning and artificial intelligence remains understudied in this field. the design of telin is a demonstration of involving artificial intelligence to facilitate conceptual modeling. we hope this effort will invite future studies in this domain. @xcite identifying influential nodes in complex networks with community structure. knowledge-based systems, 42:74-84.