LEGOB ench: Scientific Leaderboard Generation Benchmark (2024)

Abstract: The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate scientific leaderboards. LEGOBench is curated from 22 years of preprint submission data on arXiv and more than 11k machine learning leaderboards on the PapersWithCode portal. We present a language model-based and four graph-based leaderboard generation task configuration. We evaluate popular encoder-only scientific language models as well as decoder-only large language models across these task configurations. State-of-the-art models showcase significant performance gaps in automatic leaderboard generation on LEGOBench. The code is available on GitHub and the dataset is hosted on OSF.

## introduction comparison of results with prior state-of-the-art (sota) is a standard practice in experimental research papers. performance on a task using a specific metric establishes the efficacy of the paper's proposed method. however, one of the primary challenges in scientific research is keeping up with the rapid volume of research progress and staying updated with the latest sota to compare with one's work. the increasing number of manuscripts (depicted by arxiv submissions in appendix a.1) demonstrates the severity of information overload. with the continuous stream of submission, revision, and acceptance timelines of conferences and journals, researchers often struggle to keep up with the latest methods and developments. thus, being acquainted with the latest papers, sieving through the massive set, and deciding which baselines to compare with can be challenging and time-consuming. moreover, the latest papers with novel methods may have low visibility, and upcoming papers may overlook those for result comparison as citations are biased towards old compared to new papers. to address the information overload and to facilitate the comparison with meaningful baseline works, multiple previous works mine scientific tables from papers @xcite @xcite and construct scientific leaderboards @xcite @xcite @xcite . a scientific leaderboard curates performance scores of competitive models against the @xmath0, task, metric>. one of the most actively maintained platforms, paperswith-code (pwc) @xcite , hosts leaderboards in empirical machine learning. figure 6 in appendix a.2 shows a representative leaderboard sample for the image clustering task on the mnist dataset, available in pwc. while leaderboards are helpful for researchers to track the latest models, a majority of leaderboard curation initiatives are manually maintained @xcite , or are dormant @xcite @xcite . hence, the need to automate the generation of leaderboards is imperative. to streamline the process of automating leaderboard generation, we create the arxiv papers' collection (apc), a curated collection of research papers and graph data (citation network and performance comparison network) from arxiv. we also create a dataset sourced from paperswith-code (pwc), consisting of leaderboards mapped with arxiv papers. we combine apc and pwc datasets to develop a benchmark framework called legobench, that facilitates evaluation and assessment of automatic leaderboard generation models. legobench introduces the leaderboard generation task in two configurations, (i) ranking papers based on content and graph @xcite , and (ii) leaderboard entries generation by prompting language models @xcite . our contributions can be summarized as follows: ## dataset we curate two datasets, (i) pwc leaderboards (pwc-ldb) and (ii) arxiv papers' collection (apc), which are utilized for the construction of legobench. pwc-ldb is curated from the papers with code repository 3 and apc is curated from arxiv preprint repository 4 . ## legobench: automatic scientific leaderboard generation benchmark we present legobench, a benchmark specifically developed for the evaluation of scientific leaderboard generation. this benchmark tasks systems with generating a leaderboard in response to a natural language query that specifies a dataset (d), a task (t), and a metric (m), utilizing a collection of arxiv papers' (apc) datasets. to comprehensively assess the capabilities of scientific leaderboard generation systems, we design two tasks, resulting in a total of five configurations. these configurations employ multiple frameworks and use various datasets from the apc to assess the generation of diverse leaderboard formats. the two leaderboard generation tasks are: (i) leaderboard entries generation by prompting language models @xcite , and (ii) ranking papers based on content and graph @xcite . the lgplm task comprises all 9847 leaderboards presented in the ap-ldb dataset. the benchmark for the rpg task comprises 4409 leaderboards for 675 empirical ml tasks on 1697 datasets (we filter out leaderboard having less than three entries that can be mapped to arxiv or our apc dataset, as paper text from arxiv is required for the task). for both tasks, given query @xmath1, t, m>, and arxiv dataset d, the task is to generate a leaderboard l corresponding to the query q. the format of d and l depends on the task configuration. q is a natural language query consisting @xmath2, t, @xmath3.g., list the performance scores of various methods in the mnist dataset for image classification task using metric accuracy). next, we describe each task. ## preliminary baselines and results we present preliminary baselines for each task. ## conclusion we curate two dataset collections, apc and pwc-ldb, to construct legobench benchmark for automatic scientific leaderboard generation task. our apc collections features multiple datasets, including titles, abstracts, and full-text of 1.9m arxiv papers, a citation network consisting of 18m papers, and a performance comparison network of 280k papers extracted from scientific tables. the ap-ldb dataset which presents pwc leaderboards mapped with arxiv data consists of 9.8k leaderboards consisting of 41k <d, t, @xmath4. legobench largely caters to graphbased rankers and language models for leaderboard generation. we design two tasks, with five configurations, to comprehensively evaluate diverse systems for automatic scientific leaderboard generation. across both tasks, we find that existing models severely lack the capabilities to generate scientific leaderboards leveraging paper texts and paper network datasets. this opens up a new avenue for foundation models to focus on, which also helps the community by reducing the overload of comparing and organizing scientific output by generating leaderboards. in addition to the automatic leaderboard generation problem, our proposed datasets and legobench can also be used in traditional tasks such as citation recommendation and intent identification, impact prediction, novelty assessment in review generation, citation count prediction, and venue recommendation for the leaderboard dataset was curated from paper-swithcode (pwc), and any papers missing in the pwc or arxiv dataset (apc) were excluded from the dataset. our rpg task excluded leaderboards with less than three entries and also removed multiple models from the same paper as we formulated it as a paper ranking task. lgplm task, on the other hand, takes into consideration all methods instead of the best performing method. the performance comparison network (ap-pcn) is currently based on the identification of citation patterns in tables in papers. however, it should be noted that not all tables are result comparison tables. similarly, it is not necessary that all papers whose results are compared are included in tables, and sometimes, results can be compared in the text alone. such comparison papers won't be present in our performance comparison network. our pwc-ldb dataset is curated from the paperwithcode repository. an adversary can add incorrect results, leading to poor performance of good ranking models. similarly, adversaries can also add incorrect results to existing leaderboards to favor a specific group or an individual organization. lastly, we rely on pwc to correctly map the scores to the corresponding papers, even if they are reported as baselines in other papers. table entity linker for extracting leaderboards from machine learning publications. in proceedings of the first workshop on information extraction from scientific publications, pages 20-25. xu zhong, elaheh shafieibavani, and antonio jimeno yepes. 2020. image-based table recognition: data, model, and evaluation. in computer vision -eccv 2020: 16th european conference, glasgow, uk, august 23-28, 2020, proceedings, part xxi, page 564-580, berlin, heidelberg. springer-verlag. john ziman. 1969. public knowledge: an essay concerning the social dimension of science. philosophy of science, 36(2).