Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards (2024)

Abstract: Scientific leaderboards are standardized ranking systems that facilitate evaluating and comparing competitive methods. Typically, a leaderboard is defined by a task, dataset, and evaluation metric (TDM) triple, allowing objective performance assessment and fostering innovation through benchmarking. However, the exponential increase in publications has made it infeasible to construct and maintain these leaderboards manually. Automatic leaderboard construction has emerged as a solution to reduce manual labor. Existing datasets for this task are based on the community-contributed leaderboards without additional curation. Our analysis shows that a large portion of these leaderboards are incomplete, and some of them contain incorrect information. In this work, we present SciLead, a manually-curated Scientific Leaderboard dataset that overcomes the aforementioned problems. Building on this dataset, we propose three experimental settings that simulate real-world scenarios where TDM triples are fully defined, partially defined, or undefined during leaderboard construction. While previous research has only explored the first setting, the latter two are more representative of real-world applications. To address these diverse settings, we develop a comprehensive LLM-based framework for constructing leaderboards. Our experiments and analysis reveal that various LLMs often correctly identify TDM triples while struggling to extract result values from publications. We make our code and data publicly available.

## introduction the comparison of state-of-the-art scientific methods has become a major challenge for the research community due to the rapidly growing number of scientific publications @xcite @xcite . for example, around 100 papers are submitted daily to the arxiv pre-print repository under the computation and language category foot_2 alone. to facilitate monitoring of research progress and comparison of sota model performance, scientific leaderboard platforms have been introduced, such as nlp-progress foot_4 or paperswithcode foot_5 (pwc). a scientific leaderboard is typically formalized as a task, dataset, and evaluation metric (tdm) triple, ranking performance (result) of competitive methods against the triple. however, the majority of these leaderboards are manually curated and maintained, which is infeasible over time due to the ever-expanding number of publications. automatic leaderboard construction has been proposed to offer a solution @xcite , which aims to extract and compare performance from scientific publications against benchmarks. previous studies have released datasets based on community-contributed platforms with minimum quality control, such as normalizing the tdm entities across different leaderboards @xcite @xcite @xcite @xcite . however, these annotations are limited by individual researchers' varying research interests, resulting in incomplete coverage of papers within a leaderboard and a lack of tdms from each paper (see appendix a for a detailed example). such partial annotations hinder the evaluation of automatic leaderboard construction systems. furthermore, all previous work on leaderboard construction mentioned above has formulated the task as matching the extracted tdm triples from a paper against a pre-defined tdm triple taxonomy. this assumption, however, falls short in capturing the dynamic nature of real-world scenarios. in reality, the landscape of leaderboards is constantly evolving, with new ones emerging as a natural consequence of ongoing research and innovation. to this end, we introduce scilead, a manuallycurated scientific leaderboard dataset, including 27 leaderboards derived from 43 nlp papers. to avoid the pitfall of community-contributed leaderboards with incomplete and inaccurate information, we exhaustively annotate all unique tdm triples and the corresponding results from each paper and construct a complete leaderboard set for these papers. as illustrated in figure 1 , we propose a threestage framework for constructing leaderboards, harnessing the power of large language models (llms) through retrieval-augmented generation (rag) prompting to streamline the process: (i) extracting task, dataset, metric and result (tdmr) tuples from individual papers; (ii) normalizing extracted tdm triples to the existing pre-defined tdm triple taxonomy or creating new leaderboards; and (iii) ranking papers by their corresponding performance to construct leaderboards. in the second step, we design three experimental settings that mimic diverse real-world scenarios where we need to update existing leaderboards or create new ones from scratch. specifically, the tdm triples are either fully specified, partially specified, or completely unknown during the construction process. notably, the task becomes increasingly challenging in the second and third settings, which have been neglected by all prior work, as it demands reasoning across all existing leaderboards and generating novel ones. following previous work, we evaluate tdm extraction using exact match metrics. we further propose various metrics to evaluate the constructed leaderboards, including coverage of correctly assigned publications to the leaderboards and their result values. we also compare the rankings of the gold and constructed leaderboards. in summary, this work presents scilead, a manually curated dataset of scientific leaderboards with comprehensive annotations. we propose an llm-based framework for constructing scientific leaderboards in realistic scenarios. our experiments demonstrate the competitiveness of our approach in a cold start setting, where we evaluate its performance on four state-of-the-art llms. furthermore, we show that our method can effectively reconstruct scientific leaderboards in real-world scenarios, highlighting its practical applicability. related work data source norm task form extr constr (hou et al., 2019) nprog. ✓ ✓ ✓ (kardas et al., 2020) pwc ✓ ✓ ✓ (yang et al., 2022) pwc -∼ -(kabongo et al., 2023b) pwc -∼ -(singh et al., 2024) pwc -∼ ✓ scilead (ours) pub. ✓ ✓ ✓ ## framework our proposed framework is illustrated in figure 2 . the main goal is to automatically construct leaderboards given a collection of scientific publications. the framework first receives as input a set of papers in pdf form and extracts a complete set of tdmr tuples from these papers ( §4.1). this first stage is realized by prompting an llm augmented with a dense retriever. the extracted tuples are then passed into a normalization module in the second stage, which decides whether the tdm triples belong to existing leaderboards or comprise new ones ( §4.2). in particular, the normalization module maps these tuples to a pre-defined tdm taxonomy or dynamically updates the taxonomy to integrate new tdm entities. finally, the corresponding best performances of these normalized tdm triples is being used to rank the competitive methods for paper chunks tables query based text extraction table extraction system prompt paper a {task: text summarization, dataset: nyt, metric: rouge-1, result: 57.75}, {task: pos tagging, dataset: ptb, metric: f1 score, result: 93.76} paper a normalization prompt paper a {task: summarization, dataset: new york times (nyt), metric: rouge-1, result: 57.75}, {task: part-of-speech (pos) tagging, dataset: penn treebank (ptb), metric: f1, result: 93.76} repeating for all papers in corpus paper result c 60.13 a 57.75 d 55.67 task: summarization, dataset: new york times (nyt) metric: rouge-1 leaderboard: tdmr extraction normalization leaderboard construction leaderboard construction ( §4.3). ## results and analysis in this section, we first inspect the main tdmr extraction and leaderboard construction results obtained under diverse experimental settings on scilead. in addition, we augment our analysis by conducting human evaluation and error analysis on a new dataset containing recent nlp papers and papers from the medical domain, showcasing the practical utility of our proposed system in real-world scenarios where tdm triples are not pre-defined. we observe that all llms, except llama-2, outperform the axcell baseline regarding the f1 score in the fully pre-defined tdm triples setting. the low precision score of llama-2 is due to the generation of an excessive number of tdmr tuples. in contrast, axcell has high precision and low recall compared with llama-2 due to the small number of extracted tuples. we observe that normalization using llama-2 and mixtral falls short of the +cs baseline, whereas llama-3 and gpt-4 turbo demonstrate superior performance, outperforming the baseline. as expected, the performance of all models deteriorates in the more realistic, partially pre-defined tdm triples setting, which mirrors realworld application scenarios. nevertheless, gpt-4 turbo outperforms its counterparts in both settings. ## conclusion we introduce scilead, a new dataset addressing the limitations of existing leaderboard construction datasets by providing accurate and complete task-dataset-metric-result information from scientific papers. furthermore, we develop an llm-based framework to facilitate the automatic leaderboard construction process in realistic application scenarios. our results show that llms excel in extracting task, dataset, and metric names and detecting correct leaderboards but struggle with extracting result values. our work contributes to the automatic leaderboard construction problem, providing valuable insights for large-scale applications.