NSF - S ci F y: Mining the NSF Awards Database for Scientific Claims (2025)

Abstract: We introduce NSF-SciFy, a comprehensive dataset of scientific claims and investigation proposals extracted from National Science Foundation award abstracts. While previous scientific claim verification datasets have been limited in size and scope, NSF-SciFy represents a significant advance with an estimated 2.8 million claims from 400,000 abstracts spanning all science and mathematics disciplines. We present two focused subsets: NSF-SciFy-MatSci with 114,000 claims from materials science awards, and NSF-SciFy-20K with 135,000 claims across five NSF directorates. Using zero-shot prompting, we develop a scalable approach for joint extraction of scientific claims and investigation proposals. We demonstrate the dataset’s utility through three downstream tasks: non-technical abstract generation, claim extraction, and investigation proposal extraction. Fine-tuning language models on our dataset yields substantial improvements, with relative gains often exceeding 100%, particularly for claim and proposal extraction tasks. Our error analysis reveals that extracted claims exhibit high precision but lower recall, suggesting opportunities for further methodological refinement. NSF-SciFy enables new research directions in large-scale claim verification, scientific discovery tracking, and meta-scientific analysis.

## introduction the overall growth rate of scientific publications is estimated to be 4% annually, with a doubling time of 17 years @xcite . within this deluge, researchers, reviewers, and the general public struggle to separate substantiated claims from spurious ones-whether it is the "quantum supremacy" assertions in computing, the shortlived excitement over lk-99 superconductors 3 , figure 1 : a sample record from our dataset. each record contains 1) award id and title, 2) nsf directorate, 3) technical and non-technical abstracts, 4) scientific claims, 5) investigation proposals, and 6) associated publications, when present. or the misunderstanding surrounding microplastic leaches from black plastic spatulas foot_0 . manual verification of ever growing body of scientific claims has become intractable, yet the economic and societal consequences of unverified claims are increasingly severe. @xcite introduced the task of scientific claim verification with the scifact dataset, focusing primarily on automatic verification of scientific claims. follow up works (see section 2 for a detailed account) have mostly focused on the healthcare, building datasets from scientific publications, and modest-sized dataset creation. in this work, we relax all of these aspects and look at building at least an order of magnitude large-scale ## dataset analysis nsf-scify the full dataset contains 412,155 award abstracts spanning from 1970 to 2024, with 2.8 million scientific claims and corresponding investigation proposals. nsf-scify-matsci this materials science subset, which is the focus of this preprint, contains: capability/ application of technology/method (32.8%) statement of problem/ knowledge gap (21.0%) observed phenomenon/property (18.9%) established scientific fact/principle (9.8%) definition/ classification (6.8%) exp. result/ finding... (5.2%) process/ mech... (3.6%) ... (2.0%) figure 3: a treemap of the scientific claim categories in nsf awards. see section 4.2 for descriptions of these categories. nsf-scify-20k for building models across all nsf directorates, we take 20,000 sample subset of nsf-scify, by stratifying across 5 directorates. ## error analysis of the three proposed tasks, we consider the claim extraction task as a canonical task for performing error analysis. to do so, we consider another 120 awards from the test portion of nsf-scify-matsci and nsf-scify-20k. these were stratified sampled across the five areas of interest (similar to section 4.3). we then generate the claims using a mistral-7b model finetuned on nsf-scify-20k, resulting in 802 claims. a careful examination revealed around 2.6% of the generated claims were incorrect. to dive deeper, we categorized the erroneous claims into 5 categories. we list them here with examples: ## discussion and conclusion we introduced nsf-scify, a large dataset of 2.8 million scientific claims and proposals from 400,000 nsf grant abstracts across all science and mathematics disciplines. focused subsets include nsf-scify-matsci (114,000 materials science claims) and nsf-scify-20k (135,000 claims from five directorates). experiments demonstrate that fine-tuning language models on nsf-scifysignificantly improves scientific claim and proposal extraction, with relative performance gains often exceeding 100%. non-technical abstract generation saw modest improvements due to strong baselines. stylistic differences between technical and non-technical abstracts offer potential for science communication. our claim taxonomy identifies prevalent assertion types like capability/application and problem/knowledge gap statements. nsf-scify's unique advantages include its vast scale, high quality from nsf expert review, comprehensive coverage of scientific domains, a temporal span from 1970-2024 enabling longitudinal studies, and, for recent grants, links to resulting publications. error analysis showed high precision but lower recall in claim extraction, with error categories guiding future improvements. nsf-scify opens new research avenues in largescale claim verification, scientific discovery tracking, and meta-scientific analysis, a key resource for understanding scientific assertions at their origin.