Unequal Scientific Recognition in the Age of LLM s (2025)

Abstract: Large language models (LLMs) are reshaping how scientific knowledge is accessed and represented. This study evaluates the extent to which popular and frontier LLMs including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro recognize scientists, benchmarking their outputs against OpenAlex and Wikipedia. Using a dataset focusing on 100,000 physicists from OpenAlex to evaluate LLM recognition, we uncover substantial disparities: LLMs exhibit selective and inconsistent recognition patterns. Recognition correlates strongly with scholarly impact such as citations, and remains uneven across gender and geography. Women researchers, and researchers from Africa, Asia, and Latin America are significantly underrecognized. We further examine the role of training data provenance, identifying Wikipedia as a potential sources that contributes to recognition gaps. Our findings highlight how LLMs can reflect, and potentially amplify existing disparities in science, underscoring the need for more transparent and inclusive knowledge systems.

## introduction large language models (llms) are increasingly used to retrieve factual knowledge across domains. with growing capabilities in discovery, reasoning, summarization, and interpretation @xcite , in science llms emerge as knowledge agents alongside structured databases @xcite @xcite such as openalex @xcite and web of science. this broad adoption raises an important question: how well do llms actually understand the scientific community? can they accurately recognize individual scientists, a core signal of their grasp of the social structure of science and the distribution of expertise? our study contributes a novel framework for auditing scientific recognition in llms, advancing beyond prior work in three key ways. first, we move to analyze cross-model agreement of entities, beyond only coverage metrics, offering insight into mutual versus isolated recognition. second, we benchmark llms recognition against authoritative knowledge bases to assess whether models amplify or mitigate representational biases across demographic groups. third, we examine the role of training data provenance, specifically links to sources like wikipedia in shaping recognition outcomes @xcite . together, these contributions highlight how llms reflect systemic disparities and provide tools for evaluating representational fidelity in ai-generated knowledge of science and scientific communities. ## methodology we develop a multi-stage evaluation pipeline to assess how llms recognize scientists, combining profile-level prompting, mutual recognition analysis, bias auditing, and source provenance analysis (figure 1 ). this framework enables robust mea-surement of both overall coverage and subgroup disparities across models. ## results and analysis for each scientist, we prompt three frontier llms to assess whether the model can correctly identify and describe the individual. results as followed. ## conclusion our study presents a systematic evaluation of scientific recognition in llms, revealing how representation varies across models, demographics, and data provenance. while recognition correlates with scholarly impact, we uncover persistent disparities by gender and region, with llms amplifying underrepresentation relative to external baselines like wikipedia. our causal evidence further highlights the critical role of training data provenance, showing that exposure to public knowledge sources significantly boosts recognition.