Rethinking NLP for Chemistry: A Critical Look at the USPTO Benchmark (2025)

Abstract: Natural Language Processing (NLP) has catalyzed a paradigm shift in Computer-Aided Synthesis Planning (CASP), reframing chemical synthesis prediction as a sequence-to-sequence modeling problem over molecular string representations like SMILES. This framing has enabled the direct application of language models to chemistry, yielding impressive benchmark scores on the USPTO dataset, a large text corpus of reactions extracted from US patents. However, we show that USPTO’s patent-derived data are both industrially biased and incomplete. They omit many fundamental transformations essential for practical real-world synthesis. Consequently, models trained exclusively on USPTO perform poorly on simple, pharmaceutically relevant reactions despite high benchmark scores. Our findings highlight a broader concern in applying standard NLP pipelines to scientific domains without rethinking data and evaluation: models may learn dataset artifacts rather than domain reasoning. We argue for the development of chemically meaningful benchmarks, greater data diversity, and interdisciplinary dialogue between the NLP community and domain experts to ensure real-world applicability.

## introduction recent advances in natural language processing (nlp) have had a transformative impact well beyond linguistics, enabling breakthroughs across diverse domains. among these, computational chemistry has emerged as a particularly promising field, where tools and models developed for human language are increasingly repurposed to reason over structured, sequential chemical representations @xcite . a key challenge in this domain is computer-aided synthesis planning (casp), which aims to automatically design viable synthesis routes to obtain target molecules. in practice, this involves identifying a series of chemical reactions that transform simple and/or commercially available starting materials into a desired target compound @xcite . this series of reactions is known as a synthesis route. traditionally, designing such routes often requires expert intuition and extensive domain knowledge @xcite . casp models aim to automate this reasoning process, providing chemists with viable synthetic strategies. finding a synthesis route is crucial in modern drug discovery, materials design, and green chemistry. drawing on structural parallels with language modeling, researchers have increasingly reformulated synthesis planning as a sequence-to-sequence task, encoding molecules as linear string representations @xcite . this approach represents a direct recontextualization of nlp methods: language models are applied to a domain that shares some structural properties with natural language, yet differs fundamentally in semantics, evaluation, and goals. contemporary approaches today in the field of casp are predominantly data-driven. a widely held argument in the field is that rule-based systems are costly to maintain and difficult to scale. from this perspective, models that bypass such rules are seen as more scalable and are often assumed to generalize better to previously unseen reactions @xcite . however, this assumption does not always hold, especially when the training data is limited in scope, unrepresentative of the real distribution, or exhibits systematic biases. one dataset dominates the literature on single and multi-step chemical synthesis, @xcite , which remains the only large-scale and open-source reaction dataset. it is derived from patents published by the united states patent and trademark office. beyond its accessibility, the structure of the dataset, where reactions are represented as smiles strings in an input-output format, played a pivotal role in shifting the field toward data-driven, nlp-style modeling. the smiles notation @xcite offers a compact, linear representation of molecular graphs by encoding atoms and bonds as a sequence of characters. this linearization is achieved through a canonical traversal of the molecular graph, ensuring that each molecule is mapped to a unique and reproducible string while retaining sufficient information for reconstruction. such a representation makes molecules inherently compatible with tokenization and sequence-based modeling. consequently, the uspto's adoption of the smiles format naturally aligned with sequence-to-sequence learning paradigms, motivating researchers to apply language models to chemical synthesis. in doing so, the uspto dataset not only served as a foundational resource but also influenced the methodological direction of synthesis prediction, reinforcing the dominance of data-driven nlp approaches over symbolic or rule-based alternatives. while different versions of the original uspto dataset have become a standard benchmark in synthesis prediction, their scope is inherently constrained by their origin: patent literature. as such, they primarily capture reactions that are considered novel, industrially relevant, or commercially valuable enough to warrant patent protection. despite repeated efforts to clean and curate the uspto dataset @xcite , and despite growing critiques regarding its efficiency @xcite , diversity @xcite , and labeling bias @xcite , it remains the most used benchmark in the field of single-step and multi-step synthesis prediction. however, contrary to widespread assumptions, we argue that relying exclusively on this benchmark, and thus on patent-derived data, introduces significant bias during model training. this reliance systematically overlooks a broad portion of the chemical space, particularly foundational and nonpatented reactions, thereby limiting the generalizability and real-world utility of current models. the lack of real-world adoption or application of models trained exclusively on these datasets further reinforces our claim: these models often fail to generalize to diverse chemical structures and fall short of the requirements for practical synthesis planning. this reveals a deeper issue: the adoption of nlp-style benchmarking practices in scientific domains can create the illusion of progress, even when models fail to achieve meaningful domainlevel impact. in this study, we critically examine the prevailing benchmark of uspto that has been optimized for synthesis prediction research for a decade. we highlight that the problem of synthesis planning, originally rooted in assisting chemists, has drifted into a purely informatics challenge, disconnected from the experimental realities and needs of chemistry. to ground this critique, we conduct an empirical investigation that tests the generalization capacity of a state-of-the-art model trained on uspto, chemformer @xcite . to ensure that our findings are not an artifact of model choice, we also include as a sanity check a t5 model, pro-pret5 @xcite , we previously trained for the same task, which performs considerably worse than chemformer. by evaluating these models on a set of foundational, pharmaceutically relevant reactions from the hartenfeller dataset @xcite absent from the uspto benchmark, we demonstrate that high accuracy on uspto does not necessarily translate into chemical reasoning or practical utility. this disconnect, we argue, calls for a reassessment of current benchmarks and evaluation practices within the field. ## conclusion this study critically re-evaluates the uspto benchmark, a cornerstone resource for nlp-based synthesis prediction. while the dataset and its subsets have advanced the field, their widespread adoption has also introduced systemic biases. our findings demonstrate that models trained on uspto, despite strong benchmark performance, fail to generalize beyond its narrow scope. the current benchmark provides an incomplete view of the chemical transformation space, omitting much of its diversity. the aim of this work is not to propose new benchmarks or corrective methods, but to draw attention to the structural issues in the current standard. highlighting these limitations is, in itself, a timely and necessary contribution: without recognizing what is missing, progress risks being evaluated against an incomplete benchmark. more broadly, as nlp-inspired methods expand into scientific domains, benchmarks must be critically assessed to ensure they capture genuine scientific reasoning rather than artifacts of data collection. only by confronting these limitations can interdisciplinary research move toward models that meaningfully support discovery. and raised concerns about its suitability as a benchmark, the creation of a more representative and generalizable dataset remains an open challenge. constructing such a corpus would require the collaborative effort of researchers across the globe, given the breadth and diversity of synthetic chemistry. if language models are to become competent agents for synthesis planning, they must be exposed to the full landscape of chemical transformations known to human experts. a second limitation stems from the naming consistency and reaction classification across datasets. although we employed rxn-insight to standardize reaction names between the uspto-mit mixed and generated test data using hartenfeller reaction set, the tool itself is not infallible. errors in classification or missed equivalences between similar reactions could have affected the exclusion or inclusion of certain reactions in our test set. consequently, our definition of "unseen" reaction types, while methodologically principled, may still suffer from edge cases or subtle misalignments. finally, our reliance on smarts-based templates to generate the evaluation set introduces its own limitations. while templates offer control and interpretability, they encode simplified views of reactivity and do not capture all the nuances of context-dependent reactivity in real-world chemical systems. the evaluation set, while diverse, may thus underrepresent complex reaction conditions or fail to reflect the full spectrum of transformation types relevant in synthesis.