The WEAVE Corpus: Annotating Synthetic Chemical Procedures in Patents with Chemical Named Entities (2020)

Abstract: The Modern pharmaceutical industry depends on the iterative design of novel synthetic routes for drugs while not infringing on existing intellectual property rights. Such a design process calls for analyzing many existing synthetic chemical reactions and planning the synthesis of novel chemicals. These procedures have been historically available in unstructured raw text form in publications and patents. To facilitate automated synthetic chemical reactions analysis and design the novel synthetic reactions using Natural Language Processing (NLP) methods, we introduce a Named Entity Recognition (NER) dataset of the Examples section in 180 full-text patent documents with 5188 synthetic procedures annotated by domain experts. All the chemical entities which are part of the synthetic discourse were annotated with suitable class labels. We present the second-largest chemical NER corpus with 100,129 annotations and the highest IAA value of 98.73% (F-measure) on a 45 document subset. We discuss this new resource in detail and highlight some specific challenges in annotating synthetic chemical procedures with chemical named entities. We make the corpus available to the community to promote further research and development of downstream NLP systems applications. We also provide baseline results for the NER model to the community to improve on.

## introduction there is a renewed interest in academia and industry to access the information regarding chemical and chemical reactions currently available in unstructured raw text form in journal publications and patents @xcite @xcite using machine learning. also, several chemical ner datasets exist. with increasing demand in automated chemical synthesis design and planning novel chemical reactions, we need to shift away from the annotation of title and abstract of patents or reactions in isolation to the patents' core, the examples section. the chemdner-patents corpus @xcite is the only dataset focusing on titles and abstracts. the chapati corpus @xcite and biosemnatics corpus @xcite focus on the full text of patents for annotation. the reason for the insufficiency of these corpora is discussed in detail in section 3.3 and 3.5. the chemu labs introduced a named entity dataset with chemical role labels @xcite . as part of the dataset, they have annotated only snippets of reaction text from the patents' experimental section. they also acknowledge the problem of entity often referring to context beyond the current reaction text foot_0 . this context cannot be accounted for by the snippets of reaction text in isolation. as part of the weave corpus, we would like to annotate the chemical entities in their full reaction discourse. this would enable us to model the context beyond the immediate reaction text. we refer readers to the supporting information containing full-text patents to understand how the discourse varies from section to section. a patent is the grant of a legal right by a patent office to an inventor. this grant provides the inventor exclusive rights for a designated period of time in exchange for comprehensive invention disclosure. the disclosure should be complete, such that a person well versed in the field should be able to reproduce this patented process, design, or invention. this disclosure is done in the examples section of a patent. hence the examples section is fundamentally different in its linguistic structure from other sections in a patent. it is the most useful part of understanding the synthetic chemical reactions given in the patent. ## the weave patents corpus an important consideration in preparing a corpus for ner training, development, and evaluation sets is selecting documents representing the distribution of chemical named entities seen in related documents. in the weave corpus, the focus is on synthetic chemical procedures and the chemical entities present. two considerations influenced document selection in our corpus. first, the documents used in the corpus should be available without copyright protection. second, they are complementary to existing datasets. we accessed the patents from the united states patent and trademark office (uspto) 5 . following criterion were applied to further subset the patents for annotation: an additional criterion for selection within this subset was the presence of synthetic organic procedures. ## experiments to establish some baseline performance parameters for the evaluation of the weave corpus, we applied the ner model foot_5 developed by @xcite , which has been successfully applied in multilingual, clinical and drug ner. morphological features have been successfully applied in named entity recognition. in submissions to bc-iv chemdner task @xcite ## analysis to better understand the weave corpus's baseline performance, we conducted several experiments involving bc-v corpus and its combinations with the weave corpus. in tables 8 and 9 we present the results of experiments on various combinations of weave and bc-v datasets. based on the simple ner model @xcite , the best result in terms of macro-averaged f-measure was the model on standalone weave corpus and tested on weave test set with 91.37%. followed by a model trained on bc-v + weave corpus and tested on the weave test set with 91.34%. in comparison, the top-performing team in the bc-v chemdner-patents task had an fscore of 89.37% @xcite . whereas the model trained on standalone bc-v corpus and tested on bc-v test corpus had an f-measure of 80.89%. the model's worst performance was when trained on weave corpus and tested on the bc-v test set; the f-measure was 29.93%. the results validate the linguistic structure of the title and abstract of a patent is very different from that of the examples section. hence, when combined with the chemdner-patents corpus, the weave corpus are complementary; without losing precision, we have an increase in the recall of the ner model. this also supports our assertion of the need for a focused dataset covering the examples section of patents. the combined corpus can perform very close to the state-of-the-art results in chemical ner. this combination also gives us many high-quality annotations 199,763 (100,129 weave + 99,634 bc-v) to develop better chemical ner models. the iaa value of 98.73% on 45 documents subset and the best ner model with f-measure of 91.37% is instructive of the ner model's simple nature. there is good scope for researching better ner models, which can reduce this difference. ## discussion our results show that a focused annotated ner dataset with a simple ner model can achieve near state-of-the-art results. complementary datasets can achieve high recall without sacrificing the precision of the chemical ner model. this is illustrated by the rows highlighted as bold in we propose to explore more complex ner models. these models can better account for the high iaa values reported by us. in the future, we would explore the possibility of extending this dataset to chemical reaction role labeling for the identified chemical entities. ## supporting information the weave corpus described in this paper is available at github repository: https://github.com/ nv-ravindra/the-weave-corpus