Original: MedMentions: A Large Biomedical Corpus Annotated with UMLS
Revised: MedMentions: A Large Biomedical Corpus Annotated with UMLS

Original: Concepts
Revised: Concepts

Original: Anonymous authors
Revised: Anonymous authors

Original: Abstract
Revised: Abstract

Original: This paper presents the formal release of MedMentions , a new manually annotated resource for the recognition of biomedical concepts.
Revised: This paper presents the formal release of MedMentions , a new manually annotated resource for the recognition of biomedical concepts.

Original: What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines.
Revised: What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines.

Original: In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval.
Revised: In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval.

Original: To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release and a baseline model and its metrics for entity linking are also described.
Revised: To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release and a baseline model and its metrics for entity linking are also described.

Original: 1. Introduction
Revised: 1. Introduction

Original: One recognized challenge in developing automated biomedical entity extraction systems is the lack of richly annotated training datasets.
Revised: One recognized challenge in developing automated biomedical entity extraction systems is the lack of richly annotated training datasets.

Original: While there are a few such datasets available, the annotated corpus often contains no more than a few thousand annotated entity mentions.
Revised: While there are a few such datasets available, the annotated corpus often contains no more than a few thousand annotated entity mentions.

Original: Additionally, the annotated entities are limited to a few types of biomedical concepts such as diseases [Do ˘ gan and
Revised: Additionally, the annotated entities are limited to a few types of biomedical concepts such as diseases [Do˘gan and

Original: Lu, 2012], gene ontology terms [Van Auken et al., 2014], or chemicals and diseases [Li et al., 2016].
Revised: Lu, 2012], gene ontology terms [Van Auken et al., 2014], or chemicals and diseases [Li et al., 2016].

Original: Researchers targeting the recognition of multiple biomedical entity types have had to resort to specialized machine learning techniques for combining datasets labelled with subsets of the full target set, e.g. using multi-task learning [Crichton et al., 2017], or a modiﬁed Conditional Random
Revised: Researchers targeting the recognition of multiple biomedical entity types have had to resort to specialized machine learning techniques for combining datasets labelled with subsets of the full target set, e.g. using multi-task learning [Crichton et al., 2017], or a modiﬁed Conditional Random

Original: Field cost which allows un-labeled tokens to take any labels not in the current dataset’s target set
Revised: Field cost which allows un-labeled tokens to take any labels not in the current dataset’s target set

Original: [Greenberg et al., 2018].
Revised: [Greenberg et al., 2018].

Original: To promote the development of state-of-the-art entity linkers targeting a more comprehensive coverage of biomedical concepts, we decided to create a large concept-mention annotated  dataset named MedMentions .
Revised: To promote the development of state-of-the-art entity linkers targeting a more comprehensive coverage of biomedical concepts, we decided to create a large concept-mention annotated gold standard dataset named MedMentions .

Revised: With the release of MedMentions, we hope to address two key needs for developing better biomedical concept recognition systems: (i) a much broader coverage of the ﬁelds of biology and medicine through the use of UMLS as the target ontology, and (ii) a signiﬁcantly larger annotated corpus than available today, to meet the data demands of today’s more complex machine learning models for concept recognition.

Original: The paper begins with an introduction to the MedMentions annotated corpus, including a subcorpus aimed at information retrieval systems.
Revised: The paper begins with an introduction to the MedMentions annotated corpus, including a subcorpus aimed at information retrieval systems.

Original: This is followed by a comparison with a few other large datasets annotated with biomedical entities.
Revised: This is followed by a comparison with a few other large datasets annotated with biomedical entities.

Original: Finally, to promote further research on large ontology named entity recognition and linking, we present some baseline metrics.
Revised: Finally, to promote further research on large ontology named entity recognition and linking, we present metrics for a baseline end-to-end concept recognition (entity type recognition and entity linking) model trained on the MedMentions corpus.

Original: 2. Introducing MedMentions 2.1 The Documents
Revised: 2. Introducing MedMentions 2.1 The Documents

Original: We randomly selected 5,000 abstracts released in PubMed ®1 between January 2016 and January 2017.
Revised: We randomly selected 5,000 abstracts released in PubMed ®1 between January 2016 and January 2017.

Original: Upon review, some abstracts were found to be outside the biomedical ﬁelds or not written in
Revised: Upon review, some abstracts were found to be outside the biomedical ﬁelds or not written in

Original: English.
Revised: English.

Original: These were excluded, resulting in a total of 4,392 abstracts in the corpus.
Revised: These were discarded, leaving a total of 4,392 abstracts in the corpus.

Original: 2.2 Concepts in UMLS
Revised: 2.2 Concepts in UMLS

Original: The Metathesaurus of the Uniﬁed Medical Language System (UMLS)
Revised: The Metathesaurus of the Uniﬁed Medical Language System (UMLS)

Original: [Bodenreider, 2004] combines concepts from over 200 source ontologies.
Revised: [Bodenreider, 2004] combines concepts from over 200 source ontologies.

Original: It is therefore the largest single ontology of biomedical concepts, and was a natural choice for constructing an annotated resource with broad coverage in biomedical science.
Revised: It is therefore the largest single ontology of biomedical concepts, and was a natural choice for constructing an annotated resource with broad coverage in biomedical science.

Original: In this paper, we will use entities and concepts interchangeably, to refer to UMLS concepts.
Revised: In this paper, we will use entities and concepts interchangeably, to refer to UMLS concepts.

Original: The 2017 AA release of the UMLS Metathesaurus contains ∼ 3 .
Revised: The 2017 AA release of the UMLS Metathesaurus contains approximately 3 .

Original: 2 M unique concepts.
Revised: 2 M unique concepts.

Original: Each concept has a unique id (a “CUID”) and primary name and a set of aliases, and is linked to all the source ontologies it was mapped from.
Revised: Each concept has a unique id (a “CUID”) and primary name and a set of aliases, and is linked to all the source ontologies it was mapped from.

Original: Each concept is also linked to one or more Semantic Types – the UMLS guidelines are to link each concept to the most speciﬁc type(s) available.
Revised: Each concept is also linked to one or more Semantic Types – the UMLS guidelines are to link each concept to the most speciﬁc type(s) available.

Original: Each Semantic Type also has a unique identiﬁer (“TUI”) and a name.
Revised: Each Semantic Type also has a unique identiﬁer (“TUI”) and a name.

Original: The Metathesaurus contains127 Semantic Types, arranged in a “is-a” hierarchy.
Revised: The Metathesaurus contains Semantic Types, arranged in a “is-a” hierarchy.

Original: About 91.7% of the concepts are linked to exactly one semantic type, ∼ 8 % to two types, and a very small number to more than two types.
Revised: About 91.7% of the concepts are linked to exactly one semantic type, approximately 8 % to two types, and a very small number to more than two types.

Original: 2.3 Annotating Concept Mentions
Revised: 2.3 Annotating Concept Mentions

Original: We recruited a team of professional annotators with rich experience in biomedical content curation to exhaustively annotate UMLS entity mentions from the abstracts.
Revised: We recruited a team of professional annotators with rich experience in biomedical content curation to exhaustively annotate UMLS entity mentions from the abstracts.

Original: The annotators used the text processing tool GATE 2 (version 8.2) to facilitate the curation.
Revised: The annotators used the text processing tool GATE 2 (version 8.2) to facilitate the curation.

Original: All the relevant scientiﬁc terms from each abstract were manually searched in the 2017 AA (full) version of the UMLS metathesaurus 3 and the best matching concept was retrieved.
Revised: All the relevant scientiﬁc terms from each abstract were manually searched in the 2017 AA (full) version of the UMLS metathesaurus 3 and the best matching concept was retrieved.

Original: The annotators were asked to annotate the most speciﬁc concept for each mention, without any overlaps in mentions.
Revised: The annotators were asked to annotate the most speciﬁc concept for each mention, without any overlaps in mentions.

Original: To gain insight on the annotation quality of MedMentions, we randomly selected eight abstracts from the annotated corpus.
Revised: To gain insight on the annotation quality of MedMentions, we randomly selected eight abstracts from the annotated corpus.

Original: Two biologists (“Rreviewers”) who did not participate in the annotation task then each reviewed four abstracts and the corresponding concepts in MedMentions.
Revised: Two biologists (“Reviewers”) who did not participate in the annotation task then each reviewed four abstracts and the corresponding concepts in MedMentions.

Original: The abstracts contained a total of 469 concepts.
Revised: The abstracts contained a total of 469 concepts.

Original: Of these 469 concepts, the agreement between Reviewers and Annotators was 97.3%, estimating the precision of the annotation IN MEDMENTIONS.
Revised: Of these 469 concepts, the agreement between Reviewers and Annotators was 97.3%, estimating the precision of the annotation in MedMentions.

Original: Due to the size of UMLS, we reasoned that no human curators would have knowledge of the entire UMLS, so we did not perform an evaluation on the recall.
Revised: Due to the size of UMLS, we reasoned that no human curators would have knowledge of the entire UMLS, so we did not perform an evaluation on the recall.

Original: We are working on getting more detailed IAA (Inter-annotator agreement) data, which will be released when that task is completed.
Revised: We are working on getting more detailed IAA (Inter-annotator agreement) data, which will be released when that task is completed.

Original: 2.4 MedMentions ST21pv
Revised: 2.4 MedMentions ST21pv

Original: Entity linking / labeling methods have prominently been used as the ﬁrst step towards relationship extraction, e.g. the BioCreative V CDR task for Chemical-Disease relationship extraction [Li et al., http://pubmed.gov 2. https://gate.ac.uk/ 3.
Revised: Entity linking / labeling methods have prominently been used as the ﬁrst step towards relationship extraction, e.g. the BioCreative V CDR task for Chemical-Disease relationship extraction [Li et al., 2016], and for indexing for entity-based document retrieval, e.g. as described in the BioASQ Task

Original: http://umlsks.nlm.nih.gov

Original: 2016], and for indexing for entity-based document retrieval, e.g. as described in the BioASQ Task

Original: A for semantic indexing [Nentidis et al., 2018].
Revised: A for semantic indexing [Nentidis et al., 2018].

Original: One of our goals in building a more comprehensive annotated corpus was to provide indexing models with a larger ontology than MeSH (used in
Revised: One of our goals in building a more comprehensive annotated corpus was to provide indexing models with a larger ontology than MeSH (used in

Original: BioASQ Task A and PubMed) for semantic indexing, to support more speciﬁc document retrieval queries from researchers in all biomedical disciplines.
Revised: BioASQ Task A and PubMed) for semantic indexing, to support more speciﬁc document retrieval queries from researchers in all biomedical disciplines.

Original: UMLS does indeed provide a much larger ontology (see table 6).
Revised: UMLS does indeed provide a much larger ontology (see Table 6).

Original: However UMLS also contains many concepts that are not as useful for specialized document retrieval, either because they are too broad so not discriminating enough (e.g. Groups [cuid = C0441833] , Risk [C0035647] ), or cover peripheral and supplementary topics not likely to be used by a biomedical researcher in a query (e.g.
Revised: However UMLS also contains many concepts that are not as useful for specialized document retrieval, either because they are too broad so not discriminating enough (e.g. Groups [cuid = C0441833] , Risk [C0035647] ), or cover peripheral and supplementary topics not likely to be used by a biomedical researcher in a query (e.g.

Original: Rural Area [C0178837] , No difference [C3842396] ).
Revised: Rural Area [C0178837] , No difference [C3842396] ).

Original: Filtering UMLS to a subset most useful for semantic indexing is going to be an area of ongoing study, and will have different answers for different user communities.
Revised: Filtering UMLS to a subset most useful for semantic indexing is going to be an area of ongoing study, and will have different answers for different user communities.

Original: Furthermore, targeting different subsets will also impact machine learning systems designed to recognize concepts in text.
Revised: Furthermore, targeting different subsets will also impact machine learning systems designed to recognize concepts in text.

Original: As a ﬁrst step, we propose the “ ST21pv ” subset of UMLS, and the corresponding annotated subcorpus MedMentions ST21pv .
Revised: As a ﬁrst step, we propose the “ ST21pv ” subset of UMLS, and the corresponding annotated subcorpus MedMentions ST21pv .

Original: Here “ST21pv” is an acronym for “21 Semantic Types from Preferred
Revised: Here “ST21pv” is an acronym for “21 Semantic Types from Preferred

Original: Vocabularies”, and the ST21pv subset of UMLS was constructed as follows: 
Revised: Vocabularies”, and the ST21pv subset of UMLS was constructed as follows: 1.

Original: We eliminated all concepts that were only linked to semantic types at levels 1 or 2 in the UMLS Semantic Type hierarchy with the intuition that these concepts would be too broad.
Revised: We eliminated all concepts that were only linked to semantic types at levels 1 or 2 in the UMLS Semantic Type hierarchy with the intuition that these concepts would be too broad.

Original: We also limited the concepts to those in the Active subset of the 2017 AA release of UMLS.
Revised: We also limited the concepts to those in the Active subset of the 2017 AA release of UMLS.

Original: We then selected 21 semantic types at levels 3–5 based on biomedical relevance, and whether MedMentions contained sufﬁcient annotated examples.
Revised: We then selected 21 semantic types at levels 3–5 based on biomedical relevance, and whether MedMentions contained sufﬁcient annotated examples.

Original: Only concepts mapping into one of these 21 types (i.e. linked to one of these types or to a descendant in the type hierarchy) were considered for inclusion.
Revised: Only concepts mapping into one of these 21 types (i.e. linked to one of these types or to a descendant in the type hierarchy) were considered for inclusion.

Original: As an example, the semantic type Archaeon [T194] was excluded because MedMentions contains only 25 mentions for 15 of the 5,418 concepts that map into this type (table 2).
Revised: As an example, the semantic type Archaeon [T194] was excluded because MedMentions contains only 25 mentions for 15 of the 5,418 concepts that map into this type (Table 2).

Original: Similarly Clinical Drug [T200] was also eliminated. Finally, we selected 18 ‘prefered’ source vocabularies (table 1), and excluded any concepts that were not linked in UMLS to at least one of these sources. These vocabularies were selected based on usage and relevance to biomedical research 4 , with an emphasis on gene function, disease and phenotype, structure and anatomy, and drug and chemical entities. Table 2 gives a detailed breakdown of a portion of the semantic type hierarchy in UMLS
Revised: Since our primary purpose for ST21pv is to use annotations from this subset as an aid for biomedical researchers to retrieve relevant papers, some types were eliminated if most of their member concepts were considered by our staff biologists as not useful for this task.

Original: AA Active. The rows in bold are the 21 types in ST21pv, and any descendants of these types have been pruned and their counts rolled up. The counts therefore are for concepts linked to the corresponding type for the non-bold rows, and mapped to the ST21pv types for the rows in bold. Note that some concepts in UMLS are linked to multiple semantic types.
Revised: An example is Qualitative Concept [T080] , which contains frequently mentioned concepts like

Original: The preﬁx MM- in the column name indicates the counts are for concepts mentioned in MedMentions. The full MedMentions corpus contains 2,473 mentions of 685 concepts that are not members of the 2017 AA Active release. These were eliminated as part of step 1.
Revised: Associated with [C0332281] , Levels [C0441889] and High [C0205250] .

Original: The other non-bold rows in the table represent semantic types excluded in steps 1 and 2, corresponding to a total of 135,986 mentions of 6,002 unique concepts. A further 10,755 mentions of 2,618 concepts  were eliminated in step 3.
Revised: Finally, we selected 18 ‘prefered’ source vocabularies (Table 1), and excluded any concepts that were not linked in UMLS to at least one of these sources.

Original: As a result of all this ﬁltering, the target ontology for MedMentions ST21pv (MM-ST21pv) contains 2,327,250 concepts and 203,282 concept mentions. [Table] International [Table]
Revised: These vocabularies were selected based on usage and relevance to biomedical research 4 , with an emphasis on gene function, disease and phenotype, structure and anatomy, and drug and chemical entities.

Original: of Diseases, Tenth Revision

Original: [Table]
Revised: [Table]

Original: Subject Headings 
Revised: Headings MTH

Original: [Table]
Revised: [Table]

Original:  Names
Revised: Metathesaurus Names

Original: NCBI
Revised: NCBI

Revised: National

Original: [Table]
Revised: [Table]

Revised: Biotechnology Information Taxonomy

Original: NCI
Revised: NCI

Original: National [Table] Institute [Table]
Revised: National Cancer Institute Thesaurus

Original: NDDF
Revised: NDDF

Original: First DataBank MedKnowledge
Revised: First DataBank MedKnowledge

Original: NDFRT
Revised: NDFRT

Original: [Table] File – Reference Terminology
Revised: National Drug File – Reference Terminology

Original: OMIM
Revised: OMIM

Original: [Table]
Revised: [Table]

Original: RXNORM
Revised: RXNORM

Original: [Table]
Revised: [Table]

Original: Nomenclature for
Revised: Nomenclature for

Original: [Table]
Revised: [Table]

Original: Drugs for Humans
Revised: Drugs [Table] Humans

Original: SNOMEDCT US
Revised: SNOMEDCT US

Original: US edn.
Revised: US edn.

Original: of the Systematized Nomenclature of Medicine-Clinical Terms
Revised: of the Systematized Nomenclature of Medicine-Clinical Terms

Original: Examples of broad concepts eliminated by selecting semtypes at level 3 or higher:
Revised: Table 2 gives a detailed breakdown of a portion of the semantic type hierarchy in UMLS AA Active. The rows in bold are the 21 types in ST21pv, and any descendants of these types have been pruned and their counts rolled up. The counts therefore are for concepts linked to the corresponding type for the non-bold rows, and mapped to the ST21pv types for the rows in bold.

Original: • C1707689: “Design”, linked to T052: Activity, level= • C0029235: “Organism” linked to T001: Organism, level= • C0520510: “Materials” linked to T167: Substance, level=
Revised: Note that some concepts in UMLS are linked to multiple semantic types. The preﬁx MM- in the column name indicates the counts are for concepts mentioned in MedMentions. The full MedMentions corpus contains 2,473 mentions of 685 concepts that are not members of the 2017 AA Active release. These were eliminated as part of step 1.

Original: The MedMentions corpus consists of 4,392 abstracts randomly selected from those released on PubMed between January 2016 and January 2017.
Revised: The other non-bold rows in the table represent semantic types excluded in steps 1 and 2, corresponding to a total of 135,986 mentions of 6,002 unique concepts.

Original: Table 3 shows some descriptive statistics for the  MedMentionscorpus and its ST21pv subset. The tokenization and [-sentence splitting were performed using Stanford CoreNLP 
 [Manning et al., 2014].-]
Revised: A further 10,755 mentions of 2,618 concepts were eliminated in step 3. As a result of all this ﬁltering, the target ontology for MedMentions ST21pv (MM-ST21pv) contains 2,327,250 concepts and [-sentence splitting were performed using Stanford CoreNLP 
 [Manning et al., 2014].-]203,282 concept mentions.

Original: Due to the size of UMLS, only about 1% of its concepts are covered in MedMentions. So a major part of the challenge for machine learning systems trained to recognize these concepts is ‘unseen labels’ (often called “zero-shot learning”, e.g. [Palatucci et al., 2009, Srivastava et al., 2018,
Revised: Examples of broad concepts eliminated by selecting semtypes at level 3 or higher:

Original: Xian et al., 2017]). As part of the release, we also include a 60% - 20% - 20% random partitioning of the corpus into training, ‘dev’ (aka validation) and test subsets. These are described in table 4. As can be seen from the table, about 42% of the concepts in the test data do not occur in the training data, and 38% do not occur in either training or dev subsets.
Revised: • C1707689: “Design”, linked to T052: Activity, level= • C0029235: “Organism” linked to T001: Organism, level= • C0520510: “Materials” linked to T167: Substance, level=

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 350,   2,051 Archaeon
Revised: 350,   2,051 Archaeon

Original: T  5,   25 Eukaryote
Revised: T  5,   25 Eukaryote

Original: T  806, 1, 1, 8,640 Anatomical Structure
Revised: T  806, 1, 1, 8,640 Anatomical Structure

Original: T  196, 2, 2, 20,778 Manufactured Object
Revised: T  196, 2, 2, 20,778 Manufactured Object

Original: T  6,  1, 3,615 Medical Device
Revised: T  6,  1, 3,615 Medical Device

Original: T  58,   2,406 Research Device
Revised: T  58,   2,406 Research Device

Original: T     365 Clinical Drug
Revised: T     365 Clinical Drug

Original: T  129,   61 Substance
Revised: T  129,   61 Substance

Original: T  9,   1,769 Body Substance
Revised: T  9,   1,769 Body Substance

Original: T  2,   1,258 Chemical
Revised: T  2,   1,258 Chemical

Original: T  435, 5, 2, 38,225 Food
Revised: T  435, 5, 2, 38,225 Food

Original: T  7,   1,462 Conceptual Entity
Revised: T  7,   1,462 Conceptual Entity

Original: T    1, 2,997 Organism Attribute
Revised: T    1, 2,997 Organism Attribute

Original: T    1, 3,732 Clinical Attribute
Revised: T    1, 3,732 Clinical Attribute

Original: T  85,   2,027 Finding
Revised: T  85,   2,027 Finding

Original: T  308, 3, 3, 18,435 Idea or Concept
Revised: T  308, 3, 3, 18,435 Idea or Concept

Original: T  3,  2, 9,348 Temporal Concept
Revised: T  3,  2, 9,348 Temporal Concept

Original: T  3,  2, 10,169 Qualitative Concept
Revised: T  3,  2, 10,169 Qualitative Concept

Original: T  4, 1, 4, 31,485 Quantitative Concept
Revised: T  4, 1, 4, 31,485 Quantitative Concept

Original: T  9,  3, 19,995 Spatial Concept
Revised: T  9,  3, 19,995 Spatial Concept

Original: T  42, 1, 2, 13,386 Functional Concept
Revised: T  42, 1, 2, 13,386 Functional Concept

Original: T  3,  3,
Revised: T  3,  3,

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: GENIA
Revised: The MedMentions corpus consists of 4,392 abstracts randomly selected from those released on PubMed between January 2016 and January 2017.

Original: ITI TXM{+Table 3 shows some descriptive statistics for the MedMentions corpus and its ST21pv subset. The tokenization and sentence splitting were performed using Stanford CoreNLP 
+}
Revised: {+Table 3 shows some descriptive statistics for the MedMentions corpus and its ST21pv subset. The tokenization and sentence splitting were performed using Stanford CoreNLP 
+}

Original: CRAFT v1.
Revised: [Manning et al., 2014].

Original: MedMentions
Revised: Due to the size of UMLS [Table] only about 1% of its concepts are covered in MedMentions. So a major part of the challenge for machine learning systems trained to recognize these concepts is ‘unseen labels’ (often called “zero-shot learning”, e

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: g. [Palatucci et al., 2009, Srivastava et al., 2018, Xian et al., 2017]). As part of the release, we also include a 60% - 20% - 20% random partitioning of the corpus into training, development (often called ‘validation’) and test subsets. These are described in Table 4. As can be seen from the table, about 42% of the concepts in the test data do not occur in the training data, and 38% do not occur in either training or development subsets.

Original: 2.6 Accessing MedMentions
Revised: 2.6 Accessing MedMentions

Original: The MedMentions resource has been published at anonymized .
Revised: The MedMentions resource has been published at anonymized .

Original: The corpus itself is in PubTator [Wei et al., 2013] format, which is described on the release site.
Revised: The corpus itself is in PubTator [Wei et al., 2013] format, which is described on the release site.

Original: The corpus consists of PubMed abstracts, each identiﬁed with a unique PubMed identiﬁer (PMID).
Revised: The corpus consists of PubMed abstracts, each identiﬁed with a unique PubMed identiﬁer (PMID).

Original: Each PubMed abstract has Title and Abstract texts, and a series of annotations of concept mentions.
Revised: Each PubMed abstract has Title and Abstract texts, and a series of annotations of concept mentions.

Original: Each concept mention identiﬁes the portion of the document text comprising the mention, and the UMLS concept.
Revised: Each concept mention identiﬁes the portion of the document text comprising the mention, and the UMLS concept.

Original: A separate ﬁle for the ST21pv sub-corpus is also included in the release.
Revised: A separate ﬁle for the ST21pv sub-corpus is also included in the release.

Original: The release also includes three lists of PMID’s that partition the corpus into a 60% / 20% / 20% split deﬁning the ‘Training’, ‘Development’ and ‘Test’ subsets.
Revised: The release also includes three lists of PMID’s that partition the corpus into a 60% / 20% / 20% split deﬁning the Training, Development and Test subsets.

Original: Researchers are encouraged to train their  models using the Training and Development portions of the corpus, and publish test results on the held-out Test subset of the corpus.
Revised: Researchers are encouraged to train their [Table] [Table] models using the Training and Development portions of the corpus, and publish test results on the held-out Test subset of the corpus.

Original: 3. A Comparison With Some Related Corpora
Revised: 3. A Comparison With Some Related Corpora

Original: There have been several “gold-standard” (manually annotated) corpora of biomedical scientiﬁc literature made publicly available.
Revised: There have been several gold standard (manually annotated) corpora of biomedical scientiﬁc literature made publicly available.

Original: Some of the larger ones are described below.
Revised: Some of the larger ones are described below.

Original: GENIA: [Ohta et al., 2002, Kim et al., 2003]
Revised: GENIA: [Ohta et al., 2002, Kim et al., 2003]

Original: One of the earliest ‘large’ biomedical annotated corpora, it is aimed at biomedical Named Entity Recognition, where the annotations are forbiomedical Entity Types.
Revised: One of the earliest ‘large’ biomedical annotated corpora, it is aimed at biomedical Named Entity Recognition, where the annotations are forbiomedical Entity Types.

Original: The dataset consists of 2,000 MEDLINE abstracts about “biological reactions concerning transcription factors in human blood cells”, collected by searching on
Revised: The dataset consists of 2,000 MEDLINE abstracts about “biological reactions concerning transcription factors in human blood cells”, collected by searching on

Original: MEDLINE using the MeSH terms human , blood cells and transcription factors .
Revised: MEDLINE using the MeSH terms human , blood cells and transcription factors .

Original: An extended version (2,404 abstracts), with a smaller ontology (six types) was later used for the JNLPBA 2004 NER task
Revised: An extended version (2,404 abstracts), with a smaller ontology (six types) was later used for the JNLPBA 2004 NER task

Original: [Kim et al., 2004].
Revised: [Kim et al., 2004].

Original: ITI TXM Corpora:
Revised: ITI TXM Corpora:

Original: [Alex et al., 2008]
Revised: [Alex et al., 2008]

Original: Among the largest gold-standard biomedical annotated corpora previously available, this consists of two sets of full-length papers obtained from
Revised: Among the largest gold standard biomedical annotated corpora previously available, this consists of two sets of full-length papers obtained from

Original: PubMed and PubMed Central: 217 articles focusing on protein-protein interactions (PPI) and 238 articles on tissue expressions (TES).
Revised: PubMed and PubMed Central: 217 articles focusing on protein-protein interactions (PPI) and 238 articles on tissue expressions (TES).

Original: The PPI and TES corpora were annotated with entities from NCBI Taxonomy, NCBI Reference Sequence Database, and Entrez Gene.
Revised: The PPI and TES corpora were annotated with entities from NCBI Taxonomy, NCBI Reference Sequence Database, and Entrez Gene.

Original: The
Revised: The

Original: TES corpus was also annotated with entities from Chemical Entities of Biological Interest (ChEBI) and Medical Subject Headings (MeSH).
Revised: TES corpus was also annotated with entities from Chemical Entities of Biological Interest (ChEBI) and Medical Subject Headings (MeSH).

Original: The concepts were grouped into 15 entity types, and these type labels were included in the annotations.
Revised: The concepts were grouped into 15 entity types, and these type labels were included in the annotations.

Original: In addition to concept mentions, the corpus also includes relations between entities.
Revised: In addition to concept mentions, the corpus also includes relations between entities.

Original: The statistics (table 5) for this corpus [Alex et al., 2008] are a little confusing, since not all sections of the articles were annotated.
Revised: The statistics (Table 5) for this corpus [Alex et al., 2008] are a little confusing, since not all sections of the articles were annotated.

Original: Furthermore some articles were annotated by more than one biologist, and each annotated version was incorporated into the corpus as a separate document.
Revised: Furthermore some articles were annotated by more than one biologist, and each annotated version was incorporated into the corpus as a separate document.

Original: CRAFT: [Bada et al., 2012]
Revised: CRAFT: [Bada et al., 2012]

Original: The Colorado Richly Annotated Full-Text (CRAFT)
Revised: The Colorado Richly Annotated Full-Text (CRAFT)

Original: Corpus is another large gold-standard corpus annotated with a diverse set of biomedical concepts.
Revised: Corpus is another large gold standard corpus annotated with a diverse set of biomedical concepts.

Original: It consists of 67 full-text open-access biomedical journal articles, downloaded from PubMed Central, covering a wide range of disciplines, including genetics, biochemistry and molecular biology, cell biology, developmental biology, and computational biology.
Revised: It consists of 67 full-text open-access biomedical journal articles, downloaded from PubMed Central, covering a wide range of disciplines, including genetics, biochemistry and molecular biology, cell biology, developmental biology, and computational biology.

Original: The text is annotated with concepts from 9 biomedical ontologies: ChEBI, Cell Ontology, Entrez Gene, Gene Ontology (GO) Biological Process, GO Cellular Component, GO Molecular Function, NCBI Taxonomy, Protein Ontology, and Sequence Ontology.
Revised: The text is annotated with concepts from 9 biomedical ontologies: ChEBI, Cell Ontology, Entrez Gene, Gene Ontology (GO) Biological Process, GO Cellular Component, GO Molecular Function, NCBI Taxonomy, Protein Ontology, and Sequence Ontology.

Original: The latest release of CRAFT 6 reorganizes this into ten Open Biomedical Ontologies.
Revised: The latest release of CRAFT 6 reorganizes this into ten Open Biomedical Ontologies.

Original: The corpus also contains exhaustive syntactic annotations.
Revised: The corpus also contains exhaustive syntactic annotations.

Original: Table 5 gives a comparison of the sizes of CRAFT against the other corpora mentioned here.
Revised: Table 5 gives a comparison of the sizes of CRAFT against the other corpora mentioned here.

Revised: MedMentions can be viewed as a supplement to the CRAFT corpus, but with a broader coverage of biomedical research (over four thousand abstracts compared to the 67 articles in

Revised: CRAFT).

Revised: Through the larger set of ontologies included within UMLS, MedMentions also contains more comprehensive annotation of concepts from some biomedical ﬁelds, e.g. diseases and drugs (see Table 1 for a partial list of the ontologies included in UMLS).

Original: BioASQ Task A:
Revised: BioASQ Task A:

Original: [Nentidis et al., 2018]
Revised: [Nentidis et al., 2018]

Original: The Large Scale Semantic Indexing task considers assigning MeSH headings for ‘important’ concepts to each document.
Revised: The Large Scale Semantic Indexing task considers assigning MeSH headings for ‘important’ concepts to each document.

Original: The training data is very large, but with a smaller target concept vocabulary (see table 6), and annotation (by NCBI) is at the document level rather than at the mention level.
Revised: The training data is very large, but with a smaller target concept vocabulary (see Table 6), and annotation (by NCBI) is at the document level rather than at the mention level.

Original: Relation / Event Extraction Corpora: Most recently developed manually annotated datasets of biomedical scientiﬁc literature have focused on the task of extracting biomedical events or relations between entities.
Revised: Relation / Event Extraction Corpora: Most recently developed manually annotated datasets of biomedical scientiﬁc literature have focused on the task of extracting biomedical events or relations between entities.

Original: These datasets have been used for shared tasks in biomedical NLP workshops like BioCreative, e.g. BC5-CDR [Li et al., 2016] which focuses on ChemicalDisease relations, and BioNLP, e.g. the BioNLP 2013 Cancer Genetics (CG) and Pathway 
Revised: These datasets have been used for shared tasks in biomedical NLP workshops like BioCreative, e.g. BC5-CDR [Li et al., 2016] which focuses on ChemicalDisease relations, and BioNLP, e.g. the BioNLP 2013 Cancer Genetics (CG) and Pathway BioASQ Task A (2018) MedMentions ST21pv [Table]

Original:  Curation tasks [Pyysalo et al., 2015] where the main goal is to identify events involving entities.
Revised: 48M [Table] [Table] Curation tasks [Pyysalo et al., 2015] where the main goal is to identify events involving entities.

Original: While these datasets include entity mention annotations, they are typically focused on a small set of entity types, and the sizes of the corpora are also smaller (1,500 document abstracts in BC5-CDR, 600 abstracts in CG, and 525 abstracts in PC).
Revised: While these datasets include entity mention annotations, they are typically focused on a small set of entity types, and the sizes of the corpora are also smaller (1,500 document abstracts in BC5-CDR, 600 abstracts in CG, and 525 abstracts in PC).

Original: Machine learning models for relation extraction take as input a text annotated with entity mentions, so recognizing biomedical concepts remains an important foundational task.
Revised: Machine learning models for relation extraction take as input a text annotated with entity mentions, so recognizing biomedical concepts remains an important foundational task.

Original: 4. Concept Recognition with MedMentions ST21pv
Revised: 4. Concept Recognition with MedMentions ST21pv

Original: Our main goal in constructing and releasing MedMentions is to promote the development of models for recognizing biomedical concepts mentioned in scientiﬁc literature.
Revised: Our main goal in constructing and releasing MedMentions is to promote the development of models for recognizing biomedical concepts mentioned in scientiﬁc literature.

Original: To help jumpstart this BioASQ Task A (2018) MedMentions ST21pv [Table]
Revised: To help jumpstart this research, we now present a baseline modeling approach trained using the Training and Development splits of MedMentions ST21pv, and its metrics on the MM-ST21pv Test set.

Original: [Table]
Revised: We measure the performance of the model described below at both the mention level (also referred to as phrase level) and the document level . Concept annotations in MedMentions identify an exact span of text using start and end positions, and annotate that span with an entity type identiﬁer and entity identiﬁer.

Original: research, we now present some baseline modeling approaches trained using the Training and Dev splits of MedMentions ST21pv, and their metrics on the MM-ST21pv Test set.
Revised: Concept recognition models like the one described below will output predictions in a similar format. The performance of such models is usually measured using mention level precision , recall and F1 score as described in [Sang and Meulder, 2003].

Original: 4.1 End-to-end Entity Recognition and Linking with TaggerOne
Revised: Here we are interested in measuring the entity resolution performance of the model: a prediction is counted as a true positive ( tp ) only when the predicted text span as well as the linked entity (and by implication the entity type) matches with the gold standard reference. All other predicted mentions are counted as falsepositives ( fp ), and all un-matched reference entity mentions as false-negatives ( fn ). These counts [Table]

Original: TaggerOne [Leaman and Lu, 2016] is  a semi-markov model doing joint entity type recognition and entity linking, with perceptron-style parameter estimation.
Revised: are used to compute the following metrics: [Equation] Mention level metrics would be the primary concept recognition metrics of interest when, for example, the model is used as a component in a relation extraction system. As another example, the

Original: The MM-ST21pv data presented to 
Revised: Disease recognition task in BC5-CDR described above uses mention level metrics. Document level metrics are computed in a similar manner, after mapping all concept mentions as entity labels directly to the document, and discarding all associations with spans of text that identify the locations of the mentions in the document. For example, a document may contain three mentions of the concept Breast Carcinoma in three different parts (spans) of the document text; for document level metrics they are all mapped to one label on the document. Document level metrics are useful in information retrieval when the goal is simply to retrieve the entire matching document, and are used in the BioASQ Large Scale Semantic Indexing task mentioned earlier.

Revised: 4.2 End-to-end Entity Recognition and Linking with TaggerOne

Revised: TaggerOne [Leaman and Lu, 2016] is a semi-Markov model doing joint entity type recognition and entity linking, with perceptron-style parameter estimation.

Revised: It is a ﬂexible package that handles simultaneous recognition of multiple entity types, with published results near state-of-the-art, e.g. for joint Chemical and Disease recognition on the BC5-CDR corpus.

Revised: We used the package without any changes to its modeling features.

Original:  TaggerOne was modiﬁed as follows: for each mention of a concept in the data, the Semantic Type label was modiﬁed to one of the 21 semantic types (table 2) that concept mapped into.
Revised: The MM-ST21pv data presented to TaggerOne was modiﬁed as follows: for each mention of a concept in the data, the Semantic Type label was modiﬁed to one of the 21 semantic types (Table 2) that concept mapped into.

Original: Thus each mention was labeled with one of 21 entity types, as well as linked to a speciﬁc concept from our ST21pv subset of UMLS.
Revised: Thus each mention was labeled with one of 21 entity types, as well as linked to a speciﬁc concept from the ST21pv subset of UMLS.

Original: Twenty one ‘lexicons’ of primary and alias names for each concept in the 21 types, extracted from UMLS 2017 AA Active, were also provided to TaggerOne.
Revised: Twenty one lexicons of primary and alias names for each concept in the 21 types, extracted from UMLS 2017 AA Active, were also provided to TaggerOne.

Original: Training was performed on the ‘Training’ split, and the ‘Dev’ split was provided as the ‘holdout’ data (validation data, used for stopping training).
Revised: Training was performed on the Training split, and the Development split was provided as holdout data (validation data, used for stopping training).

Original: The model was trained with  REGULARIZATION = 0 , MAX STEP SIZE = 1.5 and patience ( iterationsPastLastImprovement )ofepoch, with a maximum of 10 epochs.
Revised: The model was trained with the parameters: REGULARIZATION = 0 , MAX STEP SIZE = 1.5 , for a maximum of 10 epochs with patience ( iterationsPastLastImprovement ) of 1 epoch.

Original: Both mention level (aka phrase-level) metrics, and document level metrics on the MM-ST21pv Test split are reported in table 7.
Revised: Our model took 9 days to train on a machine equipped with Intel Xeon Broadwell processors and over 900GB of RAM.

Original: Mention level metrics are calculated by only counting an exact mention phrase match (including location of phrase in  the text) and linked entity match between the prediction and reference (ground truth in MM-ST21pv) as a true-positive. All other predicted mentions are counted as falsepositives and all un-matched reference entity mentions are counted as false-negatives.
Revised: When TaggerOne detects a concept in a document it identiﬁes a span of text (start and end positions) within the document, and labels it with an entity type and links it to a concept from that type. Metrics are calculated by comparing these concept predictions against the reference or ground truth (i e. the annotations in MM-ST21pv).

Original: Document level metrics are computed in a similar manner, after mapping all mentions as entity labels (without the mention phrase) on the document. Document level metrics are useful in information retrieval when the goal is simply to retrieve the entire matching document.
Revised: As a baseline for future work on biomedical concept recognition, both mention level and document level metrics for the TaggerOne model, computed on the MM-ST21pv Test subset, are reported in Table 7.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We presented the formal release of a new resource, named MedMentions, for biomedical concept recognition, with a large manually annotated annotated corpus of over 4,000 abstracts targeting a very large ﬁne-grained concept ontology consisting of over 3 million concepts.
Revised: We presented the formal release of a new resource, named MedMentions, for biomedical concept recognition, with a large manually annotated annotated corpus of over 4,000 abstracts targeting a very large ﬁne-grained concept ontology consisting of over 3 million concepts.

Original: We also included in this release a targeted sub-corpus (MedMentions ST21pv), and standard training / dev / test splits of the data to allow researchers to compare the metrics of their models
Revised: We also included in this release a targeted sub-corpus (MedMentions ST21pv), with standard training, development and test splits of the data, and the metrics of a baseline concept recognition model trained on this subset, to allow researchers to compare the metrics of their concept recognition models.

Original: Acknowledgments
Revised: Acknowledgments

Original: Omitted for blind review.
Revised: Omitted for blind review.