Original: E ND - TO - END NAMED ENTITY RECOGNITION AND RELATION EXTRACTION USING PRE - TRAINED LANGUAGE MODELS
Revised: E ND - TO - END NAMED ENTITY RECOGNITION AND RELATION EXTRACTION USING PRE - TRAINED LANGUAGE MODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Named entity recognition (NER) and relation extraction (RE) are two important tasks in information extraction and retrieval (IE & IR).
Revised: Named entity recognition (NER) and relation extraction (RE) are two important tasks in information extraction and retrieval (IE & IR).

Original: Recent work has demonstrated that it is beneﬁcial to learn these tasks jointly, which avoids the propagation of error inherent in pipeline-based systems and improves performance.
Revised: Recent work has demonstrated that it is beneﬁcial to learn these tasks jointly, which avoids the propagation of error inherent in pipeline-based systems and improves performance.

Original: However, state-of-the-art joint models typically rely on external natural language processing (NLP) tools, such as dependency parsers, limiting their usefulness to domains (e.g. news) where those tools perform well.
Revised: However, state-of-the-art joint models typically rely on external natural language processing (NLP) tools, such as dependency parsers, limiting their usefulness to domains (e.g. news) where those tools perform well.

Original: The few neural, end-to-end models that have been proposed are trained almost completely from scratch.
Revised: The few neural, end-to-end models that have been proposed are trained almost completely from scratch.

Original: In this paper, we propose a neural, end-to-end model for jointly extracting entities and their relations which does not rely on external NLP tools and which integrates a large, pre-trained language model.
Revised: In this paper, we propose a neural, end-to-end model for jointly extracting entities and their relations which does not rely on external NLP tools and which integrates a large, pre-trained language model.

Original: Because the bulk of our model’s parameters are pre-trained and we eschew recurrence for self-attention, our model is fast to train.
Revised: Because the bulk of our model’s parameters are pre-trained and we eschew recurrence for self-attention, our model is fast to train.

Original: On 5 datasets across 3 domains, our model matches or exceeds state-of-the-art performance, sometimes by a large margin.
Revised: On 5 datasets across 3 domains, our model matches or exceeds state-of-the-art performance, sometimes by a large margin.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The extraction of named entities (named entity recognition, NER) and their semantic relations (relation extraction, RE) are key tasks in information extraction and retrieval (IE & IR).
Revised: The extraction of named entities (named entity recognition, NER) and their semantic relations (relation extraction, RE) are key tasks in information extraction and retrieval (IE & IR).

Original: Given a sequence of text (usually a sentence), the objective is to identify both the named entities and the relations between them.
Revised: Given a sequence of text (usually a sentence), the objective is to identify both the named entities and the relations between them.

Original: This information is useful in a variety of NLP tasks such asdocument summarization, question answering, knowledge base population, and semantic search (Jiang, 2012).
Revised: This information is useful in a variety of NLP tasks such as question answering, knowledge base population, and semantic search (Jiang, 2012).

Original: In the biomedical domain, NER and RE facilitate large-scale biomedical data analysis, such as network biology (Zhou et al., 2014), gene prioritization (Aerts et al., 2006), drug repositioning (Wang & Zhang, 2013) and the creation of curated databases (Li et al., 2015).
Revised: In the biomedical domain, NER and RE facilitate large-scale biomedical data analysis, such as network biology (Zhou et al., 2014), gene prioritization (Aerts et al., 2006), drug repositioning (Wang & Zhang, 2013) and the creation of curated databases (Li et al., 2015).

Original: In the clinical domain, NER and RE can aid in disease and treatment prediction, readmission prediction, de-identiﬁcation, and patient cohort identiﬁcation (Miotto et al., 2017).
Revised: In the clinical domain, NER and RE can aid in disease and treatment prediction, readmission prediction, de-identiﬁcation, and patient cohort identiﬁcation (Miotto et al., 2017).

Original: Most commonly, the tasks of NER and RE are approached as a pipeline, with NER preceding RE.
Revised: Most commonly, the tasks of NER and RE are approached as a pipeline, with NER preceding RE.

Original: There are two main drawbacks to this approach: (1) Pipeline systems are prone to error propagation between the NER and RE systems.
Revised: There are two main drawbacks to this approach: (1) Pipeline systems are prone to error propagation between the NER and RE systems.

Original: (2) One task is not able to exploit useful information from the other (e.g. the type of relation identiﬁed by the RE system may be useful to the NER system for determining the type of entities involved in the relation, and vice versa).
Revised: (2) One task is not able to exploit useful information from the other (e.g. the type of relation identiﬁed by the RE system may be useful to the NER system for determining the type of entities involved in the relation, and vice versa).

Original: More recently, joint models which simultaneously learn to extract entities and relations have been proposed, alleviating the aforementioned issues and achieving state-of-the-art performance (Miwa & Sasaki, 2014; Miwa &Bansal, 2016;
Revised: More recently, joint models that simultaneously learn to extract entities and relations have been proposed, alleviating the aforementioned issues and achieving state-of-the-art performance (Miwa & Sasaki, 2014; Miwa &

Original:  Li et al., 2016; 2017; Zhang et al., 2017; Bekoulis et al., 2018a;b; Nguyen & 
Revised: Bansal, 2016; Gupta et al., 2016; Li et al., 2016; 2017; Zhang et al., 2017; Adel & Sch¨utze, 2017;

Original:  Verspoor, 2019; Li et al., 2019).
Revised: Bekoulis et al., 2018a;b; Nguyen & Verspoor, 2019; Li et al., 2019).

Original: Many of the proposed joint models for entity and relation extraction rely heavily on external NLP tools such as dependency parsers.
Revised: Many of the proposed joint models for entity and relation extraction rely heavily on external natural language processing (NLP) tools such as dependency parsers.

Original: For instance, Miwa & Bansal (2016) propose a recurrent neural network (RNN)-based joint model that uses a bidirectional long-short term memory network (BiLSTM) to model the entities and a tree-LSTM to model the relations between entities; Li et al.
Revised: For instance, Miwa & Bansal (2016) propose a recurrent neural network (RNN)-based joint model that uses a bidirectional long-short term memory network (BiLSTM) to model the entities and a tree-LSTM to model the relations between entities; Li et al.

Original: (2017) propose a similar model for biomedical text.
Revised: (2017) propose a similar model for biomedical text.

Original: The tree-LSTM uses dependency tree information extracted using an external dependency parser to model relations between entities.
Revised: The tree-LSTM uses dependency tree information extracted using an external dependency parser to model relations between entities.

Original: The use of these external NLP tools limits the effectiveness of a model to domains (e.g. news) where those NLP tools perform well.
Revised: The use of these external NLP tools limits the effectiveness of a model to domains (e.g. news) where those NLP tools perform well.

Original: As a remedy to this problem, Bekoulis et al.
Revised: As a remedy to this problem, Bekoulis et al.

Original: (2018a) propose a neural, end-to-end system which jointly learns to extract entities and relations without relying on external NLP tools.
Revised: (2018a) proposes a neural, end-to-end system that jointly learns to extract entities and relations without relying on external NLP tools.

Original: In Bekoulis et al.
Revised: In Bekoulis et al.

Original: (2018b), they augment this model with adversarial training.
Revised: (2018b), they augment this model with adversarial training.

Original: Nguyen & Verspoor (2019) propose a different, albeit similar end-to-end neural model which makes use of deep biafﬁne attention (Dozat & Manning, 2016).
Revised: Nguyen & Verspoor (2019) propose a different, albeit similar end-to-end neural model which makes use of deep biafﬁne attention (Dozat & Manning, 2016).

Original: Li et al.
Revised: Li et al.

Original: (2019) approach the problem with multiturn question answering, posing templated queries to a BERT-based QA model (Devlin et al., 2018) whose answers constitute extracted entities and their relations and achieve state-of-the-art results on three popular benchmark datasets.
Revised: (2019) approach the problem with multi-turn question answering, posing templated queries to a BERT-based QA model (Devlin et al., 2018) whose answers constitute extracted entities and their relations and achieve state-of-the-art results on three popular benchmark datasets.

Original: While demonstrating strong performance, end-to-end systems like Bekoulis et al.
Revised: While demonstrating strong performance, end-to-end systems like Bekoulis et al.

Original: (2018a;b) and
Revised: (2018a;b) and

Original: Nguyen & Verspoor (2019) suffer from two main drawbacks.
Revised: Nguyen & Verspoor (2019) suffer from two main drawbacks.

Original: The ﬁrst is that most of the models parameters are trained from scratch.
Revised: The ﬁrst is that most of the models parameters are trained from scratch.

Original: For large datasets, this can lead to long training times.
Revised: For large datasets, this can lead to long training times.

Original: For small datasets, which are common in the biomedical and clinical domains where it is particularly challenging to acquire labelled data, this can lead to poor performance and/or overﬁtting.
Revised: For small datasets, which are common in the biomedical and clinical domains where it is particularly challenging to acquire labelled data, this can lead to poor performance and/or overﬁtting.

Original: The second is that these systems typically contain RNNs, which are sequential in nature and cannot be parallelized within training examples.
Revised: The second is that these systems typically contain RNNs, which are sequential in nature and cannot be parallelized within training examples.

Original: The multi-pass QA model proposed in Li et al.
Revised: The multi-pass QA model proposed in Li et al.

Original: (2019) alleviates these issues by incorporating a pre-trained language model, BERT (Devlin et al., 2018), which eschews recurrence for self-attention.
Revised: (2019) alleviates these issues by incorporating a pre-trained language model, BERT (Devlin et al., 2018), which eschews recurrence for self-attention.

Original: The main limitation of their approach is that it relies on handcrafted question templates to achieve maximum performance.
Revised: The main limitation of their approach is that it relies on handcrafted question templates to achieve maximum performance.

Original: This may become a limiting factor where domain expertise is required to craft such questions (e.g., for biomedical or clinical corpora).
Revised: This may become a limiting factor where domain expertise is required to craft such questions (e.g., for biomedical or clinical corpora).

Original: Additionally, one has to create a question template for each entity and relation type of interest.
Revised: Additionally, one has to create a question template for each entity and relation type of interest.

Original: In this study, we propose an end-to-end model for joint NER and RE which addresses all of these issues.
Revised: In this study, we propose an end-to-end model for joint NER and RE which addresses all of these issues.

Original: Similar to past work, our model can be viewed as a mixture of a NER module and a RE module (Figure 1).
Revised: Similar to past work, our model can be viewed as a mixture of a NER module and a RE module (Figure 1).

Original: Unlike most previous works, we include a pre-trained, transformer-based language model, speciﬁcally BERT (Devlin et al., 2018), which achieved state-of-the-art performance across many NLP tasks.
Revised: Unlike most previous works, we include a pre-trained, transformer-based language model, speciﬁcally BERT (Devlin et al., 2018), which achieved state-of-the-art performance across many NLP tasks.

Original: The weights of the BERT model are ﬁne-tuned during training, and the entire model is trained in an end-to-end fashion.
Revised: The weights of the BERT model are ﬁne-tuned during training, and the entire model is trained in an end-to-end fashion.

Original: Our main contributions are as follows: (1) Our solution is truly end-to-end, relying on no handcrafted features (e.g. templated questions) or external NLP tools (e.g. dependency parsers).
Revised: Our main contributions are as follows: (1) Our solution is truly end-to-end, relying on no handcrafted features (e.g. templated questions) or external NLP tools (e.g. dependency parsers).

Original:  Our model is fast to train (e.g. 12 minutes on a single GPU for the CoNLL04 corpus), as most of its parameters are pre-trained and we avoid recurrence.
Revised: (2) Our model is fast to train (e.g. under 10 minutes on a single GPU for the CoNLL04 corpus), as most of its parameters are pre-trained and we avoid recurrence.

Original: (3) We match or exceed state-of-the-art performance for joint NER and RE on 5 datasets across 3 domains.
Revised: (3) We match or exceed state-of-the-art performance for joint NER and RE on 5 datasets across 3 domains.

Original:  T HE M ODEL
Revised:  T HE M ODEL

Original: Figure 1 illustrates the architecture of our approach.
Revised: Figure 1 illustrates the architecture of our approach.

Original: Our model is composed of an NER module and an RE module.
Revised: Our model is composed of an NER module and an RE module.

Original: The NER module is identical to the one proposed by Devlin et al.
Revised: The NER module is identical to the one proposed by Devlin et al.

Original: In short, for a given input sequence s of N word tokens w 1 , w 2 , . . . , w N , the pre-trained BERT BASE model ﬁrst produces a sequence of vectors, x ( NER ), x ( NER ), . . .
Revised: For a given input sequence s of N word tokens w 1 , w 2 , . . . , w N , the pre-trained BERT BASE model ﬁrst produces a sequence of vectors, x ( NER ), x [Equation]

Original: , x ( NER ) N which are then fed to a feed-forward neural network (FFNN) for classiﬁcation.
Revised: , x [Equation] N which are then fed to a feed-forward neural network (FFNN) for classiﬁcation.

Original: [Equation]
Revised: [Equation]

Original: The output size of this layer is the number of BIOES-based NER labels in the training data, | C ( NER ) | .
Revised: The output size of this layer is the number of BIOES-based NER labels in the training data, | C ( NER ) | .

Original: In the BIOES tag scheme, each token is assigned a label, where the B- tag indicates the beginning of an entity span, I- the inside, E- the end and S- is used for any single-token entity.
Revised: In the BIOES tag scheme, each token is assigned a label, where the B- tag indicates the beginning of an entity span, I- the inside, E- the end and S- is used for any single-token entity.

Original: All other tokens are assigned the label O.
Revised: All other tokens are assigned the label O.

Original: During training, a cross-entropy loss is computed for the NER objective,
Revised: During training, a cross-entropy loss is computed for the NER objective,

Original: [Equation]
Revised: [Equation]

Original: where s ( NER ) n is the predicted score that token n ∈ N belongs to the ground-truth entity class and s ( NER ) n,c is the predicted score for token n belonging to the entity class c ∈ C ( NER ) .
Revised: where s ( NER ) n is the predicted score that token n ∈ N belongs to the ground-truth entity class and s ( NER ) n,c is the predicted score for token n belonging to the entity class c ∈ C ( NER ) .

Original: In the RE module, the predicted entity labels are obtained by taking the argmax of each score vector s ( NER ), s ( NER ), . . .
Revised: In the RE module, the predicted entity labels are obtained by taking the argmax of each score vector s ( NER ), s ( NER ), . . .

Original: , s ( NER ) N .
Revised: , s ( NER ) N .

Original: The predicted entity labels are then embedded to produce a sequence of vectors e ( NER ), e ( NER ), . . .
Revised: The predicted entity labels are then embedded to produce a sequence of ﬁxed-length, continuous vectors, e ( NER ), e ( NER ), . . .

Original: , e ( NER ) N and concatenated with the hidden states from the ﬁnal layer in the BERT model.
Revised: , e ( NER ) N which are concatenated with the hidden states from the ﬁnal layer in the BERT model and learned jointly with the rest of the models parameters.

Original: [Equation]
Revised: [Equation]

Original: Following Miwa & Bansal (2016) and Nguyen & Verspoor (2019), we incrementally construct the set of relation candidates, R , using all possible combinations of the last word tokens of predicted entities, i.e. words with E- or S- labels.
Revised: Following Miwa & Bansal (2016) and Nguyen & Verspoor (2019), we incrementally construct the set of relation candidates, R , using all possible combinations of the last word tokens of predicted entities, i.e. words with E- or S- labels.

Original: An entity pair is assigned to a negative relation class (NEG) when the pair has no relation or when the predicted entities are not correct.
Revised: An entity pair is assigned to a negative relation class (NEG) when the pair has no relation or when the predicted entities are not correct.

Original: Once relation candidates are constructed, classiﬁcation is performed with a deep bilinear attention mechanism (Dozat & Manning, 2016), as proposed by Nguyen & Verspoor (2019).
Revised: Once relation candidates are constructed, classiﬁcation is performed with a deep bilinear attention mechanism (Dozat & Manning, 2016), as proposed by Nguyen & Verspoor (2019).

Original: First, to encode directionality, the mechanism usestwo FFNNs to project each x ( RE ) i into head and tail vector representations, which correspond to whether the i th word serves asthe head or tail argument of the relation. 
Revised: To encode directionality, the mechanism uses FFNNs to project each x ( RE ) i into head and tail vector representations, corresponding to whether the i th word serves as head or tail argument of the relation. [Equation]

Original: [Equation]
Revised: [Equation]

Original: These projections are then fed to a biafﬁne classiﬁer,
Revised: These projections are then fed to a biafﬁne classiﬁer,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where U is an m ×| C ( RE )
Revised: where U is an m ×| C ( RE )

Original: |× m tensor, W is a | C ( RE ) |× ( [Equation] m ) matrix, and b is a bias vector.
Revised: |× m tensor, W is a | C ( RE ) |× (2 ∗ m ) matrix, and b is a bias vector.

Original: Here, m is the size of the output layers of FFNN head and FFNN tail and C ( RE ) is the set of all relation classes (including NEG).
Revised: Here, m is the size of the output layers of FFNN head and FFNN tail and C ( RE ) is the set of all relation classes (including NEG).

Original: During training, a second cross-entropy loss is computed for the RE objective
Revised: During training, a second cross-entropy loss is computed for the RE objective

Original: [Equation]
Revised: [Equation]

Original: where s ( RE ) r is the predicted score that relation candidate r ∈ R belongs to the ground-truth relation class and s ( RE ) r,c is the predicted score for relation r belonging to the relation class c ∈ C ( RE ) .
Revised: where s ( RE ) r is the predicted score that relation candidate r ∈ R belongs to the ground-truth relation class and s ( RE ) r,c is the predicted score for relation r belonging to the relation class c ∈ C ( RE ) .

Original: The model is trained in an end-to-end fashion to minimize the sum of the NER and RE losses.
Revised: The model is trained in an end-to-end fashion to minimize the sum of the NER and RE losses.

Original: [Equation]
Revised: [Equation]

Original: 2. E NTITY P RETRAINING
Revised: 2. E NTITY P RETRAINING

Original: In Miwa & Bansal (2016), entity pre-training is proposed as a solution to the problem of lowperformance entity detection in the early stages of training.
Revised: In Miwa & Bansal (2016), entity pre-training is proposed as a solution to the problem of lowperformance entity detection in the early stages of training.

Original: It is implemented by delaying the training of the RE module by some number of epochs, before training the entire model jointly.
Revised: It is implemented by delaying the training of the RE module by some number of epochs, before training the entire model jointly.

Original: Our implementation of entity pretraining is slightly different.
Revised: Our implementation of entity pretraining is slightly different.

Original: Instead of delaying training of the RE module by some number of epochs, we weight the contribution of L RE to the total loss during the ﬁrst epoch of training,
Revised: Instead of delaying training of the RE module by some number of epochs, we weight the contribution of L RE to the total loss during the ﬁrst epoch of training

Original: [Equation]
Revised: [Equation]

Original: where λ is increased linearly from 0 to 1 during the ﬁrst epoch and set to 1 for the remaining epochs.
Revised: where λ is increased linearly from 0 to 1 during the ﬁrst epoch and set to 1 for the remaining epochs.

Original: We chose this scheme because the NER module quickly achieves good performance for all datasets (i.e., within one epoch).
Revised: We chose this scheme because the NER module quickly achieves good performance for all datasets (i.e. within one epoch).

Original: In early experiments, we found this scheme to outperform a delay of a full epoch.
Revised: In early experiments, we found this scheme to outperform a delay of a full epoch.

Original: 2. I MPLEMENTATION
Revised: 2. I MPLEMENTATION

Original: We implemented our model in PyTorch (Paszke et al., 2017) using the BERT BASE model from the
Revised: We implemented our model in PyTorch (Paszke et al., 2017) using the BERT BASE model from the

Original: PyTorch Transformers library 1 .
Revised: PyTorch Transformers library 1 .

Original: Our model is available at our GitHub repository 2 .
Revised: Our model is available at our GitHub repository 2 .

Original: Furthermore, we use NVIDIAs automatic mixed precision (AMP) library Apex 3 to speed up training and reduce memory usage without affecting task-speciﬁc performance.
Revised: Furthermore, we use NVIDIAs automatic mixed precision (AMP) library Apex 3 to speed up training and reduce memory usage without affecting task-speciﬁc performance.

Original:  E XPERIMENTAL S ETUP 3. D ATASETS AND EVALUATION
Revised:  E XPERIMENTAL S ETUP 3. D ATASETS AND EVALUATION

Original: To demonstrate the generalizability of our model, we evaluate it on 5 commonly used benchmark corpora across 3 domains.
Revised: To demonstrate the generalizability of our model, we evaluate it on 5 commonly used benchmark corpora across 3 domains.

Original: All corpora are in English.
Revised: All corpora are in English.

Original: Detailed corpus statistics are presented in
Revised: Detailed corpus statistics are presented in

Original: Table A.1 of the appendix.
Revised: Table A.1 of the appendix.

Original: 3.1. ACE04/ 
Revised: 3.1. ACE04/

Original: The Automatic Content Extraction (ACE04) corpus was introduced by Doddington et al.
Revised: The Automatic Content Extraction (ACE04) corpus was introduced by Doddington et al.

Original: (2004), and is commonly used to benchmark NER and RE methods.
Revised: (2004), and is commonly used to benchmark NER and RE methods.

Original: There are 7 entity types and 7 relation types.
Revised: There are 7 entity types and 7 relation types.

Original: ACE05 builds on ACE04, splitting the Physical relation into two classes ( Physical and PartWhole ), removing the Discourse relation class and merging Employment-Membership-Subsidiary and Person-Organization-Afﬁliation into one class ( Employment-Membership-Subsidiary ).
Revised: ACE05 builds on ACE04, splitting the Physical relation into two classes ( Physical and PartWhole ), removing the Discourse relation class and merging Employment-Membership-Subsidiary and Person-Organization-Afﬁliation into one class ( Employment-Membership-Subsidiary ).

Original: For ACE04, we follow Miwa & Bansal (2016) by removing the Discourse relation and evaluating our model using 5-fold cross-validation on the bnews and nwire subsets, where 10% of the data was held out within each fold as a validation set.
Revised: For ACE04, we follow Miwa & Bansal (2016) by removing the Discourse relation and evaluating our model using 5-fold cross-validation on the bnews and nwire subsets, where 10% of the data was held out within each fold as a validation set.

Original: For ACE05, we use the same test split as Miwa & Bansal (2016).
Revised: For ACE05, we use the same test split as Miwa & Bansal (2016).

Original: We use 5-fold cross-validation on the remaining data to choose the hyperparameters.
Revised: We use 5-fold cross-validation on the remaining data to choose the hyperparameters.

Original: Once hyperparameters are chosen, we train on the combined data from all the folds and evaluate on the test set.
Revised: Once hyperparameters are chosen, we train on the combined data from all the folds and evaluate on the test set.

Original: For both corpora, we report the micro-averaged F 1 score.
Revised: For both corpora, we report the micro-averaged F 1 score.

Original: We obtained the pre-processing scripts from Miwa & Bansal (2016) 4 .
Revised: We obtained the pre-processing scripts from Miwa & Bansal (2016) 4 .

Original: 3.1. C O NLL 
Revised: 3.1. C O NLL

Original: The CoNLL04 corpus was introduced in Roth & Yih (2004)for learning the problem of simultaneously recognizing entities and relations, and consists of articles from the Wall Street Journal (WSJ) and Associated Press (AP).
Revised: The CoNLL04 corpus was introduced in Roth & Yih (2004) and consists of articles from the Wall Street Journal (WSJ) and Associated Press (AP).

Original: There are 4 entity types and 5 relation types.
Revised: There are 4 entity types and 5 relation types.

Original: As there are no ofﬁcial splits, we use the same test set split as Miwa & Sasaki (2014) 5 .
Revised: We use the same test set split as Miwa & Sasaki (2014) 5 .

Original: We use 5-fold cross-validation on the remaining data to choose hyperparameters.
Revised: We use 5-fold cross-validation on the remaining data to choose hyperparameters.

Original: Once hyperparameters are chosen, we train on the combined data from all folds and evaluate on the test set, reporting the micro-averaged F 1 score.
Revised: Once hyperparameters are chosen, we train on the combined data from all folds and evaluate on the test set, reporting the micro-averaged F 1 score.

Original: 3.1. ADE
Revised: 3.1. ADE

Original: The adverse drug event corpus was introduced by Gurulingappa et al.
Revised: The adverse drug event corpus was introduced by Gurulingappa et al.

Original: (2012) to serve as a benchmark for systems which aim to identify adverse drug events from free-text.
Revised: (2012) to serve as a benchmark for systems that aim to identify adverse drug events from free-text.

Original: It consists of the abstracts of medical case reports retrieved from PubMed 6 .
Revised: It consists of the abstracts of medical case reports retrieved from PubMed 6 .

Original: There are two entity types, Drug and Adverse effect and one relation type, Adverse drug event .
Revised: There are two entity types, Drug and Adverse effect and one relation type, Adverse drug event .

Original: Similar to previous work (Li et al., 2016; 2017; Bekoulis et al., 2018b), we remove ∼ 130 relations with overlapping entities and evaluate our model using 10-fold cross-validation, where 10% of the data within each fold was used as a validation set, 10% as a test set and the remaining data is used as a train set.
Revised: Similar to previous work (Li et al., 2016; 2017; Bekoulis et al., 2018b), we remove ∼ 130 relations with overlapping entities and evaluate our model using 10-fold cross-validation, where 10% of the data within each fold was used as a validation set, 10% as a test set and the remaining data is used as a train set.

Original: We report the macro F 1 score averaged across all folds.
Revised: We report the macro F 1 score averaged across all folds.

Original: 3.1. I 2 B
Revised: 3.1. I 2 B

Original: The 2010 i2b2/VA dataset was introduced by Uzuner et al.
Revised: The 2010 i2b2/VA dataset was introduced by Uzuner et al.

Original: (2011) for the 2010 i2b2/Va Workshop on Natural  Processing Challenges for Clinical Records.
Revised: (2011) for the 2010 i2b2/Va Workshop on Natural Language Processing Challenges for Clinical Records.

Original: The workshop contained an NER task focused on the extraction of 3 medical entity types ( Problem , Treatment , Test ) and an RE task for 8 relation types.
Revised: The workshop contained an NER task focused on the extraction of 3 medical entity types ( Problem , Treatment , Test ) and an RE task for 8 relation types.

Original: In the ofﬁcial splits, the test set contains roughly twice as many examples as the train set.
Revised: In the ofﬁcial splits, the test set contains roughly twice as many examples as the train set.

Original: To increase the number of training examples while maintaining a rigorous evaluation, we elected to perform 5fold cross-validation on the combined data from both partitions.
Revised: To increase the number of training examples while maintaining a rigorous evaluation, we elected to perform 5fold cross-validation on the combined data from both partitions.

Original: We used 10% of the data within each fold as a validation set, 20% as a test set and the remaining data was used as a train set.
Revised: We used 10% of the data within each fold as a validation set, 20% as a test set and the remaining data was used as a train set.

Original: We report the micro F 1 score averaged across all folds.
Revised: We report the micro F 1 score averaged across all folds

Original: To the best of our knowledge, we are the ﬁrst to evaluate a joint NER and RE model on thei2b2/VA dataset.
Revised: To the best of our knowledge, we are the ﬁrst to evaluate a joint NER and RE model on thei2b2/VA dataset.

Original: Therefore, we decided to compare to scores obtained by independent NER and RE systems.
Revised: Therefore, we decided to compare to scores obtained by independent NER and RE systems.

Original: We note, however, that the scores of independent RE systems are not directly comparable to the scores we report in this paper.
Revised: We note, however, that the scores of independent RE systems are not directly comparable to the scores we report in this paper.

Original: This is because RE is traditionally framed as a sentence-level classiﬁcation problem.
Revised: This is because RE is traditionally framed as a sentence-level classiﬁcation problem.

Original: During pre-processing, each example is permutated into processed examples containing two “blinded” entities and labelled for one relation class.
Revised: During pre-processing, each example is permutated into processed examples containing two “blinded” entities and labelled for one relation class.

Original: E.g. the example: “His PCP had recently started ciproﬂoxacin TREATMENT for a UTI PROBLEM ” becomes “His PCP had recently started @TREATMENT$ for a @PROBLEM$”, where the model is trained to predict the target relation type, “Treatment is administered for medical problem” (TrAP).
Revised: E.g. the example: “His PCP had recently started ciproﬂoxacin TREATMENT for a UTI PROBLEM ” becomes “His PCP had recently started @TREATMENT$ for a @PROBLEM$”, where the model is trained to predict the target relation type, “Treatment is administered for medical problem” (TrAP).

Original: We argue that this task is inherently easier than the joint setup, for two reasons: relation predictions are made on ground-truth entities, as opposed to predicted entities (which are noisy) and the model is only required to make one classiﬁcation decision per pre-processed sentence.
Revised: This task is inherently easier than the joint setup, for two reasons: relation predictions are made on ground-truth entities, as opposed to predicted entities (which are noisy) and the model is only required to make one classiﬁcation decision per pre-processed sentence.

Original: In the joint setup, a model must identify any number of relations (or the lack thereof) between all unique pairs of predicted entities in a given input sentence.
Revised: In the joint setup, a model must identify any number of relations (or the lack thereof) between all unique pairs of predicted entities in a given input sentence.

Original: To control for the ﬁrst of these differences, we report scores from our model in two settings, once when predicted entities are used as input to the RE module, and once when ground-truth entities are used.
Revised: To control for the ﬁrst of these differences, we report scores from our model in two settings, once when predicted entities are used as input to the RE module, and once when ground-truth entities are used.

Original: 3. H YPERPARAMETERS
Revised: 3. H YPERPARAMETERS

Original: Besides batch size, learning rate and number of training epochs, we used the same hyperparameters across all experiments (see Table A.2).
Revised: Besides batch size, learning rate and number of training epochs, we used the same hyperparameters across all experiments (see Table A.2).

Original: Similar to Devlin et al.
Revised: Similar to Devlin et al.

Original: (2018), learning rate and batch size were selected for each dataset using a minimal grid search (see See Table A.3).
Revised: (2018), learning rate and batch size were selected for each dataset using a minimal grid search (see See Table A.3).

Original: One hyperparameter selected by hand was the choice of the pre-trained weights used to initialize the
Revised: One hyperparameter selected by hand was the choice of the pre-trained weights used to initialize the

Original: BERT BASE model.
Revised: BERT BASE model.

Original: For general domain corpora, we found the cased BERT BASE weights from Devlin et al.
Revised: For general domain corpora, we found the cased BERT BASE weights from Devlin et al.

Original: (2018) to work well.
Revised: (2018) to work well.

Original: For biomedical corpora, we used the weights from BioBERT (Lee et al., 2019), which recently demonstrated state-of-the-art performance for biomedical NER, RE and QA.
Revised: For biomedical corpora, we used the weights from BioBERT (Lee et al., 2019), which recently demonstrated state-of-the-art performance for biomedical NER, RE and QA.

Original: Similarly, for clinical corpora we use the weights provided by Peng et al.
Revised: Similarly, for clinical corpora we use the weights provided by Peng et al.

Original: (2019), who pre-trained
Revised: (2019), who pre-trained

Original: BERT BASE on PubMed abstracts and clinical notes from MIMIC-III 7 .
Revised: BERT BASE on PubMed abstracts and clinical notes from MIMIC-III 7 .

Original:  R ESULTS 4. J OINTLY LEARNING NER AND RE
Revised:  R ESULTS 4. J OINTLY LEARNING NER AND RE

Original: Table 1 shows our results in comparison to previously published results, grouped by the domain of the evaluated corpus.
Revised: Table 1 shows our results in comparison to previously published results, grouped by the domain of the evaluated corpus.

Original: We ﬁnd that on every dataset besides i2b2, our model improves NER performance, for an average improvement of ∼ 2%.
Revised: We ﬁnd that on every dataset besides i2b2, our model improves NER performance, for an average improvement of ∼ 2%.

Original: This improvement is particularly large on the
Revised: This improvement is particularly large on the

Original: ACE04 and ACE05 corpora (3.98% and 2.41% respectively).
Revised: ACE04 and ACE05 corpora (3.98% and 2.41% respectively).

Original: On i2b2, our joint model performs within 0.29% of the best independent NER solution.
Revised: On i2b2, our joint model performs within 0.29% of the best independent NER solution.

Original: For relation extraction, we outperform previous methods on 2 datasets and come within ∼ 2% on both ACE05 and CoNLL04.
Revised: For relation extraction, we outperform previous methods on 2 datasets and come within ∼ 2% on both ACE05 and CoNLL04.

Original: In two cases, our performance improvement is substantial, with improvements of 4.59% and 10.25% on the ACE and ADE corpora respectively.
Revised: In two cases, our performance improvement is substantial, with improvements of 4.59% and 10.25% on the ACE04 and ADE corpora respectively.

Original: For i2b2, our score is not directly comparable to previous systems (as discussed in section 3.1.4) but will facilitate future comparisons of joint NER and RE methods on this dataset.
Revised: For i2b2, our score is not directly comparable to previous systems (as discussed in section 3.1.4) but will facilitate future comparisons of joint NER and RE methods on this dataset.

Original: By comparing overall performance, we ﬁnd that our approach achieves new stateof-the-art performance for 3 popular benchmark datasets (ACE04, ACE05, ADE) and comes within 0.2% for CoNLL04.
Revised: By comparing overall performance, we ﬁnd that our approach achieves new state-of-the-art performance for 3 popular benchmark datasets (ACE04, ACE05, ADE) and comes within 0.2% for CoNLL04.

Original: 4. A BLATION A NALYSIS
Revised: 4. A BLATION A NALYSIS

Original: To determine which training strategies and components are responsible for our models performance, we conduct an ablation analysis on the CoNLL04 corpus (Table 2).
Revised: To determine which training strategies and components are responsible for our models performance, we conduct an ablation analysis on the CoNLL04 corpus (Table 2).

Original: We perform ﬁve different ablations: (a) Without entity pre-training (see section 2.1), i.e. the loss function is given by equation 9.
Revised: We perform ﬁve different ablations: (a Without entity pre-training (see section 2.1), i e. the loss function is given by equation 9.

Original: (b) Without entity embeddings, i.e. equation 3 becomes x ( RE )i = x ( NER )
Revised: (b) Without entity embeddings, i e. equation 3 becomes x ( RE )

Original: i.
Revised: i

Original: (c) Replacing the two feed-forward neural networks, FFNN head and FFNN tail with a single FFNN (see equation 4 and 5).
Revised: c) Replacing the two feed-forward neural networks, FFNN head and FFNN tail with a single FFNN (see equation 4 and 5).

Original: (d ) Removing FFNN head and FFNN tail entirely.
Revised: (d) Removing FFNN head and FFNN tail entirely.

Original: (e) Without the bilinear operation
Revised: (e) Without the bilinear operation,

Original: , i.e. equationbecomes a simple linear transformation.
Revised: i e. equationbecomes a simple linear transformation.

Original: Removing FFNN head and FFNN tail(see equation 4 and 5) has, by far, the largest negative impact on performance.
Revised: Removing FFNN head and FFNN tail has, by far, the largest negative impact on performance.

Original: Interestingly, however, replacing FFNN head and FFNN tail with a single FFNN has only a small negative impact.
Revised: Interestingly, however, replacing FFNN head and FFNN tail with a single FFNN has only a small negative impact.

Original: This suggests that while these layers are very important for model performance, using distinct FFNNs for the projection of head and tail entities (as opposed to the same FFNN) is relatively much less important.
Revised: This suggests that while these layers are very important for model performance, using distinct FFNNs for the projection of head and tail entities (as opposed to the same FFNN) is relatively much less important.

Original: The next most impactful ablation was entity pre-training, suggesting that low-performance entity detection during the early stages of training is detrimental to learning (see section 2.1).
Revised: The next most impactful ablation was entity pre-training, suggesting that low-performance entity detection during the early stages of training is detrimental to learning (see section 2.1).

Original: Finally, we note that the importance of entity embeddings is surprising, as a previous study has found that entity embeddings did not help performance on the CoNLL corpusspeciﬁcally (Bekoulis et al., 2018a), although their architecture was markedly different.
Revised: Finally, we note that the importance of entity embeddings is surprising, as a previous study has found that entity embeddings did not help performance on the CoNLL04 corpus (Bekoulis et al., 2018a), although their architecture was markedly different.

Original: We conclude that each of our ablated components is necessary to achieve maximum performance.
Revised: We conclude that each of our ablated components is necessary to achieve maximum performance.

Original: 4. A NALYSIS OF THE WORD - LEVEL ATTENTION WEIGHTS
Revised: 4. A NALYSIS OF THE WORD - LEVEL ATTENTION WEIGHTS

Original: One advantage of including a transformer-based language model is that we can easily visualize the attention weights with respect to some input.
Revised: One advantage of including a transformer-based language model is that we can easily visualize the attention weights with respect to some input.

Original: This visualization is useful, for example, in detecting model bias and locating relevant attention heads (Vig, 2019).
Revised: This visualization is useful, for example, in detecting model bias and locating relevant attention heads (Vig, 2019).

Original: Previous works have used such visualizations to demonstrate that speciﬁc attention heads mark syntactic dependency relations and 
Revised: Previous works have used such visualizations to demonstrate that speciﬁc attention heads mark syntactic dependency relations and that lower layers tend to learn more about syntax while higher layers tend to encode more semantics (Raganato & Tiedemann, 2018). In Figure 2 we visualize the attention weights of select layers and attention heads from an instance of BERT ﬁne-tuned within our model on the CoNLL04 corpus. We display four patterns that are easily

Original: [Table]
Revised: [Table]

Original: To the best of our knowledge
Revised: best [Table] published

Original: [Table]
Revised: [Table]

Original: there are no published joint NER and RE modelsthat evaluate on the i2b2 2010 corpus.
Revised: NER [Table] RE models

Original: [Table]
Revised: [Table]

Original: compare our model to the state-of-the-art for each individual task (see section 3.1.4)
Revised: model to the state-of-the-

Original: [Table]
Revised: [Table]

Original: the
Revised: for

Original: [Table]
Revised: [Table]

Original: their
Revised: task

Original: [Table] [Table]
Revised: [Table] [Table]

Original: that lower layers tend to learn more about syntax while higher layers tend to encode more semantics (Raganato & Tiedemann, 2018).

Original: In Figure 2 we visualize the attention weights of select layers and attention heads from an instance of BERT ﬁne-tuned within our model on the CoNLL04 corpus.

Original: We display four patterns that are easily interpreted: paying attention to the next and previous word, paying attention to the word itself, and paying attention to the end of the sentence.
Revised: interpreted: paying attention to the next and previous words, paying attention to the word itself, and paying attention to the end of the sentence.

Original: These same patterns have been found in pre-trained
Revised: These same patterns have been found in pre-trained

Original: BERT models that have not been ﬁne-tuned on a speciﬁc, supervised task (Vig, 2019; Raganato &
Revised: BERT models that have not been ﬁne-tuned on a speciﬁc, supervised task (Vig, 2019; Raganato &

Original: Tiedemann, 2018), and therefore, are retained after our ﬁne-tuning procedure.
Revised: Tiedemann, 2018), and therefore, are retained after our ﬁne-tuning procedure.

Original: To facilitate further analysis of our learned model, we make available Jupyter and Google Colaboratory notebooks on our GitHub repository 8 , where users can use multiple views to explore the learned attention weights of our models.
Revised: To facilitate further analysis of our learned model, we make available Jupyter and Google Colaboratory notebooks on our GitHub repository 8 , where users can use multiple views to explore the learned attention weights of our models.

Original: We use the BertViz library (Vig, 2019) to render the interactive,
Revised: We use the BertViz library (Vig, 2019) to render the interactive,

Original: HTML-based views and to access the attention weights used to plot the heat maps.
Revised: HTML-based views and to access the attention weights used to plot the heat maps.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this paper, we introduced an end-to-end model for entity and relation extraction.
Revised: In this paper, we introduced an end-to-end model for entity and relation extraction.

Original: Our key contributions are: (1) No reliance on any hand-crafted features (e.g. templated questions) or external NLP tools (e.g. dependency parsers).
Revised: Our key contributions are: (1) No reliance on any hand-crafted features (e.g. templated questions) or external NLP tools (e.g. dependency parsers).

Original: (2) Integration of a pre-trained, transformer-based language model.
Revised: (2) Integration of a pre-trained, transformer-based language model.

Original: (3) State-of-the-art performance on 5 datasets across 3 domains.
Revised: (3) State-of-the-art performance on 5 datasets across 3 domains.

Original: Furthermore, our model is inherently modular.
Revised: Furthermore, our model is inherently modular.

Original: One can easily initialize the language model with pre-trained weights better suited for a domain of interest (e.g. BioBERT for biomedical corpora) or swap BERT for a comparable language model (e.g. XLNet
Revised: One can easily initialize the language model with pre-trained weights better suited for a domain of interest (e.g. BioBERT for biomedical corpora) or swap BERT for a comparable language model (e.g. XLNet

Original: (Yang et al., 2019)).
Revised: (Yang et al., 2019)).

Original: Finally, because of (2), our model is fast to train, converging in approximately 1 hour or less on a single GPU for all datasets used in this study.
Revised: Finally, because of (2), our model is fast to train, converging in approximately 1 hour or less on a single GPU for all datasets used in this study.

Original: Our model out-performed previous state-of-the-art performance on ADE by the largest margin (6.53%).
Revised: Our model out-performed previous state-of-the-art performance on ADE by the largest margin (6.53%).

Original: While exciting, we believe this corpus was particularly easy to learn.
Revised: While exciting, we believe this corpus was particularly easy to learn.

Original: The majority of sentences ( ∼ 68%) are annotated for two entities ( drug and adverse effect , and one relation ( adverse drug event ).
Revised: The majority of sentences ( ∼ 68%) are annotated for two entities ( drug and adverse effect , and one relation ( adverse drug event ).

Original: Ostensibly, a model should be able to exploit this pattern to get near-perfect performance on the majority of sentences in the corpus.
Revised: Ostensibly, a model should be able to exploit this pattern to get near-perfect performance on the majority of sentences in the corpus.

Original: As a test, we ran our model again, this time using ground-truth entities in the RE module (as opposed to predicted entities) and found that the model very quickly reached almost perfect performance for RE on the test set ( ∼ 98%).
Revised: As a test, we ran our model again, this time using ground-truth entities in the RE module (as opposed to predicted entities) and found that the model very quickly reached almost perfect performance for RE on the test set ( ∼ 98%).

Original: As such, high performance on the ADE corpus is not likely to transfer to real-world scenarios involving the large-scale annotation of diverse biomedical articles.
Revised: As such, high performance on the ADE corpus is not likely to transfer to real-world scenarios involving the large-scale annotation of diverse biomedical articles.

Original: In our experiments, we consider only intra-sentence relations.
Revised: In our experiments, we consider only intra-sentence relations.

Original: However, the multiple entities within a document generally exhibit complex, inter-sentence relations.
Revised: However, the multiple entities within a document generally exhibit complex, inter-sentence relations.

Original: Our model is not currently capable of extracting such inter-sentence relations and therefore our restriction to intra-sentence relations will limit its usefulness for certain downstream tasks, such as knowledge base creation.
Revised: Our model is not currently capable of extracting such inter-sentence relations and therefore our restriction to intra-sentence relations will limit its usefulness for certain downstream tasks, such as knowledge base creation.

Original: We also ignore the problem of nested entities, which are common in biomedical corpora.
Revised: We also ignore the problem of nested entities, which are common in biomedical corpora.

Original: In the future, we would like to extend our model to handle both nested entities and inter-sentence relations.
Revised: In the future, we would like to extend our model to handle both nested entities and inter-sentence relations.

Original: Finally, given that multilingual, pre-trained weights for BERT exist, we would also expect our model’s performance to hold across multiple languages.
Revised: Finally, given that multilingual, pre-trained weights for BERT exist, we would also expect our model’s performance to hold across multiple languages.

Original: We leave this question to future work.
Revised: We leave this question to future work.

Original: A A PPENDIX 
Revised: A A PPENDIX A. C ORPUS S TATISTICS

Revised: Table A.1 lists detailed statistics for each corpus used in this study. [Table] [Table]

Revised: Subsidiary (

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Person
Revised: [Table] Afﬁliation (141), Geopolitical [Table]

Revised: [Table]

Revised: [Table]

Revised: Entities

Original: [Table]
Revised: [Table]

Original: Organization (4405),
Revised: ), Location [Table] ), Facility

Original: [Table]
Revised: [Table]

Original: Entities (4425), Location [Table]
Revised: , Weapon (911), and [Table]

Original: [Table]
Revised: [Table]

Original: Facility (688), Weapon (119),
Revised: Physical (1612), PartWhole (1060), Person

Original: [Table]
Revised: [Table]

Original: Vehicle (209)
Revised: Social (

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Subsidiary
Revised: Agent

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: Artifact (703), Employment-MembershipSubsidiary (1922), Geopolitical Entity-Afﬁliation (730)

Original: [Table]
Revised: [Table]

Revised: Organization (

Revised: [Table]

Original:  Kill (268), Live in (521), Located in (406), OrgBased in (452),
Revised: [Table] ), Other ( [Table] Kill (268), Live in (521), Located in (406), OrgBased in (452),

Original: Work for (401)
Revised: Work for (401)

Original: [Table]
Revised: Biomedical

Original: ADE
Revised: ADE

Original: Drug (4979),
Revised: Drug (4979),

Original: [Table]
Revised: Adverse effect (5669)

Original: [Table] event (6682)
Revised: Adverse drug event (6682)

Original: [Table]
Revised: Clinical i2b

Original: Problem (19664), Test (13831),
Revised: Problem (19664), Test (13831),

Original: Treatment (14186)
Revised: Treatment (14186)

Original: PIP (2203), TeCP (504), TeRP (3053), TrAP (2617), TrCP (526),
Revised: PIP (2203), TeCP (504), TeRP (3053), TrAP (2617), TrCP (526),

Original: TrIP (203), TrNAP (174), TrWP (133)
Revised: TrIP (203), TrNAP (174), TrWP (133)

Original: [Table] Single token entities are tagged with an S- tag, the beginning of an entity span with a B- tag, the last token of an entity span with an E- tag, and tokens inside an entity span with an I- tag. [Table] [Table]
Revised: A. H YPERPARAMETERS AND M ODEL D ETAILS

Original: Dropout rate applied to the output of all FFNNs and the attention heads of the BERT model.

Original: [Table]

Original: Output dimension of the entity embedding layer.

Original: [Table]

Original: Output dimension

Original: [Table]

Original: No

Original: [Table]

Original: Number

Original: [Table] used [Table] AdamW Adam with ﬁxed weight decay regulatization (Loshchilov & Hutter, 2017).
Revised: Table A.2 lists hyperparameters and model details that were held constant across all experiments.

Original: [Table]

Original: Γ = Rescales the gradient whenever the norm goes over some threshold Γ (Pascanu et al., 2013). [Table] [Table]
Revised: Table A.3 lists those that were speciﬁc to each evaluated corpus.

Original: Devlin et al.,

Original: [Table]
Revised: [Table]

Original: ) (Lee et al., 2019)

Original: [Table][Table]
Revised: [Table]

Original: NCBI-BERT (uncased) (Peng et al., 2019)