Original: T OWARDS I NTERPRETABLE E VALUATIONS : A C ASE S TUDY OF N AMED E NTITY R ECOGNITION
Revised: T OWARDS I NTERPRETABLE E VALUATIONS : A C ASE S TUDY OF N AMED E NTITY R ECOGNITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: With the proliferation of models for natural language processing (NLP) tasks, it is even harder to understand the differences between models and their relative merits.
Revised: With the proliferation of models for natural language processing (NLP) tasks, it is even harder to understand the differences between models and their relative merits.

Original: Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 do not tell us why or how a particular method is better and how dataset biases inﬂuence the choices of model design.
Revised: Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 do not tell us why or how a particular method is better and how dataset biases inﬂuence the choices of model design.

Original: In this paper, we present a general methodology for interpretable evaluation of NLP systems and choose the task of named entity recognition (NER) as a case study, which is a core task of identifying people, places, or organizations in text.
Revised: In this paper, we present a general methodology for interpretable evaluation of NLP systems and choose the task of named entity recognition (NER) as a case study, which is a core task of identifying people, places, or organizations in text.

Original: The proposed evaluation method enables us to interpret the model biases , dataset biases , and how the differences in the datasets affect the design of the models, identifying the strengths and weaknesses of current approaches.
Revised: The proposed evaluation method enables us to interpret the model biases , dataset biases , and how the differences in the datasets affect the design of the models, identifying the strengths and weaknesses of current approaches.

Original: By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive the progress in this area.
Revised: By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive the progress in this area.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The development of deep neural networks has greatly sped the evolution of NLP systems.
Revised: The development of deep neural networks has greatly sped the evolution of NLP systems.

Original: However, these advances have also come with a plethora of design decisions: should we choose a CNN-based (Kalchbrenner et al., 2014;
Revised: However, these advances have also come with a plethora of design decisions: should we choose a CNN-based (Kalchbrenner et al., 2014;

Original: Kim, 2014), RNN-based (Sutskever et al., 2014; Bahdanau et al., 2014) or
Revised: Kim, 2014), RNN-based (Sutskever et al., 2014; Bahdanau et al., 2014) or

Original: Transformer-based (Vaswani et al., 2017; Dai et al., 2018) architecture?
Revised: Transformer-based (Vaswani et al., 2017; Dai et al., 2018) architecture?

Original: What variety of pre-training method should we use (Le & Mikolov, 2014; Peters et al., 2018;
Revised: What variety of pre-training method should we use (Le & Mikolov, 2014; Peters et al., 2018;

Original: Devlin et al., 2018; Akbik et al., 2018)?
Revised: Devlin et al., 2018; Akbik et al., 2018)?

Original: The proliferation of model variants pose a great challenge for current evaluation methodology, which are usually opaque and simply give a single holistic score (Papineni et al., 2002; Banerjee &
Revised: The proliferation of model variants pose a great challenge for current evaluation methodology, which are usually opaque and simply give a single holistic score (Papineni et al., 2002; Banerjee &

Original: Lavie, 2005; Popovi ´ c & Ney, 2011).
Revised: Lavie, 2005; Popovi ´ c & Ney, 2011).

Original: To alleviate this problem, researchers have made efforts, mainly focusing in two directions.
Revised: To alleviate this problem, researchers have made efforts, mainly focusing in two directions.

Original: First, some works (Farr ´ us Cabeceran et al., 2010; Popovi ´ c & Ney, 2011; Lommel et al., 2014) have attempted to shift the granularity of evaluation from holistic to ﬁne-grained by conducting error analysis.
Revised: First, some works (Farr ´ us Cabeceran et al., 2010; Popovi ´ c & Ney, 2011; Lommel et al., 2014) have attempted to shift the granularity of evaluation from holistic to ﬁne-grained by conducting error analysis.

Original: Despite its effectiveness, the process of error analysis usually requires manual examination and depends on some pre-existing assumptions, suffering from conﬁrmation bias, and risking ignoring new types of errors (Neubig et al., 2019).
Revised: Despite its effectiveness, the process of error analysis usually requires manual examination and depends on some pre-existing assumptions, suffering from conﬁrmation bias, and risking ignoring new types of errors (Neubig et al., 2019).

Original: Additionally, this evaluation method based on error analysis is usually applied to only a single dataset (Karpathy et al., 2015; Kummerfeld & Klein, 2013; Kummerfeld et al., 2012), lacking discussion of ﬁne-grained analysis in a multi-dataset setting .
Revised: Additionally, this evaluation method based on error analysis is usually applied to only a single dataset (Karpathy et al., 2015; Kummerfeld & Klein, 2013; Kummerfeld et al., 2012), lacking discussion of ﬁne-grained analysis in a multi-dataset setting .

Original: As a result, many important questions remain unclear: how to characterize the factors that inﬂuence the tasks for different datasets?
Revised: As a result, many important questions remain unclear: how to characterize the factors that inﬂuence the tasks for different datasets?

Original: how do the different choices of datasets inﬂuence the models’ performance?
Revised: how do the different choices of datasets inﬂuence the models’ performance?

Original: Another way to improve the evaluation strategy is common in our routine experimental design.
Revised: Another way to improve the evaluation strategy is common in our routine experimental design.

Original: That is to evaluate our models on multiple datasets with a holistic metric (Peters et al., 2018;
Revised: That is to evaluate our models on multiple datasets with a holistic metric (Peters et al., 2018;

Original: Devlin et al., 2018).
Revised: Devlin et al., 2018).

Original: Currently, researchers are making efforts along this direction by setting up general evaluation benchmarks, such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), which involve a diverse set of datasets.
Revised: Currently, researchers are making efforts along this direction by setting up general evaluation benchmarks, such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), which involve a diverse set of datasets.

Original: Although it enables us to gain a more comprehensive assessment of the models, the inﬂuence of different datasets on models is simply reﬂected by a holistic metric, which is not interpretable, and consequently, we are not clear about how different datasets inﬂuence the choices of model architectures.
Revised: Although it enables us to gain a more comprehensive assessment of the models, the inﬂuence of different datasets on models is simply reﬂected by a holistic metric, which is not interpretable, and consequently, we are not clear about how different datasets inﬂuence the choices of model architectures.

Original: In this paper, we argue that a complete evaluation method should not only reﬂect the individual performance of the model on one dataset or multiple datasets but also be able to interpret the model biases, dataset biases, and their correlation (how the difference in the datasets affects the design of the models).
Revised: In this paper, we argue that a complete evaluation method should not only reﬂect the individual performance of the model on one dataset or multiple datasets but also be able to interpret the model biases, dataset biases, and their correlation (how the difference in the datasets affects the design of the models).

Original: We draw on the complementary strengths of the ﬁne-grained evaluation and multidataset evaluation, driving ﬁne-grained analysis to the multi-dataset setting.
Revised: We draw on the complementary strengths of the ﬁne-grained evaluation and multidataset evaluation, driving ﬁne-grained analysis to the multi-dataset setting.

Original: To this end, we devise a generalized evaluation methodology and choose the NER task as a case study.
Revised: To this end, we devise a generalized evaluation methodology and choose the NER task as a case study.

Original: Concretely, we introduce the notion of attribute , which can be deﬁned ﬂexibly as the evaluation task needs.
Revised: More speciﬁcally, we introduce the notion of attribute , which can be deﬁned ﬂexibly as the evaluation task needs.

Original: Here, we utilize the attribute to describe the property of each test entity for the NER task (i.e., entity length).
Revised: Here, we utilize the attribute to describe the property of each test entity for the NER task (i.e., entity length).

Original: Then, the test set will be divided into a set of buckets by different attributes of test entities.
Revised: Then, the test set will be divided into a set of buckets by different attributes of test entities.

Original: This makes it possible to evaluate recognition accuracy of different varieties of entities, achieving much more ﬁne-grained analysis than standard corpus-level measures.[Table] [Table] [Table]
Revised: This makes it possible to evaluate recognition accuracy of different varieties of entities, achieving much more ﬁne-grained analysis than standard corpus-level measures.

Original: +EvJUOs7LiDU6Nj4xmZvKT8/Mzs0XFhbraZyJgNWCOIzFie+lLOQRq0kuQ3aSCOb1/JAd

Original: [Table]
Revised: [Table]

Original: 909FT++YCLlcXQkrxLW6nmdiJ/xwJNEtQvrTckupfZpiI7f6jslZ8M209bWoL/POyV7c5DPtwtFovWwfwLXgCLMqMaFZzRxihgBMvTAEEESDuEhpa8BFw4S4lroEycIcR1nGCBP2oyyGGV4xHZp7tCuYdiI9soz1eqATgnpF6S0sUqamPIEYXWareOZdlbsb9597anudkWrb7x6xEqcE/uXbpj5X52qReIMu7oGTjUlmlHVBcYl

Original: [Table]
Revised: [Table]

Original: /qkqSQ0KcwqcUF4QDrRy

Original: [Table]
Revised: [Table]

Original: /qltRg93s7f4J6ueQ6JfewXKxsm1bnsIwVrFE/ [Table] 4j1RoxmiV8Gdb9OxfQnQM=</latexit>
Revised: /8I74xR8IDohM2fOvefM3Ll+EvJUOs7LiDU6Nj4xmZvKT8/Mzs0XFhbraZyJgNWCOIzFie+lLOQRq0kuQ3aSCOb1/JAd+909FT++YCLlcXQkrxLW6nmdiJ/xwJNEtQvrTckupfZpiI7f6jslZ8M209bWoL/POyV7c5DPtwtFovWwfwLXgCLMqMaFZzRxihgBMvTAEEESDuEhpa8BFw4S4lroEycIcR1nGCBP2oyyGGV4xHZp7tCuYdiI9soz1eqATgnpF6S0sUqamPIEYXWareOZdlbsb9597anudkWrb7x6xEqcE/uXbpj5X52qReIMu7oGTjUlmlHVBcYl06+ibm5/qkqSQ0KcwqcUF4QDrRy+s601qa5dva2n4686U7FqH5jcDG/qltRg93s7f4J6ueQ6JfewXKxsm1bnsIwVrFE/d1DBAaqokfc1HvCIJyuxbqxb6+4j1RoxmiV8Gdb9OxfQnQM=</latexit>

Original: Additionally, the proposed attribute-aided evaluation methodology encourages us to introduce multiple attributes to ﬁnd more potential factors which affect the NER models on different datasets.
Revised: Additionally, the proposed attribute-aided evaluation methodology encourages us to introduce multiple attributes to ﬁnd more potential factors which affect the NER models on different datasets.

Original: We further propose three analytical approaches as shown in Fig.
Revised: We further propose three analytical approaches as shown in Fig.

Original: 1: attribute-wise, model-wise, and bucketwise analyses that have the following characteristics accordingly: Attribute-wise (Sec. 4.3.2) analysis could instruct us to ﬁnd which factors matter for the NER tasks and ﬁgure out the commonality of factors across different NER datasets;
Revised: 1: attribute-wise, model-wise, and bucketwise analyses that have the following characteristics accordingly: Attribute-wise (Sec. 3.3.2) analysis could instruct us to ﬁnd which factors matter for the NER tasks and ﬁgure out the commonality of factors across different NER datasets;

Original: Model-wise (Sec. 4.3.1) analysis aims to investigate how different attributes inﬂuence the performance of models with different architectures and pre-trained knowledge; Bucket-wise (Sec. 4.3.3) analysis diagnoses the strengths and weaknesses of existing models and helps us understand how different choices of datasets inﬂuence the model performance.
Revised: Model-wise (Sec. 3.3.1) analysis aims to investigate how different attributes inﬂuence the performance of models with different architectures and pre-trained knowledge; Bucket-wise (Sec. 3.3.3) analysis diagnoses the strengths and weaknesses of existing models and helps us understand how different choices of datasets inﬂuence the model performance.

Original: Our contributions can be precisely summarized as: 1) We draw on the complementary strengths of the ﬁne-grained evaluation and multi-dataset evaluation, proposing a generalized evaluation methodology to interpret model biases, dataset biases, and their correlation.
Revised: Our contributions can be precisely summarized as: 1) We draw on the complementary strengths of the ﬁne-grained evaluation and multi-dataset evaluation, proposing a generalized evaluation methodology to interpret model biases, dataset biases, and their correlation.

Original: We choose the NER task as a test case, the observations based on the extensive experiments (twelve models, ten attributes and six datasets) suggest directions for improvement and can drive the progress of this area.
Revised: We choose the NER task as a test case, the observations based on the extensive experiments (twelve models, ten attributes and six datasets) suggest directions for improvement and can drive the progress of this area.

Original: Although some attributes deﬁned in this paper are task-dependent, we claim our methodology is general since: a) for sequence labeling tasks (i.e., Partof-Speech, text chunking, and extractive summarization), this evaluation method could be transferred without much modiﬁcation.
Revised: Although some attributes deﬁned in this paper are task-dependent, we claim our methodology is general since: a) for sequence labeling tasks (i.e., Partof-Speech, text chunking, and extractive summarization), this evaluation method could be transferred without much modiﬁcation.

Original: b) for other types of NLP tasks, we could re-deﬁne the attributes and our proposed bucketization strategies, as well as three analytic approaches, are task-agnostic.
Revised: b) for other types of NLP tasks, we could re-deﬁne the attributes and our proposed bucketization strategies, as well as three analytic approaches, are task-agnostic.

Original: I NTERPRETATION OF E VALUATION M ETHODOLOGY AND R ELATED W ORK
Revised: For example, in machine translation, the attribution could be sentence length (Luong et al., 2015), word itself (n-gram) in the reference ﬁle (Kumar & Tsvetkov, 2018); In question answering, the attribution could be answer length/type/position (Seo et al., 2016), document length (Joshi et al., 2017), and query length/type (Chen et al., 2016b). Once we have determined related attributes, we could make similar analyses based on our proposed measures.

Revised:  P RELIMINARIES

Original: We ﬁrst summarize and compare past evaluation methodologies and then position our work in the context of relevant research.
Revised: We ﬁrst summarize and compare past evaluation methodologies and then describe the NER task and its current evaluation strategy.

Original: 2. P ROPERTIES OF E VALUATION M ETHODOLOGIES
Revised: 2. P ROPERTIES OF E VALUATION M ETHODOLOGIES

Original:  Evaluation is gaining increasing interest in NLP, especially on text generation tasks as exempliﬁed by machine translation (Fishel et al., 2012;
Revised: [Table] Evaluation is gaining increasing interest in NLP, especially on text generation tasks as exempliﬁed by machine translation (Fishel et al., 2012;

Original: Irvine et al., 2013; Daems et al., 2014).
Revised: Irvine et al., 2013; Daems et al., 2014).

Original: Generally, different evaluation methods can be characterized by the following three main properties:
Revised: Generally, different evaluation methods can be characterized by the following three main properties:

Original: [Table]

Original: Interpretability : Evaluation method could give interpretable results to help us understand where the weaknesses and strengths are.
Revised: Interpretability : Evaluation method could give interpretable results to help us understand where the weaknesses and strengths are.

Original: For example, “ error analysis ”(Kummerfeld et al., 2012; Kummerfeld & Klein, 2013; Karpathy et al., 2015) is an interpretable evaluation method since we could ﬁgure out detailed limitations of the evaluated systems.
Revised: For example, “ error analysis ”(Kummerfeld et al., 2012; Kummerfeld & Klein, 2013; Karpathy et al., 2015) is an interpretable evaluation method since we could ﬁgure out detailed limitations of the evaluated systems.

Original: Conﬁrmation bias : It represents a tendency to make tests consistent with the beforehand hypothesis.
Revised: Conﬁrmation bias : It represents a tendency to make tests consistent with the beforehand hypothesis.

Original: For example, Mudrakarta et al.
Revised: For example, Mudrakarta et al.

Original: (2018) assume that deep learning model are sensitive to question words in question answering tasks and verify it by carefully-designed adversarial examples, and Chen et al. 
Revised: (2018) assume that deep learning model are sensitive to question words in question answering tasks and verify it by carefully-designed adversarial examples, and Chen et al. (2016a)

Original: pre-deﬁned six error to classify error cases.
Revised: pre-deﬁned six error to classify error cases.

Original: Supplementary exam : It is an additional exam (require extra test sets ) for more comprehensive observations.
Revised: Supplementary exam : It is an additional exam (require extra test sets ) for more comprehensive observations.

Original: i.e. multi-dataset evaluation (Devlin et al., 2018) on GLUE, adversarial test (Jia & Liang, 2017) and stress test (Naik et al., 2018). 2. R ELATED W ORK Our work can be uniquely positioned in the context of the following two aspects.
Revised: i.e. multi-dataset evaluation (Devlin et al., 2018) on GLUE.

Original: Methodological perspective

Original: The development of evaluation methods is heavily driven by machine translation tasks.

Original: Popovi´c & Ney (2011); Lommel et al.

Original: (2014) pointed out the limitations of traditional evaluation with holistic metric like BLEU (Papineni et al., 2002), METEOR (Banerjee &

Original: Lavie, 2005).

Original: And the focus of MT evaluation research is gradually shifting towards error analysis (Fishel et al., 2012;

Original: Irvine et al., 2013; Daems et al., 2014), which involves manual and automatic ways.

Original: Recently, there is a trend going from traditional evaluation to diagnostic test , in which an extra test set is required.
Revised: Recently, there is a trend going from traditional evaluation to diagnostic test , in which a supplementary test set is required.

Original: For example, (Jia & Liang, 2017) propose an adversarial test for reading comprehension task and (Naik et al., 2018) present a stress test method to diagnose natural language inference systems. These methods usually focus on one dataset, not considering how models are inﬂuenced by dataset biases. Task perspective
Revised: For example, Jia & Liang (2017) propose an adversarial test for reading comprehension task and Naik et al.

Original: Existing NER systems are commonly evaluated by corpus-level metrics ( F 1score) (Sang & De Meulder, 2003) and a small amount of work will conduct some manual error analysis (Ichihara et al., 2015; Derczynski et al., 2015). With the increasing improvement in network architectures and pre-trained knowledge, the NER systems are quickly reaching a performance plateau (Akbik et al., 2018; Akbik et al.). Therefore, ﬁne-grained evaluation is required to identify the speciﬁc issues of existing NER systems.
Revised: (2018) present a stress test method to diagnose natural language inference systems.

Original: On the other hand, with the emergence of more and more NER datasets (Sang & De Meulder, 2003; Collobert et al., 2011; Weischedel et al., 2013), the time is ripe for us to bridge the gap between the insufﬁcient understanding of the nature of datasets itself and model designs.

Original: In this paper, we choose the NER task as a test case, interpreting model biases, dataset biases, as well as their correlation under a general framework.

Original: This work also takes a step towards interpretable architecture searching.

Original:  T ASK AND S TANDARD E VALUATION S TRATEGY3. T ASK
Revised: 2. NER T ASK AND C URRENT E VALUATION S TRATEGY

Original:  Named entity recognition (NER) is usually formulated as a sequence labeling problem (Huang et al., 2015; Ma & Hovy, 2016).
Revised: Task Description Named entity recognition (NER) is usually formulated as a sequence labeling problem (Huang et al., 2015; Ma & Hovy, 2016).

Original: Formally, let X = { x 1 , x 2 , . . .
Revised: Formally, let X = { x 1 , x 2 , . . .

Original: , x T } be an input sequence and 
Revised: , x T } be an input sequence and Y =

Original: Y = { y 1 , y 2 , . . .
Revised: { y 1 , y 2 , . . .

Original: , y T } be the output tags.
Revised: , y T } be the output tags.

Original: The goal of this task is to estimate the conditional probability: P ( Y | X ) =
Revised: The goal of this task is to estimate the conditional probability: P ( Y | X ) =

Original: P ( y t | X, y 1 , · · · , y t − 1 )
Revised: P ( y t | X, y 1 , · · · , y t − 1 )

Original: The common evaluation for NER  systems Sang & De Meulder (2003) is to compute a corpus-level metric using F 1 score:
Revised: Evaluation Strategy for NER Existing NER systems are commonly evaluated by corpus-level metrics ( F 1 -score) (Sang & De Meulder, 2003) and a small amount of work will conduct some manual error analysis (Ichihara et al., 2015; Derczynski et al., 2015).

Original: F 1 = 2 × P × R P
Revised: With the increasing improvement in network architectures and pre-trained knowledge, the NER systems are quickly reaching a performance plateau (Akbik et al., 2018; Akbik et al.).

Original: + R : where P is  the percentage of named entities found by learning system that are correct.
Revised: Therefore, ﬁne-grained evaluation is required to identify the speciﬁc issues of existing NER systems.

Original: R is  the percentage ofnamed entities present in the dataset that are found by the system.
Revised: On the other hand, with the emergence of more and more NER datasets (Sang & De Meulder, 2003; Collobert et al., 2011; Weischedel et al., 2013), the time is ripe for us to bridge the gap between the insufﬁcient understanding of the nature of datasets itself and model designs. In this paper, we choose the NER task as a test case, interpreting model biases, dataset biases, as well as their correlation under a general framework.

Original: Here a named entity is correct only if it is an exact match of annotated entity.
Revised: This work also takes a step towards interpretable architecture searching.

Original:  A TTRIBUTE - AIDED E VALUATION M ETHODOLOGY
Revised:  A TTRIBUTE - AIDED E VALUATION M ETHODOLOGY

Original: Our proposed evaluation methodology involves three key elements: attribute deﬁnition (Sec. 4.1), bucketization (Sec. 4.2) and the analytical approach (Sec. 4.3).
Revised: Our proposed evaluation methodology involves three key elements: attribute deﬁnition (Sec. 3.1), bucketization (Sec. 3.2) and the analytical approach (Sec. 3.3).

Original: Speciﬁcally, we ﬁrst introduce the notion of attribute and deﬁne it in NER task as some property of the test entity, bywhich the test set will be divided into different sub-sets and the overall performance could be broken down into interpretable categories.
Revised: Speciﬁcally, we ﬁrst introduce the notion of attribute and deﬁne it in NER task as some property of the test entity, bywhich the test set will be divided into different sub-sets and the overall performance could be broken down into interpretable categories.

Original: Below, we will detail the three key elements.
Revised: Below, we will detail the three key elements.

Original: 4. D EFINITION OF E NTITY A TTRIBUTES
Revised: 3. D EFINITION OF E NTITY A TTRIBUTES

Original: Entity Attributes refer to the properties that can be used to characterize a given entity.
Revised: Entity Attributes refer to the properties that can be used to characterize a given entity.

Original: Generally, different types of entity attributes provide different observation angles of system’s performances.
Revised: Generally, different types of entity attributes provide different observation angles of system’s performances.

Original: To legibly describe the studied problem, we follow the commonly used notations throughout the paper.

Original: Fig.

Original: 2 give an example of the attribute deﬁnition of the entity New York .

Original: We refer to E , P , K as the sets of entities (i.e. New York ), entity attributes (i.e. entity length ) and attributes values (i.e. 2 ).

Original: Next, we will introduce entity attributes we explored in this paper in terms of token level , span level and sentence level .
Revised: Next, we will introduce entity attributes we explored in this paper in terms of token level , span level and sentence level .

Original: We take them into consideration since they are general features and could be transferred to other tasks.[Figure] [Table]
Revised: We take them into consideration since they are general features and could be transferred to other tasks.

Original: /4FY/TfwD/QvvjFNQi+iEJGfOvefM3Hv9NAwy6TivC9bi0vLKamGtuL6xubVd2tltZ0kuGG+xJExE1/cyHgYxb8lAhrybCu5Ffsg7/vhcxTu3XGRBEjflJOX9yBvFwTBgniTqsnnSvC6VnYqjlz0PXAPKMKuRlF5whQESMOSIwBFDEg7hIaOnBxcOUuL6mBInCAU6znGPImlzyuKU4RE7pu+Idj3DxrRXnplWMzolpFeQ0sYhaRLKE4TVabaO59pZsb95T7WnutuE/r7xioiVuCH2L90s8786VYvEEGe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppWzPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAftasV1Ku5FtVw7NqMuYB8HOKJ5nqKGOhpokfcIj3jCs1W3Yiu37j5TrQWj2cO3ZT18AIopj9M=</

Original: [Figure] Token-level 1) Token itself: it denotes the words of an entity. 2) Morphology: morphological features plays an essential role in many NLP tasks.
Revised: Token-level 1) Token itself: it denotes the words of an entity. 2) Morphology: morphological features plays an essential role in many NLP tasks.

Original: Here, we deﬁne ﬁve cases for each entity token: including token, is upper case, lower case, digit, beginning with a capital letter, and others (such as punctuation).
Revised: Here, we deﬁne ﬁve cases for each entity token: including token, is upper case, lower case, digit, beginning with a capital letter, and others (such as punctuation).

Original: Span-level 1) Entity itself: each entity is regarded as a unique identiﬁer. 2) Length of entity: the number of tokens in an entity. 3) Label of entity: the NER label of an entity.
Revised: Span-level 1) Entity itself: each entity is regarded as a unique identiﬁer. 2) Entity length: the number of tokens in an entity. 3) Entity tag: the NER tag of an entity.

Original: Sentence-level 1) Sentence length; 2) Entity density: it is the number of entity tokens divided by sentence length.
Revised: Sentence-level 1) Sentence length; 2) Entity density: it is the number of entity tokens divided by sentence length.

Original: 3) OOV density: it is the number of oov word divided by sentence length.
Revised: 3) OOV density: it is the number of oov word divided by sentence length.

Original: 4. B UCKETIZATION S TRATEGIES
Revised: 3. B UCKETIZATION S TRATEGIES

Original: Bucketization is an operation that breaks down the holistic performance into different interpretable categories based on the attribute values.
Revised: Bucketization is an operation that breaks down the holistic performance into different interpretable categories based on the attribute values.

Original: This can be achieved by dividing the set of test entities into different subsets of test entities (for span- and sentence-level attributes) or test tokens (for token-level attributes).
Revised: This can be achieved by dividing the set of test entities into different subsets of test entities (for span- and sentence-level attributes) or test tokens (for token-level attributes).

Original: Without loss of generality, we describe the entity-based bucketization strategies while it can be easily applied to token-based.
Revised: Without loss of generality, we describe the entity-based bucketization strategies while it can be easily applied to token-based.

Revised: To legibly describe the studied problem, we follow the commonly used notations throughout the paper.

Revised: We refer to E , P , K as the sets of entities, entity attributes and attributes values, respectively.

Original: The bucketization process can be formulated in a general form: E te 1 , · · · , E tem =
Revised: The bucketization process can be formulated in a general form: E te 1 , · · · , E tem =

Original: Bucket( E te | E tr , p ) , where E tr , E te represent the sets of training and test entities, respectively, p denotes a type of the attribute.
Revised: Bucket( E te | E tr , p ) , where E tr , E te represent the sets of training and test entities, respectively, p denotes a type of the attribute.

Original: The basic idea is that the test entity set is divided into m buckets based on the attribute p and corresponding training set.
Revised: The basic idea is that the test entity set is divided into m buckets based on the attribute p and corresponding training set.

Original: where E tr , E te represent the sets of training and test entities, respectively, p denotes a type of the attribute.
Revised: where E tr , E te represent the sets of training and test entities, respectively, p denotes a type of the attribute.

Revised: [Table]

Original: Concretely, each subset of test entities can be obtained as follows:
Revised: Speciﬁcally, each subset of test entities can be obtained as follows:

Original: E te i =
Revised: E tei =

Original: { e | Atr( e, p ) ∈
Revised: { e | Atr( e, p ) ∈

Original: ˆ K i , ∀ e ∈ E te } , where Atr( e, p ) is to query out the value of attribute p for entity e .
Revised: ˆ K i , ∀ e ∈ E te } , where Atr( e, p ) is to query out the value of attribute p for entity e .

Original: ˆ K i is a set of attribute values, with which entities should be put into the i -th bucket: ˆ K i = { k | f ( k )
Revised: ˆ K i is a set of attribute values, with which entities should be put into the i -th bucket: ˆ K i = { k | f ( k )

Original: =
Revised: =

Original: i, ∀ k ∈ K } .
Revised: i, ∀ k ∈ K } .

Original: Above equation shows the key part for bucketization: how do we build the relationship between the value of entity attributes and the bucket number? .
Revised: Above equation shows the key part for bucketization: how do we build the relationship between the value of entity attributes and the bucket number? .

Original: That is, we need to determine a criterion f , which guides us to put test entities according to its attribute values into suitable buckets.
Revised: That is, we need to determine a criterion f , which guides us to put test entities according to its attribute values into suitable buckets.

Original: Here we explore three types of strategies for bucketization.
Revised: Here we explore three types of strategies for bucketization.

Original: Strategy-I: Range division of attribute values ( R-Bucket )
Revised: Strategy-I: Range division of attribute values ( R-Bucket )

Original: An intuitive strategy is to bucketize the test entities based on their attribute values directly.
Revised: An intuitive strategy is to bucketize the test entities based on their attribute values directly.

Original: At this time, the number of buckets is equal to the range of attribute values: m= | K | . This strategy is suitable for those attributes with discrete and ﬁnite values, such as the  length  attributes of an entity.
Revised: Speciﬁcally, we could divide the range of attribute values into m discrete parts. For example, the sentence length with range [1 , · · · , 6] could be divided into [1 , 2] , [3 , 4] , [5 , 6] three buckets. Tab. 2 shows which attributes are matched to this bucketization strategy.

Original: Strategy-II: Familiarity of attribute values ( F-Bucket )
Revised: Strategy-II: Familiarity of attribute values ( F-Bucket )

Original: The aim of the evaluation is to quantify the generalization errors of the system on the unseen samples.
Revised: The aim of the evaluation is to quantify the generalization errors of the system on the unseen samples.

Original: Therefore, by taking into account the degree to which the testing entities (or their attributes) have been seen in the training set, we can better ﬁgure out the impact of this attribute on model performance.
Revised: Therefore, by taking into account the degree to which the testing entities (or their attributes) have been seen in the training set, we can better ﬁgure out the impact of this attribute on model performance.

Original: To achieve this, here we introduce a notion of familiarity to quantify the degree to which the attribute of a test entity has been seen in the training set.
Revised: To achieve this, here we introduce a notion of familiarity to quantify the degree to which the attribute of a test entity has been seen in the training set.

Original: [Equation]
Revised: [Equation]

Original: Here, F k ( p ) ∈ [0 , 1] denotes the degree to which the test entity attribute p with value k have been seen in the training set.
Revised: Here, F k ( p ) ∈ [0 , 1] denotes the degree to which the test entity attribute p with value k have been seen in the training set.

Original: Then we can deﬁne the following criterion to achieve the bucketization:
Revised: Then we can deﬁne the following criterion to achieve the bucketization:

Original: F ˆ K i ( p ) ∈ ( im , i +1 m ] .
Revised: F ˆ K i ( p ) ∈ ( im , i +1 m ] .

Original: The basic idea behind the criterion is that test entities could be put into the i -th bucket when the familiarity of their attribute values meets the above condition.
Revised: The basic idea behind the criterion is that test entities could be put into the i -th bucket when the familiarity of their attribute values meets the above condition.

Revised: Tab. 2 shows which attributes we used for F-Bucket .

Original: Strategy-III: Multi-attribute Familiarity ( MF-Bucket )
Revised: Strategy-III: Multi-attribute Familiarity ( MF-Bucket )

Original: The beneﬁt of our general methodology for interpretable evaluation is that we can easily deﬁne new valuable measures based on old measures already deﬁned.
Revised: The beneﬁt of our general methodology for interpretable evaluation is that we can easily deﬁne new valuable measures based on old measures already deﬁned.

Original: Here, we can adapt our F-Bucket strategies to multi-attribute version and modify the calculation of F k ( p 1 , p 2 ) in the Eq.1 as follows:
Revised: Here, we can adapt our F-Bucket strategies to multi-attribute version and modify the Eq.

Original: [Equation]
Revised: [Equation]

Original: For example, when we instantiate the two entity attributes p 1 , p 2 as entity itself and tag respectively, the familiarity F k ( p 1 , p 2 ) is a measure with intriguing explanation: for each test entity with tag k , this measure quantiﬁes its category ambiguity : the probability that this entity is labeled as k in the training set.
Revised: as follows: [Equation]

Original: [Table]
Revised: [Table]

Original: Understanding Three Bucketization Strategies
Revised: /r7xioiVuCH2L90s8786VYvEEGe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppWzPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAftasV1Ku5FtVw7NqMuYB8HOKJ5nqKGOhpokfcIj3jCs1W3Yiu37j5TrQWj2cO3ZT18AIopj9M=</latexit>

Original: The process of R-Bucket is independent of the training set, and it directly divides the entity set based on attribute values. This strategy is suitable for those attributes with discrete and limited values. As shown in Tab.2, attributes entity length can be bucketized by this method. By contrast, F-bucket depends on the training set, reﬂecting the extent to which the test entity attributes have been seen in the training set.
Revised: For example, when we instantiate the two entity attributes p 1 , p 2 as entity itself and tag respectively, the familiarity F k ( p 1 , p 2 ) is a measure with intriguing explanation: for each test entity with tag k , this measure quantiﬁes its category ambiguity : the probability that this entity is labeled as k in the training set.

Original: The familiarity of attribute values is continuous. Regarding two attributes (MF-et,
Revised: To give a more intuitive understanding of the two processes: attribute deﬁnition and bucketization, Fig.

Original: MF-tt) using MF-bucket , they can quantify the degree of category ambiguity for each test entity.
Revised: 2 shows a concrete example, in which a set of attributes are deﬁned for the target entity New York , with corresponding values.

Original: 4. A NALYTIC A PPROACHES
Revised: 3. A NALYTIC A PPROACHES

Original: To better characterize the relation among attributes, models and datasets, we propose three analytical approaches: attribute-wise, model-wise, and bucket-wise, which can be used to interpret the model biases, dataset biases, and their correlation.
Revised: To better characterize the relation among attributes, models and datasets, we propose three analytical approaches: attribute-wise, model-wise, and bucket-wise, which can be used to interpret the model biases, dataset biases, and their correlation.

Original: Formally, we refer to M = m 1 , · · · , m | M | as a set of models and P = p 1 , · · · , p | P | as a set of attributes .
Revised: Formally, we refer to M = m 1 , · · · , m | M | as a set of models and P = p 1 , · · · , p | P | as a set of attributes .

Original: As describe above, the test set E could be split into different buckets E = E
Revised: As describe above, the test set E could be split into different buckets E = E

Original: j 1 , · · · , E j | E | based on a attribute p j .
Revised: j 1 , · · · , E j | E | based on a attribute p j .

Original: We introduce the notion of performance tableT ∈ R | M |×| P |×| E | , in which
Revised: We introduce the notion of performance table

Original: T  ijk represents the performance of i -th model on the k -th sub-test set (bucket) generated by j -th attribute.
Revised: T ∈ R | M |×| P |×| E | , in which T ijk represents the performance of i -th model on the k -th sub-test set (bucket) generated by j -th attribute.

Original: Next, we will show how these approaches are deﬁned based on T .
Revised: Next, we will show how these approaches are deﬁned based on T .

Original: 4.3. M ODEL - WISE
Revised: 3.3. M ODEL - WISE

Original: The model-wise analysis aims to investigate how different attributes inﬂuence the performance of models with different architectures and pre-trained knowledge.
Revised: The model-wise analysis aims to investigate how different attributes inﬂuence the performance of models with different architectures and pre-trained knowledge.

Original: For example, “does the lengths of entities inﬂuence the performances of CNN-LSTM-CRF -based NER system?”
Revised: For example, “does the lengths of entities inﬂuence the performances of CNN-LSTM-CRF -based NER system?”

Original: Here we adopt two types of statistical variables S ρi,j and S σi,j to characterize  the relationship between the i -th model and j -th attribute.
Revised: Here we adopt two types of statistical variables S ρi,j and S σi,j to characterize how the j -th attribute inﬂuences the i -th model.

Original: [Equation]
Revised: [Equation]

Original: where  R j is the rank values of buckets(Mukaka, 2012) based on j -th attribute and Std( · ) is the function to calculate standard deviation.
Revised: where Spearman is a function to calculate Spearman’s rank correlation coefﬁcient (Mukaka, 2012) and R j is the rank values of buckets based on j -th attribute. Std( · ) is the function to compute standard deviation.

Revised: Intuitively, S ρi,j reﬂects the degree to which the i -th model positively (or negatively) correlates with j -th attribute while S σi,j indicates the degree to which this attribute inﬂuences the model.

Original: 4.3. A TTRIBUTE - WISE
Revised: 3.3. A TTRIBUTE - WISE

Original: The attribute-wise analysis aims to quantify the degree to which each attribute inﬂuences the NER task.
Revised: The attribute-wise analysis aims to quantify the degree to which each attribute inﬂuences the NER task.

Original: To achieve this, we introduce four measures: task-independent variable ζ j and task-dependent variables ρ 1 j ρ 2 j and σ j based on Eq.3 and Eq.4:
Revised: To achieve this, we introduce four measures: task-independent variable ζ j and task-dependent variables ρ 1 j ρ 2 j and σ j based on Eq.3 and Eq.4:

Original: [Equation] Atr( e i , j ), where
Revised: 1) ζ j = 1 | N | (cid:80) | N | i Atr( e i , j )

Original:  N is the number test entities and Atr( e i,
Revised: , where N is the number test entities and Atr( e i

Original:  j ) represents the value of attribute j forentity e i
Revised: , j ) represents the value of attribute j for

Revised: entity e i .

Revised: For example, when j represents the attribute of sentence length, ζ j is the average sentence length of the whole dataset.

Original: [Equation]
Revised: [Equation]

Original: S σi j , where | M | is the number of evaluated models.
Revised: S σi,j , where | M | is the number of evaluated models.

Original: Compared with ρ 1 j , ρ 2 j can reﬂect whether the correlation is positive or negative.
Revised: Compared with ρ 1 j , ρ 2 j can reﬂect whether the correlation is positive or negative.

Original: [Equation]
Revised: Intuitively, a higher absolute value of ρ 1 j or ρ 2 j suggests that attribute j is a crucial factor, greatly inﬂuencing the performance of NER systems.

Original:  B UCKET - WISE
Revised: 3.3. B UCKET - WISE

Original: The bucket-wise analysis diagnoses the strengths and weaknesses of existing models.
Revised: The bucket-wise analysis diagnoses the strengths and weaknesses of existing models.

Original: Moreover, based on attribute-wise analysis, we could understand how different choices of datasets inﬂuence the models’ performance.
Revised: Moreover, based on attribute-wise analysis, we could understand how different choices of datasets inﬂuence the models’ performance.

Original: To this end, we introduce the following measures:
Revised: To this end, we introduce the following measures:

Original: [Equation]
Revised: [Equation]

Original: Usually where a , b represent two different models and usually model a has a higher performance (by dataset-level metric).
Revised: Usually where a , b represent two different models and usually model a has a higher performance (by dataset-level metric).

Original: Intuitively, a negative value of β j suggests that a worse-ranked model ( b ) outperform the best-ranked model ( a ) in some aspect (attribute j ); By contrast, a positive value shows the largest margin on the attribute j .
Revised: Intuitively, a negative value of β j suggests that a worse-ranked model ( b ) outperform the best-ranked model ( a ) in some aspect (attribute j ); By contrast, a positive value shows the largest margin on the attribute j .

Original:  E XPERIMENTAL S ETTINGS 5. M ODELS AND A TTRIBUTES
Revised:  E XPERIMENTAL S ETTINGS 4. M ODELS AND A TTRIBUTES

Original: Model Settings To evaluate the importance of different components of the NER systems, we varied our models mainly in terms of three aspects: different choices of character- ( ELMo (Peters et al., 2018),
Revised: Model Settings To evaluate the importance of different components of the NER systems, we varied our models mainly in terms of three aspects: different choices of character- ( ELMo (Peters et al., 2018),

Original: Flair (Akbik et al., 2018; Akbik et al.)), subword- ( BERT (Peters et al., 2018; Devlin et al., 2018)), word- ( GloVe (Pennington et al., 2014)), and sentence-level encoders ( LSTM (Hochreiter &
Revised: Flair (Akbik et al., 2018; Akbik et al.)), subword- ( BERT (Peters et al., 2018; Devlin et al., 2018)), word- ( GloVe (Pennington et al., 2014)), and sentence-level encoders ( LSTM (Hochreiter &

Original: Schmidhuber, 1997), CNN (Kalchbrenner et al., 2014)) and decoders ( MLP or CRF (Lample et al., 2016; Collobert et al., 2011)).
Revised: Schmidhuber, 1997), CNN (Kalchbrenner et al., 2014)) and decoders ( MLP or CRF (Lample et al., 2016; Collobert et al., 2011)).

Original: Detailed setting are shown in Tab.3.
Revised: Detailed setting are shown in Tab.3.

Original: Totally, we study 12 NER models based on deep neural networks and one traditional method utilizing CRF Lafferty et al.
Revised: Totally, we study 12 NER models based on deep neural networks and one traditional method utilizing CRF Lafferty et al.

Original: The hyper-parameter settings of our evaluated models are shown in appendix section.
Revised: The hyper-parameter settings of our evaluated models are shown in appendix section.

Revised: [Table]

Original: Attributes In our evaluation methodology, entity attributes can be deﬁned ﬂexibly and attribute values can be continuous or discrete.
Revised: Attributes In our evaluation methodology, entity attributes can be deﬁned ﬂexibly and attribute values can be continuous or discrete.

Original: In this paper, although we investigate 10 types of attributes (or their combinations) as listed in Tab.2, others can be easily introduced.
Revised: In this paper, although we investigate 10 types of attributes (or their combinations) as listed in Tab.2, others can be easily introduced.

Original: 5. NER D ATASETS FOR E VALUATION
Revised: 4. NER D ATASETS FOR E VALUATION

Original: We conduct experiments on three benchmark datasets: the CoNLL2003 NER dataset, the WNUT dataset, and Ontonotes 5.0 dataset.
Revised: We conduct experiments on three benchmark datasets: the CoNLL2003 NER dataset, the WNUT dataset, and Ontonotes 5.0 dataset.

Original: The CoNLL2003 NER dataset (Sang & De Meulder, 2003) is based on Reuters data (Collobert et al., 2011).
Revised: The CoNLL2003 NER dataset (Sang & De Meulder, 2003) is based on Reuters data (Collobert et al., 2011).

Original: WNUT16 dataset is provided by the second shared task [Table] [Table] WNUT-2016.
Revised: WNUT16 dataset is provided by the second shared task at WNUT-2016.

Original: The Ontonotes 5.0 dataset (Weischedel et al., 2013) is collected from newsgroups, broadcast news (BN), broadcast conversation (BC) and weblogs (WB) and magazine genre (MZ).
Revised: The Ontonotes 5.0 dataset (Weischedel et al., 2013) is collected from newsgroups, broadcast news (BN), broadcast conversation (BC) and weblogs (WB) and magazine genre (MZ).

Original:  A NALYSIS 6. H OLISTIC A NALYSIS
Revised:  A NALYSIS 5. H OLISTIC A NALYSIS

Original: Before giving a ﬁne-grained analysis, we present the results of different models on different datasets in the way that traditional multi-dataset evaluation does.
Revised: Before giving a ﬁne-grained analysis, we present the results of different models on different datasets in the way that traditional multi-dataset evaluation does.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, we observe that there is no one-size-ﬁts-all model, and the models with the best results on different datasets are different.
Revised: 3, we observe that there is no one-size-ﬁts-all model, and the models with the best results on different datasets are different.

Original: Naturally, the following questions are raised: 1) What factors of the datasets can distinguish themselves and inﬂuence the NER task?
Revised: Naturally, the following questions are raised: 1) What factors of the datasets can distinguish themselves and inﬂuence the NER task?

Original: How do these factors inﬂuence the choices of models? 3)
Revised: How do these factors inﬂuence the choices of models? 3)

Original: Does a worse-ranked model outperform the best-ranked model in some aspect and how the datasets inﬂuence the choices of models?
Revised: Does a worse-ranked model outperform the best-ranked model in some aspect and how the datasets inﬂuence the choices of models?

Original: The following analyses will be conducted around these questions.
Revised: The following analyses will be conducted around these questions.

Original: 6. A TTRIBUTE - WISE A NALYSIS
Revised: 5. A TTRIBUTE - WISE A NALYSIS

Original: [Figure]
Revised: [Figure]

Revised: -dependent measure ρ 2 .

Original: Attribute-wise measures enable us to characterize the dataset biases quantitatively.
Revised: Attribute-wise measures enable us to characterize the dataset biases quantitatively.

Original: Here we utilize a radar chart in Fig.
Revised: Here we utilize a radar chart in Fig.

Original: 4 to strikingly display the commonality and speciality between different datasets based on three measures η , ρ 1 , σ deﬁned in Sec.
Revised: 4 to strikingly display the commonality and speciality between different datasets based on three measures ζ , ρ 1 , σ deﬁned in Sec.

Original: And we also illustrate measure ρ 2 in Fig.
Revised: And we also illustrate measure ρ 2 in Fig.

Original: Detailed observations are listed as follows:
Revised: Detailed observations are listed as follows:

Original: Category ambiguity and entity length have more consistent inﬂuence on NER performance.
Revised: Category ambiguity and entity length have more consistent inﬂuence on NER performance.

Original: The common parts of the radar chart Fig.
Revised: The common parts of the radar chart Fig.

Original: 4 (b-c) illustrate that no matter which datasets are, the performance of
Revised: 4 (b-c) illustrate that no matter which datasets are, the performance of

Original: NER task is highly correlated with these attributes:
Revised: NER task is highly correlated with these attributes:

Original: MF-tt (token-tag with MF-buck), MF-et (entity-tag with MF-buck), R-eLen (entity length with
Revised: MF-tt (token-tag with MF-buck), MF-et (entity-tag with MF-buck), R-eLen (entity length with

Original: R-bucket).
Revised: R-bucket).

Original: Fig.
Revised: Fig.

Original: 3 shows the same result.
Revised: 3 shows the same result.

Original: This suggests that the prediction difﬁculty of named entity is commonly inﬂuenced by category ambiguity ( MF-tt , MF-et ), entity length ( R-eLen ).
Revised: This suggests that the prediction difﬁculty of named entity is commonly inﬂuenced by category ambiguity ( MF-tt , MF-et ), entity length ( R-eLen ).

Original: Occurence and sentence length matters but are minor factors.
Revised: Occurence and sentence length matters but are minor factors.

Original: The outliers in radar chart show the peculiarities of different datasets.
Revised: The outliers in radar chart show the peculiarities of different datasets.

Original: Intutively, on attributes: R-sLen , F-token , R-oov , the extent to which different datasets are affected varies greatly.
Revised: Intutively, on attributes: R-sLen , F-token , R-oov , the extent to which different datasets are affected varies greatly.

Original: Typically, as observed from Fig.
Revised: Typically, as observed from Fig.

Original: 4-(a), a sorted sequence could be obtained according to the attribute F-tok : BN > MZ > WNUT > CoNLL > BC >
Revised: 4-(a), a sorted sequence could be obtained according to the attribute F-tok : BN > MZ > WNUT > CoNLL > BC >

Original: WB .
Revised: WB .

Original: The reason why the Spearman correlations ρ 1 of BC and WB are smaller is that performance on test entities with higher-frequency tokens are lower than entities with lower-frequency tokens.
Revised: The reason why the Spearman correlations ρ 1 of BC and WB are smaller is that performance on test entities with higher-frequency tokens are lower than entities with lower-frequency tokens.

Original: This suggests that F-tok is not a decisive attribute and higher-frequency token can not guarantee a better performance since other crucial factors such as category ambiguity also matter.
Revised: This suggests that F-tok is not a decisive attribute and higher-frequency token can not guarantee a better performance since other crucial factors such as category ambiguity also matter.

Revised: Entity density is a swing factor and CoNLL dataset is an outlier.

Original: R - e D e n
Revised: As shown in Fig. 3, the measure ρ 2 enables us to know the correlation is positive or negative. We observe that most datasets except CoNLL 2003 have negative Spearman values ρ 2 on the attribute of R-eDen , which suggests that a sentence with more entities is relatively harder to process.

Revised: We can explain the unusual behavior on

Revised: CoNLL 2003 from its intrinsic value ζ of eDen , the largest one as shown in Fig. 4(a).

Revised: The dataset of CoNLL contains a lot of short sentences, such as “ Chicago 8,674 484,018 ” and “ SOFIA 1996-12-06 ”.

Revised: That’s why it distinguishes itself from other datasets.

Revised: R - e D e n

Original: R-oov
Revised: R-oov

Original: R-sLen
Revised: R-sLen

Original: MF-et
Revised: MF-et

Original: M F - tt
Revised: M F - tt

Original: F-ent
Revised: F-ent

Original: F-tok
Revised: F-tok

Original: R-eLe
Revised: R-eLen

Original: MWE (a) η
Revised: MWE (a) ζ

Original: R - e D e n
Revised: R - e D e n

Original: R-oov
Revised: R-oov

Original: R-sLen
Revised: R-sLen

Original: MF-et
Revised: MF-et

Original: M F - tt
Revised: M F - tt

Original: F-ent
Revised: F-ent

Original: F-tok
Revised: F-tok

Original: R-eLe
Revised: R-eLen

Original: MWE (b) ρ
Revised: MWE (b) ρ

Original: R - e D e n
Revised: R - e D e n

Original: R-oov
Revised: R-oov

Original: R-sLen
Revised: R-sLen

Original: MF-et
Revised: MF-et

Original: M F - tt
Revised: M F - tt

Original: F-ent
Revised: F-ent

Original: F-tok
Revised: F-tok

Original: R-eLe
Revised: R-eLen

Original: CoNLL-2003WNUT-2016OntoNote-BN
Revised: CoNLL-2003WNUT-2016OntoNote-BN

Original: OntoNote-BCOntoNote-MZOntoNote-WB
Revised: OntoNote-BCOntoNote-MZOntoNote-WB

Original: MWE (c) σ
Revised: MWE (c) σ

Original: Entity density is a swing factor and CoNLL dataset is an outlier. As shown in Fig. 3, the measure ρ 2 enables us to know the correlation is positive or negative. We observe that most datasets except
Revised: The intrinsic differences in datasets can help us to understand how different datasets inﬂuence the different choices of models, which will be explained later (Sec. 5.4).

Original: CoNLL 2003 have negative Spearmanvalues ρ 2 on the attribute of R-eDen , which suggests that a sentence with more entities is relatively harder to process.
Revised: Spearman

Original: We can explain the unusual behavior on
Revised: Standard Deviation Model

Original: CoNLL 2003 from its intrinsic value η of eDen , the largest one as shown in Fig. 4(a).
Revised: F

Original: The dataset of CoNLL contains a lot of short sentences, such as “ Chicago 8,674 484,018 ” and “ SOFIA 1996-12-06 ”. That’s why it distinguishes itself from other datasets. The intrinsic differences in datasets can help us to understand how different datasets inﬂuence the different choices of models, which will be explained later (Sec. 6.4).
Revised: [Table]

Original: [Figure]
Revised: [Table]

Original: Model
Revised: D

Original: [Equation]
Revised: [Table] oov [Table] [Table]

Revised: M

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 6. M ODEL - WISE A NALYSIS
Revised: M [Table] [Table] - oov

Revised: [Table]

Revised: [Table]

Revised: e n

Revised: M

Revised: [Table]

Revised: [Table]

Revised: t

Revised: M

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: -t o k R - e L e n

Revised: F - t a g R - m o r

Original:  Based on two model-wise measures: S ρi,j and S σi,j deﬁned in Eq.3 and Eq.4, we investigate how different attributes inﬂuence the performance of the models with different architectures and pretrained embeddings.
Revised: [Table] been seen in training set. 5. M ODEL - WISE A NALYSIS Based on two model-wise measures: S ρi,j and S σi,j deﬁned in Eq.3 and Eq.4, we investigate how different attributes inﬂuence the performance of the models with different architectures and pretrained embeddings.

Original: Fig.
Revised: Fig.

Original: 4 illustrates the case on CoNLL (other datasets are included in appendix), and we have observed that:
Revised: 4 illustrates the case on CoNLL (other datasets are included in appendix), and we have observed that:

Original: 1) Char-unaware models are more sensitive to the degree of category ambiguity and occurrence of entities.
Revised: 1) Char-unaware models are more sensitive to the degree of category ambiguity and occurrence of entities.

Original: We observe that “ CnonWrandlstmCrf ” is negatively related to MF-et , MF-tt , F-ent and F-tok with high values of ρ and σ , suggesting the importance of the character-level encoder, which plays a major role in generalization to rare entities and entities with multiple categories.
Revised: We observe that “ CnonWrandlstmCrf ” is negatively related to MF-et , MF-tt , F-ent and F-tok with high values of ρ and σ , suggesting the importance of the character-level encoder, which plays a major role in generalization to rare entities and entities with multiple categories.

Original: More importantly, this observation still holds on other datasets.
Revised: More importantly, this observation still holds on other datasets.

Original: (See appendix) 2) Sentence length is a swing factor, whose contribution depends on what types of pre-trained embeddings are used.
Revised: (See appendix) 2) Sentence length is a swing factor, whose contribution depends on what types of pre-trained embeddings are used.

Original: There is a strong negative correlation between models with contextindependent embeddings and the attribute R-sLen (sentence length).
Revised: There is a strong negative correlation between models with contextindependent embeddings and the attribute R-sLen (sentence length).

Original: The relationship is reversed when contextualized embeddings are used.
Revised: The relationship is reversed when contextualized embeddings are used.

Original: The reason we believe is that long sentences could provide sufﬁcient context information for contextualized models.
Revised: The reason we believe is that long sentences could provide sufﬁcient context information for contextualized models.

Original: It is noticeable, however that ﬂair-related models “ CﬂairWnonelstmCrf ” behave differently compared with other contextualized models, which we will explain later in Sec 6.4 (Flair performs worse than ELMo when dealing with long sentences due to its structural bias).
Revised: It is noticeable, however that ﬂair-related models “ CﬂairWnonelstmCrf ” behave differently compared with other contextualized models, which we will explain later in Sec 5.4 (Flair performs worse than ELMo when dealing with long sentences due to its structural bias).

Original: 3) Character encoders favor the sentences with high entity-density.
Revised: 3) Character encoders favor the sentences with high entity-density.

Original: Only using character-level
Revised: Only using character-level

Original: CNN is apt to overﬁt the feature of capital letters.
Revised: CNN is easy to overﬁt the feature of capital letters.

Original: As a result, more non-entities are mis-predicted as entities.
Revised: As a result, more non-entities are mis-predicted as entities.

Original: Based on the understanding of the previous analysis (Sec. 6.2) of the entity density’s inﬂuence on CoNLL, we could better explain why “ CcnnWnonelstmCrf ” achieves the highest value of ρ and σ in “ R-eDen (entity density) attribute in Tab.4.
Revised: Based on the understanding of the previous analysis (Sec. 5.2) of the entity density’s inﬂuence on CoNLL, we could better explain why “ CcnnWnonelstmCrf ” achieves the highest value of ρ and σ in “ R-eDen (entity density) attribute in Tab.4.

Original: 6. B UCKET - WISE A NALYSIS
Revised: 5. B UCKET - WISE A NALYSIS

Original: In this section, we choose several typical models (others are shown in the appendix) as analyzed cases, aiming to seek answers to the following questions: 1) What are the strengths and weakness of different architectural designs?
Revised: In this section, we choose several typical models (others are shown in the appendix) as analyzed cases, aiming to seek answers to the following questions: 1) What are the strengths and weakness of different architectural designs?

Original: In other words, does a worse-ranked model outperform the best-ranked model in some aspect?
Revised: In other words, does a worse-ranked model outperform the best-ranked model in some aspect?

Original: How do the different choices of datasets inﬂuence model performance? Tab.illustrates the bucket-wise measure β , which is computed based on any pair of models M1 and M2.
Revised: How do the different choices of datasets inﬂuence model performance? Tab.illustrates the bucket-wise measure β , which is computed based on any pair of models M1 and M2.

Original: Next, we list some of our observations.
Revised: Next, we list some of our observations.

Original: Others are illustrated in the appendix.
Revised: Others are illustrated in the appendix.

Original: CNN v.s. LSTM
Revised: CNN v.s. LSTM

Original: The sentence encoder of CNN is better at dealing with short sentences, which holds in all datasets we evaluated in this paper.
Revised: The sentence encoder of CNN is better at dealing with short sentences, which holds in all datasets we evaluated in this paper.

Original: Strikingly, CNN outperforms LSTM by a large margin (dataset-level F1) on MZ dataset while is signiﬁcantly worse than LSTM on WB (refer to the appendix.).
Revised: Strikingly, CNN outperforms LSTM by a large margin (dataset-level F1) on MZ dataset while is signiﬁcantly worse than LSTM on WB (refer to the appendix.).

Original: We attempt to explain these discrepancies based on above attribute-wise metric ζ in
Revised: We attempt to explain these discrepancies based on above attribute-wise metric ζ in

Original: Fig.
Revised: Fig density

Original: 4(a): sentence length and entity density are two major factors for the choices of CNN and LSTM. CNNis better than LSTM when the dataset with higher value of ζ sLen and ζ eDen .
Revised: CNN

Original: By contrast, when a dataset with lower ζ eDen , LSTM is a priority (The ζ eDen of WB is the lowest).
Revised: By contrast, when a dataset with lower ζ eDen , LSTM is a priority (The ζ eDen of WB is the lowest).

Original: CRF v.s. MLP The beneﬁts of using CRF on short sentences are very stable, and improvement can be seen in all datasets.
Revised: CRF v.s. MLP The beneﬁts of using CRF on short sentences are very stable, and improvement can be seen in all datasets.

Original: Similarly, based on attribute-wise metric ζ in Fig. 4, we ﬁnd category ambiguity ( MF-et , MF-tt ) is a major factor for the choices of CRF and MLP: if a dataset with higher ζ MF − et , in which longer entities can beneﬁt more from CRF-based models.
Revised: Similarly, based on attribute-wise metric ζ in Fig. 4, we ﬁnd category ambiguity ( MF-et , MF-tt ) is a major factor for the choices of CRF and MLP: if a dataset with higher ζ MF − et , in which longer entities can beneﬁt more from CRF-based models.

Original: In comparison, introducing CRF will lead to more errors on long entities once the dataset (i.e. BN , MZ ) has a lower ζ MF − et .
Revised: In comparison, introducing CRF will lead to more errors on long entities once the dataset (i.e. BN , MZ ) has a lower ζ MF − et .

Original: CcnnWrand v.s.
Revised: CcnnWrand v.s.

Original: CcnnWnone
Revised: CcnnWnone

Original: The question has been little studied whether we need an extra wordlevel method (i.e., word-level look-up table) to get word representations when we have already used
Revised: The question has been little studied whether we need an extra wordlevel method (i.e., word-level look-up table) to get word representations when we have already used

Original: CNN to obtain word representations.
Revised: CNN to obtain word representations.

Original: Here, we show that CcnnWrand is not always better than
Revised: Here, we show that CcnnWrand is not always better than

Original: CcnnWnone and entity length matters for the choice.
Revised: CcnnWnone and entity length matters for the choice.

Original: Speciﬁcally, CcnnWnone could achieve better performance on on the WNUT and MZ datasets.
Revised: Speciﬁcally, CcnnWnone could achieve better performance on on the WNUT and MZ datasets.

Original: With the help of Fig. 4-(a), we ﬁnd the two datasets share a property of much higher value of ζ R − ele (entity length).
Revised: With the help of Fig. 4-(a), we ﬁnd the two datasets share a property of much higher value of ζ R − ele (entity length).

Original: Additionally, another commonality between the two datasets can be observed from Tab. 5: the gain of CcnnWnone mainly comes from the entities with the longer length.
Revised: Additionally, another commonality between the two datasets can be observed from Tab. 5: the gain of CcnnWnone mainly comes from the entities with the longer length.

Original: ELMo v.s GloVe ELMo consistently outperforms GloVe on all datasets using the holistic F1 score, but it is worse than GloVe when modeling short sentences.
Revised: ELMo v.s GloVe ELMo consistently outperforms GloVe on all datasets using the holistic F1 score, but it is worse than GloVe when modeling short sentences.

Original: Another interesting ﬁnding is that for those sentences containing more OOV tokens, GloVe could achieve better performance on all datasets.
Revised: Another interesting ﬁnding is that for those sentences containing more OOV tokens, GloVe could achieve better performance on all datasets.

Original: These phenomena indicate the complementarity between ELMo and GloVe, and our further combination of these two embeddings ( CelmWglovelstmCrf in Tab. 3) indeed works better.
Revised: These phenomena indicate the complementarity between ELMo and GloVe, and our further combination of these two embeddings ( CelmWglovelstmCrf in Tab. 3) indeed works better.

Original: Flair v.s. ELMo While the current state-of-the-art NER model Flair has achieved the best performance in terms of dataset-level F1 score, a worse-ranked model (ELMo) could outperform it in some aspects.
Revised: Flair v.s. ELMo While the current state-of-the-art NER model Flair has achieved the best performance in terms of dataset-level F1 score, a worse-ranked model (ELMo) could outperform it in some aspects.

Original: Typically, Flair performs worse when dealing with long sentences.
Revised: Typically, Flair performs worse when dealing with long sentences.

Original: The reason can be attributed to the its structure design, which adopts a LSTM-based encoder for character language modeling, suffering from long-term dependency problem (especially for character-level language 
Revised: The reason can be attributed to the its structure design, which adopts a LSTM-based encoder for character language modeling, suffering from long-term dependency problem (especially for character-level language model).

Original: [Table]
Revised: A promising improvement is to use the Transformer-based architecture for character language model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: To bridge the gap between the insufﬁcient understanding of the nature of datasets and model designs, this paper proposes a generalized evaluation methodology to interpret model biases, dataset biases,  and their correlation, drawing on the complementary strengths of the ﬁne-grained evaluation and multi-dataset evaluation.
Revised: To bridge the gap between the insufﬁcient understanding of the nature of datasets and model designs, this paper proposes a generalized evaluation methodology to interpret model biases, dataset biases, [Table] and their correlation, drawing on the complementary strengths of the ﬁne-grained evaluation and multi-dataset evaluation.

Original: We choose the NER task as a test case, the observations based on the extensive experiments suggest directions for improvement and can drive the progress of this area.
Revised: We choose the NER task as a test case, the observations based on the extensive experiments suggest directions for improvement and can drive the progress of this area.

Original: B M ODEL - WISE ANALYSIS ON THE OTHER DATASETS
Revised: B M ODEL - WISE ANALYSIS ON THE OTHER DATASETS

Original: [Table]
Revised: [Table]

Original: e n
Revised: e n

Original: Roov
Revised: Roov

Original: R - s L e n
Revised: R - s L e n

Original: M F - e t
Revised: M F - e t

Original: F - e n t M F - tt F -t
Revised: F - e n t M F - tt F -

Original: ok
Revised: [Table]

Original: R - e L e n
Revised: R - e L e n

Original: R - e D e n
Revised: R - e D e n

Original: Roov
Revised: Roov

Original: R - s L e n 
Revised: R - s L e n M F - e t F - e n t M F - tt F - t ok R - e L e n

Original: M F -e t F - e n t M F - tt F - t
Revised: F - t a g R - m o r [Table]

Original: ok R - e L e n F - t a g R - m o r [Table] [Table]
Revised: [Table]

Original: R - e D e n
Revised: R - e D e n

Original: Roov
Revised: Roov

Original: R - s L e n
Revised: R - s L e n

Original: M F - e t
Revised: M F - e t

Original: F - e n t M F - tt F - t
Revised: F - e n t M F - tt F - t

Original: ok
Revised: ok

Original: R - e L e n
Revised: R - e L e n

Original: R - e D e n
Revised: R - e D e n

Original: Roov
Revised: Roov

Original: R - s L e n
Revised: R - s L e n

Original: M F - e t
Revised: M F - e t

Original: F - e n t M F - tt F - t
Revised: F - e n t M F - tt F - t

Original: ok R - e L e n
Revised: ok R - e L e n

Original: F - t a g R - m o r
Revised: F - t a g R - m o r

Original: [Table]
Revised: [Table]

Original: C B UCKET - WISE ANALYSISON O NTO N OTE -BN AND O NTO N OTE -WB
Revised: B UCKET - WISE ANALYSIS

Original: [Table]
Revised: [Table]

Original: oov R - s L e n MF
Revised: M

Original: [Table]
Revised: [Table]

Original: e
Revised: n

Original: [Table]
Revised: [Table]

Original: F

Original: [Table]

Original: e n t M F

Original: [Table]

Original: tt F

Original: [Table]

Original: R
Revised: n

Original: [Table]
Revised: [Table]

Original: e L e n

Original: R - e

Original: [Table]

Original: e n Roov R
Revised: oov

Original: [Table]
Revised: [Table]

Original: L e n

Original: M F -

Original: [Table]t F [Table]
Revised: [Table]

Original: e n t M F - tt

Original: [Table]
Revised: [Table]

Original: t

Original: [Table]

Original:  e n[Table] - t a g [Table]
Revised: R - e D e n

Original: [Table]

Original: [Table]

Original: [Table]

Original: D e n

Original: Roov
Revised: Roov

Original: R - s L e n
Revised: R - s L e n

Original: M F - e t
Revised: M F - e t

Original: F - e n t M F - tt F - t
Revised: F - e n t M F - tt F - t

Original: [Table]
Revised: ok

Original: R - e L e n
Revised: R - e L e n

Original: R - e D e n
Revised: R - e D e n

Original: Roov
Revised: Roov

Original: R - s L e n
Revised: R - s L e n

Original: M F - e t
Revised: M F - e t

Original: F - e n t M F - tt F - t
Revised: F - e n t M F - tt F - t

Original: ok R - e L e n
Revised: ok R - e L e n

Original: F - t a g R - m o r
Revised: F - t a g R - m o r

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: classify some attribute values into three categories: small(S), middle(M) and large(L).
Revised: classify some attribute values into three categories: small(S), middle(M) and large(L).