Interpretable Multi-dataset Evaluation for Named Entity Recognition (2020)

Abstract: With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 does not tell us why or how particular methods perform differently and how diverse datasets influence the model design choices. In this paper, we present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them, identifying the strengths and weaknesses of current systems. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area: https://github.com/neulab/InterpretEval

## introduction with improvements in model architectures @xcite @xcite and learning of pre-trained embeddings @xcite @xcite @xcite , named entity recognition (ner) systems are evolving rapidly but also quickly reaching a performance plateau @xcite . this proliferation of methods poses a great challenge for the current evaluation methodology, which usually is based on comparing systems on a single holistic score assessing accuracy (usually entity f 1-score). there are several issues with this practice. first, a single evaluation number does not allow us to distinguish on a fine-grained level the strengths and weaknesses 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " h / p n w r r 6 s c i f s s m g u 5 4 g g 3 u a c h @xmath0... < l a t e x i t s h a 1 _ b a s e 6 4 = " f w a 7 q 9 1 l 9 w e 2 s d 7 among diverse systems. second, it is hard to improve what we do not understand; if an engineer or researcher looking to make model improvements cannot tell where the model is failing, it is also hard to decide which methodological improvements to try next. to alleviate this problem, a few works @xcite have attempted to perform fine-grained error analysis of ner systems. while a step in the right direction, these analyses frequently rely upon labor-intensive manual examination and also customarily depend on pre-existing error typologies encoding assumptions about the errors a system is likely to make. orthogonally, some other works @xcite @xcite @xcite evaluate holistic metrics such as f1 across multiple datasets that differ in domain, language, or other characteristics @xcite @xcite . although this enables us to more comprehensively assess the models, the reliance on holistic metrics precludes a finer-grained view of how various aspects of the model performance vary across the different settings. in this paper, we argue that an ideal evaluation methodology should be (1) fully or partially automatic, (2) allow evaluation and comparison across multiple datasets, and (3) allow users to dig deeper into fine-grained strengths and weaknesses of each model. to this end, we devise a generalized, fine- r d f j y q 5 c + 4 9 z + b e a 0 e e m 3 b d f 8 8 p c 4 t l y y v 5 1 c l a + s b m v n f 7 p 5 o e a e y w t h n 6 y d y z r y r 5 b s d a 3 o u e 6 0 u x s 3 z b y 1 1 7 f c l i 3 t s w j 2 4 y t p g k y n 3 f g g x u 0 h u s t l s n z d m a w r 4 q l n r n l 0 s l c f y p v u w c 1 b p 5 z n z v i w u v k k 1 m w h z d j q y i 4 s c f d 4 y a n l a h c w k 9 f z c g i y k u j y l x m s f x x h k e u s b t s l m m m i x i x / q d 0 e 4 i y w p a c 8 9 e q h 0 6 x a m 3 j q w k a 9 k e l b c t f q e p m p 5 k z 8 h o 8 5 5 k t 3 g 3 c f 3 t z m s n l u o a 2 l 9 0 s 8 z / 6 k q t h e p u z a 0 u 1 r r j r l t n z c 6 p 7 i q 4 u f q j k k 4 o e x e c d y g e e 3 a k c t z n v w o s w b v o r s x j h z j t s g l v z l k p p s u t a c c z k a r z q a e s g b p m n j d l j w o 2 6 j z 2 s i 9 d m q e j b k 7 r r j u 8 b / c m f 7 w q z 8 q t c q 8 8 f k c q u u y z i 1 9 l e f o c b r g r a @xmath1. 3 r d f j y q 5 c + 4 9 z + b e a 0 e e m 3 b d f 8 8 p c 4 t l y y v 5 1 c l a + s b m v n f 7 p 5 o e a e y w t h n 6 y d y z r y r 5 b s d a 3 o u e 6 0 u x s 3 z b y 1 1 7 f c l i 3 t s w j 2 4 y t p g k y n 3 f g g x u 0 h u s t l s n z d m a w r k q l n r n l 0 s l c f y p v u w c 1 b p 5 z n z v i w u v k k 1 m w h z d j q y i 4 s c f d 4 y a n l a h c w k 9 f z c g i y k u j y l x m s f x x h k e u s b t s l m m m i x i x / q d 0 e 4 i y w p a c 8 9 e q h 0 6 x a m 3 j q w k a 9 k e l b c t f q e p m p 5 k z 8 h o 8 5 5 k t 3 g 3 c f 3 t z m s n l u o a 2 l 9 0 s 8 z / 6 k q t h e p u z a 0 u 1 r r j r l t n z c 6 p 7 i q 4 u f q j k k 4 o e x e c d y g e e 3 a k c t z n v w o s w b v o r s x j h z j t s g l v z l k p p s u t a c c z k a r z q a e s g b p m n j d l j w o 2 6 j z 2 s i 9 d m q e j b k 7 r r j u 8 b / c m f 7 w q z 8 q t c q 8 8 f k c q u u y z i 1 9 l e f o c b 3 g r a @xmath2 r d f j y q 5 c + 4 9 z + b e a 0 e e m 3 b d f 8 8 p c 4 t l y y v 5 1 c l a + s b m v n f 7 p 5 o e a e y w t h n 6 y d y z r y r 5 b s d a 3 o u e 6 0 u x s 3 z b y 1 1 7 f c l i 3 t s w j 2 4 y t p g k y n 3 f g g x u 0 h u s t l s n z d m a w r k q l n r n l 0 s l c f y p v u w c 1 b p 5 z n z v i w u v k k 1 m w h z d j q y i 4 s c f d 4 y a n l a h c w k 9 f z c g i y k u j y l x m s f x x h k e u s b t s l m m m i x i x / q d 0 e 4 i y w p a c 8 9 e q h 0 6 x a m 3 j q w k a 9 k e l b c t f q e p m p 5 k z 8 h o 8 5 5 k t 3 g 3 c f 3 t z m s n l u o a 2 l 9 0 s 8 z / 6 k q t h e p u z a 0 u 1 r r j r l t n z c 6 p 7 i q 4 u f q j k k 4 o e x e c d y g e e 3 a k c t z n v w o s w b v o r s x j h z j t s g l v z l k p p s u t a c c z k a r z q a e s g b p m n j d l j w o 2 6 j z 2 s i 9 d m q e j b k 7 r r j u 8 b / c m f 7 w q z 8 q t c q 8 8 f k c q u u y z i 1 9 l e f o c b 3 g r a @xmath3 r d f j y q 5 c + 4 9 z + b e a 0 e e m 3 b d f 8 8 p c 4 t l y y v 5 1 c l a + s b m v n f 7 p 5 o e a e y w t h n 6 y d y z r y r 5 b s d a 3 o u e 6 0 u x s 3 z b y 1 1 7 f c l i 3 t s w j 2 4 y t p g k y n 3 f g g x u 0 h u s t l s n z d m a w r k q l n r n l 0 s l c f y p v u w c 1 b p 5 z n z v i w u v k k 1 m w h z d j q y i 4 s c f d 4 y a n l a h c w k 9 f z c g i y k u j y l x m s f x x h k e u s b t s l m m m i x i x / q d 0 e 4 i y w p a c 8 9 e q h 0 6 x a m 3 j q w k a 9 k e l b c t f q e p m p 5 k z 8 h o 8 5 5 k t 3 g 3 c f 3 t z m s n l u o a 2 l 9 0 s 8 z / 6 k q t h e p u z a 0 u 1 r r j r l t n z c 6 p 7 i q 4 u f q j k k 4 o e x e c d y g e e 3 a k c t z n v w o s w b v o r s x j h z j t s g l v z l k p p s u t a c c z k a r z q a e s g b p m n j d l j w o 2 6 j z 2 s i 9 d m q e j b k 7 r r j u 8 b / c m f 7 w q z 8 q t c q 8 8 f k c q u u y z i 1 9 l e f o c b 3 g r a @xmath4. 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " q i 8 4 i 3 4 m c c f v s t g t i l z 1 n g c b k 5 @xmath5 methodologically, our evaluation framework allows for three analytical views as elucidated in fig. 2 . model-wise (sec. 5.1) analysis investigates how the performances of different models vary according to attribute value (e.g. "is a model with a crf layer better at dealing with long entities?"). attribute-wise (sec. 5.2) analysis compares how different attributes affect performance on different datasets (e.g. "does entity length correlate with model performance on all datasets or just some?"). bucket-wise (sec. 5.3) compares among all possible analysis dimensions, and can diagnose the strengths and weaknesses of existing models (e.g. "what entity attributes indicate that a bertbased model will likely fail?"), or help us understand how different choices of datasets influence model performance (e.g. "on which datasets is using a crf layer more appropriate?"). experimentally, we conduct a comprehensive analysis over twelve models, eight attributes, and six datasets. proposed quantifiable measures allow us to draw several qualitative conclusions as highlighted below: 1) label consistency (the degree of label agreement of an entity on the training set) and entity length have a consistent influence on ner task's performance (sec. 5.2.2); 2) crf-based systems are more likely to make a mistake compared with mlp-based systems when dealing with long entities (sec. 5.1.2); 3) higher-frequency tokens of a test entity cannot guarantee better performance since other crucial factors such as label consistency also matter (sec. 5.2.2); 4) even more advanced models (e.g., bert, flair) fail to predict entities with low label consistency (sec. 5.3.2). finally, motivated by observation 4), we present an effective solution to improve current ner systems. quantitative and qualitative experiments demonstrate that introducing larger context is an effective method, obtaining improvements of up to 10 points in f1 score on some datasets. standard evaluation strategy for ner the common evaluation metric for ner systems @xcite is to compute a corpuslevel metric using micro-averaged f 1 score: f 1 = 2×p ×r p +r : where p is the percentage of named entities output by learning system that are correct. r is the percentage of gold entities identified by the system. here a named entity is correct only if it is an exact match of an annotated entity. ## attribute-aided evaluation our proposed attribute-aided evaluation methodology involves two key elements: attribute definition and bucketing. we first define diverse attributes for each entity, by which test entities are partitioned into different buckets. we then calculate the performance for each bucket of test entities. ## experimental settings in this section we describe our experimental settings, each of which is followed by an experiment and analysis results. ## fine-grained analysis to better characterize the relationship among models, attributes, and datasets, we propose three analysis approaches: model-, attribute-, and bucket-wise. ## discussion this paper has provided a framework where we can covert our understanding of the ner task (i.e., which attributes matter for the current task?) into interpretable evaluation aspects, and define axes through which we can apply them to acquire insights and make model improvements. this is just a first step towards the goal of fully-automated interpretable evaluation, and applications to new attributes and tasks beyond ner are promising future directions.