WRF : Weighted Rouge-F1 Metric for Entity Recognition (2023)

Abstract: The continuous progress in Named Entity Recognition allows the identification of complex entities in multiple domains. The traditionally used metrics like precision, recall, and F1-score can only reflect the classification quality of the underlying NER model to a limited extent. Existing metrics do not distinguish between a non-recognition of an entity and a misclassification of an entity. Additionally, the dealing with redundant entities remains unaddressed. We propose WRF, a Weighted Rouge F1 metric for Entity Recognition, to solve the mentioned gaps in currently available metrics. We successfully employ the WRF metric for automotive entity recognition, followed by a comprehensive qualitative and quantitative analysis of the obtained results.

## introduction the continuous progress in named entity recognition (ner) allows the identification of complex entities in multiple domains @xcite . the traditionally used metrics like precision, recall, and f1-score @xcite ) can only reflect the classification quality of the underlying ner model to a limited extent @xcite . the limitation of the entity recognition evaluation metrics has been studied by many researchers, which motivated them to modify the existing or create new metrics @xcite @xcite to tackle many corner cases @xcite . this research work shows that still many corner cases are not being addressed by the existing metrics to date, to evaluate the true prediction performance of the model. in the ner task, the model needs to identify the entity and classify it. after tokenizing the input text, all the tokens that do not represent an entity of our interest are usually * work done during an internship at mercedes-benz ag. labeled as other (o). existing metrics do not distinguish between a non-recognition of an entity and a misclassification of an entity. non-recognition is the wrong classification of an entity as other, whereas a misclassification is the wrong classification of an entity as any of the other classes, apart from other. furthermore, the dealing of redundant entities which are present in the predicted or target labels are not tackled by the above-mentioned metrics and therefore should take into account too. in this work, we show that the existing metrics do not fit well for automotive entity recognition (aer). aer is the automotive domain-specific entity recognition task. we propose wrf, a weighted rouge f1 metric for entity recognition, to solve the gaps in currently available metrics. the scientific contribution is structured as follows: in section 2, we give insights into related work. the currently available metrics, the identified metric gap, and our proposed method wrf are explained in section 3. section 4 deals with the fine-tuning of a pretrained language model with an aer dataset and the quantitative and qualitative evaluation comparison between existing metric and wrf. we will end up this contribution with a conclusion in section 5. ## related work the evaluation of entity recognition models is a crucial task in the field of nlp. several forums have addressed meaningful entity recognition evaluation metrics in the past. the entity recognition challenge @xcite at the conference on computational natural language learning 2003 (conll2003) introduced the idea of measuring the performance of the systems in terms of precision, recall, f1, and its variations like f1micro, which considers the entity prediction to be correct, only when the sequence of predicted labels for the entire entity precisely matches the sequence prediction: repair costs (parts and labour) are often very high, since the workshop does not know which is the faulty fp location and then also replaces the ekmv fp or replaces because of the consequential damage to the scroll fp , (scroll tip tp is partially melted tp ) by too high temperatures tp . target: repair costs (parts and labour) are often very high, since the workshop does not know which is the faulty location and then also replaces the ekmv or replaces because of the consequential damage to the scroll, (scroll tip is partially melted) by too high temperatures. ## conclusion we present to the research community a new metric called wrf to fill the evaluation gap in the entity recognition evaluation. we used a weighted form of the rouge unigram f1, which differentiates between misclassification and non-recognition of entities. wrf is also able to handle redundant entities. the newly developed metric was applied successfully within aer. it is beneficial for the practical use case to make it more focused on correct classification or just the identification of entities. it is possible to optimize the weights of wrf according to its practical use case by the parameters γ 1,2,3 .