This paper presents the area under the Receiver Operating Characteristics (ROC) curve as an alternative metric for evaluating word sense disambiguation performance. The current metrics -accuracy, precision and recallwhile suitable for two-way classification, are shown to be inadequate when disambiguating between three or more senses. Specifically, these measures do not facilitate comparison with baseline performance nor are they sensitive to non-uniform misclassification costs. Both of these issues can be addressed using ROC analysis.
REF introduces the concept of (non-uniform) misclassification costs.
Performance Metrics for Word Sense Disambiguation
ALTA
Computer Science