Dice Loss for Data-imbalanced NLP Tasks (2020)

Abstract: Many NLP tasks such as tagging and machine reading comprehension are faced with the severe data imbalance issue: negative examples significantly outnumber positive examples, and the huge number of easy-negative examples overwhelms the training. The most commonly used cross entropy (CE) criteria is actually an accuracy-oriented objective, and thus creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sørensen–Dice coefficient or Tversky index , which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples. Theoretical analysis shows that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boost on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task; SOTA results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification.

## introduction data imbalance is a common issue in a variety of nlp tasks such as tagging and machine reading comprehension. data imbalance results in the following two issues: to handle the first issue, we propose to replace ce or mle with losses based on the sørensen-dice coefficient @xcite or @xcite . the sørensen-dice coefficient, dice loss for short, is the harmonic mean of precision and recall. it attaches equal importance to false positives (fps) and false negatives (fns) and is thus more immune to data-imbalanced datasets. tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the f β score, and thus comes with more flexibility. therefore, we use dice loss or tversky index to replace ce loss to address the first issue. only using dice loss or tversky index is not enough since they are unable to address the dominating influence of easy-negative examples. this is intrinsically because dice loss is actually a soft version of the f1 score. taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0.5, but training will push the value to 0 as much as possible. this gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easynegative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. meanwhile, the model can hardly distinguish between hard-negative examples and positive ones. inspired by the idea of focal loss @xcite in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to (1 -p), and this weight dynamically changes as training proceeds. this strategy helps deemphasize confident examples during training as their probability p approaches 1, making the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative exam-ples. combing both strategies, we observe significant performance boosts on a wide range of data imbalanced nlp tasks. the rest of this paper is organized as follows: related work is presented in section 2. we describe different proposed losses in section 3. experimental results are presented in section 4. we perform ablation studies in section 5, followed by a brief conclusion in section 6. ## experiments we evaluated the proposed method on four nlp tasks, part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification. hyperparameters are tuned on the corresponding development set of each dataset. more experiment details including datasets and hyperparameters are shown in supplementary material. ## conclusion in this paper, we propose the dice-based loss to narrow down the gap between training objective and evaluation metrics (f1 score). experimental results show that the proposed loss function help