Calibrating Imbalanced Classifiers with Focal Loss: An Empirical Study (2022)

Abstract: Imbalanced data distribution is a practical and common challenge in building production-level machine learning (ML) models in industry, where data usually exhibits long-tail distributions. For instance, in virtual AI Assistants, such as Google Assistant, Amazon Alexa and Apple Siri, the “play music” or “set timer” utterance is exposed to an order of magnitude more traffic than other skills. This can easily cause trained models to overfit to the majority classes, categories or intents, lead to model miscalibration. The uncalibrated models output unreliable (mostly overconfident) predictions, which are at high risk of affecting downstream decision-making systems. In this work, we study the calibration of production models in the industry use-case of predicting product return reason codes in customer service conversations of an online retail store; The returns reasons also exhibit class imbalance. To alleviate the resulting miscalibration in the production ML model, we streamline the model development and deployment using focal loss (CITATION).We empirically show the effectiveness of model training with focal loss in learning better calibrated models, as compared to standard cross-entropy loss. Better calibration, in turn, enables better control of the precision-recall trade-off for the models deployed in production.

## introduction building and developing ml models in industry has many practical challenges. imbalanced data distributions @xcite , particularly long-tail distributions @xcite , make models overfit to majority data classes and lead to miscalibration @xcite , i.e. the model-predicted probability fails to estimate the likelihood of true correctness and provides over-or under-confident predictions. to address imbalanced data, some mainstream strategies are rebalancing the dataset through upsampling minority and/or downsampling majority classes. figure 1: our procedure to calibrate the ml model that trained with an imbalanced dataset d. with focal loss, model m is learned to be a better calibrated model m c . in a follow-up stage, a dataset d g is sampled from actual conversations (the inference distribution) and manually annotated as a golden dataset. it is used to further calibrate models and tune precision-recall threshold to achieve specific precision or recall to serve customer requests. see @xcite for an example. however, miscalibration caused by imbalanced data cannot be easily handled by these methods. @xcite recently found that negative log likelihood (nll) trained deep neural networks (dnns) tend to be poorly calibrated as compared to traditional ml methods @xcite . calibrating models that have been trained with imbalanced data plays an important role in industry. in applications such as medical diagnosis @xcite , decisions do not only depend on a predicted class, but also on the predicted probabilities, e.g. to quantify patients' risks @xcite in order to determine appropriate treatments. therefore, it is of particular importance that models are well calibrated in high-risk domains such as medicine, financial management @xcite , self-driving cars @xcite , etc., to assure that the predicted probabilities reflect the true values. in this work, we empirically study the effectiveness of focal loss @xcite in building reliable ml models for a practical text classification task -return reason code prediction in customer service chatbots. formally, focal loss is: ## datasets and implementation details we use historical logged data on return reasons for past human-customer service interactions (which is sometimes noisy), and use human annotated reliable data for model calibration before deployment. concretely, we prepared 1013 human annotated samples as a golden dataset for the binary reason code model, and a set of 1839 annotated samples for the multi-reason code model. the numbers reported in the tables in section 4 are based on the annotated dataset. figure 2 gives the statistics of randomly sampled datasets from historical logs. we considered 4 widely used return reasons in online retail stores (for details see link below foot_0 ), subsequently referred as "item is defective or does not work" (label 0), "missing parts or accessories" (label 1), "performance or quality not adequate" (label 2), and "customer preference" (label 3). in the binary case, we aim to detect one particular return type label 0, and discriminate it from any other return reason (referred as others (label 1)). similarly in the multi-class case, we consider the four broad return categories (labels 0 .. 3) and gloss all other return types under others (label 4). in both settings others is the most frequent class. as we can see the label distribution in both use-cases is imbalanced. for a full dataset d, we split it into {d train , d val , d test } with 8:1:1 into train/val/test splits. we train models with standard cross-entropy (ce) and focal loss with different γ values (fl γ ). we implemented our models with py-torch @xcite , each model consists of 2 bidirectional lstm layers and 2 dense layers. the embedding dimension is 1024. the hidden layer dimension is set to 128 and 512 for the binary and multi-reason code models respectively. we apply dropout with rates of 0.1 and 0.2 to the embedding and dense layers respectively. the models are trained with adam (kingma and ba, 2014) as the optimization algorithm ## model selection and calibration this section describes how we build a model which is well calibrated while retaining its original predictive performance, e.g., 85% precision or 90% recall. we adapt the focal loss @xcite and empirically evaluate its calibration effectiveness @xcite in both binary and multi-class text classification tasks in a practical setting. we observed the two types of trade-offs which are crucial for using models in practice: ## experimental results to serve customers' requests, we use the binary reason code model for three conversational usecases foot_1 . the model detects whether a product return