Repository: scikit-learn/scikit-learn
File: examples/model_selection/plot_cost_sensitive_learning.py (Python)

Code before:

# %%
# In addition, the original research [1]_ defines a custom business metric. We
# call a "business metric" any metric function that aims at quantifying how the
# predictions (correct or wrong) might impact the business value of deploying a
# given machine learning model in a specific application context. For our
# credit prediction task, the authors provide a custom cost-matrix which
# encodes that classifying a a "bad" credit as "good" is 5 times more costly on
# average than the opposite: it is less costly for the financing institution to
# not grant a credit to a potential customer that will not default (and
# therefore miss a good customer that would have otherwise both reimbursed the
# credit and paid interests) than to grant a credit to a customer that will
# default.
#
# We define a python function that weighs the confusion matrix and returns the
# overall cost.
# The rows of the confusion matrix hold the counts of observed classes
# while the columns hold counts of predicted classes. Recall that here we
# consider "bad" as the positive class (second row and column).
# Scikit-learn model selection tools expect that we follow a convention
# that "higher" means "better", hence the following gain matrix assigns
# negative gains (costs) to the two kinds of prediction errors:
# - a gain of -1 for each false positive ("good" credit labeled as "bad"),
# - a gain of -5 for each false negative ("bad" credit labeled as "good"),
# The true positives and true negatives are assigned null gains in this
# metric.
#
# Note that theoretically, given that our model is calibrated and our data
# set representative and large enough, we do not need to tune the
# threshold, but can safely set it to 1/5 of the cost ratio, as stated by
# Eq. (2) in Elkan's paper [2]_.
import numpy as np


def credit_gain_score(y, y_pred, neg_label, pos_label):
    cm = confusion_matrix(y, y_pred, labels=[neg_label, pos_label])

    gain_matrix = np.array(
        [
            [0, -1],  # -1 gain for false positives
            [-5, 0],  # -5 gain for false negatives
        ]
    )
    return np.sum(cm * gain_matrix)


scoring["credit_gain"] = make_scorer(
    credit_gain_score, neg_label=neg_label, pos_label=pos_label
)
# %%
# Vanilla predictive model

Review comment (refactor):
This part is not rendering properly. Also some wording can simplify the last phrase.
```suggestion
#
# - a gain of `-1` for each false positive ("good" credit labeled as "bad"),
# - a gain of `-5` for each false negative ("bad" credit labeled as "good"),
# - a `0` gain for true positives and true negatives.
#
```

Code after:

# %%
# In addition, the original research [1]_ defines a custom business metric. We
# call a "business metric" any metric function that aims at quantifying how the
# predictions (correct or wrong) might impact the business value of deploying a
# given machine learning model in a specific application context. For our
# credit prediction task, the authors provide a custom cost-matrix which
# encodes that classifying a a "bad" credit as "good" is 5 times more costly on
# average than the opposite: it is less costly for the financing institution to
# not grant a credit to a potential customer that will not default (and
# therefore miss a good customer that would have otherwise both reimbursed the
# credit and paid interests) than to grant a credit to a customer that will
# default.
#
# We define a python function that weighs the confusion matrix and returns the
# overall cost.
# The rows of the confusion matrix hold the counts of observed classes
# while the columns hold counts of predicted classes. Recall that here we
# consider "bad" as the positive class (second row and column).
# Scikit-learn model selection tools expect that we follow a convention
# that "higher" means "better", hence the following gain matrix assigns
# negative gains (costs) to the two kinds of prediction errors:
#
# - a gain of `-1` for each false positive ("good" credit labeled as "bad"),
# - a gain of `-5` for each false negative ("bad" credit labeled as "good"),
# - a `0` gain for true positives and true negatives.
#
# Note that theoretically, given that our model is calibrated and our data
# set representative and large enough, we do not need to tune the
# threshold, but can safely set it to 1/5 of the cost ratio, as stated by
# Eq. (2) in Elkan's paper [2]_.
import numpy as np


def credit_gain_score(y, y_pred, neg_label, pos_label):
    cm = confusion_matrix(y, y_pred, labels=[neg_label, pos_label])

    gain_matrix = np.array(
        [
            [0, -1],  # -1 gain for false positives
            [-5, 0],  # -5 gain for false negatives
        ]
    )
    return np.sum(cm * gain_matrix)


scoring["credit_gain"] = make_scorer(
    credit_gain_score, neg_label=neg_label, pos_label=pos_label
)
# %%
# Vanilla predictive model