Repository: scikit-learn/scikit-learn
File: sklearn/metrics/_classification.py (Python)

Code before:
           computational linguistics". Computational Linguistics 34(4):555-596
           <https://www.mitpressjournals.org/doi/pdf/10.1162/coli.07-034-R2>`_.
    .. [3] `Wikipedia entry for the Cohen's kappa
            <https://en.wikipedia.org/wiki/Cohen%27s_kappa>`_.

    Examples
    --------
    >>> from sklearn.metrics import cohen_kappa_score
    >>> y1 = ["negative", "positive", "negative", "neutral", "positive"]
    >>> y2 = ["negative", "positive", "negative", "neutral", "negative"]
    >>> cohen_kappa_score(y1, y2)
    0.6875
    """
    confusion = confusion_matrix(y1, y2, labels=labels, sample_weight=sample_weight)
    n_classes = confusion.shape[0]
    sum0 = np.sum(confusion, axis=0)
    sum1 = np.sum(confusion, axis=1)

    numerator = np.outer(sum0, sum1)
    denominator = np.sum(sum0)
    if np.isclose(denominator, 0):
        msg = (
            "`y2` does not contain any label that is also both present in `y1` and in "
            "`labels`. cohen_kappa_score is undefined and set to the value defined in "
            "the `replace_undefined_by` param, which defaults to 0.0."
        )
        warnings.warn(msg, UndefinedMetricWarning, stacklevel=2)
        return replace_undefined_by
    expected = numerator / denominator

    if weights is None:
        w_mat = np.ones([n_classes, n_classes], dtype=int)
        w_mat.flat[:: n_classes + 1] = 0
    else:  # "linear" or "quadratic"
        w_mat = np.zeros([n_classes, n_classes], dtype=int)
        w_mat += np.arange(n_classes)
        if weights == "linear":
            w_mat = np.abs(w_mat - w_mat.T)
        else:
            w_mat = (w_mat - w_mat.T) ** 2

    numerator = np.sum(w_mat * confusion)
    denominator = np.sum(w_mat * expected)
    if np.isclose(denominator, 0):
        msg = (
            "`y1` and `y2` only have one label in common that is also in `labels`. "
            "cohen_kappa_score is undefined and set to the value defined in the "
            "`replace_undefined_by` param, which defaults to 0.0."
        )
        warnings.warn(msg, UndefinedMetricWarning, stacklevel=2)
        return replace_undefined_by

Review comment (suggestion):
In my thinking, this warning message also covers cases where both y1 and y2 are empty (as in "test case: empty inputs"). At least it would motivate people to inspect their data and then they might find out if their data is empty.

However, we could also branch here and make a separate warning message for empty inputs.

Code after:
    else:
        tn = y_true.shape[0] - tp - fp - fn

    return xp.reshape(xp.stack([tn, fp, fn, tp]).T, (-1, 2, 2))


@validate_params(
    {
        "y1": ["array-like"],
        "y2": ["array-like"],
        "labels": ["array-like", None],
        "weights": [StrOptions({"linear", "quadratic"}), None],
        "sample_weight": ["array-like", None],
        "replace_undefined_by": [
            Interval(Real, -1.0, 1.0, closed="both"),
            np.nan,
        ],
    },
    prefer_skip_nested_validation=True,
)
def cohen_kappa_score(
    y1,
    y2,
    *,
    labels=None,
    weights=None,
    sample_weight=None,
    replace_undefined_by=np.nan,
):
    r"""Compute Cohen's kappa: a statistic that measures inter-annotator agreement.

    This function computes Cohen's kappa [1]_, a score that expresses the level
    of agreement between two annotators on a classification problem. It is
    defined as

    .. math::
        \kappa = (p_o - p_e) / (1 - p_e)

    where :math:`p_o` is the empirical probability of agreement on the label
    assigned to any sample (the observed agreement ratio), and :math:`p_e` is
    the expected agreement when both annotators assign labels randomly.
    :math:`p_e` is estimated using a per-annotator empirical prior over the
    class labels [2]_.

    Read more in the :ref:`User Guide <cohen_kappa>`.

    Parameters
    ----------
    y1 : array-like of shape (n_samples,)
        Labels assigned by the first annotator.