Repository: scikit-learn/scikit-learn
File: sklearn/metrics/_classification.py (Python)

Code before:
        raise

    n_classes = confusion.shape[0]
    sum0 = np.sum(confusion, axis=0)
    sum1 = np.sum(confusion, axis=1)

    mgs_changing_default = (
        "The default return value of `cohen_kappa_score` in case of a division "
        "by zero has been deprecated in 1.7 and will be changed to 0.0 in version "
        "1.9. Set `replace_undefined_by=0.0` to use the new default and to silence "
        "this Warning."
    )

    numerator = np.outer(sum0, sum1)
    denominator = np.sum(sum0)
    if np.isclose(denominator, 0):
        if replace_undefined_by == "deprecated":
            replace_undefined_by = np.nan
            warnings.warn(mgs_changing_default, FutureWarning)
        msg = (
            "`y2` contains no labels that are presented in both `y1` and `labels`."
            "cohen_kappa_score is undefined and set to the value defined in "
            "the `replace_undefined_by` param, which defaults to `np.nan`."
        )
        warnings.warn(msg, UndefinedMetricWarning, stacklevel=2)
        return replace_undefined_by
    expected = numerator / denominator

    if weights is None:
        w_mat = np.ones([n_classes, n_classes], dtype=int)
        w_mat.flat[:: n_classes + 1] = 0
    else:  # "linear" or "quadratic"
        w_mat = np.zeros([n_classes, n_classes], dtype=int)
        w_mat += np.arange(n_classes)
        if weights == "linear":
            w_mat = np.abs(w_mat - w_mat.T)
        else:
            w_mat = (w_mat - w_mat.T) ** 2

    numerator = np.sum(w_mat * confusion)
    denominator = np.sum(w_mat * expected)
    if np.isclose(denominator, 0):
        if replace_undefined_by == "deprecated":
            replace_undefined_by = np.nan
            warnings.warn(mgs_changing_default, FutureWarning)
        msg = (
            "`y1`, `y2` and `labels` have only one label in common. "
            "cohen_kappa_score is undefined and set to the value defined in the "
            "`replace_undefined_by` param, which defaults to `np.nan`."
        )
        warnings.warn(msg, UndefinedMetricWarning, stacklevel=2)

Review comment (suggestion):
This part can be in a function defined here (inside `kohen_kappa_score`) to avoid duplicating the logic twice here.

Code after:

    where :math:`p_o` is the empirical probability of agreement on the label
    assigned to any sample (the observed agreement ratio), and :math:`p_e` is
    the expected agreement when both annotators assign labels randomly.
    :math:`p_e` is estimated using a per-annotator empirical prior over the
    class labels [2]_.

    Read more in the :ref:`User Guide <cohen_kappa>`.

    Parameters
    ----------
    y1 : array-like of shape (n_samples,)
        Labels assigned by the first annotator.

    y2 : array-like of shape (n_samples,)
        Labels assigned by the second annotator. The kappa statistic is
        symmetric, so swapping ``y1`` and ``y2`` doesn't change the value.

    labels : array-like of shape (n_classes,), default=None
        List of labels to index the matrix. This may be used to select a
        subset of labels. If `None`, all labels that appear at least once in
        ``y1`` or ``y2`` are used. Note that at least one label in `labels` must be
        present in `y1`, even though this function is otherwise agnostic to the order
        of `y1` and `y2`.

    weights : {'linear', 'quadratic'}, default=None
        Weighting type to calculate the score. `None` means not weighted;
        "linear" means linear weighting; "quadratic" means quadratic weighting.

    sample_weight : array-like of shape (n_samples,), default=None
        Sample weights.

    replace_undefined_by : np.nan, float in [-1.0, 1.0], default=np.nan
        Sets the return value when the metric is undefined. This can happen when no
        label of interest (as defined in the `labels` param) is assigned by the second
        annotator, or when both `y1` and `y2`only have one label in common that is also
        in `labels`. In these cases, an
        :class:`~sklearn.exceptions.UndefinedMetricWarning` is raised. Can take the
        following values:

        - `np.nan` to return `np.nan`
        - a floating point value in the range of [-1.0, 1.0] to return a specific value

        .. versionadded:: 1.9

    Returns
    -------
    kappa : float
        The kappa statistic, which is a number between -1.0 and 1.0. The maximum value
        means complete agreement; the minimum value means complete disagreement; 0.0
        indicates no agreement beyond what would be expected by chance.