Repository: scikit-learn/scikit-learn
File: sklearn/metrics/_classification.py (Python)

Code before:
           <10.1177/001316446002000104>`
    .. [2] `R. Artstein and M. Poesio (2008). "Inter-coder agreement for
           computational linguistics". Computational Linguistics 34(4):555-596
           <https://www.mitpressjournals.org/doi/pdf/10.1162/coli.07-034-R2>`_.
    .. [3] `Wikipedia entry for the Cohen's kappa
            <https://en.wikipedia.org/wiki/Cohen%27s_kappa>`_.

    Examples
    --------
    >>> from sklearn.metrics import cohen_kappa_score
    >>> y1 = ["negative", "positive", "negative", "neutral", "positive"]
    >>> y2 = ["negative", "positive", "negative", "neutral", "negative"]
    >>> cohen_kappa_score(y1, y2)
    0.6875
    """
    confusion = confusion_matrix(y1, y2, labels=labels, sample_weight=sample_weight)
    n_classes = confusion.shape[0]
    sum0 = np.sum(confusion, axis=0)
    sum1 = np.sum(confusion, axis=1)

    mgs_changing_default = (
        "The default return value of `cohen_kappa_score` in case of a division "
        "by zero has been deprecated in 1.7 and will be changed to 0.0 in version "
        "1.9. Set `replace_undefined_by=0.0` to use the new default and to silence "
        "this Warning."
    )

    numerator = np.outer(sum0, sum1)
    denominator = np.sum(sum0)
    if np.isclose(denominator, 0):
        if replace_undefined_by == "deprecated":
            replace_undefined_by = np.nan
            warnings.warn(mgs_changing_default, FutureWarning)
        msg = (
            "`y2` does not contain any label that is also both present in `y1` and in "
            "`labels`. cohen_kappa_score is undefined and set to the value defined in "
            "the `replace_undefined_by` param, which defaults to 0.0."
        )
        warnings.warn(msg, UndefinedMetricWarning, stacklevel=2)
        return replace_undefined_by
    expected = numerator / denominator

    if weights is None:
        w_mat = np.ones([n_classes, n_classes], dtype=int)
        w_mat.flat[:: n_classes + 1] = 0
    else:  # "linear" or "quadratic"
        w_mat = np.zeros([n_classes, n_classes], dtype=int)
        w_mat += np.arange(n_classes)
        if weights == "linear":
            w_mat = np.abs(w_mat - w_mat.T)
        else:

Review comment (suggestion):
I would pick 0.0 as a future default (instead of -1.0 which is the worst score), because it is the least expressive of the scores, representing matching labels by chance. 

If users would use `cohen_kappa_score` as part of their custom metric, that calculates the mean over several cohen_kappa_scores,  0.0 would be a neutral element like the "ignore" option that we have talked about in this comment: https://github.com/scikit-learn/scikit-learn/issues/29048#issuecomment-2460062084

Code after:
        "labels": ["array-like", None],
        "weights": [StrOptions({"linear", "quadratic"}), None],
        "sample_weight": ["array-like", None],
        "replace_undefined_by": [
            Interval(Real, -1.0, 1.0, closed="both"),
            np.nan,
        ],
    },
    prefer_skip_nested_validation=True,
)
def cohen_kappa_score(
    y1,
    y2,
    *,
    labels=None,
    weights=None,
    sample_weight=None,
    replace_undefined_by=np.nan,
):
    r"""Compute Cohen's kappa: a statistic that measures inter-annotator agreement.

    This function computes Cohen's kappa [1]_, a score that expresses the level
    of agreement between two annotators on a classification problem. It is
    defined as

    .. math::
        \kappa = (p_o - p_e) / (1 - p_e)

    where :math:`p_o` is the empirical probability of agreement on the label
    assigned to any sample (the observed agreement ratio), and :math:`p_e` is
    the expected agreement when both annotators assign labels randomly.
    :math:`p_e` is estimated using a per-annotator empirical prior over the
    class labels [2]_.

    Read more in the :ref:`User Guide <cohen_kappa>`.

    Parameters
    ----------
    y1 : array-like of shape (n_samples,)
        Labels assigned by the first annotator.

    y2 : array-like of shape (n_samples,)
        Labels assigned by the second annotator. The kappa statistic is
        symmetric, so swapping ``y1`` and ``y2`` doesn't change the value.

    labels : array-like of shape (n_classes,), default=None
        List of labels to index the matrix. This may be used to select a
        subset of labels. If `None`, all labels that appear at least once in
        ``y1`` or ``y2`` are used. Note that at least one label in `labels` must be
        present in `y1`, even though this function is otherwise agnostic to the order
        of `y1` and `y2`.