Repository: scikit-learn/scikit-learn
File: sklearn/metrics/_classification.py (Python)

Code before:
    ----------
    .. [1] :doi:`J. Cohen (1960). "A coefficient of agreement for nominal scales".
           Educational and Psychological Measurement 20(1):37-46.
           <10.1177/001316446002000104>`
    .. [2] `R. Artstein and M. Poesio (2008). "Inter-coder agreement for
           computational linguistics". Computational Linguistics 34(4):555-596
           <https://www.mitpressjournals.org/doi/pdf/10.1162/coli.07-034-R2>`_.
    .. [3] `Wikipedia entry for the Cohen's kappa
            <https://en.wikipedia.org/wiki/Cohen%27s_kappa>`_.

    Examples
    --------
    >>> from sklearn.metrics import cohen_kappa_score
    >>> y1 = ["negative", "positive", "negative", "neutral", "positive"]
    >>> y2 = ["negative", "positive", "negative", "neutral", "negative"]
    >>> cohen_kappa_score(y1, y2)
    0.6875
    """

    def _check_zero_division(denominator, replace_undefined_by, msg):
        if np.isclose(denominator, 0):
            if replace_undefined_by == "deprecated":
                replace_undefined_by = np.nan
                warnings.warn(mgs_changing_default, FutureWarning)
            warnings.warn(msg, UndefinedMetricWarning, stacklevel=2)
            return True

    try:
        confusion = confusion_matrix(y1, y2, labels=labels, sample_weight=sample_weight)
    except ValueError as e:
        if "At least one label specified must be in y_true" in str(e):
            msg = (
                "At least one label in `labels` must be present in `y1` (even though "
                "`cohen_kappa_score` is otherwise agnostic to the order of `y1` and "
                "`y2`)."
            )
            raise ValueError(msg) from e
        raise

    n_classes = confusion.shape[0]
    sum0 = np.sum(confusion, axis=0)
    sum1 = np.sum(confusion, axis=1)

    mgs_changing_default = (
        "`np.nan` as the default return value of `cohen_kappa_score` in case of a "
        "division by zero has been deprecated in 1.7 and will be changed to 0.0 in "
        "version 1.9. Set `replace_undefined_by=0.0` to use the new default and to "
        "silence this Warning."
    )

    numerator = np.outer(sum0, sum1)

Review comment (bug):
I find this check quite brittle. Take for instance
```py
import numpy as np
from sklearn.metrics import cohen_kappa_score

y1 = [1,1,1,1,1,2,2,2,2,2]
y2 = [2,3,3,3,3,3,3,3,3,3]
sw = [1e-10,1,1,1,1,1,1,1,1,1]

cohen_kappa_score(y1, y2, labels=[1,2], sample_weight=sw)
/home/jeremie/R/scikit-learn/sklearn/metrics/_classification.py:946: FutureWarning: `np.nan` as the default return value of `cohen_kappa_score` in case of a division by zero has been deprecated in 1.7 and will be changed to 0.0 in version 1.9. Set `replace_undefined_by=0.0` to use the new default and to silence this Warning.
  warnings.warn(mgs_changing_default, FutureWarning)
/home/jeremie/R/scikit-learn/sklearn/metrics/_classification.py:980: UndefinedMetricWarning: `y2` contains no labels that are presented in both `y1` and `labels`.`cohen_kappa_score` is undefined and set to the value defined by the `replace_undefined_by` param, which defaults to `np.nan`.
  if _check_zero_division(denominator, replace_undefined_by, msg_zero_division):
```
The warning is wrong here. "2" is in y1 and labels. It's just that its weight is so low that the `isclose` misses it.

I guess the most robust solution would be to check for the actual condition of the undefined behavior, i.e. check that y2 only contains labels that are not in both y1 and labels. (and same for the other warning)

Code after:

    y2 : array-like of shape (n_samples,)
        Labels assigned by the second annotator. The kappa statistic is
        symmetric, so swapping ``y1`` and ``y2`` doesn't change the value.

    labels : array-like of shape (n_classes,), default=None
        List of labels to index the matrix. This may be used to select a
        subset of labels. If `None`, all labels that appear at least once in
        ``y1`` or ``y2`` are used. Note that at least one label in `labels` must be
        present in `y1`, even though this function is otherwise agnostic to the order
        of `y1` and `y2`.

    weights : {'linear', 'quadratic'}, default=None
        Weighting type to calculate the score. `None` means not weighted;
        "linear" means linear weighting; "quadratic" means quadratic weighting.

    sample_weight : array-like of shape (n_samples,), default=None
        Sample weights.

    replace_undefined_by : np.nan, float in [-1.0, 1.0], default=np.nan
        Sets the return value when the metric is undefined. This can happen when no
        label of interest (as defined in the `labels` param) is assigned by the second
        annotator, or when both `y1` and `y2`only have one label in common that is also
        in `labels`. In these cases, an
        :class:`~sklearn.exceptions.UndefinedMetricWarning` is raised. Can take the
        following values:

        - `np.nan` to return `np.nan`
        - a floating point value in the range of [-1.0, 1.0] to return a specific value

        .. versionadded:: 1.9

    Returns
    -------
    kappa : float
        The kappa statistic, which is a number between -1.0 and 1.0. The maximum value
        means complete agreement; the minimum value means complete disagreement; 0.0
        indicates no agreement beyond what would be expected by chance.

    References
    ----------
    .. [1] :doi:`J. Cohen (1960). "A coefficient of agreement for nominal scales".
           Educational and Psychological Measurement 20(1):37-46.
           <10.1177/001316446002000104>`
    .. [2] `R. Artstein and M. Poesio (2008). "Inter-coder agreement for
           computational linguistics". Computational Linguistics 34(4):555-596
           <https://www.mitpressjournals.org/doi/pdf/10.1162/coli.07-034-R2>`_.
    .. [3] `Wikipedia entry for the Cohen's kappa
            <https://en.wikipedia.org/wiki/Cohen%27s_kappa>`_.

    Examples