Repository: scikit-learn/scikit-learn
File: sklearn/metrics/_classification.py (Python)

Code before:

    Parameters
    ----------
    y1 : array-like of shape (n_samples,)
        Labels assigned by the first annotator.

    y2 : array-like of shape (n_samples,)
        Labels assigned by the second annotator. The kappa statistic is
        symmetric, so swapping ``y1`` and ``y2`` doesn't change the value.

    labels : array-like of shape (n_classes,), default=None
        List of labels to index the matrix. This may be used to select a
        subset of labels. If `None`, all labels that appear at least once in
        ``y1`` or ``y2`` are used. Note that at least one label in `labels` must be
        present in `y1`, even though this function is otherwise agnostic to the order
        of `y1` and `y2`.

    weights : {'linear', 'quadratic'}, default=None
        Weighting type to calculate the score. `None` means not weighted;
        "linear" means linear weighting; "quadratic" means quadratic weighting.

    sample_weight : array-like of shape (n_samples,), default=None
        Sample weights.

    replace_undefined_by : np.nan, float in [-1.0, 1.0], default=np.nan
        Sets the return value when a division by zero would occur. This can happen for
        instance on empty input arrays, or when no label of interest (as defined in the
        `labels` param) is assigned by the second annotator, or when both `y1` and `y2`
        only have one label in common that is also in `labels`. In these cases, an
        :class:`~sklearn.exceptions.UndefinedMetricWarning` is raised. Can take the
        following values:

        - `np.nan` to return `np.nan`
        - a floating point value in the range of [-1.0, 1.0] to return a specific value

        .. versionadded:: 1.7

    Returns
    -------
    kappa : float
        The kappa statistic, which is a number between -1.0 and 1.0. The maximum value
        means complete agreement; the minimum value means complete disagreement; 0.0
        indicates no agreement beyond what would be expected by chance.

    References
    ----------
    .. [1] :doi:`J. Cohen (1960). "A coefficient of agreement for nominal scales".
           Educational and Psychological Measurement 20(1):37-46.
           <10.1177/001316446002000104>`
    .. [2] `R. Artstein and M. Poesio (2008). "Inter-coder agreement for
           computational linguistics". Computational Linguistics 34(4):555-596

Review comment (suggestion):
I'd prefer a more generic phrasing such that the param has a common meaning for all metrics
```suggestion
        Sets the return value when the metric is undefined. This can happen for
```

Code after:

@validate_params(
    {
        "y1": ["array-like"],
        "y2": ["array-like"],
        "labels": ["array-like", None],
        "weights": [StrOptions({"linear", "quadratic"}), None],
        "sample_weight": ["array-like", None],
        "replace_undefined_by": [
            Interval(Real, -1.0, 1.0, closed="both"),
            np.nan,
        ],
    },
    prefer_skip_nested_validation=True,
)
def cohen_kappa_score(
    y1,
    y2,
    *,
    labels=None,
    weights=None,
    sample_weight=None,
    replace_undefined_by=np.nan,
):
    r"""Compute Cohen's kappa: a statistic that measures inter-annotator agreement.

    This function computes Cohen's kappa [1]_, a score that expresses the level
    of agreement between two annotators on a classification problem. It is
    defined as

    .. math::
        \kappa = (p_o - p_e) / (1 - p_e)

    where :math:`p_o` is the empirical probability of agreement on the label
    assigned to any sample (the observed agreement ratio), and :math:`p_e` is
    the expected agreement when both annotators assign labels randomly.
    :math:`p_e` is estimated using a per-annotator empirical prior over the
    class labels [2]_.

    Read more in the :ref:`User Guide <cohen_kappa>`.

    Parameters
    ----------
    y1 : array-like of shape (n_samples,)
        Labels assigned by the first annotator.

    y2 : array-like of shape (n_samples,)
        Labels assigned by the second annotator. The kappa statistic is
        symmetric, so swapping ``y1`` and ``y2`` doesn't change the value.

    labels : array-like of shape (n_classes,), default=None