metadata

title: McNemar
emoji: 🤗
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 3.0.2
app_file: app.py
pinned: false
tags:
  - evaluate
  - comparison
description: >-
  McNemar's test is a diagnostic test over a contingency table resulting from
  the predictions of two classifiers. The test compares the sensitivity and
  specificity of the diagnostic tests on the same group reference labels. It can
  be computed with: McNemar = (SE - SP)**2 / SE + SP Where: SE: Sensitivity
  (Test 1 positive; Test 2 negative) SP: Specificity (Test 1 negative; Test 2
  positive)

Comparison Card for McNemar

Comparison description

McNemar's test is a non-parametric diagnostic test over a contingency table resulting from the predictions of two classifiers. The test compares the sensitivity and specificity of the diagnostic tests on the same group reference labels. It can be computed with:

McNemar = (SE - SP)**2 / SE + SP

Where:

SE: Sensitivity (Test 1 positive; Test 2 negative)
SP: Specificity (Test 1 negative; Test 2 positive)

In other words, SE and SP are the diagonal elements of the contingency table for the classifier predictions (predictions1 and predictions2) with respect to the ground truth references.

How to use

The McNemar comparison calculates the proportions of responses that exhibit disagreement between two classifiers. It is used to analyze paired nominal data.

Inputs

Its arguments are:

predictions1: a list of predictions from the first model.

predictions2: a list of predictions from the second model.

references: a list of the ground truth reference labels.

Output values

The McNemar comparison outputs two things:

stat: The McNemar statistic.

p: The p value.

Examples

Example comparison:

mcnemar = evaluate.load("mcnemar")
results = mcnemar.compute(references=[1, 0, 1], predictions1=[1, 1, 1], predictions2=[1, 0, 1])
print(results)
{'stat': 1.0, 'p': 0.31731050786291115}

Limitations and bias

The McNemar test is a non-parametric test, so it has relatively few assumptions (basically only that the observations are independent). It should be used to analyze paired nominal data only.

Citations

@article{mcnemar1947note,
  title={Note on the sampling error of the difference between correlated proportions or percentages},
  author={McNemar, Quinn},
  journal={Psychometrika},
  volume={12},
  number={2},
  pages={153--157},
  year={1947},
  publisher={Springer-Verlag}
}