Update Space (evaluate main: 828c6327)

README.md

@@ -1,12 +1,113 @@
 ---
-title: Xnli
-emoji: 😻
-colorFrom: gray
-colorTo: pink
+title: XNLI
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
 sdk: gradio
 sdk_version: 3.0.2
 app_file: app.py
 pinned: false
+tags:
+- evaluate
+- metric
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces#reference
+# Metric Card for XNLI
+
+## Metric description
+
+The XNLI metric allows to evaluate a model's score on the [XNLI dataset](https://huggingface.co/datasets/xnli), which is a subset of a few thousand examples from the [MNLI dataset](https://huggingface.co/datasets/glue/viewer/mnli) that have been translated into a 14 different languages, some of which are relatively low resource such as Swahili and Urdu.
+
+As with MNLI, the task is to predict textual entailment (does sentence A imply/contradict/neither sentence B) and is a classification task (given two sentences, predict one of three labels).
+
+## How to use 
+
+The XNLI metric is computed based on the `predictions` (a list of predicted labels) and the `references` (a list of ground truth labels).
+
+```python
+from evaluate import load
+xnli_metric = load("xnli")
+predictions = [0, 1]
+references = [0, 1]
+results = xnli_metric.compute(predictions=predictions, references=references)
+```
+
+## Output values
+
+The output of the XNLI metric is simply the `accuracy`, i.e. the proportion of correct predictions among the total number of cases processed, with a range between 0 and 1 (see [accuracy](https://huggingface.co/metrics/accuracy) for more information). 
+
+### Values from popular papers
+The [original XNLI paper](https://arxiv.org/pdf/1809.05053.pdf) reported accuracies ranging from 59.3 (for `ur`) to 73.7 (for `en`) for the BiLSTM-max model.
+
+For more recent model performance, see the [dataset leaderboard](https://paperswithcode.com/dataset/xnli).
+
+## Examples 
+
+Maximal values:
+
+```python
+>>> from evaluate import load
+>>> xnli_metric = load("xnli")
+>>> predictions = [0, 1]
+>>> references = [0, 1]
+>>> results = xnli_metric.compute(predictions=predictions, references=references)
+>>> print(results)
+{'accuracy': 1.0}
+```
+
+Minimal values:
+
+```python
+>>> from evaluate import load
+>>> xnli_metric = load("xnli")
+>>> predictions = [1, 0]
+>>> references = [0, 1]
+>>> results = xnli_metric.compute(predictions=predictions, references=references)
+>>> print(results)
+{'accuracy': 0.0}
+```
+
+Partial match:
+
+```python
+>>> from evaluate import load
+>>> xnli_metric = load("xnli")
+>>> predictions = [1, 0, 1]
+>>> references = [1, 0, 0]
+>>> results = xnli_metric.compute(predictions=predictions, references=references)
+>>> print(results)
+{'accuracy': 0.6666666666666666}
+```
+
+## Limitations and bias
+
+While accuracy alone does give a certain indication of performance, it can be supplemented by error analysis and a better understanding of the model's mistakes on each of the categories represented in the dataset, especially if they are unbalanced. 
+
+While the XNLI dataset is multilingual and represents a diversity of languages, in reality, cross-lingual sentence understanding goes beyond translation, given that there are many cultural differences that have an impact on human sentiment annotations. Since the XNLI dataset was obtained by translation based on English sentences, it does not capture these cultural differences. 
+
+
+
+## Citation
+
+```bibtex
+@InProceedings{conneau2018xnli,
+  author = "Conneau, Alexis
+                 and Rinott, Ruty
+                 and Lample, Guillaume
+                 and Williams, Adina
+                 and Bowman, Samuel R.
+                 and Schwenk, Holger
+                 and Stoyanov, Veselin",
+  title = "XNLI: Evaluating Cross-lingual Sentence Representations",
+  booktitle = "Proceedings of the 2018 Conference on Empirical Methods
+               in Natural Language Processing",
+  year = "2018",
+  publisher = "Association for Computational Linguistics",
+  location = "Brussels, Belgium",
+}
+```
+    
+## Further References 
+
+- [XNI Dataset GitHub](https://github.com/facebookresearch/XNLI)
+- [HuggingFace Tasks -- Text Classification](https://huggingface.co/tasks/text-classification)

app.py

@@ -0,0 +1,6 @@
+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("xnli")
+launch_gradio_widget(module)

requirements.txt

@@ -0,0 +1,3 @@
+# TODO: fix github to release
+git+https://github.com/huggingface/evaluate.git@b6e6ed7f3e6844b297bff1b43a1b4be0709b9671
+datasets~=2.0

xnli.py

@@ -0,0 +1,88 @@
+# Copyright 2020 The HuggingFace Evaluate Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" XNLI benchmark metric. """
+
+import datasets
+
+import evaluate
+
+
+_CITATION = """\
+@InProceedings{conneau2018xnli,
+  author = "Conneau, Alexis
+                 and Rinott, Ruty
+                 and Lample, Guillaume
+                 and Williams, Adina
+                 and Bowman, Samuel R.
+                 and Schwenk, Holger
+                 and Stoyanov, Veselin",
+  title = "XNLI: Evaluating Cross-lingual Sentence Representations",
+  booktitle = "Proceedings of the 2018 Conference on Empirical Methods
+               in Natural Language Processing",
+  year = "2018",
+  publisher = "Association for Computational Linguistics",
+  location = "Brussels, Belgium",
+}
+"""
+
+_DESCRIPTION = """\
+XNLI is a subset of a few thousand examples from MNLI which has been translated
+into a 14 different languages (some low-ish resource). As with MNLI, the goal is
+to predict textual entailment (does sentence A imply/contradict/neither sentence
+B) and is a classification task (given two sentences, predict one of three
+labels).
+"""
+
+_KWARGS_DESCRIPTION = """
+Computes XNLI score which is just simple accuracy.
+Args:
+    predictions: Predicted labels.
+    references: Ground truth labels.
+Returns:
+    'accuracy': accuracy
+Examples:
+
+    >>> predictions = [0, 1]
+    >>> references = [0, 1]
+    >>> xnli_metric = evaluate.load("xnli")
+    >>> results = xnli_metric.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'accuracy': 1.0}
+"""
+
+
+def simple_accuracy(preds, labels):
+    return (preds == labels).mean()
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Xnli(evaluate.EvaluationModule):
+    def _info(self):
+        return evaluate.EvaluationModuleInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Value("int64" if self.config_name != "sts-b" else "float32"),
+                    "references": datasets.Value("int64" if self.config_name != "sts-b" else "float32"),
+                }
+            ),
+            codebase_urls=[],
+            reference_urls=[],
+            format="numpy",
+        )
+
+    def _compute(self, predictions, references):
+        return {"accuracy": simple_accuracy(predictions, references)}