Updating module

.gitattributes

@@ -1,27 +0,0 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zstandard filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,50 +1,104 @@
 ---
 title: Geometric Mean
-datasets:
--  
-tags:
-- evaluate
-- metric
-description: "TODO: add a description here"
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
 sdk: gradio
 sdk_version: 3.0.2
 app_file: app.py
 pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+ The geometric mean (G-mean) is the root of the product of class-wise sensitivity.  
 ---
 
 # Metric Card for Geometric Mean
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+
+The geometric mean (G-mean) is the root of the product of class-wise sensitivity. 
+This measure tries to maximize the accuracy on each of the classes while keeping these accuracies balanced. 
+For binary classification G-mean is the squared root of the product of the sensitivity and specificity. 
 
 ## How to Use
-*Give general statement of how to use the metric*
 
-*Provide simplest possible example for using the metric*
+At minimum, this metric requires predictions and references as input
+
+```python
+>>> gmean_metric = evaluate.load("geometric_mean")
+>>> results = gmean_metric.compute(predictions=[0, 1], references=[0, 1])
+>>> print(results)
+["{'geometric-mean': 1.0}"]
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+- **predictions** (`list` of `int`): Predicted labels.
+- **references** (`list` of `int`): Ground truth labels.
+- **labels** (`list` of `int`): The set of labels to include when average != 'binary', and their order if average is None. Labels present in the data can be excluded, for example to calculate a multiclass average ignoring a majority negative class, while labels not present in the data will result in 0 components in a macro average. Defaults to None.
+- **pos_label** (`string` or `int`): The class to report if average='binary' and the data is binary. If the data are multiclass, this will be ignored; setting labels=[pos_label] and average != 'binary' will report scores for that label only. Defaults to 1.
+- **average** (`string`): If None, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data. Defaults to `'multiclass'`.
+    - 'binary': Only report results for the class specified by pos_label. This is applicable only if targets (y_{true,pred}) are binary.
+    - 'micro': Calculate metrics globally by counting the total true positives, false negatives and false positives.
+    - 'macro': Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
+    - 'weighted': Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label).
+    - 'samples': Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).
+- **sample_weight** (`list` of `float`): Sample weights. Defaults to None.
+- **correction** (`float`): Substitutes sensitivity of unrecognized classes from zero to a given value. Defaults to 0.0.
 
 ### Output Values
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+- **geometric_mean** (`float` or `array` of `float`): geometric mean score or list of geometric mean scores, depending on the value passed to `average`. Minimum possible value is 0. Maximum possible value is 1. Higher geometric mean scores are better.
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+Output Example:
+```python
+{'geometric_mean': 0.4714045207910317}
+```
 
-#### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+
+Example 1-A simple binary example
+```python
+>>> geometric_mean = evaluate.load("geometric_mean")
+>>> results = geometric_mean.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0])
+>>> print(round(res['geometric-mean'], 2))
+0.58
+```
+
+Example 2-The same simple binary example as in Example 1, but with `sample_weight` included.
+```python
+>>> geometric_mean = evaluate.load("geometric_mean")
+>>> results = geometric_mean.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0], sample_weight=[0.9, 0.5, 3.9, 1.2, 0.3])
+>>> print(round(results['geometric-mean'], 2))
+0.35
+```
+
+Example 3-A multiclass example, with `average` equal to `macro`.
+```python
+>>> predictions = [0, 2, 1, 0, 0, 1]
+>>> references = [0, 1, 2, 0, 1, 2]
+>>> results = geometric_mean.compute(predictions=predictions, references=references, average="macro")
+>>> print(round(results['geometric-mean'], 2))
+0.47
+```
 
 ## Limitations and Bias
 *Note any known limitations or biases that the metric has, with links and references if possible.*
 
-## Citation
-*Cite the source where this metric was introduced.*
+## Citation(s)
+```bibtex
+@article{imbalanced-learn,
+  title={Imbalanced-learn: A Python Toolbox to Tackle the Curse of
+Imbalanced Datasets in Machine Learning},
+  author={Lemaˆıtre, G. and Nogueira, F. and Aridas, C.},
+  journal={Journal of Machine Learning Research},
+  volume={18},
+  pages={1-5},
+  year={2017}
+}
+```
 
 ## Further References
 *Add any useful further references.*

geometric_mean.py

@@ -11,85 +11,97 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""TODO: Add a description here."""
+"""Geometric mean metric."""
 
-import evaluate
 import datasets
+from imblearn.metrics import geometric_mean_score
+import evaluate
 
-
-# TODO: Add BibTeX citation
-_CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
-}
-"""
-
-# TODO: Add description of the module here
-_DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
+_DESCRIPTION = """
+The geometric mean (G-mean) is the root of the product of class-wise sensitivity. This measure 
+tries to maximize the accuracy on each of the classes while keeping these accuracies balanced. For binary 
+classification G-mean is the squared root of the product of the sensitivity and specificity. For multi-class problems 
+it is a higher root of the product of sensitivity for each class. 
 """
 
-
-# TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
 Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+    predictions (`list` of `int`): Predicted labels.
+    references (`list` of `int`): Ground truth labels.
+    labels (`list` of `int`): The set of labels to include when average != 'binary', and their order if average is None. Labels present in the data can be excluded, for example to calculate a multiclass average ignoring a majority negative class, while labels not present in the data will result in 0 components in a macro average. Defaults to None.
+    pos_label ('string' or `int`): The class to report if average='binary' and the data is binary. If the data are multiclass, this will be ignored; setting labels=[pos_label] and average != 'binary' will report scores for that label only. Defaults to 1.
+    average (`string`): If None, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data. Defaults to `'multiclass'`.
+    
+    - 'binary': Only report results for the class specified by pos_label. This is applicable only if targets (y_{true,pred}) are binary.
+    - 'micro': Calculate metrics globally by counting the total true positives, false negatives and false positives.
+    - 'macro': Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
+    - 'weighted': Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label).
+    - 'samples': Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).
+    
+    sample_weight (`list` of `float`): Sample weights. Defaults to None.
+    correction (`float`): Substitutes sensitivity of unrecognized classes from zero to a given value. Defaults to 0.0.
+    
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
-Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
+    geometric_mean (`float` or `array` of `float`): geometric mean score or list of geometric mean scores, depending on the value passed to `average`. Minimum possible value is 0. Maximum possible value is 1. Higher geometric mean scores are better.
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
+Examples:
+    Example 1-A simple binary example
+        >>> geometric_mean = evaluate.load("geometric_mean")
+        >>> results = geometric_mean.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0])
+        >>> print(round(res['geometric-mean'], 2))
+        0.58
+    Example 2-The same simple binary example as in Example 1, but with `sample_weight` included.
+        >>> geometric_mean = evaluate.load("geometric_mean")
+        >>> results = geometric_mean.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0], sample_weight=[0.9, 0.5, 3.9, 1.2, 0.3])
+        >>> print(round(results['geometric-mean'], 2))
+        0.35
+    Example 3-A multiclass example, with `average` equal to `macro`.
+        >>> predictions = [0, 2, 1, 0, 0, 1]
+        >>> references = [0, 1, 2, 0, 1, 2]
+        >>> results = geometric_mean.compute(predictions=predictions, references=references, average="macro")
+        >>> print(round(results['geometric-mean'], 2))
+        0.47
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
+_CITATION = """
+@article{imbalanced-learn,
+  title={Imbalanced-learn: A Python Toolbox to Tackle the Curse of
+Imbalanced Datasets in Machine Learning},
+  author={Lemaˆıtre, G. and Nogueira, F. and Aridas, C.},
+  journal={Journal of Machine Learning Research},
+  volume={18},
+  pages={1-5},
+  year={2017}
+}
+"""
 
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class GeometricMean(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
-
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
             module_type="metric",
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
             # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("int32")),
+                    "references": datasets.Sequence(datasets.Value("int32")),
+                }
+                if self.config_name == "multilabel"
+                else {
+                    "predictions": datasets.Value("int32"),
+                    "references": datasets.Value("int32"),
+                }
+            ),
+            reference_urls=["http://glemaitre.github.io/imbalanced-learn/generated/imblearn.metrics.geometric_mean_score.html#:~:text=The%20geometric%20mean%20(G%2Dmean,of%20the%20sensitivity%20and%20specificity."],
         )
 
-    def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass
-
-    def _compute(self, predictions, references):
-        """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
-        return {
-            "accuracy": accuracy,
-        }
+    def _compute(self, predictions, references, labels=None, pos_label=1, average="multiclass", sample_weight=None, correction=0.0):
+        score = geometric_mean_score(
+            references, predictions, labels=labels, pos_label=pos_label, average=average, sample_weight=sample_weight, correction=correction
+        )
+        return {"geometric-mean": float(score) if score.size == 1 else score}

requirements.txt

@@ -1,2 +1,3 @@
 git+https://github.com/huggingface/evaluate@a45df1eb9996eec64ec3282ebe554061cb366388
-datasets~=2.0
+datasets~=2.0
+imblearn==0.0

tests.py

@@ -1,17 +1,19 @@
-test_cases = [
-    {
-        "predictions": [0, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0}
-    },
-    {
-        "predictions": [1, 1],
-        "references": [1, 1],
-        "result": {"metric_score": 1}
-    },
-    {
-        "predictions": [1, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0.5}
-    }
-]
+import unittest
+
+from metrics.geometric_mean.geometric_mean import GeometricMean
+
+geometric_mean = GeometricMean()
+
+
+class TestGeometricMean(unittest.TestCase):
+    def test_gmean(self):
+        refs = [0, 1, 2, 0, 1, 2]
+        preds = [0, 1, 2, 0, 1, 2]
+        geometric_mean_score = geometric_mean.compute(predictions=preds, references=refs)
+        print(geometric_mean_score)
+        self.assertTrue(geometric_mean_score == {'geometric-mean': 1.0})
+
+        refs = [0, 2, 1, 0, 0, 1]
+        preds = [0, 1, 2, 0, 1, 2]
+        geometric_mean_score = geometric_mean.compute(predictions=preds, references=refs)
+        self.assertTrue(geometric_mean_score == {'geometric-mean': 0.0})