lvwerra HF staff commited on
Commit
36e833c
1 Parent(s): f1834c2

Update Space (evaluate main: 7e21410f)

Browse files
Files changed (4) hide show
  1. README.md +106 -6
  2. app.py +6 -0
  3. requirements.txt +2 -0
  4. smape.py +158 -0
README.md CHANGED
@@ -1,12 +1,112 @@
1
  ---
2
- title: Smape
3
- emoji: 👁
4
- colorFrom: purple
5
- colorTo: blue
6
  sdk: gradio
7
- sdk_version: 3.9
8
  app_file: app.py
9
  pinned: false
 
 
 
 
 
10
  ---
11
 
12
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: sMAPE
3
+ emoji: 🤗
4
+ colorFrom: blue
5
+ colorTo: red
6
  sdk: gradio
7
+ sdk_version: 3.0.2
8
  app_file: app.py
9
  pinned: false
10
+ tags:
11
+ - evaluate
12
+ - metric
13
+ description: >-
14
+ Symmetric Mean Absolute Percentage Error (sMAPE) is the symmetric mean percentage error difference between the predicted and actual values defined by Chen and Yang (2004).
15
  ---
16
 
17
+ # Metric Card for sMAPE
18
+
19
+
20
+ ## Metric Description
21
+
22
+ Symmetric Mean Absolute Error (sMAPE) is the symmetric mean of the percentage error of difference between the predicted $x_i$ and actual $y_i$ numeric values:
23
+
24
+ ![image](https://user-images.githubusercontent.com/8100/200009801-ae8be6c8-facf-401b-8df0-3f80a458b9f4.png)
25
+
26
+
27
+ ## How to Use
28
+
29
+ At minimum, this metric requires predictions and references as inputs.
30
+
31
+ ```python
32
+ >>> smape_metric = evaluate.load("smape")
33
+ >>> predictions = [2.5, 0.0, 2, 8]
34
+ >>> references = [3, -0.5, 2, 7]
35
+ >>> results = smape_metric.compute(predictions=predictions, references=references)
36
+ ```
37
+
38
+ ### Inputs
39
+
40
+ Mandatory inputs:
41
+ - `predictions`: numeric array-like of shape (`n_samples,`) or (`n_samples`, `n_outputs`), representing the estimated target values.
42
+ - `references`: numeric array-like of shape (`n_samples,`) or (`n_samples`, `n_outputs`), representing the ground truth (correct) target values.
43
+
44
+ Optional arguments:
45
+ - `sample_weight`: numeric array-like of shape (`n_samples,`) representing sample weights. The default is `None`.
46
+ - `multioutput`: `raw_values`, `uniform_average` or numeric array-like of shape (`n_outputs,`), which defines the aggregation of multiple output values. The default value is `uniform_average`.
47
+ - `raw_values` returns a full set of errors in case of multioutput input.
48
+ - `uniform_average` means that the errors of all outputs are averaged with uniform weight.
49
+ - the array-like value defines weights used to average errors.
50
+
51
+ ### Output Values
52
+ This metric outputs a dictionary, containing the mean absolute error score, which is of type:
53
+ - `float`: if multioutput is `uniform_average` or an ndarray of weights, then the weighted average of all output errors is returned.
54
+ - numeric array-like of shape (`n_outputs,`): if multioutput is `raw_values`, then the score is returned for each output separately.
55
+
56
+ Each sMAPE `float` value ranges from `0.0` to `2.0`, with the best value being 0.0.
57
+
58
+ Output Example(s):
59
+ ```python
60
+ {'smape': 0.5}
61
+ ```
62
+
63
+ If `multioutput="raw_values"`:
64
+ ```python
65
+ {'smape': array([0.5, 1.5 ])}
66
+ ```
67
+
68
+ #### Values from Popular Papers
69
+
70
+
71
+ ### Examples
72
+
73
+ Example with the `uniform_average` config:
74
+ ```python
75
+ >>> smape_metric = evaluate.load("smape")
76
+ >>> predictions = [2.5, 0.0, 2, 8]
77
+ >>> references = [3, -0.5, 2, 7]
78
+ >>> results = smape_metric.compute(predictions=predictions, references=references)
79
+ >>> print(results)
80
+ {'smape': 0.5787...}
81
+ ```
82
+
83
+ Example with multi-dimensional lists, and the `raw_values` config:
84
+ ```python
85
+ >>> smape_metric = evaluate.load("smape", "multilist")
86
+ >>> predictions = [[0.5, 1], [-1, 1], [7, -6]]
87
+ >>> references = [[0.1, 2], [-1, 2], [8, -5]]
88
+ >>> results = smape_metric.compute(predictions=predictions, references=references)
89
+ >>> print(results)
90
+ {'smape': 0.8874...}
91
+ >>> results = smape_metric.compute(predictions=predictions, references=references, multioutput='raw_values')
92
+ >>> print(results)
93
+ {'smape': array([1.3749..., 0.4])}
94
+ ```
95
+
96
+ ## Limitations and Bias
97
+ This metric is called a measure of "percentage error" even though there is no multiplier of 100. The range is between (0, 2) with it being two when the target and prediction are both zero.
98
+
99
+ ## Citation(s)
100
+
101
+ ```bibtex
102
+ @article{article,
103
+ author = {Chen, Zhuo and Yang, Yuhong},
104
+ year = {2004},
105
+ month = {04},
106
+ pages = {},
107
+ title = {Assessing forecast accuracy measures}
108
+ }
109
+ ```
110
+
111
+ ## Further References
112
+ - [Symmetric Mean absolute percentage error - Wikipedia](https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error)
app.py ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ import evaluate
2
+ from evaluate.utils import launch_gradio_widget
3
+
4
+
5
+ module = evaluate.load("smape")
6
+ launch_gradio_widget(module)
requirements.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ git+https://github.com/huggingface/evaluate@7e21410f9bcff651452f188b702cc80ecd3530e6
2
+ sklearn
smape.py ADDED
@@ -0,0 +1,158 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2022 The HuggingFace Datasets Authors and the current dataset script contributor.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ """sMAPE - Symmetric Mean Absolute Percentage Error Metric"""
15
+
16
+ import datasets
17
+ import numpy as np
18
+ from sklearn.metrics._regression import _check_reg_targets
19
+ from sklearn.utils.validation import check_consistent_length
20
+
21
+ import evaluate
22
+
23
+
24
+ _CITATION = """\
25
+ @article{article,
26
+ author = {Chen, Zhuo and Yang, Yuhong},
27
+ year = {2004},
28
+ month = {04},
29
+ pages = {},
30
+ title = {Assessing forecast accuracy measures}
31
+ }
32
+ """
33
+
34
+ _DESCRIPTION = """\
35
+ Symmetric Mean Absolute Percentage Error (sMAPE) is the symmetric mean percentage error
36
+ difference between the predicted and actual values as defined by Chen and Yang (2004),
37
+ based on the metric by Armstrong (1985) and Makridakis (1993).
38
+ """
39
+
40
+
41
+ _KWARGS_DESCRIPTION = """
42
+ Args:
43
+ predictions: array-like of shape (n_samples,) or (n_samples, n_outputs)
44
+ Estimated target values.
45
+ references: array-like of shape (n_samples,) or (n_samples, n_outputs)
46
+ Ground truth (correct) target values.
47
+ sample_weight: array-like of shape (n_samples,), default=None
48
+ Sample weights.
49
+ multioutput: {"raw_values", "uniform_average"} or array-like of shape (n_outputs,), default="uniform_average"
50
+ Defines aggregating of multiple output values. Array-like value defines weights used to average errors.
51
+
52
+ "raw_values" : Returns a full set of errors in case of multioutput input.
53
+
54
+ "uniform_average" : Errors of all outputs are averaged with uniform weight.
55
+
56
+ Returns:
57
+ smape : symmetric mean absolute percentage error.
58
+ If multioutput is "raw_values", then symmetric mean absolute percentage error is returned for each output separately. If multioutput is "uniform_average" or an ndarray of weights, then the weighted average of all output errors is returned.
59
+ sMAPE output is non-negative floating point in the range (0, 2). The best value is 0.0.
60
+ Examples:
61
+
62
+ >>> smape_metric = evaluate.load("smape")
63
+ >>> predictions = [2.5, 0.0, 2, 8]
64
+ >>> references = [3, -0.5, 2, 7]
65
+ >>> results = smape_metric.compute(predictions=predictions, references=references)
66
+ >>> print(results)
67
+ {'smape': 0.5787878787878785}
68
+
69
+ If you're using multi-dimensional lists, then set the config as follows :
70
+
71
+ >>> smape_metric = evaluate.load("smape", "multilist")
72
+ >>> predictions = [[0.5, 1], [-1, 1], [7, -6]]
73
+ >>> references = [[0.1, 2], [-1, 2], [8, -5]]
74
+ >>> results = smape_metric.compute(predictions=predictions, references=references)
75
+ >>> print(results)
76
+ {'smape': 0.49696969558995985}
77
+ >>> results = smape_metric.compute(predictions=predictions, references=references, multioutput='raw_values')
78
+ >>> print(results)
79
+ {'smape': array([0.48888889, 0.50505051])}
80
+ """
81
+
82
+
83
+ def symmetric_mean_absolute_percentage_error(y_true, y_pred, *, sample_weight=None, multioutput="uniform_average"):
84
+ """Symmetric Mean absolute percentage error (sMAPE) metric using sklearn's api and helpers.
85
+
86
+ Parameters
87
+ ----------
88
+ y_true : array-like of shape (n_samples,) or (n_samples, n_outputs)
89
+ Ground truth (correct) target values.
90
+ y_pred : array-like of shape (n_samples,) or (n_samples, n_outputs)
91
+ Estimated target values.
92
+ sample_weight : array-like of shape (n_samples,), default=None
93
+ Sample weights.
94
+ multioutput : {'raw_values', 'uniform_average'} or array-like
95
+ Defines aggregating of multiple output values.
96
+ Array-like value defines weights used to average errors.
97
+ If input is list then the shape must be (n_outputs,).
98
+ 'raw_values' :
99
+ Returns a full set of errors in case of multioutput input.
100
+ 'uniform_average' :
101
+ Errors of all outputs are averaged with uniform weight.
102
+ Returns
103
+ -------
104
+ loss : float or ndarray of floats
105
+ If multioutput is 'raw_values', then mean absolute percentage error
106
+ is returned for each output separately.
107
+ If multioutput is 'uniform_average' or an ndarray of weights, then the
108
+ weighted average of all output errors is returned.
109
+ sMAPE output is non-negative floating point. The best value is 0.0.
110
+ """
111
+ y_type, y_true, y_pred, multioutput = _check_reg_targets(y_true, y_pred, multioutput)
112
+ check_consistent_length(y_true, y_pred, sample_weight)
113
+ epsilon = np.finfo(np.float64).eps
114
+ smape = 2 * np.abs(y_pred - y_true) / (np.maximum(np.abs(y_true), epsilon) + np.maximum(np.abs(y_pred), epsilon))
115
+ output_errors = np.average(smape, weights=sample_weight, axis=0)
116
+ if isinstance(multioutput, str):
117
+ if multioutput == "raw_values":
118
+ return output_errors
119
+ elif multioutput == "uniform_average":
120
+ # pass None as weights to np.average: uniform mean
121
+ multioutput = None
122
+
123
+ return np.average(output_errors, weights=multioutput)
124
+
125
+
126
+ @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
127
+ class Smape(evaluate.Metric):
128
+ def _info(self):
129
+ return evaluate.MetricInfo(
130
+ description=_DESCRIPTION,
131
+ citation=_CITATION,
132
+ inputs_description=_KWARGS_DESCRIPTION,
133
+ features=datasets.Features(self._get_feature_types()),
134
+ reference_urls=["https://robjhyndman.com/hyndsight/smape/"],
135
+ )
136
+
137
+ def _get_feature_types(self):
138
+ if self.config_name == "multilist":
139
+ return {
140
+ "predictions": datasets.Sequence(datasets.Value("float")),
141
+ "references": datasets.Sequence(datasets.Value("float")),
142
+ }
143
+ else:
144
+ return {
145
+ "predictions": datasets.Value("float"),
146
+ "references": datasets.Value("float"),
147
+ }
148
+
149
+ def _compute(self, predictions, references, sample_weight=None, multioutput="uniform_average"):
150
+
151
+ smape_score = symmetric_mean_absolute_percentage_error(
152
+ references,
153
+ predictions,
154
+ sample_weight=sample_weight,
155
+ multioutput=multioutput,
156
+ )
157
+
158
+ return {"smape": smape_score}