Spaces:
Running
Running
Added initial setup for SEM-F1 metric
Browse files- README.md +51 -10
- requirements.txt +3 -1
- semf1.py +138 -43
README.md
CHANGED
@@ -1,28 +1,49 @@
|
|
1 |
---
|
2 |
title: SemF1
|
3 |
-
datasets:
|
4 |
-
-
|
5 |
tags:
|
6 |
- evaluate
|
7 |
- metric
|
8 |
-
description: "TODO: add a description here"
|
9 |
sdk: gradio
|
10 |
sdk_version: 3.19.1
|
11 |
app_file: app.py
|
12 |
pinned: false
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
13 |
---
|
14 |
|
15 |
# Metric Card for SemF1
|
16 |
|
17 |
-
***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
|
18 |
-
|
19 |
## Metric Description
|
20 |
-
|
|
|
|
|
21 |
|
22 |
## How to Use
|
23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
24 |
|
25 |
-
|
|
|
26 |
|
27 |
### Inputs
|
28 |
*List all input arguments in the format below*
|
@@ -44,7 +65,27 @@ pinned: false
|
|
44 |
*Note any known limitations or biases that the metric has, with links and references if possible.*
|
45 |
|
46 |
## Citation
|
47 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
48 |
|
49 |
## Further References
|
50 |
-
|
|
|
|
1 |
---
|
2 |
title: SemF1
|
|
|
|
|
3 |
tags:
|
4 |
- evaluate
|
5 |
- metric
|
|
|
6 |
sdk: gradio
|
7 |
sdk_version: 3.19.1
|
8 |
app_file: app.py
|
9 |
pinned: false
|
10 |
+
description: >-
|
11 |
+
SEM-F1 metric leverages the pre-trained contextual embeddings and evaluates the model generated semantic overlap
|
12 |
+
summary with the reference overlap summary. It evaluates the semantic overlap summary at the sentence level and
|
13 |
+
computes precision, recall and F1 scores.
|
14 |
+
|
15 |
+
Refer to the paper `SEM-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale`
|
16 |
+
for more details.
|
17 |
---
|
18 |
|
19 |
# Metric Card for SemF1
|
20 |
|
|
|
|
|
21 |
## Metric Description
|
22 |
+
SEM-F1 metric leverages the pre-trained contextual embeddings and evaluates the model generated semantic overlap
|
23 |
+
summary with the reference overlap summary. It evaluates the semantic overlap summary at the sentence level and
|
24 |
+
computes precision, recall and F1 scores.
|
25 |
|
26 |
## How to Use
|
27 |
+
SEM-F1 takes 2 mandatory arguments:
|
28 |
+
`predictions`: (a list of system generated documents in the form of sentences i.e. List[List[str]]),
|
29 |
+
`references`: (a list of ground-truth documents in the form of sentences i.e. List[List[str]])
|
30 |
+
|
31 |
+
```python
|
32 |
+
from evaluate import load
|
33 |
+
predictions = [
|
34 |
+
["I go to School.", "You are stupid."],
|
35 |
+
["I love adventure sports."],
|
36 |
+
]
|
37 |
+
references = [
|
38 |
+
["I go to School.", "You are stupid."],
|
39 |
+
["I love adventure sports."],
|
40 |
+
]
|
41 |
+
metric = load("semf1")
|
42 |
+
results = metric.compute(predictions=predictions, references=references)
|
43 |
+
```
|
44 |
|
45 |
+
It also accepts multiple optional arguments:
|
46 |
+
TODO: List optional arguments
|
47 |
|
48 |
### Inputs
|
49 |
*List all input arguments in the format below*
|
|
|
65 |
*Note any known limitations or biases that the metric has, with links and references if possible.*
|
66 |
|
67 |
## Citation
|
68 |
+
```bibtex
|
69 |
+
@inproceedings{bansal-etal-2022-sem,
|
70 |
+
title = "{SEM}-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale",
|
71 |
+
author = "Bansal, Naman and
|
72 |
+
Akter, Mousumi and
|
73 |
+
Karmaker Santu, Shubhra Kanti",
|
74 |
+
editor = "Goldberg, Yoav and
|
75 |
+
Kozareva, Zornitsa and
|
76 |
+
Zhang, Yue",
|
77 |
+
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
|
78 |
+
month = dec,
|
79 |
+
year = "2022",
|
80 |
+
address = "Abu Dhabi, United Arab Emirates",
|
81 |
+
publisher = "Association for Computational Linguistics",
|
82 |
+
url = "https://aclanthology.org/2022.emnlp-main.49",
|
83 |
+
doi = "10.18653/v1/2022.emnlp-main.49",
|
84 |
+
pages = "780--792",
|
85 |
+
abstract = "Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives. Previous work also published a benchmark dataset for this task by collecting 2,925 alternative narrative pairs from the web and manually annotating 411 different reference summaries by engaging human annotators. In this paper, we exclusively focus on the automated evaluation of the SOS task using the benchmark dataset. More specifically, we first use the popular ROUGE metric from text-summarization literature and conduct a systematic study to evaluate the SOS task. Our experiments discover that ROUGE is not suitable for this novel task and therefore, we propose a new sentence-level precision-recall style automated evaluation metric, called SEM-F1 (Semantic F1). It is inspired by the benefits of the sentence-wise annotation technique using overlap labels reported by the previous work. Our experiments show that the proposed SEM-F1 metric yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.",
|
86 |
+
}
|
87 |
+
```
|
88 |
|
89 |
## Further References
|
90 |
+
- [Paper](https://aclanthology.org/2022.emnlp-main.49/)
|
91 |
+
- [Presentation Slides]()
|
requirements.txt
CHANGED
@@ -1 +1,3 @@
|
|
1 |
-
git+https://github.com/huggingface/evaluate@main
|
|
|
|
|
|
1 |
+
git+https://github.com/huggingface/evaluate@main
|
2 |
+
scikit-learn
|
3 |
+
sentence-transformers
|
semf1.py
CHANGED
@@ -11,58 +11,137 @@
|
|
11 |
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12 |
# See the License for the specific language governing permissions and
|
13 |
# limitations under the License.
|
14 |
-
|
|
|
15 |
|
16 |
-
import evaluate
|
17 |
-
import datasets
|
18 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
19 |
|
20 |
-
# TODO: Add BibTeX citation
|
21 |
_CITATION = """\
|
22 |
-
@
|
23 |
-
title = {
|
24 |
-
|
25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
26 |
}
|
27 |
"""
|
28 |
|
29 |
-
# TODO: Add description of the module here
|
30 |
_DESCRIPTION = """\
|
31 |
-
|
|
|
|
|
32 |
"""
|
33 |
|
34 |
-
|
35 |
-
# TODO: Add description of the arguments of the module here
|
36 |
_KWARGS_DESCRIPTION = """
|
37 |
-
|
|
|
38 |
Args:
|
39 |
-
predictions:
|
40 |
-
|
41 |
-
references: list of reference for each prediction. Each
|
42 |
reference should be a string with tokens separated by spaces.
|
|
|
|
|
|
|
|
|
|
|
43 |
Returns:
|
44 |
accuracy: description of the first score,
|
45 |
another_score: description of the second score,
|
46 |
Examples:
|
47 |
-
Examples should be written in doctest format, and should illustrate how
|
48 |
-
to use the function.
|
49 |
|
50 |
-
>>>
|
51 |
-
>>>
|
52 |
-
|
53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
54 |
"""
|
55 |
|
56 |
-
|
57 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
58 |
|
59 |
|
60 |
@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
|
61 |
class SemF1(evaluate.Metric):
|
62 |
-
|
|
|
|
|
|
|
|
|
63 |
|
64 |
def _info(self):
|
65 |
-
# TODO: Specifies the evaluate.EvaluationModuleInfo object
|
66 |
return evaluate.MetricInfo(
|
67 |
# This is the description that will appear on the modules page.
|
68 |
module_type="metric",
|
@@ -71,25 +150,41 @@ class SemF1(evaluate.Metric):
|
|
71 |
inputs_description=_KWARGS_DESCRIPTION,
|
72 |
# This defines the format of each prediction and reference
|
73 |
features=datasets.Features({
|
74 |
-
'predictions': datasets.Value(
|
75 |
-
'references': datasets.Value(
|
76 |
}),
|
77 |
-
# Homepage of the module for documentation
|
78 |
-
homepage="http://module.homepage",
|
79 |
# Additional links to the codebase or references
|
80 |
-
|
81 |
-
reference_urls=["http://path.to.reference.url/new_module"]
|
82 |
)
|
83 |
|
84 |
-
def
|
85 |
-
|
86 |
-
|
87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
88 |
|
89 |
-
|
90 |
-
"""Returns the scores"""
|
91 |
-
# TODO: Compute the different scores of the module
|
92 |
-
accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
|
93 |
-
return {
|
94 |
-
"accuracy": accuracy,
|
95 |
-
}
|
|
|
11 |
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12 |
# See the License for the specific language governing permissions and
|
13 |
# limitations under the License.
|
14 |
+
# TODO: Add test cases
|
15 |
+
"""SEM-F1 metric"""
|
16 |
|
|
|
|
|
17 |
|
18 |
+
import abc
|
19 |
+
import sys
|
20 |
+
from typing import List, Optional, Tuple
|
21 |
+
|
22 |
+
import datasets
|
23 |
+
import evaluate
|
24 |
+
import numpy as np
|
25 |
+
from numpy.typing import NDArray
|
26 |
+
from sentence_transformers import SentenceTransformer
|
27 |
+
from sklearn.metrics.pairwise import cosine_similarity
|
28 |
|
|
|
29 |
_CITATION = """\
|
30 |
+
@inproceedings{bansal-etal-2022-sem,
|
31 |
+
title = "{SEM}-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale",
|
32 |
+
author = "Bansal, Naman and
|
33 |
+
Akter, Mousumi and
|
34 |
+
Karmaker Santu, Shubhra Kanti",
|
35 |
+
editor = "Goldberg, Yoav and
|
36 |
+
Kozareva, Zornitsa and
|
37 |
+
Zhang, Yue",
|
38 |
+
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
|
39 |
+
month = dec,
|
40 |
+
year = "2022",
|
41 |
+
address = "Abu Dhabi, United Arab Emirates",
|
42 |
+
publisher = "Association for Computational Linguistics",
|
43 |
+
url = "https://aclanthology.org/2022.emnlp-main.49",
|
44 |
+
doi = "10.18653/v1/2022.emnlp-main.49",
|
45 |
+
pages = "780--792",
|
46 |
+
abstract = "Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives. Previous work also published a benchmark dataset for this task by collecting 2,925 alternative narrative pairs from the web and manually annotating 411 different reference summaries by engaging human annotators. In this paper, we exclusively focus on the automated evaluation of the SOS task using the benchmark dataset. More specifically, we first use the popular ROUGE metric from text-summarization literature and conduct a systematic study to evaluate the SOS task. Our experiments discover that ROUGE is not suitable for this novel task and therefore, we propose a new sentence-level precision-recall style automated evaluation metric, called SEM-F1 (Semantic F1). It is inspired by the benefits of the sentence-wise annotation technique using overlap labels reported by the previous work. Our experiments show that the proposed SEM-F1 metric yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.",
|
47 |
}
|
48 |
"""
|
49 |
|
|
|
50 |
_DESCRIPTION = """\
|
51 |
+
SEM-F1 metric leverages the pre-trained contextual embeddings and evaluates the model generated
|
52 |
+
semantic overlap summary with the reference overlap summary. It evaluates the semantic overlap summary at the
|
53 |
+
sentence level and computes precision, recall and F1 scores.
|
54 |
"""
|
55 |
|
|
|
|
|
56 |
_KWARGS_DESCRIPTION = """
|
57 |
+
SEM-F1 compares the system generated overlap summary with ground truth reference overlap.
|
58 |
+
|
59 |
Args:
|
60 |
+
predictions: List[List(str)] - List of predictions where each prediction is a list of sentences.
|
61 |
+
references: List[List(str)] - List of references where each reference is a list of sentences.
|
|
|
62 |
reference should be a string with tokens separated by spaces.
|
63 |
+
model_type: str - Model to use. [pv1, stsb, use]
|
64 |
+
Options:
|
65 |
+
pv1 - paraphrase-distilroberta-base-v1
|
66 |
+
stsb - stsb-roberta-large
|
67 |
+
use - Universal Sentence Encoder
|
68 |
Returns:
|
69 |
accuracy: description of the first score,
|
70 |
another_score: description of the second score,
|
71 |
Examples:
|
|
|
|
|
72 |
|
73 |
+
>>> import evaluate
|
74 |
+
>>> predictions = [
|
75 |
+
["I go to School.", "You are stupid."],
|
76 |
+
["I love adventure sports."],
|
77 |
+
]
|
78 |
+
>>> references = [
|
79 |
+
["I go to School.", "You are stupid."],
|
80 |
+
["I love adventure sports."],
|
81 |
+
]
|
82 |
+
>>> metric = evaluate.load("semf1")
|
83 |
+
>>> results = metric.compute(predictions=predictions, references=references)
|
84 |
+
>>> print([round(v, 2) for v in results["f1"]])
|
85 |
+
[0.77, 0.56]
|
86 |
"""
|
87 |
|
88 |
+
|
89 |
+
class Encoder(metaclass=abc.ABCMeta):
|
90 |
+
@abc.abstractmethod
|
91 |
+
def encode(self, prediction: List[str]) -> NDArray:
|
92 |
+
pass
|
93 |
+
|
94 |
+
|
95 |
+
class USE(Encoder):
|
96 |
+
def __init__(self):
|
97 |
+
pass
|
98 |
+
|
99 |
+
def encode(self, prediction: List[str]) -> NDArray:
|
100 |
+
pass
|
101 |
+
|
102 |
+
|
103 |
+
class SBertEncoder(Encoder):
|
104 |
+
def __init__(self, model_name: str):
|
105 |
+
self.model = SentenceTransformer(model_name)
|
106 |
+
|
107 |
+
def encode(self, prediction: List[str]) -> NDArray:
|
108 |
+
return self.model.encode(prediction)
|
109 |
+
|
110 |
+
|
111 |
+
def _get_encoder(model_name: str):
|
112 |
+
if model_name == "use":
|
113 |
+
return USE()
|
114 |
+
else:
|
115 |
+
return SBertEncoder(model_name)
|
116 |
+
|
117 |
+
|
118 |
+
def _compute_f1(p, r, eps=sys.float_info.epsilon):
|
119 |
+
'''
|
120 |
+
Computes F1 value
|
121 |
+
:param p: Precision Value
|
122 |
+
:param r: Recall Value
|
123 |
+
:return:
|
124 |
+
'''
|
125 |
+
f1 = 2 * p * r / (p + r + eps)
|
126 |
+
return f1
|
127 |
+
|
128 |
+
|
129 |
+
def _compute_cosine_similarity(pred_embeds: NDArray, ref_embeds: NDArray) -> Tuple[float, float]:
|
130 |
+
cosine_scores = cosine_similarity(pred_embeds, ref_embeds)
|
131 |
+
precision_per_sentence_sim = np.max(cosine_scores, axis=-1)
|
132 |
+
recall_per_sentence_sim = np.max(cosine_scores, axis=0)
|
133 |
+
return np.mean(precision_per_sentence_sim).item(), np.mean(recall_per_sentence_sim).item()
|
134 |
|
135 |
|
136 |
@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
|
137 |
class SemF1(evaluate.Metric):
|
138 |
+
_MODEL_TYPE_TO_NAME = {
|
139 |
+
"pv1": "paraphrase-distilroberta-base-v1",
|
140 |
+
"stsb": "stsb-roberta-large",
|
141 |
+
"use": "use",
|
142 |
+
}
|
143 |
|
144 |
def _info(self):
|
|
|
145 |
return evaluate.MetricInfo(
|
146 |
# This is the description that will appear on the modules page.
|
147 |
module_type="metric",
|
|
|
150 |
inputs_description=_KWARGS_DESCRIPTION,
|
151 |
# This defines the format of each prediction and reference
|
152 |
features=datasets.Features({
|
153 |
+
'predictions': datasets.Sequence(datasets.Value("string", id="sequence"), id="predictions"),
|
154 |
+
'references': datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
|
155 |
}),
|
156 |
+
# # Homepage of the module for documentation
|
|
|
157 |
# Additional links to the codebase or references
|
158 |
+
reference_urls=["https://aclanthology.org/2022.emnlp-main.49/"]
|
|
|
159 |
)
|
160 |
|
161 |
+
def _get_model_name(self, model_type: Optional[str] = None) -> str:
|
162 |
+
# TODO: make it work with USE as well
|
163 |
+
if model_type not in self._MODEL_TYPE_TO_NAME.keys():
|
164 |
+
raise ValueError(f"Provide a correct model_type.\n"
|
165 |
+
f"Options: {self._MODEL_TYPE_TO_NAME.keys()}\n"
|
166 |
+
f"Currently provided: {model_type}")
|
167 |
+
|
168 |
+
if model_type is None:
|
169 |
+
model_type = "pv1" # Change it to use
|
170 |
+
|
171 |
+
return self._MODEL_TYPE_TO_NAME[model_type]
|
172 |
+
|
173 |
+
def _compute(self, predictions, references, model_type: Optional[str] = None):
|
174 |
+
model_name = self._get_model_name(model_type)
|
175 |
+
encoder = _get_encoder(model_name)
|
176 |
+
|
177 |
+
precisions = [0] * len(predictions)
|
178 |
+
recalls = [0] * len(predictions)
|
179 |
+
f1_scores = [0] * len(predictions)
|
180 |
+
|
181 |
+
for idx, (preds, refs) in enumerate(zip(predictions, references)):
|
182 |
+
pred_embeddings = encoder.encode(preds)
|
183 |
+
ref_embeddings = encoder.encode(refs)
|
184 |
+
p, r = _compute_cosine_similarity(pred_embeddings, ref_embeddings)
|
185 |
+
f1 = _compute_f1(p, r)
|
186 |
+
precisions[idx] = p
|
187 |
+
recalls[idx] = r
|
188 |
+
f1_scores[idx] = f1
|
189 |
|
190 |
+
return {"precision": precisions, "recall": recalls, "f1": f1_scores}
|
|
|
|
|
|
|
|
|
|
|
|